2024-07-26T07:22:11.248 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-26T07:22:11.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T07:22:11.369 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818695 branch: wip-vshankar-testing-20240725.121331-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/blogbench}} email: vshankar@redhat.com first_in_suite: false job_id: '7818695' kernel: flavor: default kdb: 1 sha1: 3e9255e907100f88049f320ca33d977e29155d12 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240725.121331-debug ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: mds: mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240725.121331-debug sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5532 sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 sleep_before_teardown: 0 subset: 1001/1024 suite: fs suite_branch: wip-vshankar-testing-20240725.121331-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPuSvVSK23f4UBnzdEdCurdQJ1ilQi8qiSag5ZiDmgpAoXnzfRfvDe58CFECxQ+jQ8uXOycQuVUOypO8QeqV21U= smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0ukVig1k+Zilo9puaEbpiUp9XjsTr+FoIz10z2RAZxw0YPMTBlR+AskQxRuzBLXBG3jSnOV9cGrpGpPYIawF8= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvslcFteNQx8StA84ajXYkk+j4vu397uxPS697QNoTgJaKI3Taz6Xb3au/G8mHyV08hDtLxNon5+VVYOtI2Mbo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_04:18:59 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-26T07:22:11.369 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa; will attempt to use it 2024-07-26T07:22:11.370 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks 2024-07-26T07:22:11.370 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-26T07:22:11.372 INFO:teuthology.task.internal:Checking packages... 2024-07-26T07:22:11.394 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528' 2024-07-26T07:22:11.394 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-26T07:22:11.394 INFO:teuthology.packaging:ref: None 2024-07-26T07:22:11.394 INFO:teuthology.packaging:tag: None 2024-07-26T07:22:11.394 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240725.121331-debug 2024-07-26T07:22:11.394 INFO:teuthology.packaging:sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:22:11.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240725.121331-debug 2024-07-26T07:22:11.536 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5320.ge8f88bb1 2024-07-26T07:22:11.538 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-26T07:22:11.544 INFO:teuthology.task.internal:no buildpackages task found 2024-07-26T07:22:11.544 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-26T07:22:11.565 INFO:teuthology.task.internal:Saving configuration 2024-07-26T07:22:11.586 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-26T07:22:11.592 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-26T07:22:11.617 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818695', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 07:17:10.842628', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPuSvVSK23f4UBnzdEdCurdQJ1ilQi8qiSag5ZiDmgpAoXnzfRfvDe58CFECxQ+jQ8uXOycQuVUOypO8QeqV21U='} 2024-07-26T07:22:11.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818695', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 07:17:10.845474', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0ukVig1k+Zilo9puaEbpiUp9XjsTr+FoIz10z2RAZxw0YPMTBlR+AskQxRuzBLXBG3jSnOV9cGrpGpPYIawF8='} 2024-07-26T07:22:11.657 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818695', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 07:17:10.844200', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvslcFteNQx8StA84ajXYkk+j4vu397uxPS697QNoTgJaKI3Taz6Xb3au/G8mHyV08hDtLxNon5+VVYOtI2Mbo='} 2024-07-26T07:22:11.657 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-26T07:22:11.663 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-07-26T07:22:11.663 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-07-26T07:22:11.663 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-07-26T07:22:11.663 INFO:teuthology.run_tasks:Running task console_log... 2024-07-26T07:22:11.750 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4d0d992950>, signals=[15]) 2024-07-26T07:22:11.750 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-26T07:22:11.756 INFO:teuthology.task.internal:Opening connections... 2024-07-26T07:22:11.756 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-07-26T07:22:11.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:22:11.842 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-07-26T07:22:11.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:22:11.920 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-07-26T07:22:11.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:22:12.000 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-26T07:22:12.006 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-07-26T07:22:12.024 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-07-26T07:22:12.024 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-07-26T07:22:12.081 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-07-26T07:22:12.081 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T07:22:12.082 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T07:22:12.083 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-07-26T07:22:12.101 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-07-26T07:22:12.119 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-07-26T07:22:12.119 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-07-26T07:22:12.176 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2024-07-26T07:22:12.176 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="9" 2024-07-26T07:22:12.176 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2024-07-26T07:22:12.176 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2024-07-26T07:22:12.176 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="9" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el9" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:LOGO="fedora-logo-icon" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T07:22:12.177 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T07:22:12.177 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-07-26T07:22:12.201 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-07-26T07:22:12.219 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-07-26T07:22:12.220 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-07-26T07:22:12.278 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-07-26T07:22:12.279 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T07:22:12.279 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-07-26T07:22:12.279 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T07:22:12.279 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T07:22:12.279 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T07:22:12.279 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-07-26T07:22:12.301 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-26T07:22:12.311 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-26T07:22:12.327 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-26T07:22:12.327 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-07-26T07:22:12.330 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-07-26T07:22:12.335 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-07-26T07:22:12.352 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-26T07:22:12.359 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-26T07:22:12.359 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-07-26T07:22:12.386 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-07-26T07:22:12.392 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-07-26T07:22:12.489 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-26T07:22:12.564 INFO:teuthology.run_tasks:Running task kernel... 2024-07-26T07:22:12.579 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'} 2024-07-26T07:22:12.579 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'}, 'mon.b': {'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'}, 'mon.c': {'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'}}, timeout 300 2024-07-26T07:22:12.579 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T07:22:12.580 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T07:22:12.581 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T07:22:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T07:22:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T07:22:12.598 DEBUG:teuthology.task.kernel:sha1 for mon.b is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.599 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T07:22:12.599 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-07-26T07:22:12.599 DEBUG:teuthology.task.kernel:sha1 for mon.a is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.599 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T07:22:12.599 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-07-26T07:22:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T07:22:12.600 DEBUG:teuthology.task.kernel:sha1 for mon.c is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.601 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T07:22:12.601 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-07-26T07:22:12.656 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-437.el9.x86_64 2024-07-26T07:22:12.656 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-437.el9.x86_64 2024-07-26T07:22:12.657 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.657 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T07:22:12.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.658 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.658 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T07:22:12.658 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.663 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-437.el9.x86_64 2024-07-26T07:22:12.663 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.663 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T07:22:12.664 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:22:12.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum upgrade -y linux-firmware 2024-07-26T07:22:12.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum upgrade -y linux-firmware 2024-07-26T07:22:12.811 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum upgrade -y linux-firmware 2024-07-26T07:22:14.905 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-07-26T07:22:15.651 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.2 MB 00:01 2024-07-26T07:22:16.627 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.2 MB 00:02 2024-07-26T07:22:17.444 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-07-26T07:22:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 20 MB 00:03 2024-07-26T07:22:20.529 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 20 MB 00:02 2024-07-26T07:22:23.344 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.5 MB 00:02 2024-07-26T07:22:24.438 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.5 MB 00:00 2024-07-26T07:22:25.053 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 18 kB 00:00 2024-07-26T07:22:26.024 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-07-26T07:22:26.489 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 18 kB 00:00 2024-07-26T07:22:26.621 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.5 MB 00:02 2024-07-26T07:22:27.643 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-07-26T07:22:29.221 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 18 kB 00:01 2024-07-26T07:22:30.108 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 29 MB/s | 22 MB 00:00 2024-07-26T07:22:31.372 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-07-26T07:22:32.956 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-07-26T07:22:33.505 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-07-26T07:22:33.512 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-07-26T07:22:33.512 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-07-26T07:22:33.512 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-07-26T07:22:33.512 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-07-26T07:22:33.512 INFO:teuthology.orchestra.run.smithi181.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T07:22:33.513 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:22:33.513 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-07-26T07:22:33.513 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-07-26T07:22:33.513 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 1 Package 2024-07-26T07:22:33.513 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:22:33.513 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 383 M 2024-07-26T07:22:33.513 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-07-26T07:22:35.084 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-07-26T07:22:35.094 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-07-26T07:22:35.094 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-07-26T07:22:35.094 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-07-26T07:22:35.094 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2024-07-26T07:22:35.095 INFO:teuthology.orchestra.run.smithi115.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T07:22:35.095 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:22:35.095 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-07-26T07:22:35.095 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-07-26T07:22:35.095 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 1 Package 2024-07-26T07:22:35.095 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:22:35.095 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 383 M 2024-07-26T07:22:35.095 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-07-26T07:22:35.422 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-07-26T07:22:37.529 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-07-26T07:22:37.535 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 1 Package 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 383 M 2024-07-26T07:22:37.536 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-07-26T07:22:42.799 INFO:teuthology.orchestra.run.smithi097.stdout:linux-firmware-20240716-145.el9.noarch.rpm 77 MB/s | 383 MB 00:04 2024-07-26T07:22:42.800 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-07-26T07:22:42.800 INFO:teuthology.orchestra.run.smithi097.stdout:Total 73 MB/s | 383 MB 00:05 2024-07-26T07:22:45.868 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-07-26T07:22:45.938 INFO:teuthology.orchestra.run.smithi181.stdout:linux-firmware-20240716-145.el9.noarch.rpm 31 MB/s | 383 MB 00:12 2024-07-26T07:22:45.939 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-07-26T07:22:45.939 INFO:teuthology.orchestra.run.smithi181.stdout:Total 31 MB/s | 383 MB 00:12 2024-07-26T07:22:45.942 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-07-26T07:22:45.942 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-07-26T07:22:45.981 INFO:teuthology.orchestra.run.smithi115.stdout:linux-firmware-20240716-145.el9.noarch.rpm 36 MB/s | 383 MB 00:10 2024-07-26T07:22:45.981 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-07-26T07:22:45.982 INFO:teuthology.orchestra.run.smithi115.stdout:Total 35 MB/s | 383 MB 00:10 2024-07-26T07:22:49.017 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-07-26T07:22:49.058 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-07-26T07:22:49.097 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-07-26T07:22:49.097 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-07-26T07:22:49.154 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-07-26T07:22:49.154 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-07-26T07:22:49.175 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-07-26T07:22:49.176 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-07-26T07:22:52.368 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-07-26T07:22:52.368 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-07-26T07:22:52.402 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-07-26T07:22:52.402 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-07-26T07:22:58.021 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-07-26T07:22:58.672 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T07:22:58.756 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:22:59.632 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:22:59.632 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T07:23:00.343 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:23:00.343 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:23:00.343 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-07-26T07:23:00.344 INFO:teuthology.orchestra.run.smithi097.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T07:23:00.344 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:23:00.344 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-07-26T07:23:00.696 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on mon.a... 2024-07-26T07:23:00.697 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:23:00.846 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T07:23:00.847 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T07:23:00.849 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi097.front.sepia.ceph.com to complete... 2024-07-26T07:23:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-07-26T07:23:01.554 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-07-26T07:23:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T07:23:02.204 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:23:02.273 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T07:23:02.355 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:23:02.950 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:23:02.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T07:23:03.157 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:23:03.157 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T07:23:03.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:23:03.668 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:23:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-07-26T07:23:03.668 INFO:teuthology.orchestra.run.smithi181.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T07:23:03.668 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:23:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-07-26T07:23:03.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T07:23:03.905 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:23:03.905 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2024-07-26T07:23:03.905 INFO:teuthology.orchestra.run.smithi115.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T07:23:03.905 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:23:03.905 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-07-26T07:23:04.073 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on mon.c... 2024-07-26T07:23:04.074 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:23:04.224 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T07:23:04.224 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T07:23:04.226 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi181.front.sepia.ceph.com to complete... 2024-07-26T07:23:04.229 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on mon.b... 2024-07-26T07:23:04.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T07:23:04.378 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T07:23:04.378 DEBUG:teuthology.orchestra.run.smithi115:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T07:23:04.381 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi115.front.sepia.ceph.com to complete... 2024-07-26T07:23:09.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26 07:23:09 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T07:23:09.245 INFO:teuthology.orchestra.run.smithi097.stderr:FINISHED --2024-07-26 07:23:09-- 2024-07-26T07:23:09.245 INFO:teuthology.orchestra.run.smithi097.stderr:Total wall clock time: 8.4s 2024-07-26T07:23:09.245 INFO:teuthology.orchestra.run.smithi097.stderr:Downloaded: 1 files, 342M in 8.1s (42.1 MB/s) 2024-07-26T07:23:09.248 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on mon.a... 2024-07-26T07:23:09.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T07:23:11.305 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying... ######################################## 2024-07-26T07:23:11.348 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing... ######################################## 2024-07-26T07:23:11.397 INFO:teuthology.orchestra.run.smithi097.stdout:Updating / installing... 2024-07-26T07:23:13.719 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26 07:23:13 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T07:23:13.719 INFO:teuthology.orchestra.run.smithi181.stderr:FINISHED --2024-07-26 07:23:13-- 2024-07-26T07:23:13.720 INFO:teuthology.orchestra.run.smithi181.stderr:Total wall clock time: 9.5s 2024-07-26T07:23:13.720 INFO:teuthology.orchestra.run.smithi181.stderr:Downloaded: 1 files, 342M in 9.1s (37.4 MB/s) 2024-07-26T07:23:13.723 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on mon.c... 2024-07-26T07:23:13.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T07:23:15.774 INFO:teuthology.orchestra.run.smithi181.stdout:Verifying... ######################################## 2024-07-26T07:23:15.839 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing... ######################################## 2024-07-26T07:23:15.889 INFO:teuthology.orchestra.run.smithi181.stdout:Updating / installing... 2024-07-26T07:23:19.390 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T07:23:23.804 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T07:23:24.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26 07:23:24 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T07:23:24.295 INFO:teuthology.orchestra.run.smithi115.stderr:FINISHED --2024-07-26 07:23:24-- 2024-07-26T07:23:24.295 INFO:teuthology.orchestra.run.smithi115.stderr:Total wall clock time: 20s 2024-07-26T07:23:24.295 INFO:teuthology.orchestra.run.smithi115.stderr:Downloaded: 1 files, 342M in 20s (17.5 MB/s) 2024-07-26T07:23:24.298 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on mon.b... 2024-07-26T07:23:24.299 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T07:23:26.351 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying... ######################################## 2024-07-26T07:23:26.486 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing... ######################################## 2024-07-26T07:23:26.537 INFO:teuthology.orchestra.run.smithi115.stdout:Updating / installing... 2024-07-26T07:23:34.506 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T07:23:59.150 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T07:23:59.150 DEBUG:teuthology.orchestra.run.smithi097:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T07:23:59.227 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T07:23:59.228 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T07:23:59.229 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T07:23:59.230 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T07:23:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T07:23:59.232 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T07:23:59.233 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T07:23:59.234 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T07:23:59.235 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T07:23:59.236 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T07:23:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T07:23:59.238 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T07:23:59.239 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T07:23:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T07:23:59.241 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T07:23:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T07:23:59.243 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T07:23:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T07:23:59.245 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T07:23:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T07:23:59.247 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T07:23:59.248 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T07:23:59.249 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T07:23:59.250 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T07:23:59.251 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T07:23:59.252 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T07:23:59.253 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T07:23:59.254 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T07:23:59.255 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T07:23:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T07:23:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T07:23:59.258 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T07:23:59.259 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T07:23:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T07:23:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T07:23:59.262 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T07:23:59.263 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T07:23:59.264 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T07:23:59.265 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T07:23:59.266 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T07:23:59.267 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T07:23:59.268 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T07:23:59.269 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T07:23:59.270 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T07:23:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T07:23:59.272 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T07:23:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T07:23:59.274 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T07:23:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T07:23:59.276 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T07:23:59.277 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T07:23:59.278 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T07:23:59.279 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T07:23:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T07:23:59.281 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T07:23:59.282 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T07:23:59.283 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T07:23:59.284 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T07:23:59.285 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T07:23:59.286 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T07:23:59.287 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T07:23:59.288 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T07:23:59.289 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T07:23:59.290 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T07:23:59.291 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T07:23:59.292 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T07:23:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T07:23:59.294 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T07:23:59.295 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T07:23:59.296 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T07:23:59.297 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T07:23:59.297 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T07:23:59.297 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T07:23:59.297 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T07:23:59.297 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T07:23:59.297 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T07:23:59.297 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T07:23:59.297 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T07:23:59.298 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T07:23:59.299 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T07:23:59.300 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T07:23:59.301 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T07:23:59.302 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T07:23:59.303 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T07:23:59.304 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T07:23:59.305 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T07:23:59.306 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T07:23:59.307 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T07:23:59.308 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T07:23:59.309 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T07:23:59.310 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T07:23:59.311 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T07:23:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T07:23:59.313 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T07:23:59.314 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T07:23:59.315 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T07:23:59.316 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T07:23:59.317 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T07:23:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T07:23:59.319 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T07:23:59.320 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T07:23:59.321 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T07:23:59.322 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T07:23:59.323 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T07:23:59.324 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T07:23:59.325 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T07:23:59.326 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T07:23:59.327 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T07:23:59.328 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T07:23:59.329 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T07:23:59.330 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T07:23:59.331 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T07:23:59.332 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T07:23:59.333 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T07:23:59.334 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T07:23:59.335 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T07:23:59.336 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T07:23:59.337 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T07:23:59.338 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T07:23:59.339 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T07:23:59.339 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T07:23:59.339 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T07:23:59.339 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T07:23:59.342 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T07:23:59.342 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T07:23:59.342 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T07:23:59.343 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T07:23:59.344 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T07:23:59.345 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T07:23:59.346 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T07:23:59.347 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T07:23:59.348 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T07:23:59.349 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T07:23:59.350 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T07:23:59.351 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T07:23:59.352 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T07:23:59.353 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T07:23:59.354 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T07:23:59.355 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T07:23:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T07:23:59.357 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T07:23:59.358 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T07:23:59.359 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T07:23:59.360 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T07:23:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T07:23:59.362 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T07:23:59.363 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T07:23:59.364 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T07:23:59.365 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T07:23:59.366 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T07:23:59.367 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T07:23:59.368 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T07:23:59.369 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T07:23:59.370 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T07:23:59.371 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T07:23:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T07:23:59.373 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T07:23:59.374 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T07:23:59.375 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T07:23:59.376 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T07:23:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T07:23:59.378 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T07:23:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T07:23:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T07:23:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T07:23:59.382 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T07:23:59.383 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T07:23:59.383 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T07:23:59.383 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T07:23:59.387 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T07:23:59.387 DEBUG:teuthology.orchestra.run.smithi097:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T07:23:59.409 INFO:teuthology.orchestra.run.smithi097.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T07:23:59.409 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T07:23:59.409 INFO:teuthology.orchestra.run.smithi097.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T07:23:59.409 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-07-26T07:23:59.409 INFO:teuthology.orchestra.run.smithi097.stdout:for file in vmlinuz System.map config; do 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout: fi 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout:done 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T07:23:59.410 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-07-26T07:23:59.411 INFO:teuthology.orchestra.run.smithi097.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T07:23:59.411 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T07:23:59.411 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T07:23:59.411 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-07-26T07:23:59.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-07-26T07:23:59.505 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-07-26T07:23:59.505 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-07-26T07:23:59.505 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2024-07-26T07:23:59.505 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2024-07-26T07:23:59.505 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-07-26T07:23:59.505 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T07:23:59.505 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T07:23:59.506 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T07:23:59.507 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T07:23:59.507 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-07-26T07:23:59.507 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:23:59.507 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T07:23:59.508 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T07:23:59.508 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 6.10.0-g3e9255e90710 2024-07-26T07:23:59.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T07:24:00.117 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-07-26T07:24:01.466 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T07:24:01.496 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-07-26T07:24:01.499 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-07-26T07:24:01.513 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.KcdayA1XSw 2024-07-26T07:24:01.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KcdayA1XSw 2024-07-26T07:24:01.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.KcdayA1XSw 2024-07-26T07:24:01.719 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.KcdayA1XSw is 6KB 2024-07-26T07:24:01.738 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.KcdayA1XSw 2024-07-26T07:24:01.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T07:24:01.814 INFO:teuthology.orchestra.run.smithi097.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T07:24:01.873 INFO:teuthology.orchestra.run.smithi097.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T07:24:01.873 INFO:teuthology.orchestra.run.smithi097.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T07:24:01.873 INFO:teuthology.orchestra.run.smithi097.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T07:24:01.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T07:24:02.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-07-26T07:24:03.506 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T07:24:03.506 DEBUG:teuthology.orchestra.run.smithi181:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T07:24:03.583 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T07:24:03.584 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T07:24:03.585 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T07:24:03.586 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T07:24:03.587 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T07:24:03.588 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T07:24:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T07:24:03.590 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T07:24:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T07:24:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T07:24:03.593 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T07:24:03.594 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T07:24:03.595 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T07:24:03.596 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T07:24:03.597 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T07:24:03.598 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T07:24:03.599 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T07:24:03.600 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T07:24:03.601 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T07:24:03.602 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T07:24:03.603 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T07:24:03.604 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T07:24:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T07:24:03.606 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T07:24:03.607 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T07:24:03.608 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T07:24:03.609 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T07:24:03.610 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T07:24:03.611 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T07:24:03.612 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T07:24:03.613 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T07:24:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T07:24:03.615 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T07:24:03.616 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T07:24:03.617 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T07:24:03.618 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T07:24:03.619 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T07:24:03.620 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T07:24:03.621 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T07:24:03.622 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T07:24:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T07:24:03.624 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T07:24:03.625 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T07:24:03.626 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T07:24:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T07:24:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T07:24:03.629 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T07:24:03.630 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T07:24:03.631 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T07:24:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T07:24:03.633 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T07:24:03.634 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T07:24:03.635 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T07:24:03.636 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T07:24:03.637 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T07:24:03.638 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T07:24:03.639 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T07:24:03.640 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T07:24:03.641 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T07:24:03.642 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T07:24:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T07:24:03.644 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T07:24:03.645 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T07:24:03.646 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T07:24:03.647 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T07:24:03.648 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T07:24:03.649 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T07:24:03.650 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T07:24:03.651 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T07:24:03.652 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T07:24:03.653 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T07:24:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T07:24:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T07:24:03.656 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T07:24:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T07:24:03.658 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T07:24:03.659 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T07:24:03.660 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T07:24:03.661 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T07:24:03.662 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T07:24:03.663 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T07:24:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T07:24:03.665 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T07:24:03.666 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T07:24:03.667 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T07:24:03.668 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T07:24:03.669 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T07:24:03.670 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T07:24:03.671 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T07:24:03.672 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T07:24:03.673 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T07:24:03.674 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T07:24:03.675 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T07:24:03.676 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T07:24:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T07:24:03.678 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T07:24:03.679 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T07:24:03.680 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T07:24:03.681 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T07:24:03.682 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T07:24:03.683 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T07:24:03.684 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T07:24:03.685 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T07:24:03.686 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T07:24:03.687 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T07:24:03.688 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T07:24:03.689 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T07:24:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T07:24:03.691 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T07:24:03.692 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T07:24:03.693 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T07:24:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T07:24:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T07:24:03.696 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T07:24:03.697 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T07:24:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T07:24:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T07:24:03.700 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T07:24:03.701 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T07:24:03.702 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T07:24:03.703 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T07:24:03.704 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T07:24:03.705 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T07:24:03.706 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T07:24:03.707 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T07:24:03.708 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T07:24:03.709 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T07:24:03.710 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T07:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T07:24:03.712 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T07:24:03.713 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T07:24:03.714 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T07:24:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T07:24:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T07:24:03.717 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T07:24:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T07:24:03.719 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T07:24:03.720 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T07:24:03.721 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T07:24:03.722 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T07:24:03.723 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T07:24:03.724 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T07:24:03.725 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T07:24:03.726 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T07:24:03.727 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T07:24:03.728 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T07:24:03.729 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T07:24:03.729 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T07:24:03.729 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T07:24:03.729 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T07:24:03.732 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T07:24:03.732 DEBUG:teuthology.orchestra.run.smithi181:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout:fi 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout:for file in vmlinuz System.map config; do 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout: fi 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout:done 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T07:24:03.757 INFO:teuthology.orchestra.run.smithi181.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout:fi 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout:fi 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T07:24:03.758 INFO:teuthology.orchestra.run.smithi181.stdout:fi 2024-07-26T07:24:03.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-07-26T07:24:03.834 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-07-26T07:24:03.834 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-07-26T07:24:03.834 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-07-26T07:24:03.834 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 68.el9 2024-07-26T07:24:03.834 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-07-26T07:24:03.834 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T07:24:03.834 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8264254 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T07:24:03.835 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T07:24:03.836 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T07:24:03.836 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-07-26T07:24:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:24:03.836 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T07:24:03.837 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T07:24:03.837 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 6.10.0-g3e9255e90710 2024-07-26T07:24:03.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T07:24:04.433 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-07-26T07:24:05.829 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T07:24:05.859 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-07-26T07:24:05.862 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-07-26T07:24:05.879 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.ALUQtClCEQ 2024-07-26T07:24:05.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ALUQtClCEQ 2024-07-26T07:24:05.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.ALUQtClCEQ 2024-07-26T07:24:06.088 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.ALUQtClCEQ is 6KB 2024-07-26T07:24:06.108 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.ALUQtClCEQ 2024-07-26T07:24:06.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T07:24:06.246 INFO:teuthology.orchestra.run.smithi181.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T07:24:06.246 INFO:teuthology.orchestra.run.smithi181.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T07:24:06.246 INFO:teuthology.orchestra.run.smithi181.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T07:24:06.246 INFO:teuthology.orchestra.run.smithi181.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T07:24:06.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T07:24:06.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-07-26T07:24:14.152 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T07:24:14.153 DEBUG:teuthology.orchestra.run.smithi115:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T07:24:14.256 INFO:teuthology.orchestra.run.smithi115.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T07:24:14.257 INFO:teuthology.orchestra.run.smithi115.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T07:24:14.257 INFO:teuthology.orchestra.run.smithi115.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T07:24:14.257 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T07:24:14.257 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T07:24:14.257 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T07:24:14.257 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T07:24:14.258 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T07:24:14.258 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T07:24:14.258 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T07:24:14.258 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T07:24:14.258 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T07:24:14.258 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T07:24:14.259 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T07:24:14.259 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T07:24:14.259 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T07:24:14.259 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T07:24:14.259 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T07:24:14.259 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T07:24:14.260 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T07:24:14.260 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T07:24:14.260 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T07:24:14.260 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T07:24:14.260 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T07:24:14.260 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T07:24:14.261 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T07:24:14.261 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T07:24:14.261 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T07:24:14.261 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T07:24:14.261 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T07:24:14.261 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T07:24:14.262 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T07:24:14.262 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T07:24:14.262 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T07:24:14.262 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T07:24:14.262 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T07:24:14.262 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T07:24:14.263 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T07:24:14.263 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T07:24:14.263 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T07:24:14.263 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T07:24:14.263 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T07:24:14.263 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T07:24:14.264 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T07:24:14.264 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T07:24:14.264 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T07:24:14.264 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T07:24:14.264 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T07:24:14.264 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T07:24:14.265 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T07:24:14.265 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T07:24:14.265 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T07:24:14.265 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T07:24:14.265 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T07:24:14.265 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T07:24:14.266 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T07:24:14.266 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T07:24:14.266 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T07:24:14.266 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T07:24:14.266 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T07:24:14.267 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T07:24:14.267 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T07:24:14.267 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T07:24:14.267 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T07:24:14.267 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T07:24:14.267 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T07:24:14.268 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T07:24:14.268 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T07:24:14.268 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T07:24:14.268 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T07:24:14.268 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T07:24:14.268 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T07:24:14.268 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T07:24:14.269 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T07:24:14.269 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T07:24:14.269 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T07:24:14.269 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T07:24:14.269 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T07:24:14.270 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T07:24:14.270 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T07:24:14.270 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T07:24:14.270 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T07:24:14.270 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T07:24:14.271 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T07:24:14.271 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T07:24:14.271 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T07:24:14.271 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T07:24:14.271 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T07:24:14.272 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T07:24:14.272 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T07:24:14.272 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T07:24:14.272 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T07:24:14.273 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T07:24:14.273 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T07:24:14.273 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T07:24:14.273 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T07:24:14.273 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T07:24:14.274 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T07:24:14.274 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T07:24:14.274 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T07:24:14.274 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T07:24:14.275 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T07:24:14.275 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T07:24:14.275 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T07:24:14.275 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T07:24:14.275 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T07:24:14.276 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T07:24:14.276 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T07:24:14.276 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T07:24:14.276 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T07:24:14.277 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T07:24:14.277 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T07:24:14.277 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T07:24:14.277 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T07:24:14.277 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T07:24:14.278 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T07:24:14.278 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T07:24:14.278 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T07:24:14.278 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T07:24:14.279 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T07:24:14.279 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T07:24:14.279 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T07:24:14.279 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T07:24:14.279 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T07:24:14.280 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T07:24:14.280 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T07:24:14.280 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T07:24:14.280 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T07:24:14.281 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T07:24:14.281 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T07:24:14.281 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T07:24:14.281 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T07:24:14.281 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T07:24:14.282 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T07:24:14.282 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T07:24:14.282 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T07:24:14.282 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T07:24:14.282 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T07:24:14.283 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T07:24:14.283 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T07:24:14.283 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T07:24:14.283 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T07:24:14.284 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T07:24:14.284 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T07:24:14.284 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T07:24:14.284 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T07:24:14.284 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T07:24:14.285 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T07:24:14.285 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T07:24:14.285 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T07:24:14.285 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T07:24:14.286 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T07:24:14.286 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T07:24:14.286 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T07:24:14.286 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T07:24:14.286 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T07:24:14.287 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T07:24:14.287 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T07:24:14.287 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T07:24:14.287 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T07:24:14.288 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T07:24:14.288 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T07:24:14.288 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T07:24:14.288 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T07:24:14.288 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T07:24:14.289 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T07:24:14.289 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T07:24:14.289 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T07:24:14.289 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T07:24:14.289 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T07:24:14.290 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T07:24:14.290 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T07:24:14.290 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T07:24:14.290 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T07:24:14.290 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T07:24:14.290 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T07:24:14.291 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T07:24:14.291 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T07:24:14.291 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T07:24:14.291 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T07:24:14.291 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T07:24:14.291 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T07:24:14.291 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T07:24:14.292 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T07:24:14.292 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T07:24:14.292 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T07:24:14.292 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T07:24:14.292 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T07:24:14.292 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T07:24:14.293 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T07:24:14.293 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T07:24:14.293 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T07:24:14.293 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T07:24:14.293 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T07:24:14.293 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T07:24:14.293 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T07:24:14.294 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T07:24:14.294 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T07:24:14.294 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T07:24:14.294 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T07:24:14.294 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T07:24:14.294 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T07:24:14.295 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T07:24:14.295 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T07:24:14.295 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T07:24:14.295 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T07:24:14.295 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T07:24:14.295 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T07:24:14.295 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T07:24:14.296 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T07:24:14.296 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T07:24:14.296 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T07:24:14.296 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T07:24:14.296 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T07:24:14.297 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T07:24:14.297 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T07:24:14.297 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T07:24:14.297 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T07:24:14.297 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T07:24:14.297 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T07:24:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T07:24:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T07:24:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T07:24:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T07:24:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T07:24:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T07:24:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T07:24:14.298 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T07:24:14.299 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T07:24:14.300 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T07:24:14.301 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T07:24:14.302 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T07:24:14.303 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T07:24:14.304 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T07:24:14.305 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T07:24:14.306 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T07:24:14.307 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T07:24:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T07:24:14.309 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T07:24:14.310 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T07:24:14.311 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T07:24:14.312 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T07:24:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T07:24:14.314 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T07:24:14.315 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T07:24:14.316 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T07:24:14.317 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T07:24:14.318 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T07:24:14.319 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T07:24:14.320 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T07:24:14.321 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T07:24:14.322 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T07:24:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T07:24:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T07:24:14.325 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T07:24:14.326 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T07:24:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T07:24:14.328 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T07:24:14.329 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T07:24:14.330 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T07:24:14.331 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T07:24:14.332 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T07:24:14.333 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T07:24:14.334 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T07:24:14.335 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T07:24:14.336 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T07:24:14.337 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T07:24:14.338 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T07:24:14.339 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T07:24:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T07:24:14.341 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T07:24:14.342 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T07:24:14.343 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T07:24:14.344 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T07:24:14.345 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T07:24:14.346 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T07:24:14.347 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T07:24:14.348 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T07:24:14.349 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T07:24:14.350 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T07:24:14.351 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T07:24:14.352 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T07:24:14.353 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T07:24:14.354 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T07:24:14.355 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T07:24:14.356 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T07:24:14.357 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T07:24:14.358 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T07:24:14.359 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T07:24:14.360 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T07:24:14.361 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T07:24:14.362 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T07:24:14.363 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T07:24:14.364 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T07:24:14.365 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T07:24:14.366 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T07:24:14.367 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T07:24:14.368 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T07:24:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T07:24:14.370 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T07:24:14.371 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T07:24:14.372 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T07:24:14.373 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T07:24:14.374 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T07:24:14.375 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T07:24:14.376 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T07:24:14.377 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T07:24:14.378 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T07:24:14.379 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T07:24:14.380 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T07:24:14.381 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T07:24:14.382 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T07:24:14.383 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T07:24:14.384 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T07:24:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T07:24:14.386 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T07:24:14.387 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T07:24:14.388 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T07:24:14.389 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T07:24:14.390 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T07:24:14.391 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T07:24:14.392 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T07:24:14.393 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T07:24:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T07:24:14.395 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T07:24:14.396 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T07:24:14.397 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T07:24:14.398 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T07:24:14.399 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T07:24:14.400 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T07:24:14.401 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T07:24:14.402 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T07:24:14.403 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T07:24:14.404 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T07:24:14.405 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T07:24:14.406 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T07:24:14.407 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T07:24:14.408 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T07:24:14.409 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T07:24:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T07:24:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T07:24:14.412 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T07:24:14.413 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T07:24:14.414 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T07:24:14.415 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T07:24:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T07:24:14.417 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T07:24:14.418 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T07:24:14.419 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T07:24:14.420 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T07:24:14.421 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T07:24:14.422 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T07:24:14.423 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T07:24:14.424 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T07:24:14.425 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T07:24:14.426 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T07:24:14.427 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T07:24:14.428 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T07:24:14.429 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T07:24:14.429 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T07:24:14.429 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T07:24:14.429 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T07:24:14.429 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T07:24:14.429 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T07:24:14.429 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T07:24:14.429 INFO:teuthology.orchestra.run.smithi115.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T07:24:14.433 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T07:24:14.433 DEBUG:teuthology.orchestra.run.smithi115:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T07:24:14.458 INFO:teuthology.orchestra.run.smithi115.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T07:24:14.458 INFO:teuthology.orchestra.run.smithi115.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T07:24:14.458 INFO:teuthology.orchestra.run.smithi115.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T07:24:14.458 INFO:teuthology.orchestra.run.smithi115.stdout:fi 2024-07-26T07:24:14.458 INFO:teuthology.orchestra.run.smithi115.stdout:for file in vmlinuz System.map config; do 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout: fi 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout:done 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout:fi 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout:fi 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T07:24:14.459 INFO:teuthology.orchestra.run.smithi115.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T07:24:14.460 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T07:24:14.460 INFO:teuthology.orchestra.run.smithi115.stdout:fi 2024-07-26T07:24:14.460 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2024-07-26T07:24:14.545 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2024-07-26T07:24:14.545 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2024-07-26T07:24:14.545 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.06 2024-07-26T07:24:14.545 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 68.el9 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Group : Unspecified 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 8264254 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : builder@centos.org 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T07:24:14.546 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2024-07-26T07:24:14.547 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2024-07-26T07:24:14.547 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:24:14.547 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T07:24:14.547 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T07:24:14.547 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T07:24:14.547 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2024-07-26T07:24:14.547 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:24:14.547 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T07:24:14.548 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T07:24:14.548 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 6.10.0-g3e9255e90710 2024-07-26T07:24:14.548 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T07:24:15.089 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2024-07-26T07:24:16.448 INFO:teuthology.orchestra.run.smithi115.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T07:24:16.485 INFO:teuthology.orchestra.run.smithi115.stderr:done 2024-07-26T07:24:16.488 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-07-26T07:24:16.503 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.3wkjTxOoQG 2024-07-26T07:24:16.503 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3wkjTxOoQG 2024-07-26T07:24:16.569 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.3wkjTxOoQG 2024-07-26T07:24:16.706 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.3wkjTxOoQG is 6KB 2024-07-26T07:24:16.754 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.3wkjTxOoQG 2024-07-26T07:24:16.769 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T07:24:16.830 INFO:teuthology.orchestra.run.smithi115.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T07:24:16.830 INFO:teuthology.orchestra.run.smithi115.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T07:24:16.830 INFO:teuthology.orchestra.run.smithi115.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T07:24:16.830 INFO:teuthology.orchestra.run.smithi115.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T07:24:16.831 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T07:24:16.980 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2024-07-26T07:24:32.034 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-07-26T07:24:32.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-07-26T07:24:32.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:24:36.390 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-07-26T07:24:36.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-07-26T07:24:36.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:24:46.992 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-07-26T07:24:46.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-07-26T07:24:46.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:24:50.527 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-07-26T07:24:59.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-07-26T07:24:59.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:25:02.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-07-26T07:25:05.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-07-26T07:25:11.007 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-07-26T07:25:14.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-07-26T07:25:14.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:25:14.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-07-26T07:25:14.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:25:17.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-07-26T07:25:20.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-07-26T07:25:20.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:25:29.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-07-26T07:25:29.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:25:38.399 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-07-26T07:25:46.460 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-07-26T07:25:47.004 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-07-26T07:25:47.004 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.10.0-g3e9255e90710"... 2024-07-26T07:25:47.004 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-07-26T07:25:47.046 INFO:teuthology.orchestra.run.smithi097.stdout:6.10.0-g3e9255e90710 2024-07-26T07:25:47.047 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T07:25:47.047 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T07:25:47.047 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T07:25:47.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-07-26T07:25:48.048 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-07-26T07:25:48.048 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-07-26T07:25:48.170 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2024-07-26T07:25:48.227 DEBUG:teuthology.parallel:result is None 2024-07-26T07:25:50.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-07-26T07:25:50.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:25:53.337 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-07-26T07:25:53.924 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-07-26T07:25:53.924 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "6.10.0-g3e9255e90710"... 2024-07-26T07:25:53.925 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-07-26T07:25:53.966 INFO:teuthology.orchestra.run.smithi181.stdout:6.10.0-g3e9255e90710 2024-07-26T07:25:53.966 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T07:25:53.967 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T07:25:53.967 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T07:25:54.968 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-07-26T07:25:54.968 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-07-26T07:25:55.077 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-07-26T07:25:55.116 DEBUG:teuthology.parallel:result is None 2024-07-26T07:26:02.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-07-26T07:26:02.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:26:05.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-07-26T07:26:23.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-07-26T07:26:23.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:26:24.338 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-07-26T07:26:24.936 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-07-26T07:26:24.936 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.10.0-g3e9255e90710"... 2024-07-26T07:26:24.937 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-07-26T07:26:24.979 INFO:teuthology.orchestra.run.smithi115.stdout:6.10.0-g3e9255e90710 2024-07-26T07:26:24.979 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T07:26:24.979 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T07:26:24.980 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T07:26:25.980 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-07-26T07:26:25.980 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-07-26T07:26:26.185 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2024-07-26T07:26:26.231 DEBUG:teuthology.parallel:result is None 2024-07-26T07:26:26.232 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-26T07:26:26.239 INFO:teuthology.task.internal:Creating test directory... 2024-07-26T07:26:26.239 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T07:26:26.248 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T07:26:26.252 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T07:26:26.313 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-26T07:26:26.320 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-26T07:26:26.329 INFO:teuthology.task.internal:Creating archive directory... 2024-07-26T07:26:26.330 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T07:26:26.335 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T07:26:26.340 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T07:26:26.437 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-26T07:26:26.444 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-26T07:26:26.444 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T07:26:26.452 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T07:26:26.457 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T07:26:26.536 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T07:26:26.537 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T07:26:26.552 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T07:26:26.567 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T07:26:26.567 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T07:26:26.582 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T07:26:26.587 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-26T07:26:26.595 INFO:teuthology.task.internal:Configuring sudo... 2024-07-26T07:26:26.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T07:26:26.600 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T07:26:26.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T07:26:26.691 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-26T07:26:26.702 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-26T07:26:26.702 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T07:26:26.710 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T07:26:26.717 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T07:26:26.773 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T07:26:26.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T07:26:26.953 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T07:26:27.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T07:26:27.120 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:26:27.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T07:26:27.201 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T07:26:27.290 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T07:26:27.386 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T07:26:27.468 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T07:26:27.562 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:26:27.562 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T07:26:27.640 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T07:26:27.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T07:26:27.823 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T07:26:27.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T07:26:27.994 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:26:27.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T07:26:28.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-07-26T07:26:28.085 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-07-26T07:26:28.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-07-26T07:26:28.199 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T07:26:28.202 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T07:26:28.238 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T07:26:28.579 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-26T07:26:28.588 INFO:teuthology.task.internal:Starting timer... 2024-07-26T07:26:28.588 INFO:teuthology.run_tasks:Running task pcp... 2024-07-26T07:26:28.603 INFO:teuthology.run_tasks:Running task selinux... 2024-07-26T07:26:28.614 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-07-26T07:26:28.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-07-26T07:26:28.752 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-07-26T07:26:28.757 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2024-07-26T07:26:28.870 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2024-07-26T07:26:28.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-07-26T07:26:29.022 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-07-26T07:26:29.026 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-26T07:26:29.027 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-07-26T07:26:29.099 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-07-26T07:26:29.100 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2024-07-26T07:26:29.173 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2024-07-26T07:26:29.173 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-07-26T07:26:29.229 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-07-26T07:26:29.229 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi115.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-07-26T07:26:29.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T07:26:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T07:26:29.304 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T07:26:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T07:26:29.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T07:26:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T07:26:29.458 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-26T07:26:29.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-07-26T07:26:29.538 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2024-07-26T07:26:29.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-07-26T07:26:29.681 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-26T07:26:29.691 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-07-26T07:26:29.695 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-07-26T07:26:29.811 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-26T07:26:29.825 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-26T07:26:30.368 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi115.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-26T07:33:03.726 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-07-26T07:33:03.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-07-26T07:33:03.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:33:03.864 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-07-26T07:33:04.009 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-07-26T07:33:04.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-07-26T07:33:04.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:33:04.122 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-07-26T07:33:04.279 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-07-26T07:33:04.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-07-26T07:33:04.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T07:33:04.394 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-07-26T07:33:04.555 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-07-26T07:33:04.555 INFO:teuthology.run_tasks:Running task clock... 2024-07-26T07:33:04.566 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-26T07:33:04.566 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T07:33:04.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T07:33:04.575 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T07:33:04.575 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T07:33:04.583 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T07:33:04.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T07:33:04.662 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T07:33:04.668 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T07:33:04.671 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T07:33:04.705 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T07:33:04.707 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T07:33:04.712 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T07:33:04.774 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-07-26T07:33:04.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-07-26T07:33:04.786 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2024-07-26T07:33:04.806 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-07-26T07:33:04.812 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-07-26T07:33:04.818 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2024-07-26T07:33:04.847 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T07:33:04.853 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T07:33:04.858 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T07:33:04.888 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T07:33:04.894 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T07:33:04.898 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T07:33:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-07-26T07:33:05.023 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T07:33:05.023 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-07-26T07:33:05.023 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.024 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.024 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.024 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.032 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-07-26T07:33:05.042 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-07-26T07:33:05.042 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T07:33:05.042 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-07-26T07:33:05.042 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.043 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.043 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.043 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.049 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T07:33:05.049 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-07-26T07:33:05.049 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.049 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.049 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.049 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T07:33:05.050 INFO:teuthology.run_tasks:Running task install... 2024-07-26T07:33:05.062 DEBUG:teuthology.task.install:project ceph 2024-07-26T07:33:05.063 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'}} 2024-07-26T07:33:05.063 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'} 2024-07-26T07:33:05.063 INFO:teuthology.task.install:Using flavor: default 2024-07-26T07:33:05.071 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-07-26T07:33:05.071 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-07-26T07:33:05.071 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T07:33:05.072 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:33:05.074 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T07:33:05.075 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:33:05.075 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T07:33:05.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:33:05.238 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T07:33:05.238 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T07:33:05.242 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T07:33:05.242 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T07:33:05.243 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T07:33:05.243 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T07:33:05.355 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T07:33:05.356 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:33:05.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T07:33:05.366 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T07:33:05.366 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:33:05.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T07:33:05.383 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T07:33:05.383 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:33:05.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T07:33:05.447 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T07:33:05.447 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T07:33:05.455 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T07:33:05.455 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T07:33:05.471 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T07:33:05.471 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T07:33:05.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T07:33:05.582 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T07:33:05.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T07:33:05.682 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T07:33:05.691 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T07:33:05.709 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T07:33:05.761 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-07-26T07:33:05.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-07-26T07:33:05.769 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2024-07-26T07:33:05.773 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2024-07-26T07:33:05.786 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-07-26T07:33:05.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-07-26T07:33:06.080 INFO:teuthology.orchestra.run.smithi097.stdout:45 files removed 2024-07-26T07:33:06.101 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-07-26T07:33:06.101 INFO:teuthology.orchestra.run.smithi115.stdout:45 files removed 2024-07-26T07:33:06.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T07:33:06.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T07:33:06.140 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T07:33:06.943 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 220 kB/s | 83 kB 00:00 2024-07-26T07:33:06.944 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 221 kB/s | 83 kB 00:00 2024-07-26T07:33:06.952 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 209 kB/s | 83 kB 00:00 2024-07-26T07:33:07.411 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 64 kB/s | 24 kB 00:00 2024-07-26T07:33:07.411 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 66 kB/s | 24 kB 00:00 2024-07-26T07:33:07.419 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 63 kB/s | 24 kB 00:00 2024-07-26T07:33:07.845 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-07-26T07:33:07.861 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-07-26T07:33:07.895 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-07-26T07:33:08.910 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.2 MB 00:00 2024-07-26T07:33:09.343 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.2 MB 00:01 2024-07-26T07:33:12.075 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.2 MB 00:04 2024-07-26T07:33:12.890 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 20 MB 00:02 2024-07-26T07:33:12.969 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 20 MB 00:02 2024-07-26T07:33:16.143 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 20 MB 00:02 2024-07-26T07:33:17.770 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.5 MB 00:00 2024-07-26T07:33:18.561 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-07-26T07:33:19.629 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 18 kB 00:00 2024-07-26T07:33:20.154 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 18 kB 00:00 2024-07-26T07:33:20.762 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 24 MB/s | 22 MB 00:00 2024-07-26T07:33:21.375 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-07-26T07:33:22.601 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-07-26T07:33:25.335 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 18 kB 00:01 2024-07-26T07:33:26.357 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-07-26T07:33:27.131 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-07-26T07:33:27.597 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-07-26T07:33:29.794 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T07:33:29.796 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T07:33:29.818 INFO:teuthology.orchestra.run.smithi097.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T07:33:29.819 INFO:teuthology.orchestra.run.smithi097.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T07:33:29.821 INFO:teuthology.orchestra.run.smithi097.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T07:33:29.822 INFO:teuthology.orchestra.run.smithi097.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T07:33:29.824 INFO:teuthology.orchestra.run.smithi097.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T07:33:29.828 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T07:33:29.829 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T07:33:29.904 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-07-26T07:33:29.928 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2024-07-26T07:33:29.928 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-07-26T07:33:29.928 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2024-07-26T07:33:29.928 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-07-26T07:33:29.928 INFO:teuthology.orchestra.run.smithi097.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T07:33:29.928 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T07:33:29.928 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T07:33:29.929 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T07:33:29.930 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T07:33:29.930 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T07:33:29.930 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T07:33:29.930 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T07:33:29.930 INFO:teuthology.orchestra.run.smithi097.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T07:33:29.930 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T07:33:29.930 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T07:33:29.930 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T07:33:29.931 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T07:33:29.932 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T07:33:29.933 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T07:33:29.934 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T07:33:29.935 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T07:33:29.936 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T07:33:29.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T07:33:29.938 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T07:33:29.939 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T07:33:29.940 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T07:33:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:33:29.942 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-07-26T07:33:29.943 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2024-07-26T07:33:29.943 INFO:teuthology.orchestra.run.smithi097.stdout:Install 148 Packages 2024-07-26T07:33:29.943 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 12 Packages 2024-07-26T07:33:29.943 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:33:29.943 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 233 M 2024-07-26T07:33:29.943 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-07-26T07:33:30.148 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T07:33:30.150 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T07:33:30.168 INFO:teuthology.orchestra.run.smithi181.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T07:33:30.170 INFO:teuthology.orchestra.run.smithi181.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T07:33:30.171 INFO:teuthology.orchestra.run.smithi181.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T07:33:30.172 INFO:teuthology.orchestra.run.smithi181.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T07:33:30.174 INFO:teuthology.orchestra.run.smithi181.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T07:33:30.177 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T07:33:30.178 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T07:33:30.249 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-07-26T07:33:30.263 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-07-26T07:33:30.263 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-07-26T07:33:30.263 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-07-26T07:33:30.263 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-07-26T07:33:30.263 INFO:teuthology.orchestra.run.smithi181.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T07:33:30.264 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T07:33:30.265 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T07:33:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T07:33:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T07:33:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T07:33:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T07:33:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T07:33:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T07:33:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T07:33:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T07:33:30.267 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T07:33:30.268 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T07:33:30.269 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T07:33:30.270 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T07:33:30.271 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T07:33:30.272 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T07:33:30.273 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T07:33:30.274 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T07:33:30.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T07:33:30.276 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout:Install 148 Packages 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 12 Packages 2024-07-26T07:33:30.277 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:33:30.278 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 233 M 2024-07-26T07:33:30.278 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-07-26T07:33:30.941 INFO:teuthology.orchestra.run.smithi097.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-07-26T07:33:31.133 INFO:teuthology.orchestra.run.smithi097.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 4.5 MB/s | 872 kB 00:00 2024-07-26T07:33:31.200 INFO:teuthology.orchestra.run.smithi097.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 2.2 MB/s | 148 kB 00:00 2024-07-26T07:33:31.331 INFO:teuthology.orchestra.run.smithi181.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-07-26T07:33:31.342 INFO:teuthology.orchestra.run.smithi097.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 8.2 MB/s | 5.5 MB 00:00 2024-07-26T07:33:31.442 INFO:teuthology.orchestra.run.smithi097.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 9.4 MB/s | 2.3 MB 00:00 2024-07-26T07:33:31.509 INFO:teuthology.orchestra.run.smithi097.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 9.1 MB/s | 1.5 MB 00:00 2024-07-26T07:33:31.815 INFO:teuthology.orchestra.run.smithi181.stdout:(2/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 7.3 MB/s | 5.5 MB 00:00 2024-07-26T07:33:31.907 INFO:teuthology.orchestra.run.smithi181.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 1.6 MB/s | 148 kB 00:00 2024-07-26T07:33:31.917 INFO:teuthology.orchestra.run.smithi097.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 10 MB/s | 4.9 MB 00:00 2024-07-26T07:33:31.949 INFO:teuthology.orchestra.run.smithi181.stdout:(4/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.4 MB/s | 872 kB 00:00 2024-07-26T07:33:32.282 INFO:teuthology.orchestra.run.smithi181.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 6.0 MB/s | 2.3 MB 00:00 2024-07-26T07:33:32.376 INFO:teuthology.orchestra.run.smithi097.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 12 MB/s | 20 MB 00:01 2024-07-26T07:33:32.426 INFO:teuthology.orchestra.run.smithi097.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 512 kB/s | 25 kB 00:00 2024-07-26T07:33:32.499 INFO:teuthology.orchestra.run.smithi181.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-07-26T07:33:32.577 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-07-26T07:33:32.725 INFO:teuthology.orchestra.run.smithi181.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 11 MB/s | 4.9 MB 00:00 2024-07-26T07:33:32.777 INFO:teuthology.orchestra.run.smithi097.stdout:(10/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 14 MB/s | 18 MB 00:01 2024-07-26T07:33:32.835 INFO:teuthology.orchestra.run.smithi097.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 3.9 MB/s | 225 kB 00:00 2024-07-26T07:33:33.002 INFO:teuthology.orchestra.run.smithi097.stdout:(12/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 9.6 MB/s | 10 MB 00:01 2024-07-26T07:33:33.028 INFO:teuthology.orchestra.run.smithi097.stdout:(13/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 141 kB/s | 27 kB 00:00 2024-07-26T07:33:33.053 INFO:teuthology.orchestra.run.smithi097.stdout:(14/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 641 kB/s | 32 kB 00:00 2024-07-26T07:33:33.128 INFO:teuthology.orchestra.run.smithi097.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 7.5 MB/s | 765 kB 00:00 2024-07-26T07:33:33.161 INFO:teuthology.orchestra.run.smithi097.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 1.4 MB/s | 158 kB 00:00 2024-07-26T07:33:33.204 INFO:teuthology.orchestra.run.smithi097.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.7 MB/s | 127 kB 00:00 2024-07-26T07:33:33.246 INFO:teuthology.orchestra.run.smithi097.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 5.9 MB/s | 503 kB 00:00 2024-07-26T07:33:33.304 INFO:teuthology.orchestra.run.smithi097.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 787 kB/s | 45 kB 00:00 2024-07-26T07:33:33.342 INFO:teuthology.orchestra.run.smithi181.stdout:(8/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el9 17 MB/s | 10 MB 00:00 2024-07-26T07:33:33.363 INFO:teuthology.orchestra.run.smithi097.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.4 MB/s | 138 kB 00:00 2024-07-26T07:33:33.417 INFO:teuthology.orchestra.run.smithi181.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 338 kB/s | 25 kB 00:00 2024-07-26T07:33:33.468 INFO:teuthology.orchestra.run.smithi097.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 1.8 MB/s | 188 kB 00:00 2024-07-26T07:33:33.572 INFO:teuthology.orchestra.run.smithi097.stdout:(22/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 14 MB/s | 5.3 MB 00:00 2024-07-26T07:33:33.614 INFO:teuthology.orchestra.run.smithi097.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 2.5 MB/s | 372 kB 00:00 2024-07-26T07:33:33.664 INFO:teuthology.orchestra.run.smithi097.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 3.8 MB/s | 355 kB 00:00 2024-07-26T07:33:33.697 INFO:teuthology.orchestra.run.smithi097.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.4 MB/s | 116 kB 00:00 2024-07-26T07:33:33.734 INFO:teuthology.orchestra.run.smithi097.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.3 MB/s | 86 kB 00:00 2024-07-26T07:33:33.823 INFO:teuthology.orchestra.run.smithi097.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 1.8 MB/s | 170 kB 00:00 2024-07-26T07:33:33.982 INFO:teuthology.orchestra.run.smithi097.stdout:(28/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 3.1 MB 00:00 2024-07-26T07:33:34.007 INFO:teuthology.orchestra.run.smithi097.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 188 kB/s | 34 kB 00:00 2024-07-26T07:33:34.058 INFO:teuthology.orchestra.run.smithi097.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 3.0 MB/s | 149 kB 00:00 2024-07-26T07:33:34.100 INFO:teuthology.orchestra.run.smithi181.stdout:(10/160): ceph-common-19.0.0-5320.ge8f88bb1.el9 6.5 MB/s | 20 MB 00:03 2024-07-26T07:33:34.355 INFO:teuthology.orchestra.run.smithi181.stdout:(11/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 9.6 MB/s | 18 MB 00:01 2024-07-26T07:33:34.384 INFO:teuthology.orchestra.run.smithi181.stdout:(12/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 797 kB/s | 225 kB 00:00 2024-07-26T07:33:34.434 INFO:teuthology.orchestra.run.smithi181.stdout:(13/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 325 kB/s | 27 kB 00:00 2024-07-26T07:33:34.468 INFO:teuthology.orchestra.run.smithi181.stdout:(14/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 382 kB/s | 32 kB 00:00 2024-07-26T07:33:34.568 INFO:teuthology.orchestra.run.smithi181.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 1.6 MB/s | 158 kB 00:00 2024-07-26T07:33:34.618 INFO:teuthology.orchestra.run.smithi181.stdout:(16/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 4.1 MB/s | 765 kB 00:00 2024-07-26T07:33:34.625 INFO:teuthology.orchestra.run.smithi097.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-07-26T07:33:34.652 INFO:teuthology.orchestra.run.smithi181.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.5 MB/s | 127 kB 00:00 2024-07-26T07:33:34.694 INFO:teuthology.orchestra.run.smithi181.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 6.6 MB/s | 503 kB 00:00 2024-07-26T07:33:34.708 INFO:teuthology.orchestra.run.smithi097.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 3.4 MB/s | 284 kB 00:00 2024-07-26T07:33:34.769 INFO:teuthology.orchestra.run.smithi181.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 608 kB/s | 45 kB 00:00 2024-07-26T07:33:34.829 INFO:teuthology.orchestra.run.smithi181.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.3 MB/s | 138 kB 00:00 2024-07-26T07:33:34.887 INFO:teuthology.orchestra.run.smithi181.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.3 MB/s | 188 kB 00:00 2024-07-26T07:33:34.892 INFO:teuthology.orchestra.run.smithi097.stdout:(33/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 16 MB/s | 14 MB 00:00 2024-07-26T07:33:34.917 INFO:teuthology.orchestra.run.smithi097.stdout:(34/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 237 kB/s | 49 kB 00:00 2024-07-26T07:33:34.954 INFO:teuthology.orchestra.run.smithi181.stdout:(22/160): python3-rados-19.0.0-5320.ge8f88bb1.e 5.5 MB/s | 372 kB 00:00 2024-07-26T07:33:34.959 INFO:teuthology.orchestra.run.smithi097.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5320.ge 250 kB/s | 16 kB 00:00 2024-07-26T07:33:35.001 INFO:teuthology.orchestra.run.smithi097.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 3.2 MB/s | 274 kB 00:00 2024-07-26T07:33:35.020 INFO:teuthology.orchestra.run.smithi181.stdout:(23/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 5.3 MB/s | 355 kB 00:00 2024-07-26T07:33:35.068 INFO:teuthology.orchestra.run.smithi097.stdout:(37/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 7.0 MB/s | 771 kB 00:00 2024-07-26T07:33:35.079 INFO:teuthology.orchestra.run.smithi181.stdout:(24/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.0 MB/s | 116 kB 00:00 2024-07-26T07:33:35.093 INFO:teuthology.orchestra.run.smithi097.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 200 kB/s | 18 kB 00:00 2024-07-26T07:33:35.126 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T07:33:35.128 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T07:33:35.148 INFO:teuthology.orchestra.run.smithi115.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T07:33:35.149 INFO:teuthology.orchestra.run.smithi115.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T07:33:35.150 INFO:teuthology.orchestra.run.smithi115.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T07:33:35.152 INFO:teuthology.orchestra.run.smithi115.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T07:33:35.154 INFO:teuthology.orchestra.run.smithi115.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T07:33:35.155 INFO:teuthology.orchestra.run.smithi181.stdout:(25/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.1 MB/s | 86 kB 00:00 2024-07-26T07:33:35.157 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T07:33:35.159 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T07:33:35.231 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-07-26T07:33:35.250 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================= 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================= 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T07:33:35.251 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T07:33:35.252 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T07:33:35.253 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T07:33:35.253 INFO:teuthology.orchestra.run.smithi115.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T07:33:35.253 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T07:33:35.253 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T07:33:35.253 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T07:33:35.253 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T07:33:35.253 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T07:33:35.253 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T07:33:35.254 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T07:33:35.255 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T07:33:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T07:33:35.257 INFO:teuthology.orchestra.run.smithi115.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T07:33:35.258 INFO:teuthology.orchestra.run.smithi115.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T07:33:35.259 INFO:teuthology.orchestra.run.smithi115.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T07:33:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T07:33:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T07:33:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T07:33:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout:Installing weak dependencies: 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================= 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout:Install 148 Packages 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 12 Packages 2024-07-26T07:33:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:33:35.265 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 233 M 2024-07-26T07:33:35.265 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-07-26T07:33:35.297 INFO:teuthology.orchestra.run.smithi181.stdout:(26/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 8.2 MB/s | 5.3 MB 00:00 2024-07-26T07:33:35.364 INFO:teuthology.orchestra.run.smithi181.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 2.5 MB/s | 170 kB 00:00 2024-07-26T07:33:35.414 INFO:teuthology.orchestra.run.smithi181.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 697 kB/s | 34 kB 00:00 2024-07-26T07:33:35.520 INFO:teuthology.orchestra.run.smithi181.stdout:(29/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 8.5 MB/s | 3.1 MB 00:00 2024-07-26T07:33:35.556 INFO:teuthology.orchestra.run.smithi181.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.0 MB/s | 149 kB 00:00 2024-07-26T07:33:35.852 INFO:teuthology.orchestra.run.smithi097.stdout:(39/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 13 MB/s | 44 MB 00:03 2024-07-26T07:33:35.961 INFO:teuthology.orchestra.run.smithi097.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 89 kB/s | 80 kB 00:00 2024-07-26T07:33:36.011 INFO:teuthology.orchestra.run.smithi097.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-07-26T07:33:36.014 INFO:teuthology.orchestra.run.smithi181.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 16 MB/s | 7.4 MB 00:00 2024-07-26T07:33:36.065 INFO:teuthology.orchestra.run.smithi181.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 5.6 MB/s | 284 kB 00:00 2024-07-26T07:33:36.102 INFO:teuthology.orchestra.run.smithi097.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 796 kB/s | 803 kB 00:01 2024-07-26T07:33:36.123 INFO:teuthology.orchestra.run.smithi181.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 854 kB/s | 49 kB 00:00 2024-07-26T07:33:36.170 INFO:teuthology.orchestra.run.smithi097.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-07-26T07:33:36.174 INFO:teuthology.orchestra.run.smithi181.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 333 kB/s | 16 kB 00:00 2024-07-26T07:33:36.253 INFO:teuthology.orchestra.run.smithi097.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 710 kB/s | 106 kB 00:00 2024-07-26T07:33:36.320 INFO:teuthology.orchestra.run.smithi097.stdout:(45/160): python3-cryptography-36.0.1-4.el9.x86 4.0 MB/s | 1.2 MB 00:00 2024-07-26T07:33:36.346 INFO:teuthology.orchestra.run.smithi097.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 774 kB/s | 135 kB 00:00 2024-07-26T07:33:36.446 INFO:teuthology.orchestra.run.smithi097.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 653 kB/s | 125 kB 00:00 2024-07-26T07:33:36.520 INFO:teuthology.orchestra.run.smithi115.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-07-26T07:33:36.546 INFO:teuthology.orchestra.run.smithi097.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 956 kB/s | 215 kB 00:00 2024-07-26T07:33:36.650 INFO:teuthology.orchestra.run.smithi181.stdout:(35/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 14 MB/s | 44 MB 00:03 2024-07-26T07:33:36.699 INFO:teuthology.orchestra.run.smithi181.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 523 kB/s | 274 kB 00:00 2024-07-26T07:33:36.729 INFO:teuthology.orchestra.run.smithi115.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 4.1 MB/s | 872 kB 00:00 2024-07-26T07:33:36.758 INFO:teuthology.orchestra.run.smithi181.stdout:(37/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 316 kB/s | 18 kB 00:00 2024-07-26T07:33:36.804 INFO:teuthology.orchestra.run.smithi115.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 2.0 MB/s | 148 kB 00:00 2024-07-26T07:33:36.833 INFO:teuthology.orchestra.run.smithi181.stdout:(38/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 4.1 MB/s | 771 kB 00:00 2024-07-26T07:33:36.905 INFO:teuthology.orchestra.run.smithi115.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 8.6 MB/s | 5.5 MB 00:00 2024-07-26T07:33:37.013 INFO:teuthology.orchestra.run.smithi115.stdout:(5/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 14 MB/s | 1.5 MB 00:00 2024-07-26T07:33:37.114 INFO:teuthology.orchestra.run.smithi181.stdout:(39/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 8.8 MB/s | 14 MB 00:01 2024-07-26T07:33:37.122 INFO:teuthology.orchestra.run.smithi115.stdout:(6/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 7.1 MB/s | 2.3 MB 00:00 2024-07-26T07:33:37.138 INFO:teuthology.orchestra.run.smithi097.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 181 kB/s | 107 kB 00:00 2024-07-26T07:33:37.280 INFO:teuthology.orchestra.run.smithi115.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 18 MB/s | 4.9 MB 00:00 2024-07-26T07:33:37.313 INFO:teuthology.orchestra.run.smithi097.stdout:(50/160): automake-1.16.2-8.el9.noarch.rpm 713 kB/s | 689 kB 00:00 2024-07-26T07:33:37.355 INFO:teuthology.orchestra.run.smithi097.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-07-26T07:33:37.380 INFO:teuthology.orchestra.run.smithi097.stdout:(52/160): elfutils-libelf-devel-0.191-4.el9.x86 90 kB/s | 22 kB 00:00 2024-07-26T07:33:37.447 INFO:teuthology.orchestra.run.smithi097.stdout:(53/160): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-07-26T07:33:37.450 INFO:teuthology.orchestra.run.smithi181.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 115 kB/s | 80 kB 00:00 2024-07-26T07:33:37.498 INFO:teuthology.orchestra.run.smithi097.stdout:(54/160): flexiblas-3.0.4-8.el9.x86_64.rpm 282 kB/s | 33 kB 00:00 2024-07-26T07:33:37.584 INFO:teuthology.orchestra.run.smithi181.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 416 kB/s | 195 kB 00:00 2024-07-26T07:33:37.701 INFO:teuthology.orchestra.run.smithi181.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 927 kB/s | 803 kB 00:00 2024-07-26T07:33:37.740 INFO:teuthology.orchestra.run.smithi097.stdout:(55/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.4 MB/s | 3.0 MB 00:00 2024-07-26T07:33:37.765 INFO:teuthology.orchestra.run.smithi097.stdout:(56/160): flexiblas-openblas-openmp-3.0.4-8.el9 57 kB/s | 18 kB 00:00 2024-07-26T07:33:37.790 INFO:teuthology.orchestra.run.smithi097.stdout:(57/160): libaio-devel-0.3.111-13.el9.x86_64.rp 39 kB/s | 11 kB 00:00 2024-07-26T07:33:37.831 INFO:teuthology.orchestra.run.smithi115.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 13 MB/s | 20 MB 00:01 2024-07-26T07:33:37.873 INFO:teuthology.orchestra.run.smithi115.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 620 kB/s | 25 kB 00:00 2024-07-26T07:33:37.940 INFO:teuthology.orchestra.run.smithi097.stdout:(58/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-07-26T07:33:37.957 INFO:teuthology.orchestra.run.smithi097.stdout:(59/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 273 kB/s | 45 kB 00:00 2024-07-26T07:33:37.999 INFO:teuthology.orchestra.run.smithi097.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-07-26T07:33:38.035 INFO:teuthology.orchestra.run.smithi181.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 434 kB/s | 253 kB 00:00 2024-07-26T07:33:38.056 INFO:teuthology.orchestra.run.smithi115.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 13 MB/s | 10 MB 00:00 2024-07-26T07:33:38.083 INFO:teuthology.orchestra.run.smithi097.stdout:(61/160): libtool-2.4.6-46.el9.x86_64.rpm 4.8 MB/s | 581 kB 00:00 2024-07-26T07:33:38.099 INFO:teuthology.orchestra.run.smithi097.stdout:(62/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 187 kB/s | 25 kB 00:00 2024-07-26T07:33:38.125 INFO:teuthology.orchestra.run.smithi097.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 380 kB/s | 47 kB 00:00 2024-07-26T07:33:38.138 INFO:teuthology.orchestra.run.smithi115.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 2.7 MB/s | 225 kB 00:00 2024-07-26T07:33:38.182 INFO:teuthology.orchestra.run.smithi115.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 634 kB/s | 27 kB 00:00 2024-07-26T07:33:38.193 INFO:teuthology.orchestra.run.smithi181.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 217 kB/s | 106 kB 00:00 2024-07-26T07:33:38.217 INFO:teuthology.orchestra.run.smithi097.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 292 kB 00:00 2024-07-26T07:33:38.233 INFO:teuthology.orchestra.run.smithi115.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 645 kB/s | 32 kB 00:00 2024-07-26T07:33:38.242 INFO:teuthology.orchestra.run.smithi097.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-07-26T07:33:38.259 INFO:teuthology.orchestra.run.smithi097.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 264 kB/s | 35 kB 00:00 2024-07-26T07:33:38.316 INFO:teuthology.orchestra.run.smithi115.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 9.1 MB/s | 765 kB 00:00 2024-07-26T07:33:38.366 INFO:teuthology.orchestra.run.smithi115.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 3.1 MB/s | 158 kB 00:00 2024-07-26T07:33:38.416 INFO:teuthology.orchestra.run.smithi115.stdout:(16/160): librados-devel-19.0.0-5320.ge8f88bb1. 2.5 MB/s | 127 kB 00:00 2024-07-26T07:33:38.426 INFO:teuthology.orchestra.run.smithi181.stdout:(45/160): python3-cryptography-36.0.1-4.el9.x86 1.5 MB/s | 1.2 MB 00:00 2024-07-26T07:33:38.494 INFO:teuthology.orchestra.run.smithi181.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 295 kB/s | 135 kB 00:00 2024-07-26T07:33:38.508 INFO:teuthology.orchestra.run.smithi115.stdout:(17/160): libradosstriper1-19.0.0-5320.ge8f88bb 5.4 MB/s | 503 kB 00:00 2024-07-26T07:33:38.534 INFO:teuthology.orchestra.run.smithi097.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-07-26T07:33:38.686 INFO:teuthology.orchestra.run.smithi181.stdout:(47/160): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-07-26T07:33:38.710 INFO:teuthology.orchestra.run.smithi097.stdout:(68/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 9.5 MB/s | 4.4 MB 00:00 2024-07-26T07:33:38.711 INFO:teuthology.orchestra.run.smithi181.stdout:(48/160): python3-requests-2.25.1-8.el9.noarch. 242 kB/s | 125 kB 00:00 2024-07-26T07:33:38.777 INFO:teuthology.orchestra.run.smithi097.stdout:(69/160): postgresql-13.14-1.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-07-26T07:33:38.784 INFO:teuthology.orchestra.run.smithi115.stdout:(18/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 18 MB 00:01 2024-07-26T07:33:38.819 INFO:teuthology.orchestra.run.smithi181.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 995 kB/s | 107 kB 00:00 2024-07-26T07:33:38.835 INFO:teuthology.orchestra.run.smithi097.stdout:(70/160): postgresql-contrib-13.14-1.el9.x86_64 3.1 MB/s | 911 kB 00:00 2024-07-26T07:33:38.859 INFO:teuthology.orchestra.run.smithi115.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 609 kB/s | 45 kB 00:00 2024-07-26T07:33:38.860 INFO:teuthology.orchestra.run.smithi097.stdout:(71/160): postgresql-private-libs-13.14-1.el9.x 960 kB/s | 133 kB 00:00 2024-07-26T07:33:38.861 INFO:teuthology.orchestra.run.smithi181.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 533 kB/s | 22 kB 00:00 2024-07-26T07:33:38.917 INFO:teuthology.orchestra.run.smithi115.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.3 MB/s | 138 kB 00:00 2024-07-26T07:33:38.928 INFO:teuthology.orchestra.run.smithi181.stdout:(51/160): python3-urllib3-1.26.5-6.el9.noarch.r 429 kB/s | 215 kB 00:00 2024-07-26T07:33:38.971 INFO:teuthology.orchestra.run.smithi181.stdout:(52/160): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-07-26T07:33:38.976 INFO:teuthology.orchestra.run.smithi115.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.2 MB/s | 188 kB 00:00 2024-07-26T07:33:39.054 INFO:teuthology.orchestra.run.smithi181.stdout:(53/160): bison-3.7.4-5.el9.x86_64.rpm 2.5 MB/s | 943 kB 00:00 2024-07-26T07:33:39.080 INFO:teuthology.orchestra.run.smithi181.stdout:(54/160): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 33 kB 00:00 2024-07-26T07:33:39.101 INFO:teuthology.orchestra.run.smithi115.stdout:(22/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 8.9 MB/s | 5.3 MB 00:00 2024-07-26T07:33:39.105 INFO:teuthology.orchestra.run.smithi181.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-07-26T07:33:39.130 INFO:teuthology.orchestra.run.smithi181.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 226 kB/s | 11 kB 00:00 2024-07-26T07:33:39.135 INFO:teuthology.orchestra.run.smithi097.stdout:(72/160): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.8 MB 00:00 2024-07-26T07:33:39.143 INFO:teuthology.orchestra.run.smithi115.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 2.2 MB/s | 372 kB 00:00 2024-07-26T07:33:39.161 INFO:teuthology.orchestra.run.smithi097.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 292 kB/s | 87 kB 00:00 2024-07-26T07:33:39.172 INFO:teuthology.orchestra.run.smithi181.stdout:(57/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-07-26T07:33:39.197 INFO:teuthology.orchestra.run.smithi181.stdout:(58/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-07-26T07:33:39.211 INFO:teuthology.orchestra.run.smithi097.stdout:(74/160): protobuf-3.14.0-13.el9.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00 2024-07-26T07:33:39.218 INFO:teuthology.orchestra.run.smithi115.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 3.0 MB/s | 355 kB 00:00 2024-07-26T07:33:39.252 INFO:teuthology.orchestra.run.smithi115.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.1 MB/s | 116 kB 00:00 2024-07-26T07:33:39.278 INFO:teuthology.orchestra.run.smithi115.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.4 MB/s | 86 kB 00:00 2024-07-26T07:33:39.289 INFO:teuthology.orchestra.run.smithi181.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 5.6 MB/s | 662 kB 00:00 2024-07-26T07:33:39.323 INFO:teuthology.orchestra.run.smithi181.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 4.5 MB/s | 581 kB 00:00 2024-07-26T07:33:39.336 INFO:teuthology.orchestra.run.smithi115.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 2.9 MB/s | 170 kB 00:00 2024-07-26T07:33:39.348 INFO:teuthology.orchestra.run.smithi181.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 438 kB/s | 25 kB 00:00 2024-07-26T07:33:39.353 INFO:teuthology.orchestra.run.smithi097.stdout:(75/160): python3-colorama-0.4.6-3.el9.noarch.r 333 kB/s | 61 kB 00:00 2024-07-26T07:33:39.374 INFO:teuthology.orchestra.run.smithi181.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 962 kB/s | 47 kB 00:00 2024-07-26T07:33:39.403 INFO:teuthology.orchestra.run.smithi115.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 522 kB/s | 34 kB 00:00 2024-07-26T07:33:39.431 INFO:teuthology.orchestra.run.smithi181.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.5 MB/s | 292 kB 00:00 2024-07-26T07:33:39.466 INFO:teuthology.orchestra.run.smithi181.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-07-26T07:33:39.470 INFO:teuthology.orchestra.run.smithi097.stdout:(76/160): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-07-26T07:33:39.491 INFO:teuthology.orchestra.run.smithi181.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 596 kB/s | 35 kB 00:00 2024-07-26T07:33:39.495 INFO:teuthology.orchestra.run.smithi097.stdout:(77/160): python3-devel-3.9.18-3.el9.x86_64.rpm 862 kB/s | 244 kB 00:00 2024-07-26T07:33:39.496 INFO:teuthology.orchestra.run.smithi115.stdout:(29/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 13 MB/s | 3.1 MB 00:00 2024-07-26T07:33:39.528 INFO:teuthology.orchestra.run.smithi097.stdout:(78/160): python3-jinja2-2.11.3-6.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-07-26T07:33:39.529 INFO:teuthology.orchestra.run.smithi115.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.2 MB/s | 149 kB 00:00 2024-07-26T07:33:39.604 INFO:teuthology.orchestra.run.smithi097.stdout:(79/160): python3-jsonpatch-1.21-16.el9.noarch. 244 kB/s | 26 kB 00:00 2024-07-26T07:33:39.621 INFO:teuthology.orchestra.run.smithi097.stdout:(80/160): python3-jsonpointer-2.0-4.el9.noarch. 157 kB/s | 19 kB 00:00 2024-07-26T07:33:39.655 INFO:teuthology.orchestra.run.smithi097.stdout:(81/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-07-26T07:33:39.747 INFO:teuthology.orchestra.run.smithi097.stdout:(82/160): python3-markupsafe-1.1.1-12.el9.x86_6 283 kB/s | 35 kB 00:00 2024-07-26T07:33:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.5 MB/s | 3.0 MB 00:00 2024-07-26T07:33:39.947 INFO:teuthology.orchestra.run.smithi097.stdout:(83/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-07-26T07:33:39.972 INFO:teuthology.orchestra.run.smithi097.stdout:(84/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 792 kB/s | 250 kB 00:00 2024-07-26T07:33:39.997 INFO:teuthology.orchestra.run.smithi097.stdout:(85/160): python3-oauthlib-3.1.1-5.el9.noarch.r 924 kB/s | 222 kB 00:00 2024-07-26T07:33:40.013 INFO:teuthology.orchestra.run.smithi115.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-07-26T07:33:40.071 INFO:teuthology.orchestra.run.smithi115.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 4.8 MB/s | 284 kB 00:00 2024-07-26T07:33:40.106 INFO:teuthology.orchestra.run.smithi097.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 722 kB/s | 77 kB 00:00 2024-07-26T07:33:40.121 INFO:teuthology.orchestra.run.smithi115.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 1.0 MB/s | 49 kB 00:00 2024-07-26T07:33:40.131 INFO:teuthology.orchestra.run.smithi097.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 320 kB/s | 42 kB 00:00 2024-07-26T07:33:40.165 INFO:teuthology.orchestra.run.smithi097.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 1.6 MB/s | 269 kB 00:00 2024-07-26T07:33:40.172 INFO:teuthology.orchestra.run.smithi115.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 334 kB/s | 16 kB 00:00 2024-07-26T07:33:40.231 INFO:teuthology.orchestra.run.smithi115.stdout:(35/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 4.7 MB/s | 274 kB 00:00 2024-07-26T07:33:40.267 INFO:teuthology.orchestra.run.smithi181.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-07-26T07:33:40.314 INFO:teuthology.orchestra.run.smithi115.stdout:(36/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 9.1 MB/s | 771 kB 00:00 2024-07-26T07:33:40.315 INFO:teuthology.orchestra.run.smithi097.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 773 kB/s | 159 kB 00:00 2024-07-26T07:33:40.349 INFO:teuthology.orchestra.run.smithi097.stdout:(90/160): python3-pyasn1-modules-0.4.8-6.el9.no 1.5 MB/s | 279 kB 00:00 2024-07-26T07:33:40.374 INFO:teuthology.orchestra.run.smithi097.stdout:(91/160): python3-pytz-2021.1-5.el9.noarch.rpm 245 kB/s | 51 kB 00:00 2024-07-26T07:33:40.489 INFO:teuthology.orchestra.run.smithi115.stdout:(37/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 14 MB/s | 14 MB 00:00 2024-07-26T07:33:40.515 INFO:teuthology.orchestra.run.smithi115.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 91 kB/s | 18 kB 00:00 2024-07-26T07:33:40.542 INFO:teuthology.orchestra.run.smithi181.stdout:(68/160): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-07-26T07:33:40.625 INFO:teuthology.orchestra.run.smithi097.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 215 kB/s | 54 kB 00:00 2024-07-26T07:33:40.633 INFO:teuthology.orchestra.run.smithi181.stdout:(69/160): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-07-26T07:33:40.724 INFO:teuthology.orchestra.run.smithi097.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 119 kB/s | 42 kB 00:00 2024-07-26T07:33:40.759 INFO:teuthology.orchestra.run.smithi181.stdout:(70/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2024-07-26T07:33:40.783 INFO:teuthology.orchestra.run.smithi097.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 298 kB/s | 44 kB 00:00 2024-07-26T07:33:40.925 INFO:teuthology.orchestra.run.smithi097.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 272 kB/s | 36 kB 00:00 2024-07-26T07:33:41.026 INFO:teuthology.orchestra.run.smithi181.stdout:(71/160): openblas-openmp-0.3.21-2.el9.x86_64.r 3.0 MB/s | 4.7 MB 00:01 2024-07-26T07:33:41.034 INFO:teuthology.orchestra.run.smithi097.stdout:(96/160): qatlib-24.02.0-1.el9.x86_64.rpm 748 kB/s | 221 kB 00:00 2024-07-26T07:33:41.076 INFO:teuthology.orchestra.run.smithi181.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-07-26T07:33:41.118 INFO:teuthology.orchestra.run.smithi181.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 962 kB/s | 87 kB 00:00 2024-07-26T07:33:41.165 INFO:teuthology.orchestra.run.smithi115.stdout:(39/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 13 MB/s | 44 MB 00:03 2024-07-26T07:33:41.168 INFO:teuthology.orchestra.run.smithi181.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-07-26T07:33:41.267 INFO:teuthology.orchestra.run.smithi097.stdout:(97/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-07-26T07:33:41.292 INFO:teuthology.orchestra.run.smithi097.stdout:(98/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 133 kB/s | 47 kB 00:00 2024-07-26T07:33:41.326 INFO:teuthology.orchestra.run.smithi097.stdout:(99/160): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-07-26T07:33:41.327 INFO:teuthology.orchestra.run.smithi181.stdout:(75/160): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 6.0 MB 00:00 2024-07-26T07:33:41.385 INFO:teuthology.orchestra.run.smithi181.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-07-26T07:33:41.419 INFO:teuthology.orchestra.run.smithi181.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 2.7 MB/s | 248 kB 00:00 2024-07-26T07:33:41.435 INFO:teuthology.orchestra.run.smithi097.stdout:(100/160): uuid-1.6.2-55.el9.x86_64.rpm 419 kB/s | 58 kB 00:00 2024-07-26T07:33:41.444 INFO:teuthology.orchestra.run.smithi181.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 444 kB/s | 26 kB 00:00 2024-07-26T07:33:41.451 INFO:teuthology.orchestra.run.smithi097.stdout:(101/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 618 kB/s | 96 kB 00:00 2024-07-26T07:33:41.470 INFO:teuthology.orchestra.run.smithi181.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 387 kB/s | 19 kB 00:00 2024-07-26T07:33:41.477 INFO:teuthology.orchestra.run.smithi097.stdout:(102/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 424 kB/s | 64 kB 00:00 2024-07-26T07:33:41.499 INFO:teuthology.orchestra.run.smithi115.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 79 kB/s | 80 kB 00:01 2024-07-26T07:33:41.520 INFO:teuthology.orchestra.run.smithi181.stdout:(80/160): python3-markupsafe-1.1.1-12.el9.x86_6 709 kB/s | 35 kB 00:00 2024-07-26T07:33:41.544 INFO:teuthology.orchestra.run.smithi097.stdout:(103/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 423 kB/s | 45 kB 00:00 2024-07-26T07:33:41.554 INFO:teuthology.orchestra.run.smithi181.stdout:(81/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-07-26T07:33:41.652 INFO:teuthology.orchestra.run.smithi097.stdout:(104/160): lua-devel-5.4.4-4.el9.x86_64.rpm 128 kB/s | 22 kB 00:00 2024-07-26T07:33:41.657 INFO:teuthology.orchestra.run.smithi115.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 409 kB/s | 195 kB 00:00 2024-07-26T07:33:41.686 INFO:teuthology.orchestra.run.smithi097.stdout:(105/160): protobuf-compiler-3.14.0-13.el9.x86_ 4.1 MB/s | 863 kB 00:00 2024-07-26T07:33:41.738 INFO:teuthology.orchestra.run.smithi181.stdout:(82/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-07-26T07:33:41.749 INFO:teuthology.orchestra.run.smithi115.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-07-26T07:33:41.797 INFO:teuthology.orchestra.run.smithi181.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-07-26T07:33:41.816 INFO:teuthology.orchestra.run.smithi115.stdout:(43/160): libgfortran-11.4.1-3.el9.x86_64.rpm 618 kB/s | 803 kB 00:01 2024-07-26T07:33:41.830 INFO:teuthology.orchestra.run.smithi181.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-07-26T07:33:41.836 INFO:teuthology.orchestra.run.smithi097.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-07-26T07:33:41.858 INFO:teuthology.orchestra.run.smithi181.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-07-26T07:33:41.861 INFO:teuthology.orchestra.run.smithi097.stdout:(107/160): gperftools-libs-2.9.1-2.el9.x86_64.r 1.8 MB/s | 309 kB 00:00 2024-07-26T07:33:41.880 INFO:teuthology.orchestra.run.smithi181.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 855 kB/s | 42 kB 00:00 2024-07-26T07:33:41.895 INFO:teuthology.orchestra.run.smithi097.stdout:(108/160): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-07-26T07:33:41.914 INFO:teuthology.orchestra.run.smithi181.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-07-26T07:33:41.920 INFO:teuthology.orchestra.run.smithi097.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-07-26T07:33:41.945 INFO:teuthology.orchestra.run.smithi097.stdout:(110/160): liboath-2.6.7-2.el9.x86_64.rpm 977 kB/s | 49 kB 00:00 2024-07-26T07:33:41.948 INFO:teuthology.orchestra.run.smithi181.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-07-26T07:33:41.950 INFO:teuthology.orchestra.run.smithi115.stdout:(44/160): python3-cryptography-36.0.1-4.el9.x86 4.2 MB/s | 1.2 MB 00:00 2024-07-26T07:33:41.962 INFO:teuthology.orchestra.run.smithi097.stdout:(111/160): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-07-26T07:33:41.983 INFO:teuthology.orchestra.run.smithi115.stdout:(45/160): python3-ply-3.11-14.el9.noarch.rpm 460 kB/s | 106 kB 00:00 2024-07-26T07:33:41.990 INFO:teuthology.orchestra.run.smithi181.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 279 kB 00:00 2024-07-26T07:33:41.996 INFO:teuthology.orchestra.run.smithi097.stdout:(112/160): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-07-26T07:33:42.009 INFO:teuthology.orchestra.run.smithi115.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 715 kB/s | 135 kB 00:00 2024-07-26T07:33:42.015 INFO:teuthology.orchestra.run.smithi181.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 773 kB/s | 51 kB 00:00 2024-07-26T07:33:42.040 INFO:teuthology.orchestra.run.smithi181.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-07-26T07:33:42.059 INFO:teuthology.orchestra.run.smithi115.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-07-26T07:33:42.063 INFO:teuthology.orchestra.run.smithi097.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 8.2 MB/s | 548 kB 00:00 2024-07-26T07:33:42.088 INFO:teuthology.orchestra.run.smithi097.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T07:33:42.099 INFO:teuthology.orchestra.run.smithi181.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 725 kB/s | 42 kB 00:00 2024-07-26T07:33:42.114 INFO:teuthology.orchestra.run.smithi097.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T07:33:42.154 INFO:teuthology.orchestra.run.smithi115.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 1.5 MB/s | 215 kB 00:00 2024-07-26T07:33:42.156 INFO:teuthology.orchestra.run.smithi097.stdout:(116/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 839 kB 00:00 2024-07-26T07:33:42.168 INFO:teuthology.orchestra.run.smithi181.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 650 kB/s | 44 kB 00:00 2024-07-26T07:33:42.181 INFO:teuthology.orchestra.run.smithi097.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 214 kB/s | 14 kB 00:00 2024-07-26T07:33:42.223 INFO:teuthology.orchestra.run.smithi097.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 4.1 MB/s | 172 kB 00:00 2024-07-26T07:33:42.233 INFO:teuthology.orchestra.run.smithi181.stdout:(94/160): qatlib-24.02.0-1.el9.x86_64.rpm 3.4 MB/s | 221 kB 00:00 2024-07-26T07:33:42.260 INFO:teuthology.orchestra.run.smithi115.stdout:(49/160): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 689 kB 00:00 2024-07-26T07:33:42.264 INFO:teuthology.orchestra.run.smithi097.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 4.3 MB/s | 358 kB 00:00 2024-07-26T07:33:42.283 INFO:teuthology.orchestra.run.smithi181.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 732 kB/s | 36 kB 00:00 2024-07-26T07:33:42.298 INFO:teuthology.orchestra.run.smithi097.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-07-26T07:33:42.310 INFO:teuthology.orchestra.run.smithi115.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 442 kB/s | 22 kB 00:00 2024-07-26T07:33:42.325 INFO:teuthology.orchestra.run.smithi181.stdout:(96/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-07-26T07:33:42.335 INFO:teuthology.orchestra.run.smithi115.stdout:(51/160): boost-program-options-1.75.0-8.el9.x8 594 kB/s | 107 kB 00:00 2024-07-26T07:33:42.340 INFO:teuthology.orchestra.run.smithi097.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 2.8 MB/s | 216 kB 00:00 2024-07-26T07:33:42.374 INFO:teuthology.orchestra.run.smithi097.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T07:33:42.385 INFO:teuthology.orchestra.run.smithi115.stdout:(52/160): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-07-26T07:33:42.411 INFO:teuthology.orchestra.run.smithi115.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 440 kB/s | 33 kB 00:00 2024-07-26T07:33:42.422 INFO:teuthology.orchestra.run.smithi097.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 230 kB/s | 11 kB 00:00 2024-07-26T07:33:42.444 INFO:teuthology.orchestra.run.smithi115.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-07-26T07:33:42.450 INFO:teuthology.orchestra.run.smithi097.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 661 kB/s | 18 kB 00:00 2024-07-26T07:33:42.467 INFO:teuthology.orchestra.run.smithi097.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 1.4 MB/s | 23 kB 00:00 2024-07-26T07:33:42.470 INFO:teuthology.orchestra.run.smithi115.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-07-26T07:33:42.484 INFO:teuthology.orchestra.run.smithi097.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 1.2 MB/s | 19 kB 00:00 2024-07-26T07:33:42.501 INFO:teuthology.orchestra.run.smithi097.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 1.2 MB/s | 20 kB 00:00 2024-07-26T07:33:42.516 INFO:teuthology.orchestra.run.smithi115.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 158 kB/s | 11 kB 00:00 2024-07-26T07:33:42.554 INFO:teuthology.orchestra.run.smithi115.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-07-26T07:33:42.579 INFO:teuthology.orchestra.run.smithi115.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 729 kB/s | 45 kB 00:00 2024-07-26T07:33:42.595 INFO:teuthology.orchestra.run.smithi097.stdout:(128/160): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-07-26T07:33:42.609 INFO:teuthology.orchestra.run.smithi181.stdout:(97/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-07-26T07:33:42.618 INFO:teuthology.orchestra.run.smithi097.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.9 MB/s | 46 kB 00:00 2024-07-26T07:33:42.629 INFO:teuthology.orchestra.run.smithi115.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-07-26T07:33:42.642 INFO:teuthology.orchestra.run.smithi181.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 962 kB/s | 304 kB 00:00 2024-07-26T07:33:42.652 INFO:teuthology.orchestra.run.smithi097.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.4 MB/s | 79 kB 00:00 2024-07-26T07:33:42.667 INFO:teuthology.orchestra.run.smithi181.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-07-26T07:33:42.677 INFO:teuthology.orchestra.run.smithi097.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-26T07:33:42.680 INFO:teuthology.orchestra.run.smithi115.stdout:(60/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 515 kB/s | 25 kB 00:00 2024-07-26T07:33:42.701 INFO:teuthology.orchestra.run.smithi181.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.6 MB/s | 96 kB 00:00 2024-07-26T07:33:42.719 INFO:teuthology.orchestra.run.smithi097.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 6.5 MB/s | 272 kB 00:00 2024-07-26T07:33:42.722 INFO:teuthology.orchestra.run.smithi115.stdout:(61/160): libtool-2.4.6-46.el9.x86_64.rpm 4.0 MB/s | 581 kB 00:00 2024-07-26T07:33:42.726 INFO:teuthology.orchestra.run.smithi181.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-07-26T07:33:42.744 INFO:teuthology.orchestra.run.smithi097.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 681 kB/s | 16 kB 00:00 2024-07-26T07:33:42.747 INFO:teuthology.orchestra.run.smithi115.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 716 kB/s | 47 kB 00:00 2024-07-26T07:33:42.752 INFO:teuthology.orchestra.run.smithi181.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-07-26T07:33:42.803 INFO:teuthology.orchestra.run.smithi097.stdout:(134/160): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-07-26T07:33:42.822 INFO:teuthology.orchestra.run.smithi115.stdout:(63/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.9 MB/s | 3.0 MB 00:00 2024-07-26T07:33:42.828 INFO:teuthology.orchestra.run.smithi097.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-26T07:33:42.854 INFO:teuthology.orchestra.run.smithi097.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 831 kB/s | 90 kB 00:00 2024-07-26T07:33:42.860 INFO:teuthology.orchestra.run.smithi181.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 168 kB/s | 22 kB 00:00 2024-07-26T07:33:42.879 INFO:teuthology.orchestra.run.smithi097.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 624 kB/s | 31 kB 00:00 2024-07-26T07:33:42.879 INFO:teuthology.orchestra.run.smithi115.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2024-07-26T07:33:42.913 INFO:teuthology.orchestra.run.smithi097.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-07-26T07:33:42.915 INFO:teuthology.orchestra.run.smithi115.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-07-26T07:33:42.938 INFO:teuthology.orchestra.run.smithi097.stdout:(139/160): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-07-26T07:33:42.940 INFO:teuthology.orchestra.run.smithi115.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 303 kB/s | 35 kB 00:00 2024-07-26T07:33:42.971 INFO:teuthology.orchestra.run.smithi097.stdout:(140/160): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-07-26T07:33:42.997 INFO:teuthology.orchestra.run.smithi097.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 882 kB/s | 51 kB 00:00 2024-07-26T07:33:43.019 INFO:teuthology.orchestra.run.smithi181.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 3.4 MB/s | 548 kB 00:00 2024-07-26T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 2.7 MB/s | 90 kB 00:00 2024-07-26T07:33:43.048 INFO:teuthology.orchestra.run.smithi115.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-07-26T07:33:43.061 INFO:teuthology.orchestra.run.smithi181.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 7.4 MB/s | 309 kB 00:00 2024-07-26T07:33:43.064 INFO:teuthology.orchestra.run.smithi097.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-07-26T07:33:43.086 INFO:teuthology.orchestra.run.smithi181.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 808 kB/s | 19 kB 00:00 2024-07-26T07:33:43.089 INFO:teuthology.orchestra.run.smithi097.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 829 kB/s | 20 kB 00:00 2024-07-26T07:33:43.146 INFO:teuthology.orchestra.run.smithi181.stdout:(107/160): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-07-26T07:33:43.165 INFO:teuthology.orchestra.run.smithi115.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-07-26T07:33:43.181 INFO:teuthology.orchestra.run.smithi097.stdout:(145/160): libarrow-9.0.0-11.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-07-26T07:33:43.215 INFO:teuthology.orchestra.run.smithi097.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-07-26T07:33:43.240 INFO:teuthology.orchestra.run.smithi115.stdout:(69/160): postgresql-contrib-13.14-1.el9.x86_64 4.7 MB/s | 911 kB 00:00 2024-07-26T07:33:43.248 INFO:teuthology.orchestra.run.smithi097.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 1.9 MB/s | 427 kB 00:00 2024-07-26T07:33:43.274 INFO:teuthology.orchestra.run.smithi115.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-07-26T07:33:43.312 INFO:teuthology.orchestra.run.smithi181.stdout:(108/160): postgresql-server-13.14-1.el9.x86_64 2.2 MB/s | 5.8 MB 00:02 2024-07-26T07:33:43.340 INFO:teuthology.orchestra.run.smithi097.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-07-26T07:33:43.391 INFO:teuthology.orchestra.run.smithi115.stdout:(71/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 9.9 MB/s | 4.4 MB 00:00 2024-07-26T07:33:43.412 INFO:teuthology.orchestra.run.smithi181.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 98 kB/s | 26 kB 00:00 2024-07-26T07:33:43.446 INFO:teuthology.orchestra.run.smithi181.stdout:(110/160): liboath-2.6.7-2.el9.x86_64.rpm 387 kB/s | 49 kB 00:00 2024-07-26T07:33:43.449 INFO:teuthology.orchestra.run.smithi115.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-07-26T07:33:43.475 INFO:teuthology.orchestra.run.smithi115.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-07-26T07:33:43.480 INFO:teuthology.orchestra.run.smithi181.stdout:(111/160): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-07-26T07:33:43.513 INFO:teuthology.orchestra.run.smithi181.stdout:(112/160): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-07-26T07:33:43.516 INFO:teuthology.orchestra.run.smithi097.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 3.4 MB 00:00 2024-07-26T07:33:43.525 INFO:teuthology.orchestra.run.smithi115.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-07-26T07:33:43.588 INFO:teuthology.orchestra.run.smithi181.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 7.2 MB/s | 548 kB 00:00 2024-07-26T07:33:43.609 INFO:teuthology.orchestra.run.smithi115.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-07-26T07:33:43.614 INFO:teuthology.orchestra.run.smithi181.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T07:33:43.616 INFO:teuthology.orchestra.run.smithi097.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 8.6 MB/s | 3.1 MB 00:00 2024-07-26T07:33:43.639 INFO:teuthology.orchestra.run.smithi181.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T07:33:43.668 INFO:teuthology.orchestra.run.smithi115.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-07-26T07:33:43.681 INFO:teuthology.orchestra.run.smithi181.stdout:(116/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.1 MB/s | 839 kB 00:00 2024-07-26T07:33:43.706 INFO:teuthology.orchestra.run.smithi181.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 213 kB/s | 14 kB 00:00 2024-07-26T07:33:43.725 INFO:teuthology.orchestra.run.smithi097.stdout:(151/160): NetworkManager-1.48.2-2.el9.x86_64.r 6.0 MB/s | 2.3 MB 00:00 2024-07-26T07:33:43.750 INFO:teuthology.orchestra.run.smithi097.stdout:(152/160): NetworkManager-team-1.48.2-2.el9.x86 308 kB/s | 41 kB 00:00 2024-07-26T07:33:43.773 INFO:teuthology.orchestra.run.smithi181.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-07-26T07:33:43.785 INFO:teuthology.orchestra.run.smithi115.stdout:(77/160): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-07-26T07:33:43.810 INFO:teuthology.orchestra.run.smithi115.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 181 kB/s | 26 kB 00:00 2024-07-26T07:33:43.815 INFO:teuthology.orchestra.run.smithi181.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 3.3 MB/s | 358 kB 00:00 2024-07-26T07:33:43.825 INFO:teuthology.orchestra.run.smithi097.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 5.9 MB/s | 1.8 MB 00:00 2024-07-26T07:33:43.835 INFO:teuthology.orchestra.run.smithi115.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 387 kB/s | 19 kB 00:00 2024-07-26T07:33:43.849 INFO:teuthology.orchestra.run.smithi181.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-07-26T07:33:43.869 INFO:teuthology.orchestra.run.smithi115.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-07-26T07:33:43.882 INFO:teuthology.orchestra.run.smithi181.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 3.2 MB/s | 216 kB 00:00 2024-07-26T07:33:43.894 INFO:teuthology.orchestra.run.smithi115.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 600 kB/s | 35 kB 00:00 2024-07-26T07:33:43.909 INFO:teuthology.orchestra.run.smithi097.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 232 kB/s | 37 kB 00:00 2024-07-26T07:33:43.916 INFO:teuthology.orchestra.run.smithi181.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.4 MB/s | 144 kB 00:00 2024-07-26T07:33:43.941 INFO:teuthology.orchestra.run.smithi181.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 439 kB/s | 11 kB 00:00 2024-07-26T07:33:43.943 INFO:teuthology.orchestra.run.smithi097.stdout:(155/160): NetworkManager-tui-1.48.2-2.el9.x86_ 1.1 MB/s | 247 kB 00:00 2024-07-26T07:33:43.961 INFO:teuthology.orchestra.run.smithi115.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-07-26T07:33:43.967 INFO:teuthology.orchestra.run.smithi181.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 732 kB/s | 18 kB 00:00 2024-07-26T07:33:43.992 INFO:teuthology.orchestra.run.smithi181.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 962 kB/s | 23 kB 00:00 2024-07-26T07:33:44.001 INFO:teuthology.orchestra.run.smithi097.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 1.2 MB/s | 207 kB 00:00 2024-07-26T07:33:44.017 INFO:teuthology.orchestra.run.smithi181.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 808 kB/s | 19 kB 00:00 2024-07-26T07:33:44.042 INFO:teuthology.orchestra.run.smithi181.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 817 kB/s | 20 kB 00:00 2024-07-26T07:33:44.078 INFO:teuthology.orchestra.run.smithi115.stdout:(83/160): python3-babel-2.9.1-2.el9.noarch.rpm 9.5 MB/s | 6.0 MB 00:00 2024-07-26T07:33:44.093 INFO:teuthology.orchestra.run.smithi097.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 1.4 MB/s | 260 kB 00:00 2024-07-26T07:33:44.112 INFO:teuthology.orchestra.run.smithi115.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-07-26T07:33:44.137 INFO:teuthology.orchestra.run.smithi115.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-07-26T07:33:44.160 INFO:teuthology.orchestra.run.smithi181.stdout:(128/160): python3-kubernetes-26.1.0-2.el9.noar 8.8 MB/s | 1.0 MB 00:00 2024-07-26T07:33:44.165 INFO:teuthology.orchestra.run.smithi115.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 852 kB/s | 42 kB 00:00 2024-07-26T07:33:44.185 INFO:teuthology.orchestra.run.smithi181.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.9 MB/s | 46 kB 00:00 2024-07-26T07:33:44.196 INFO:teuthology.orchestra.run.smithi115.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-07-26T07:33:44.218 INFO:teuthology.orchestra.run.smithi181.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.4 MB/s | 79 kB 00:00 2024-07-26T07:33:44.230 INFO:teuthology.orchestra.run.smithi115.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-07-26T07:33:44.243 INFO:teuthology.orchestra.run.smithi097.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 4.5 MB/s | 1.4 MB 00:00 2024-07-26T07:33:44.252 INFO:teuthology.orchestra.run.smithi181.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-07-26T07:33:44.290 INFO:teuthology.orchestra.run.smithi115.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 2.9 MB/s | 279 kB 00:00 2024-07-26T07:33:44.294 INFO:teuthology.orchestra.run.smithi181.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 6.5 MB/s | 272 kB 00:00 2024-07-26T07:33:44.322 INFO:teuthology.orchestra.run.smithi115.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 559 kB/s | 51 kB 00:00 2024-07-26T07:33:44.335 INFO:teuthology.orchestra.run.smithi097.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 7.2 MB/s | 2.4 MB 00:00 2024-07-26T07:33:44.347 INFO:teuthology.orchestra.run.smithi115.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 945 kB/s | 54 kB 00:00 2024-07-26T07:33:44.361 INFO:teuthology.orchestra.run.smithi181.stdout:(133/160): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-07-26T07:33:44.385 INFO:teuthology.orchestra.run.smithi097.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 3.7 MB/s | 1.1 MB 00:00 2024-07-26T07:33:44.386 INFO:teuthology.orchestra.run.smithi181.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 180 kB/s | 16 kB 00:00 2024-07-26T07:33:44.398 INFO:teuthology.orchestra.run.smithi115.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 847 kB/s | 42 kB 00:00 2024-07-26T07:33:44.405 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-07-26T07:33:44.405 INFO:teuthology.orchestra.run.smithi097.stdout:Total 16 MB/s | 233 MB 00:14 2024-07-26T07:33:44.411 INFO:teuthology.orchestra.run.smithi181.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.2 MB/s | 28 kB 00:00 2024-07-26T07:33:44.437 INFO:teuthology.orchestra.run.smithi181.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-07-26T07:33:44.448 INFO:teuthology.orchestra.run.smithi115.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 899 kB/s | 44 kB 00:00 2024-07-26T07:33:44.462 INFO:teuthology.orchestra.run.smithi181.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 624 kB/s | 31 kB 00:00 2024-07-26T07:33:44.520 INFO:teuthology.orchestra.run.smithi181.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-07-26T07:33:44.555 INFO:teuthology.orchestra.run.smithi181.stdout:(139/160): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-07-26T07:33:44.557 INFO:teuthology.orchestra.run.smithi115.stdout:(94/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.4 MB/s | 5.1 MB 00:00 2024-07-26T07:33:44.588 INFO:teuthology.orchestra.run.smithi181.stdout:(140/160): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-07-26T07:33:44.590 INFO:teuthology.orchestra.run.smithi115.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-07-26T07:33:44.613 INFO:teuthology.orchestra.run.smithi181.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 883 kB/s | 51 kB 00:00 2024-07-26T07:33:44.615 INFO:teuthology.orchestra.run.smithi115.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 622 kB/s | 36 kB 00:00 2024-07-26T07:33:44.647 INFO:teuthology.orchestra.run.smithi181.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 2.7 MB/s | 90 kB 00:00 2024-07-26T07:33:44.674 INFO:teuthology.orchestra.run.smithi115.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 572 kB/s | 47 kB 00:00 2024-07-26T07:33:44.681 INFO:teuthology.orchestra.run.smithi181.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-07-26T07:33:44.697 INFO:teuthology.orchestra.run.smithi181.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-07-26T07:33:44.708 INFO:teuthology.orchestra.run.smithi115.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 3.3 MB/s | 304 kB 00:00 2024-07-26T07:33:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-07-26T07:33:44.759 INFO:teuthology.orchestra.run.smithi115.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-07-26T07:33:44.784 INFO:teuthology.orchestra.run.smithi115.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-07-26T07:33:44.789 INFO:teuthology.orchestra.run.smithi181.stdout:(145/160): libarrow-9.0.0-11.el9.x86_64.rpm 2.6 MB/s | 4.4 MB 00:01 2024-07-26T07:33:44.809 INFO:teuthology.orchestra.run.smithi115.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-07-26T07:33:44.823 INFO:teuthology.orchestra.run.smithi181.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-07-26T07:33:44.857 INFO:teuthology.orchestra.run.smithi181.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.0 MB/s | 427 kB 00:00 2024-07-26T07:33:44.949 INFO:teuthology.orchestra.run.smithi181.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-07-26T07:33:45.110 INFO:teuthology.orchestra.run.smithi115.stdout:(103/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-07-26T07:33:45.163 INFO:teuthology.orchestra.run.smithi181.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 3.4 MB 00:00 2024-07-26T07:33:45.164 INFO:teuthology.orchestra.run.smithi115.stdout:(104/160): lua-devel-5.4.4-4.el9.x86_64.rpm 64 kB/s | 22 kB 00:00 2024-07-26T07:33:45.224 INFO:teuthology.orchestra.run.smithi181.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 8.6 MB/s | 3.1 MB 00:00 2024-07-26T07:33:45.301 INFO:teuthology.orchestra.run.smithi115.stdout:(105/160): abseil-cpp-20211102.0-3.el9.x86_64.r 2.8 MB/s | 548 kB 00:00 2024-07-26T07:33:45.327 INFO:teuthology.orchestra.run.smithi115.stdout:(106/160): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-07-26T07:33:45.369 INFO:teuthology.orchestra.run.smithi115.stdout:(107/160): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-07-26T07:33:45.394 INFO:teuthology.orchestra.run.smithi115.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-07-26T07:33:45.428 INFO:teuthology.orchestra.run.smithi115.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-07-26T07:33:45.462 INFO:teuthology.orchestra.run.smithi115.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-07-26T07:33:45.495 INFO:teuthology.orchestra.run.smithi115.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.6 MB/s | 151 kB 00:00 2024-07-26T07:33:45.525 INFO:teuthology.orchestra.run.smithi181.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 137 kB/s | 41 kB 00:00 2024-07-26T07:33:45.562 INFO:teuthology.orchestra.run.smithi115.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-07-26T07:33:45.613 INFO:teuthology.orchestra.run.smithi115.stdout:(113/160): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-07-26T07:33:45.754 INFO:teuthology.orchestra.run.smithi115.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 309 kB/s | 43 kB 00:00 2024-07-26T07:33:45.779 INFO:teuthology.orchestra.run.smithi115.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T07:33:45.814 INFO:teuthology.orchestra.run.smithi115.stdout:(116/160): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-07-26T07:33:45.839 INFO:teuthology.orchestra.run.smithi115.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 242 kB/s | 14 kB 00:00 2024-07-26T07:33:45.872 INFO:teuthology.orchestra.run.smithi115.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-07-26T07:33:45.914 INFO:teuthology.orchestra.run.smithi115.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-07-26T07:33:45.933 INFO:teuthology.orchestra.run.smithi181.stdout:(152/160): NetworkManager-libnm-1.48.2-2.el9.x8 2.3 MB/s | 1.8 MB 00:00 2024-07-26T07:33:45.948 INFO:teuthology.orchestra.run.smithi115.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-07-26T07:33:45.981 INFO:teuthology.orchestra.run.smithi115.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 3.2 MB/s | 216 kB 00:00 2024-07-26T07:33:46.015 INFO:teuthology.orchestra.run.smithi115.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T07:33:46.025 INFO:teuthology.orchestra.run.smithi181.stdout:(153/160): NetworkManager-tui-1.48.2-2.el9.x86_ 495 kB/s | 247 kB 00:00 2024-07-26T07:33:46.066 INFO:teuthology.orchestra.run.smithi115.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 218 kB/s | 11 kB 00:00 2024-07-26T07:33:46.091 INFO:teuthology.orchestra.run.smithi115.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 729 kB/s | 18 kB 00:00 2024-07-26T07:33:46.116 INFO:teuthology.orchestra.run.smithi115.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 955 kB/s | 23 kB 00:00 2024-07-26T07:33:46.141 INFO:teuthology.orchestra.run.smithi115.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 805 kB/s | 19 kB 00:00 2024-07-26T07:33:46.167 INFO:teuthology.orchestra.run.smithi115.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 813 kB/s | 20 kB 00:00 2024-07-26T07:33:46.225 INFO:teuthology.orchestra.run.smithi181.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 126 kB/s | 37 kB 00:00 2024-07-26T07:33:46.275 INFO:teuthology.orchestra.run.smithi115.stdout:(128/160): python3-kubernetes-26.1.0-2.el9.noar 9.5 MB/s | 1.0 MB 00:00 2024-07-26T07:33:46.292 INFO:teuthology.orchestra.run.smithi181.stdout:(155/160): NetworkManager-1.48.2-2.el9.x86_64.r 1.7 MB/s | 2.3 MB 00:01 2024-07-26T07:33:46.301 INFO:teuthology.orchestra.run.smithi115.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.9 MB/s | 46 kB 00:00 2024-07-26T07:33:46.334 INFO:teuthology.orchestra.run.smithi115.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.4 MB/s | 79 kB 00:00 2024-07-26T07:33:46.368 INFO:teuthology.orchestra.run.smithi115.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-07-26T07:33:46.435 INFO:teuthology.orchestra.run.smithi115.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-07-26T07:33:46.544 INFO:teuthology.orchestra.run.smithi115.stdout:(133/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-07-26T07:33:46.569 INFO:teuthology.orchestra.run.smithi115.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2024-07-26T07:33:46.661 INFO:teuthology.orchestra.run.smithi115.stdout:(135/160): libarrow-9.0.0-11.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-07-26T07:33:46.678 INFO:teuthology.orchestra.run.smithi115.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 262 kB/s | 28 kB 00:00 2024-07-26T07:33:46.684 INFO:teuthology.orchestra.run.smithi181.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 314 kB/s | 207 kB 00:00 2024-07-26T07:33:46.703 INFO:teuthology.orchestra.run.smithi115.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 742 kB/s | 31 kB 00:00 2024-07-26T07:33:46.726 INFO:teuthology.orchestra.run.smithi181.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 519 kB/s | 260 kB 00:00 2024-07-26T07:33:46.728 INFO:teuthology.orchestra.run.smithi115.stdout:(138/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 568 kB/s | 90 kB 00:00 2024-07-26T07:33:46.782 INFO:teuthology.orchestra.run.smithi115.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 759 kB/s | 59 kB 00:00 2024-07-26T07:33:46.813 INFO:teuthology.orchestra.run.smithi115.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 431 kB/s | 36 kB 00:00 2024-07-26T07:33:46.838 INFO:teuthology.orchestra.run.smithi115.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 937 kB/s | 51 kB 00:00 2024-07-26T07:33:46.871 INFO:teuthology.orchestra.run.smithi115.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 980 kB/s | 188 kB 00:00 2024-07-26T07:33:46.897 INFO:teuthology.orchestra.run.smithi115.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-07-26T07:33:46.922 INFO:teuthology.orchestra.run.smithi115.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 822 kB/s | 20 kB 00:00 2024-07-26T07:33:46.956 INFO:teuthology.orchestra.run.smithi115.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-07-26T07:33:46.989 INFO:teuthology.orchestra.run.smithi115.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-07-26T07:33:47.031 INFO:teuthology.orchestra.run.smithi115.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-07-26T07:33:47.136 INFO:teuthology.orchestra.run.smithi115.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 8.8 MB/s | 1.6 MB 00:00 2024-07-26T07:33:47.299 INFO:teuthology.orchestra.run.smithi115.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 3.4 MB 00:00 2024-07-26T07:33:47.365 INFO:teuthology.orchestra.run.smithi115.stdout:(150/160): NetworkManager-1.48.2-2.el9.x86_64.r 10 MB/s | 2.3 MB 00:00 2024-07-26T07:33:47.441 INFO:teuthology.orchestra.run.smithi115.stdout:(151/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-07-26T07:33:47.525 INFO:teuthology.orchestra.run.smithi115.stdout:(152/160): NetworkManager-libnm-1.48.2-2.el9.x8 8.3 MB/s | 1.8 MB 00:00 2024-07-26T07:33:47.549 INFO:teuthology.orchestra.run.smithi115.stdout:(153/160): NetworkManager-team-1.48.2-2.el9.x86 223 kB/s | 41 kB 00:00 2024-07-26T07:33:47.583 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-07-26T07:33:47.583 INFO:teuthology.orchestra.run.smithi115.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 1.8 MB/s | 247 kB 00:00 2024-07-26T07:33:47.658 INFO:teuthology.orchestra.run.smithi115.stdout:(155/160): elfutils-debuginfod-client-0.191-4.e 347 kB/s | 37 kB 00:00 2024-07-26T07:33:47.683 INFO:teuthology.orchestra.run.smithi115.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 1.5 MB/s | 207 kB 00:00 2024-07-26T07:33:47.717 INFO:teuthology.orchestra.run.smithi115.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 1.9 MB/s | 260 kB 00:00 2024-07-26T07:33:47.801 INFO:teuthology.orchestra.run.smithi115.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 9.6 MB/s | 1.4 MB 00:00 2024-07-26T07:33:47.871 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-07-26T07:33:47.872 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-07-26T07:33:47.884 INFO:teuthology.orchestra.run.smithi115.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-07-26T07:33:47.910 INFO:teuthology.orchestra.run.smithi181.stdout:(158/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 937 kB/s | 1.1 MB 00:01 2024-07-26T07:33:47.935 INFO:teuthology.orchestra.run.smithi115.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 5.0 MB/s | 1.1 MB 00:00 2024-07-26T07:33:47.950 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-07-26T07:33:47.950 INFO:teuthology.orchestra.run.smithi115.stdout:Total 18 MB/s | 233 MB 00:12 2024-07-26T07:33:48.448 INFO:teuthology.orchestra.run.smithi181.stdout:(159/160): openssl-3.2.2-2.el9.x86_64.rpm 643 kB/s | 1.4 MB 00:02 2024-07-26T07:33:49.623 INFO:teuthology.orchestra.run.smithi181.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 833 kB/s | 2.4 MB 00:02 2024-07-26T07:33:49.635 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-07-26T07:33:49.636 INFO:teuthology.orchestra.run.smithi181.stdout:Total 12 MB/s | 233 MB 00:19 2024-07-26T07:33:50.499 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-07-26T07:33:50.516 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-07-26T07:33:51.363 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-07-26T07:33:51.664 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-07-26T07:33:51.664 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-07-26T07:33:52.859 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-07-26T07:33:53.148 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-07-26T07:33:53.148 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-07-26T07:33:54.237 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-07-26T07:33:54.238 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-07-26T07:33:55.802 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-07-26T07:33:55.802 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-07-26T07:33:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-07-26T07:33:57.434 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T07:33:57.511 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T07:33:57.613 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T07:33:57.694 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T07:33:57.730 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T07:33:57.836 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T07:33:58.271 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T07:33:58.385 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:33:58.550 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:33:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:33:58.729 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:33:58.769 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T07:33:58.838 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T07:33:59.142 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T07:33:59.216 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T07:33:59.241 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:33:59.380 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:33:59.409 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T07:33:59.486 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T07:33:59.527 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T07:33:59.600 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T07:33:59.831 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T07:34:00.009 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T07:34:00.381 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T07:34:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T07:34:00.688 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T07:34:00.857 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:00.971 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:01.245 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-07-26T07:34:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:01.367 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T07:34:01.446 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T07:34:01.465 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T07:34:01.502 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T07:34:01.551 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T07:34:01.566 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T07:34:01.645 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T07:34:01.665 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T07:34:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T07:34:01.727 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T07:34:01.774 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T07:34:01.786 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T07:34:01.838 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T07:34:01.884 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T07:34:01.917 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T07:34:02.028 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T07:34:02.108 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T07:34:02.161 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T07:34:02.219 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T07:34:02.220 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T07:34:02.326 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T07:34:02.358 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:34:02.364 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T07:34:02.426 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T07:34:02.461 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T07:34:02.536 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:34:02.566 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:34:02.571 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T07:34:02.633 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T07:34:02.698 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:34:02.732 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T07:34:02.740 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T07:34:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T07:34:02.940 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-07-26T07:34:03.037 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T07:34:03.042 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T07:34:03.103 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T07:34:03.120 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T07:34:03.128 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T07:34:03.189 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T07:34:03.213 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:34:03.260 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T07:34:03.322 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T07:34:03.362 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T07:34:03.386 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:34:03.439 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T07:34:03.503 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T07:34:03.533 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T07:34:03.565 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T07:34:03.620 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T07:34:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T07:34:03.884 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T07:34:03.944 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T07:34:04.014 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T07:34:04.015 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T07:34:04.015 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T07:34:04.071 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T07:34:04.102 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:34:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T07:34:04.140 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T07:34:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T07:34:04.279 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:34:04.325 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:34:04.412 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T07:34:04.459 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:34:04.482 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T07:34:04.490 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T07:34:04.495 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T07:34:04.547 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T07:34:04.602 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T07:34:04.675 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T07:34:04.699 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T07:34:04.758 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T07:34:04.786 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T07:34:04.806 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T07:34:04.814 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T07:34:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T07:34:04.880 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T07:34:04.905 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:34:04.930 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T07:34:04.988 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T07:34:05.035 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:05.051 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T07:34:05.053 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:34:05.090 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T07:34:05.118 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T07:34:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T07:34:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T07:34:05.259 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T07:34:05.322 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T07:34:05.373 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T07:34:05.373 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:05.425 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T07:34:05.556 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T07:34:05.569 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T07:34:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T07:34:05.596 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T07:34:05.608 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T07:34:05.648 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T07:34:05.659 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T07:34:05.687 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T07:34:05.745 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T07:34:05.756 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T07:34:05.768 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T07:34:05.835 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T07:34:05.904 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T07:34:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T07:34:05.979 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T07:34:06.025 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T07:34:06.065 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T07:34:06.095 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T07:34:06.134 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T07:34:06.162 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T07:34:06.172 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T07:34:06.184 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T07:34:06.209 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T07:34:06.264 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T07:34:06.346 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T07:34:06.346 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T07:34:06.346 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T07:34:06.352 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T07:34:06.362 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T07:34:06.393 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T07:34:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T07:34:06.446 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T07:34:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T07:34:06.501 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T07:34:06.513 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T07:34:06.537 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T07:34:06.557 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T07:34:06.595 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T07:34:06.607 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:06.643 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T07:34:06.696 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T07:34:06.713 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:06.762 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T07:34:06.763 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T07:34:06.806 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T07:34:06.821 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T07:34:06.921 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T07:34:06.958 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T07:34:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T07:34:07.058 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T07:34:07.249 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T07:34:07.285 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T07:34:07.338 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T07:34:07.393 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T07:34:07.443 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T07:34:07.466 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T07:34:07.502 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T07:34:07.548 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T07:34:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T07:34:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T07:34:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T07:34:07.792 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T07:34:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T07:34:07.928 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T07:34:07.970 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T07:34:07.987 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T07:34:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T07:34:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T07:34:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T07:34:08.127 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T07:34:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T07:34:08.180 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T07:34:08.204 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T07:34:08.231 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T07:34:08.239 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T07:34:08.250 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T07:34:08.268 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T07:34:08.306 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T07:34:08.316 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T07:34:08.389 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T07:34:08.403 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T07:34:08.415 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T07:34:08.453 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T07:34:08.461 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T07:34:08.461 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T07:34:08.538 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T07:34:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T07:34:08.684 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T07:34:08.716 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T07:34:08.735 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T07:34:08.750 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T07:34:08.813 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T07:34:08.820 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T07:34:08.830 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T07:34:08.959 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T07:34:09.044 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T07:34:09.164 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T07:34:09.214 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T07:34:09.253 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T07:34:09.322 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T07:34:09.370 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T07:34:09.433 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T07:34:09.493 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T07:34:09.498 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T07:34:09.593 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T07:34:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T07:34:09.629 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T07:34:09.670 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T07:34:09.691 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T07:34:09.731 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T07:34:09.745 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T07:34:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T07:34:09.777 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T07:34:09.810 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T07:34:09.891 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T07:34:09.949 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T07:34:09.995 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T07:34:10.048 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T07:34:10.086 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T07:34:10.119 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T07:34:10.121 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T07:34:10.183 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T07:34:10.320 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T07:34:10.326 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T07:34:10.404 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T07:34:10.491 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T07:34:10.502 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T07:34:10.534 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T07:34:10.549 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T07:34:10.572 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T07:34:10.578 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T07:34:10.616 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T07:34:10.662 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T07:34:10.671 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T07:34:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T07:34:10.726 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T07:34:10.769 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T07:34:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T07:34:10.814 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T07:34:10.844 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T07:34:10.898 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T07:34:10.976 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T07:34:11.022 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T07:34:11.041 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T07:34:11.091 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T07:34:11.145 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T07:34:11.162 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T07:34:11.251 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T07:34:11.278 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T07:34:11.296 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T07:34:11.314 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T07:34:11.321 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T07:34:11.369 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T07:34:11.406 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T07:34:11.414 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T07:34:11.439 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T07:34:11.449 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T07:34:11.450 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T07:34:11.485 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T07:34:11.525 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T07:34:11.614 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T07:34:11.801 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T07:34:11.845 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T07:34:11.882 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T07:34:11.883 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T07:34:11.920 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T07:34:11.935 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T07:34:11.976 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T07:34:12.030 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T07:34:12.074 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T07:34:12.118 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T07:34:12.187 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T07:34:12.187 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T07:34:12.187 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T07:34:12.213 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T07:34:12.261 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T07:34:12.309 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T07:34:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T07:34:12.504 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T07:34:12.515 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T07:34:12.565 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T07:34:12.586 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T07:34:12.599 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T07:34:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T07:34:12.746 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T07:34:12.780 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T07:34:12.812 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T07:34:12.815 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T07:34:12.821 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T07:34:12.870 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T07:34:12.948 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T07:34:12.973 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T07:34:12.977 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T07:34:13.026 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T07:34:13.255 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T07:34:13.289 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T07:34:13.421 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T07:34:13.790 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T07:34:13.845 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T07:34:13.878 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T07:34:14.009 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T07:34:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T07:34:14.151 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T07:34:14.248 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T07:34:14.268 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T07:34:14.306 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T07:34:14.395 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T07:34:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T07:34:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T07:34:14.664 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T07:34:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T07:34:15.557 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:15.633 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T07:34:15.962 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T07:34:16.061 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T07:34:16.098 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T07:34:16.132 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T07:34:16.258 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:16.306 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T07:34:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T07:34:16.461 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T07:34:16.461 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T07:34:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:16.499 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T07:34:16.567 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T07:34:16.615 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T07:34:16.747 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T07:34:16.781 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T07:34:16.838 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T07:34:17.230 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T07:34:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T07:34:17.362 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T07:34:17.398 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T07:34:17.469 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T07:34:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T07:34:17.637 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T07:34:17.670 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T07:34:17.786 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T07:34:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T07:34:18.020 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T07:34:18.056 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T07:34:18.116 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T07:34:18.700 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T07:34:18.734 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T07:34:18.826 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T07:34:18.852 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T07:34:20.287 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:20.333 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:20.988 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:21.028 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T07:34:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T07:34:21.211 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T07:34:21.211 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T07:34:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:21.253 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T07:34:21.463 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:21.507 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:22.169 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T07:34:22.219 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T07:34:22.360 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T07:34:22.361 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T07:34:22.361 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T07:34:22.361 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:22.395 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T07:34:29.320 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T07:34:29.320 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-07-26T07:34:29.320 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-07-26T07:34:29.320 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-07-26T07:34:29.320 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-07-26T07:34:29.320 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-07-26T07:34:29.320 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2024-07-26T07:34:29.320 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-07-26T07:34:29.321 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:29.552 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T07:34:29.598 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T07:34:29.598 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:29.598 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T07:34:29.598 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:31.312 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T07:34:31.417 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T07:34:31.418 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:31.418 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T07:34:31.418 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T07:34:31.418 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T07:34:31.422 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:31.679 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T07:34:31.770 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T07:34:31.771 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:31.771 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T07:34:31.771 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T07:34:31.771 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T07:34:31.771 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:32.288 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T07:34:32.381 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T07:34:32.381 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:32.381 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T07:34:32.381 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T07:34:32.381 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T07:34:32.381 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:32.521 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T07:34:32.585 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T07:34:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T07:34:32.905 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T07:34:32.984 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T07:34:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T07:34:33.414 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T07:34:33.595 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T07:34:33.643 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T07:34:33.686 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T07:34:33.716 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T07:34:33.830 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T07:34:33.830 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2024-07-26T07:34:33.830 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2024-07-26T07:34:33.830 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2024-07-26T07:34:33.830 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2024-07-26T07:34:33.830 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2024-07-26T07:34:33.830 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /root 2024-07-26T07:34:33.831 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2024-07-26T07:34:33.831 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:34.073 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T07:34:34.122 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T07:34:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:34.122 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T07:34:34.122 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:34.373 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T07:34:34.410 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T07:34:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T07:34:34.579 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T07:34:34.613 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T07:34:35.337 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T07:34:35.337 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-07-26T07:34:35.337 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-07-26T07:34:35.337 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-07-26T07:34:35.337 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-07-26T07:34:35.337 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-07-26T07:34:35.337 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-07-26T07:34:35.337 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-07-26T07:34:35.338 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:35.612 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T07:34:35.661 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T07:34:35.661 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:35.661 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T07:34:35.661 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:35.849 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T07:34:35.930 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T07:34:35.930 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:35.930 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T07:34:35.930 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T07:34:35.930 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T07:34:35.930 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:36.191 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T07:34:36.269 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T07:34:36.269 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:36.269 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T07:34:36.269 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T07:34:36.269 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T07:34:36.269 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:36.775 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T07:34:36.853 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T07:34:36.853 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:36.853 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T07:34:36.853 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T07:34:36.853 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T07:34:36.853 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:36.983 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T07:34:37.047 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T07:34:37.121 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T07:34:37.353 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T07:34:37.401 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T07:34:37.412 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T07:34:37.478 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T07:34:37.479 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:37.479 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T07:34:37.479 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T07:34:37.479 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T07:34:37.479 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:37.498 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T07:34:37.607 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T07:34:37.735 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T07:34:37.772 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T07:34:37.804 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T07:34:37.805 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:37.805 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T07:34:37.805 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T07:34:37.805 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T07:34:37.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:37.829 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T07:34:37.868 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T07:34:37.894 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T07:34:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T07:34:38.336 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T07:34:38.337 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T07:34:38.405 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T07:34:38.405 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:38.405 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T07:34:38.405 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T07:34:38.405 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T07:34:38.405 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:38.535 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T07:34:38.578 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T07:34:38.605 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T07:34:38.655 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T07:34:38.681 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T07:34:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T07:34:38.828 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T07:34:38.858 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T07:34:38.933 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T07:34:38.990 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T07:34:39.075 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T07:34:39.183 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T07:34:39.347 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T07:34:39.389 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T07:34:39.426 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T07:34:39.437 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T07:34:39.461 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T07:34:39.506 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T07:34:39.622 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T07:34:39.797 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T07:34:39.830 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T07:34:39.900 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T07:34:39.900 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:39.900 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T07:34:39.900 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T07:34:39.900 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T07:34:39.900 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T07:34:40.003 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T07:34:40.076 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T07:34:40.116 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T07:34:40.154 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T07:34:40.168 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T07:34:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T07:34:40.272 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T07:34:40.310 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T07:34:41.282 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T07:34:41.317 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T07:34:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T07:34:41.387 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:41.387 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T07:34:41.387 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T07:34:41.387 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T07:34:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:41.460 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T07:34:41.530 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T07:34:41.531 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:41.531 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T07:34:41.531 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:41.602 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T07:34:41.677 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T07:34:41.677 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:41.677 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T07:34:41.677 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T07:34:41.677 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T07:34:41.677 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:42.086 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T07:34:42.156 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T07:34:42.156 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:42.156 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T07:34:42.156 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T07:34:42.157 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T07:34:42.157 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:42.550 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T07:34:42.622 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T07:34:43.719 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T07:34:43.746 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T07:34:43.909 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T07:34:43.983 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T07:34:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T07:34:44.117 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T07:34:44.149 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T07:34:44.218 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T07:34:44.218 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:44.218 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T07:34:44.218 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T07:34:44.218 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T07:34:44.218 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:44.307 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T07:34:44.346 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T07:34:44.477 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T07:34:44.547 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T07:34:45.129 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T07:34:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T07:34:45.315 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T07:34:45.509 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T07:34:45.543 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T07:34:45.610 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T07:34:45.610 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:45.611 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T07:34:45.611 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T07:34:45.611 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T07:34:45.611 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:45.623 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T07:34:45.652 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T07:34:45.694 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T07:34:45.724 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T07:34:45.724 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:45.724 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T07:34:45.724 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T07:34:45.724 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T07:34:45.724 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:45.731 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T07:34:45.778 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T07:34:45.846 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T07:34:45.847 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:45.847 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T07:34:45.847 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:45.883 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T07:34:45.911 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T07:34:45.959 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T07:34:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T07:34:46.003 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:46.003 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T07:34:46.003 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T07:34:46.004 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T07:34:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:46.435 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T07:34:46.530 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T07:34:46.531 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:46.531 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T07:34:46.531 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T07:34:46.531 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T07:34:46.531 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:46.916 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T07:34:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T07:34:47.022 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T07:34:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T07:34:47.055 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T07:34:47.057 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T07:34:47.091 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T07:34:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T07:34:47.122 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:47.122 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T07:34:47.122 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T07:34:47.123 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T07:34:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T07:34:47.264 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T07:34:47.264 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:47.264 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T07:34:47.265 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:47.339 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T07:34:47.361 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T07:34:47.415 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T07:34:47.416 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:47.416 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T07:34:47.416 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T07:34:47.416 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T07:34:47.416 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:47.421 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T07:34:47.455 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T07:34:47.494 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T07:34:47.541 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T07:34:47.587 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T07:34:47.790 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T07:34:47.859 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T07:34:47.859 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T07:34:47.859 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T07:34:47.859 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T07:34:47.860 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T07:34:47.860 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:48.520 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:50.028 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T07:34:50.135 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T07:34:50.190 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T07:34:50.321 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T07:34:50.369 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T07:34:50.403 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T07:34:50.437 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T07:34:50.487 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T07:34:50.487 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T07:34:50.620 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T07:34:50.674 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T07:34:50.675 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T07:34:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T07:34:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T07:34:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:51.184 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:51.185 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T07:34:51.335 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T07:34:51.338 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T07:34:51.383 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T07:34:51.413 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T07:34:51.421 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T07:34:51.469 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T07:34:51.509 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T07:34:51.548 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T07:34:51.552 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:34:51.815 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:34:51.815 INFO:teuthology.orchestra.run.smithi097.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T07:34:51.816 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:34:51.861 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T07:34:51.914 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T07:34:51.949 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T07:34:51.995 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T07:34:52.044 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T07:34:52.113 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T07:34:52.647 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T07:34:52.705 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T07:34:52.776 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T07:34:52.813 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T07:34:52.851 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T07:34:53.036 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:53.084 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:53.139 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:53.158 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T07:34:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T07:34:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T07:34:53.280 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T07:34:53.320 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T07:34:53.390 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T07:34:54.250 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:54.320 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:54.387 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T07:34:54.519 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T07:34:54.617 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T07:34:54.681 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T07:34:54.805 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T07:34:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T07:34:54.907 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T07:34:54.945 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T07:34:54.987 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T07:34:54.988 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T07:34:55.128 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T07:34:55.182 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T07:34:55.182 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T07:34:55.357 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T07:34:55.394 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T07:34:55.430 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:55.431 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:55.753 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:55.754 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T07:34:55.764 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T07:34:55.879 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T07:34:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T07:34:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T07:34:55.945 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T07:34:55.982 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T07:34:56.087 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T07:34:56.113 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:34:56.129 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T07:34:56.163 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T07:34:56.198 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T07:34:56.240 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T07:34:56.240 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T07:34:56.373 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:34:56.373 INFO:teuthology.orchestra.run.smithi115.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T07:34:56.373 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:34:56.392 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T07:34:56.472 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T07:34:56.472 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T07:34:56.609 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T07:34:56.648 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T07:34:56.691 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:56.691 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:56.995 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T07:34:56.995 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T07:34:57.123 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T07:34:57.168 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T07:34:57.231 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T07:34:57.363 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:34:57.623 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:34:57.623 INFO:teuthology.orchestra.run.smithi181.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T07:34:57.623 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T07:34:58.322 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T07:34:58.323 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T07:34:58.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T07:34:58.325 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T07:34:58.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T07:34:58.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T07:34:58.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T07:34:58.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T07:34:58.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T07:34:58.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T07:34:58.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T07:34:58.333 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T07:34:58.334 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T07:34:58.335 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T07:35:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:35:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:35:00.597 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-07-26T07:35:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-07-26T07:35:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T07:35:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T07:35:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T07:35:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T07:35:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T07:35:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T07:35:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T07:35:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T07:35:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T07:35:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T07:35:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T07:35:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T07:35:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T07:35:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T07:35:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T07:35:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T07:35:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T07:35:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T07:35:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T07:35:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:35:00.611 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-07-26T07:35:01.232 DEBUG:teuthology.parallel:result is None 2024-07-26T07:35:02.881 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T07:35:02.881 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T07:35:02.881 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T07:35:02.881 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T07:35:02.881 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T07:35:02.881 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T07:35:02.881 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T07:35:02.882 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T07:35:02.883 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T07:35:02.883 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T07:35:02.883 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T07:35:02.883 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T07:35:02.883 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T07:35:02.883 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T07:35:02.895 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T07:35:02.895 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T07:35:02.895 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T07:35:02.895 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T07:35:02.895 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T07:35:02.896 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T07:35:02.898 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T07:35:02.898 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T07:35:02.898 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T07:35:02.898 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T07:35:02.898 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T07:35:02.898 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T07:35:02.898 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T07:35:02.899 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T07:35:02.899 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T07:35:02.899 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T07:35:02.899 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T07:35:02.899 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T07:35:02.899 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T07:35:02.899 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T07:35:02.899 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T07:35:02.900 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T07:35:02.901 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T07:35:02.902 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T07:35:02.903 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T07:35:02.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T07:35:02.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T07:35:02.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T07:35:02.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T07:35:02.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T07:35:02.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T07:35:02.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T07:35:02.904 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T07:35:02.905 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T07:35:02.906 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T07:35:02.907 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T07:35:02.908 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T07:35:02.908 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T07:35:02.908 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T07:35:02.908 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T07:35:02.909 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T07:35:02.909 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T07:35:02.909 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T07:35:02.910 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T07:35:02.911 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T07:35:02.911 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T07:35:02.911 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T07:35:02.911 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T07:35:02.911 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T07:35:02.911 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T07:35:03.974 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T07:35:03.974 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T07:35:03.974 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T07:35:03.974 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T07:35:03.974 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T07:35:03.974 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T07:35:03.975 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T07:35:03.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T07:35:03.977 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T07:35:03.978 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T07:35:03.979 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T07:35:03.980 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T07:35:03.981 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T07:35:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T07:35:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T07:35:03.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T07:35:03.985 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T07:35:03.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T07:35:03.987 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T07:35:03.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T07:35:03.992 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T07:35:03.993 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T07:35:03.993 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T07:35:05.333 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T07:35:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.335 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T07:35:05.336 INFO:teuthology.orchestra.run.smithi115.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.337 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T07:35:05.338 INFO:teuthology.orchestra.run.smithi115.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T07:35:05.339 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T07:35:05.340 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T07:35:05.341 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T07:35:05.342 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T07:35:05.343 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T07:35:05.344 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T07:35:05.345 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T07:35:05.346 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T07:35:05.346 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-07-26T07:35:05.346 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-07-26T07:35:06.088 DEBUG:teuthology.parallel:result is None 2024-07-26T07:35:06.406 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T07:35:06.406 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:35:06.407 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-07-26T07:35:06.407 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:06.407 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:06.407 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:06.407 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T07:35:06.407 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T07:35:06.407 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T07:35:06.407 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.408 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.409 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T07:35:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T07:35:06.411 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T07:35:06.412 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T07:35:06.413 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T07:35:06.415 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T07:35:06.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T07:35:06.417 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T07:35:06.418 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.419 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T07:35:06.420 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T07:35:06.420 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T07:35:06.420 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T07:35:06.420 INFO:teuthology.orchestra.run.smithi181.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T07:35:06.420 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T07:35:06.420 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T07:35:06.421 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T07:35:06.421 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:35:06.421 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-07-26T07:35:07.057 DEBUG:teuthology.parallel:result is None 2024-07-26T07:35:07.058 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:35:07.219 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T07:35:07.292 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T07:35:07.292 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T07:35:07.293 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T07:35:07.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:35:07.439 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T07:35:07.488 INFO:teuthology.orchestra.run.smithi115.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T07:35:07.488 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T07:35:07.489 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T07:35:07.490 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:35:07.632 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T07:35:07.681 INFO:teuthology.orchestra.run.smithi181.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T07:35:07.681 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T07:35:07.681 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T07:35:07.683 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-26T07:35:07.684 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:35:07.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T07:35:07.775 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:35:07.776 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T07:35:07.836 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:35:07.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T07:35:07.898 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-26T07:35:07.898 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:35:07.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T07:35:07.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T07:35:08.069 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:35:08.069 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T07:35:08.131 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T07:35:08.238 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:35:08.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T07:35:08.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T07:35:08.404 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-26T07:35:08.404 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:35:08.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T07:35:08.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T07:35:08.623 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:35:08.623 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T07:35:08.692 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T07:35:08.813 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:35:08.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T07:35:08.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T07:35:08.982 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-26T07:35:08.982 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:35:08.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T07:35:09.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T07:35:09.144 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:35:09.145 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T07:35:09.214 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T07:35:09.338 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:35:09.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T07:35:09.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T07:35:09.507 INFO:teuthology.run_tasks:Running task cephadm... 2024-07-26T07:35:09.655 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'mds': {'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-07-26T07:35:09.655 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:35:09.656 INFO:tasks.cephadm:Cluster fsid is 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:35:09.656 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-07-26T07:35:09.656 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.97', 'mon.b': '172.21.15.115', 'mon.c': '172.21.15.181'} 2024-07-26T07:35:09.657 INFO:tasks.cephadm:First mon is mon.a on smithi097 2024-07-26T07:35:09.657 INFO:tasks.cephadm:First mgr is x 2024-07-26T07:35:09.657 INFO:tasks.cephadm:Normalizing hostnames... 2024-07-26T07:35:09.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2024-07-26T07:35:09.746 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2024-07-26T07:35:09.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-07-26T07:35:09.905 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-07-26T07:35:09.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:35:10.051 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 16:18:49.953482', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['arm64', 'x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 15:10:42.298781', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-07-26T07:35:10.193 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240725.121331-debug, sha1 e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:35:10.194 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T07:35:10.194 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T07:35:10.194 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:10.620 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 07:35 /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:10.620 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:11.099 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 07:35 /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:11.099 DEBUG:teuthology.orchestra.run.smithi181:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:11.525 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 07:35 /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:11.525 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:11.587 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:11.647 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T07:35:11.739 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 on all hosts... 2024-07-26T07:35:11.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T07:35:11.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T07:35:11.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T07:35:12.937 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T07:35:13.011 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T07:35:13.393 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T07:35:56.579 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-07-26T07:35:56.580 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T07:35:56.580 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T07:35:56.580 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2024-07-26T07:35:56.580 INFO:teuthology.orchestra.run.smithi181.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T07:35:56.581 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-07-26T07:35:56.581 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-07-26T07:35:58.441 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:35:58.441 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T07:35:58.442 INFO:teuthology.orchestra.run.smithi097.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T07:35:58.442 INFO:teuthology.orchestra.run.smithi097.stdout: "repo_digests": [ 2024-07-26T07:35:58.442 INFO:teuthology.orchestra.run.smithi097.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T07:35:58.442 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-07-26T07:35:58.442 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:35:59.991 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-07-26T07:35:59.991 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T07:35:59.991 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T07:35:59.991 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2024-07-26T07:35:59.992 INFO:teuthology.orchestra.run.smithi115.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T07:35:59.992 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-07-26T07:35:59.992 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-07-26T07:36:00.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2024-07-26T07:36:00.152 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2024-07-26T07:36:00.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-07-26T07:36:00.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2024-07-26T07:36:00.423 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2024-07-26T07:36:00.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-07-26T07:36:00.599 INFO:tasks.cephadm:Writing seed config... 2024-07-26T07:36:00.615 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-07-26T07:36:00.615 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-07-26T07:36:00.615 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-07-26T07:36:00.615 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-07-26T07:36:00.615 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-07-26T07:36:00.615 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-07-26T07:36:00.615 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-07-26T07:36:00.616 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-07-26T07:36:00.617 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-07-26T07:36:00.617 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-07-26T07:36:00.617 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-07-26T07:36:00.617 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-07-26T07:36:00.617 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-07-26T07:36:00.617 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-07-26T07:36:00.617 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-07-26T07:36:00.618 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:36:00.618 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-07-26T07:36:00.675 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9621a072-4b21-11ef-bcb6-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [mds] mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 2024-07-26T07:36:00.675 DEBUG:teuthology.orchestra.run.smithi097:mon.a> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.a.service 2024-07-26T07:36:00.680 DEBUG:teuthology.orchestra.run.smithi097:mgr.x> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mgr.x.service 2024-07-26T07:36:00.725 INFO:tasks.cephadm:Bootstrapping... 2024-07-26T07:36:00.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 -v bootstrap --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.97 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-07-26T07:36:01.007 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-07-26T07:36:01.007 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', '-v', 'bootstrap', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.97', '--skip-admin-label'] 2024-07-26T07:36:01.007 INFO:teuthology.orchestra.run.smithi097.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-07-26T07:36:01.008 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying podman|docker is present... 2024-07-26T07:36:01.065 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stdout 5.1.2 2024-07-26T07:36:01.065 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying lvm2 is present... 2024-07-26T07:36:01.065 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying time synchronization is in place... 2024-07-26T07:36:01.082 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T07:36:01.082 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T07:36:01.103 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T07:36:01.103 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-07-26T07:36:01.125 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout enabled 2024-07-26T07:36:01.145 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout active 2024-07-26T07:36:01.145 INFO:teuthology.orchestra.run.smithi097.stdout:Unit chronyd.service is enabled and running 2024-07-26T07:36:01.146 INFO:teuthology.orchestra.run.smithi097.stdout:Repeating the final host check... 2024-07-26T07:36:01.202 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stdout 5.1.2 2024-07-26T07:36:01.203 INFO:teuthology.orchestra.run.smithi097.stdout:podman (/bin/podman) version 5.1.2 is present 2024-07-26T07:36:01.203 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl is present 2024-07-26T07:36:01.203 INFO:teuthology.orchestra.run.smithi097.stdout:lvcreate is present 2024-07-26T07:36:01.219 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T07:36:01.219 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T07:36:01.234 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T07:36:01.234 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-07-26T07:36:01.253 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout enabled 2024-07-26T07:36:01.268 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout active 2024-07-26T07:36:01.268 INFO:teuthology.orchestra.run.smithi097.stdout:Unit chronyd.service is enabled and running 2024-07-26T07:36:01.269 INFO:teuthology.orchestra.run.smithi097.stdout:Host looks OK 2024-07-26T07:36:01.269 INFO:teuthology.orchestra.run.smithi097.stdout:Cluster fsid: 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:01.269 INFO:teuthology.orchestra.run.smithi097.stdout:Acquiring lock 140259314824000 on /run/cephadm/9621a072-4b21-11ef-bcb6-c7b262605968.lock 2024-07-26T07:36:01.269 INFO:teuthology.orchestra.run.smithi097.stdout:Lock 140259314824000 acquired on /run/cephadm/9621a072-4b21-11ef-bcb6-c7b262605968.lock 2024-07-26T07:36:01.269 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying IP 172.21.15.97 port 3300 ... 2024-07-26T07:36:01.270 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying IP 172.21.15.97 port 6789 ... 2024-07-26T07:36:01.270 INFO:teuthology.orchestra.run.smithi097.stdout:Base mon IP(s) is [172.21.15.97:3300, 172.21.15.97:6789], mon addrv is [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2024-07-26T07:36:01.278 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.97 metric 100 2024-07-26T07:36:01.278 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.97 metric 100 2024-07-26T07:36:01.286 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-07-26T07:36:01.286 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-07-26T07:36:01.286 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2024-07-26T07:36:01.293 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-07-26T07:36:01.293 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-07-26T07:36:01.293 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T07:36:01.293 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-07-26T07:36:01.293 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d805/64 scope link 2024-07-26T07:36:01.293 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T07:36:01.294 INFO:teuthology.orchestra.run.smithi097.stdout:Mon IP `172.21.15.97` is in CIDR network `172.21.0.0/20` 2024-07-26T07:36:01.294 INFO:teuthology.orchestra.run.smithi097.stdout:Mon IP `172.21.15.97` is in CIDR network `172.21.0.0/20` 2024-07-26T07:36:01.294 INFO:teuthology.orchestra.run.smithi097.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-07-26T07:36:01.295 INFO:teuthology.orchestra.run.smithi097.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-07-26T07:36:01.296 INFO:teuthology.orchestra.run.smithi097.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T07:36:02.104 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stdout ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T07:36:02.104 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T07:36:02.104 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Getting image source signatures 2024-07-26T07:36:02.105 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Copying blob sha256:27b44fafaf0d8ddf6712bf077b0e069fd1632e81c2016bcdf9194ad64e762303 2024-07-26T07:36:02.105 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Copying blob sha256:43735fbce9214f061a6894341d7822563f5aafe16fc7532189f4415cff6d2d91 2024-07-26T07:36:02.105 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Copying config sha256:ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T07:36:02.105 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Writing manifest to image destination 2024-07-26T07:36:03.326 INFO:teuthology.orchestra.run.smithi097.stdout:ceph: stdout ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:36:03.327 INFO:teuthology.orchestra.run.smithi097.stdout:Ceph version: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:36:03.327 INFO:teuthology.orchestra.run.smithi097.stdout:Extracting ceph user uid/gid from container image... 2024-07-26T07:36:04.545 INFO:teuthology.orchestra.run.smithi097.stdout:stat: stdout 167 167 2024-07-26T07:36:04.545 INFO:teuthology.orchestra.run.smithi097.stdout:Creating initial keys... 2024-07-26T07:36:05.682 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph-authtool: stdout AQDlUaNmHeQVBBAAqEH7dpXu4I0GdSYdEoCaqg== 2024-07-26T07:36:06.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph-authtool: stdout AQDmUaNmhz+vDRAA4U2FOipaVnvdbESBitKvDg== 2024-07-26T07:36:07.950 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph-authtool: stdout AQDnUaNm5B6VExAAbZ+eEm0J7S/6ICj6311QGg== 2024-07-26T07:36:07.953 INFO:teuthology.orchestra.run.smithi097.stdout:Creating initial monmap... 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool for a [v2:172.21.15.97:3300,v1:172.21.15.97:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:setting min_mon_release = quincy 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: set fsid to 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:36:09.084 INFO:teuthology.orchestra.run.smithi097.stdout:Creating mon... 2024-07-26T07:36:10.356 INFO:teuthology.orchestra.run.smithi097.stdout:create mon.a on 2024-07-26T07:36:10.735 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-07-26T07:36:10.977 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-07-26T07:36:11.272 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9621a072-4b21-11ef-bcb6-c7b262605968.target → /etc/systemd/system/ceph-9621a072-4b21-11ef-bcb6-c7b262605968.target. 2024-07-26T07:36:11.272 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9621a072-4b21-11ef-bcb6-c7b262605968.target → /etc/systemd/system/ceph-9621a072-4b21-11ef-bcb6-c7b262605968.target. 2024-07-26T07:36:11.665 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.a 2024-07-26T07:36:11.666 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.a.service: Unit ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.a.service not loaded. 2024-07-26T07:36:11.977 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9621a072-4b21-11ef-bcb6-c7b262605968.target.wants/ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.a.service → /etc/systemd/system/ceph-9621a072-4b21-11ef-bcb6-c7b262605968@.service. 2024-07-26T07:36:12.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:12 smithi097 podman[31841]: 2024-07-26 07:36:12.632980114 +0000 UTC m=+0.386869732 container init 54f2f715762b4cf69cad5b964be2c1c0133c44d02db3b3abfbaf6fb590362120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:36:12.766 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T07:36:12.766 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout disabled 2024-07-26T07:36:12.782 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T07:36:12.782 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-07-26T07:36:12.782 INFO:teuthology.orchestra.run.smithi097.stdout:firewalld.service is not enabled 2024-07-26T07:36:12.782 INFO:teuthology.orchestra.run.smithi097.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T07:36:12.783 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mon to start... 2024-07-26T07:36:12.783 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mon... 2024-07-26T07:36:12.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:12 smithi097 podman[31841]: 2024-07-26 07:36:12.642425953 +0000 UTC m=+0.396315487 container start 54f2f715762b4cf69cad5b964be2c1c0133c44d02db3b3abfbaf6fb590362120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T07:36:12.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:12 smithi097 bash[31841]: 54f2f715762b4cf69cad5b964be2c1c0133c44d02db3b3abfbaf6fb590362120 2024-07-26T07:36:12.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:12 smithi097 systemd[1]: Started Ceph mon.a for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:36:12.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:12 smithi097 ceph-mon[31860]: mkfs 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:12.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:12 smithi097 ceph-mon[31860]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: monmap epoch 1 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: last_changed 2024-07-26T07:36:08.466808+0000 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: created 2024-07-26T07:36:08.466808+0000 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: min_mon_release 19 (squid) 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: election_strategy: 1 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: fsmap 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: mgrmap e1: no daemons active 2024-07-26T07:36:14.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:13 smithi097 ceph-mon[31860]: from='client.? 172.21.15.97:0/380941830' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-07-26T07:36:14.147 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout cluster: 2024-07-26T07:36:14.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout id: 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:14.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-07-26T07:36:14.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:14.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout services: 2024-07-26T07:36:14.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.689834s) 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout data: 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout pgs: 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:mon is available 2024-07-26T07:36:14.150 INFO:teuthology.orchestra.run.smithi097.stdout:Assimilating anything we can from ceph.conf... 2024-07-26T07:36:15.210 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:14 smithi097 ceph-mon[31860]: from='client.? 172.21.15.97:0/4107744147' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T07:36:15.210 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:14 smithi097 ceph-mon[31860]: from='client.? 172.21.15.97:0/4107744147' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout fsid = 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:15.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T07:36:15.472 INFO:teuthology.orchestra.run.smithi097.stdout:Generating new minimal ceph.conf... 2024-07-26T07:36:16.483 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:16 smithi097 ceph-mon[31860]: from='client.? 172.21.15.97:0/1203454845' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:36:16.752 INFO:teuthology.orchestra.run.smithi097.stdout:Restarting the monitor... 2024-07-26T07:36:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:16 smithi097 systemd[1]: Stopping Ceph mon.a for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:36:17.473 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:17 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a[31856]: 2024-07-26T07:36:17.204+0000 7ff55a600640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-07-26T07:36:17.474 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:17 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a[31856]: 2024-07-26T07:36:17.204+0000 7ff55a600640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-26T07:36:17.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:17 smithi097 podman[32100]: 2024-07-26 07:36:17.473161536 +0000 UTC m=+0.400532596 container died 54f2f715762b4cf69cad5b964be2c1c0133c44d02db3b3abfbaf6fb590362120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:36:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:27 smithi097 podman[32100]: 2024-07-26 07:36:27.082185479 +0000 UTC m=+10.009556536 container cleanup 54f2f715762b4cf69cad5b964be2c1c0133c44d02db3b3abfbaf6fb590362120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T07:36:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:27 smithi097 bash[32100]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a 2024-07-26T07:36:27.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:27 smithi097 podman[32111]: 2024-07-26 07:36:27.333677639 +0000 UTC m=+9.843100664 container remove 54f2f715762b4cf69cad5b964be2c1c0133c44d02db3b3abfbaf6fb590362120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-07-26T07:36:27.964 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:27 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.a.service: Deactivated successfully. 2024-07-26T07:36:27.964 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:27 smithi097 systemd[1]: Stopped Ceph mon.a for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:36:27.964 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:27 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.a.service: Consumed 1.832s CPU time. 2024-07-26T07:36:27.964 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:27 smithi097 systemd[1]: Starting Ceph mon.a for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:36:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 podman[32206]: 2024-07-26 07:36:27.973950749 +0000 UTC m=+0.035239421 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:36:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 podman[32206]: 2024-07-26 07:36:28.075575244 +0000 UTC m=+0.136863916 container create aa7ca637e85bdf8539018a4a9ac6d4c8f5f58ecae7272fdf26bbdb8c181e4431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-07-26T07:36:28.449 INFO:teuthology.orchestra.run.smithi097.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-07-26T07:36:28.608 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 podman[32206]: 2024-07-26 07:36:28.337323347 +0000 UTC m=+0.398612061 container init aa7ca637e85bdf8539018a4a9ac6d4c8f5f58ecae7272fdf26bbdb8c181e4431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 podman[32206]: 2024-07-26 07:36:28.348742902 +0000 UTC m=+0.410031616 container start aa7ca637e85bdf8539018a4a9ac6d4c8f5f58ecae7272fdf26bbdb8c181e4431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-a, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 bash[32206]: aa7ca637e85bdf8539018a4a9ac6d4c8f5f58ecae7272fdf26bbdb8c181e4431 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 systemd[1]: Started Ceph mon.a for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: pidfile_write: ignore empty --pid-file 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: load: jerasure load: lrc 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: RocksDB version: 7.9.2 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Git sha 0 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T07:36:28.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: DB SUMMARY 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: DB Session ID: LQKX3JTRM1JUJOG3CE01 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: CURRENT file: CURRENT 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95371 ; 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.error_if_exists: 0 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.create_if_missing: 0 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.paranoid_checks: 1 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T07:36:28.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T07:36:28.611 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T07:36:28.611 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.env: 0x564068d43280 2024-07-26T07:36:28.611 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T07:36:28.611 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.info_log: 0x56408c71d060 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.statistics: (nil) 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.use_fsync: 0 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_log_file_size: 0 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.allow_fallocate: 1 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.use_direct_reads: 0 2024-07-26T07:36:28.612 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.db_log_dir: 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.wal_dir: 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T07:36:28.613 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.write_buffer_manager: 0x56408c7215e0 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T07:36:28.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.unordered_write: 0 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.row_cache: None 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.wal_filter: None 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.two_write_queues: 0 2024-07-26T07:36:28.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.wal_compression: 0 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.atomic_flush: 0 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.log_readahead_size: 0 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T07:36:28.616 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_background_jobs: 2 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_background_compactions: -1 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_subcompactions: 1 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T07:36:28.617 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_open_files: -1 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_background_flushes: -1 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Compression algorithms supported: 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: kZSTD supported: 0 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: kXpressCompression supported: 0 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: kBZip2Compression supported: 0 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: kLZ4Compression supported: 1 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: kZlibCompression supported: 1 2024-07-26T07:36:28.618 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: kSnappyCompression supported: 1 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.merge_operator: 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_filter: None 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56408c71cc00) 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout: cache_index_and_filter_blocks: 1 2024-07-26T07:36:28.619 INFO:journalctl@ceph.mon.a.smithi097.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: pin_top_level_index_and_filter: 1 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_type: 0 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: data_block_index_type: 0 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_shortening: 1 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: checksum: 4 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: no_block_cache: 0 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache: 0x5640886e1610 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_name: BinnedLRUCache 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_options: 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: capacity : 536870912 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: num_shard_bits : 4 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: strict_capacity_limit : 0 2024-07-26T07:36:28.620 INFO:journalctl@ceph.mon.a.smithi097.stdout: high_pri_pool_ratio: 0.000 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_compressed: (nil) 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: persistent_cache: (nil) 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_size: 4096 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_size_deviation: 10 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_restart_interval: 16 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_block_restart_interval: 1 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: metadata_block_size: 4096 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: partition_filters: 0 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: use_delta_encoding: 1 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: filter_policy: bloomfilter 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: whole_key_filtering: 1 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: verify_compression: 0 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: read_amp_bytes_per_bit: 0 2024-07-26T07:36:28.621 INFO:journalctl@ceph.mon.a.smithi097.stdout: format_version: 5 2024-07-26T07:36:28.622 INFO:journalctl@ceph.mon.a.smithi097.stdout: enable_index_compression: 1 2024-07-26T07:36:28.622 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_align: 0 2024-07-26T07:36:28.622 INFO:journalctl@ceph.mon.a.smithi097.stdout: max_auto_readahead_size: 262144 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout: prepopulate_block_cache: 0 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout: initial_auto_readahead_size: 8192 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression: NoCompression 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T07:36:28.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.num_levels: 7 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T07:36:28.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T07:36:28.625 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T07:36:28.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T07:36:28.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.inplace_update_support: 0 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T07:36:28.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.bloom_locality: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.max_successive_merges: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.ttl: 2592000 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.enable_blob_files: false 2024-07-26T07:36:28.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.min_blob_size: 0 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ec812d5f-7fc5-4b86-9ae1-78831a82a0c1 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979388465718, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-07-26T07:36:28.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979388467169, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91510, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 284, "table_properties": {"data_size": 89551, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11728, "raw_average_key_size": 48, "raw_value_size": 82945, "raw_average_value_size": 341, "num_data_blocks": 12, "num_entries": 243, "num_filter_entries": 243, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721979388, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ec812d5f-7fc5-4b86-9ae1-78831a82a0c1", "db_session_id": "LQKX3JTRM1JUJOG3CE01", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979388467465, "job": 1, "event": "recovery_finished"} 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56408c740a80 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: DB pointer 0x56408d4e0000 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** DB Stats ** 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T07:36:28.631 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: L0 2/0 91.23 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 99.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: Sum 2/0 91.23 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 99.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 99.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T07:36:28.632 INFO:journalctl@ceph.mon.a.smithi097.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 99.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative compaction: 0.00 GB write, 12.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval compaction: 0.00 GB write, 12.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: Block cache BinnedLRUCache@0x5640886e1610#2 capacity: 512.00 MB usage: 1.31 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-07-26T07:36:28.633 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: starting mon.a rank 0 at public addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] at bind addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a@-1(???) e1 preinit fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a@-1(???).mds e1 new map 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a@-1(???).mds e1 print_map 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout: e1 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout: btime 2024-07-26T07:36:12:782518+0000 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout: legacy client fscid: -1 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout: No filesystems configured 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T07:36:28.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: monmap epoch 1 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: last_changed 2024-07-26T07:36:08.466808+0000 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: created 2024-07-26T07:36:08.466808+0000 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: min_mon_release 19 (squid) 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: election_strategy: 1 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: fsmap 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T07:36:28.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:28 smithi097 ceph-mon[32231]: mgrmap e1: no daemons active 2024-07-26T07:36:29.884 INFO:teuthology.orchestra.run.smithi097.stdout:Wrote config to /etc/ceph/ceph.conf 2024-07-26T07:36:29.934 INFO:teuthology.orchestra.run.smithi097.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-07-26T07:36:29.934 INFO:teuthology.orchestra.run.smithi097.stdout:Creating mgr... 2024-07-26T07:36:29.936 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying port 0.0.0.0:9283 ... 2024-07-26T07:36:29.937 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying port 0.0.0.0:8765 ... 2024-07-26T07:36:30.365 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mgr.x 2024-07-26T07:36:30.366 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mgr.x.service: Unit ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mgr.x.service not loaded. 2024-07-26T07:36:30.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:30 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2574082484' entity='client.admin' 2024-07-26T07:36:30.632 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9621a072-4b21-11ef-bcb6-c7b262605968.target.wants/ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-9621a072-4b21-11ef-bcb6-c7b262605968@.service. 2024-07-26T07:36:31.001 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:30 smithi097 systemd[1]: Starting Ceph mgr.x for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:36:31.291 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:31 smithi097 podman[32444]: 2024-07-26 07:36:30.937425348 +0000 UTC m=+0.037755854 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:36:31.292 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:31 smithi097 podman[32444]: 2024-07-26 07:36:31.045093083 +0000 UTC m=+0.145423609 container create bd9819eb29f0e40338761276244e39d416370683d99f17a136189abb21129b44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:36:31.292 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:31 smithi097 podman[32444]: 2024-07-26 07:36:31.29027795 +0000 UTC m=+0.390608479 container init bd9819eb29f0e40338761276244e39d416370683d99f17a136189abb21129b44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=) 2024-07-26T07:36:31.421 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T07:36:31.421 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout disabled 2024-07-26T07:36:31.437 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T07:36:31.437 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-07-26T07:36:31.437 INFO:teuthology.orchestra.run.smithi097.stdout:firewalld.service is not enabled 2024-07-26T07:36:31.437 INFO:teuthology.orchestra.run.smithi097.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T07:36:31.455 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T07:36:31.455 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout disabled 2024-07-26T07:36:31.472 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T07:36:31.472 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-07-26T07:36:31.472 INFO:teuthology.orchestra.run.smithi097.stdout:firewalld.service is not enabled 2024-07-26T07:36:31.472 INFO:teuthology.orchestra.run.smithi097.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-07-26T07:36:31.473 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mgr to start... 2024-07-26T07:36:31.473 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mgr... 2024-07-26T07:36:31.615 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:31 smithi097 podman[32444]: 2024-07-26 07:36:31.300459677 +0000 UTC m=+0.400790206 container start bd9819eb29f0e40338761276244e39d416370683d99f17a136189abb21129b44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:36:31.616 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:31 smithi097 bash[32444]: bd9819eb29f0e40338761276244e39d416370683d99f17a136189abb21129b44 2024-07-26T07:36:31.616 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:31 smithi097 systemd[1]: Started Ceph mgr.x for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:36:33.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:32 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/627025961' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T07:36:33.379 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 0 2024-07-26T07:36:33.382 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T07:36:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T07:36:33.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T07:36:33.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T07:36:33.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T07:36:33.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:33.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:33.388 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T07:36:12:782518+0000", 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T07:36:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:33.390 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T07:36:12.783563+0000", 2024-07-26T07:36:33.390 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T07:36:33.390 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:33.390 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T07:36:33.390 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-07-26T07:36:33.390 INFO:teuthology.orchestra.run.smithi097.stdout:mgr not available, waiting (1/15)... 2024-07-26T07:36:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:36 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2514254851' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T07:36:36.782 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:36.783 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-07-26T07:36:36.783 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:36:36.783 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T07:36:36.783 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T07:36:36.783 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T07:36:36.784 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T07:36:36.784 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:36.784 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T07:36:36.784 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T07:36:36.784 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 0 2024-07-26T07:36:36.784 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:36.784 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T07:36:36.785 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T07:36:36.785 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:36.786 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-07-26T07:36:36.786 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T07:36:36.786 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:36.786 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T07:36:36.786 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T07:36:36.788 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:36.788 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T07:36:36.788 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:36.788 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T07:36:36.788 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T07:36:36.788 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T07:36:36.788 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T07:36:36.789 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T07:36:12:782518+0000", 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T07:36:36.792 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T07:36:12.783563+0000", 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-07-26T07:36:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:mgr not available, waiting (2/15)... 2024-07-26T07:36:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: Activating manager daemon x 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: mgrmap e2: x(active, starting, since 0.0122227s) 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T07:36:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: Manager daemon x is now available 2024-07-26T07:36:38.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T07:36:38.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T07:36:38.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' 2024-07-26T07:36:38.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:37 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' 2024-07-26T07:36:38.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:38 smithi097 ceph-mon[32231]: from='mgr.14100 172.21.15.97:0/2996474022' entity='mgr.x' 2024-07-26T07:36:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:38 smithi097 ceph-mon[32231]: mgrmap e3: x(active, since 1.0235s) 2024-07-26T07:36:40.319 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:40.319 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-07-26T07:36:40.319 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:36:40.319 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T07:36:40.319 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T07:36:40.319 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T07:36:40.319 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T07:36:40.321 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:40.321 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T07:36:40.321 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T07:36:40.321 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 0 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:40.322 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T07:36:40.323 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T07:36:40.323 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T07:36:40.326 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T07:36:40.327 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T07:36:12:782518+0000", 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T07:36:40.328 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T07:36:12.783563+0000", 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-07-26T07:36:40.329 INFO:teuthology.orchestra.run.smithi097.stdout:mgr is available 2024-07-26T07:36:40.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:40 smithi097 ceph-mon[32231]: mgrmap e4: x(active, since 2s) 2024-07-26T07:36:40.635 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:40 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3961109174' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T07:36:41.855 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:41 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/717300737' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T07:36:41.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:41.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T07:36:41.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout fsid = 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:41.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T07:36:41.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2024-07-26T07:36:41.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T07:36:41.870 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T07:36:41.870 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T07:36:41.870 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T07:36:41.870 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:41.870 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T07:36:41.870 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T07:36:41.870 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-07-26T07:36:41.870 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T07:36:41.871 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-07-26T07:36:41.871 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T07:36:41.871 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T07:36:41.871 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T07:36:41.871 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T07:36:41.871 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T07:36:41.871 INFO:teuthology.orchestra.run.smithi097.stdout:Enabling cephadm module... 2024-07-26T07:36:43.113 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:42 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/765038286' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-07-26T07:36:43.113 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:42 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: ignoring --setuser ceph since I am not root 2024-07-26T07:36:43.114 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:42 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: ignoring --setgroup ceph since I am not root 2024-07-26T07:36:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:43 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/765038286' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-07-26T07:36:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:43 smithi097 ceph-mon[32231]: mgrmap e5: x(active, since 5s) 2024-07-26T07:36:44.909 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:44 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/230401160' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T07:36:44.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-07-26T07:36:44.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-07-26T07:36:44.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T07:36:44.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T07:36:44.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T07:36:44.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-07-26T07:36:44.918 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for the mgr to restart... 2024-07-26T07:36:44.918 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mgr epoch 5... 2024-07-26T07:36:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: Active manager daemon x restarted 2024-07-26T07:36:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: Activating manager daemon x 2024-07-26T07:36:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: osdmap e2: 0 total, 0 up, 0 in 2024-07-26T07:36:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: mgrmap e6: x(active, starting, since 0.00851391s) 2024-07-26T07:36:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:36:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T07:36:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T07:36:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T07:36:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T07:36:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: Manager daemon x is now available 2024-07-26T07:36:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:36:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:48 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:36:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:49 smithi097 ceph-mon[32231]: Found migration_current of "None". Setting to last migration. 2024-07-26T07:36:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:49 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:36:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:49 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:36:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:49 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T07:36:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:49 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T07:36:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:49 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:36:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:49 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:36:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:49 smithi097 ceph-mon[32231]: mgrmap e7: x(active, since 1.01622s) 2024-07-26T07:36:50.502 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:36:50.325+0000 7fe57c200640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:07:36:50] ENGINE Error in HTTPServer.serve 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: Traceback (most recent call last): 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: self._connections.run(self.expiration_interval) 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: self._run(expiration_interval) 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T07:36:50.503 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: s = self.context.wrap_socket( 2024-07-26T07:36:50.504 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T07:36:50.504 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: return self.sslsocket_class._create( 2024-07-26T07:36:50.504 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T07:36:50.504 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: self.do_handshake() 2024-07-26T07:36:50.504 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T07:36:50.504 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: self._sslobj.do_handshake() 2024-07-26T07:36:50.504 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T07:36:50.504 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:36:50.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-07-26T07:36:50.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-07-26T07:36:50.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T07:36:50.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-07-26T07:36:50.530 INFO:teuthology.orchestra.run.smithi097.stdout:mgr epoch 5 is available 2024-07-26T07:36:50.530 INFO:teuthology.orchestra.run.smithi097.stdout:Setting orchestrator backend to cephadm... 2024-07-26T07:36:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-mon[32231]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T07:36:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-mon[32231]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T07:36:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:50 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:36:52.114 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:51 smithi097 ceph-mon[32231]: [26/Jul/2024:07:36:50] ENGINE Bus STARTING 2024-07-26T07:36:52.114 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:51 smithi097 ceph-mon[32231]: [26/Jul/2024:07:36:50] ENGINE Serving on http://172.21.15.97:8765 2024-07-26T07:36:52.114 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:51 smithi097 ceph-mon[32231]: [26/Jul/2024:07:36:50] ENGINE Serving on https://172.21.15.97:7150 2024-07-26T07:36:52.114 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:51 smithi097 ceph-mon[32231]: [26/Jul/2024:07:36:50] ENGINE Bus STARTED 2024-07-26T07:36:52.114 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:51 smithi097 ceph-mon[32231]: [26/Jul/2024:07:36:50] ENGINE Error in HTTPServer.serve 2024-07-26T07:36:52.114 INFO:journalctl@ceph.mon.a.smithi097.stdout: Traceback (most recent call last): 2024-07-26T07:36:52.114 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: self._connections.run(self.expiration_interval) 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: self._run(expiration_interval) 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: s = self.context.wrap_socket( 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: return self.sslsocket_class._create( 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T07:36:52.115 INFO:journalctl@ceph.mon.a.smithi097.stdout: self.do_handshake() 2024-07-26T07:36:52.116 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T07:36:52.116 INFO:journalctl@ceph.mon.a.smithi097.stdout: self._sslobj.do_handshake() 2024-07-26T07:36:52.116 INFO:journalctl@ceph.mon.a.smithi097.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T07:36:52.116 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:51 smithi097 ceph-mon[32231]: mgrmap e8: x(active, since 2s) 2024-07-26T07:36:52.116 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:51 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:36:52.116 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:51 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:36:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:52 smithi097 ceph-mon[32231]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:36:53.579 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout value unchanged 2024-07-26T07:36:53.579 INFO:teuthology.orchestra.run.smithi097.stdout:Generating ssh key... 2024-07-26T07:36:54.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:53 smithi097 ceph-mon[32231]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:36:54.904 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: Generating public/private rsa key pair. 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: Your identification has been saved in /tmp/tmpoyuwc7xw/key 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: Your public key has been saved in /tmp/tmpoyuwc7xw/key.pub 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: The key fingerprint is: 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: SHA256:NNq/fvM/5R1++B7hoY+wBuLb/xB4z3ANPjToNAJDmTY ceph-9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: The key's randomart image is: 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: +---[RSA 3072]----+ 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | .+o | 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | Eo . | 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | . +. + + | 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | + .= + + | 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | . S. = + + | 2024-07-26T07:36:54.905 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | . o. * +.+| 2024-07-26T07:36:54.906 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | . . oo +.=+| 2024-07-26T07:36:54.906 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | .. o* +o=| 2024-07-26T07:36:54.906 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: | ..o=+.=.=*| 2024-07-26T07:36:54.906 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:36:54 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: +----[SHA256]-----+ 2024-07-26T07:36:56.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:55 smithi097 ceph-mon[32231]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:36:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:55 smithi097 ceph-mon[32231]: Generating ssh key... 2024-07-26T07:36:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:55 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:36:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:55 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:36:56.754 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQMLvn6RDaAwYjo/+khTFqD0+aB7wd+z6z4GUYPse1u1+3Fu448uRSIBXrDcVW+ZtehsWV0c1xc/89i610+uXKHdQkmh/MBlK+OSWYxnR/pFG+6RT8q3G5J5QkmuX6gOC6Q+QpEg/nMTpW101iBu7MkrXXKPdkzfa+8XY1J/+zIGrXgvwKp7PRGdLbSplHPs6OtUU76JMAEFAs+8rm4liZOEZrmpJSiH0aG6dtt6cd6xxlCs0XiNftENiv0V5nOyvlTFtZ1rxn1OEc6eWLyr8x9MG08vPUzKv7YHzstDedvLFfjiezVtCbkJetZVN2h3+j53tbr+S9/8cQogPgr0c05/922/RfIWfmrZiRctY/ymEK4oq0NqHwVTWSrnns9nP1577sx9XOf6G5N7px9qV9YabE6N9BhKgGGBL/dFG9bOSy7jvF8TTGhHheMKd5o4YeLxRLxMcZQn/fOW/nq+hHLEnVIV6C8N+z4rRPlYjtQ+Sl+mrApBGFcL9fF8Vvtd8= ceph-9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:36:56.754 INFO:teuthology.orchestra.run.smithi097.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-07-26T07:36:56.755 INFO:teuthology.orchestra.run.smithi097.stdout:Adding key to root@localhost authorized_keys... 2024-07-26T07:36:56.756 INFO:teuthology.orchestra.run.smithi097.stdout:Adding host smithi097... 2024-07-26T07:36:57.300 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:57 smithi097 ceph-mon[32231]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:36:58.361 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:36:58 smithi097 ceph-mon[32231]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "addr": "172.21.15.97", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:37:00.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:00 smithi097 ceph-mon[32231]: Deploying cephadm binary to smithi097 2024-07-26T07:37:03.016 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout Added host 'smithi097' with addr '172.21.15.97' 2024-07-26T07:37:03.018 INFO:teuthology.orchestra.run.smithi097.stdout:Deploying unmanaged mon service... 2024-07-26T07:37:03.641 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:03 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:37:03.642 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:03 smithi097 ceph-mon[32231]: Added host smithi097 2024-07-26T07:37:03.642 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:03 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:05.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:05 smithi097 ceph-mon[32231]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:37:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:05 smithi097 ceph-mon[32231]: Saving service mon spec with placement count:5 2024-07-26T07:37:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:05 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:37:05.580 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-07-26T07:37:05.580 INFO:teuthology.orchestra.run.smithi097.stdout:Deploying unmanaged mgr service... 2024-07-26T07:37:06.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:06 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:37:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:06 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:37:07.144 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-07-26T07:37:07.679 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:07 smithi097 ceph-mon[32231]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:37:07.680 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:07 smithi097 ceph-mon[32231]: Saving service mgr spec with placement count:2 2024-07-26T07:37:07.680 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:07 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:37:09.573 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:09 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:37:09.574 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:09 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4237006493' entity='client.admin' 2024-07-26T07:37:09.574 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:09 smithi097 ceph-mon[32231]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:10.538 INFO:teuthology.orchestra.run.smithi097.stdout:Enabling the dashboard module... 2024-07-26T07:37:10.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:10 smithi097 ceph-mon[32231]: from='mgr.14118 172.21.15.97:0/2069190654' entity='mgr.x' 2024-07-26T07:37:10.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:10 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3744568921' entity='client.admin' 2024-07-26T07:37:12.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:11 smithi097 ceph-mon[32231]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:12.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:11 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3695442997' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-07-26T07:37:12.163 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:11 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: ignoring --setuser ceph since I am not root 2024-07-26T07:37:12.163 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:11 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: ignoring --setgroup ceph since I am not root 2024-07-26T07:37:12.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:12 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3695442997' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-07-26T07:37:12.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:12 smithi097 ceph-mon[32231]: mgrmap e9: x(active, since 22s) 2024-07-26T07:37:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:13 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/849483452' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T07:37:14.470 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-07-26T07:37:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-07-26T07:37:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T07:37:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T07:37:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T07:37:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-07-26T07:37:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for the mgr to restart... 2024-07-26T07:37:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mgr epoch 9... 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: Active manager daemon x restarted 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: Activating manager daemon x 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: osdmap e3: 0 total, 0 up, 0 in 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: mgrmap e10: x(active, starting, since 0.00974998s) 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T07:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:17 smithi097 ceph-mon[32231]: Manager daemon x is now available 2024-07-26T07:37:19.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:19.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T07:37:19.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T07:37:19.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:18 smithi097 ceph-mon[32231]: mgrmap e11: x(active, since 1.01076s) 2024-07-26T07:37:19.470 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:37:19.196+0000 7fcfe3800640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:07:37:19] ENGINE Error in HTTPServer.serve 2024-07-26T07:37:19.470 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: Traceback (most recent call last): 2024-07-26T07:37:19.470 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: self._connections.run(self.expiration_interval) 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: self._run(expiration_interval) 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: s = self.context.wrap_socket( 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T07:37:19.471 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: return self.sslsocket_class._create( 2024-07-26T07:37:19.472 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T07:37:19.472 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: self.do_handshake() 2024-07-26T07:37:19.472 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T07:37:19.472 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: self._sslobj.do_handshake() 2024-07-26T07:37:19.472 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T07:37:19.472 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:37:20.149 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-mon[32231]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T07:37:20.149 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-mon[32231]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T07:37:20.149 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:20.158 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-07-26T07:37:20.158 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-07-26T07:37:20.158 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T07:37:20.158 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-07-26T07:37:20.159 INFO:teuthology.orchestra.run.smithi097.stdout:mgr epoch 9 is available 2024-07-26T07:37:20.159 INFO:teuthology.orchestra.run.smithi097.stdout:Generating a dashboard self-signed certificate... 2024-07-26T07:37:21.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:20 smithi097 ceph-mon[32231]: [26/Jul/2024:07:37:19] ENGINE Bus STARTING 2024-07-26T07:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:20 smithi097 ceph-mon[32231]: [26/Jul/2024:07:37:19] ENGINE Serving on https://172.21.15.97:7150 2024-07-26T07:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:20 smithi097 ceph-mon[32231]: [26/Jul/2024:07:37:19] ENGINE Error in HTTPServer.serve 2024-07-26T07:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout: Traceback (most recent call last): 2024-07-26T07:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T07:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout: self._connections.run(self.expiration_interval) 2024-07-26T07:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T07:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout: self._run(expiration_interval) 2024-07-26T07:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: s = self.context.wrap_socket( 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: return self.sslsocket_class._create( 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: self.do_handshake() 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: self._sslobj.do_handshake() 2024-07-26T07:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T07:37:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:20 smithi097 ceph-mon[32231]: [26/Jul/2024:07:37:19] ENGINE Serving on http://172.21.15.97:8765 2024-07-26T07:37:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:20 smithi097 ceph-mon[32231]: [26/Jul/2024:07:37:19] ENGINE Bus STARTED 2024-07-26T07:37:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:20 smithi097 ceph-mon[32231]: mgrmap e12: x(active, since 2s) 2024-07-26T07:37:21.680 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-07-26T07:37:21.680 INFO:teuthology.orchestra.run.smithi097.stdout:Creating initial admin user... 2024-07-26T07:37:22.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:21 smithi097 ceph-mon[32231]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:37:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:23.674 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$n915Qz2.E8HFieHtqmNr2uy.4gwwoEFGq3pNWvGR9U.UytVvKze8u", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1721979442, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-07-26T07:37:23.675 INFO:teuthology.orchestra.run.smithi097.stdout:Fetching dashboard port number... 2024-07-26T07:37:23.968 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:23 smithi097 ceph-mon[32231]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:37:23.969 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:25.240 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:24 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/371653520' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-07-26T07:37:25.256 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 8443 2024-07-26T07:37:25.293 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T07:37:25.293 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout disabled 2024-07-26T07:37:25.318 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T07:37:25.318 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-07-26T07:37:25.319 INFO:teuthology.orchestra.run.smithi097.stdout:firewalld.service is not enabled 2024-07-26T07:37:25.319 INFO:teuthology.orchestra.run.smithi097.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-07-26T07:37:25.326 INFO:teuthology.orchestra.run.smithi097.stdout:Ceph Dashboard is now available at: 2024-07-26T07:37:25.326 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:25.326 INFO:teuthology.orchestra.run.smithi097.stdout: URL: https://smithi097.front.sepia.ceph.com:8443/ 2024-07-26T07:37:25.326 INFO:teuthology.orchestra.run.smithi097.stdout: User: admin 2024-07-26T07:37:25.326 INFO:teuthology.orchestra.run.smithi097.stdout: Password: 4iabzyydim 2024-07-26T07:37:25.326 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:25.328 INFO:teuthology.orchestra.run.smithi097.stdout:Saving cluster configuration to /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config directory 2024-07-26T07:37:26.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:25 smithi097 ceph-mon[32231]: mgrmap e13: x(active, since 6s) 2024-07-26T07:37:26.964 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-07-26T07:37:26.964 INFO:teuthology.orchestra.run.smithi097.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout:Or, if you are only running a single cluster on this host: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout:Please consider enabling telemetry to help improve Ceph: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: ceph telemetry on 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout:For more information see: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-07-26T07:37:26.966 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:26.966 INFO:teuthology.orchestra.run.smithi097.stdout:Bootstrap complete. 2024-07-26T07:37:27.054 INFO:tasks.cephadm:Fetching config... 2024-07-26T07:37:27.055 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:37:27.055 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-07-26T07:37:27.101 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-07-26T07:37:27.101 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:37:27.102 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-07-26T07:37:27.184 INFO:tasks.cephadm:Fetching mon keyring... 2024-07-26T07:37:27.184 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:37:27.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/keyring of=/dev/stdout 2024-07-26T07:37:27.315 INFO:tasks.cephadm:Fetching pub ssh key... 2024-07-26T07:37:27.315 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:37:27.315 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-07-26T07:37:27.363 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-07-26T07:37:27.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQMLvn6RDaAwYjo/+khTFqD0+aB7wd+z6z4GUYPse1u1+3Fu448uRSIBXrDcVW+ZtehsWV0c1xc/89i610+uXKHdQkmh/MBlK+OSWYxnR/pFG+6RT8q3G5J5QkmuX6gOC6Q+QpEg/nMTpW101iBu7MkrXXKPdkzfa+8XY1J/+zIGrXgvwKp7PRGdLbSplHPs6OtUU76JMAEFAs+8rm4liZOEZrmpJSiH0aG6dtt6cd6xxlCs0XiNftENiv0V5nOyvlTFtZ1rxn1OEc6eWLyr8x9MG08vPUzKv7YHzstDedvLFfjiezVtCbkJetZVN2h3+j53tbr+S9/8cQogPgr0c05/922/RfIWfmrZiRctY/ymEK4oq0NqHwVTWSrnns9nP1577sx9XOf6G5N7px9qV9YabE6N9BhKgGGBL/dFG9bOSy7jvF8TTGhHheMKd5o4YeLxRLxMcZQn/fOW/nq+hHLEnVIV6C8N+z4rRPlYjtQ+Sl+mrApBGFcL9fF8Vvtd8= ceph-9621a072-4b21-11ef-bcb6-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T07:37:27.508 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:27 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/632391136' entity='client.admin' 2024-07-26T07:37:27.518 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQMLvn6RDaAwYjo/+khTFqD0+aB7wd+z6z4GUYPse1u1+3Fu448uRSIBXrDcVW+ZtehsWV0c1xc/89i610+uXKHdQkmh/MBlK+OSWYxnR/pFG+6RT8q3G5J5QkmuX6gOC6Q+QpEg/nMTpW101iBu7MkrXXKPdkzfa+8XY1J/+zIGrXgvwKp7PRGdLbSplHPs6OtUU76JMAEFAs+8rm4liZOEZrmpJSiH0aG6dtt6cd6xxlCs0XiNftENiv0V5nOyvlTFtZ1rxn1OEc6eWLyr8x9MG08vPUzKv7YHzstDedvLFfjiezVtCbkJetZVN2h3+j53tbr+S9/8cQogPgr0c05/922/RfIWfmrZiRctY/ymEK4oq0NqHwVTWSrnns9nP1577sx9XOf6G5N7px9qV9YabE6N9BhKgGGBL/dFG9bOSy7jvF8TTGhHheMKd5o4YeLxRLxMcZQn/fOW/nq+hHLEnVIV6C8N+z4rRPlYjtQ+Sl+mrApBGFcL9fF8Vvtd8= ceph-9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:37:27.550 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQMLvn6RDaAwYjo/+khTFqD0+aB7wd+z6z4GUYPse1u1+3Fu448uRSIBXrDcVW+ZtehsWV0c1xc/89i610+uXKHdQkmh/MBlK+OSWYxnR/pFG+6RT8q3G5J5QkmuX6gOC6Q+QpEg/nMTpW101iBu7MkrXXKPdkzfa+8XY1J/+zIGrXgvwKp7PRGdLbSplHPs6OtUU76JMAEFAs+8rm4liZOEZrmpJSiH0aG6dtt6cd6xxlCs0XiNftENiv0V5nOyvlTFtZ1rxn1OEc6eWLyr8x9MG08vPUzKv7YHzstDedvLFfjiezVtCbkJetZVN2h3+j53tbr+S9/8cQogPgr0c05/922/RfIWfmrZiRctY/ymEK4oq0NqHwVTWSrnns9nP1577sx9XOf6G5N7px9qV9YabE6N9BhKgGGBL/dFG9bOSy7jvF8TTGhHheMKd5o4YeLxRLxMcZQn/fOW/nq+hHLEnVIV6C8N+z4rRPlYjtQ+Sl+mrApBGFcL9fF8Vvtd8= ceph-9621a072-4b21-11ef-bcb6-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T07:37:27.668 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQMLvn6RDaAwYjo/+khTFqD0+aB7wd+z6z4GUYPse1u1+3Fu448uRSIBXrDcVW+ZtehsWV0c1xc/89i610+uXKHdQkmh/MBlK+OSWYxnR/pFG+6RT8q3G5J5QkmuX6gOC6Q+QpEg/nMTpW101iBu7MkrXXKPdkzfa+8XY1J/+zIGrXgvwKp7PRGdLbSplHPs6OtUU76JMAEFAs+8rm4liZOEZrmpJSiH0aG6dtt6cd6xxlCs0XiNftENiv0V5nOyvlTFtZ1rxn1OEc6eWLyr8x9MG08vPUzKv7YHzstDedvLFfjiezVtCbkJetZVN2h3+j53tbr+S9/8cQogPgr0c05/922/RfIWfmrZiRctY/ymEK4oq0NqHwVTWSrnns9nP1577sx9XOf6G5N7px9qV9YabE6N9BhKgGGBL/dFG9bOSy7jvF8TTGhHheMKd5o4YeLxRLxMcZQn/fOW/nq+hHLEnVIV6C8N+z4rRPlYjtQ+Sl+mrApBGFcL9fF8Vvtd8= ceph-9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:37:27.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQMLvn6RDaAwYjo/+khTFqD0+aB7wd+z6z4GUYPse1u1+3Fu448uRSIBXrDcVW+ZtehsWV0c1xc/89i610+uXKHdQkmh/MBlK+OSWYxnR/pFG+6RT8q3G5J5QkmuX6gOC6Q+QpEg/nMTpW101iBu7MkrXXKPdkzfa+8XY1J/+zIGrXgvwKp7PRGdLbSplHPs6OtUU76JMAEFAs+8rm4liZOEZrmpJSiH0aG6dtt6cd6xxlCs0XiNftENiv0V5nOyvlTFtZ1rxn1OEc6eWLyr8x9MG08vPUzKv7YHzstDedvLFfjiezVtCbkJetZVN2h3+j53tbr+S9/8cQogPgr0c05/922/RfIWfmrZiRctY/ymEK4oq0NqHwVTWSrnns9nP1577sx9XOf6G5N7px9qV9YabE6N9BhKgGGBL/dFG9bOSy7jvF8TTGhHheMKd5o4YeLxRLxMcZQn/fOW/nq+hHLEnVIV6C8N+z4rRPlYjtQ+Sl+mrApBGFcL9fF8Vvtd8= ceph-9621a072-4b21-11ef-bcb6-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T07:37:27.836 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQMLvn6RDaAwYjo/+khTFqD0+aB7wd+z6z4GUYPse1u1+3Fu448uRSIBXrDcVW+ZtehsWV0c1xc/89i610+uXKHdQkmh/MBlK+OSWYxnR/pFG+6RT8q3G5J5QkmuX6gOC6Q+QpEg/nMTpW101iBu7MkrXXKPdkzfa+8XY1J/+zIGrXgvwKp7PRGdLbSplHPs6OtUU76JMAEFAs+8rm4liZOEZrmpJSiH0aG6dtt6cd6xxlCs0XiNftENiv0V5nOyvlTFtZ1rxn1OEc6eWLyr8x9MG08vPUzKv7YHzstDedvLFfjiezVtCbkJetZVN2h3+j53tbr+S9/8cQogPgr0c05/922/RfIWfmrZiRctY/ymEK4oq0NqHwVTWSrnns9nP1577sx9XOf6G5N7px9qV9YabE6N9BhKgGGBL/dFG9bOSy7jvF8TTGhHheMKd5o4YeLxRLxMcZQn/fOW/nq+hHLEnVIV6C8N+z4rRPlYjtQ+Sl+mrApBGFcL9fF8Vvtd8= ceph-9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:37:27.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-07-26T07:37:28.307 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:37:31.733 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-07-26T07:37:31.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-07-26T07:37:32.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:32 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/727807683' entity='client.admin' 2024-07-26T07:37:32.707 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:37:33.975 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:33.975 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:33.975 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:37:33.975 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:33.975 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:33.976 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:35.784 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi115 2024-07-26T07:37:35.784 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:37:35.784 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.conf 2024-07-26T07:37:35.841 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:37:35.841 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:37:35.936 INFO:tasks.cephadm:Adding host smithi115 to orchestrator... 2024-07-26T07:37:35.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch host add smithi115 2024-07-26T07:37:36.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:35 smithi097 ceph-mon[32231]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:37:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:37:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:37:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:35 smithi097 ceph-mon[32231]: Updating smithi097:/etc/ceph/ceph.conf 2024-07-26T07:37:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:35 smithi097 ceph-mon[32231]: Updating smithi097:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:37:36.358 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:37:37.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:36 smithi097 ceph-mon[32231]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:37:37.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:36 smithi097 ceph-mon[32231]: Updating smithi097:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T07:37:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:38 smithi097 ceph-mon[32231]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:39 smithi097 ceph-mon[32231]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:37:40.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:40 smithi097 ceph-mon[32231]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:41 smithi097 ceph-mon[32231]: Deploying cephadm binary to smithi115 2024-07-26T07:37:42.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:42 smithi097 ceph-mon[32231]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:43.621 INFO:teuthology.orchestra.run.smithi097.stdout:Added host 'smithi115' with addr '172.21.15.115' 2024-07-26T07:37:44.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T07:37:44.746 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:37:44.895 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:44.895 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:44 smithi097 ceph-mon[32231]: Added host smithi115 2024-07-26T07:37:44.895 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:44.895 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:44 smithi097 ceph-mon[32231]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:44.895 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:45.961 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:46 smithi097 ceph-mon[32231]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:47.013 INFO:teuthology.orchestra.run.smithi097.stdout:[{"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}, {"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}] 2024-07-26T07:37:47.758 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-07-26T07:37:47.758 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:37:47.758 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-07-26T07:37:47.815 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:37:47.815 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:37:47.913 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-07-26T07:37:47.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch host add smithi181 2024-07-26T07:37:48.368 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:37:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:48 smithi097 ceph-mon[32231]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:37:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:48 smithi097 ceph-mon[32231]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:50.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:49 smithi097 ceph-mon[32231]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:50 smithi097 ceph-mon[32231]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:37:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:37:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:37:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:37:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: Updating smithi115:/etc/ceph/ceph.conf 2024-07-26T07:37:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:52 smithi097 ceph-mon[32231]: Deploying cephadm binary to smithi181 2024-07-26T07:37:53.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:53 smithi097 ceph-mon[32231]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:37:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:53 smithi097 ceph-mon[32231]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:37:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:54 smithi097 ceph-mon[32231]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:55.384 INFO:teuthology.orchestra.run.smithi097.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-07-26T07:37:55.729 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:55 smithi097 ceph-mon[32231]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T07:37:55.730 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:55.730 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:55.730 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:55.730 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:56.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T07:37:56.599 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:37:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:56 smithi097 ceph-mon[32231]: Added host smithi181 2024-07-26T07:37:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:56 smithi097 ceph-mon[32231]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:58.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:58.748 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:37:58.748 INFO:teuthology.orchestra.run.smithi097.stdout:[{"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}, {"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-07-26T07:37:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:58 smithi097 ceph-mon[32231]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:37:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:37:59.460 INFO:tasks.cephadm:Setting crush tunables to default 2024-07-26T07:37:59.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd crush tunables default 2024-07-26T07:37:59.888 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:00.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:37:59 smithi097 ceph-mon[32231]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:38:00.960 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:00 smithi097 ceph-mon[32231]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:02.024 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2024-07-26T07:38:02.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:02 smithi097 ceph-mon[32231]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:02.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:02 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2523733456' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T07:38:02.701 INFO:tasks.cephadm:Adding mon.a on smithi097 2024-07-26T07:38:02.702 INFO:tasks.cephadm:Adding mon.b on smithi115 2024-07-26T07:38:02.702 INFO:tasks.cephadm:Adding mon.c on smithi181 2024-07-26T07:38:02.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch apply mon '3;smithi097:172.21.15.97=a;smithi115:172.21.15.115=b;smithi181:172.21.15.181=c' 2024-07-26T07:38:03.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:03 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2523733456' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T07:38:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:03 smithi097 ceph-mon[32231]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:03.520 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-07-26T07:38:03.624 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-07-26T07:38:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:38:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: Updating smithi181:/etc/ceph/ceph.conf 2024-07-26T07:38:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:04 smithi097 ceph-mon[32231]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:05.996 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-07-26T07:38:06.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:05 smithi097 ceph-mon[32231]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:06.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:05 smithi097 ceph-mon[32231]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:38:06.965 DEBUG:teuthology.orchestra.run.smithi115:mon.b> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.b.service 2024-07-26T07:38:06.973 DEBUG:teuthology.orchestra.run.smithi181:mon.c> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mon.c.service 2024-07-26T07:38:06.980 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T07:38:06.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph mon dump -f json 2024-07-26T07:38:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T07:38:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi115:172.21.15.115=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: Saving service mon spec with placement smithi097:172.21.15.97=a;smithi115:172.21.15.115=b;smithi181:172.21.15.181=c;count:3 2024-07-26T07:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:07 smithi097 ceph-mon[32231]: Deploying daemon mon.c on smithi181 2024-07-26T07:38:08.551 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:09 smithi097 ceph-mon[32231]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:10.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:10 smithi097 ceph-mon[32231]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:12.185 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:38:12.185 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","modified":"2024-07-26T07:36:08.466808Z","created":"2024-07-26T07:36:08.466808Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-07-26T07:38:12.185 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-07-26T07:38:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:12 smithi181 systemd[1]: Starting Ceph mon.c for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:38:12.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:12 smithi097 ceph-mon[32231]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 podman[35850]: 2024-07-26 07:38:12.989944016 +0000 UTC m=+0.040423677 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:38:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 podman[35850]: 2024-07-26 07:38:13.090914164 +0000 UTC m=+0.141393853 container create 2c20e9d8ef7f6465bf41aec87071cc1ab2ba5c426835351451707f2094709515 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-c, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:38:13.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:13 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/557426293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 podman[35850]: 2024-07-26 07:38:13.358755484 +0000 UTC m=+0.409235180 container init 2c20e9d8ef7f6465bf41aec87071cc1ab2ba5c426835351451707f2094709515 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-c, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T07:38:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 podman[35850]: 2024-07-26 07:38:13.3729864 +0000 UTC m=+0.423466079 container start 2c20e9d8ef7f6465bf41aec87071cc1ab2ba5c426835351451707f2094709515 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.build-date=20240724) 2024-07-26T07:38:13.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 bash[35850]: 2c20e9d8ef7f6465bf41aec87071cc1ab2ba5c426835351451707f2094709515 2024-07-26T07:38:13.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 systemd[1]: Started Ceph mon.c for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:38:13.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T07:38:13.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T07:38:13.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pidfile_write: ignore empty --pid-file 2024-07-26T07:38:13.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: load: jerasure load: lrc 2024-07-26T07:38:13.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: RocksDB version: 7.9.2 2024-07-26T07:38:13.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Git sha 0 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: DB SUMMARY 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: DB Session ID: 8A1JCTXCMRI3JW18E91W 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: CURRENT file: CURRENT 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.error_if_exists: 0 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.create_if_missing: 0 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.paranoid_checks: 1 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T07:38:13.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T07:38:13.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.env: 0x5557f58fd280 2024-07-26T07:38:13.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T07:38:13.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.info_log: 0x55580ffb5060 2024-07-26T07:38:13.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T07:38:13.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.statistics: (nil) 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.use_fsync: 0 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_log_file_size: 0 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.allow_fallocate: 1 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.use_direct_reads: 0 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T07:38:13.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.db_log_dir: 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.wal_dir: 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T07:38:13.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.write_buffer_manager: 0x55580ffb95e0 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.unordered_write: 0 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T07:38:13.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.row_cache: None 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.wal_filter: None 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.two_write_queues: 0 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.wal_compression: 0 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.atomic_flush: 0 2024-07-26T07:38:13.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.log_readahead_size: 0 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T07:38:13.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_background_jobs: 2 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_background_compactions: -1 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_subcompactions: 1 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T07:38:13.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T07:38:13.743 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T07:38:13.743 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_open_files: -1 2024-07-26T07:38:13.743 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T07:38:13.743 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T07:38:13.743 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T07:38:13.743 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_background_flushes: -1 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Compression algorithms supported: 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: kZSTD supported: 0 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: kXpressCompression supported: 0 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: kBZip2Compression supported: 0 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: kLZ4Compression supported: 1 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: kZlibCompression supported: 1 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: kSnappyCompression supported: 1 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T07:38:13.744 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.merge_operator: 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_filter: None 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55580ffb4c00) 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout: cache_index_and_filter_blocks: 1 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T07:38:13.745 INFO:journalctl@ceph.mon.c.smithi181.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: pin_top_level_index_and_filter: 1 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_type: 0 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: data_block_index_type: 0 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_shortening: 1 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: checksum: 4 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: no_block_cache: 0 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache: 0x55580bf79610 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_name: BinnedLRUCache 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_options: 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: capacity : 536870912 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: num_shard_bits : 4 2024-07-26T07:38:13.746 INFO:journalctl@ceph.mon.c.smithi181.stdout: strict_capacity_limit : 0 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: high_pri_pool_ratio: 0.000 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_compressed: (nil) 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: persistent_cache: (nil) 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_size: 4096 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_size_deviation: 10 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_restart_interval: 16 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_block_restart_interval: 1 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: metadata_block_size: 4096 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: partition_filters: 0 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: use_delta_encoding: 1 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: filter_policy: bloomfilter 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: whole_key_filtering: 1 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: verify_compression: 0 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: read_amp_bytes_per_bit: 0 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: format_version: 5 2024-07-26T07:38:13.747 INFO:journalctl@ceph.mon.c.smithi181.stdout: enable_index_compression: 1 2024-07-26T07:38:13.748 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_align: 0 2024-07-26T07:38:13.749 INFO:journalctl@ceph.mon.c.smithi181.stdout: max_auto_readahead_size: 262144 2024-07-26T07:38:13.749 INFO:journalctl@ceph.mon.c.smithi181.stdout: prepopulate_block_cache: 0 2024-07-26T07:38:13.749 INFO:journalctl@ceph.mon.c.smithi181.stdout: initial_auto_readahead_size: 8192 2024-07-26T07:38:13.749 INFO:journalctl@ceph.mon.c.smithi181.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T07:38:13.749 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T07:38:13.749 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T07:38:13.749 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression: NoCompression 2024-07-26T07:38:13.749 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.num_levels: 7 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T07:38:13.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T07:38:13.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T07:38:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T07:38:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.inplace_update_support: 0 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T07:38:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.bloom_locality: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.max_successive_merges: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.ttl: 2592000 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T07:38:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T07:38:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.enable_blob_files: false 2024-07-26T07:38:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.min_blob_size: 0 2024-07-26T07:38:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T07:38:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T07:38:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T07:38:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0a780246-b17a-47f8-9b6c-ac96c2aa6a6d 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979493496399, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979493497586, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721979493, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0a780246-b17a-47f8-9b6c-ac96c2aa6a6d", "db_session_id": "8A1JCTXCMRI3JW18E91W", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979493497897, "job": 1, "event": "recovery_finished"} 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-07-26T07:38:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T07:38:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55580ffd8a80 2024-07-26T07:38:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: DB pointer 0x555810d78000 2024-07-26T07:38:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T07:38:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T07:38:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** DB Stats ** 2024-07-26T07:38:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T07:38:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T07:38:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T07:38:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T07:38:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T07:38:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T07:38:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T07:38:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-07-26T07:38:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** Compaction Stats [default] ** 2024-07-26T07:38:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T07:38:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T07:38:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** Compaction Stats [default] ** 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T07:38:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: Block cache BinnedLRUCache@0x55580bf79610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: using public_addr v2:172.21.15.181:0/0 -> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-07-26T07:38:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: starting mon.c rank -1 at public addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] at bind addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(???) e0 preinit fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).mds e1 new map 2024-07-26T07:38:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).mds e1 print_map 2024-07-26T07:38:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout: e1 2024-07-26T07:38:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout: btime 2024-07-26T07:36:12:782518+0000 2024-07-26T07:38:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T07:38:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T07:38:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout: legacy client fscid: -1 2024-07-26T07:38:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-07-26T07:38:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout: No filesystems configured 2024-07-26T07:38:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T07:38:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T07:38:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T07:38:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T07:38:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T07:38:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T07:38:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T07:38:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:38:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:38:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:38:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Deploying cephadm binary to smithi115 2024-07-26T07:38:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Added host smithi115 2024-07-26T07:38:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:38:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Updating smithi115:/etc/ceph/ceph.conf 2024-07-26T07:38:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Deploying cephadm binary to smithi181 2024-07-26T07:38:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:38:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T07:38:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Added host smithi181 2024-07-26T07:38:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:38:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2523733456' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T07:38:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2523733456' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T07:38:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:38:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Updating smithi181:/etc/ceph/ceph.conf 2024-07-26T07:38:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:38:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T07:38:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi115:172.21.15.115=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:38:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Saving service mon spec with placement smithi097:172.21.15.97=a;smithi115:172.21.15.115=b;smithi181:172.21.15.181=c;count:3 2024-07-26T07:38:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: Deploying daemon mon.c on smithi181 2024-07-26T07:38:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/557426293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:13 smithi181 ceph-mon[35869]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-26T07:38:13.968 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T07:38:13.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph mon dump -f json 2024-07-26T07:38:14.423 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:38:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: mon.a calling monitor election 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: mon.c calling monitor election 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: monmap epoch 2 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: last_changed 2024-07-26T07:38:13.551390+0000 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: created 2024-07-26T07:36:08.466808+0000 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: min_mon_release 19 (squid) 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: election_strategy: 1 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-07-26T07:38:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: fsmap 2024-07-26T07:38:18.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:18.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: mgrmap e13: x(active, since 60s) 2024-07-26T07:38:18.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: overall HEALTH_OK 2024-07-26T07:38:18.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:18.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:18.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:18.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:18.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: mon.a calling monitor election 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: mon.c calling monitor election 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T07:38:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: monmap epoch 2 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: last_changed 2024-07-26T07:38:13.551390+0000 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: created 2024-07-26T07:36:08.466808+0000 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: min_mon_release 19 (squid) 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: election_strategy: 1 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: fsmap 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: mgrmap e13: x(active, since 60s) 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: overall HEALTH_OK 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:19.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:19.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:19.939 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:38:19.939 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","modified":"2024-07-26T07:38:13.551390Z","created":"2024-07-26T07:36:08.466808Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T07:38:19.940 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-07-26T07:38:19.940 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:19 smithi181 ceph-mon[35869]: Deploying daemon mon.b on smithi115 2024-07-26T07:38:19.940 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:19 smithi097 ceph-mon[32231]: Deploying daemon mon.b on smithi115 2024-07-26T07:38:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:20 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/3401396403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:20 smithi181 ceph-mon[35869]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:20 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/3401396403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:20 smithi097 ceph-mon[32231]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:21.001 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:38:20 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:38:20.553+0000 7fd023e00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-07-26T07:38:21.714 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T07:38:21.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph mon dump -f json 2024-07-26T07:38:22.160 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:38:22.306 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:22 smithi181 ceph-mon[35869]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:22.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:22 smithi097 ceph-mon[32231]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:24.478 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:38:24.478 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","modified":"2024-07-26T07:38:13.551390Z","created":"2024-07-26T07:36:08.466808Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T07:38:24.479 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-07-26T07:38:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 podman[35346]: 2024-07-26 07:38:24.479938162 +0000 UTC m=+0.191307254 container create 15eec0fbd5e25b740136bf842c8544861d425043ab560efb577151980e7786a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-26T07:38:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:24 smithi181 ceph-mon[35869]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:24 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/3305241520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:24 smithi097 ceph-mon[32231]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:24 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/3305241520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 podman[35346]: 2024-07-26 07:38:24.756512428 +0000 UTC m=+0.467881537 container init 15eec0fbd5e25b740136bf842c8544861d425043ab560efb577151980e7786a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:38:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 podman[35346]: 2024-07-26 07:38:24.766970508 +0000 UTC m=+0.478339606 container start 15eec0fbd5e25b740136bf842c8544861d425043ab560efb577151980e7786a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-07-26T07:38:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 bash[35346]: 15eec0fbd5e25b740136bf842c8544861d425043ab560efb577151980e7786a9 2024-07-26T07:38:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 systemd[1]: Started Ceph mon.b for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:38:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T07:38:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T07:38:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pidfile_write: ignore empty --pid-file 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: load: jerasure load: lrc 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: RocksDB version: 7.9.2 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Git sha 0 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: DB SUMMARY 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: DB Session ID: 0BZU17IMSR3E22VN7DYQ 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: CURRENT file: CURRENT 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-07-26T07:38:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.error_if_exists: 0 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.create_if_missing: 0 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.paranoid_checks: 1 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.env: 0x55a3b6f31280 2024-07-26T07:38:25.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T07:38:25.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.info_log: 0x55a3f119d060 2024-07-26T07:38:25.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T07:38:25.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.statistics: (nil) 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.use_fsync: 0 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_log_file_size: 0 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.allow_fallocate: 1 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.use_direct_reads: 0 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T07:38:25.229 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.db_log_dir: 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.wal_dir: 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.write_buffer_manager: 0x55a3f11a15e0 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.unordered_write: 0 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T07:38:25.231 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.row_cache: None 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.wal_filter: None 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.two_write_queues: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.wal_compression: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.atomic_flush: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.log_readahead_size: 0 2024-07-26T07:38:25.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_background_jobs: 2 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_background_compactions: -1 2024-07-26T07:38:25.233 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_subcompactions: 1 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_open_files: -1 2024-07-26T07:38:25.234 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T07:38:25.235 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T07:38:25.235 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T07:38:25.235 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T07:38:25.235 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_background_flushes: -1 2024-07-26T07:38:25.235 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Compression algorithms supported: 2024-07-26T07:38:25.235 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: kZSTD supported: 0 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: kXpressCompression supported: 0 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: kBZip2Compression supported: 0 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: kLZ4Compression supported: 1 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: kZlibCompression supported: 1 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: kSnappyCompression supported: 1 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T07:38:25.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.merge_operator: 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_filter: None 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a3f119cc00) 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout: cache_index_and_filter_blocks: 1 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout: pin_top_level_index_and_filter: 1 2024-07-26T07:38:25.237 INFO:journalctl@ceph.mon.b.smithi115.stdout: index_type: 0 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: data_block_index_type: 0 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: index_shortening: 1 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: checksum: 4 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: no_block_cache: 0 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_cache: 0x55a3ed161610 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_cache_name: BinnedLRUCache 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_cache_options: 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: capacity : 536870912 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: num_shard_bits : 4 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: strict_capacity_limit : 0 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: high_pri_pool_ratio: 0.000 2024-07-26T07:38:25.238 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_cache_compressed: (nil) 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: persistent_cache: (nil) 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_size: 4096 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_size_deviation: 10 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_restart_interval: 16 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: index_block_restart_interval: 1 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: metadata_block_size: 4096 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: partition_filters: 0 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: use_delta_encoding: 1 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: filter_policy: bloomfilter 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: whole_key_filtering: 1 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: verify_compression: 0 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: read_amp_bytes_per_bit: 0 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: format_version: 5 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: enable_index_compression: 1 2024-07-26T07:38:25.239 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_align: 0 2024-07-26T07:38:25.241 INFO:journalctl@ceph.mon.b.smithi115.stdout: max_auto_readahead_size: 262144 2024-07-26T07:38:25.241 INFO:journalctl@ceph.mon.b.smithi115.stdout: prepopulate_block_cache: 0 2024-07-26T07:38:25.241 INFO:journalctl@ceph.mon.b.smithi115.stdout: initial_auto_readahead_size: 8192 2024-07-26T07:38:25.241 INFO:journalctl@ceph.mon.b.smithi115.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T07:38:25.241 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T07:38:25.241 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T07:38:25.241 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression: NoCompression 2024-07-26T07:38:25.241 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.num_levels: 7 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T07:38:25.242 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T07:38:25.243 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T07:38:25.244 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T07:38:25.245 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.inplace_update_support: 0 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.bloom_locality: 0 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.max_successive_merges: 0 2024-07-26T07:38:25.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.ttl: 2592000 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.enable_blob_files: false 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.min_blob_size: 0 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T07:38:25.247 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T07:38:25.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T07:38:25.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T07:38:25.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T07:38:25.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-07-26T07:38:25.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-07-26T07:38:25.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fdb01a9a-0be6-4d96-94fd-6acf98d3af00 2024-07-26T07:38:25.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979504891615, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-07-26T07:38:25.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-07-26T07:38:25.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979504892931, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721979504, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fdb01a9a-0be6-4d96-94fd-6acf98d3af00", "db_session_id": "0BZU17IMSR3E22VN7DYQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-07-26T07:38:25.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721979504893222, "job": 1, "event": "recovery_finished"} 2024-07-26T07:38:25.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-07-26T07:38:25.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T07:38:25.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a3f11c0a80 2024-07-26T07:38:25.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: DB pointer 0x55a3f1f60000 2024-07-26T07:38:25.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: ** DB Stats ** 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T07:38:25.250 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: ** Compaction Stats [default] ** 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: ** Compaction Stats [default] ** 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T07:38:25.251 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: Cumulative compaction: 0.00 GB write, 0.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: Interval compaction: 0.00 GB write, 0.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T07:38:25.252 INFO:journalctl@ceph.mon.b.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout: Block cache BinnedLRUCache@0x55a3ed161610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: using public_addr v2:172.21.15.115:0/0 -> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: starting mon.b rank -1 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(???) e0 preinit fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).mds e1 new map 2024-07-26T07:38:25.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).mds e1 print_map 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout: e1 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout: btime 2024-07-26T07:36:12:782518+0000 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout: legacy client fscid: -1 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout: No filesystems configured 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T07:38:25.254 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Deploying cephadm binary to smithi115 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Added host smithi115 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:25.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Updating smithi115:/etc/ceph/ceph.conf 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Deploying cephadm binary to smithi181 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Added host smithi181 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.257 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2523733456' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2523733456' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:38:25.258 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Updating smithi181:/etc/ceph/ceph.conf 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi115:172.21.15.115=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Saving service mon spec with placement smithi097:172.21.15.97=a;smithi115:172.21.15.115=b;smithi181:172.21.15.181=c;count:3 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.259 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Deploying daemon mon.c on smithi181 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/557426293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.a calling monitor election 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:25.260 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.c calling monitor election 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: monmap epoch 2 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: last_changed 2024-07-26T07:38:13.551390+0000 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: created 2024-07-26T07:36:08.466808+0000 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: min_mon_release 19 (squid) 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: election_strategy: 1 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-07-26T07:38:25.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: fsmap 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mgrmap e13: x(active, since 60s) 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: overall HEALTH_OK 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:25.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: Deploying daemon mon.b on smithi115 2024-07-26T07:38:25.263 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:25.263 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/3401396403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:25.263 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.263 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.263 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:25.263 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/3305241520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:25.264 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:24 smithi115 ceph-mon[35364]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-07-26T07:38:26.241 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T07:38:26.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph mon dump -f json 2024-07-26T07:38:26.688 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:38:30.319 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-07-26T07:38:30.319 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","modified":"2024-07-26T07:38:24.945556Z","created":"2024-07-26T07:36:08.466808Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-07-26T07:38:30.320 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2024-07-26T07:38:30.320 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:38:30.320 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: mon.a calling monitor election 2024-07-26T07:38:30.320 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.320 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:30.320 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: mon.c calling monitor election 2024-07-26T07:38:30.320 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: mon.b calling monitor election 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: monmap epoch 3 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: last_changed 2024-07-26T07:38:24.945556+0000 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: created 2024-07-26T07:36:08.466808+0000 2024-07-26T07:38:30.321 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: min_mon_release 19 (squid) 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: election_strategy: 1 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: 2: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.b 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: fsmap 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: mgrmap e13: x(active, since 72s) 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: overall HEALTH_OK 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:38:30.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: mon.a calling monitor election 2024-07-26T07:38:30.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:30.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: mon.c calling monitor election 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: mon.b calling monitor election 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: monmap epoch 3 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: last_changed 2024-07-26T07:38:24.945556+0000 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: created 2024-07-26T07:36:08.466808+0000 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: min_mon_release 19 (squid) 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: election_strategy: 1 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: 2: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.b 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: fsmap 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: mgrmap e13: x(active, since 72s) 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: overall HEALTH_OK 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:38:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: mon.a calling monitor election 2024-07-26T07:38:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: mon.c calling monitor election 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: mon.b calling monitor election 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: monmap epoch 3 2024-07-26T07:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: last_changed 2024-07-26T07:38:24.945556+0000 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: created 2024-07-26T07:36:08.466808+0000 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: min_mon_release 19 (squid) 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: election_strategy: 1 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: 2: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.b 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: fsmap 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: mgrmap e13: x(active, since 72s) 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: overall HEALTH_OK 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:30.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:31.093 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-07-26T07:38:31.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config generate-minimal-conf 2024-07-26T07:38:31.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:31 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/2219125509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:31.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:31.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:31 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/2219125509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:31.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:31 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/2219125509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T07:38:31.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:38:31.509 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:32.250 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:38:31 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:38:31.945+0000 7fd023e00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-07-26T07:38:33.015 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:32 smithi097 ceph-mon[32231]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:33.015 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.015 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.015 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.015 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.015 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:33.015 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:33.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:32 smithi115 ceph-mon[35364]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:32 smithi181 ceph-mon[35869]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:33.796 INFO:teuthology.orchestra.run.smithi097.stdout:# minimal ceph.conf for 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:33.796 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-07-26T07:38:33.796 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:38:33.796 INFO:teuthology.orchestra.run.smithi097.stdout: mon_host = [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-07-26T07:38:34.191 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: Updating smithi097:/etc/ceph/ceph.conf 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: Updating smithi115:/etc/ceph/ceph.conf 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: Updating smithi181:/etc/ceph/ceph.conf 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: Updating smithi097:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.192 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4043561444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:34.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:34.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:34.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: Updating smithi097:/etc/ceph/ceph.conf 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: Updating smithi115:/etc/ceph/ceph.conf 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: Updating smithi181:/etc/ceph/ceph.conf 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: Updating smithi097:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4043561444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: Updating smithi097:/etc/ceph/ceph.conf 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: Updating smithi115:/etc/ceph/ceph.conf 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: Updating smithi181:/etc/ceph/ceph.conf 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: Updating smithi097:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: Updating smithi115:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: Updating smithi181:/var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4043561444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:34.577 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-07-26T07:38:34.577 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:38:34.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T07:38:34.660 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:38:34.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:38:34.791 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:38:34.791 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T07:38:34.884 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:38:34.884 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:38:35.013 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:38:35.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T07:38:35.109 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:38:35.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T07:38:35.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:34 smithi115 ceph-mon[35364]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T07:38:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:34 smithi115 ceph-mon[35364]: Reconfiguring daemon mon.a on smithi097 2024-07-26T07:38:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:34 smithi115 ceph-mon[35364]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:35.234 INFO:tasks.cephadm:Adding mgr.x on smithi097 2024-07-26T07:38:35.234 INFO:tasks.cephadm:Adding mgr.y on smithi115 2024-07-26T07:38:35.234 INFO:tasks.cephadm:Adding mgr.z on smithi181 2024-07-26T07:38:35.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch apply mgr '3;smithi097=x;smithi115=y;smithi181=z' 2024-07-26T07:38:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:34 smithi097 ceph-mon[32231]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T07:38:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:34 smithi097 ceph-mon[32231]: Reconfiguring daemon mon.a on smithi097 2024-07-26T07:38:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:34 smithi097 ceph-mon[32231]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:35.348 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:34 smithi181 ceph-mon[35869]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T07:38:35.348 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:34 smithi181 ceph-mon[35869]: Reconfiguring daemon mon.a on smithi097 2024-07-26T07:38:35.348 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:34 smithi181 ceph-mon[35869]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:35.710 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:38:36.965 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:36.965 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:36.965 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:36 smithi181 ceph-mon[35869]: Reconfiguring mon.b (monmap changed)... 2024-07-26T07:38:36.965 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:36.965 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:36.965 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:36.965 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:36 smithi181 ceph-mon[35869]: Reconfiguring daemon mon.b on smithi115 2024-07-26T07:38:36.965 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:36 smithi181 ceph-mon[35869]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:36 smithi115 ceph-mon[35364]: Reconfiguring mon.b (monmap changed)... 2024-07-26T07:38:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:36 smithi115 ceph-mon[35364]: Reconfiguring daemon mon.b on smithi115 2024-07-26T07:38:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:36 smithi115 ceph-mon[35364]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:36 smithi097 ceph-mon[32231]: Reconfiguring mon.b (monmap changed)... 2024-07-26T07:38:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:36 smithi097 ceph-mon[32231]: Reconfiguring daemon mon.b on smithi115 2024-07-26T07:38:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:36 smithi097 ceph-mon[32231]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:38.080 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2024-07-26T07:38:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: Reconfiguring mon.c (monmap changed)... 2024-07-26T07:38:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: Reconfiguring daemon mon.c on smithi181 2024-07-26T07:38:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.858 DEBUG:teuthology.orchestra.run.smithi115:mgr.y> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mgr.y.service 2024-07-26T07:38:38.865 DEBUG:teuthology.orchestra.run.smithi181:mgr.z> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mgr.z.service 2024-07-26T07:38:38.871 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-07-26T07:38:38.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-07-26T07:38:38.879 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-07-26T07:38:38.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set client debug_client 20 2024-07-26T07:38:38.926 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-07-26T07:38:38.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set client debug_ms 1 2024-07-26T07:38:38.950 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.950 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.950 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: Reconfiguring mon.c (monmap changed)... 2024-07-26T07:38:38.950 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:38.950 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:38.950 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:38.950 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: Reconfiguring daemon mon.c on smithi181 2024-07-26T07:38:38.951 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:38.951 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.954 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-07-26T07:38:38.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: Reconfiguring mon.c (monmap changed)... 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: Reconfiguring daemon mon.c on smithi181 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:38.999 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-07-26T07:38:39.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-07-26T07:38:39.046 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-07-26T07:38:39.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds debug_mds 20 2024-07-26T07:38:39.093 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-07-26T07:38:39.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-07-26T07:38:39.141 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-07-26T07:38:39.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds debug_ms 2 2024-07-26T07:38:39.191 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-07-26T07:38:39.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-07-26T07:38:39.248 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-07-26T07:38:39.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds mds_bal_merge_size 5 2024-07-26T07:38:39.300 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-07-26T07:38:39.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds mds_bal_split_bits 3 2024-07-26T07:38:39.354 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-07-26T07:38:39.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds mds_bal_split_size 100 2024-07-26T07:38:39.407 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-07-26T07:38:39.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-07-26T07:38:39.470 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-07-26T07:38:39.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-07-26T07:38:39.530 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-07-26T07:38:39.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-07-26T07:38:39.598 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-07-26T07:38:39.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-07-26T07:38:39.665 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-07-26T07:38:39.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-07-26T07:38:39.678 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:39.697 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-07-26T07:38:39.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-07-26T07:38:39.729 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:39.739 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-07-26T07:38:39.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-07-26T07:38:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:39 smithi097 ceph-mon[32231]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi097=x;smithi115=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:38:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:39 smithi097 ceph-mon[32231]: Saving service mgr spec with placement smithi097=x;smithi115=y;smithi181=z;count:3 2024-07-26T07:38:39.772 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-07-26T07:38:39.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-07-26T07:38:39.773 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:39.824 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-07-26T07:38:39.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-07-26T07:38:39.880 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-07-26T07:38:39.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-07-26T07:38:39.943 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:39 smithi181 ceph-mon[35869]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi097=x;smithi115=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:38:39.944 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:39 smithi181 ceph-mon[35869]: Saving service mgr spec with placement smithi097=x;smithi115=y;smithi181=z;count:3 2024-07-26T07:38:39.944 DEBUG:tasks.cephadm:set 22 configs 2024-07-26T07:38:39.944 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi097') 2024-07-26T07:38:39.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:39 smithi115 ceph-mon[35364]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi097=x;smithi115=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:38:39.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:39 smithi115 ceph-mon[35364]: Saving service mgr spec with placement smithi097=x;smithi115=y;smithi181=z;count:3 2024-07-26T07:38:39.977 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:40.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T07:38:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:40.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T07:38:40.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:40.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:41.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:41 smithi181 ceph-mon[35869]: Deploying daemon mgr.z on smithi181 2024-07-26T07:38:41.947 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:41 smithi097 ceph-mon[32231]: Deploying daemon mgr.z on smithi181 2024-07-26T07:38:41.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:41 smithi115 ceph-mon[35364]: Deploying daemon mgr.z on smithi181 2024-07-26T07:38:42.371 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:42.390 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:42.859 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:42 smithi181 ceph-mon[35869]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:42.867 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:42 smithi097 ceph-mon[32231]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:43.223 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Jul 26 07:38:42 smithi181 systemd[1]: Starting Ceph mgr.z for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:38:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:42 smithi115 ceph-mon[35364]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:43.483 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Jul 26 07:38:43 smithi181 podman[37617]: 2024-07-26 07:38:43.12627203 +0000 UTC m=+0.031710693 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:38:43.483 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Jul 26 07:38:43 smithi181 podman[37617]: 2024-07-26 07:38:43.225779777 +0000 UTC m=+0.131218453 container create 5b65aa30ade1eab2754323a83c5c8554aff0b1b4aa29e127df5b9dc3d501f85c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-07-26T07:38:43.484 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Jul 26 07:38:43 smithi181 podman[37617]: 2024-07-26 07:38:43.429750063 +0000 UTC m=+0.335188761 container init 5b65aa30ade1eab2754323a83c5c8554aff0b1b4aa29e127df5b9dc3d501f85c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-z, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-07-26T07:38:43.484 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Jul 26 07:38:43 smithi181 podman[37617]: 2024-07-26 07:38:43.440676337 +0000 UTC m=+0.346114990 container start 5b65aa30ade1eab2754323a83c5c8554aff0b1b4aa29e127df5b9dc3d501f85c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-26T07:38:43.982 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Jul 26 07:38:43 smithi181 bash[37617]: 5b65aa30ade1eab2754323a83c5c8554aff0b1b4aa29e127df5b9dc3d501f85c 2024-07-26T07:38:43.983 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Jul 26 07:38:43 smithi181 systemd[1]: Started Ceph mgr.z for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:38:43.983 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Jul 26 07:38:43 smithi181 ceph-mgr[37635]: -- 172.21.15.181:0/910778790 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56434facd0e0 con 0x56434faeba80 2024-07-26T07:38:44.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:44.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T07:38:44.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:44.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:44.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: Deploying daemon mgr.y on smithi115 2024-07-26T07:38:44.745 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:44 smithi115 ceph-mon[35364]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:44.925 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.925 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.925 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.926 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:44.926 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T07:38:44.926 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:44.926 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:44.926 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: Deploying daemon mgr.y on smithi115 2024-07-26T07:38:44.926 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:44 smithi097 ceph-mon[32231]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T07:38:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: Deploying daemon mgr.y on smithi115 2024-07-26T07:38:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:44 smithi181 ceph-mon[35869]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:46.185 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:46 smithi097 ceph-mon[32231]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:46.451 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:46 smithi115 ceph-mon[35364]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:46 smithi181 ceph-mon[35869]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:46.723 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Jul 26 07:38:46 smithi115 systemd[1]: Starting Ceph mgr.y for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:38:47.135 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Jul 26 07:38:46 smithi115 podman[36498]: 2024-07-26 07:38:46.710160986 +0000 UTC m=+0.030877725 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:38:47.135 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Jul 26 07:38:46 smithi115 podman[36498]: 2024-07-26 07:38:46.821431455 +0000 UTC m=+0.142148244 container create d2f28bd7fa4f443e53c8fc8595bb42287df0421fc0f6dd039946f7982aa35fab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-y, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:38:47.135 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Jul 26 07:38:47 smithi115 podman[36498]: 2024-07-26 07:38:47.133320315 +0000 UTC m=+0.454037094 container init d2f28bd7fa4f443e53c8fc8595bb42287df0421fc0f6dd039946f7982aa35fab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T07:38:47.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:47 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1616985524' entity='client.admin' 2024-07-26T07:38:47.474 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Jul 26 07:38:47 smithi115 podman[36498]: 2024-07-26 07:38:47.145294078 +0000 UTC m=+0.466010860 container start d2f28bd7fa4f443e53c8fc8595bb42287df0421fc0f6dd039946f7982aa35fab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-y, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-07-26T07:38:47.474 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Jul 26 07:38:47 smithi115 bash[36498]: d2f28bd7fa4f443e53c8fc8595bb42287df0421fc0f6dd039946f7982aa35fab 2024-07-26T07:38:47.474 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Jul 26 07:38:47 smithi115 systemd[1]: Started Ceph mgr.y for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:38:47.474 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Jul 26 07:38:47 smithi115 ceph-mgr[36517]: -- 172.21.15.115:0/3074829296 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560df1f110e0 con 0x560df1f2fa80 2024-07-26T07:38:47.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:47 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1616985524' entity='client.admin' 2024-07-26T07:38:47.640 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:47.656 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:47 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1616985524' entity='client.admin' 2024-07-26T07:38:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:48 smithi097 ceph-mon[32231]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:48.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.672 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.672 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.672 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:48.672 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:48 smithi181 ceph-mon[35869]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:48.672 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:48 smithi115 ceph-mon[35364]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3911695003' entity='client.admin' 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: Standby manager daemon z started 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T07:38:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T07:38:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3698318665' entity='client.admin' 2024-07-26T07:38:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:50 smithi115 ceph-mon[35364]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3911695003' entity='client.admin' 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: Standby manager daemon z started 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3698318665' entity='client.admin' 2024-07-26T07:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:50 smithi181 ceph-mon[35869]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3911695003' entity='client.admin' 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: Standby manager daemon z started 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='mgr.? 172.21.15.181:0/382385294' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3698318665' entity='client.admin' 2024-07-26T07:38:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:50 smithi097 ceph-mon[32231]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:51.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 -- 172.21.15.97:0/2328897962 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f258403ed60 msgr2=0x7f2584041270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 --2- 172.21.15.97:0/2328897962 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f258403ed60 0x7f2584041270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f258c005f70 tx=0x7f258c005fd0 comp rx=0 tx=0).stop 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 -- 172.21.15.97:0/2328897962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a0102fa0 msgr2=0x7f25a019b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 --2- 172.21.15.97:0/2328897962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a0102fa0 0x7f25a019b730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f259000e9f0 tx=0x7f259000eeb0 comp rx=0 tx=0).stop 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 -- 172.21.15.97:0/2328897962 shutdown_connections 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 --2- 172.21.15.97:0/2328897962 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f258403ed60 0x7f2584041270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 --2- 172.21.15.97:0/2328897962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25a01049d0 0x7f25a019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 --2- 172.21.15.97:0/2328897962 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f25a0104030 0x7f25a019bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 --2- 172.21.15.97:0/2328897962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a0102fa0 0x7f25a019b730 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 -- 172.21.15.97:0/2328897962 >> 172.21.15.97:0/2328897962 conn(0x7f25a00fe8f0 msgr2=0x7f25a0075020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.244+0000 7f259f200640 1 -- 172.21.15.97:0/2328897962 shutdown_connections 2024-07-26T07:38:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:50.245+0000 7f259f200640 1 -- 172.21.15.97:0/2328897962 wait complete. 2024-07-26T07:38:51.426 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.427 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.428 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2328897962' entity='client.admin' 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: mgrmap e14: x(active, since 92s), standbys: z 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:51 smithi097 ceph-mon[32231]: Reconfiguring daemon mgr.x on smithi097 2024-07-26T07:38:51.594 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.595 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.596 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.609 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.625 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.626 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2328897962' entity='client.admin' 2024-07-26T07:38:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T07:38:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: mgrmap e14: x(active, since 92s), standbys: z 2024-07-26T07:38:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T07:38:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:51 smithi115 ceph-mon[35364]: Reconfiguring daemon mgr.x on smithi097 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2328897962' entity='client.admin' 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: mgrmap e14: x(active, since 92s), standbys: z 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:51 smithi181 ceph-mon[35869]: Reconfiguring daemon mgr.x on smithi097 2024-07-26T07:38:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.374+0000 7f27de200640 1 -- 172.21.15.97:0/3993487168 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8102fe0 msgr2=0x7f27d8103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.374+0000 7f27de200640 1 --2- 172.21.15.97:0/3993487168 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8102fe0 0x7f27d8103450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009900 tx=0x7f27cc02a480 comp rx=0 tx=0).stop 2024-07-26T07:38:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.374+0000 7f27de200640 1 -- 172.21.15.97:0/3993487168 shutdown_connections 2024-07-26T07:38:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.374+0000 7f27de200640 1 --2- 172.21.15.97:0/3993487168 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104a10 0x7f27d810b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.374+0000 7f27de200640 1 --2- 172.21.15.97:0/3993487168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8104070 0x7f27d8104480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.374+0000 7f27de200640 1 --2- 172.21.15.97:0/3993487168 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8102fe0 0x7f27d8103450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.374+0000 7f27de200640 1 -- 172.21.15.97:0/3993487168 >> 172.21.15.97:0/3993487168 conn(0x7f27d80fe950 msgr2=0x7f27d8100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.374+0000 7f27de200640 1 -- 172.21.15.97:0/3993487168 shutdown_connections 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.375+0000 7f27de200640 1 -- 172.21.15.97:0/3993487168 wait complete. 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.376+0000 7f27de200640 1 Processor -- start 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.376+0000 7f27de200640 1 -- start start 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.376+0000 7f27de200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8102fe0 0x7f27d81995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.376+0000 7f27de200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104070 0x7f27d8199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.376+0000 7f27de200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8104a10 0x7f27d819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.376+0000 7f27de200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f27d810e8b0 con 0x7f27d8104a10 2024-07-26T07:38:52.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.376+0000 7f27de200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27d810e730 con 0x7f27d8102fe0 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.376+0000 7f27de200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f27d810ea30 con 0x7f27d8104070 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8104a10 0x7f27d819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8104a10 0x7f27d819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51152/0 (socket says 172.21.15.97:51152) 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 -- 172.21.15.97:0/2129347766 learned_addr learned my addr 172.21.15.97:0/2129347766 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dce00640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8102fe0 0x7f27d81995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27d7e00640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104070 0x7f27d8199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 -- 172.21.15.97:0/2129347766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104070 msgr2=0x7f27d8199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104070 0x7f27d8199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 -- 172.21.15.97:0/2129347766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8102fe0 msgr2=0x7f27d81995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:52.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8102fe0 0x7f27d81995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 -- 172.21.15.97:0/2129347766 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d819aff0 con 0x7f27d8104a10 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dce00640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8102fe0 0x7f27d81995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.377+0000 7f27dd800640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8104a10 0x7f27d819a7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f27d000eb10 tx=0x7f27d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.378+0000 7f27d5600640 1 -- 172.21.15.97:0/2129347766 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d0004b20 con 0x7f27d8104a10 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.378+0000 7f27d5600640 1 -- 172.21.15.97:0/2129347766 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f27d0004cc0 con 0x7f27d8104a10 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.378+0000 7f27d5600640 1 -- 172.21.15.97:0/2129347766 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d0010430 con 0x7f27d8104a10 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.378+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d819b2e0 con 0x7f27d8104a10 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.378+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d81a19e0 con 0x7f27d8104a10 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.379+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27d810a1c0 con 0x7f27d8104a10 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.380+0000 7f27d5600640 1 -- 172.21.15.97:0/2129347766 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 14) ==== 100875+0+0 (secure 0 0 0) 0x7f27d0005020 con 0x7f27d8104a10 2024-07-26T07:38:52.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.381+0000 7f27d5600640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27c0079450 0x7f27c007b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.381+0000 7f27d5600640 1 -- 172.21.15.97:0/2129347766 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f27d009af30 con 0x7f27d8104a10 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.382+0000 7f27dce00640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27c0079450 0x7f27c007b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.382+0000 7f27dce00640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27c0079450 0x7f27c007b960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005d90 tx=0x7f27cc0315c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.386+0000 7f27d5600640 1 -- 172.21.15.97:0/2129347766 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f27d0064350 con 0x7f27d8104a10 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.617+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f27d802e0c0 con 0x7f27d8104a10 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.625+0000 7f27d5600640 1 -- 172.21.15.97:0/2129347766 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f27d0063c50 con 0x7f27d8104a10 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.627+0000 7f27d5600640 1 -- 172.21.15.97:0/2129347766 <== mon.0 v2:172.21.15.97:3300/0 8 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f27d005f3c0 con 0x7f27d8104a10 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27c0079450 msgr2=0x7f27c007b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27c0079450 0x7f27c007b960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005d90 tx=0x7f27cc0315c0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8104a10 msgr2=0x7f27d819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8104a10 0x7f27d819a7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f27d000eb10 tx=0x7f27d000c2d0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 shutdown_connections 2024-07-26T07:38:52.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27c0079450 0x7f27c007b960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d8104a10 0x7f27d819a7d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104070 0x7f27d8199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 --2- 172.21.15.97:0/2129347766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d8102fe0 0x7f27d81995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:52.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.630+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 >> 172.21.15.97:0/2129347766 conn(0x7f27d80fe950 msgr2=0x7f27d8107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:38:52.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.631+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 shutdown_connections 2024-07-26T07:38:52.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:51.631+0000 7f27de200640 1 -- 172.21.15.97:0/2129347766 wait complete. 2024-07-26T07:38:52.845 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:52 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2129347766' entity='client.admin' 2024-07-26T07:38:52.845 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:52 smithi097 ceph-mon[32231]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:52.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:52 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2129347766' entity='client.admin' 2024-07-26T07:38:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:52 smithi115 ceph-mon[35364]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:52 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2129347766' entity='client.admin' 2024-07-26T07:38:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:52 smithi181 ceph-mon[35869]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:53.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.156+0000 7f54ce600640 1 -- 172.21.15.97:0/1061856164 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 msgr2=0x7f54c810b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:53.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.156+0000 7f54ce600640 1 --2- 172.21.15.97:0/1061856164 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 0x7f54c810b3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f54c0009ca0 tx=0x7f54c002a8e0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.157+0000 7f54ce600640 1 -- 172.21.15.97:0/1061856164 shutdown_connections 2024-07-26T07:38:53.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.157+0000 7f54ce600640 1 --2- 172.21.15.97:0/1061856164 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 0x7f54c810b3c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.157+0000 7f54ce600640 1 --2- 172.21.15.97:0/1061856164 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54c81063d0 0x7f54c81087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.157+0000 7f54ce600640 1 --2- 172.21.15.97:0/1061856164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54c806bf70 0x7f54c8105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.157+0000 7f54ce600640 1 -- 172.21.15.97:0/1061856164 >> 172.21.15.97:0/1061856164 conn(0x7f54c80fd9d0 msgr2=0x7f54c80ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.157+0000 7f54ce600640 1 -- 172.21.15.97:0/1061856164 shutdown_connections 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.157+0000 7f54ce600640 1 -- 172.21.15.97:0/1061856164 wait complete. 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.158+0000 7f54ce600640 1 Processor -- start 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.158+0000 7f54ce600640 1 -- start start 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.158+0000 7f54ce600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54c806bf70 0x7f54c81034c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.158+0000 7f54ce600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54c81063d0 0x7f54c8101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.158+0000 7f54ce600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 0x7f54c81022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.158+0000 7f54ce600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f54c810ebf0 con 0x7f54c8108eb0 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.158+0000 7f54ce600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f54c810ea70 con 0x7f54c806bf70 2024-07-26T07:38:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.158+0000 7f54ce600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f54c810ed70 con 0x7f54c81063d0 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 0x7f54c81022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 0x7f54c81022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51176/0 (socket says 172.21.15.97:51176) 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c7e00640 1 -- 172.21.15.97:0/2985797088 learned_addr learned my addr 172.21.15.97:0/2985797088 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c6a00640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54c81063d0 0x7f54c8101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c7e00640 1 -- 172.21.15.97:0/2985797088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54c81063d0 msgr2=0x7f54c8101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c7e00640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54c81063d0 0x7f54c8101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c7e00640 1 -- 172.21.15.97:0/2985797088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54c806bf70 msgr2=0x7f54c81034c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c7e00640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54c806bf70 0x7f54c81034c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.159+0000 7f54c7e00640 1 -- 172.21.15.97:0/2985797088 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c81029d0 con 0x7f54c8108eb0 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.160+0000 7f54c7e00640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 0x7f54c81022d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f54c0031040 tx=0x7f54c0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:38:53.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.160+0000 7f54bfe00640 1 -- 172.21.15.97:0/2985797088 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0004040 con 0x7f54c8108eb0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.160+0000 7f54bfe00640 1 -- 172.21.15.97:0/2985797088 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f54c0004650 con 0x7f54c8108eb0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.160+0000 7f54bfe00640 1 -- 172.21.15.97:0/2985797088 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0030d20 con 0x7f54c8108eb0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.160+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c81a38d0 con 0x7f54c8108eb0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.160+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c81a3da0 con 0x7f54c8108eb0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.161+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54c80ff4f0 con 0x7f54c8108eb0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.163+0000 7f54bfe00640 1 -- 172.21.15.97:0/2985797088 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 14) ==== 100875+0+0 (secure 0 0 0) 0x7f54c00041e0 con 0x7f54c8108eb0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.164+0000 7f54bfe00640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54ac079450 0x7f54ac07b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.164+0000 7f54bfe00640 1 -- 172.21.15.97:0/2985797088 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f54c00bbd30 con 0x7f54c8108eb0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.164+0000 7f54c7400640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54ac079450 0x7f54ac07b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.165+0000 7f54c7400640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54ac079450 0x7f54ac07b960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009670 tx=0x7f54b8006bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:38:53.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.168+0000 7f54bfe00640 1 -- 172.21.15.97:0/2985797088 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f54c0085150 con 0x7f54c8108eb0 2024-07-26T07:38:53.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.370+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f54c80639f0 con 0x7f54c8108eb0 2024-07-26T07:38:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.379+0000 7f54bfe00640 1 -- 172.21.15.97:0/2985797088 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v15)=0 v15) ==== 123+0+0 (secure 0 0 0) 0x7f54c0084a50 con 0x7f54c8108eb0 2024-07-26T07:38:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.382+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54ac079450 msgr2=0x7f54ac07b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.382+0000 7f54ce600640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54ac079450 0x7f54ac07b960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009670 tx=0x7f54b8006bd0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.382+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 msgr2=0x7f54c81022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:38:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.382+0000 7f54ce600640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 0x7f54c81022d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f54c0031040 tx=0x7f54c0003ba0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.383+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 shutdown_connections 2024-07-26T07:38:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.383+0000 7f54ce600640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54ac079450 0x7f54ac07b960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.383+0000 7f54ce600640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54c8108eb0 0x7f54c81022d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.383+0000 7f54ce600640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54c81063d0 0x7f54c8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.383+0000 7f54ce600640 1 --2- 172.21.15.97:0/2985797088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54c806bf70 0x7f54c81034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:38:53.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.383+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 >> 172.21.15.97:0/2985797088 conn(0x7f54c80fd9d0 msgr2=0x7f54c8104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:38:53.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.383+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 shutdown_connections 2024-07-26T07:38:53.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:38:53.383+0000 7f54ce600640 1 -- 172.21.15.97:0/2985797088 wait complete. 2024-07-26T07:38:54.680 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:54 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2985797088' entity='client.admin' 2024-07-26T07:38:54.680 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:54 smithi097 ceph-mon[32231]: Standby manager daemon y started 2024-07-26T07:38:54.680 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:54 smithi097 ceph-mon[32231]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T07:38:54.681 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:54 smithi097 ceph-mon[32231]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T07:38:54.681 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:54 smithi097 ceph-mon[32231]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T07:38:54.681 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:54 smithi097 ceph-mon[32231]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T07:38:54.681 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:54 smithi097 ceph-mon[32231]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:54 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2985797088' entity='client.admin' 2024-07-26T07:38:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:54 smithi115 ceph-mon[35364]: Standby manager daemon y started 2024-07-26T07:38:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:54 smithi115 ceph-mon[35364]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T07:38:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:54 smithi115 ceph-mon[35364]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T07:38:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:54 smithi115 ceph-mon[35364]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T07:38:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:54 smithi115 ceph-mon[35364]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T07:38:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:54 smithi115 ceph-mon[35364]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:54 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2985797088' entity='client.admin' 2024-07-26T07:38:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:54 smithi181 ceph-mon[35869]: Standby manager daemon y started 2024-07-26T07:38:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:54 smithi181 ceph-mon[35869]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T07:38:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:54 smithi181 ceph-mon[35869]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T07:38:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:54 smithi181 ceph-mon[35869]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T07:38:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:54 smithi181 ceph-mon[35869]: from='mgr.? 172.21.15.115:0/3456525123' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T07:38:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:54 smithi181 ceph-mon[35869]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:54.747 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi097') 2024-07-26T07:38:54.747 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi097') 2024-07-26T07:38:54.747 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi097') 2024-07-26T07:38:54.747 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi097') 2024-07-26T07:38:55.037 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:38:55.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:55 smithi097 ceph-mon[32231]: mgrmap e15: x(active, since 96s), standbys: z, y 2024-07-26T07:38:55.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T07:38:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:55 smithi115 ceph-mon[35364]: mgrmap e15: x(active, since 96s), standbys: z, y 2024-07-26T07:38:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T07:38:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:55 smithi181 ceph-mon[35869]: mgrmap e15: x(active, since 96s), standbys: z, y 2024-07-26T07:38:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T07:38:56.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:56 smithi097 ceph-mon[32231]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:56.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:56 smithi115 ceph-mon[35364]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:56 smithi181 ceph-mon[35869]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:58.242 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:38:58 smithi097 ceph-mon[32231]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:58.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:38:58 smithi115 ceph-mon[35364]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:38:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:38:58 smithi181 ceph-mon[35869]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:00.178 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:00 smithi181 ceph-mon[35869]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:00.246 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:00 smithi097 ceph-mon[32231]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:00.310 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:00 smithi115 ceph-mon[35364]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 -- 172.21.15.97:0/3056253124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5230106c90 msgr2=0x7f5230107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 --2- 172.21.15.97:0/3056253124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5230106c90 0x7f5230107190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5228009ca0 tx=0x7f522802a8d0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 -- 172.21.15.97:0/3056253124 shutdown_connections 2024-07-26T07:39:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 --2- 172.21.15.97:0/3056253124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5230106c90 0x7f5230107190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 --2- 172.21.15.97:0/3056253124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52301040f0 0x7f5230106550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 --2- 172.21.15.97:0/3056253124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f523006b3b0 0x7f5230103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 -- 172.21.15.97:0/3056253124 >> 172.21.15.97:0/3056253124 conn(0x7f52300fd9e0 msgr2=0x7f52300ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 -- 172.21.15.97:0/3056253124 shutdown_connections 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.673+0000 7f5237800640 1 -- 172.21.15.97:0/3056253124 wait complete. 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.679+0000 7f5237800640 1 Processor -- start 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.679+0000 7f5237800640 1 -- start start 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.679+0000 7f5237800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523006b3b0 0x7f523019fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5237800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52301040f0 0x7f52301a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5237800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5230106c90 0x7f52301a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5237800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5230115200 con 0x7f523006b3b0 2024-07-26T07:39:01.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5237800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5230115080 con 0x7f5230106c90 2024-07-26T07:39:01.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5237800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5230115380 con 0x7f52301040f0 2024-07-26T07:39:01.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523006b3b0 0x7f523019fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:01.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523006b3b0 0x7f523019fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50768/0 (socket says 172.21.15.97:50768) 2024-07-26T07:39:01.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 -- 172.21.15.97:0/339169916 learned_addr learned my addr 172.21.15.97:0/339169916 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:01.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 -- 172.21.15.97:0/339169916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52301040f0 msgr2=0x7f52301a0510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T07:39:01.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52301040f0 0x7f52301a0510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 -- 172.21.15.97:0/339169916 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5230106c90 msgr2=0x7f52301a10b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5230106c90 0x7f52301a10b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 -- 172.21.15.97:0/339169916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52301a18d0 con 0x7f523006b3b0 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5234e00640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523006b3b0 0x7f523019fea0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f522400ea60 tx=0x7f522400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f522d600640 1 -- 172.21.15.97:0/339169916 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522400cde0 con 0x7f523006b3b0 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f522d600640 1 -- 172.21.15.97:0/339169916 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52240035f0 con 0x7f523006b3b0 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f522d600640 1 -- 172.21.15.97:0/339169916 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5224010690 con 0x7f523006b3b0 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52301a1bc0 con 0x7f523006b3b0 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52301a82c0 con 0x7f523006b3b0 2024-07-26T07:39:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.680+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52300ff500 con 0x7f523006b3b0 2024-07-26T07:39:01.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.687+0000 7f522d600640 1 -- 172.21.15.97:0/339169916 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f5224004020 con 0x7f523006b3b0 2024-07-26T07:39:01.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.687+0000 7f522d600640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f52180b3bc0 0x7f52180b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:01.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.687+0000 7f522d600640 1 -- 172.21.15.97:0/339169916 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f52240e2340 con 0x7f523006b3b0 2024-07-26T07:39:01.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.687+0000 7f522fe00640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f52180b3bc0 0x7f52180b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:01.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.687+0000 7f522fe00640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f52180b3bc0 0x7f52180b60d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f521c009a60 tx=0x7f521c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:01.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.694+0000 7f522d600640 1 -- 172.21.15.97:0/339169916 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f52240ab5c0 con 0x7f523006b3b0 2024-07-26T07:39:01.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.898+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f52300639f0 con 0x7f523006b3b0 2024-07-26T07:39:01.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.905+0000 7f522d600640 1 -- 172.21.15.97:0/339169916 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v17)=0 v17) ==== 116+0+0 (secure 0 0 0) 0x7f52240aaec0 con 0x7f523006b3b0 2024-07-26T07:39:01.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f52180b3bc0 msgr2=0x7f52180b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f52180b3bc0 0x7f52180b60d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f521c009a60 tx=0x7f521c005f90 comp rx=0 tx=0).stop 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523006b3b0 msgr2=0x7f523019fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523006b3b0 0x7f523019fea0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f522400ea60 tx=0x7f522400ef20 comp rx=0 tx=0).stop 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 shutdown_connections 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f52180b3bc0 0x7f52180b60d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5230106c90 0x7f52301a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52301040f0 0x7f52301a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 --2- 172.21.15.97:0/339169916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f523006b3b0 0x7f523019fea0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 >> 172.21.15.97:0/339169916 conn(0x7f52300fd9e0 msgr2=0x7f5230105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 shutdown_connections 2024-07-26T07:39:01.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.908+0000 7f5237800640 1 -- 172.21.15.97:0/339169916 wait complete. 2024-07-26T07:39:02.741 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi097') 2024-07-26T07:39:03.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:02 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1754964936' entity='client.admin' 2024-07-26T07:39:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:02 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/339169916' entity='client.admin' 2024-07-26T07:39:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:02 smithi097 ceph-mon[32231]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:02 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1754964936' entity='client.admin' 2024-07-26T07:39:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:02 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/339169916' entity='client.admin' 2024-07-26T07:39:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:02 smithi115 ceph-mon[35364]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:02 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1754964936' entity='client.admin' 2024-07-26T07:39:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:02 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/339169916' entity='client.admin' 2024-07-26T07:39:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:02 smithi181 ceph-mon[35869]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.548+0000 7f818be00640 1 -- 172.21.15.97:0/687940005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c10af40 msgr2=0x7f818c10d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.548+0000 7f818be00640 1 --2- 172.21.15.97:0/687940005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c10af40 0x7f818c10d3e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8184009ca0 tx=0x7f818402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.548+0000 7f818be00640 1 -- 172.21.15.97:0/687940005 shutdown_connections 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.548+0000 7f818be00640 1 --2- 172.21.15.97:0/687940005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c10af40 0x7f818c10d3e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.548+0000 7f818be00640 1 --2- 172.21.15.97:0/687940005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f818c075720 0x7f818c075b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.548+0000 7f818be00640 1 --2- 172.21.15.97:0/687940005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f818c076b10 0x7f818c0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.548+0000 7f818be00640 1 -- 172.21.15.97:0/687940005 >> 172.21.15.97:0/687940005 conn(0x7f818c0fe8f0 msgr2=0x7f818c100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.549+0000 7f818be00640 1 -- 172.21.15.97:0/687940005 shutdown_connections 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.549+0000 7f818be00640 1 -- 172.21.15.97:0/687940005 wait complete. 2024-07-26T07:39:03.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818be00640 1 Processor -- start 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818be00640 1 -- start start 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c075720 0x7f818c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818be00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f818c076b10 0x7f818c19bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f818c10af40 0x7f818c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f818c110cf0 con 0x7f818c075720 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818be00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f818c110b70 con 0x7f818c076b10 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f818c110e70 con 0x7f818c10af40 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.550+0000 7f818aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c075720 0x7f818c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c075720 0x7f818c19b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50740/0 (socket says 172.21.15.97:50740) 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818aa00640 1 -- 172.21.15.97:0/1754964936 learned_addr learned my addr 172.21.15.97:0/1754964936 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:03.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818a000640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f818c076b10 0x7f818c19bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818b400640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f818c10af40 0x7f818c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818aa00640 1 -- 172.21.15.97:0/1754964936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f818c10af40 msgr2=0x7f818c19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818aa00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f818c10af40 0x7f818c19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818aa00640 1 -- 172.21.15.97:0/1754964936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f818c076b10 msgr2=0x7f818c19bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818aa00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f818c076b10 0x7f818c19bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.551+0000 7f818aa00640 1 -- 172.21.15.97:0/1754964936 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818c19d270 con 0x7f818c075720 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.552+0000 7f818aa00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c075720 0x7f818c19b8d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f817c00c920 tx=0x7f817c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.552+0000 7f8183400640 1 -- 172.21.15.97:0/1754964936 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c007c70 con 0x7f818c075720 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.552+0000 7f8183400640 1 -- 172.21.15.97:0/1754964936 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f817c007e10 con 0x7f818c075720 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.552+0000 7f8183400640 1 -- 172.21.15.97:0/1754964936 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c018710 con 0x7f818c075720 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.552+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818c19d560 con 0x7f818c075720 2024-07-26T07:39:03.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.552+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818c1a3b20 con 0x7f818c075720 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.554+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f818c109d50 con 0x7f818c075720 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.559+0000 7f8183400640 1 -- 172.21.15.97:0/1754964936 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f817c00f450 con 0x7f818c075720 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.561+0000 7f8183400640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81700b3b40 0x7f81700b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.561+0000 7f8183400640 1 -- 172.21.15.97:0/1754964936 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f817c0e2110 con 0x7f818c075720 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.562+0000 7f818a000640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81700b3b40 0x7f81700b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.562+0000 7f818a000640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81700b3b40 0x7f81700b6050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f817800bbe0 tx=0x7f817800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.569+0000 7f8183400640 1 -- 172.21.15.97:0/1754964936 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f817c014030 con 0x7f818c075720 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.819+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f818c02e0c0 con 0x7f818c075720 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.824+0000 7f8183400640 1 -- 172.21.15.97:0/1754964936 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v16)=0 v16) ==== 127+0+0 (secure 0 0 0) 0x7f817c090e20 con 0x7f818c075720 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.827+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81700b3b40 msgr2=0x7f81700b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.827+0000 7f818be00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81700b3b40 0x7f81700b6050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f817800bbe0 tx=0x7f817800b390 comp rx=0 tx=0).stop 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c075720 msgr2=0x7f818c19b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c075720 0x7f818c19b8d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f817c00c920 tx=0x7f817c00cde0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 shutdown_connections 2024-07-26T07:39:03.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81700b3b40 0x7f81700b6050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f818c10af40 0x7f818c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f818c076b10 0x7f818c19bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 --2- 172.21.15.97:0/1754964936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f818c075720 0x7f818c19b8d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:03.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 >> 172.21.15.97:0/1754964936 conn(0x7f818c0fe8f0 msgr2=0x7f818c106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:03.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 shutdown_connections 2024-07-26T07:39:03.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:01.828+0000 7f818be00640 1 -- 172.21.15.97:0/1754964936 wait complete. 2024-07-26T07:39:04.373 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:04 smithi097 ceph-mon[32231]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:04 smithi115 ceph-mon[35364]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:04 smithi181 ceph-mon[35869]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:04.804 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:39:05.035 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:39:06.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:06 smithi115 ceph-mon[35364]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:06 smithi181 ceph-mon[35869]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:06 smithi097 ceph-mon[32231]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:07.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.668+0000 7f426a800640 1 -- 172.21.15.97:0/3358009990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4264108d00 msgr2=0x7f426410b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:07.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.668+0000 7f426a800640 1 --2- 172.21.15.97:0/3358009990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4264108d00 0x7f426410b210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4254009900 tx=0x7f425402a410 comp rx=0 tx=0).stop 2024-07-26T07:39:07.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.669+0000 7f426a800640 1 -- 172.21.15.97:0/3358009990 shutdown_connections 2024-07-26T07:39:07.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.669+0000 7f426a800640 1 --2- 172.21.15.97:0/3358009990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4264108d00 0x7f426410b210 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.669+0000 7f426a800640 1 --2- 172.21.15.97:0/3358009990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4264106160 0x7f42641085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.669+0000 7f426a800640 1 --2- 172.21.15.97:0/3358009990 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f426406bd70 0x7f4264105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.669+0000 7f426a800640 1 -- 172.21.15.97:0/3358009990 >> 172.21.15.97:0/3358009990 conn(0x7f42640fd880 msgr2=0x7f42640ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.669+0000 7f426a800640 1 -- 172.21.15.97:0/3358009990 shutdown_connections 2024-07-26T07:39:07.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.669+0000 7f426a800640 1 -- 172.21.15.97:0/3358009990 wait complete. 2024-07-26T07:39:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.670+0000 7f426a800640 1 Processor -- start 2024-07-26T07:39:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.670+0000 7f426a800640 1 -- start start 2024-07-26T07:39:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f426a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f426406bd70 0x7f42641996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f426a800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4264106160 0x7f4264199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f426a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4264108d00 0x7f426419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f426a800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f426410ead0 con 0x7f426406bd70 2024-07-26T07:39:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f426a800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f426410e950 con 0x7f4264106160 2024-07-26T07:39:07.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f426a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f426410ec50 con 0x7f4264108d00 2024-07-26T07:39:07.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f4269400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f426406bd70 0x7f42641996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:07.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f4269400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f426406bd70 0x7f42641996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40286/0 (socket says 172.21.15.97:40286) 2024-07-26T07:39:07.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f4269400640 1 -- 172.21.15.97:0/988461629 learned_addr learned my addr 172.21.15.97:0/988461629 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:07.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f4269e00640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4264108d00 0x7f426419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:07.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.671+0000 7f4268a00640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4264106160 0x7f4264199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:07.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4269400640 1 -- 172.21.15.97:0/988461629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4264108d00 msgr2=0x7f426419a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:07.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4269400640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4264108d00 0x7f426419a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4269400640 1 -- 172.21.15.97:0/988461629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4264106160 msgr2=0x7f4264199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:07.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4269400640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4264106160 0x7f4264199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4269400640 1 -- 172.21.15.97:0/988461629 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426419b100 con 0x7f426406bd70 2024-07-26T07:39:07.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4269400640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f426406bd70 0x7f42641996d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f425c00eab0 tx=0x7f425c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:07.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4262000640 1 -- 172.21.15.97:0/988461629 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c00ce40 con 0x7f426406bd70 2024-07-26T07:39:07.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4262000640 1 -- 172.21.15.97:0/988461629 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f425c011900 con 0x7f426406bd70 2024-07-26T07:39:07.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f4262000640 1 -- 172.21.15.97:0/988461629 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c010990 con 0x7f426406bd70 2024-07-26T07:39:07.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.672+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f426419b3f0 con 0x7f426406bd70 2024-07-26T07:39:07.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.673+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42641a1af0 con 0x7f426406bd70 2024-07-26T07:39:07.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.674+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42640ff3a0 con 0x7f426406bd70 2024-07-26T07:39:07.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.676+0000 7f4262000640 1 -- 172.21.15.97:0/988461629 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f425c011470 con 0x7f426406bd70 2024-07-26T07:39:07.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.679+0000 7f4262000640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f42480b3c40 0x7f42480b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:07.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.679+0000 7f4262000640 1 -- 172.21.15.97:0/988461629 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f425c0e25c0 con 0x7f426406bd70 2024-07-26T07:39:07.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.679+0000 7f4268a00640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f42480b3c40 0x7f42480b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:07.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.680+0000 7f4268a00640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f42480b3c40 0x7f42480b6150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4250005fd0 tx=0x7f4250005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:07.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.683+0000 7f4262000640 1 -- 172.21.15.97:0/988461629 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f425c016030 con 0x7f426406bd70 2024-07-26T07:39:07.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.883+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f426402e0c0 con 0x7f426406bd70 2024-07-26T07:39:07.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.890+0000 7f4262000640 1 -- 172.21.15.97:0/988461629 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v18)=0 v18) ==== 106+0+0 (secure 0 0 0) 0x7f425c0ab840 con 0x7f426406bd70 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.893+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f42480b3c40 msgr2=0x7f42480b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.893+0000 7f426a800640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f42480b3c40 0x7f42480b6150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4250005fd0 tx=0x7f4250005850 comp rx=0 tx=0).stop 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.893+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f426406bd70 msgr2=0x7f42641996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.893+0000 7f426a800640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f426406bd70 0x7f42641996d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f425c00eab0 tx=0x7f425c00ef70 comp rx=0 tx=0).stop 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.894+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 shutdown_connections 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.894+0000 7f426a800640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f42480b3c40 0x7f42480b6150 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.894+0000 7f426a800640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4264108d00 0x7f426419a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.894+0000 7f426a800640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4264106160 0x7f4264199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.894+0000 7f426a800640 1 --2- 172.21.15.97:0/988461629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f426406bd70 0x7f42641996d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:07.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.894+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 >> 172.21.15.97:0/988461629 conn(0x7f42640fd880 msgr2=0x7f4264104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:07.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.894+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 shutdown_connections 2024-07-26T07:39:07.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:07.894+0000 7f426a800640 1 -- 172.21.15.97:0/988461629 wait complete. 2024-07-26T07:39:09.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:08 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/988461629' entity='client.admin' 2024-07-26T07:39:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:08 smithi115 ceph-mon[35364]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:08 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/988461629' entity='client.admin' 2024-07-26T07:39:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:08 smithi181 ceph-mon[35869]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:09.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:08 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/988461629' entity='client.admin' 2024-07-26T07:39:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:08 smithi097 ceph-mon[32231]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:10.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.556+0000 7f6e5ca00640 1 -- 172.21.15.97:0/2830499110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 msgr2=0x7f6e5810d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.556+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/2830499110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 0x7f6e5810d5a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009ca0 tx=0x7f6e5002a950 comp rx=0 tx=0).stop 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.556+0000 7f6e5ca00640 1 -- 172.21.15.97:0/2830499110 shutdown_connections 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.556+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/2830499110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 0x7f6e5810d5a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.556+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/2830499110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e58106260 0x7f6e58106670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.556+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/2830499110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e581051d0 0x7f6e58105640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.556+0000 7f6e5ca00640 1 -- 172.21.15.97:0/2830499110 >> 172.21.15.97:0/2830499110 conn(0x7f6e58100b60 msgr2=0x7f6e58102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.556+0000 7f6e5ca00640 1 -- 172.21.15.97:0/2830499110 shutdown_connections 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.557+0000 7f6e5ca00640 1 -- 172.21.15.97:0/2830499110 wait complete. 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e5ca00640 1 Processor -- start 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e5ca00640 1 -- start start 2024-07-26T07:39:10.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e581051d0 0x7f6e5819b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e58106260 0x7f6e5819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 0x7f6e5819cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e5ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6e58110d30 con 0x7f6e58106c00 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e5ca00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6e58110bb0 con 0x7f6e58106260 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e5ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6e58110eb0 con 0x7f6e581051d0 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e57200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 0x7f6e5819cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e57200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 0x7f6e5819cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40298/0 (socket says 172.21.15.97:40298) 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.558+0000 7f6e57200640 1 -- 172.21.15.97:0/1015961956 learned_addr learned my addr 172.21.15.97:0/1015961956 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.559+0000 7f6e56800640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e581051d0 0x7f6e5819b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.559+0000 7f6e55e00640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e58106260 0x7f6e5819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.559+0000 7f6e57200640 1 -- 172.21.15.97:0/1015961956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e581051d0 msgr2=0x7f6e5819b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:10.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.559+0000 7f6e57200640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e581051d0 0x7f6e5819b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.559+0000 7f6e57200640 1 -- 172.21.15.97:0/1015961956 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e58106260 msgr2=0x7f6e5819bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:10.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.559+0000 7f6e57200640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e58106260 0x7f6e5819bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.559+0000 7f6e57200640 1 -- 172.21.15.97:0/1015961956 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5819d330 con 0x7f6e58106c00 2024-07-26T07:39:10.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.559+0000 7f6e57200640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 0x7f6e5819cb10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e5002af40 tx=0x7f6e50003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:10.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.560+0000 7f6e4f400640 1 -- 172.21.15.97:0/1015961956 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50034040 con 0x7f6e58106c00 2024-07-26T07:39:10.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.560+0000 7f6e4f400640 1 -- 172.21.15.97:0/1015961956 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e50030850 con 0x7f6e58106c00 2024-07-26T07:39:10.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.560+0000 7f6e4f400640 1 -- 172.21.15.97:0/1015961956 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50004760 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.560+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e5819d5c0 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.560+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e581a3b50 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.561+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5810c3b0 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.563+0000 7f6e4f400640 1 -- 172.21.15.97:0/1015961956 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f6e50037020 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.565+0000 7f6e4f400640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6e3c0b3bc0 0x7f6e3c0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.565+0000 7f6e4f400640 1 -- 172.21.15.97:0/1015961956 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6e50103190 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.565+0000 7f6e56800640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6e3c0b3bc0 0x7f6e3c0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.566+0000 7f6e56800640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6e3c0b3bc0 0x7f6e3c0b60d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005fd0 tx=0x7f6e48005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.567+0000 7f6e4f400640 1 -- 172.21.15.97:0/1015961956 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6e50082600 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.769+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f6e5802e0c0 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.779+0000 7f6e4f400640 1 -- 172.21.15.97:0/1015961956 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v19)=0 v19) ==== 122+0+0 (secure 0 0 0) 0x7f6e500cc4b0 con 0x7f6e58106c00 2024-07-26T07:39:10.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6e3c0b3bc0 msgr2=0x7f6e3c0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6e3c0b3bc0 0x7f6e3c0b60d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005fd0 tx=0x7f6e48005850 comp rx=0 tx=0).stop 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 msgr2=0x7f6e5819cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 0x7f6e5819cb10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e5002af40 tx=0x7f6e50003f10 comp rx=0 tx=0).stop 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 shutdown_connections 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6e3c0b3bc0 0x7f6e3c0b60d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6e58106c00 0x7f6e5819cb10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e58106260 0x7f6e5819bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 --2- 172.21.15.97:0/1015961956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e581051d0 0x7f6e5819b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 >> 172.21.15.97:0/1015961956 conn(0x7f6e58100b60 msgr2=0x7f6e58109580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 shutdown_connections 2024-07-26T07:39:10.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:09.782+0000 7f6e5ca00640 1 -- 172.21.15.97:0/1015961956 wait complete. 2024-07-26T07:39:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:10 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1015961956' entity='client.admin' 2024-07-26T07:39:11.050 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:10 smithi181 ceph-mon[35869]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:11.051 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:10 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1015961956' entity='client.admin' 2024-07-26T07:39:11.051 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:10 smithi097 ceph-mon[32231]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:10 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1015961956' entity='client.admin' 2024-07-26T07:39:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:10 smithi115 ceph-mon[35364]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:12.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:12 smithi115 ceph-mon[35364]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:12.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:12 smithi097 ceph-mon[32231]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:12 smithi181 ceph-mon[35869]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:14.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.146+0000 7fdab6c00640 1 -- 172.21.15.97:0/3110080545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104bb0 msgr2=0x7fdab010b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:14.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.146+0000 7fdab6c00640 1 --2- 172.21.15.97:0/3110080545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104bb0 0x7fdab010b550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009900 tx=0x7fdaa802a480 comp rx=0 tx=0).stop 2024-07-26T07:39:14.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.146+0000 7fdab6c00640 1 -- 172.21.15.97:0/3110080545 shutdown_connections 2024-07-26T07:39:14.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.146+0000 7fdab6c00640 1 --2- 172.21.15.97:0/3110080545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104bb0 0x7fdab010b550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.146+0000 7fdab6c00640 1 --2- 172.21.15.97:0/3110080545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdab0104210 0x7fdab0104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.146+0000 7fdab6c00640 1 --2- 172.21.15.97:0/3110080545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab0103180 0x7fdab01035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.146+0000 7fdab6c00640 1 -- 172.21.15.97:0/3110080545 >> 172.21.15.97:0/3110080545 conn(0x7fdab00feaf0 msgr2=0x7fdab0100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:14.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.147+0000 7fdab6c00640 1 -- 172.21.15.97:0/3110080545 shutdown_connections 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.147+0000 7fdab6c00640 1 -- 172.21.15.97:0/3110080545 wait complete. 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.148+0000 7fdab6c00640 1 Processor -- start 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.148+0000 7fdab6c00640 1 -- start start 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.148+0000 7fdab6c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdab0103180 0x7fdab019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.148+0000 7fdab6c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104210 0x7fdab019c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.148+0000 7fdab6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab0104bb0 0x7fdab019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.148+0000 7fdab6c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdab010ec90 con 0x7fdab0104210 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.148+0000 7fdab6c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdab010eb10 con 0x7fdab0103180 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.148+0000 7fdab6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdab010ee10 con 0x7fdab0104bb0 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104210 0x7fdab019c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104210 0x7fdab019c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40344/0 (socket says 172.21.15.97:40344) 2024-07-26T07:39:14.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 -- 172.21.15.97:0/504246872 learned_addr learned my addr 172.21.15.97:0/504246872 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaafe00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdab0103180 0x7fdab019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdab4c00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab0104bb0 0x7fdab019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 -- 172.21.15.97:0/504246872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab0104bb0 msgr2=0x7fdab019cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab0104bb0 0x7fdab019cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 -- 172.21.15.97:0/504246872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdab0103180 msgr2=0x7fdab019ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdab0103180 0x7fdab019ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 -- 172.21.15.97:0/504246872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab019d4b0 con 0x7fdab0104210 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.149+0000 7fdaa7400640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104210 0x7fdab019c0f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fda9800d950 tx=0x7fda9800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.150+0000 7fdaad600640 1 -- 172.21.15.97:0/504246872 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98017070 con 0x7fdab0104210 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.150+0000 7fdaad600640 1 -- 172.21.15.97:0/504246872 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fda98009c30 con 0x7fdab0104210 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.150+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab019d7a0 con 0x7fdab0104210 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.150+0000 7fdaad600640 1 -- 172.21.15.97:0/504246872 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9800f620 con 0x7fdab0104210 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.150+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab01a3ea0 con 0x7fdab0104210 2024-07-26T07:39:14.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.151+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab01a4070 con 0x7fdab0104210 2024-07-26T07:39:14.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.153+0000 7fdaad600640 1 -- 172.21.15.97:0/504246872 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fda98010460 con 0x7fdab0104210 2024-07-26T07:39:14.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.155+0000 7fdaad600640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fda940b3c40 0x7fda940b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:14.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.155+0000 7fdaad600640 1 -- 172.21.15.97:0/504246872 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fda980e1bd0 con 0x7fdab0104210 2024-07-26T07:39:14.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.155+0000 7fdaafe00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fda940b3c40 0x7fda940b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:14.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.156+0000 7fdaafe00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fda940b3c40 0x7fda940b6150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0005fd0 tx=0x7fdaa0009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:14.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.159+0000 7fdaad600640 1 -- 172.21.15.97:0/504246872 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fda980aae50 con 0x7fdab0104210 2024-07-26T07:39:14.302 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:14 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2333744569' entity='client.admin' 2024-07-26T07:39:14.302 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:14.302 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:14.302 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:39:14.302 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:14 smithi097 ceph-mon[32231]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:14.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.405+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fdab01a4240 con 0x7fdab0104210 2024-07-26T07:39:14.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.411+0000 7fdaad600640 1 -- 172.21.15.97:0/504246872 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v22)=0 v22) ==== 108+0+0 (secure 0 0 0) 0x7fda98009de0 con 0x7fdab0104210 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fda940b3c40 msgr2=0x7fda940b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fda940b3c40 0x7fda940b6150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0005fd0 tx=0x7fdaa0009870 comp rx=0 tx=0).stop 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104210 msgr2=0x7fdab019c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104210 0x7fdab019c0f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fda9800d950 tx=0x7fda9800de10 comp rx=0 tx=0).stop 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 shutdown_connections 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fda940b3c40 0x7fda940b6150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab0104bb0 0x7fdab019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab0104210 0x7fdab019c0f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 --2- 172.21.15.97:0/504246872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdab0103180 0x7fdab019ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:14.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 >> 172.21.15.97:0/504246872 conn(0x7fdab00feaf0 msgr2=0x7fdab0075020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:14.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 shutdown_connections 2024-07-26T07:39:14.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.414+0000 7fdab6c00640 1 -- 172.21.15.97:0/504246872 wait complete. 2024-07-26T07:39:14.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:14 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2333744569' entity='client.admin' 2024-07-26T07:39:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:39:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:14 smithi115 ceph-mon[35364]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:14 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2333744569' entity='client.admin' 2024-07-26T07:39:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:39:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:14 smithi181 ceph-mon[35869]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:15.129 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:15 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2122195793' entity='client.admin' 2024-07-26T07:39:15.130 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:15.130 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:39:15.130 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:15.130 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:15 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/504246872' entity='client.admin' 2024-07-26T07:39:15.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 -- 172.21.15.97:0/2918606600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103100 msgr2=0x7f8540103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 --2- 172.21.15.97:0/2918606600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103100 0x7f8540103510 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f852c009900 tx=0x7f852c02a480 comp rx=0 tx=0).stop 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 -- 172.21.15.97:0/2918606600 shutdown_connections 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 --2- 172.21.15.97:0/2918606600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8540103aa0 0x7f854010a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 --2- 172.21.15.97:0/2918606600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103100 0x7f8540103510 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 --2- 172.21.15.97:0/2918606600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8540102070 0x7f85401024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 -- 172.21.15.97:0/2918606600 >> 172.21.15.97:0/2918606600 conn(0x7f85400fd9e0 msgr2=0x7f85400ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 -- 172.21.15.97:0/2918606600 shutdown_connections 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.805+0000 7f8546200640 1 -- 172.21.15.97:0/2918606600 wait complete. 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.806+0000 7f8546200640 1 Processor -- start 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f8546200640 1 -- start start 2024-07-26T07:39:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f8546200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8540102070 0x7f854019bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f8546200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8540103100 0x7f854019c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f8546200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103aa0 0x7f854019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f8546200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f854010ecf0 con 0x7f8540103aa0 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f8546200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f854010eb70 con 0x7f8540102070 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f8546200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f854010ee70 con 0x7f8540103100 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f853fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103aa0 0x7f854019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f853fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103aa0 0x7f854019cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40322/0 (socket says 172.21.15.97:40322) 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f853fe00640 1 -- 172.21.15.97:0/2333744569 learned_addr learned my addr 172.21.15.97:0/2333744569 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:15.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f853ea00640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8540103100 0x7f854019c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.807+0000 7f853f400640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8540102070 0x7f854019bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f853fe00640 1 -- 172.21.15.97:0/2333744569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8540103100 msgr2=0x7f854019c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f853fe00640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8540103100 0x7f854019c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f853fe00640 1 -- 172.21.15.97:0/2333744569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8540102070 msgr2=0x7f854019bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f853fe00640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8540102070 0x7f854019bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f853fe00640 1 -- 172.21.15.97:0/2333744569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854019d470 con 0x7f8540103aa0 2024-07-26T07:39:15.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f853f400640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8540102070 0x7f854019bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f853ea00640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8540103100 0x7f854019c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f853fe00640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103aa0 0x7f854019cce0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f853800cce0 tx=0x7f853800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f8537e00640 1 -- 172.21.15.97:0/2333744569 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853800f450 con 0x7f8540103aa0 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f8537e00640 1 -- 172.21.15.97:0/2333744569 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f853800fa80 con 0x7f8540103aa0 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854019d760 con 0x7f8540103aa0 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f8537e00640 1 -- 172.21.15.97:0/2333744569 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538018ad0 con 0x7f8540103aa0 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.808+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85401a3d20 con 0x7f8540103aa0 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.809+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85401a3fb0 con 0x7f8540103aa0 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.812+0000 7f8537e00640 1 -- 172.21.15.97:0/2333744569 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f853800f5f0 con 0x7f8540103aa0 2024-07-26T07:39:15.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.814+0000 7f8537e00640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f85240b3bc0 0x7f85240b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.815+0000 7f8537e00640 1 -- 172.21.15.97:0/2333744569 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f85380e28f0 con 0x7f8540103aa0 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.815+0000 7f853f400640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f85240b3bc0 0x7f85240b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.815+0000 7f853f400640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f85240b3bc0 0x7f85240b60d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f853000baa0 tx=0x7f8530005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:12.816+0000 7f8537e00640 1 -- 172.21.15.97:0/2333744569 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8538014030 con 0x7f8540103aa0 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.016+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f85401a4180 con 0x7f8540103aa0 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.027+0000 7f8537e00640 1 -- 172.21.15.97:0/2333744569 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v20)=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f85380abc00 con 0x7f8540103aa0 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.030+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f85240b3bc0 msgr2=0x7f85240b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.030+0000 7f8546200640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f85240b3bc0 0x7f85240b60d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f853000baa0 tx=0x7f8530005d90 comp rx=0 tx=0).stop 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.030+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103aa0 msgr2=0x7f854019cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.030+0000 7f8546200640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103aa0 0x7f854019cce0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f853800cce0 tx=0x7f853800a620 comp rx=0 tx=0).stop 2024-07-26T07:39:15.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.031+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 shutdown_connections 2024-07-26T07:39:15.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.031+0000 7f8546200640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f85240b3bc0 0x7f85240b60d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.031+0000 7f8546200640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8540103aa0 0x7f854019cce0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.031+0000 7f8546200640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8540103100 0x7f854019c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.031+0000 7f8546200640 1 --2- 172.21.15.97:0/2333744569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8540102070 0x7f854019bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:15.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.031+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 >> 172.21.15.97:0/2333744569 conn(0x7f85400fd9e0 msgr2=0x7f854010af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:15.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.031+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 shutdown_connections 2024-07-26T07:39:15.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.031+0000 7f8546200640 1 -- 172.21.15.97:0/2333744569 wait complete. 2024-07-26T07:39:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:15 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2122195793' entity='client.admin' 2024-07-26T07:39:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:39:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:15 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/504246872' entity='client.admin' 2024-07-26T07:39:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:15 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2122195793' entity='client.admin' 2024-07-26T07:39:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:39:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:39:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:15 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/504246872' entity='client.admin' 2024-07-26T07:39:16.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.024+0000 7fa59a800640 1 -- 172.21.15.97:0/474121931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 msgr2=0x7fa594105640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.024+0000 7fa59a800640 1 --2- 172.21.15.97:0/474121931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 0x7fa594105640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa584009900 tx=0x7fa58402a480 comp rx=0 tx=0).stop 2024-07-26T07:39:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.024+0000 7fa59a800640 1 -- 172.21.15.97:0/474121931 shutdown_connections 2024-07-26T07:39:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.024+0000 7fa59a800640 1 --2- 172.21.15.97:0/474121931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa594106c00 0x7fa59410d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.024+0000 7fa59a800640 1 --2- 172.21.15.97:0/474121931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa594106260 0x7fa594106670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.024+0000 7fa59a800640 1 --2- 172.21.15.97:0/474121931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 0x7fa594105640 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.024+0000 7fa59a800640 1 -- 172.21.15.97:0/474121931 >> 172.21.15.97:0/474121931 conn(0x7fa594100b60 msgr2=0x7fa594102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.024+0000 7fa59a800640 1 -- 172.21.15.97:0/474121931 shutdown_connections 2024-07-26T07:39:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.025+0000 7fa59a800640 1 -- 172.21.15.97:0/474121931 wait complete. 2024-07-26T07:39:16.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.026+0000 7fa59a800640 1 Processor -- start 2024-07-26T07:39:16.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.026+0000 7fa59a800640 1 -- start start 2024-07-26T07:39:16.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.026+0000 7fa59a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 0x7fa59419b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:16.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.026+0000 7fa59a800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa594106260 0x7fa59419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.026+0000 7fa59a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa594106c00 0x7fa59419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.026+0000 7fa59a800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa594110ce0 con 0x7fa5941051d0 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.026+0000 7fa59a800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa594110b60 con 0x7fa594106260 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.026+0000 7fa59a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa594110e60 con 0x7fa594106c00 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 0x7fa59419b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 0x7fa59419b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48164/0 (socket says 172.21.15.97:48164) 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599400640 1 -- 172.21.15.97:0/3505849551 learned_addr learned my addr 172.21.15.97:0/3505849551 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa598a00640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa594106260 0x7fa59419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599e00640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa594106c00 0x7fa59419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599400640 1 -- 172.21.15.97:0/3505849551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa594106c00 msgr2=0x7fa59419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599400640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa594106c00 0x7fa59419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599400640 1 -- 172.21.15.97:0/3505849551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa594106260 msgr2=0x7fa59419c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599400640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa594106260 0x7fa59419c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.027+0000 7fa599400640 1 -- 172.21.15.97:0/3505849551 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59419d3c0 con 0x7fa5941051d0 2024-07-26T07:39:16.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.028+0000 7fa599400640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 0x7fa59419b990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa584009230 tx=0x7fa5840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:16.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.028+0000 7fa592000640 1 -- 172.21.15.97:0/3505849551 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58403f030 con 0x7fa5941051d0 2024-07-26T07:39:16.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.028+0000 7fa592000640 1 -- 172.21.15.97:0/3505849551 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa584003c90 con 0x7fa5941051d0 2024-07-26T07:39:16.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.028+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59419d650 con 0x7fa5941051d0 2024-07-26T07:39:16.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.028+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa594077400 con 0x7fa5941051d0 2024-07-26T07:39:16.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.028+0000 7fa592000640 1 -- 172.21.15.97:0/3505849551 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584035070 con 0x7fa5941051d0 2024-07-26T07:39:16.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.030+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa578005290 con 0x7fa5941051d0 2024-07-26T07:39:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.033+0000 7fa592000640 1 -- 172.21.15.97:0/3505849551 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fa584042030 con 0x7fa5941051d0 2024-07-26T07:39:16.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.034+0000 7fa592000640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa57c0b3bc0 0x7fa57c0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:16.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.035+0000 7fa592000640 1 -- 172.21.15.97:0/3505849551 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa5841023e0 con 0x7fa5941051d0 2024-07-26T07:39:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.035+0000 7fa598a00640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa57c0b3bc0 0x7fa57c0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:16.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.036+0000 7fa598a00640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa57c0b3bc0 0x7fa57c0b60d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa58800bbe0 tx=0x7fa58800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:16.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.039+0000 7fa592000640 1 -- 172.21.15.97:0/3505849551 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa5840cb010 con 0x7fa5941051d0 2024-07-26T07:39:16.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:16 smithi097 ceph-mon[32231]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:16.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 -- 172.21.15.97:0/2488202509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac1051c0 msgr2=0x7f6aac105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 --2- 172.21.15.97:0/2488202509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac1051c0 0x7f6aac105630 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009900 tx=0x7f6a9c02a480 comp rx=0 tx=0).stop 2024-07-26T07:39:16.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 -- 172.21.15.97:0/2488202509 shutdown_connections 2024-07-26T07:39:16.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 --2- 172.21.15.97:0/2488202509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6aac106bf0 0x7f6aac10d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 --2- 172.21.15.97:0/2488202509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aac106250 0x7f6aac106660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 --2- 172.21.15.97:0/2488202509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac1051c0 0x7f6aac105630 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 -- 172.21.15.97:0/2488202509 >> 172.21.15.97:0/2488202509 conn(0x7f6aac100b30 msgr2=0x7f6aac102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:16.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 -- 172.21.15.97:0/2488202509 shutdown_connections 2024-07-26T07:39:16.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.816+0000 7f6aab200640 1 -- 172.21.15.97:0/2488202509 wait complete. 2024-07-26T07:39:16.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.817+0000 7f6aab200640 1 Processor -- start 2024-07-26T07:39:16.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aab200640 1 -- start start 2024-07-26T07:39:16.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aab200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6aac1051c0 0x7f6aac19b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:16.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aab200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aac106250 0x7f6aac19c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:16.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aab200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac106bf0 0x7f6aac19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:16.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aab200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac110cd0 con 0x7f6aac106bf0 2024-07-26T07:39:16.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aab200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac110b50 con 0x7f6aac106250 2024-07-26T07:39:16.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aab200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac110e50 con 0x7f6aac1051c0 2024-07-26T07:39:16.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aaa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac106bf0 0x7f6aac19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:16.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aaa800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac106bf0 0x7f6aac19cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40336/0 (socket says 172.21.15.97:40336) 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aa9e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6aac1051c0 0x7f6aac19b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.818+0000 7f6aaa800640 1 -- 172.21.15.97:0/2122195793 learned_addr learned my addr 172.21.15.97:0/2122195793 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aa9400640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aac106250 0x7f6aac19c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aaa800640 1 -- 172.21.15.97:0/2122195793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6aac1051c0 msgr2=0x7f6aac19b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aaa800640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6aac1051c0 0x7f6aac19b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aaa800640 1 -- 172.21.15.97:0/2122195793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aac106250 msgr2=0x7f6aac19c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aaa800640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aac106250 0x7f6aac19c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aaa800640 1 -- 172.21.15.97:0/2122195793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aac19d380 con 0x7f6aac106bf0 2024-07-26T07:39:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aaa800640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac106bf0 0x7f6aac19cb60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400eb10 tx=0x7f6aa400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aa2a00640 1 -- 172.21.15.97:0/2122195793 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4004b20 con 0x7f6aac106bf0 2024-07-26T07:39:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aa2a00640 1 -- 172.21.15.97:0/2122195793 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6aa4004cc0 con 0x7f6aac106bf0 2024-07-26T07:39:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aa2a00640 1 -- 172.21.15.97:0/2122195793 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4010690 con 0x7f6aac106bf0 2024-07-26T07:39:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.819+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aac19d670 con 0x7f6aac106bf0 2024-07-26T07:39:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.820+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aac1a3d70 con 0x7f6aac106bf0 2024-07-26T07:39:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.821+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aac10c3a0 con 0x7f6aac106bf0 2024-07-26T07:39:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.823+0000 7f6aa2a00640 1 -- 172.21.15.97:0/2122195793 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f6aa4005000 con 0x7f6aac106bf0 2024-07-26T07:39:16.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.826+0000 7f6aa2a00640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a900b3bc0 0x7f6a900b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.826+0000 7f6aa2a00640 1 -- 172.21.15.97:0/2122195793 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6aa40e2730 con 0x7f6aac106bf0 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.826+0000 7f6aa9e00640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a900b3bc0 0x7f6a900b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.827+0000 7f6aa9e00640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a900b3bc0 0x7f6a900b60d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c0057c0 tx=0x7f6a9c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:13.829+0000 7f6aa2a00640 1 -- 172.21.15.97:0/2122195793 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6aa4055600 con 0x7f6aac106bf0 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.032+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f6aac02e0c0 con 0x7f6aac106bf0 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.041+0000 7f6aa2a00640 1 -- 172.21.15.97:0/2122195793 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v21)=0 v21) ==== 118+0+0 (secure 0 0 0) 0x7f6aa40ab9b0 con 0x7f6aac106bf0 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.043+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a900b3bc0 msgr2=0x7f6a900b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.043+0000 7f6aab200640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a900b3bc0 0x7f6a900b60d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c0057c0 tx=0x7f6a9c000bc0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac106bf0 msgr2=0x7f6aac19cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac106bf0 0x7f6aac19cb60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400eb10 tx=0x7f6aa400c2d0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 shutdown_connections 2024-07-26T07:39:16.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a900b3bc0 0x7f6a900b60d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6aac106bf0 0x7f6aac19cb60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aac106250 0x7f6aac19c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 --2- 172.21.15.97:0/2122195793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6aac1051c0 0x7f6aac19b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 >> 172.21.15.97:0/2122195793 conn(0x7f6aac100b30 msgr2=0x7f6aac109570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.044+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 shutdown_connections 2024-07-26T07:39:16.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:14.045+0000 7f6aab200640 1 -- 172.21.15.97:0/2122195793 wait complete. 2024-07-26T07:39:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.257+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7fa578005580 con 0x7fa5941051d0 2024-07-26T07:39:16.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.264+0000 7fa592000640 1 -- 172.21.15.97:0/3505849551 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v23)=0 v23) ==== 116+0+0 (secure 0 0 0) 0x7fa5840cf560 con 0x7fa5941051d0 2024-07-26T07:39:16.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa57c0b3bc0 msgr2=0x7fa57c0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa57c0b3bc0 0x7fa57c0b60d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa58800bbe0 tx=0x7fa58800b390 comp rx=0 tx=0).stop 2024-07-26T07:39:16.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 msgr2=0x7fa59419b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:16.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 0x7fa59419b990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa584009230 tx=0x7fa5840042f0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 shutdown_connections 2024-07-26T07:39:16.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa57c0b3bc0 0x7fa57c0b60d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa594106c00 0x7fa59419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa594106260 0x7fa59419c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 --2- 172.21.15.97:0/3505849551 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5941051d0 0x7fa59419b990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:16.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 >> 172.21.15.97:0/3505849551 conn(0x7fa594100b60 msgr2=0x7fa594109580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:16.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 shutdown_connections 2024-07-26T07:39:16.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:16.267+0000 7fa59a800640 1 -- 172.21.15.97:0/3505849551 wait complete. 2024-07-26T07:39:16.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:16 smithi115 ceph-mon[35364]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:16 smithi181 ceph-mon[35869]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:17 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3505849551' entity='client.admin' 2024-07-26T07:39:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:17 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3505849551' entity='client.admin' 2024-07-26T07:39:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:17 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3505849551' entity='client.admin' 2024-07-26T07:39:18.700 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi097') 2024-07-26T07:39:18.700 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi097') 2024-07-26T07:39:18.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 -- 172.21.15.97:0/1994320630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c1031a0 msgr2=0x7f337c103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:18.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 --2- 172.21.15.97:0/1994320630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c1031a0 0x7f337c103610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3370009900 tx=0x7f337002a480 comp rx=0 tx=0).stop 2024-07-26T07:39:18.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 -- 172.21.15.97:0/1994320630 shutdown_connections 2024-07-26T07:39:18.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 --2- 172.21.15.97:0/1994320630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f337c104bd0 0x7f337c10b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 --2- 172.21.15.97:0/1994320630 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f337c104230 0x7f337c104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 --2- 172.21.15.97:0/1994320630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c1031a0 0x7f337c103610 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 -- 172.21.15.97:0/1994320630 >> 172.21.15.97:0/1994320630 conn(0x7f337c0feb10 msgr2=0x7f337c100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 -- 172.21.15.97:0/1994320630 shutdown_connections 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.093+0000 7f3384800640 1 -- 172.21.15.97:0/1994320630 wait complete. 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.094+0000 7f3384800640 1 Processor -- start 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3384800640 1 -- start start 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3384800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f337c1031a0 0x7f337c1998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3384800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c104230 0x7f337c199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3384800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f337c104bd0 0x7f337c19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3384800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f337c10ed00 con 0x7f337c104230 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3384800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f337c10eb80 con 0x7f337c1031a0 2024-07-26T07:39:18.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3384800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f337c10ee80 con 0x7f337c104bd0 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3381400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c104230 0x7f337c199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3381400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c104230 0x7f337c199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48214/0 (socket says 172.21.15.97:48214) 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.095+0000 7f3381400640 1 -- 172.21.15.97:0/1985425268 learned_addr learned my addr 172.21.15.97:0/1985425268 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3381e00640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f337c1031a0 0x7f337c1998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3382800640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f337c104bd0 0x7f337c19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3381400640 1 -- 172.21.15.97:0/1985425268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f337c104bd0 msgr2=0x7f337c19aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3381400640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f337c104bd0 0x7f337c19aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3381400640 1 -- 172.21.15.97:0/1985425268 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f337c1031a0 msgr2=0x7f337c1998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3381400640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f337c1031a0 0x7f337c1998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3381400640 1 -- 172.21.15.97:0/1985425268 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c19b300 con 0x7f337c104230 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3381400640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c104230 0x7f337c199f40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f336c00d950 tx=0x7f336c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:18.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3376a00640 1 -- 172.21.15.97:0/1985425268 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c017070 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337c19b5f0 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3376a00640 1 -- 172.21.15.97:0/1985425268 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f336c009c30 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.096+0000 7f3376a00640 1 -- 172.21.15.97:0/1985425268 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c00f620 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.097+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337c1a1cf0 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.098+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337c10a380 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.100+0000 7f3376a00640 1 -- 172.21.15.97:0/1985425268 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f336c010460 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.101+0000 7f3376a00640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f33640b3bc0 0x7f33640b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.102+0000 7f3376a00640 1 -- 172.21.15.97:0/1985425268 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f336c0e1bd0 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.102+0000 7f3381e00640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f33640b3bc0 0x7f33640b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.102+0000 7f3381e00640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f33640b3bc0 0x7f33640b60d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3370006010 tx=0x7f3370005720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.107+0000 7f3376a00640 1 -- 172.21.15.97:0/1985425268 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f336c0aae50 con 0x7f337c104230 2024-07-26T07:39:18.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.295+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f337c02e0c0 con 0x7f337c104230 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.305+0000 7f3376a00640 1 -- 172.21.15.97:0/1985425268 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v25)=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7f336c009de0 con 0x7f337c104230 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.307+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f33640b3bc0 msgr2=0x7f33640b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.307+0000 7f3384800640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f33640b3bc0 0x7f33640b60d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3370006010 tx=0x7f3370005720 comp rx=0 tx=0).stop 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.307+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c104230 msgr2=0x7f337c199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.307+0000 7f3384800640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c104230 0x7f337c199f40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f336c00d950 tx=0x7f336c00de10 comp rx=0 tx=0).stop 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.308+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 shutdown_connections 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.308+0000 7f3384800640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f33640b3bc0 0x7f33640b60d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.308+0000 7f3384800640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f337c104bd0 0x7f337c19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.308+0000 7f3384800640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f337c104230 0x7f337c199f40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.308+0000 7f3384800640 1 --2- 172.21.15.97:0/1985425268 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f337c1031a0 0x7f337c1998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.308+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 >> 172.21.15.97:0/1985425268 conn(0x7f337c0feb10 msgr2=0x7f337c107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.308+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 shutdown_connections 2024-07-26T07:39:18.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:18.308+0000 7f3384800640 1 -- 172.21.15.97:0/1985425268 wait complete. 2024-07-26T07:39:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:18 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/649009492' entity='client.admin' 2024-07-26T07:39:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:18 smithi115 ceph-mon[35364]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:18 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1985425268' entity='client.admin' 2024-07-26T07:39:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:18 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/649009492' entity='client.admin' 2024-07-26T07:39:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:18 smithi181 ceph-mon[35869]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:18 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1985425268' entity='client.admin' 2024-07-26T07:39:19.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:18 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/649009492' entity='client.admin' 2024-07-26T07:39:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:18 smithi097 ceph-mon[32231]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:18 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1985425268' entity='client.admin' 2024-07-26T07:39:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.302+0000 7f7497200640 1 -- 172.21.15.97:0/3124792338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7498106c10 msgr2=0x7f749810d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.302+0000 7f7497200640 1 --2- 172.21.15.97:0/3124792338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7498106c10 0x7f749810d5b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7490009900 tx=0x7f749002a480 comp rx=0 tx=0).stop 2024-07-26T07:39:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.302+0000 7f7497200640 1 -- 172.21.15.97:0/3124792338 shutdown_connections 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.302+0000 7f7497200640 1 --2- 172.21.15.97:0/3124792338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7498106c10 0x7f749810d5b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.302+0000 7f7497200640 1 --2- 172.21.15.97:0/3124792338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7498106270 0x7f7498106680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.302+0000 7f7497200640 1 --2- 172.21.15.97:0/3124792338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74981051e0 0x7f7498105650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.302+0000 7f7497200640 1 -- 172.21.15.97:0/3124792338 >> 172.21.15.97:0/3124792338 conn(0x7f7498100b50 msgr2=0x7f7498103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.303+0000 7f7497200640 1 -- 172.21.15.97:0/3124792338 shutdown_connections 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.303+0000 7f7497200640 1 -- 172.21.15.97:0/3124792338 wait complete. 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7497200640 1 Processor -- start 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7497200640 1 -- start start 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7497200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74981051e0 0x7f749819b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7497200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7498106270 0x7f749819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7497200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7498106c10 0x7f749819cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7497200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7498110cf0 con 0x7f74981051e0 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7497200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7498110b70 con 0x7f7498106270 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7497200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7498110e70 con 0x7f7498106c10 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7495e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74981051e0 0x7f749819b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7495e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74981051e0 0x7f749819b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48184/0 (socket says 172.21.15.97:48184) 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.304+0000 7f7495e00640 1 -- 172.21.15.97:0/649009492 learned_addr learned my addr 172.21.15.97:0/649009492 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f7495e00640 1 -- 172.21.15.97:0/649009492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7498106c10 msgr2=0x7f749819cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f7495e00640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7498106c10 0x7f749819cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f7495400640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7498106270 0x7f749819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f7495e00640 1 -- 172.21.15.97:0/649009492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7498106270 msgr2=0x7f749819bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f7495e00640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7498106270 0x7f749819bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f7495e00640 1 -- 172.21.15.97:0/649009492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749819d330 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f7495e00640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74981051e0 0x7f749819b900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f748800eab0 tx=0x7f748800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f748ea00640 1 -- 172.21.15.97:0/649009492 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748800ce40 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f748ea00640 1 -- 172.21.15.97:0/649009492 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7488011900 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f748ea00640 1 -- 172.21.15.97:0/649009492 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488010990 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.305+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f749819d620 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.306+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74981a3d20 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.307+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f749810c3c0 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.309+0000 7f748ea00640 1 -- 172.21.15.97:0/649009492 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f7488011470 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.310+0000 7f748ea00640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f747c0b3bc0 0x7f747c0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.311+0000 7f748ea00640 1 -- 172.21.15.97:0/649009492 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f74880e25a0 con 0x7f74981051e0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.311+0000 7f7495400640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f747c0b3bc0 0x7f747c0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.311+0000 7f7495400640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f747c0b3bc0 0x7f747c0b60d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7480005fd0 tx=0x7f7480005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.312+0000 7f748ea00640 1 -- 172.21.15.97:0/649009492 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7488016030 con 0x7f74981051e0 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.513+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f749802e0c0 con 0x7f74981051e0 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.524+0000 7f748ea00640 1 -- 172.21.15.97:0/649009492 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v24)=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7f74880ab8a0 con 0x7f74981051e0 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f747c0b3bc0 msgr2=0x7f747c0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f747c0b3bc0 0x7f747c0b60d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7480005fd0 tx=0x7f7480005850 comp rx=0 tx=0).stop 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74981051e0 msgr2=0x7f749819b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74981051e0 0x7f749819b900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f748800eab0 tx=0x7f748800ef70 comp rx=0 tx=0).stop 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 shutdown_connections 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f747c0b3bc0 0x7f747c0b60d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7498106c10 0x7f749819cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7498106270 0x7f749819bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 --2- 172.21.15.97:0/649009492 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f74981051e0 0x7f749819b900 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 >> 172.21.15.97:0/649009492 conn(0x7f7498100b50 msgr2=0x7f7498109590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 shutdown_connections 2024-07-26T07:39:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:17.527+0000 7f7497200640 1 -- 172.21.15.97:0/649009492 wait complete. 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.590+0000 7f54a4800640 1 -- 172.21.15.97:0/967875905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 msgr2=0x7f549c112c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.590+0000 7f54a4800640 1 --2- 172.21.15.97:0/967875905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 0x7f549c112c60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5498009ca0 tx=0x7f549802a950 comp rx=0 tx=0).stop 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.591+0000 7f54a4800640 1 -- 172.21.15.97:0/967875905 shutdown_connections 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.591+0000 7f54a4800640 1 --2- 172.21.15.97:0/967875905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 0x7f549c112c60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.591+0000 7f54a4800640 1 --2- 172.21.15.97:0/967875905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f549c10b920 0x7f549c10bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.591+0000 7f54a4800640 1 --2- 172.21.15.97:0/967875905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f549c10a890 0x7f549c10ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.591+0000 7f54a4800640 1 -- 172.21.15.97:0/967875905 >> 172.21.15.97:0/967875905 conn(0x7f549c071430 msgr2=0x7f549c0738e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.591+0000 7f54a4800640 1 -- 172.21.15.97:0/967875905 shutdown_connections 2024-07-26T07:39:19.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.591+0000 7f54a4800640 1 -- 172.21.15.97:0/967875905 wait complete. 2024-07-26T07:39:19.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.592+0000 7f54a4800640 1 Processor -- start 2024-07-26T07:39:19.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.592+0000 7f54a4800640 1 -- start start 2024-07-26T07:39:19.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.592+0000 7f54a4800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f549c10a890 0x7f549c1a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:19.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.592+0000 7f54a4800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f549c10b920 0x7f549c1a2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:19.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.592+0000 7f54a4800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 0x7f549c1a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:19.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.592+0000 7f54a4800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f549c116370 con 0x7f549c10c2c0 2024-07-26T07:39:19.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.592+0000 7f54a4800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1161f0 con 0x7f549c10a890 2024-07-26T07:39:19.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.592+0000 7f54a4800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1164f0 con 0x7f549c10b920 2024-07-26T07:39:19.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 0x7f549c1a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:19.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 0x7f549c1a32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48234/0 (socket says 172.21.15.97:48234) 2024-07-26T07:39:19.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a2800640 1 -- 172.21.15.97:0/567596607 learned_addr learned my addr 172.21.15.97:0/567596607 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:19.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a1e00640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f549c10a890 0x7f549c1a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:19.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a1400640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f549c10b920 0x7f549c1a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a2800640 1 -- 172.21.15.97:0/567596607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f549c10b920 msgr2=0x7f549c1a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a2800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f549c10b920 0x7f549c1a2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a2800640 1 -- 172.21.15.97:0/567596607 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f549c10a890 msgr2=0x7f549c1a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a2800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f549c10a890 0x7f549c1a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a2800640 1 -- 172.21.15.97:0/567596607 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c1a3b00 con 0x7f549c10c2c0 2024-07-26T07:39:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.593+0000 7f54a1400640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f549c10b920 0x7f549c1a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:39:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.594+0000 7f54a2800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 0x7f549c1a32e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f549802f800 tx=0x7f5498003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.594+0000 7f5496a00640 1 -- 172.21.15.97:0/567596607 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498034040 con 0x7f549c10c2c0 2024-07-26T07:39:19.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.594+0000 7f5496a00640 1 -- 172.21.15.97:0/567596607 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5498030850 con 0x7f549c10c2c0 2024-07-26T07:39:19.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.594+0000 7f5496a00640 1 -- 172.21.15.97:0/567596607 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498039430 con 0x7f549c10c2c0 2024-07-26T07:39:19.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.594+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549c1a3d90 con 0x7f549c10c2c0 2024-07-26T07:39:19.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.594+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549c1aa320 con 0x7f549c10c2c0 2024-07-26T07:39:19.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.595+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549c1aa610 con 0x7f549c10c2c0 2024-07-26T07:39:19.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.598+0000 7f5496a00640 1 -- 172.21.15.97:0/567596607 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f5498037020 con 0x7f549c10c2c0 2024-07-26T07:39:19.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.599+0000 7f5496a00640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54840b3bc0 0x7f54840b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:19.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.599+0000 7f5496a00640 1 -- 172.21.15.97:0/567596607 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5498103190 con 0x7f549c10c2c0 2024-07-26T07:39:19.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.600+0000 7f54a1e00640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54840b3bc0 0x7f54840b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:19.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.600+0000 7f54a1e00640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54840b3bc0 0x7f54840b60d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f549000bbe0 tx=0x7f549000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:19.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.602+0000 7f5496a00640 1 -- 172.21.15.97:0/567596607 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f54980cc4a0 con 0x7f549c10c2c0 2024-07-26T07:39:19.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.810+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f549c1aa990 con 0x7f549c10c2c0 2024-07-26T07:39:19.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.818+0000 7f5496a00640 1 -- 172.21.15.97:0/567596607 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v26)=0 v26) ==== 125+0+0 (secure 0 0 0) 0x7f54980cbda0 con 0x7f549c10c2c0 2024-07-26T07:39:19.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54840b3bc0 msgr2=0x7f54840b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54840b3bc0 0x7f54840b60d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f549000bbe0 tx=0x7f549000b390 comp rx=0 tx=0).stop 2024-07-26T07:39:19.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 msgr2=0x7f549c1a32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:19.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 0x7f549c1a32e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f549802f800 tx=0x7f5498003f90 comp rx=0 tx=0).stop 2024-07-26T07:39:19.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 shutdown_connections 2024-07-26T07:39:19.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f54840b3bc0 0x7f54840b60d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f549c10c2c0 0x7f549c1a32e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f549c10b920 0x7f549c1a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 --2- 172.21.15.97:0/567596607 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f549c10a890 0x7f549c1a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:19.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 >> 172.21.15.97:0/567596607 conn(0x7f549c071430 msgr2=0x7f549c072d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:19.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 shutdown_connections 2024-07-26T07:39:19.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:19.822+0000 7f54a4800640 1 -- 172.21.15.97:0/567596607 wait complete. 2024-07-26T07:39:20.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.905+0000 7fde1d200640 1 -- 172.21.15.97:0/1951698402 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 msgr2=0x7fde18109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:20.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.905+0000 7fde1d200640 1 --2- 172.21.15.97:0/1951698402 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 0x7fde18109a40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fde08009900 tx=0x7fde0802a480 comp rx=0 tx=0).stop 2024-07-26T07:39:20.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.905+0000 7fde1d200640 1 -- 172.21.15.97:0/1951698402 shutdown_connections 2024-07-26T07:39:20.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.905+0000 7fde1d200640 1 --2- 172.21.15.97:0/1951698402 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde1810b000 0x7fde181119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:20.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.905+0000 7fde1d200640 1 --2- 172.21.15.97:0/1951698402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde1810a660 0x7fde1810aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:20.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.905+0000 7fde1d200640 1 --2- 172.21.15.97:0/1951698402 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 0x7fde18109a40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.906+0000 7fde1d200640 1 -- 172.21.15.97:0/1951698402 >> 172.21.15.97:0/1951698402 conn(0x7fde18078970 msgr2=0x7fde1807ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.906+0000 7fde1d200640 1 -- 172.21.15.97:0/1951698402 shutdown_connections 2024-07-26T07:39:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.906+0000 7fde1d200640 1 -- 172.21.15.97:0/1951698402 wait complete. 2024-07-26T07:39:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.907+0000 7fde1d200640 1 Processor -- start 2024-07-26T07:39:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.907+0000 7fde1d200640 1 -- start start 2024-07-26T07:39:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.907+0000 7fde1d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 0x7fde1819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:20.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.907+0000 7fde1d200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde1810a660 0x7fde181a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:20.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.907+0000 7fde1d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde1810b000 0x7fde181a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:20.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.907+0000 7fde1d200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fde18115110 con 0x7fde181095d0 2024-07-26T07:39:20.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.907+0000 7fde1d200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fde18114f90 con 0x7fde1810a660 2024-07-26T07:39:20.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.907+0000 7fde1d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fde18115290 con 0x7fde1810b000 2024-07-26T07:39:20.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 0x7fde1819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:20.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 0x7fde1819fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48258/0 (socket says 172.21.15.97:48258) 2024-07-26T07:39:20.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 -- 172.21.15.97:0/3299955764 learned_addr learned my addr 172.21.15.97:0/3299955764 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:20.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde16a00640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde1810a660 0x7fde181a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:20.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17e00640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde1810b000 0x7fde181a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:20.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 -- 172.21.15.97:0/3299955764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde1810b000 msgr2=0x7fde181a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:20.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde1810b000 0x7fde181a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:20.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 -- 172.21.15.97:0/3299955764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde1810a660 msgr2=0x7fde181a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:20.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde1810a660 0x7fde181a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:20.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 -- 172.21.15.97:0/3299955764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde181a1800 con 0x7fde181095d0 2024-07-26T07:39:20.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde16a00640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde1810a660 0x7fde181a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:39:20.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17e00640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde1810b000 0x7fde181a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:39:20.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.908+0000 7fde17400640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 0x7fde1819fdd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fde08005760 tx=0x7fde08003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:20.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.909+0000 7fde0fe00640 1 -- 172.21.15.97:0/3299955764 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0803f070 con 0x7fde181095d0 2024-07-26T07:39:20.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.909+0000 7fde0fe00640 1 -- 172.21.15.97:0/3299955764 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde08031da0 con 0x7fde181095d0 2024-07-26T07:39:20.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.909+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde181a1a90 con 0x7fde181095d0 2024-07-26T07:39:20.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.909+0000 7fde0fe00640 1 -- 172.21.15.97:0/3299955764 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde08035030 con 0x7fde181095d0 2024-07-26T07:39:20.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.909+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde181a8020 con 0x7fde181095d0 2024-07-26T07:39:20.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.910+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde181107b0 con 0x7fde181095d0 2024-07-26T07:39:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.913+0000 7fde0fe00640 1 -- 172.21.15.97:0/3299955764 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fde08042070 con 0x7fde181095d0 2024-07-26T07:39:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.914+0000 7fde0fe00640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fddfc0b3b40 0x7fddfc0b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.914+0000 7fde0fe00640 1 -- 172.21.15.97:0/3299955764 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fde08103050 con 0x7fde181095d0 2024-07-26T07:39:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.914+0000 7fde16a00640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fddfc0b3b40 0x7fddfc0b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.915+0000 7fde16a00640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fddfc0b3b40 0x7fddfc0b6050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde00009b40 tx=0x7fde00005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:20.916+0000 7fde0fe00640 1 -- 172.21.15.97:0/3299955764 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fde080cc370 con 0x7fde181095d0 2024-07-26T07:39:21.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.116+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fde1802e0c0 con 0x7fde181095d0 2024-07-26T07:39:21.117 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:20 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/567596607' entity='client.admin' 2024-07-26T07:39:21.117 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:20 smithi097 ceph-mon[32231]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:21.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.127+0000 7fde0fe00640 1 -- 172.21.15.97:0/3299955764 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v27)=0 v27) ==== 121+0+0 (secure 0 0 0) 0x7fde080cbc70 con 0x7fde181095d0 2024-07-26T07:39:21.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.130+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fddfc0b3b40 msgr2=0x7fddfc0b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:21.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.130+0000 7fde1d200640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fddfc0b3b40 0x7fddfc0b6050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde00009b40 tx=0x7fde00005e50 comp rx=0 tx=0).stop 2024-07-26T07:39:21.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.130+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 msgr2=0x7fde1819fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:21.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.130+0000 7fde1d200640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 0x7fde1819fdd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fde08005760 tx=0x7fde08003d50 comp rx=0 tx=0).stop 2024-07-26T07:39:21.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.131+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 shutdown_connections 2024-07-26T07:39:21.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.131+0000 7fde1d200640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fddfc0b3b40 0x7fddfc0b6050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:21.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.131+0000 7fde1d200640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde1810b000 0x7fde181a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:21.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.131+0000 7fde1d200640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde1810a660 0x7fde181a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:21.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.131+0000 7fde1d200640 1 --2- 172.21.15.97:0/3299955764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fde181095d0 0x7fde1819fdd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:21.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.131+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 >> 172.21.15.97:0/3299955764 conn(0x7fde18078970 msgr2=0x7fde1810d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:21.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.131+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 shutdown_connections 2024-07-26T07:39:21.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:21.131+0000 7fde1d200640 1 -- 172.21.15.97:0/3299955764 wait complete. 2024-07-26T07:39:21.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:20 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/567596607' entity='client.admin' 2024-07-26T07:39:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:20 smithi115 ceph-mon[35364]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:20 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/567596607' entity='client.admin' 2024-07-26T07:39:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:20 smithi181 ceph-mon[35869]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:22 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3299955764' entity='client.admin' 2024-07-26T07:39:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:22 smithi115 ceph-mon[35364]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:22 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3299955764' entity='client.admin' 2024-07-26T07:39:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:22 smithi181 ceph-mon[35869]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:22.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:22 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3299955764' entity='client.admin' 2024-07-26T07:39:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:22 smithi097 ceph-mon[32231]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.261+0000 7f7f10a00640 1 -- 172.21.15.97:0/4216441078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 msgr2=0x7f7f0c078ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.261+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4216441078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 0x7f7f0c078ae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009900 tx=0x7f7ef402a480 comp rx=0 tx=0).stop 2024-07-26T07:39:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.261+0000 7f7f10a00640 1 -- 172.21.15.97:0/4216441078 shutdown_connections 2024-07-26T07:39:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.261+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4216441078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f0c079070 0x7f7f0c0799e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.261+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4216441078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 0x7f7f0c078ae0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.261+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4216441078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f0c077640 0x7f7f0c077ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.261+0000 7f7f10a00640 1 -- 172.21.15.97:0/4216441078 >> 172.21.15.97:0/4216441078 conn(0x7f7f0c100b10 msgr2=0x7f7f0c102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:23.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.262+0000 7f7f10a00640 1 -- 172.21.15.97:0/4216441078 shutdown_connections 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.262+0000 7f7f10a00640 1 -- 172.21.15.97:0/4216441078 wait complete. 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f10a00640 1 Processor -- start 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f10a00640 1 -- start start 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f10a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f0c077640 0x7f7f0c19fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f10a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 0x7f7f0c1a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f10a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f0c079070 0x7f7f0c1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f10a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c115120 con 0x7f7f0c0786d0 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f10a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c114fa0 con 0x7f7f0c077640 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f10a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c1152a0 con 0x7f7f0c079070 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f02a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 0x7f7f0c1a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f02a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 0x7f7f0c1a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48278/0 (socket says 172.21.15.97:48278) 2024-07-26T07:39:23.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.263+0000 7f7f02a00640 1 -- 172.21.15.97:0/4132796621 learned_addr learned my addr 172.21.15.97:0/4132796621 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.264+0000 7f7f0b400640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f0c077640 0x7f7f0c19fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.264+0000 7f7f0be00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f0c079070 0x7f7f0c1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.264+0000 7f7f02a00640 1 -- 172.21.15.97:0/4132796621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f0c079070 msgr2=0x7f7f0c1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.264+0000 7f7f02a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f0c079070 0x7f7f0c1a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.264+0000 7f7f02a00640 1 -- 172.21.15.97:0/4132796621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f0c077640 msgr2=0x7f7f0c19fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.264+0000 7f7f02a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f0c077640 0x7f7f0c19fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.264+0000 7f7f02a00640 1 -- 172.21.15.97:0/4132796621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c1a1760 con 0x7f7f0c0786d0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.264+0000 7f7f02a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 0x7f7f0c1a0430 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4005760 tx=0x7f7ef4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.265+0000 7f7f08c00640 1 -- 172.21.15.97:0/4132796621 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef403f070 con 0x7f7f0c0786d0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.265+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0c1a19f0 con 0x7f7f0c0786d0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.265+0000 7f7f08c00640 1 -- 172.21.15.97:0/4132796621 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ef4031da0 con 0x7f7f0c0786d0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.265+0000 7f7f08c00640 1 -- 172.21.15.97:0/4132796621 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4035030 con 0x7f7f0c0786d0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.265+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0c1a8010 con 0x7f7f0c0786d0 2024-07-26T07:39:23.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.266+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0c1123e0 con 0x7f7f0c0786d0 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.268+0000 7f7f08c00640 1 -- 172.21.15.97:0/4132796621 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f7ef4042070 con 0x7f7f0c0786d0 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.271+0000 7f7f08c00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7ef00b3b40 0x7f7ef00b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.271+0000 7f7f08c00640 1 -- 172.21.15.97:0/4132796621 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7ef4102970 con 0x7f7f0c0786d0 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.271+0000 7f7f0b400640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7ef00b3b40 0x7f7ef00b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.272+0000 7f7f0b400640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7ef00b3b40 0x7f7ef00b6050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7efc005fd0 tx=0x7f7efc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.275+0000 7f7f08c00640 1 -- 172.21.15.97:0/4132796621 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7ef40cbbf0 con 0x7f7f0c0786d0 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.475+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f7f0c02e0c0 con 0x7f7f0c0786d0 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.482+0000 7f7f08c00640 1 -- 172.21.15.97:0/4132796621 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v28)=0 v28) ==== 120+0+0 (secure 0 0 0) 0x7f7ef40cb4f0 con 0x7f7f0c0786d0 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7ef00b3b40 msgr2=0x7f7ef00b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7ef00b3b40 0x7f7ef00b6050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7efc005fd0 tx=0x7f7efc005850 comp rx=0 tx=0).stop 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 msgr2=0x7f7f0c1a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 0x7f7f0c1a0430 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4005760 tx=0x7f7ef4003d50 comp rx=0 tx=0).stop 2024-07-26T07:39:23.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 shutdown_connections 2024-07-26T07:39:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7ef00b3b40 0x7f7ef00b6050 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f0c079070 0x7f7f0c1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f0c0786d0 0x7f7f0c1a0430 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 --2- 172.21.15.97:0/4132796621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f0c077640 0x7f7f0c19fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 >> 172.21.15.97:0/4132796621 conn(0x7f7f0c100b10 msgr2=0x7f7f0c10f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 shutdown_connections 2024-07-26T07:39:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:23.485+0000 7f7f10a00640 1 -- 172.21.15.97:0/4132796621 wait complete. 2024-07-26T07:39:24.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:24 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4132796621' entity='client.admin' 2024-07-26T07:39:24.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:24 smithi097 ceph-mon[32231]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:24.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:24 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4132796621' entity='client.admin' 2024-07-26T07:39:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:24 smithi115 ceph-mon[35364]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:24 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4132796621' entity='client.admin' 2024-07-26T07:39:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:24 smithi181 ceph-mon[35869]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:25.210 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi097') 2024-07-26T07:39:25.210 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi097') 2024-07-26T07:39:25.210 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi097') 2024-07-26T07:39:25.210 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi097') 2024-07-26T07:39:25.210 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi097') 2024-07-26T07:39:25.210 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi097') 2024-07-26T07:39:25.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.834+0000 7fd35ae00640 1 -- 172.21.15.97:0/137752323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106260 msgr2=0x7fd354106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:25.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.834+0000 7fd35ae00640 1 --2- 172.21.15.97:0/137752323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106260 0x7fd354106670 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd340009900 tx=0x7fd34002a480 comp rx=0 tx=0).stop 2024-07-26T07:39:25.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.835+0000 7fd35ae00640 1 -- 172.21.15.97:0/137752323 shutdown_connections 2024-07-26T07:39:25.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.835+0000 7fd35ae00640 1 --2- 172.21.15.97:0/137752323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354106c00 0x7fd35410d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.835+0000 7fd35ae00640 1 --2- 172.21.15.97:0/137752323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106260 0x7fd354106670 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.835+0000 7fd35ae00640 1 --2- 172.21.15.97:0/137752323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3541051d0 0x7fd354105640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.835+0000 7fd35ae00640 1 -- 172.21.15.97:0/137752323 >> 172.21.15.97:0/137752323 conn(0x7fd354100b60 msgr2=0x7fd354102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.835+0000 7fd35ae00640 1 -- 172.21.15.97:0/137752323 shutdown_connections 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.835+0000 7fd35ae00640 1 -- 172.21.15.97:0/137752323 wait complete. 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.836+0000 7fd35ae00640 1 Processor -- start 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.836+0000 7fd35ae00640 1 -- start start 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.836+0000 7fd35ae00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3541051d0 0x7fd35419b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.836+0000 7fd35ae00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354106260 0x7fd35419c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.836+0000 7fd35ae00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106c00 0x7fd35419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.836+0000 7fd35ae00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd354110ce0 con 0x7fd354106c00 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.836+0000 7fd35ae00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd354110b60 con 0x7fd3541051d0 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.836+0000 7fd35ae00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd354110e60 con 0x7fd354106260 2024-07-26T07:39:25.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106c00 0x7fd35419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106c00 0x7fd35419cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48294/0 (socket says 172.21.15.97:48294) 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 -- 172.21.15.97:0/1916239702 learned_addr learned my addr 172.21.15.97:0/1916239702 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd359a00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3541051d0 0x7fd35419b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd359000640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354106260 0x7fd35419c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 -- 172.21.15.97:0/1916239702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354106260 msgr2=0x7fd35419c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354106260 0x7fd35419c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 -- 172.21.15.97:0/1916239702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3541051d0 msgr2=0x7fd35419b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3541051d0 0x7fd35419b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 -- 172.21.15.97:0/1916239702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35419d3e0 con 0x7fd354106c00 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.837+0000 7fd35a400640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106c00 0x7fd35419cbc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00cad0 tx=0x7fd34c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.838+0000 7fd352000640 1 -- 172.21.15.97:0/1916239702 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c00ae20 con 0x7fd354106c00 2024-07-26T07:39:25.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.838+0000 7fd352000640 1 -- 172.21.15.97:0/1916239702 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd34c005020 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.838+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35419d6d0 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.838+0000 7fd352000640 1 -- 172.21.15.97:0/1916239702 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c00f6b0 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.838+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3541a3dd0 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.839+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd35410c390 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.841+0000 7fd352000640 1 -- 172.21.15.97:0/1916239702 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fd34c0051d0 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.843+0000 7fd352000640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd33c0b3bc0 0x7fd33c0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.843+0000 7fd352000640 1 -- 172.21.15.97:0/1916239702 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd34c0e22d0 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.843+0000 7fd359a00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd33c0b3bc0 0x7fd33c0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.844+0000 7fd359a00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd33c0b3bc0 0x7fd33c0b60d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd34800bbe0 tx=0x7fd34800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:24.847+0000 7fd352000640 1 -- 172.21.15.97:0/1916239702 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd34c014030 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.074+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fd35402e0c0 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.081+0000 7fd352000640 1 -- 172.21.15.97:0/1916239702 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v29)=0 v29) ==== 119+0+0 (secure 0 0 0) 0x7fd34c0ab550 con 0x7fd354106c00 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd33c0b3bc0 msgr2=0x7fd33c0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:25.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd33c0b3bc0 0x7fd33c0b60d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd34800bbe0 tx=0x7fd34800b390 comp rx=0 tx=0).stop 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106c00 msgr2=0x7fd35419cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106c00 0x7fd35419cbc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00cad0 tx=0x7fd34c00cf90 comp rx=0 tx=0).stop 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 shutdown_connections 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd33c0b3bc0 0x7fd33c0b60d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd354106c00 0x7fd35419cbc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354106260 0x7fd35419c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 --2- 172.21.15.97:0/1916239702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3541051d0 0x7fd35419b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 >> 172.21.15.97:0/1916239702 conn(0x7fd354100b60 msgr2=0x7fd354109560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 shutdown_connections 2024-07-26T07:39:25.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:25.084+0000 7fd35ae00640 1 -- 172.21.15.97:0/1916239702 wait complete. 2024-07-26T07:39:25.499 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi097') 2024-07-26T07:39:25.500 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi097') 2024-07-26T07:39:25.695 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:39:26.243 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi097') 2024-07-26T07:39:26.243 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi097') 2024-07-26T07:39:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:26 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1916239702' entity='client.admin' 2024-07-26T07:39:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:26 smithi115 ceph-mon[35364]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:26 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1916239702' entity='client.admin' 2024-07-26T07:39:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:26 smithi181 ceph-mon[35869]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:26.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:26 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1916239702' entity='client.admin' 2024-07-26T07:39:26.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:26 smithi097 ceph-mon[32231]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:27.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.377+0000 7f4a8ba00640 1 -- 172.21.15.97:0/3952797377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a841053b0 msgr2=0x7f4a84105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:27.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.377+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/3952797377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a841053b0 0x7f4a84105820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009900 tx=0x7f4a7802a480 comp rx=0 tx=0).stop 2024-07-26T07:39:27.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.378+0000 7f4a8ba00640 1 -- 172.21.15.97:0/3952797377 shutdown_connections 2024-07-26T07:39:27.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.378+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/3952797377 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a84106de0 0x7f4a8410d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.378+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/3952797377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a84106440 0x7f4a84106850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.378+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/3952797377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a841053b0 0x7f4a84105820 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.378+0000 7f4a8ba00640 1 -- 172.21.15.97:0/3952797377 >> 172.21.15.97:0/3952797377 conn(0x7f4a84100d20 msgr2=0x7f4a841031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:27.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.378+0000 7f4a8ba00640 1 -- 172.21.15.97:0/3952797377 shutdown_connections 2024-07-26T07:39:27.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.378+0000 7f4a8ba00640 1 -- 172.21.15.97:0/3952797377 wait complete. 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.379+0000 7f4a8ba00640 1 Processor -- start 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.379+0000 7f4a8ba00640 1 -- start start 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.379+0000 7f4a8ba00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a841053b0 0x7f4a8419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.379+0000 7f4a8ba00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a84106440 0x7f4a8419c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.379+0000 7f4a8ba00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a84106de0 0x7f4a8419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.379+0000 7f4a8ba00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4a84110ec0 con 0x7f4a84106de0 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.379+0000 7f4a8ba00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4a84110d40 con 0x7f4a841053b0 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.379+0000 7f4a8ba00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4a84111040 con 0x7f4a84106440 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a84106de0 0x7f4a8419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a84106de0 0x7f4a8419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33178/0 (socket says 172.21.15.97:33178) 2024-07-26T07:39:27.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89a00640 1 -- 172.21.15.97:0/1282790627 learned_addr learned my addr 172.21.15.97:0/1282790627 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:27.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89000640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a841053b0 0x7f4a8419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:27.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a83e00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a84106440 0x7f4a8419c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89a00640 1 -- 172.21.15.97:0/1282790627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a84106440 msgr2=0x7f4a8419c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89a00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a84106440 0x7f4a8419c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89a00640 1 -- 172.21.15.97:0/1282790627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a841053b0 msgr2=0x7f4a8419bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89a00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a841053b0 0x7f4a8419bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89a00640 1 -- 172.21.15.97:0/1282790627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8419d560 con 0x7f4a84106de0 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a89000640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a841053b0 0x7f4a8419bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.380+0000 7f4a83e00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a84106440 0x7f4a8419c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.381+0000 7f4a89a00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a84106de0 0x7f4a8419cd40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00cad0 tx=0x7f4a7c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.381+0000 7f4a81600640 1 -- 172.21.15.97:0/1282790627 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c00ae20 con 0x7f4a84106de0 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.381+0000 7f4a81600640 1 -- 172.21.15.97:0/1282790627 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a7c005020 con 0x7f4a84106de0 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.381+0000 7f4a81600640 1 -- 172.21.15.97:0/1282790627 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c00f6b0 con 0x7f4a84106de0 2024-07-26T07:39:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.381+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8419d850 con 0x7f4a84106de0 2024-07-26T07:39:27.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.381+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a841a3f50 con 0x7f4a84106de0 2024-07-26T07:39:27.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.382+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8410c590 con 0x7f4a84106de0 2024-07-26T07:39:27.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.385+0000 7f4a81600640 1 -- 172.21.15.97:0/1282790627 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f4a7c0051d0 con 0x7f4a84106de0 2024-07-26T07:39:27.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.386+0000 7f4a81600640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4a6c0b3c40 0x7f4a6c0b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:27.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.387+0000 7f4a81600640 1 -- 172.21.15.97:0/1282790627 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4a7c0e23c0 con 0x7f4a84106de0 2024-07-26T07:39:27.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.387+0000 7f4a89000640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4a6c0b3c40 0x7f4a6c0b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:27.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.388+0000 7f4a89000640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4a6c0b3c40 0x7f4a6c0b6150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4a78000c00 tx=0x7f4a78031420 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:27.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.390+0000 7f4a81600640 1 -- 172.21.15.97:0/1282790627 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4a7c014030 con 0x7f4a84106de0 2024-07-26T07:39:27.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.591+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f4a8402e0c0 con 0x7f4a84106de0 2024-07-26T07:39:27.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.598+0000 7f4a81600640 1 -- 172.21.15.97:0/1282790627 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v30)=0 v30) ==== 121+0+0 (secure 0 0 0) 0x7f4a7c0ab640 con 0x7f4a84106de0 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.601+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4a6c0b3c40 msgr2=0x7f4a6c0b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.601+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4a6c0b3c40 0x7f4a6c0b6150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4a78000c00 tx=0x7f4a78031420 comp rx=0 tx=0).stop 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.601+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a84106de0 msgr2=0x7f4a8419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.601+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a84106de0 0x7f4a8419cd40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00cad0 tx=0x7f4a7c00cf90 comp rx=0 tx=0).stop 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.601+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 shutdown_connections 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.601+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4a6c0b3c40 0x7f4a6c0b6150 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.601+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a84106de0 0x7f4a8419cd40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.601+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a84106440 0x7f4a8419c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.602+0000 7f4a8ba00640 1 --2- 172.21.15.97:0/1282790627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4a841053b0 0x7f4a8419bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.602+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 >> 172.21.15.97:0/1282790627 conn(0x7f4a84100d20 msgr2=0x7f4a84109760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:27.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.602+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 shutdown_connections 2024-07-26T07:39:27.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:27.602+0000 7f4a8ba00640 1 -- 172.21.15.97:0/1282790627 wait complete. 2024-07-26T07:39:28.953 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:28 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1282790627' entity='client.admin' 2024-07-26T07:39:28.953 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:28 smithi097 ceph-mon[32231]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:28.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:28 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1282790627' entity='client.admin' 2024-07-26T07:39:28.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:28 smithi115 ceph-mon[35364]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:28 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1282790627' entity='client.admin' 2024-07-26T07:39:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:28 smithi181 ceph-mon[35869]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:29.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 -- 172.21.15.97:0/1663196008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e010af60 msgr2=0x7f12e010d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:29.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 --2- 172.21.15.97:0/1663196008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e010af60 0x7f12e010d400 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f12d8009ca0 tx=0x7f12d802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 -- 172.21.15.97:0/1663196008 shutdown_connections 2024-07-26T07:39:29.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 --2- 172.21.15.97:0/1663196008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e010af60 0x7f12e010d400 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 --2- 172.21.15.97:0/1663196008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12e0075720 0x7f12e0075b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 --2- 172.21.15.97:0/1663196008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12e0076b10 0x7f12e00751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 -- 172.21.15.97:0/1663196008 >> 172.21.15.97:0/1663196008 conn(0x7f12e00fe930 msgr2=0x7f12e0100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:29.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 -- 172.21.15.97:0/1663196008 shutdown_connections 2024-07-26T07:39:29.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.287+0000 7f12e6200640 1 -- 172.21.15.97:0/1663196008 wait complete. 2024-07-26T07:39:29.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.288+0000 7f12e6200640 1 Processor -- start 2024-07-26T07:39:29.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12e6200640 1 -- start start 2024-07-26T07:39:29.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12e6200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12e0075720 0x7f12e019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:29.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12e6200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e0076b10 0x7f12e019bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12e6200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12e010af60 0x7f12e019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12e6200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0110d10 con 0x7f12e0076b10 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12e6200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0110b90 con 0x7f12e010af60 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12e6200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0110e90 con 0x7f12e0075720 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e0076b10 0x7f12e019bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e0076b10 0x7f12e019bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33188/0 (socket says 172.21.15.97:33188) 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12dfe00640 1 -- 172.21.15.97:0/4267312909 learned_addr learned my addr 172.21.15.97:0/4267312909 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.289+0000 7f12e4e00640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12e0075720 0x7f12e019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12e5800640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12e010af60 0x7f12e019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12dfe00640 1 -- 172.21.15.97:0/4267312909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12e0075720 msgr2=0x7f12e019b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12dfe00640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12e0075720 0x7f12e019b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12dfe00640 1 -- 172.21.15.97:0/4267312909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12e010af60 msgr2=0x7f12e019cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12dfe00640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12e010af60 0x7f12e019cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12dfe00640 1 -- 172.21.15.97:0/4267312909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e019d2c0 con 0x7f12e0076b10 2024-07-26T07:39:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12dfe00640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e0076b10 0x7f12e019bf90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00c920 tx=0x7f12cc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:29.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12dd600640 1 -- 172.21.15.97:0/4267312909 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc007c70 con 0x7f12e0076b10 2024-07-26T07:39:29.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.290+0000 7f12dd600640 1 -- 172.21.15.97:0/4267312909 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f12cc007e10 con 0x7f12e0076b10 2024-07-26T07:39:29.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.291+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e019d5b0 con 0x7f12e0076b10 2024-07-26T07:39:29.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.291+0000 7f12dd600640 1 -- 172.21.15.97:0/4267312909 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc018710 con 0x7f12e0076b10 2024-07-26T07:39:29.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.291+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e01a3d40 con 0x7f12e0076b10 2024-07-26T07:39:29.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.292+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e0109d70 con 0x7f12e0076b10 2024-07-26T07:39:29.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.295+0000 7f12dd600640 1 -- 172.21.15.97:0/4267312909 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f12cc00f450 con 0x7f12e0076b10 2024-07-26T07:39:29.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.296+0000 7f12dd600640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f12c80b3b40 0x7f12c80b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:29.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.297+0000 7f12dd600640 1 -- 172.21.15.97:0/4267312909 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f12cc0e2490 con 0x7f12e0076b10 2024-07-26T07:39:29.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.297+0000 7f12e4e00640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f12c80b3b40 0x7f12c80b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:29.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.298+0000 7f12e4e00640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f12c80b3b40 0x7f12c80b6050 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12d4005fd0 tx=0x7f12d4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:29.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.298+0000 7f12dd600640 1 -- 172.21.15.97:0/4267312909 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f12cc014030 con 0x7f12e0076b10 2024-07-26T07:39:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.495+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f12e002e0c0 con 0x7f12e0076b10 2024-07-26T07:39:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.504+0000 7f12dd600640 1 -- 172.21.15.97:0/4267312909 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v31)=0 v31) ==== 120+0+0 (secure 0 0 0) 0x7f12cc0ab7a0 con 0x7f12e0076b10 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.507+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f12c80b3b40 msgr2=0x7f12c80b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.507+0000 7f12e6200640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f12c80b3b40 0x7f12c80b6050 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12d4005fd0 tx=0x7f12d4005850 comp rx=0 tx=0).stop 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.507+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e0076b10 msgr2=0x7f12e019bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.507+0000 7f12e6200640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e0076b10 0x7f12e019bf90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00c920 tx=0x7f12cc00cde0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.507+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 shutdown_connections 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.508+0000 7f12e6200640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f12c80b3b40 0x7f12c80b6050 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.508+0000 7f12e6200640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12e010af60 0x7f12e019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.508+0000 7f12e6200640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12e0076b10 0x7f12e019bf90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.508+0000 7f12e6200640 1 --2- 172.21.15.97:0/4267312909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12e0075720 0x7f12e019b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.508+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 >> 172.21.15.97:0/4267312909 conn(0x7f12e00fe930 msgr2=0x7f12e0106f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:29.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.508+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 shutdown_connections 2024-07-26T07:39:29.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:29.508+0000 7f12e6200640 1 -- 172.21.15.97:0/4267312909 wait complete. 2024-07-26T07:39:30.278 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi097') 2024-07-26T07:39:30.278 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi097') 2024-07-26T07:39:30.278 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi097') 2024-07-26T07:39:30.278 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9621a072-4b21-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi097') 2024-07-26T07:39:30.279 INFO:tasks.cephadm:Deploying OSDs... 2024-07-26T07:39:30.279 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:39:30.279 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T07:39:30.337 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T07:39:30.337 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-07-26 07:37:31.499315256 +0000 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-07-26 07:32:10.724533009 +0000 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-07-26 07:32:10.724533009 +0000 2024-07-26T07:39:30.388 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-07-26 07:32:10.724533009 +0000 2024-07-26T07:39:30.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T07:39:30.471 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-07-26T07:39:30.471 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-07-26T07:39:30.471 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000381479 s, 1.3 MB/s 2024-07-26T07:39:30.475 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T07:39:30.529 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-07-26T07:39:30.571 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T07:39:30.571 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:30.571 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-07-26T07:39:30.571 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:30.571 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:30.571 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-07-26 07:37:31.601313434 +0000 2024-07-26T07:39:30.571 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-07-26 07:32:11.312522501 +0000 2024-07-26T07:39:30.571 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-07-26 07:32:11.312522501 +0000 2024-07-26T07:39:30.572 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-07-26 07:32:11.312522501 +0000 2024-07-26T07:39:30.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T07:39:30.688 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-07-26T07:39:30.689 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-07-26T07:39:30.689 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000337786 s, 1.5 MB/s 2024-07-26T07:39:30.692 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T07:39:30.747 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-07-26T07:39:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:30 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4267312909' entity='client.admin' 2024-07-26T07:39:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:30 smithi097 ceph-mon[32231]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:30.776 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:30 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4267312909' entity='client.admin' 2024-07-26T07:39:30.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:30 smithi115 ceph-mon[35364]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:30.794 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T07:39:30.794 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:30.795 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-07-26T07:39:30.795 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:30.795 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:30.795 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-07-26 07:37:31.687311899 +0000 2024-07-26T07:39:30.795 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-07-26 07:32:11.897512048 +0000 2024-07-26T07:39:30.795 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-07-26 07:32:11.897512048 +0000 2024-07-26T07:39:30.795 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-07-26 07:32:11.897512048 +0000 2024-07-26T07:39:30.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T07:39:30.874 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-07-26T07:39:30.874 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-07-26T07:39:30.874 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000356192 s, 1.4 MB/s 2024-07-26T07:39:30.878 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T07:39:30.932 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-07-26T07:39:30.982 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T07:39:30.982 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:30.982 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-07-26T07:39:30.982 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:30.982 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:30.983 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-07-26 07:37:31.755310685 +0000 2024-07-26T07:39:30.983 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-07-26 07:32:12.497501326 +0000 2024-07-26T07:39:30.983 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-07-26 07:32:12.497501326 +0000 2024-07-26T07:39:30.983 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-07-26 07:32:12.497501326 +0000 2024-07-26T07:39:30.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T07:39:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:30 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4267312909' entity='client.admin' 2024-07-26T07:39:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:30 smithi181 ceph-mon[35869]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:31.060 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-07-26T07:39:31.060 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-07-26T07:39:31.060 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000336933 s, 1.5 MB/s 2024-07-26T07:39:31.063 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T07:39:31.110 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-07-26T07:39:31.110 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T07:39:31.168 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T07:39:31.169 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2024-07-26T07:39:31.220 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T07:39:31.220 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:31.221 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-07-26T07:39:31.221 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:31.221 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:31.221 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-07-26 07:38:46.328211758 +0000 2024-07-26T07:39:31.221 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-07-26 07:32:13.294459015 +0000 2024-07-26T07:39:31.221 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-07-26 07:32:13.294459015 +0000 2024-07-26T07:39:31.221 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-07-26 07:32:13.294459015 +0000 2024-07-26T07:39:31.221 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T07:39:31.295 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-07-26T07:39:31.295 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-07-26T07:39:31.295 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000347936 s, 1.5 MB/s 2024-07-26T07:39:31.298 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T07:39:31.350 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2024-07-26T07:39:31.399 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T07:39:31.400 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:31.400 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-07-26T07:39:31.400 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:31.400 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:31.400 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-07-26 07:38:46.328211758 +0000 2024-07-26T07:39:31.400 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-07-26 07:32:13.840448720 +0000 2024-07-26T07:39:31.400 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-07-26 07:32:13.840448720 +0000 2024-07-26T07:39:31.400 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-07-26 07:32:13.840448720 +0000 2024-07-26T07:39:31.401 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T07:39:31.477 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-07-26T07:39:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-07-26T07:39:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000338329 s, 1.5 MB/s 2024-07-26T07:39:31.481 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T07:39:31.538 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2024-07-26T07:39:31.586 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T07:39:31.586 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:31.586 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-07-26T07:39:31.586 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:31.586 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:31.586 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-07-26 07:38:46.329211739 +0000 2024-07-26T07:39:31.586 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-07-26 07:32:14.442437370 +0000 2024-07-26T07:39:31.587 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-07-26 07:32:14.442437370 +0000 2024-07-26T07:39:31.587 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-07-26 07:32:14.442437370 +0000 2024-07-26T07:39:31.587 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T07:39:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-07-26T07:39:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-07-26T07:39:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000354773 s, 1.4 MB/s 2024-07-26T07:39:31.665 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T07:39:31.721 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-07-26 07:38:46.330211721 +0000 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-07-26 07:32:15.031426264 +0000 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-07-26 07:32:15.031426264 +0000 2024-07-26T07:39:31.769 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-07-26 07:32:15.031426264 +0000 2024-07-26T07:39:31.770 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T07:39:31.842 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-07-26T07:39:31.842 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-07-26T07:39:31.842 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000380182 s, 1.3 MB/s 2024-07-26T07:39:31.845 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T07:39:31.898 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-07-26T07:39:31.899 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T07:39:31.958 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T07:39:31.958 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-07-26T07:39:32.011 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T07:39:32.011 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:32.011 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-07-26T07:39:32.011 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:32.011 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:32.011 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-07-26 07:38:42.721969477 +0000 2024-07-26T07:39:32.012 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-07-26 07:32:12.665443967 +0000 2024-07-26T07:39:32.012 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-07-26 07:32:12.665443967 +0000 2024-07-26T07:39:32.012 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-07-26 07:32:12.665443967 +0000 2024-07-26T07:39:32.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T07:39:32.088 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-07-26T07:39:32.088 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-07-26T07:39:32.088 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.00033118 s, 1.5 MB/s 2024-07-26T07:39:32.091 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T07:39:32.142 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-07-26T07:39:32.191 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T07:39:32.191 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:32.191 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-07-26T07:39:32.191 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:32.191 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:32.191 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-07-26 07:38:42.721969477 +0000 2024-07-26T07:39:32.191 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-07-26 07:32:13.205433735 +0000 2024-07-26T07:39:32.191 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-07-26 07:32:13.205433735 +0000 2024-07-26T07:39:32.192 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-07-26 07:32:13.205433735 +0000 2024-07-26T07:39:32.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T07:39:32.268 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-07-26T07:39:32.268 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-07-26T07:39:32.268 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000339491 s, 1.5 MB/s 2024-07-26T07:39:32.272 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T07:39:32.327 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-07-26T07:39:32.375 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T07:39:32.375 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:32.375 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-07-26T07:39:32.375 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:32.375 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:32.375 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-07-26 07:38:42.723969439 +0000 2024-07-26T07:39:32.376 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-07-26 07:32:13.800422461 +0000 2024-07-26T07:39:32.376 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-07-26 07:32:13.800422461 +0000 2024-07-26T07:39:32.376 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-07-26 07:32:13.800422461 +0000 2024-07-26T07:39:32.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T07:39:32.441 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:32 smithi181 ceph-mon[35869]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:32.449 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-07-26T07:39:32.450 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-07-26T07:39:32.450 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000307513 s, 1.7 MB/s 2024-07-26T07:39:32.453 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T07:39:32.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:32 smithi115 ceph-mon[35364]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:32.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:32 smithi097 ceph-mon[32231]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:32.509 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-07-26T07:39:32.556 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T07:39:32.556 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T07:39:32.557 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-07-26T07:39:32.557 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:39:32.557 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T07:39:32.557 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-07-26 07:38:42.723969439 +0000 2024-07-26T07:39:32.557 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-07-26 07:32:14.352412002 +0000 2024-07-26T07:39:32.557 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-07-26 07:32:14.352412002 +0000 2024-07-26T07:39:32.557 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-07-26 07:32:14.352412002 +0000 2024-07-26T07:39:32.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T07:39:32.682 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-07-26T07:39:32.682 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-07-26T07:39:32.682 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000358569 s, 1.4 MB/s 2024-07-26T07:39:32.685 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T07:39:32.740 INFO:tasks.cephadm:Deploying osd.0 on smithi097 with /dev/vg_nvme/lv_4... 2024-07-26T07:39:32.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T07:39:33.182 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:39:34.339 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:34 smithi097 ceph-mon[32231]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:34.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:34 smithi115 ceph-mon[35364]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:34 smithi181 ceph-mon[35869]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:35.446 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T07:39:35.647 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T07:39:35.690 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-07-26T07:39:35.690 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-07-26T07:39:35.690 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0312931 s, 335 MB/s 2024-07-26T07:39:35.690 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 2024-07-26T07:39:35.691 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-07-26T07:39:36.428 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:36 smithi097 ceph-mon[32231]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:36.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:36 smithi115 ceph-mon[35364]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:36.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_4 2024-07-26T07:39:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:36 smithi181 ceph-mon[35869]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:36.938 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:39:38.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:38 smithi115 ceph-mon[35364]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:38.476 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:38 smithi097 ceph-mon[32231]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:38 smithi181 ceph-mon[35869]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:38.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.961+0000 7f140f200640 1 -- 172.21.15.97:0/2025941921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 msgr2=0x7f1410109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:38.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.961+0000 7f140f200640 1 --2- 172.21.15.97:0/2025941921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 0x7f1410109a40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1400009900 tx=0x7f140002a480 comp rx=0 tx=0).stop 2024-07-26T07:39:38.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.961+0000 7f140f200640 1 -- 172.21.15.97:0/2025941921 shutdown_connections 2024-07-26T07:39:38.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.961+0000 7f140f200640 1 --2- 172.21.15.97:0/2025941921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f141010b000 0x7f14101119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:38.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.961+0000 7f140f200640 1 --2- 172.21.15.97:0/2025941921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141010a660 0x7f141010aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:38.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.961+0000 7f140f200640 1 --2- 172.21.15.97:0/2025941921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 0x7f1410109a40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:38.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.961+0000 7f140f200640 1 -- 172.21.15.97:0/2025941921 >> 172.21.15.97:0/2025941921 conn(0x7f1410078970 msgr2=0x7f141007ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:39:38.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.961+0000 7f140f200640 1 -- 172.21.15.97:0/2025941921 shutdown_connections 2024-07-26T07:39:38.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.962+0000 7f140f200640 1 -- 172.21.15.97:0/2025941921 wait complete. 2024-07-26T07:39:38.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140f200640 1 Processor -- start 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140f200640 1 -- start start 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 0x7f141019fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141010a660 0x7f14101a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140f200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f141010b000 0x7f14101a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1410115110 con 0x7f14101095d0 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140f200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1410114f90 con 0x7f141010b000 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1410115290 con 0x7f141010a660 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 0x7f141019fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.963+0000 7f140de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 0x7f141019fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37994/0 (socket says 172.21.15.97:37994) 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f140de00640 1 -- 172.21.15.97:0/3648174403 learned_addr learned my addr 172.21.15.97:0/3648174403 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f1407e00640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141010a660 0x7f14101a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:38.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f140e800640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f141010b000 0x7f14101a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f140de00640 1 -- 172.21.15.97:0/3648174403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141010a660 msgr2=0x7f14101a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f140de00640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141010a660 0x7f14101a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f140de00640 1 -- 172.21.15.97:0/3648174403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f141010b000 msgr2=0x7f14101a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f140de00640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f141010b000 0x7f14101a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f140de00640 1 -- 172.21.15.97:0/3648174403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14101a1800 con 0x7f14101095d0 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.964+0000 7f140de00640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 0x7f141019fdd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1400005760 tx=0x7f1400003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.965+0000 7f1406a00640 1 -- 172.21.15.97:0/3648174403 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140003f070 con 0x7f14101095d0 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.965+0000 7f1406a00640 1 -- 172.21.15.97:0/3648174403 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1400031da0 con 0x7f14101095d0 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.965+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14101a1a90 con 0x7f14101095d0 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.965+0000 7f1406a00640 1 -- 172.21.15.97:0/3648174403 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400035030 con 0x7f14101095d0 2024-07-26T07:39:38.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.965+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14101a81f0 con 0x7f14101095d0 2024-07-26T07:39:38.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.966+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14101107b0 con 0x7f14101095d0 2024-07-26T07:39:38.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.970+0000 7f1406a00640 1 -- 172.21.15.97:0/3648174403 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f1400042070 con 0x7f14101095d0 2024-07-26T07:39:38.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.972+0000 7f1406a00640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13f00b3c40 0x7f13f00b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:39:38.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.972+0000 7f1406a00640 1 -- 172.21.15.97:0/3648174403 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1400103080 con 0x7f14101095d0 2024-07-26T07:39:38.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.972+0000 7f1407e00640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13f00b3c40 0x7f13f00b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:39:38.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.973+0000 7f1407e00640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13f00b3c40 0x7f13f00b6150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f13f8005fd0 tx=0x7f13f8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:39:38.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:38.974+0000 7f1406a00640 1 -- 172.21.15.97:0/3648174403 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f14000cc300 con 0x7f14101095d0 2024-07-26T07:39:39.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:39:39.181+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1410063230 con 0x7f13f00b3c40 2024-07-26T07:39:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:39:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:39:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:39:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:39:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:39:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:39:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:40 smithi181 ceph-mon[35869]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:39:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:40 smithi181 ceph-mon[35869]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:40 smithi097 ceph-mon[32231]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:39:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:40 smithi097 ceph-mon[32231]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:40 smithi115 ceph-mon[35364]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:39:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:40 smithi115 ceph-mon[35364]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:42.319 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:42 smithi097 ceph-mon[32231]: pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:42.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:42 smithi115 ceph-mon[35364]: pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:42 smithi181 ceph-mon[35869]: pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:43.445 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:43 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2411246376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d860fc33-42a6-4123-af2c-5adff0e9b852"}]: dispatch 2024-07-26T07:39:43.445 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:43 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2411246376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d860fc33-42a6-4123-af2c-5adff0e9b852"}]': finished 2024-07-26T07:39:43.445 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:43 smithi097 ceph-mon[32231]: osdmap e5: 1 total, 0 up, 1 in 2024-07-26T07:39:43.445 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:39:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:43 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2411246376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d860fc33-42a6-4123-af2c-5adff0e9b852"}]: dispatch 2024-07-26T07:39:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:43 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2411246376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d860fc33-42a6-4123-af2c-5adff0e9b852"}]': finished 2024-07-26T07:39:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:43 smithi115 ceph-mon[35364]: osdmap e5: 1 total, 0 up, 1 in 2024-07-26T07:39:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:39:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:43 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2411246376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d860fc33-42a6-4123-af2c-5adff0e9b852"}]: dispatch 2024-07-26T07:39:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:43 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2411246376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d860fc33-42a6-4123-af2c-5adff0e9b852"}]': finished 2024-07-26T07:39:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:43 smithi181 ceph-mon[35869]: osdmap e5: 1 total, 0 up, 1 in 2024-07-26T07:39:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:39:44.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:44 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2944056423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:39:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:44 smithi115 ceph-mon[35364]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:44 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2944056423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:39:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:44 smithi181 ceph-mon[35869]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:44 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2944056423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:39:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:44 smithi097 ceph-mon[32231]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:46.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:46 smithi115 ceph-mon[35364]: pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:46 smithi181 ceph-mon[35869]: pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:46.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:46 smithi097 ceph-mon[32231]: pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:48.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:48 smithi115 ceph-mon[35364]: pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:48 smithi181 ceph-mon[35869]: pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:48.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:48 smithi097 ceph-mon[32231]: pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:50 smithi097 ceph-mon[32231]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:50.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:50 smithi115 ceph-mon[35364]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:50 smithi181 ceph-mon[35869]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:52 smithi097 ceph-mon[32231]: pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T07:39:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:52 smithi097 ceph-mon[32231]: Deploying daemon osd.0 on smithi097 2024-07-26T07:39:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:52 smithi115 ceph-mon[35364]: pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T07:39:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:52 smithi115 ceph-mon[35364]: Deploying daemon osd.0 on smithi097 2024-07-26T07:39:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:52 smithi181 ceph-mon[35869]: pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T07:39:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:52 smithi181 ceph-mon[35869]: Deploying daemon osd.0 on smithi097 2024-07-26T07:39:54.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:54 smithi097 ceph-mon[32231]: pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:54 smithi115 ceph-mon[35364]: pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:54 smithi181 ceph-mon[35869]: pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:56.342 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:56 smithi097 ceph-mon[32231]: pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:56.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:56 smithi115 ceph-mon[35364]: pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:56 smithi181 ceph-mon[35869]: pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:58.362 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:39:58 smithi097 ceph-mon[32231]: pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:58.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:39:58 smithi115 ceph-mon[35364]: pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:39:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:39:58 smithi181 ceph-mon[35869]: pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:00.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:00 smithi097 ceph-mon[32231]: pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:00.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:00 smithi097 ceph-mon[32231]: overall HEALTH_OK 2024-07-26T07:40:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:00 smithi115 ceph-mon[35364]: pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:00 smithi115 ceph-mon[35364]: overall HEALTH_OK 2024-07-26T07:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:00 smithi181 ceph-mon[35869]: pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:00 smithi181 ceph-mon[35869]: overall HEALTH_OK 2024-07-26T07:40:01.298 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:01.299 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:01.299 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:02 smithi115 ceph-mon[35364]: pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:02 smithi181 ceph-mon[35869]: pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:02.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:02 smithi097 ceph-mon[32231]: pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:05.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:04 smithi097 ceph-mon[32231]: pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:05.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:04 smithi097 ceph-mon[32231]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T07:40:05.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:04 smithi115 ceph-mon[35364]: pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:04 smithi115 ceph-mon[35364]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T07:40:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:04 smithi181 ceph-mon[35869]: pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:04 smithi181 ceph-mon[35869]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T07:40:05.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.316+0000 7f1406a00640 1 -- 172.21.15.97:0/3648174403 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1410063230 con 0x7f13f00b3c40 2024-07-26T07:40:05.320 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 0 on host 'smithi097' 2024-07-26T07:40:05.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13f00b3c40 msgr2=0x7f13f00b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:05.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13f00b3c40 0x7f13f00b6150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f13f8005fd0 tx=0x7f13f8005850 comp rx=0 tx=0).stop 2024-07-26T07:40:05.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 msgr2=0x7f141019fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:05.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 0x7f141019fdd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1400005760 tx=0x7f1400003d50 comp rx=0 tx=0).stop 2024-07-26T07:40:05.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 shutdown_connections 2024-07-26T07:40:05.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13f00b3c40 0x7f13f00b6150 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:05.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f141010b000 0x7f14101a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:05.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141010a660 0x7f14101a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:05.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 --2- 172.21.15.97:0/3648174403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14101095d0 0x7f141019fdd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:05.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 >> 172.21.15.97:0/3648174403 conn(0x7f1410078970 msgr2=0x7f141010d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:40:05.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 shutdown_connections 2024-07-26T07:40:05.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:40:05.321+0000 7f140f200640 1 -- 172.21.15.97:0/3648174403 wait complete. 2024-07-26T07:40:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T07:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:06.099 DEBUG:teuthology.orchestra.run.smithi097:osd.0> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.0.service 2024-07-26T07:40:06.104 INFO:tasks.cephadm:Deploying osd.1 on smithi115 with /dev/vg_nvme/lv_4... 2024-07-26T07:40:06.105 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T07:40:06.181 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T07:40:06.182 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T07:40:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T07:40:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:40:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:40:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T07:40:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:06.548 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T07:40:06.954 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:06.954 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:06 smithi097 ceph-mon[32231]: pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:06.954 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:40:06 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-0[50136]: 2024-07-26T07:40:06.918+0000 7ff0d5e00640 -1 osd.0 0 waiting for initial osdmap 2024-07-26T07:40:06.955 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:40:06 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-0[50136]: 2024-07-26T07:40:06.952+0000 7ff0cbe00640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T07:40:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:06 smithi115 ceph-mon[35364]: pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:06 smithi181 ceph-mon[35869]: pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:08.117 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:07 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:40:08.117 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:07 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:40:08.117 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:08.117 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:07 smithi097 ceph-mon[32231]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' 2024-07-26T07:40:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:07 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:40:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:07 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:40:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:07 smithi115 ceph-mon[35364]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' 2024-07-26T07:40:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:07 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:40:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:07 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:40:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:07 smithi181 ceph-mon[35869]: from='osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072]' entity='osd.0' 2024-07-26T07:40:09.128 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T07:40:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:08 smithi115 ceph-mon[35364]: osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] boot 2024-07-26T07:40:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:08 smithi115 ceph-mon[35364]: osdmap e8: 1 total, 1 up, 1 in 2024-07-26T07:40:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:08 smithi115 ceph-mon[35364]: pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:08 smithi181 ceph-mon[35869]: osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] boot 2024-07-26T07:40:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:08 smithi181 ceph-mon[35869]: osdmap e8: 1 total, 1 up, 1 in 2024-07-26T07:40:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:08 smithi181 ceph-mon[35869]: pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:08 smithi097 ceph-mon[32231]: osd.0 [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] boot 2024-07-26T07:40:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:08 smithi097 ceph-mon[32231]: osdmap e8: 1 total, 1 up, 1 in 2024-07-26T07:40:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T07:40:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:08 smithi097 ceph-mon[32231]: pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T07:40:09.334 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T07:40:09.379 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-07-26T07:40:09.379 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-07-26T07:40:09.380 INFO:teuthology.orchestra.run.smithi115.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0351736 s, 298 MB/s 2024-07-26T07:40:09.380 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 2024-07-26T07:40:09.380 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-07-26T07:40:10.086 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:09 smithi115 ceph-mon[35364]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T07:40:10.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi115:vg_nvme/lv_4 2024-07-26T07:40:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:09 smithi181 ceph-mon[35869]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T07:40:10.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:09 smithi097 ceph-mon[32231]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T07:40:10.646 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T07:40:11.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:10 smithi115 ceph-mon[35364]: pgmap v85: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:11.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:40:11.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:11.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:11.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:10 smithi181 ceph-mon[35869]: pgmap v85: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:40:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:10 smithi097 ceph-mon[32231]: pgmap v85: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:40:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:11 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi097 2024-07-26T07:40:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:11 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 17089M 2024-07-26T07:40:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:11 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi097 2024-07-26T07:40:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:11 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 17089M 2024-07-26T07:40:12.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:11 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi097 2024-07-26T07:40:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:11 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 17089M 2024-07-26T07:40:12.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 -- 172.21.15.115:0/3573167650 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae881051d0 msgr2=0x7fae88105640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:12.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 --2- 172.21.15.115:0/3573167650 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae881051d0 0x7fae88105640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae70009900 tx=0x7fae7002a480 comp rx=0 tx=0).stop 2024-07-26T07:40:12.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 -- 172.21.15.115:0/3573167650 shutdown_connections 2024-07-26T07:40:12.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 --2- 172.21.15.115:0/3573167650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae88106c00 0x7fae8810d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:12.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 --2- 172.21.15.115:0/3573167650 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae88106260 0x7fae88106670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:12.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 --2- 172.21.15.115:0/3573167650 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae881051d0 0x7fae88105640 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:12.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 -- 172.21.15.115:0/3573167650 >> 172.21.15.115:0/3573167650 conn(0x7fae88100b60 msgr2=0x7fae88102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:40:12.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 -- 172.21.15.115:0/3573167650 shutdown_connections 2024-07-26T07:40:12.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.784+0000 7fae8de00640 1 -- 172.21.15.115:0/3573167650 wait complete. 2024-07-26T07:40:12.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.785+0000 7fae8de00640 1 Processor -- start 2024-07-26T07:40:12.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.785+0000 7fae8de00640 1 -- start start 2024-07-26T07:40:12.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8de00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae881051d0 0x7fae8819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae88106260 0x7fae8819bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8de00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae88106c00 0x7fae8819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8de00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fae88110d30 con 0x7fae88106260 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8de00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fae88110bb0 con 0x7fae88106c00 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8de00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae88110eb0 con 0x7fae881051d0 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8d400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae88106c00 0x7fae8819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8d400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae88106c00 0x7fae8819cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44772/0 (socket says 172.21.15.115:44772) 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8d400640 1 -- 172.21.15.115:0/2985597439 learned_addr learned my addr 172.21.15.115:0/2985597439 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae7fe00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae88106260 0x7fae8819bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:12.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.786+0000 7fae8ca00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae881051d0 0x7fae8819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae8d400640 1 -- 172.21.15.115:0/2985597439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae881051d0 msgr2=0x7fae8819b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae8d400640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae881051d0 0x7fae8819b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae8d400640 1 -- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae88106260 msgr2=0x7fae8819bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae8d400640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae88106260 0x7fae8819bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae8d400640 1 -- 172.21.15.115:0/2985597439 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8819d3a0 con 0x7fae88106c00 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae7fe00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae88106260 0x7fae8819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae8ca00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae881051d0 0x7fae8819b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae8d400640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae88106c00 0x7fae8819cb80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fae8000eaa0 tx=0x7fae8000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae86000640 1 -- 172.21.15.115:0/2985597439 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8000cdb0 con 0x7fae88106c00 2024-07-26T07:40:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae86000640 1 -- 172.21.15.115:0/2985597439 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fae80005020 con 0x7fae88106c00 2024-07-26T07:40:12.791 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae86000640 1 -- 172.21.15.115:0/2985597439 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae800116d0 con 0x7fae88106c00 2024-07-26T07:40:12.791 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.787+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae8819d690 con 0x7fae88106c00 2024-07-26T07:40:12.791 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.788+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae881a3d90 con 0x7fae88106c00 2024-07-26T07:40:12.791 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.789+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae8810c3b0 con 0x7fae88106c00 2024-07-26T07:40:12.792 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.791+0000 7fae86000640 1 -- 172.21.15.115:0/2985597439 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fae800051d0 con 0x7fae88106c00 2024-07-26T07:40:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.793+0000 7fae86000640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae680b3bc0 0x7fae680b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.793+0000 7fae86000640 1 -- 172.21.15.115:0/2985597439 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fae800e26e0 con 0x7fae88106c00 2024-07-26T07:40:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.794+0000 7fae8ca00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae680b3bc0 0x7fae680b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:12.795 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.795+0000 7fae8ca00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae680b3bc0 0x7fae680b60d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fae70006010 tx=0x7fae70000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:40:12.796 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:12.796+0000 7fae86000640 1 -- 172.21.15.115:0/2985597439 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fae80016030 con 0x7fae88106c00 2024-07-26T07:40:13.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:13.003+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fae88063230 con 0x7fae680b3bc0 2024-07-26T07:40:13.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:12 smithi115 ceph-mon[35364]: pgmap v86: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:12 smithi181 ceph-mon[35869]: pgmap v86: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:13.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:12 smithi097 ceph-mon[32231]: pgmap v86: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:14.073 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:13 smithi115 ceph-mon[35364]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:40:14.073 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:40:14.073 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:40:14.073 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:13 smithi181 ceph-mon[35869]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:40:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:40:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:40:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:14.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:13 smithi097 ceph-mon[32231]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:40:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:40:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:40:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:15.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:14 smithi115 ceph-mon[35364]: pgmap v87: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:14 smithi181 ceph-mon[35869]: pgmap v87: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:15.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:14 smithi097 ceph-mon[32231]: pgmap v87: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:16 smithi097 ceph-mon[32231]: pgmap v88: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:16 smithi097 ceph-mon[32231]: from='client.? 172.21.15.115:0/2790352027' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]: dispatch 2024-07-26T07:40:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:16 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]: dispatch 2024-07-26T07:40:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:16 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]': finished 2024-07-26T07:40:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:16 smithi097 ceph-mon[32231]: osdmap e10: 2 total, 1 up, 2 in 2024-07-26T07:40:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:17.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:16 smithi115 ceph-mon[35364]: pgmap v88: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:17.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:16 smithi115 ceph-mon[35364]: from='client.? 172.21.15.115:0/2790352027' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]: dispatch 2024-07-26T07:40:17.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:16 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]: dispatch 2024-07-26T07:40:17.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:16 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]': finished 2024-07-26T07:40:17.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:16 smithi115 ceph-mon[35364]: osdmap e10: 2 total, 1 up, 2 in 2024-07-26T07:40:17.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:16 smithi181 ceph-mon[35869]: pgmap v88: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:16 smithi181 ceph-mon[35869]: from='client.? 172.21.15.115:0/2790352027' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]: dispatch 2024-07-26T07:40:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:16 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]: dispatch 2024-07-26T07:40:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:16 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2130819-8356-471c-b82d-89ce0ac47dc6"}]': finished 2024-07-26T07:40:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:16 smithi181 ceph-mon[35869]: osdmap e10: 2 total, 1 up, 2 in 2024-07-26T07:40:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:18.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:18 smithi115 ceph-mon[35364]: from='client.? 172.21.15.115:0/673333634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:40:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:18 smithi181 ceph-mon[35869]: from='client.? 172.21.15.115:0/673333634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:40:18.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:18 smithi097 ceph-mon[32231]: from='client.? 172.21.15.115:0/673333634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:40:19.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:19 smithi115 ceph-mon[35364]: pgmap v90: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:19 smithi181 ceph-mon[35869]: pgmap v90: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:19.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:19 smithi097 ceph-mon[32231]: pgmap v90: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:20.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:20 smithi115 ceph-mon[35364]: pgmap v91: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:20 smithi181 ceph-mon[35869]: pgmap v91: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:20.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:20 smithi097 ceph-mon[32231]: pgmap v91: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:22.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:22 smithi115 ceph-mon[35364]: pgmap v92: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:22 smithi181 ceph-mon[35869]: pgmap v92: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:22.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:22 smithi097 ceph-mon[32231]: pgmap v92: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:24 smithi115 ceph-mon[35364]: pgmap v93: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:24 smithi181 ceph-mon[35869]: pgmap v93: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:24.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:24 smithi097 ceph-mon[32231]: pgmap v93: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:26.180 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:26 smithi115 ceph-mon[35364]: pgmap v94: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:26 smithi181 ceph-mon[35869]: pgmap v94: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:26.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:26 smithi097 ceph-mon[32231]: pgmap v94: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:27.273 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T07:40:27.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:27.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:27 smithi115 ceph-mon[35364]: Deploying daemon osd.1 on smithi115 2024-07-26T07:40:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T07:40:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:27 smithi181 ceph-mon[35869]: Deploying daemon osd.1 on smithi115 2024-07-26T07:40:27.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T07:40:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:27 smithi097 ceph-mon[32231]: Deploying daemon osd.1 on smithi115 2024-07-26T07:40:28.161 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:28 smithi115 ceph-mon[35364]: pgmap v95: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:28.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:28 smithi181 ceph-mon[35869]: pgmap v95: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:28.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:28 smithi097 ceph-mon[32231]: pgmap v95: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:30.325 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:30 smithi097 ceph-mon[32231]: pgmap v96: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:30 smithi115 ceph-mon[35364]: pgmap v96: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:30 smithi181 ceph-mon[35869]: pgmap v96: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:32 smithi115 ceph-mon[35364]: pgmap v97: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:32 smithi097 ceph-mon[32231]: pgmap v97: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:32 smithi181 ceph-mon[35869]: pgmap v97: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:34.296 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:34 smithi115 ceph-mon[35364]: pgmap v98: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:34 smithi181 ceph-mon[35869]: pgmap v98: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:34.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:34 smithi097 ceph-mon[32231]: pgmap v98: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:35.306 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:35.306 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:35.306 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:36.323 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:36 smithi115 ceph-mon[35364]: pgmap v99: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:36 smithi181 ceph-mon[35869]: pgmap v99: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:36.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:36 smithi097 ceph-mon[32231]: pgmap v99: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:38 smithi115 ceph-mon[35364]: pgmap v100: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:38 smithi181 ceph-mon[35869]: pgmap v100: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:38 smithi097 ceph-mon[32231]: pgmap v100: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:39.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.507+0000 7fae86000640 1 -- 172.21.15.115:0/2985597439 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fae88063230 con 0x7fae680b3bc0 2024-07-26T07:40:39.509 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 1 on host 'smithi115' 2024-07-26T07:40:39.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae680b3bc0 msgr2=0x7fae680b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:39.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae680b3bc0 0x7fae680b60d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fae70006010 tx=0x7fae70000bc0 comp rx=0 tx=0).stop 2024-07-26T07:40:39.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae88106c00 msgr2=0x7fae8819cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:39.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae88106c00 0x7fae8819cb80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fae8000eaa0 tx=0x7fae8000ef60 comp rx=0 tx=0).stop 2024-07-26T07:40:39.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 shutdown_connections 2024-07-26T07:40:39.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae680b3bc0 0x7fae680b60d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:39.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae88106c00 0x7fae8819cb80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:39.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae88106260 0x7fae8819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:39.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 --2- 172.21.15.115:0/2985597439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae881051d0 0x7fae8819b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:39.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 >> 172.21.15.115:0/2985597439 conn(0x7fae88100b60 msgr2=0x7fae88109580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:40:39.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 shutdown_connections 2024-07-26T07:40:39.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:40:39.511+0000 7fae8de00640 1 -- 172.21.15.115:0/2985597439 wait complete. 2024-07-26T07:40:39.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:39 smithi115 ceph-mon[35364]: from='osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T07:40:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:39 smithi115 ceph-mon[35364]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T07:40:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:39 smithi181 ceph-mon[35869]: from='osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T07:40:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:39 smithi181 ceph-mon[35869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T07:40:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:40.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:39 smithi097 ceph-mon[32231]: from='osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T07:40:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:39 smithi097 ceph-mon[32231]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T07:40:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:40.320 DEBUG:teuthology.orchestra.run.smithi115:osd.1> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.1.service 2024-07-26T07:40:40.325 INFO:tasks.cephadm:Deploying osd.2 on smithi181 with /dev/vg_nvme/lv_4... 2024-07-26T07:40:40.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T07:40:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:40 smithi181 ceph-mon[35869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T07:40:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:40 smithi181 ceph-mon[35869]: from='osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:40:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:40 smithi181 ceph-mon[35869]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T07:40:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:40 smithi181 ceph-mon[35869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:40:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:40 smithi181 ceph-mon[35869]: pgmap v102: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:40.788 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:40:40.866 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:40 smithi115 ceph-mon[35364]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T07:40:40.866 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:40 smithi115 ceph-mon[35364]: from='osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:40:40.867 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:40 smithi115 ceph-mon[35364]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T07:40:40.867 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:40.867 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:40 smithi115 ceph-mon[35364]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:40:40.867 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:40 smithi115 ceph-mon[35364]: pgmap v102: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:41.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:40 smithi097 ceph-mon[32231]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T07:40:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:40 smithi097 ceph-mon[32231]: from='osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:40:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:40 smithi097 ceph-mon[32231]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T07:40:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:40 smithi097 ceph-mon[32231]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:40:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:40 smithi097 ceph-mon[32231]: pgmap v102: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:41.676 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:41 smithi115 ceph-mon[35364]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:40:41.676 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:41 smithi115 ceph-mon[35364]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T07:40:41.677 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:41.677 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:41.932 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:41 smithi097 ceph-mon[32231]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:40:41.932 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:41 smithi097 ceph-mon[32231]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T07:40:41.932 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:41.932 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:41.961 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:41 smithi181 ceph-mon[35869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:40:41.961 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:41 smithi181 ceph-mon[35869]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T07:40:41.961 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:41.961 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:41.974 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:40:41 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-1[39980]: 2024-07-26T07:40:41.674+0000 7fb93be00640 -1 osd.1 0 waiting for initial osdmap 2024-07-26T07:40:41.974 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:40:41 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-1[39980]: 2024-07-26T07:40:41.718+0000 7fb931e00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T07:40:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:42 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:40:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:42 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:40:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:42 smithi115 ceph-mon[35364]: from='osd.1 ' entity='osd.1' 2024-07-26T07:40:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:42 smithi115 ceph-mon[35364]: pgmap v104: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:42 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:40:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:42 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:40:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:42 smithi181 ceph-mon[35869]: from='osd.1 ' entity='osd.1' 2024-07-26T07:40:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:42 smithi181 ceph-mon[35869]: pgmap v104: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:42 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:40:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:42 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:40:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:42 smithi097 ceph-mon[32231]: from='osd.1 ' entity='osd.1' 2024-07-26T07:40:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:42 smithi097 ceph-mon[32231]: pgmap v104: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-07-26T07:40:43.115 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T07:40:43.286 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T07:40:43.326 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-07-26T07:40:43.326 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-07-26T07:40:43.326 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0317599 s, 330 MB/s 2024-07-26T07:40:43.326 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-07-26T07:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:43 smithi181 ceph-mon[35869]: osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] boot 2024-07-26T07:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:43 smithi181 ceph-mon[35869]: osdmap e13: 2 total, 2 up, 2 in 2024-07-26T07:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:43.961 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:43.962 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:43 smithi115 ceph-mon[35364]: osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] boot 2024-07-26T07:40:43.962 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:43 smithi115 ceph-mon[35364]: osdmap e13: 2 total, 2 up, 2 in 2024-07-26T07:40:43.962 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:43 smithi097 ceph-mon[32231]: osd.1 [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] boot 2024-07-26T07:40:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:43 smithi097 ceph-mon[32231]: osdmap e13: 2 total, 2 up, 2 in 2024-07-26T07:40:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T07:40:44.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-07-26T07:40:44.556 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:40:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:44 smithi115 ceph-mon[35364]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T07:40:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:44 smithi115 ceph-mon[35364]: pgmap v107: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:44 smithi181 ceph-mon[35869]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T07:40:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:44 smithi181 ceph-mon[35869]: pgmap v107: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:45.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:44 smithi097 ceph-mon[32231]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T07:40:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:44 smithi097 ceph-mon[32231]: pgmap v107: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi115 2024-07-26T07:40:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 17089M 2024-07-26T07:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi115 2024-07-26T07:40:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:40:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 17089M 2024-07-26T07:40:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi115 2024-07-26T07:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 17089M 2024-07-26T07:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:40:46.670 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.668+0000 7f37b9200640 1 -- 172.21.15.181:0/2028176906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 msgr2=0x7f37b4104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:46.670 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.668+0000 7f37b9200640 1 --2- 172.21.15.181:0/2028176906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 0x7f37b4104440 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009900 tx=0x7f37a002a480 comp rx=0 tx=0).stop 2024-07-26T07:40:46.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.669+0000 7f37b9200640 1 -- 172.21.15.181:0/2028176906 shutdown_connections 2024-07-26T07:40:46.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.669+0000 7f37b9200640 1 --2- 172.21.15.181:0/2028176906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b41049d0 0x7f37b410b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:46.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.669+0000 7f37b9200640 1 --2- 172.21.15.181:0/2028176906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 0x7f37b4104440 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:46.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.669+0000 7f37b9200640 1 --2- 172.21.15.181:0/2028176906 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4102fa0 0x7f37b4103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:46.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.669+0000 7f37b9200640 1 -- 172.21.15.181:0/2028176906 >> 172.21.15.181:0/2028176906 conn(0x7f37b40fe8f0 msgr2=0x7f37b4100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:40:46.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.669+0000 7f37b9200640 1 -- 172.21.15.181:0/2028176906 shutdown_connections 2024-07-26T07:40:46.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.669+0000 7f37b9200640 1 -- 172.21.15.181:0/2028176906 wait complete. 2024-07-26T07:40:46.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.670+0000 7f37b9200640 1 Processor -- start 2024-07-26T07:40:46.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.670+0000 7f37b9200640 1 -- start start 2024-07-26T07:40:46.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4102fa0 0x7f37b4199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:46.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b9200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 0x7f37b4199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:46.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b9200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b41049d0 0x7f37b419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:46.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b9200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f37b410eb00 con 0x7f37b4102fa0 2024-07-26T07:40:46.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b9200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f37b410e980 con 0x7f37b41049d0 2024-07-26T07:40:46.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b9200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f37b410ec80 con 0x7f37b4104030 2024-07-26T07:40:46.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 0x7f37b4199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:46.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 0x7f37b4199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46534/0 (socket says 172.21.15.181:46534) 2024-07-26T07:40:46.673 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b2a00640 1 -- 172.21.15.181:0/2705485059 learned_addr learned my addr 172.21.15.181:0/2705485059 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-07-26T07:40:46.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.671+0000 7f37b3400640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4102fa0 0x7f37b4199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:46.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b3e00640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b41049d0 0x7f37b419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:46.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b2a00640 1 -- 172.21.15.181:0/2705485059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b41049d0 msgr2=0x7f37b419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:46.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b2a00640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b41049d0 0x7f37b419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:46.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b2a00640 1 -- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4102fa0 msgr2=0x7f37b4199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:40:46.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b2a00640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4102fa0 0x7f37b4199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b2a00640 1 -- 172.21.15.181:0/2705485059 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b419b190 con 0x7f37b4104030 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b3400640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4102fa0 0x7f37b4199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b2a00640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 0x7f37b4199dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f37a0005760 tx=0x7f37a0003d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37abe00640 1 -- 172.21.15.181:0/2705485059 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a003f070 con 0x7f37b4104030 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37abe00640 1 -- 172.21.15.181:0/2705485059 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37a0031da0 con 0x7f37b4104030 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37abe00640 1 -- 172.21.15.181:0/2705485059 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0035030 con 0x7f37b4104030 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.672+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b419b420 con 0x7f37b4104030 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.673+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b41a1b80 con 0x7f37b4104030 2024-07-26T07:40:46.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.674+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b4103410 con 0x7f37b4104030 2024-07-26T07:40:46.679 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.678+0000 7f37abe00640 1 -- 172.21.15.181:0/2705485059 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f37a0042070 con 0x7f37b4104030 2024-07-26T07:40:46.680 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.679+0000 7f37abe00640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f37980b3c40 0x7f37980b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:46.681 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.680+0000 7f37abe00640 1 -- 172.21.15.181:0/2705485059 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f37a0103500 con 0x7f37b4104030 2024-07-26T07:40:46.681 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.680+0000 7f37b3400640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f37980b3c40 0x7f37980b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:46.682 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.681+0000 7f37b3400640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f37980b3c40 0x7f37980b6150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37a40058c0 tx=0x7f37a4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:40:46.683 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.682+0000 7f37abe00640 1 -- 172.21.15.181:0/2705485059 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f37a00cc300 con 0x7f37b4104030 2024-07-26T07:40:46.893 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:40:46.892+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f37b4063230 con 0x7f37980b3c40 2024-07-26T07:40:47.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:46 smithi115 ceph-mon[35364]: pgmap v108: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:46 smithi181 ceph-mon[35869]: pgmap v108: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:47.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:46 smithi097 ceph-mon[32231]: pgmap v108: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:47.931 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:47 smithi181 ceph-mon[35869]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:40:47.931 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:40:47.932 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:40:47.932 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:47 smithi115 ceph-mon[35364]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:40:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:40:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:40:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:47 smithi097 ceph-mon[32231]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:40:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:40:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:40:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:48 smithi181 ceph-mon[35869]: pgmap v109: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:49.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:48 smithi115 ceph-mon[35364]: pgmap v109: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:49.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:48 smithi097 ceph-mon[32231]: pgmap v109: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:50.116 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:50 smithi181 ceph-mon[35869]: pgmap v110: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:50.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:50 smithi115 ceph-mon[35364]: pgmap v110: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:50.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:50 smithi097 ceph-mon[32231]: pgmap v110: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:51 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/109706427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]: dispatch 2024-07-26T07:40:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:51 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]: dispatch 2024-07-26T07:40:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:51 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]': finished 2024-07-26T07:40:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:51 smithi181 ceph-mon[35869]: osdmap e15: 3 total, 2 up, 3 in 2024-07-26T07:40:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:40:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:51 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/109706427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]: dispatch 2024-07-26T07:40:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:51 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]: dispatch 2024-07-26T07:40:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:51 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]': finished 2024-07-26T07:40:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:51 smithi115 ceph-mon[35364]: osdmap e15: 3 total, 2 up, 3 in 2024-07-26T07:40:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:40:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:51 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/109706427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]: dispatch 2024-07-26T07:40:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:51 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]: dispatch 2024-07-26T07:40:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:51 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9"}]': finished 2024-07-26T07:40:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:51 smithi097 ceph-mon[32231]: osdmap e15: 3 total, 2 up, 3 in 2024-07-26T07:40:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:40:52.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:52 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/4006612780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:40:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:52 smithi115 ceph-mon[35364]: pgmap v112: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:52 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/4006612780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:40:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:52 smithi181 ceph-mon[35869]: pgmap v112: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:52.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:52 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/4006612780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:40:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:52 smithi097 ceph-mon[32231]: pgmap v112: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:54.412 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:54 smithi181 ceph-mon[35869]: pgmap v113: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:54.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:54 smithi115 ceph-mon[35364]: pgmap v113: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:54.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:54 smithi097 ceph-mon[32231]: pgmap v113: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:56 smithi181 ceph-mon[35869]: pgmap v114: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:56 smithi115 ceph-mon[35364]: pgmap v114: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:56.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:56 smithi097 ceph-mon[32231]: pgmap v114: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:58.191 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:58 smithi181 ceph-mon[35869]: pgmap v115: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:58.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:58 smithi115 ceph-mon[35364]: pgmap v115: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:58.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:58 smithi097 ceph-mon[32231]: pgmap v115: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:40:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T07:40:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:40:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T07:40:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:40:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:00.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T07:41:00.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:40:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:00.841 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:00 smithi181 ceph-mon[35869]: Deploying daemon osd.2 on smithi181 2024-07-26T07:41:00.841 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:00 smithi181 ceph-mon[35869]: pgmap v116: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:00 smithi115 ceph-mon[35364]: Deploying daemon osd.2 on smithi181 2024-07-26T07:41:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:00 smithi115 ceph-mon[35364]: pgmap v116: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:00 smithi097 ceph-mon[32231]: Deploying daemon osd.2 on smithi181 2024-07-26T07:41:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:00 smithi097 ceph-mon[32231]: pgmap v116: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:02.348 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:02 smithi181 ceph-mon[35869]: pgmap v117: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:02.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:02 smithi115 ceph-mon[35364]: pgmap v117: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:02.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:02 smithi097 ceph-mon[32231]: pgmap v117: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:04.425 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:04 smithi181 ceph-mon[35869]: pgmap v118: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:04 smithi115 ceph-mon[35364]: pgmap v118: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:04.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:04 smithi097 ceph-mon[32231]: pgmap v118: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:06.467 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:06 smithi181 ceph-mon[35869]: pgmap v119: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:06.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:06 smithi115 ceph-mon[35364]: pgmap v119: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:06.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:06 smithi097 ceph-mon[32231]: pgmap v119: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:08.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:08 smithi115 ceph-mon[35364]: pgmap v120: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:08 smithi181 ceph-mon[35869]: pgmap v120: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:08.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:08 smithi097 ceph-mon[32231]: pgmap v120: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:09.341 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:09.341 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:09.341 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:10.145 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:10 smithi181 ceph-mon[35869]: pgmap v121: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:10 smithi115 ceph-mon[35364]: pgmap v121: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:10.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:10 smithi097 ceph-mon[32231]: pgmap v121: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:12.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:12.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:12.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:12.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:12.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:12.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:12 smithi181 ceph-mon[35869]: pgmap v122: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:12.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:12 smithi181 ceph-mon[35869]: from='osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T07:41:12.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:12 smithi181 ceph-mon[35869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T07:41:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:12 smithi115 ceph-mon[35364]: pgmap v122: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:12 smithi115 ceph-mon[35364]: from='osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T07:41:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:12 smithi115 ceph-mon[35364]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T07:41:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:12 smithi097 ceph-mon[32231]: pgmap v122: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:12 smithi097 ceph-mon[32231]: from='osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T07:41:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:12 smithi097 ceph-mon[32231]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T07:41:13.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.463+0000 7f37abe00640 1 -- 172.21.15.181:0/2705485059 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f37b4063230 con 0x7f37980b3c40 2024-07-26T07:41:13.464 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 2 on host 'smithi181' 2024-07-26T07:41:13.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f37980b3c40 msgr2=0x7f37980b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:13.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f37980b3c40 0x7f37980b6150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37a40058c0 tx=0x7f37a4005850 comp rx=0 tx=0).stop 2024-07-26T07:41:13.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 msgr2=0x7f37b4199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:13.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 0x7f37b4199dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f37a0005760 tx=0x7f37a0003d50 comp rx=0 tx=0).stop 2024-07-26T07:41:13.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 shutdown_connections 2024-07-26T07:41:13.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f37980b3c40 0x7f37980b6150 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:13.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b41049d0 0x7f37b419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:13.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37b4104030 0x7f37b4199dd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:13.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 --2- 172.21.15.181:0/2705485059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b4102fa0 0x7f37b4199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:13.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.467+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 >> 172.21.15.181:0/2705485059 conn(0x7f37b40fe8f0 msgr2=0x7f37b4107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:41:13.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.468+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 shutdown_connections 2024-07-26T07:41:13.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:41:13.468+0000 7f37b9200640 1 -- 172.21.15.181:0/2705485059 wait complete. 2024-07-26T07:41:13.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:13 smithi181 ceph-mon[35869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T07:41:13.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:13 smithi181 ceph-mon[35869]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T07:41:13.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:13.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:13 smithi181 ceph-mon[35869]: from='osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:41:13.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:13 smithi181 ceph-mon[35869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:41:13.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:13.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:13.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:13 smithi115 ceph-mon[35364]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T07:41:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:13 smithi115 ceph-mon[35364]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T07:41:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:13 smithi115 ceph-mon[35364]: from='osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:41:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:13 smithi115 ceph-mon[35364]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:41:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:13 smithi097 ceph-mon[32231]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T07:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:13 smithi097 ceph-mon[32231]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T07:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:13 smithi097 ceph-mon[32231]: from='osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:13 smithi097 ceph-mon[32231]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:14.248 DEBUG:teuthology.orchestra.run.smithi181:osd.2> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.2.service 2024-07-26T07:41:14.253 INFO:tasks.cephadm:Deploying osd.3 on smithi097 with /dev/vg_nvme/lv_3... 2024-07-26T07:41:14.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T07:41:14.710 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:41:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:14 smithi181 ceph-mon[35869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:41:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:14 smithi181 ceph-mon[35869]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T07:41:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:14 smithi181 ceph-mon[35869]: pgmap v125: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:14 smithi115 ceph-mon[35364]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:41:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:14 smithi115 ceph-mon[35364]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T07:41:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:14 smithi115 ceph-mon[35364]: pgmap v125: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:14 smithi097 ceph-mon[32231]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:41:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:14 smithi097 ceph-mon[32231]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T07:41:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:14 smithi097 ceph-mon[32231]: pgmap v125: 0 pgs: ; 0 B data, 69 MiB used, 179 GiB / 179 GiB avail 2024-07-26T07:41:15.877 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:15 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:41:15.877 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:15 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:41:15.877 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:15 smithi097 ceph-mon[32231]: osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] boot 2024-07-26T07:41:15.877 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:15 smithi097 ceph-mon[32231]: osdmap e18: 3 total, 3 up, 3 in 2024-07-26T07:41:15.877 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:15.943 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:15 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:41:15.943 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:15 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:41:15.943 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:15 smithi181 ceph-mon[35869]: osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] boot 2024-07-26T07:41:15.943 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:15 smithi181 ceph-mon[35869]: osdmap e18: 3 total, 3 up, 3 in 2024-07-26T07:41:15.943 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:15 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:41:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:15 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:41:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:15 smithi115 ceph-mon[35364]: osd.2 [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] boot 2024-07-26T07:41:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:15 smithi115 ceph-mon[35364]: osdmap e18: 3 total, 3 up, 3 in 2024-07-26T07:41:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T07:41:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:16 smithi115 ceph-mon[35364]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T07:41:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:16 smithi115 ceph-mon[35364]: pgmap v128: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T07:41:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:16 smithi181 ceph-mon[35869]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T07:41:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:16 smithi181 ceph-mon[35869]: pgmap v128: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T07:41:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:16 smithi097 ceph-mon[32231]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T07:41:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:16 smithi097 ceph-mon[32231]: pgmap v128: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T07:41:17.032 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T07:41:17.245 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T07:41:17.287 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-07-26T07:41:17.287 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-07-26T07:41:17.287 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0331181 s, 317 MB/s 2024-07-26T07:41:17.288 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 2024-07-26T07:41:17.288 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-07-26T07:41:17.876 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T07:41:17.877 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:17 smithi097 ceph-mon[32231]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T07:41:17.877 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T07:41:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T07:41:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:17 smithi115 ceph-mon[35364]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T07:41:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T07:41:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T07:41:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:17 smithi181 ceph-mon[35869]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T07:41:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T07:41:18.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_3 2024-07-26T07:41:18.140 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:41:17 smithi097 sudo[54287]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T07:41:18.140 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:41:17 smithi097 sudo[54287]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.140 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:41:17 smithi097 sudo[54287]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.140 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:41:17 smithi097 sudo[54287]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.140 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:41:17 smithi097 sudo[54290]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T07:41:18.141 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:41:17 smithi097 sudo[54290]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.141 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:41:17 smithi097 sudo[54290]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.141 INFO:journalctl@ceph.osd.0.smithi097.stdout:Jul 26 07:41:17 smithi097 sudo[54290]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.474 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44125]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T07:41:18.474 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44125]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.474 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44125]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.474 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44125]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.475 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44128]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T07:41:18.475 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44128]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.475 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44128]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.475 INFO:journalctl@ceph.osd.1.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44128]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.482 INFO:journalctl@ceph.osd.2.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44883]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T07:41:18.482 INFO:journalctl@ceph.osd.2.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44883]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.482 INFO:journalctl@ceph.osd.2.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44883]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.482 INFO:journalctl@ceph.osd.2.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44883]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.483 INFO:journalctl@ceph.osd.2.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44886]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T07:41:18.483 INFO:journalctl@ceph.osd.2.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44886]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.483 INFO:journalctl@ceph.osd.2.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44886]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.483 INFO:journalctl@ceph.osd.2.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44886]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.503 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:41:18.746 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 sudo[54337]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T07:41:18.746 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 sudo[54337]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.746 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 sudo[54337]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.746 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 sudo[54337]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.746 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 sudo[54348]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T07:41:18.746 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 sudo[54348]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.747 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 sudo[54348]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.747 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 sudo[54348]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44131]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T07:41:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44131]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44131]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44131]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44134]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44134]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44134]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 sudo[44134]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: pgmap v131: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:18.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44891]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44891]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44891]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44891]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44894]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44894]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44894]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 sudo[44894]: pam_unix(sudo:session): session closed for user root 2024-07-26T07:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: pgmap v131: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:18.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:18.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: pgmap v131: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T07:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T07:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T07:41:19.930 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi181 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 17089M 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:19.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:19 smithi097 ceph-mon[32231]: mgrmap e16: x(active, since 4m), standbys: z, y 2024-07-26T07:41:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi181 2024-07-26T07:41:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 17089M 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:19 smithi115 ceph-mon[35364]: mgrmap e16: x(active, since 4m), standbys: z, y 2024-07-26T07:41:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T07:41:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi181 2024-07-26T07:41:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T07:41:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 17089M 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:19 smithi181 ceph-mon[35869]: mgrmap e16: x(active, since 4m), standbys: z, y 2024-07-26T07:41:20.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 -- 172.21.15.97:0/705686260 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c10b000 msgr2=0x7fef0c1119a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 --2- 172.21.15.97:0/705686260 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c10b000 0x7fef0c1119a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fef04009ca0 tx=0x7fef0402a950 comp rx=0 tx=0).stop 2024-07-26T07:41:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 -- 172.21.15.97:0/705686260 shutdown_connections 2024-07-26T07:41:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 --2- 172.21.15.97:0/705686260 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c10b000 0x7fef0c1119a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 --2- 172.21.15.97:0/705686260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef0c10a660 0x7fef0c10aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 --2- 172.21.15.97:0/705686260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef0c1095d0 0x7fef0c109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 -- 172.21.15.97:0/705686260 >> 172.21.15.97:0/705686260 conn(0x7fef0c078970 msgr2=0x7fef0c07ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:41:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 -- 172.21.15.97:0/705686260 shutdown_connections 2024-07-26T07:41:20.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.495+0000 7fef0be00640 1 -- 172.21.15.97:0/705686260 wait complete. 2024-07-26T07:41:20.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.496+0000 7fef0be00640 1 Processor -- start 2024-07-26T07:41:20.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0be00640 1 -- start start 2024-07-26T07:41:20.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1095d0 0x7fef0c19fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef0c10a660 0x7fef0c1a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0be00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef0c10b000 0x7fef0c1a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c115120 con 0x7fef0c1095d0 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0be00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c114fa0 con 0x7fef0c10b000 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c1152a0 con 0x7fef0c10a660 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1095d0 0x7fef0c19fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1095d0 0x7fef0c19fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50740/0 (socket says 172.21.15.97:50740) 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.497+0000 7fef0aa00640 1 -- 172.21.15.97:0/1120617436 learned_addr learned my addr 172.21.15.97:0/1120617436 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef02000640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef0c10a660 0x7fef0c1a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef0b400640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef0c10b000 0x7fef0c1a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:41:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef0aa00640 1 -- 172.21.15.97:0/1120617436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef0c10a660 msgr2=0x7fef0c1a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef0aa00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef0c10a660 0x7fef0c1a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef0aa00640 1 -- 172.21.15.97:0/1120617436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef0c10b000 msgr2=0x7fef0c1a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef0aa00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef0c10b000 0x7fef0c1a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef0aa00640 1 -- 172.21.15.97:0/1120617436 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0c1a1780 con 0x7fef0c1095d0 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef0b400640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef0c10b000 0x7fef0c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.498+0000 7fef0aa00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1095d0 0x7fef0c19fd50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feefc00d950 tx=0x7feefc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.499+0000 7fef03e00640 1 -- 172.21.15.97:0/1120617436 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc017070 con 0x7fef0c1095d0 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.499+0000 7fef03e00640 1 -- 172.21.15.97:0/1120617436 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feefc009c30 con 0x7fef0c1095d0 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.499+0000 7fef03e00640 1 -- 172.21.15.97:0/1120617436 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc00f620 con 0x7fef0c1095d0 2024-07-26T07:41:20.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.499+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0c1a1a70 con 0x7fef0c1095d0 2024-07-26T07:41:20.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.499+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0c1a8170 con 0x7fef0c1095d0 2024-07-26T07:41:20.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.500+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef0c110790 con 0x7fef0c1095d0 2024-07-26T07:41:20.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.503+0000 7fef03e00640 1 -- 172.21.15.97:0/1120617436 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7feefc010460 con 0x7fef0c1095d0 2024-07-26T07:41:20.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.504+0000 7fef03e00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feef00b3c50 0x7feef00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:41:20.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.504+0000 7fef03e00640 1 -- 172.21.15.97:0/1120617436 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 3237+0+0 (secure 0 0 0) 0x7feefc0e2530 con 0x7fef0c1095d0 2024-07-26T07:41:20.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.505+0000 7fef02000640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feef00b3c50 0x7feef00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:41:20.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.505+0000 7fef02000640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feef00b3c50 0x7feef00b6160 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feef4005fd0 tx=0x7feef4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:41:20.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.508+0000 7fef03e00640 1 -- 172.21.15.97:0/1120617436 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feefc0aaf30 con 0x7fef0c1095d0 2024-07-26T07:41:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:20.718+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fef0c063230 con 0x7feef00b3c50 2024-07-26T07:41:21.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:20 smithi115 ceph-mon[35364]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:41:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:41:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:20 smithi181 ceph-mon[35869]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:41:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:41:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:21.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:20 smithi097 ceph-mon[32231]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:21.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:41:21.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:41:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:21 smithi097 ceph-mon[32231]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:41:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:21 smithi115 ceph-mon[35364]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:41:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:21 smithi181 ceph-mon[35869]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:41:23.113 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:22 smithi097 ceph-mon[32231]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:23.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:22 smithi115 ceph-mon[35364]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:22 smithi181 ceph-mon[35869]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:24.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:24 smithi097 ceph-mon[32231]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 50 KiB/s wr, 4 op/s 2024-07-26T07:41:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:24 smithi115 ceph-mon[35364]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 50 KiB/s wr, 4 op/s 2024-07-26T07:41:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:24 smithi181 ceph-mon[35869]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 50 KiB/s wr, 4 op/s 2024-07-26T07:41:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:25 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2721296424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "144e0dff-895b-471b-bd0c-8fcf3f51ac36"}]: dispatch 2024-07-26T07:41:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:25 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2721296424' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "144e0dff-895b-471b-bd0c-8fcf3f51ac36"}]': finished 2024-07-26T07:41:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:25 smithi115 ceph-mon[35364]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T07:41:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:25 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2721296424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "144e0dff-895b-471b-bd0c-8fcf3f51ac36"}]: dispatch 2024-07-26T07:41:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:25 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2721296424' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "144e0dff-895b-471b-bd0c-8fcf3f51ac36"}]': finished 2024-07-26T07:41:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:25 smithi181 ceph-mon[35869]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T07:41:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:25 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2721296424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "144e0dff-895b-471b-bd0c-8fcf3f51ac36"}]: dispatch 2024-07-26T07:41:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:25 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2721296424' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "144e0dff-895b-471b-bd0c-8fcf3f51ac36"}]': finished 2024-07-26T07:41:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:25 smithi097 ceph-mon[32231]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T07:41:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:26 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2903132079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:41:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:26 smithi181 ceph-mon[35869]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 3 op/s 2024-07-26T07:41:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:26 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2903132079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:41:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:26 smithi115 ceph-mon[35364]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 3 op/s 2024-07-26T07:41:26.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:26 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2903132079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:41:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:26 smithi097 ceph-mon[32231]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 3 op/s 2024-07-26T07:41:28.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:28 smithi097 ceph-mon[32231]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 996 B/s rd, 28 KiB/s wr, 2 op/s 2024-07-26T07:41:28.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:28 smithi115 ceph-mon[35364]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 996 B/s rd, 28 KiB/s wr, 2 op/s 2024-07-26T07:41:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:28 smithi181 ceph-mon[35869]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 996 B/s rd, 28 KiB/s wr, 2 op/s 2024-07-26T07:41:30.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:30 smithi097 ceph-mon[32231]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-07-26T07:41:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:30 smithi115 ceph-mon[35364]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-07-26T07:41:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:30 smithi181 ceph-mon[35869]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-07-26T07:41:32.383 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:32 smithi097 ceph-mon[32231]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-07-26T07:41:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:32 smithi115 ceph-mon[35364]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-07-26T07:41:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:32 smithi181 ceph-mon[35869]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-07-26T07:41:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T07:41:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T07:41:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:34.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T07:41:34.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:34 smithi115 ceph-mon[35364]: Deploying daemon osd.3 on smithi097 2024-07-26T07:41:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:34 smithi115 ceph-mon[35364]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:34 smithi181 ceph-mon[35869]: Deploying daemon osd.3 on smithi097 2024-07-26T07:41:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:34 smithi181 ceph-mon[35869]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:34 smithi097 ceph-mon[32231]: Deploying daemon osd.3 on smithi097 2024-07-26T07:41:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:34 smithi097 ceph-mon[32231]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:36.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:36 smithi115 ceph-mon[35364]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:36 smithi181 ceph-mon[35869]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:36.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:36 smithi097 ceph-mon[32231]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:38.470 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:38 smithi097 ceph-mon[32231]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:38.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:38 smithi115 ceph-mon[35364]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:38 smithi181 ceph-mon[35869]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:40.371 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:40 smithi097 ceph-mon[32231]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:40 smithi115 ceph-mon[35364]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:40 smithi181 ceph-mon[35869]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:42.461 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:42 smithi097 ceph-mon[32231]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:42.462 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:42.462 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:42.462 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:42 smithi115 ceph-mon[35364]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:42 smithi181 ceph-mon[35869]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:44 smithi115 ceph-mon[35364]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:44 smithi181 ceph-mon[35869]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:44 smithi097 ceph-mon[32231]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:46.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:46.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:46.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:46 smithi097 ceph-mon[32231]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:46 smithi115 ceph-mon[35364]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:46 smithi181 ceph-mon[35869]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:46.919 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 3 on host 'smithi097' 2024-07-26T07:41:46.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.917+0000 7fef03e00640 1 -- 172.21.15.97:0/1120617436 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fef0c063230 con 0x7feef00b3c50 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feef00b3c50 msgr2=0x7feef00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feef00b3c50 0x7feef00b6160 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feef4005fd0 tx=0x7feef4005850 comp rx=0 tx=0).stop 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1095d0 msgr2=0x7fef0c19fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1095d0 0x7fef0c19fd50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feefc00d950 tx=0x7feefc00de10 comp rx=0 tx=0).stop 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 shutdown_connections 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feef00b3c50 0x7feef00b6160 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef0c10b000 0x7fef0c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef0c10a660 0x7fef0c1a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 --2- 172.21.15.97:0/1120617436 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef0c1095d0 0x7fef0c19fd50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:46.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.922+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 >> 172.21.15.97:0/1120617436 conn(0x7fef0c078970 msgr2=0x7fef0c10d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:41:46.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.923+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 shutdown_connections 2024-07-26T07:41:46.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:41:46.923+0000 7fef0be00640 1 -- 172.21.15.97:0/1120617436 wait complete. 2024-07-26T07:41:47.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:47 smithi097 ceph-mon[32231]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T07:41:47.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:47 smithi097 ceph-mon[32231]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T07:41:47.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:47 smithi097 ceph-mon[32231]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T07:41:47.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:47 smithi097 ceph-mon[32231]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:41:47.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:47.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:47.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:47.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:47 smithi115 ceph-mon[35364]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T07:41:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:47 smithi115 ceph-mon[35364]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T07:41:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:47 smithi115 ceph-mon[35364]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T07:41:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:47 smithi115 ceph-mon[35364]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:41:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:47 smithi181 ceph-mon[35869]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T07:41:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:47 smithi181 ceph-mon[35869]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T07:41:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:47 smithi181 ceph-mon[35869]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T07:41:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:47 smithi181 ceph-mon[35869]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:41:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:47.721 DEBUG:teuthology.orchestra.run.smithi097:osd.3> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.3.service 2024-07-26T07:41:47.726 INFO:tasks.cephadm:Deploying osd.4 on smithi115 with /dev/vg_nvme/lv_3... 2024-07-26T07:41:47.727 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T07:41:48.190 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T07:41:48.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:48 smithi097 ceph-mon[32231]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:41:48.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:48 smithi097 ceph-mon[32231]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T07:41:48.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:48.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:48.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:48 smithi097 ceph-mon[32231]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:48.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:48 smithi115 ceph-mon[35364]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:41:48.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:48 smithi115 ceph-mon[35364]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T07:41:48.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:48.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:48.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:48 smithi115 ceph-mon[35364]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:48 smithi181 ceph-mon[35869]: from='osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:41:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:48 smithi181 ceph-mon[35869]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T07:41:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:48 smithi181 ceph-mon[35869]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-07-26T07:41:49.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:49 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:41:49.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:49 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:41:49.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:49 smithi115 ceph-mon[35364]: osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] boot 2024-07-26T07:41:49.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:49 smithi115 ceph-mon[35364]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T07:41:49.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:49.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:49 smithi115 ceph-mon[35364]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T07:41:49.392 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:49 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:41:49.392 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:49 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:41:49.392 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:49 smithi097 ceph-mon[32231]: osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] boot 2024-07-26T07:41:49.392 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:49 smithi097 ceph-mon[32231]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T07:41:49.392 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:49.392 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:49 smithi097 ceph-mon[32231]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T07:41:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:49 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:41:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:49 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:41:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:49 smithi181 ceph-mon[35869]: osd.3 [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] boot 2024-07-26T07:41:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:49 smithi181 ceph-mon[35869]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T07:41:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T07:41:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:49 smithi181 ceph-mon[35869]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T07:41:50.758 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T07:41:50.940 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:50 smithi097 ceph-mon[32231]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T07:41:50.940 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:50 smithi097 ceph-mon[32231]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:41:50.957 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:50 smithi115 ceph-mon[35364]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T07:41:50.958 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:50 smithi115 ceph-mon[35364]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:41:50.958 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T07:41:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:50 smithi181 ceph-mon[35869]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T07:41:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:50 smithi181 ceph-mon[35869]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:41:50.999 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-07-26T07:41:50.999 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-07-26T07:41:50.999 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.030128 s, 348 MB/s 2024-07-26T07:41:51.000 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-07-26T07:41:51.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi115:vg_nvme/lv_3 2024-07-26T07:41:52.245 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T07:41:53.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:41:53.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi097 2024-07-26T07:41:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:41:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi097 2024-07-26T07:41:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi097 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:41:54.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:54 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 8544M 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.351+0000 7fc38d400640 1 -- 172.21.15.115:0/2994280990 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3881095d0 msgr2=0x7fc388109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.351+0000 7fc38d400640 1 --2- 172.21.15.115:0/2994280990 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3881095d0 0x7fc388109a40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc378009900 tx=0x7fc37802a480 comp rx=0 tx=0).stop 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.352+0000 7fc38d400640 1 -- 172.21.15.115:0/2994280990 shutdown_connections 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.352+0000 7fc38d400640 1 --2- 172.21.15.115:0/2994280990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc38810b000 0x7fc3881119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.352+0000 7fc38d400640 1 --2- 172.21.15.115:0/2994280990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38810a660 0x7fc38810aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.352+0000 7fc38d400640 1 --2- 172.21.15.115:0/2994280990 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3881095d0 0x7fc388109a40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.352+0000 7fc38d400640 1 -- 172.21.15.115:0/2994280990 >> 172.21.15.115:0/2994280990 conn(0x7fc388078970 msgr2=0x7fc38807ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.352+0000 7fc38d400640 1 -- 172.21.15.115:0/2994280990 shutdown_connections 2024-07-26T07:41:54.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.352+0000 7fc38d400640 1 -- 172.21.15.115:0/2994280990 wait complete. 2024-07-26T07:41:54.354 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.353+0000 7fc38d400640 1 Processor -- start 2024-07-26T07:41:54.354 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.353+0000 7fc38d400640 1 -- start start 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.353+0000 7fc38d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3881095d0 0x7fc38819fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.353+0000 7fc38d400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc38810a660 0x7fc3881a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.353+0000 7fc38d400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38810b000 0x7fc3881a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc38d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc388115110 con 0x7fc3881095d0 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc38d400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc388114f90 con 0x7fc38810a660 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc38d400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc388115290 con 0x7fc38810b000 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc38810a660 0x7fc3881a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc38810a660 0x7fc3881a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56536/0 (socket says 172.21.15.115:56536) 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387400640 1 -- 172.21.15.115:0/4036742101 learned_addr learned my addr 172.21.15.115:0/4036742101 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-07-26T07:41:54.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387e00640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3881095d0 0x7fc38819fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:41:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc38ca00640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38810b000 0x7fc3881a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:41:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387400640 1 -- 172.21.15.115:0/4036742101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38810b000 msgr2=0x7fc3881a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38810b000 0x7fc3881a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387400640 1 -- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3881095d0 msgr2=0x7fc38819fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:41:54.356 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3881095d0 0x7fc38819fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:41:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.354+0000 7fc387400640 1 -- 172.21.15.115:0/4036742101 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3881a17f0 con 0x7fc38810a660 2024-07-26T07:41:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.355+0000 7fc387400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc38810a660 0x7fc3881a0430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc37400e9f0 tx=0x7fc37400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:41:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.355+0000 7fc384c00640 1 -- 172.21.15.115:0/4036742101 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37400cd80 con 0x7fc38810a660 2024-07-26T07:41:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.355+0000 7fc384c00640 1 -- 172.21.15.115:0/4036742101 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3740054b0 con 0x7fc38810a660 2024-07-26T07:41:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.355+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3881a1ae0 con 0x7fc38810a660 2024-07-26T07:41:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.355+0000 7fc384c00640 1 -- 172.21.15.115:0/4036742101 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc374010430 con 0x7fc38810a660 2024-07-26T07:41:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.355+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3881a81e0 con 0x7fc38810a660 2024-07-26T07:41:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.356+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3881107b0 con 0x7fc38810a660 2024-07-26T07:41:54.360 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.360+0000 7fc384c00640 1 -- 172.21.15.115:0/4036742101 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fc3740046c0 con 0x7fc38810a660 2024-07-26T07:41:54.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.361+0000 7fc384c00640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc36c0b3bd0 0x7fc36c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:41:54.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.361+0000 7fc384c00640 1 -- 172.21.15.115:0/4036742101 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fc3740e3160 con 0x7fc38810a660 2024-07-26T07:41:54.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.362+0000 7fc387e00640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc36c0b3bd0 0x7fc36c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:41:54.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.363+0000 7fc387e00640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc36c0b3bd0 0x7fc36c0b60e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc378006010 tx=0x7fc378005720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:41:54.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.363+0000 7fc384c00640 1 -- 172.21.15.115:0/4036742101 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc374055600 con 0x7fc38810a660 2024-07-26T07:41:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:54 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 8544M 2024-07-26T07:41:54.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:54 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 8544M 2024-07-26T07:41:54.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:41:54.578+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc388063230 con 0x7fc36c0b3bd0 2024-07-26T07:41:55.328 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:55 smithi115 ceph-mon[35364]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T07:41:55.328 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:55 smithi115 ceph-mon[35364]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:41:55.328 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:41:55.328 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:41:55.329 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:55 smithi181 ceph-mon[35869]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T07:41:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:55 smithi181 ceph-mon[35869]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:41:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:41:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:41:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:55 smithi097 ceph-mon[32231]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T07:41:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:55 smithi097 ceph-mon[32231]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:41:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:41:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:41:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:56.338 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:56 smithi115 ceph-mon[35364]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-26T07:41:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:56 smithi181 ceph-mon[35869]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-26T07:41:56.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:56 smithi097 ceph-mon[32231]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-26T07:41:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:58 smithi115 ceph-mon[35364]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-26T07:41:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:58 smithi181 ceph-mon[35869]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-26T07:41:58.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:58 smithi097 ceph-mon[32231]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-26T07:41:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:59 smithi115 ceph-mon[35364]: from='client.? 172.21.15.115:0/947075450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]: dispatch 2024-07-26T07:41:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:59 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]: dispatch 2024-07-26T07:41:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:59 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]': finished 2024-07-26T07:41:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:59 smithi115 ceph-mon[35364]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T07:41:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:41:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:41:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:59 smithi181 ceph-mon[35869]: from='client.? 172.21.15.115:0/947075450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]: dispatch 2024-07-26T07:41:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:59 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]: dispatch 2024-07-26T07:41:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:59 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]': finished 2024-07-26T07:41:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:59 smithi181 ceph-mon[35869]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T07:41:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:41:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:41:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:59 smithi097 ceph-mon[32231]: from='client.? 172.21.15.115:0/947075450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]: dispatch 2024-07-26T07:41:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:59 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]: dispatch 2024-07-26T07:41:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:59 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fddb4b8-44aa-42d7-8df4-c304b85c69b1"}]': finished 2024-07-26T07:41:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:59 smithi097 ceph-mon[32231]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T07:41:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:41:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:00 smithi181 ceph-mon[35869]: from='client.? 172.21.15.115:0/3783250921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:42:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:00 smithi181 ceph-mon[35869]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:42:00.384 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:00 smithi115 ceph-mon[35364]: from='client.? 172.21.15.115:0/3783250921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:42:00.384 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:00 smithi115 ceph-mon[35364]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:42:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:00 smithi097 ceph-mon[32231]: from='client.? 172.21.15.115:0/3783250921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:42:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:00 smithi097 ceph-mon[32231]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:42:02.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:02 smithi115 ceph-mon[35364]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:42:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:02 smithi181 ceph-mon[35869]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:42:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:02 smithi097 ceph-mon[32231]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:42:04.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:04 smithi115 ceph-mon[35364]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:04 smithi181 ceph-mon[35869]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:04.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:04 smithi097 ceph-mon[32231]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:06.205 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:06 smithi115 ceph-mon[35364]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:06 smithi181 ceph-mon[35869]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:06.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:06 smithi097 ceph-mon[32231]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:08.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:08 smithi115 ceph-mon[35364]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T07:42:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:08 smithi181 ceph-mon[35869]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T07:42:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:08 smithi097 ceph-mon[32231]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T07:42:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:09 smithi115 ceph-mon[35364]: Deploying daemon osd.4 on smithi115 2024-07-26T07:42:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:09 smithi181 ceph-mon[35869]: Deploying daemon osd.4 on smithi115 2024-07-26T07:42:09.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:09 smithi097 ceph-mon[32231]: Deploying daemon osd.4 on smithi115 2024-07-26T07:42:10.458 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:10 smithi115 ceph-mon[35364]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:10 smithi181 ceph-mon[35869]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:10.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:10 smithi097 ceph-mon[32231]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:12.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:12 smithi115 ceph-mon[35364]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:12 smithi181 ceph-mon[35869]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:12.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:12 smithi097 ceph-mon[32231]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:14 smithi115 ceph-mon[35364]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:14 smithi181 ceph-mon[35869]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:14 smithi097 ceph-mon[32231]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:16.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:16 smithi115 ceph-mon[35364]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:16 smithi181 ceph-mon[35869]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:16.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:16 smithi097 ceph-mon[32231]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:17.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:17.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:17.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:17.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:18 smithi115 ceph-mon[35364]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:18 smithi181 ceph-mon[35869]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:18 smithi097 ceph-mon[32231]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:20 smithi115 ceph-mon[35364]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:20 smithi181 ceph-mon[35869]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:20 smithi097 ceph-mon[32231]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:22.172 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.171+0000 7fc384c00640 1 -- 172.21.15.115:0/4036742101 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc388063230 con 0x7fc36c0b3bd0 2024-07-26T07:42:22.173 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 4 on host 'smithi115' 2024-07-26T07:42:22.176 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc36c0b3bd0 msgr2=0x7fc36c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:42:22.176 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc36c0b3bd0 0x7fc36c0b60e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc378006010 tx=0x7fc378005720 comp rx=0 tx=0).stop 2024-07-26T07:42:22.176 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc38810a660 msgr2=0x7fc3881a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:42:22.176 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc38810a660 0x7fc3881a0430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc37400e9f0 tx=0x7fc37400edf0 comp rx=0 tx=0).stop 2024-07-26T07:42:22.176 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 shutdown_connections 2024-07-26T07:42:22.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc36c0b3bd0 0x7fc36c0b60e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:22.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38810b000 0x7fc3881a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:22.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc38810a660 0x7fc3881a0430 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:22.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 --2- 172.21.15.115:0/4036742101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3881095d0 0x7fc38819fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:22.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 >> 172.21.15.115:0/4036742101 conn(0x7fc388078970 msgr2=0x7fc38810d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:42:22.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.175+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 shutdown_connections 2024-07-26T07:42:22.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:42:22.176+0000 7fc38d400640 1 -- 172.21.15.115:0/4036742101 wait complete. 2024-07-26T07:42:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:21 smithi181 ceph-mon[35869]: from='osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T07:42:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:21 smithi181 ceph-mon[35869]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T07:42:22.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:21 smithi097 ceph-mon[32231]: from='osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T07:42:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:21 smithi097 ceph-mon[32231]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T07:42:22.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:21 smithi115 ceph-mon[35364]: from='osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T07:42:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:21 smithi115 ceph-mon[35364]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T07:42:23.006 DEBUG:teuthology.orchestra.run.smithi115:osd.4> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.4.service 2024-07-26T07:42:23.012 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2024-07-26T07:42:23.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T07:42:23.194 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T07:42:23.194 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: from='osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:42:23.194 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T07:42:23.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:23.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:42:23.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:23.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:23.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:23.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:23.462 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: from='osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T07:42:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: from='osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:42:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T07:42:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:42:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:24 smithi181 ceph-mon[35869]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:42:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:24 smithi181 ceph-mon[35869]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T07:42:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:24.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:24 smithi115 ceph-mon[35364]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:42:24.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:24 smithi115 ceph-mon[35364]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T07:42:24.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:24.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:24 smithi097 ceph-mon[32231]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:42:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:24 smithi097 ceph-mon[32231]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T07:42:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:25.271 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:25 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:42:25.271 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:25 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:42:25.271 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:25 smithi181 ceph-mon[35869]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:25.271 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:25.271 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:25 smithi181 ceph-mon[35869]: osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] boot 2024-07-26T07:42:25.271 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:25 smithi181 ceph-mon[35869]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T07:42:25.271 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:25 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:42:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:25 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:42:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:25 smithi115 ceph-mon[35364]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:25 smithi115 ceph-mon[35364]: osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] boot 2024-07-26T07:42:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:25 smithi115 ceph-mon[35364]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T07:42:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:25 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:42:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:25 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:42:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:25 smithi097 ceph-mon[32231]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-07-26T07:42:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:25 smithi097 ceph-mon[32231]: osd.4 [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] boot 2024-07-26T07:42:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:25 smithi097 ceph-mon[32231]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T07:42:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T07:42:25.915 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T07:42:26.112 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T07:42:26.150 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-07-26T07:42:26.150 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-07-26T07:42:26.151 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0311051 s, 337 MB/s 2024-07-26T07:42:26.151 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-07-26T07:42:26.188 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:26 smithi181 ceph-mon[35869]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T07:42:26.189 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:26 smithi181 ceph-mon[35869]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:26 smithi115 ceph-mon[35364]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T07:42:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:26 smithi115 ceph-mon[35364]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:26 smithi097 ceph-mon[32231]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T07:42:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:26 smithi097 ceph-mon[32231]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:27.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-07-26T07:42:27.508 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi115 2024-07-26T07:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 8544M 2024-07-26T07:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:28 smithi097 ceph-mon[32231]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi115 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 8544M 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:28 smithi181 ceph-mon[35869]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi115 2024-07-26T07:42:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:42:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:42:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 8544M 2024-07-26T07:42:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:28 smithi115 ceph-mon[35364]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:29.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 -- 172.21.15.181:0/2600371451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f4106c00 msgr2=0x7f67f410d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:42:29.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 --2- 172.21.15.181:0/2600371451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f4106c00 0x7f67f410d5a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009ca0 tx=0x7f67ec02a950 comp rx=0 tx=0).stop 2024-07-26T07:42:29.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 -- 172.21.15.181:0/2600371451 shutdown_connections 2024-07-26T07:42:29.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 --2- 172.21.15.181:0/2600371451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f4106c00 0x7f67f410d5a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:29.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 --2- 172.21.15.181:0/2600371451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67f4106260 0x7f67f4106670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:29.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 --2- 172.21.15.181:0/2600371451 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67f41051d0 0x7f67f4105640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:29.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 -- 172.21.15.181:0/2600371451 >> 172.21.15.181:0/2600371451 conn(0x7f67f4100b60 msgr2=0x7f67f4102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:42:29.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 -- 172.21.15.181:0/2600371451 shutdown_connections 2024-07-26T07:42:29.692 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.689+0000 7f67f9c00640 1 -- 172.21.15.181:0/2600371451 wait complete. 2024-07-26T07:42:29.692 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.691+0000 7f67f9c00640 1 Processor -- start 2024-07-26T07:42:29.692 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.691+0000 7f67f9c00640 1 -- start start 2024-07-26T07:42:29.694 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.691+0000 7f67f9c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f41051d0 0x7f67f419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:42:29.694 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67f4106260 0x7f67f419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:42:29.694 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f9c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67f4106c00 0x7f67f419cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:42:29.695 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f9c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f67f4110d30 con 0x7f67f4106260 2024-07-26T07:42:29.695 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f9c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f67f4110bb0 con 0x7f67f4106c00 2024-07-26T07:42:29.695 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f9c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f67f4110eb0 con 0x7f67f41051d0 2024-07-26T07:42:29.695 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f41051d0 0x7f67f419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:42:29.695 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f41051d0 0x7f67f419b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:38418/0 (socket says 172.21.15.181:38418) 2024-07-26T07:42:29.695 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f3e00640 1 -- 172.21.15.181:0/1129553058 learned_addr learned my addr 172.21.15.181:0/1129553058 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-07-26T07:42:29.695 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67eb400640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67f4106260 0x7f67f419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:42:29.695 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f9200640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67f4106c00 0x7f67f419cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f3e00640 1 -- 172.21.15.181:0/1129553058 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67f4106c00 msgr2=0x7f67f419cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f3e00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67f4106c00 0x7f67f419cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f3e00640 1 -- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67f4106260 msgr2=0x7f67f419bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f3e00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67f4106260 0x7f67f419bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67f3e00640 1 -- 172.21.15.181:0/1129553058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f419d340 con 0x7f67f41051d0 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.692+0000 7f67eb400640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67f4106260 0x7f67f419bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.693+0000 7f67f3e00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f41051d0 0x7f67f419b910 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f67e400d770 tx=0x7f67e400dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.693+0000 7f67f1600640 1 -- 172.21.15.181:0/1129553058 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4017070 con 0x7f67f41051d0 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.693+0000 7f67f1600640 1 -- 172.21.15.181:0/1129553058 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f67e40054b0 con 0x7f67f41051d0 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.693+0000 7f67f1600640 1 -- 172.21.15.181:0/1129553058 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4004c60 con 0x7f67f41051d0 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.693+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f419d630 con 0x7f67f41051d0 2024-07-26T07:42:29.696 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.693+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f41a3d30 con 0x7f67f41051d0 2024-07-26T07:42:29.697 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.694+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f4105640 con 0x7f67f41051d0 2024-07-26T07:42:29.699 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.699+0000 7f67f1600640 1 -- 172.21.15.181:0/1129553058 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f67e4004e00 con 0x7f67f41051d0 2024-07-26T07:42:29.701 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.700+0000 7f67f1600640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f67d80b3bd0 0x7f67d80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:42:29.701 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.700+0000 7f67f1600640 1 -- 172.21.15.181:0/1129553058 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f67e40e2ad0 con 0x7f67f41051d0 2024-07-26T07:42:29.702 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.701+0000 7f67eb400640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f67d80b3bd0 0x7f67d80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:42:29.703 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.702+0000 7f67eb400640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f67d80b3bd0 0x7f67d80b60e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009b40 tx=0x7f67dc005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:42:29.704 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.703+0000 7f67f1600640 1 -- 172.21.15.181:0/1129553058 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f67e40ab190 con 0x7f67f41051d0 2024-07-26T07:42:29.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:29.908+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f67f4063230 con 0x7f67d80b3bd0 2024-07-26T07:42:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:42:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:42:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:42:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:42:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:30.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:42:30.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:42:30.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:31.241 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:30 smithi115 ceph-mon[35364]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:42:31.241 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:30 smithi115 ceph-mon[35364]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:30 smithi097 ceph-mon[32231]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:42:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:30 smithi097 ceph-mon[32231]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:31.416 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:30 smithi181 ceph-mon[35869]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:42:31.417 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:30 smithi181 ceph-mon[35869]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:32 smithi097 ceph-mon[32231]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:32.361 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:32 smithi181 ceph-mon[35869]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:32 smithi115 ceph-mon[35364]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:33 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/3057010859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]: dispatch 2024-07-26T07:42:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:33 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]: dispatch 2024-07-26T07:42:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:33 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]': finished 2024-07-26T07:42:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:33 smithi115 ceph-mon[35364]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T07:42:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:33 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/3057010859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]: dispatch 2024-07-26T07:42:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:33 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]: dispatch 2024-07-26T07:42:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:33 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]': finished 2024-07-26T07:42:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:33 smithi181 ceph-mon[35869]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T07:42:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:33 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/3057010859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]: dispatch 2024-07-26T07:42:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:33 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]: dispatch 2024-07-26T07:42:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:33 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1272d9a0-8173-47b6-bfe2-017da6dc526d"}]': finished 2024-07-26T07:42:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:33 smithi097 ceph-mon[32231]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T07:42:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:34 smithi115 ceph-mon[35364]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:34 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/1335419424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:42:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:34 smithi181 ceph-mon[35869]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:34 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/1335419424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:42:35.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:34 smithi097 ceph-mon[32231]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:35.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:34 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/1335419424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:42:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:36 smithi115 ceph-mon[35364]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:36 smithi181 ceph-mon[35869]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:36 smithi097 ceph-mon[32231]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:38.355 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:38 smithi181 ceph-mon[35869]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:38.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:38 smithi115 ceph-mon[35364]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:38.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:38 smithi097 ceph-mon[32231]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:40.362 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:40 smithi181 ceph-mon[35869]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:40.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:40 smithi115 ceph-mon[35364]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:40.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:40 smithi097 ceph-mon[32231]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:42 smithi181 ceph-mon[35869]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:42.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:42 smithi115 ceph-mon[35364]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:42.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:42 smithi097 ceph-mon[32231]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:43.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T07:42:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T07:42:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:43.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T07:42:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:44 smithi115 ceph-mon[35364]: Deploying daemon osd.5 on smithi181 2024-07-26T07:42:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:44 smithi115 ceph-mon[35364]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:44.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:44 smithi181 ceph-mon[35869]: Deploying daemon osd.5 on smithi181 2024-07-26T07:42:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:44 smithi181 ceph-mon[35869]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:44.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:44 smithi097 ceph-mon[32231]: Deploying daemon osd.5 on smithi181 2024-07-26T07:42:44.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:44 smithi097 ceph-mon[32231]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:46 smithi181 ceph-mon[35869]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:46.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:46 smithi115 ceph-mon[35364]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:46.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:46 smithi097 ceph-mon[32231]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:48 smithi181 ceph-mon[35869]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:48.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:48 smithi115 ceph-mon[35364]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:48.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:48 smithi097 ceph-mon[32231]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:50 smithi115 ceph-mon[35364]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:50 smithi181 ceph-mon[35869]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:50.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:50 smithi097 ceph-mon[32231]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:52 smithi115 ceph-mon[35364]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:52 smithi181 ceph-mon[35869]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:53.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:52 smithi097 ceph-mon[32231]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:54.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:54 smithi115 ceph-mon[35364]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:54 smithi181 ceph-mon[35869]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:54.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:54 smithi097 ceph-mon[32231]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:56.425 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.425 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.425 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:56 smithi181 ceph-mon[35869]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:56.425 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:56 smithi181 ceph-mon[35869]: from='osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T07:42:56.425 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:56 smithi181 ceph-mon[35869]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T07:42:56.426 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:56.426 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:56.426 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:56 smithi115 ceph-mon[35364]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:56 smithi115 ceph-mon[35364]: from='osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T07:42:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:56 smithi115 ceph-mon[35364]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T07:42:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:56 smithi097 ceph-mon[32231]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:56 smithi097 ceph-mon[32231]: from='osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T07:42:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:56 smithi097 ceph-mon[32231]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T07:42:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:56.848 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.847+0000 7f67f1600640 1 -- 172.21.15.181:0/1129553058 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f67f4063230 con 0x7f67d80b3bd0 2024-07-26T07:42:56.848 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-07-26T07:42:56.852 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f67d80b3bd0 msgr2=0x7f67d80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f67d80b3bd0 0x7f67d80b60e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009b40 tx=0x7f67dc005e50 comp rx=0 tx=0).stop 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f41051d0 msgr2=0x7f67f419b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f41051d0 0x7f67f419b910 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f67e400d770 tx=0x7f67e400dc30 comp rx=0 tx=0).stop 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 shutdown_connections 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f67d80b3bd0 0x7f67d80b60e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67f4106c00 0x7f67f419cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f67f4106260 0x7f67f419bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 --2- 172.21.15.181:0/1129553058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67f41051d0 0x7f67f419b910 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 >> 172.21.15.181:0/1129553058 conn(0x7f67f4100b60 msgr2=0x7f67f4109580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 shutdown_connections 2024-07-26T07:42:56.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:42:56.852+0000 7f67f9c00640 1 -- 172.21.15.181:0/1129553058 wait complete. 2024-07-26T07:42:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:57 smithi181 ceph-mon[35869]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T07:42:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:57 smithi181 ceph-mon[35869]: from='osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:42:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:57 smithi181 ceph-mon[35869]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T07:42:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:57 smithi181 ceph-mon[35869]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:42:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:57 smithi097 ceph-mon[32231]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T07:42:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:57 smithi097 ceph-mon[32231]: from='osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:42:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:57 smithi097 ceph-mon[32231]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T07:42:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:57 smithi097 ceph-mon[32231]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:42:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:57.642 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.5.service 2024-07-26T07:42:57.649 INFO:tasks.cephadm:Deploying osd.6 on smithi097 with /dev/vg_nvme/lv_2... 2024-07-26T07:42:57.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T07:42:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:57 smithi115 ceph-mon[35364]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T07:42:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:57 smithi115 ceph-mon[35364]: from='osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:42:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:57 smithi115 ceph-mon[35364]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T07:42:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:57 smithi115 ceph-mon[35364]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:42:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:42:58.105 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:42:58.440 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:58 smithi181 ceph-mon[35869]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:42:58.441 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:58 smithi181 ceph-mon[35869]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T07:42:58.441 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:58.441 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:58.441 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:58 smithi181 ceph-mon[35869]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:58.441 INFO:journalctl@ceph.osd.5.smithi181.stdout:Jul 26 07:42:58 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-5[47806]: 2024-07-26T07:42:58.317+0000 7fb4d8000640 -1 osd.5 0 waiting for initial osdmap 2024-07-26T07:42:58.442 INFO:journalctl@ceph.osd.5.smithi181.stdout:Jul 26 07:42:58 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-5[47806]: 2024-07-26T07:42:58.351+0000 7fb4ce000640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T07:42:58.572 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:58 smithi097 ceph-mon[32231]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:42:58.572 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:58 smithi097 ceph-mon[32231]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T07:42:58.572 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:58.572 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:58.572 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:58 smithi097 ceph-mon[32231]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:58 smithi115 ceph-mon[35364]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:42:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:58 smithi115 ceph-mon[35364]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T07:42:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:58 smithi115 ceph-mon[35364]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:42:59.296 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:59 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:42:59.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:59 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:42:59.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:59.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:59 smithi097 ceph-mon[32231]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T07:42:59.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:59.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:59 smithi097 ceph-mon[32231]: from='osd.5 ' entity='osd.5' 2024-07-26T07:42:59.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:42:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:59.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:59 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:42:59.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:59 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:42:59.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:59.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:59 smithi181 ceph-mon[35869]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T07:42:59.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:59.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:59 smithi181 ceph-mon[35869]: from='osd.5 ' entity='osd.5' 2024-07-26T07:42:59.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:42:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:59 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:42:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:59 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:42:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:59 smithi115 ceph-mon[35364]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T07:42:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:42:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:59 smithi115 ceph-mon[35364]: from='osd.5 ' entity='osd.5' 2024-07-26T07:42:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:42:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:43:00.493 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T07:43:00.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:00 smithi115 ceph-mon[35364]: osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] boot 2024-07-26T07:43:00.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:00 smithi115 ceph-mon[35364]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T07:43:00.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:43:00.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:00 smithi115 ceph-mon[35364]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:43:00.684 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T07:43:00.685 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:00 smithi097 ceph-mon[32231]: osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] boot 2024-07-26T07:43:00.685 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:00 smithi097 ceph-mon[32231]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T07:43:00.685 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:43:00.685 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:00 smithi097 ceph-mon[32231]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:43:00.725 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-07-26T07:43:00.726 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-07-26T07:43:00.726 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0328945 s, 319 MB/s 2024-07-26T07:43:00.726 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 2024-07-26T07:43:00.726 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-07-26T07:43:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:00 smithi181 ceph-mon[35869]: osd.5 [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] boot 2024-07-26T07:43:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:00 smithi181 ceph-mon[35869]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T07:43:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T07:43:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:00 smithi181 ceph-mon[35869]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-07-26T07:43:01.442 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:01 smithi097 ceph-mon[32231]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T07:43:01.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_2 2024-07-26T07:43:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:01 smithi115 ceph-mon[35364]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T07:43:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:01 smithi181 ceph-mon[35869]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T07:43:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:43:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T07:43:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi181 2024-07-26T07:43:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 8544M 2024-07-26T07:43:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:02 smithi097 ceph-mon[32231]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T07:43:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi181 2024-07-26T07:43:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 8544M 2024-07-26T07:43:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:02 smithi115 ceph-mon[35364]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi181 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 8544M 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:02 smithi181 ceph-mon[35869]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:04.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.224+0000 7fa74e800640 1 -- 172.21.15.97:0/112257201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7481095d0 msgr2=0x7fa748109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:43:04.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.224+0000 7fa74e800640 1 --2- 172.21.15.97:0/112257201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7481095d0 0x7fa748109a40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009900 tx=0x7fa73c02a480 comp rx=0 tx=0).stop 2024-07-26T07:43:04.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.225+0000 7fa74e800640 1 -- 172.21.15.97:0/112257201 shutdown_connections 2024-07-26T07:43:04.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.225+0000 7fa74e800640 1 --2- 172.21.15.97:0/112257201 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa74810b000 0x7fa7481119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:04.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.225+0000 7fa74e800640 1 --2- 172.21.15.97:0/112257201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa74810a660 0x7fa74810aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:04.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.225+0000 7fa74e800640 1 --2- 172.21.15.97:0/112257201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7481095d0 0x7fa748109a40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:04.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.225+0000 7fa74e800640 1 -- 172.21.15.97:0/112257201 >> 172.21.15.97:0/112257201 conn(0x7fa748078970 msgr2=0x7fa74807ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:43:04.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.225+0000 7fa74e800640 1 -- 172.21.15.97:0/112257201 shutdown_connections 2024-07-26T07:43:04.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.225+0000 7fa74e800640 1 -- 172.21.15.97:0/112257201 wait complete. 2024-07-26T07:43:04.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.226+0000 7fa74e800640 1 Processor -- start 2024-07-26T07:43:04.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.226+0000 7fa74e800640 1 -- start start 2024-07-26T07:43:04.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.226+0000 7fa74e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7481095d0 0x7fa74819fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:43:04.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.226+0000 7fa74e800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa74810a660 0x7fa7481a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:43:04.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.226+0000 7fa74e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa74810b000 0x7fa7481a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:43:04.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.226+0000 7fa74e800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa7481150d0 con 0x7fa74810b000 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.226+0000 7fa74e800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa748114f50 con 0x7fa74810a660 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.226+0000 7fa74e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa748115250 con 0x7fa7481095d0 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa74810b000 0x7fa7481a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa74810b000 0x7fa7481a0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43842/0 (socket says 172.21.15.97:43842) 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74de00640 1 -- 172.21.15.97:0/2279526972 learned_addr learned my addr 172.21.15.97:0/2279526972 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74ca00640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa74810a660 0x7fa7481a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74d400640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7481095d0 0x7fa74819fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74de00640 1 -- 172.21.15.97:0/2279526972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7481095d0 msgr2=0x7fa74819fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74de00640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7481095d0 0x7fa74819fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74de00640 1 -- 172.21.15.97:0/2279526972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa74810a660 msgr2=0x7fa7481a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74de00640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa74810a660 0x7fa7481a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:04.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.227+0000 7fa74de00640 1 -- 172.21.15.97:0/2279526972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7481a17c0 con 0x7fa74810b000 2024-07-26T07:43:04.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.228+0000 7fa74de00640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa74810b000 0x7fa7481a0fa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa74000eaa0 tx=0x7fa74000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:43:04.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.228+0000 7fa746000640 1 -- 172.21.15.97:0/2279526972 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74000cdb0 con 0x7fa74810b000 2024-07-26T07:43:04.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.228+0000 7fa746000640 1 -- 172.21.15.97:0/2279526972 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa740005020 con 0x7fa74810b000 2024-07-26T07:43:04.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.228+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7481a1ab0 con 0x7fa74810b000 2024-07-26T07:43:04.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.228+0000 7fa746000640 1 -- 172.21.15.97:0/2279526972 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7400116d0 con 0x7fa74810b000 2024-07-26T07:43:04.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.228+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7481a81b0 con 0x7fa74810b000 2024-07-26T07:43:04.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.230+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7481107b0 con 0x7fa74810b000 2024-07-26T07:43:04.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.232+0000 7fa746000640 1 -- 172.21.15.97:0/2279526972 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fa7400051d0 con 0x7fa74810b000 2024-07-26T07:43:04.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.235+0000 7fa746000640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa7300b3bd0 0x7fa7300b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:43:04.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.235+0000 7fa746000640 1 -- 172.21.15.97:0/2279526972 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4533+0+0 (secure 0 0 0) 0x7fa7400e32e0 con 0x7fa74810b000 2024-07-26T07:43:04.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.235+0000 7fa74d400640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa7300b3bd0 0x7fa7300b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:43:04.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.236+0000 7fa74d400640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa7300b3bd0 0x7fa7300b60e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0057c0 tx=0x7fa73c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:43:04.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.237+0000 7fa746000640 1 -- 172.21.15.97:0/2279526972 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa740016030 con 0x7fa74810b000 2024-07-26T07:43:04.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:04.444+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa748063230 con 0x7fa7300b3bd0 2024-07-26T07:43:04.446 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:04 smithi097 ceph-mon[32231]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-07-26T07:43:04.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:04 smithi115 ceph-mon[35364]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-07-26T07:43:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:04 smithi181 ceph-mon[35869]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-07-26T07:43:05.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:05 smithi097 ceph-mon[32231]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:43:05.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:43:05.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:43:05.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:05 smithi115 ceph-mon[35364]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:43:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:43:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:43:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:05 smithi181 ceph-mon[35869]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:43:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:43:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:43:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:06.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:06 smithi097 ceph-mon[32231]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:43:06.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:06 smithi115 ceph-mon[35364]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:43:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:06 smithi181 ceph-mon[35869]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:43:08.459 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:08 smithi097 ceph-mon[32231]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:43:08.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:08 smithi115 ceph-mon[35364]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:43:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:08 smithi181 ceph-mon[35869]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:09 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2414503309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9567a0ec-9e3d-4b09-afc6-050d6ef59a83"}]: dispatch 2024-07-26T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:09 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2414503309' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9567a0ec-9e3d-4b09-afc6-050d6ef59a83"}]': finished 2024-07-26T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:09 smithi097 ceph-mon[32231]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:09 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4177743299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:43:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:09 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2414503309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9567a0ec-9e3d-4b09-afc6-050d6ef59a83"}]: dispatch 2024-07-26T07:43:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:09 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2414503309' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9567a0ec-9e3d-4b09-afc6-050d6ef59a83"}]': finished 2024-07-26T07:43:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:09 smithi115 ceph-mon[35364]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T07:43:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:09 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4177743299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:43:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:09 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2414503309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9567a0ec-9e3d-4b09-afc6-050d6ef59a83"}]: dispatch 2024-07-26T07:43:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:09 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2414503309' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9567a0ec-9e3d-4b09-afc6-050d6ef59a83"}]': finished 2024-07-26T07:43:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:09 smithi181 ceph-mon[35869]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T07:43:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:09 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4177743299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:43:10.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:10 smithi115 ceph-mon[35364]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T07:43:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:10 smithi181 ceph-mon[35869]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T07:43:10.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:10 smithi097 ceph-mon[32231]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T07:43:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:12 smithi115 ceph-mon[35364]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:43:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:12 smithi181 ceph-mon[35869]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:43:12.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:12 smithi097 ceph-mon[32231]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:43:14.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:14 smithi097 ceph-mon[32231]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:14.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:14 smithi115 ceph-mon[35364]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:14 smithi181 ceph-mon[35869]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:16.328 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:16 smithi097 ceph-mon[32231]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:16.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:16 smithi115 ceph-mon[35364]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:16 smithi181 ceph-mon[35869]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T07:43:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:17.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T07:43:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T07:43:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:18.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:18 smithi115 ceph-mon[35364]: Deploying daemon osd.6 on smithi097 2024-07-26T07:43:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:18 smithi115 ceph-mon[35364]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:18 smithi181 ceph-mon[35869]: Deploying daemon osd.6 on smithi097 2024-07-26T07:43:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:18 smithi181 ceph-mon[35869]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:18 smithi097 ceph-mon[32231]: Deploying daemon osd.6 on smithi097 2024-07-26T07:43:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:18 smithi097 ceph-mon[32231]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:20.384 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:20 smithi097 ceph-mon[32231]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:20.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:20 smithi115 ceph-mon[35364]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:20 smithi181 ceph-mon[35869]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:22.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:22 smithi097 ceph-mon[32231]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:22.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:22 smithi115 ceph-mon[35364]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:22 smithi181 ceph-mon[35869]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:24.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:24 smithi115 ceph-mon[35364]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:24 smithi181 ceph-mon[35869]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:24 smithi097 ceph-mon[32231]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:26.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:43:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:43:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:43:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:27.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:26 smithi115 ceph-mon[35364]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:26 smithi181 ceph-mon[35869]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:26 smithi097 ceph-mon[32231]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:28.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:28 smithi115 ceph-mon[35364]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:28 smithi181 ceph-mon[35869]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:28.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:28 smithi097 ceph-mon[32231]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:30 smithi115 ceph-mon[35364]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T07:43:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:30 smithi115 ceph-mon[35364]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:30 smithi181 ceph-mon[35869]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T07:43:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:30 smithi181 ceph-mon[35869]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:30 smithi097 ceph-mon[32231]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T07:43:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:30 smithi097 ceph-mon[32231]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.888+0000 7fa746000640 1 -- 172.21.15.97:0/2279526972 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa748063230 con 0x7fa7300b3bd0 2024-07-26T07:43:30.889 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 6 on host 'smithi097' 2024-07-26T07:43:30.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa7300b3bd0 msgr2=0x7fa7300b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:43:30.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa7300b3bd0 0x7fa7300b60e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0057c0 tx=0x7fa73c000bc0 comp rx=0 tx=0).stop 2024-07-26T07:43:30.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa74810b000 msgr2=0x7fa7481a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:43:30.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa74810b000 0x7fa7481a0fa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa74000eaa0 tx=0x7fa74000ef60 comp rx=0 tx=0).stop 2024-07-26T07:43:30.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 shutdown_connections 2024-07-26T07:43:30.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa7300b3bd0 0x7fa7300b60e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:30.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa74810b000 0x7fa7481a0fa0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:30.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa74810a660 0x7fa7481a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:30.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 --2- 172.21.15.97:0/2279526972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7481095d0 0x7fa74819fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:30.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 >> 172.21.15.97:0/2279526972 conn(0x7fa748078970 msgr2=0x7fa74810d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:43:30.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.892+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 shutdown_connections 2024-07-26T07:43:30.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:43:30.893+0000 7fa74e800640 1 -- 172.21.15.97:0/2279526972 wait complete. 2024-07-26T07:43:31.621 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:31 smithi097 ceph-mon[32231]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T07:43:31.621 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:31 smithi097 ceph-mon[32231]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T07:43:31.622 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:31 smithi097 ceph-mon[32231]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:43:31.622 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:31.622 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:43:31.622 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:31.622 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:31.677 DEBUG:teuthology.orchestra.run.smithi097:osd.6> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.6.service 2024-07-26T07:43:31.682 INFO:tasks.cephadm:Deploying osd.7 on smithi115 with /dev/vg_nvme/lv_2... 2024-07-26T07:43:31.682 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T07:43:31.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:31 smithi115 ceph-mon[35364]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T07:43:31.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:31 smithi115 ceph-mon[35364]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T07:43:31.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:31 smithi115 ceph-mon[35364]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:43:31.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:43:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:31 smithi181 ceph-mon[35869]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T07:43:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:31 smithi181 ceph-mon[35869]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T07:43:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:31 smithi181 ceph-mon[35869]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:43:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:43:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:32.136 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T07:43:32.490 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:32 smithi097 ceph-mon[32231]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:43:32.490 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:32 smithi097 ceph-mon[32231]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T07:43:32.490 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:32.490 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:32.491 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:32 smithi097 ceph-mon[32231]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:32.491 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jul 26 07:43:32 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-6[63218]: 2024-07-26T07:43:32.351+0000 7f3579a00640 -1 osd.6 0 waiting for initial osdmap 2024-07-26T07:43:32.491 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jul 26 07:43:32 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-6[63218]: 2024-07-26T07:43:32.381+0000 7f356fa00640 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T07:43:32.593 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:32 smithi115 ceph-mon[35364]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:43:32.593 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:32 smithi115 ceph-mon[35364]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T07:43:32.594 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:32.594 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:32.594 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:32 smithi115 ceph-mon[35364]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:32 smithi181 ceph-mon[35869]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:43:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:32 smithi181 ceph-mon[35869]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T07:43:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:32 smithi181 ceph-mon[35869]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:33 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:43:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:33 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:43:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:33 smithi115 ceph-mon[35364]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' 2024-07-26T07:43:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:33.574 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:33 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:43:33.575 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:33 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:43:33.575 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:33.575 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:33 smithi097 ceph-mon[32231]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' 2024-07-26T07:43:33.575 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:33 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:33 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:33 smithi181 ceph-mon[35869]: from='osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113]' entity='osd.6' 2024-07-26T07:43:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:34 smithi115 ceph-mon[35364]: osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] boot 2024-07-26T07:43:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:34 smithi115 ceph-mon[35364]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T07:43:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:34 smithi115 ceph-mon[35364]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:34 smithi181 ceph-mon[35869]: osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] boot 2024-07-26T07:43:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:34 smithi181 ceph-mon[35869]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T07:43:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:34 smithi181 ceph-mon[35869]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:34.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:34 smithi097 ceph-mon[32231]: osd.6 [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] boot 2024-07-26T07:43:34.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:34 smithi097 ceph-mon[32231]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T07:43:34.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T07:43:34.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:34 smithi097 ceph-mon[32231]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-07-26T07:43:34.796 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T07:43:34.972 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T07:43:35.021 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-07-26T07:43:35.021 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-07-26T07:43:35.021 INFO:teuthology.orchestra.run.smithi115.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0340861 s, 308 MB/s 2024-07-26T07:43:35.022 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-07-26T07:43:35.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:35 smithi115 ceph-mon[35364]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T07:43:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:35 smithi181 ceph-mon[35869]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T07:43:35.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:35 smithi097 ceph-mon[32231]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T07:43:35.899 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi115:vg_nvme/lv_2 2024-07-26T07:43:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T07:43:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi097 2024-07-26T07:43:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 5696M 2024-07-26T07:43:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi097 2024-07-26T07:43:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 5696M 2024-07-26T07:43:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi097 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 5696M 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:43:38.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:38 smithi115 ceph-mon[35364]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:38 smithi181 ceph-mon[35869]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:38 smithi097 ceph-mon[32231]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.744+0000 7f5ec2200640 1 -- 172.21.15.115:0/165943028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc106de0 msgr2=0x7f5ebc10d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.744+0000 7f5ec2200640 1 --2- 172.21.15.115:0/165943028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc106de0 0x7f5ebc10d780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009ca0 tx=0x7f5eb402a950 comp rx=0 tx=0).stop 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.744+0000 7f5ec2200640 1 -- 172.21.15.115:0/165943028 shutdown_connections 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.744+0000 7f5ec2200640 1 --2- 172.21.15.115:0/165943028 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc106de0 0x7f5ebc10d780 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.745+0000 7f5ec2200640 1 --2- 172.21.15.115:0/165943028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ebc106440 0x7f5ebc106850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.745+0000 7f5ec2200640 1 --2- 172.21.15.115:0/165943028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ebc1053b0 0x7f5ebc105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.745+0000 7f5ec2200640 1 -- 172.21.15.115:0/165943028 >> 172.21.15.115:0/165943028 conn(0x7f5ebc100d20 msgr2=0x7f5ebc1031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.745+0000 7f5ec2200640 1 -- 172.21.15.115:0/165943028 shutdown_connections 2024-07-26T07:43:38.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.745+0000 7f5ec2200640 1 -- 172.21.15.115:0/165943028 wait complete. 2024-07-26T07:43:38.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ec2200640 1 Processor -- start 2024-07-26T07:43:38.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ec2200640 1 -- start start 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ec2200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc1053b0 0x7f5ebc19ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ec2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ebc106440 0x7f5ebc19c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ec2200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ebc106de0 0x7f5ebc19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ec2200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5ebc110ec0 con 0x7f5ebc106de0 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ec2200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5ebc110d40 con 0x7f5ebc1053b0 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ec2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5ebc111040 con 0x7f5ebc106440 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.746+0000 7f5ebb400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc1053b0 0x7f5ebc19ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebb400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc1053b0 0x7f5ebc19ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:55840/0 (socket says 172.21.15.115:55840) 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebb400640 1 -- 172.21.15.115:0/3397117708 learned_addr learned my addr 172.21.15.115:0/3397117708 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebaa00640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ebc106440 0x7f5ebc19c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:43:38.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebbe00640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ebc106de0 0x7f5ebc19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebb400640 1 -- 172.21.15.115:0/3397117708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ebc106440 msgr2=0x7f5ebc19c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebb400640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ebc106440 0x7f5ebc19c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebb400640 1 -- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ebc106de0 msgr2=0x7f5ebc19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebb400640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ebc106de0 0x7f5ebc19cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebb400640 1 -- 172.21.15.115:0/3397117708 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ebc19d4c0 con 0x7f5ebc1053b0 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebbe00640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ebc106de0 0x7f5ebc19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.747+0000 7f5ebb400640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc1053b0 0x7f5ebc19ba90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400d950 tx=0x7f5ea400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.748+0000 7f5eb3e00640 1 -- 172.21.15.115:0/3397117708 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4017070 con 0x7f5ebc1053b0 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.748+0000 7f5eb3e00640 1 -- 172.21.15.115:0/3397117708 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ea4009c30 con 0x7f5ebc1053b0 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.748+0000 7f5eb3e00640 1 -- 172.21.15.115:0/3397117708 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea400f620 con 0x7f5ebc1053b0 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.748+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ebc19d7b0 con 0x7f5ebc1053b0 2024-07-26T07:43:38.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.748+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ebc1a3eb0 con 0x7f5ebc1053b0 2024-07-26T07:43:38.752 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.749+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ebc10c570 con 0x7f5ebc1053b0 2024-07-26T07:43:38.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.752+0000 7f5eb3e00640 1 -- 172.21.15.115:0/3397117708 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f5ea4010460 con 0x7f5ebc1053b0 2024-07-26T07:43:38.755 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.754+0000 7f5eb3e00640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5ea00b3c50 0x7f5ea00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:43:38.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.754+0000 7f5eb3e00640 1 -- 172.21.15.115:0/3397117708 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 4965+0+0 (secure 0 0 0) 0x7f5ea40e2bf0 con 0x7f5ebc1053b0 2024-07-26T07:43:38.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.755+0000 7f5ebaa00640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5ea00b3c50 0x7f5ea00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:43:38.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.755+0000 7f5eb3e00640 1 -- 172.21.15.115:0/3397117708 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5ea40aaf30 con 0x7f5ebc1053b0 2024-07-26T07:43:38.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.756+0000 7f5ebaa00640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5ea00b3c50 0x7f5ea00b6160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5eac005fd0 tx=0x7f5eac005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:43:38.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:43:38.965+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5ebc063230 con 0x7f5ea00b3c50 2024-07-26T07:43:39.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:39 smithi115 ceph-mon[35364]: from='client.34385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:43:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:43:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:43:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:39 smithi181 ceph-mon[35869]: from='client.34385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:43:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:43:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:43:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:39 smithi097 ceph-mon[32231]: from='client.34385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:43:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:43:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:43:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:40.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:40 smithi115 ceph-mon[35364]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:40 smithi181 ceph-mon[35869]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:40.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:40 smithi097 ceph-mon[32231]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:42.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:42 smithi115 ceph-mon[35364]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:42 smithi181 ceph-mon[35869]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:42.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:42 smithi097 ceph-mon[32231]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:43 smithi115 ceph-mon[35364]: from='client.? 172.21.15.115:0/3329104955' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]: dispatch 2024-07-26T07:43:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:43 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]: dispatch 2024-07-26T07:43:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:43 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]': finished 2024-07-26T07:43:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:43 smithi115 ceph-mon[35364]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T07:43:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:43:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:43 smithi181 ceph-mon[35869]: from='client.? 172.21.15.115:0/3329104955' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]: dispatch 2024-07-26T07:43:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:43 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]: dispatch 2024-07-26T07:43:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:43 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]': finished 2024-07-26T07:43:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:43 smithi181 ceph-mon[35869]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T07:43:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:43:43.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:43 smithi097 ceph-mon[32231]: from='client.? 172.21.15.115:0/3329104955' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]: dispatch 2024-07-26T07:43:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:43 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]: dispatch 2024-07-26T07:43:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:43 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "256172cc-f069-4599-bd65-14deca123b42"}]': finished 2024-07-26T07:43:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:43 smithi097 ceph-mon[32231]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T07:43:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:43:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:44 smithi115 ceph-mon[35364]: from='client.? 172.21.15.115:0/1248445400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:43:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:44 smithi115 ceph-mon[35364]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:44 smithi181 ceph-mon[35869]: from='client.? 172.21.15.115:0/1248445400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:43:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:44 smithi181 ceph-mon[35869]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:44.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:44 smithi097 ceph-mon[32231]: from='client.? 172.21.15.115:0/1248445400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:43:44.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:44 smithi097 ceph-mon[32231]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:46.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:46 smithi115 ceph-mon[35364]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:46 smithi181 ceph-mon[35869]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:46.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:46 smithi097 ceph-mon[32231]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:48.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:48 smithi115 ceph-mon[35364]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:48 smithi181 ceph-mon[35869]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:48.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:48 smithi097 ceph-mon[32231]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:50.454 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:50 smithi115 ceph-mon[35364]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:50 smithi181 ceph-mon[35869]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:50.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:50 smithi097 ceph-mon[32231]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:52 smithi115 ceph-mon[35364]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:52 smithi181 ceph-mon[35869]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:52.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:52 smithi097 ceph-mon[32231]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:53.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T07:43:53.254 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:53.254 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:53 smithi115 ceph-mon[35364]: Deploying daemon osd.7 on smithi115 2024-07-26T07:43:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T07:43:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:53 smithi181 ceph-mon[35869]: Deploying daemon osd.7 on smithi115 2024-07-26T07:43:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T07:43:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:53 smithi097 ceph-mon[32231]: Deploying daemon osd.7 on smithi115 2024-07-26T07:43:54.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:54 smithi115 ceph-mon[35364]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:54 smithi181 ceph-mon[35869]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:54.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:54 smithi097 ceph-mon[32231]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:56.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:56 smithi115 ceph-mon[35364]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:56 smithi181 ceph-mon[35869]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:56.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:56 smithi097 ceph-mon[32231]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:43:58 smithi115 ceph-mon[35364]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:43:58 smithi181 ceph-mon[35869]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:43:58.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:43:58 smithi097 ceph-mon[32231]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:00 smithi181 ceph-mon[35869]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:00.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:00 smithi115 ceph-mon[35364]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:00 smithi097 ceph-mon[32231]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:01.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:01.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:01.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:01.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:02 smithi097 ceph-mon[32231]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:02.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:02 smithi115 ceph-mon[35364]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:02 smithi181 ceph-mon[35869]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:04.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:04 smithi115 ceph-mon[35364]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:04 smithi181 ceph-mon[35869]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:04 smithi097 ceph-mon[32231]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:05.673 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.673 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.673 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:05.673 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:05.673 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.673 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:05 smithi115 ceph-mon[35364]: from='osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T07:44:05.673 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:05 smithi115 ceph-mon[35364]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T07:44:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:05 smithi181 ceph-mon[35869]: from='osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T07:44:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:05 smithi181 ceph-mon[35869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T07:44:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:05 smithi097 ceph-mon[32231]: from='osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T07:44:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:05 smithi097 ceph-mon[32231]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T07:44:06.114 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.113+0000 7f5eb3e00640 1 -- 172.21.15.115:0/3397117708 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5ebc063230 con 0x7f5ea00b3c50 2024-07-26T07:44:06.114 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 7 on host 'smithi115' 2024-07-26T07:44:06.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.116+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5ea00b3c50 msgr2=0x7f5ea00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:06.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.116+0000 7f5ec2200640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5ea00b3c50 0x7f5ea00b6160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5eac005fd0 tx=0x7f5eac005850 comp rx=0 tx=0).stop 2024-07-26T07:44:06.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.116+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc1053b0 msgr2=0x7f5ebc19ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:06.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.116+0000 7f5ec2200640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc1053b0 0x7f5ebc19ba90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400d950 tx=0x7f5ea400de10 comp rx=0 tx=0).stop 2024-07-26T07:44:06.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.117+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 shutdown_connections 2024-07-26T07:44:06.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.117+0000 7f5ec2200640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5ea00b3c50 0x7f5ea00b6160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:06.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.117+0000 7f5ec2200640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5ebc106de0 0x7f5ebc19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:06.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.117+0000 7f5ec2200640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ebc106440 0x7f5ebc19c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:06.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.117+0000 7f5ec2200640 1 --2- 172.21.15.115:0/3397117708 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5ebc1053b0 0x7f5ebc19ba90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:06.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.117+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 >> 172.21.15.115:0/3397117708 conn(0x7f5ebc100d20 msgr2=0x7f5ebc109740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:44:06.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.117+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 shutdown_connections 2024-07-26T07:44:06.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:44:06.117+0000 7f5ec2200640 1 -- 172.21.15.115:0/3397117708 wait complete. 2024-07-26T07:44:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T07:44:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T07:44:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: from='osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:44:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:44:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T07:44:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T07:44:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: from='osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:44:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:44:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: from='osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:06.867 DEBUG:teuthology.orchestra.run.smithi115:osd.7> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.7.service 2024-07-26T07:44:06.872 INFO:tasks.cephadm:Deploying osd.8 on smithi181 with /dev/vg_nvme/lv_2... 2024-07-26T07:44:06.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T07:44:07.326 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:44:07.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:07 smithi115 ceph-mon[35364]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:44:07.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:07 smithi115 ceph-mon[35364]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T07:44:07.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:07.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:07.667 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:07 smithi181 ceph-mon[35869]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:44:07.667 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:07 smithi181 ceph-mon[35869]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T07:44:07.667 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:07.668 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:07 smithi097 ceph-mon[32231]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:44:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:07 smithi097 ceph-mon[32231]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T07:44:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:08.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:08 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:44:08.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:08 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:44:08.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:08.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:08 smithi115 ceph-mon[35364]: osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] boot 2024-07-26T07:44:08.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:08 smithi115 ceph-mon[35364]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T07:44:08.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:08.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:08 smithi115 ceph-mon[35364]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:08 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:44:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:08 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:44:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:08 smithi181 ceph-mon[35869]: osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] boot 2024-07-26T07:44:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:08 smithi181 ceph-mon[35869]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T07:44:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:08 smithi181 ceph-mon[35869]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:08 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:44:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:08 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:44:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:08 smithi097 ceph-mon[32231]: osd.7 [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] boot 2024-07-26T07:44:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:08 smithi097 ceph-mon[32231]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T07:44:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T07:44:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:08 smithi097 ceph-mon[32231]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-07-26T07:44:09.434 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:09 smithi181 ceph-mon[35869]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T07:44:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:09 smithi115 ceph-mon[35364]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T07:44:09.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:09 smithi097 ceph-mon[32231]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T07:44:09.953 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T07:44:10.164 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T07:44:10.203 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-07-26T07:44:10.203 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-07-26T07:44:10.203 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.031143 s, 337 MB/s 2024-07-26T07:44:10.203 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-07-26T07:44:10.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:10 smithi181 ceph-mon[35869]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T07:44:10.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:10 smithi181 ceph-mon[35869]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:10.635 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:10 smithi115 ceph-mon[35364]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T07:44:10.635 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:10 smithi115 ceph-mon[35364]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:10 smithi097 ceph-mon[32231]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T07:44:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:10 smithi097 ceph-mon[32231]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:11.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-07-26T07:44:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:44:12.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi115 2024-07-26T07:44:12.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 5696M 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:12 smithi181 ceph-mon[35869]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi115 2024-07-26T07:44:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 5696M 2024-07-26T07:44:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:12 smithi097 ceph-mon[32231]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi115 2024-07-26T07:44:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 5696M 2024-07-26T07:44:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:12 smithi115 ceph-mon[35364]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:13.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.619+0000 7fb09e200640 1 -- 172.21.15.181:0/1394322547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098103180 msgr2=0x7fb0981035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:13.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.619+0000 7fb09e200640 1 --2- 172.21.15.181:0/1394322547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098103180 0x7fb0981035f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb088009ca0 tx=0x7fb08802a950 comp rx=0 tx=0).stop 2024-07-26T07:44:13.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.620+0000 7fb09e200640 1 -- 172.21.15.181:0/1394322547 shutdown_connections 2024-07-26T07:44:13.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.620+0000 7fb09e200640 1 --2- 172.21.15.181:0/1394322547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb098104bb0 0x7fb09810b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:13.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.620+0000 7fb09e200640 1 --2- 172.21.15.181:0/1394322547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb098104210 0x7fb098104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:13.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.620+0000 7fb09e200640 1 --2- 172.21.15.181:0/1394322547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098103180 0x7fb0981035f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:13.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.620+0000 7fb09e200640 1 -- 172.21.15.181:0/1394322547 >> 172.21.15.181:0/1394322547 conn(0x7fb0980feaf0 msgr2=0x7fb098100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:44:13.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.620+0000 7fb09e200640 1 -- 172.21.15.181:0/1394322547 shutdown_connections 2024-07-26T07:44:13.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.620+0000 7fb09e200640 1 -- 172.21.15.181:0/1394322547 wait complete. 2024-07-26T07:44:13.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.621+0000 7fb09e200640 1 Processor -- start 2024-07-26T07:44:13.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb09e200640 1 -- start start 2024-07-26T07:44:13.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb09e200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb098103180 0x7fb0981998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:44:13.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb09e200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb098104210 0x7fb098199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:44:13.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb09e200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098104bb0 0x7fb09819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:44:13.623 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb09e200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb09810ece0 con 0x7fb098103180 2024-07-26T07:44:13.624 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb09e200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb09810eb60 con 0x7fb098104210 2024-07-26T07:44:13.624 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb09e200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb09810ee60 con 0x7fb098104bb0 2024-07-26T07:44:13.624 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb097e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098104bb0 0x7fb09819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:44:13.624 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb097e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098104bb0 0x7fb09819ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34764/0 (socket says 172.21.15.181:34764) 2024-07-26T07:44:13.624 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.622+0000 7fb097e00640 1 -- 172.21.15.181:0/2368499224 learned_addr learned my addr 172.21.15.181:0/2368499224 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb097400640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb098103180 0x7fb0981998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb08ea00640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb098104210 0x7fb098199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb097e00640 1 -- 172.21.15.181:0/2368499224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb098104210 msgr2=0x7fb098199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb097e00640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb098104210 0x7fb098199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb097e00640 1 -- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb098103180 msgr2=0x7fb0981998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb097e00640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb098103180 0x7fb0981998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb097e00640 1 -- 172.21.15.181:0/2368499224 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09819b320 con 0x7fb098104bb0 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb097e00640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098104bb0 0x7fb09819ab00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb09000d750 tx=0x7fb09000dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb094c00640 1 -- 172.21.15.181:0/2368499224 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090017070 con 0x7fb098104bb0 2024-07-26T07:44:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09819b610 con 0x7fb098104bb0 2024-07-26T07:44:13.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb094c00640 1 -- 172.21.15.181:0/2368499224 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0900054b0 con 0x7fb098104bb0 2024-07-26T07:44:13.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.623+0000 7fb094c00640 1 -- 172.21.15.181:0/2368499224 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090004b90 con 0x7fb098104bb0 2024-07-26T07:44:13.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.624+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0981a1d10 con 0x7fb098104bb0 2024-07-26T07:44:13.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.625+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0981035f0 con 0x7fb098104bb0 2024-07-26T07:44:13.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.628+0000 7fb094c00640 1 -- 172.21.15.181:0/2368499224 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fb090004d30 con 0x7fb098104bb0 2024-07-26T07:44:13.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.629+0000 7fb094c00640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb07c0b3bd0 0x7fb07c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:44:13.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.629+0000 7fb094c00640 1 -- 172.21.15.181:0/2368499224 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5397+0+0 (secure 0 0 0) 0x7fb0900e2ff0 con 0x7fb098104bb0 2024-07-26T07:44:13.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.630+0000 7fb097400640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb07c0b3bd0 0x7fb07c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:44:13.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.631+0000 7fb097400640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb07c0b3bd0 0x7fb07c0b60e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb08802f800 tx=0x7fb08802f880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:44:13.632 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.631+0000 7fb094c00640 1 -- 172.21.15.181:0/2368499224 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb0900ab160 con 0x7fb098104bb0 2024-07-26T07:44:13.847 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:13.846+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb098063230 con 0x7fb07c0b3bd0 2024-07-26T07:44:14.882 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:14 smithi181 ceph-mon[35869]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:44:14.883 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:44:14.883 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:44:14.883 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:14.883 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:14 smithi181 ceph-mon[35869]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:44:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:14 smithi115 ceph-mon[35364]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:44:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:44:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:44:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:14 smithi115 ceph-mon[35364]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:44:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:14 smithi097 ceph-mon[32231]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:44:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:44:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:44:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:14 smithi097 ceph-mon[32231]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:44:16.429 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:16 smithi181 ceph-mon[35869]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:44:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:16 smithi115 ceph-mon[35364]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:44:16.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:16 smithi097 ceph-mon[32231]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:44:17.973 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:17 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/3085844744' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]: dispatch 2024-07-26T07:44:17.973 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:17 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]: dispatch 2024-07-26T07:44:17.973 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:17 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]': finished 2024-07-26T07:44:17.973 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:17 smithi181 ceph-mon[35869]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T07:44:17.973 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:17 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/3085844744' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]: dispatch 2024-07-26T07:44:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:17 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]: dispatch 2024-07-26T07:44:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:17 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]': finished 2024-07-26T07:44:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:17 smithi115 ceph-mon[35364]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T07:44:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:17 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/3085844744' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]: dispatch 2024-07-26T07:44:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:17 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]: dispatch 2024-07-26T07:44:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:17 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63b664e5-d348-4a9a-afb9-fe1fd1892a27"}]': finished 2024-07-26T07:44:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:17 smithi097 ceph-mon[32231]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T07:44:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:18 smithi115 ceph-mon[35364]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T07:44:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:18 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/375517906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:44:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:18 smithi181 ceph-mon[35869]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T07:44:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:18 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/375517906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:44:19.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:18 smithi097 ceph-mon[32231]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-07-26T07:44:19.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:18 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/375517906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:44:20.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:20 smithi115 ceph-mon[35364]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:20 smithi181 ceph-mon[35869]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:20.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:20 smithi097 ceph-mon[32231]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:22.469 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:22 smithi181 ceph-mon[35869]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:22.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:22 smithi115 ceph-mon[35364]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:22.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:22 smithi097 ceph-mon[32231]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:24.410 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:24 smithi181 ceph-mon[35869]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:24 smithi115 ceph-mon[35364]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:24.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:24 smithi097 ceph-mon[32231]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:26 smithi181 ceph-mon[35869]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:26 smithi115 ceph-mon[35364]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:26.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:26 smithi097 ceph-mon[32231]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T07:44:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T07:44:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:27.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T07:44:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:28 smithi115 ceph-mon[35364]: Deploying daemon osd.8 on smithi181 2024-07-26T07:44:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:28 smithi115 ceph-mon[35364]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:28 smithi181 ceph-mon[35869]: Deploying daemon osd.8 on smithi181 2024-07-26T07:44:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:28 smithi181 ceph-mon[35869]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:28.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:28 smithi097 ceph-mon[32231]: Deploying daemon osd.8 on smithi181 2024-07-26T07:44:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:28 smithi097 ceph-mon[32231]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:30 smithi115 ceph-mon[35364]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:30 smithi181 ceph-mon[35869]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:30.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:30 smithi097 ceph-mon[32231]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:32 smithi115 ceph-mon[35364]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:32 smithi097 ceph-mon[32231]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:32.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:32 smithi181 ceph-mon[35869]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:34.387 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:34 smithi181 ceph-mon[35869]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:34 smithi115 ceph-mon[35364]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:34.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:34 smithi097 ceph-mon[32231]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:36.074 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:36.074 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:36.074 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:36.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:36 smithi097 ceph-mon[32231]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:36 smithi115 ceph-mon[35364]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:36 smithi181 ceph-mon[35869]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:38 smithi115 ceph-mon[35364]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:38 smithi181 ceph-mon[35869]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:38 smithi097 ceph-mon[32231]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:40.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:40.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:40.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:40 smithi181 ceph-mon[35869]: from='osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T07:44:40.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:40 smithi181 ceph-mon[35869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T07:44:40.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:40 smithi181 ceph-mon[35869]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:40 smithi115 ceph-mon[35364]: from='osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T07:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:40 smithi115 ceph-mon[35364]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T07:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:40 smithi115 ceph-mon[35364]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:40 smithi097 ceph-mon[32231]: from='osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T07:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:40 smithi097 ceph-mon[32231]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T07:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:40 smithi097 ceph-mon[32231]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:40.898 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 8 on host 'smithi181' 2024-07-26T07:44:40.899 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.896+0000 7fb094c00640 1 -- 172.21.15.181:0/2368499224 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb098063230 con 0x7fb07c0b3bd0 2024-07-26T07:44:40.901 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb07c0b3bd0 msgr2=0x7fb07c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:40.901 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb07c0b3bd0 0x7fb07c0b60e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb08802f800 tx=0x7fb08802f880 comp rx=0 tx=0).stop 2024-07-26T07:44:40.901 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098104bb0 msgr2=0x7fb09819ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:40.901 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098104bb0 0x7fb09819ab00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb09000d750 tx=0x7fb09000dc10 comp rx=0 tx=0).stop 2024-07-26T07:44:40.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 shutdown_connections 2024-07-26T07:44:40.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb07c0b3bd0 0x7fb07c0b60e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:40.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb098104bb0 0x7fb09819ab00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:40.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb098104210 0x7fb098199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:40.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 --2- 172.21.15.181:0/2368499224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb098103180 0x7fb0981998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:40.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 >> 172.21.15.181:0/2368499224 conn(0x7fb0980feaf0 msgr2=0x7fb098107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:44:40.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 shutdown_connections 2024-07-26T07:44:40.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:44:40.901+0000 7fb09e200640 1 -- 172.21.15.181:0/2368499224 wait complete. 2024-07-26T07:44:41.640 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:41 smithi181 ceph-mon[35869]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T07:44:41.641 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:41 smithi181 ceph-mon[35869]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T07:44:41.641 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:41.641 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:41 smithi181 ceph-mon[35869]: from='osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:44:41.641 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:41 smithi181 ceph-mon[35869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:44:41.641 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:41.641 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:41.641 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:41.702 DEBUG:teuthology.orchestra.run.smithi181:osd.8> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.8.service 2024-07-26T07:44:41.707 INFO:tasks.cephadm:Deploying osd.9 on smithi097 with /dev/vg_nvme/lv_1... 2024-07-26T07:44:41.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:41 smithi115 ceph-mon[35364]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:41 smithi115 ceph-mon[35364]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:41 smithi115 ceph-mon[35364]: from='osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:41 smithi115 ceph-mon[35364]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:41 smithi097 ceph-mon[32231]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T07:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:41 smithi097 ceph-mon[32231]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T07:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:41 smithi097 ceph-mon[32231]: from='osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:41 smithi097 ceph-mon[32231]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:42.164 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:44:42.450 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:42 smithi181 ceph-mon[35869]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:44:42.453 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:42 smithi181 ceph-mon[35869]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T07:44:42.453 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:42.454 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:42.454 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:42 smithi181 ceph-mon[35869]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:42.454 INFO:journalctl@ceph.osd.8.smithi181.stdout:Jul 26 07:44:42 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-8[54409]: 2024-07-26T07:44:42.418+0000 7f4e24800640 -1 osd.8 0 waiting for initial osdmap 2024-07-26T07:44:42.454 INFO:journalctl@ceph.osd.8.smithi181.stdout:Jul 26 07:44:42 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-8[54409]: 2024-07-26T07:44:42.449+0000 7f4e1a800640 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T07:44:42.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:42 smithi097 ceph-mon[32231]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:44:42.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:42 smithi097 ceph-mon[32231]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T07:44:42.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:42.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:42.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:42 smithi097 ceph-mon[32231]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:42 smithi115 ceph-mon[35364]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:44:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:42 smithi115 ceph-mon[35364]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T07:44:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:42 smithi115 ceph-mon[35364]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:43 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:44:43.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:43 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:44:43.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:43.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:43 smithi181 ceph-mon[35869]: osdmap e55: 9 total, 8 up, 9 in 2024-07-26T07:44:43.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:43.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:43 smithi181 ceph-mon[35869]: from='osd.8 ' entity='osd.8' 2024-07-26T07:44:43.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:43.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:43 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:44:43.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:43 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:44:43.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:43.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:43 smithi097 ceph-mon[32231]: osdmap e55: 9 total, 8 up, 9 in 2024-07-26T07:44:43.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:43.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:43 smithi097 ceph-mon[32231]: from='osd.8 ' entity='osd.8' 2024-07-26T07:44:43.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:43 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:44:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:43 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:44:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:43 smithi115 ceph-mon[35364]: osdmap e55: 9 total, 8 up, 9 in 2024-07-26T07:44:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:43 smithi115 ceph-mon[35364]: from='osd.8 ' entity='osd.8' 2024-07-26T07:44:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:44 smithi097 ceph-mon[32231]: osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] boot 2024-07-26T07:44:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:44 smithi097 ceph-mon[32231]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T07:44:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:44 smithi097 ceph-mon[32231]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T07:44:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:44 smithi097 ceph-mon[32231]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:44 smithi115 ceph-mon[35364]: osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] boot 2024-07-26T07:44:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:44 smithi115 ceph-mon[35364]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T07:44:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:44 smithi115 ceph-mon[35364]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T07:44:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:44 smithi115 ceph-mon[35364]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:44 smithi181 ceph-mon[35869]: osd.8 [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] boot 2024-07-26T07:44:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:44 smithi181 ceph-mon[35869]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T07:44:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T07:44:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:44 smithi181 ceph-mon[35869]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T07:44:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:44 smithi181 ceph-mon[35869]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-07-26T07:44:44.741 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T07:44:44.916 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T07:44:44.957 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-07-26T07:44:44.957 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-07-26T07:44:44.957 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.031646 s, 331 MB/s 2024-07-26T07:44:44.958 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 2024-07-26T07:44:44.958 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-07-26T07:44:45.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_1 2024-07-26T07:44:45.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:45 smithi115 ceph-mon[35364]: osdmap e58: 9 total, 9 up, 9 in 2024-07-26T07:44:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:45 smithi181 ceph-mon[35869]: osdmap e58: 9 total, 9 up, 9 in 2024-07-26T07:44:46.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:45 smithi097 ceph-mon[32231]: osdmap e58: 9 total, 9 up, 9 in 2024-07-26T07:44:46.212 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:44:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi181 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 5696M 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:47.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:46 smithi097 ceph-mon[32231]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 284 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:44:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi181 2024-07-26T07:44:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 5696M 2024-07-26T07:44:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:46 smithi115 ceph-mon[35364]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 284 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:44:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi181 2024-07-26T07:44:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:44:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 5696M 2024-07-26T07:44:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:44:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:46 smithi181 ceph-mon[35869]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 284 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:44:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.332+0000 7f5845200640 1 -- 172.21.15.97:0/1863326781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010a660 msgr2=0x7f584010aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.332+0000 7f5845200640 1 --2- 172.21.15.97:0/1863326781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010a660 0x7f584010aa70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f582c009900 tx=0x7f582c02a480 comp rx=0 tx=0).stop 2024-07-26T07:44:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.332+0000 7f5845200640 1 -- 172.21.15.97:0/1863326781 shutdown_connections 2024-07-26T07:44:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.332+0000 7f5845200640 1 --2- 172.21.15.97:0/1863326781 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f584010b000 0x7f58401119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.332+0000 7f5845200640 1 --2- 172.21.15.97:0/1863326781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010a660 0x7f584010aa70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.332+0000 7f5845200640 1 --2- 172.21.15.97:0/1863326781 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58401095d0 0x7f5840109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.332+0000 7f5845200640 1 -- 172.21.15.97:0/1863326781 >> 172.21.15.97:0/1863326781 conn(0x7f5840078970 msgr2=0x7f584007ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:44:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.333+0000 7f5845200640 1 -- 172.21.15.97:0/1863326781 shutdown_connections 2024-07-26T07:44:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.333+0000 7f5845200640 1 -- 172.21.15.97:0/1863326781 wait complete. 2024-07-26T07:44:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.334+0000 7f5845200640 1 Processor -- start 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.334+0000 7f5845200640 1 -- start start 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.334+0000 7f5845200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58401095d0 0x7f584019fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.334+0000 7f5845200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f584010a660 0x7f58401a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.334+0000 7f5845200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010b000 0x7f58401a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.334+0000 7f5845200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f58401150c0 con 0x7f584010b000 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.334+0000 7f5845200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5840114f40 con 0x7f58401095d0 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.334+0000 7f5845200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5840115240 con 0x7f584010a660 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010b000 0x7f58401a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:44:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010b000 0x7f58401a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54612/0 (socket says 172.21.15.97:54612) 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583fe00640 1 -- 172.21.15.97:0/335191909 learned_addr learned my addr 172.21.15.97:0/335191909 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583f400640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58401095d0 0x7f584019fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583ea00640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f584010a660 0x7f58401a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583fe00640 1 -- 172.21.15.97:0/335191909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f584010a660 msgr2=0x7f58401a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583fe00640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f584010a660 0x7f58401a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583fe00640 1 -- 172.21.15.97:0/335191909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58401095d0 msgr2=0x7f584019fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583fe00640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58401095d0 0x7f584019fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.335+0000 7f583fe00640 1 -- 172.21.15.97:0/335191909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58401a17a0 con 0x7f584010b000 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.336+0000 7f583fe00640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010b000 0x7f58401a1010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f583800cad0 tx=0x7f583800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.336+0000 7f5837e00640 1 -- 172.21.15.97:0/335191909 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583800ad60 con 0x7f584010b000 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.336+0000 7f5837e00640 1 -- 172.21.15.97:0/335191909 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5838005020 con 0x7f584010b000 2024-07-26T07:44:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.336+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58401a1a90 con 0x7f584010b000 2024-07-26T07:44:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.336+0000 7f5837e00640 1 -- 172.21.15.97:0/335191909 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583800f6b0 con 0x7f584010b000 2024-07-26T07:44:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.336+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58401a8050 con 0x7f584010b000 2024-07-26T07:44:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.337+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58401107b0 con 0x7f584010b000 2024-07-26T07:44:48.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.340+0000 7f5837e00640 1 -- 172.21.15.97:0/335191909 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f58380051d0 con 0x7f584010b000 2024-07-26T07:44:48.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.341+0000 7f5837e00640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58240b3bd0 0x7f58240b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:44:48.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.342+0000 7f5837e00640 1 -- 172.21.15.97:0/335191909 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5829+0+0 (secure 0 0 0) 0x7f58380e3690 con 0x7f584010b000 2024-07-26T07:44:48.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.342+0000 7f583f400640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58240b3bd0 0x7f58240b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:44:48.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.343+0000 7f583f400640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58240b3bd0 0x7f58240b60e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f583000abe0 tx=0x7f583000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:44:48.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.343+0000 7f5837e00640 1 -- 172.21.15.97:0/335191909 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5838014030 con 0x7f584010b000 2024-07-26T07:44:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:48 smithi115 ceph-mon[35364]: pgmap v273: 1 pgs: 1 remapped+peering; 577 KiB data, 284 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:44:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:48 smithi181 ceph-mon[35869]: pgmap v273: 1 pgs: 1 remapped+peering; 577 KiB data, 284 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:44:48.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:48 smithi097 ceph-mon[32231]: pgmap v273: 1 pgs: 1 remapped+peering; 577 KiB data, 284 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:44:48.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:44:48.556+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5840063230 con 0x7f58240b3bd0 2024-07-26T07:44:49.430 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:49 smithi097 ceph-mon[32231]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:44:49.430 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:44:49.430 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:44:49.430 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:49 smithi115 ceph-mon[35364]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:44:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:44:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:44:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:49 smithi181 ceph-mon[35869]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:44:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:44:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:44:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:50.316 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:50 smithi097 ceph-mon[32231]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:44:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:50 smithi115 ceph-mon[35364]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:44:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:50 smithi181 ceph-mon[35869]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-07-26T07:44:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:52 smithi115 ceph-mon[35364]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-26T07:44:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:52 smithi181 ceph-mon[35869]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-26T07:44:52.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:52 smithi097 ceph-mon[32231]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-26T07:44:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:53 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3493483684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "774b752e-42fc-4201-9051-9a9cdb1f495c"}]: dispatch 2024-07-26T07:44:53.547 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:53 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3493483684' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "774b752e-42fc-4201-9051-9a9cdb1f495c"}]': finished 2024-07-26T07:44:53.547 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:53 smithi115 ceph-mon[35364]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T07:44:53.547 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:44:53.547 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:53 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/54935857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:44:53.548 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:53 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3493483684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "774b752e-42fc-4201-9051-9a9cdb1f495c"}]: dispatch 2024-07-26T07:44:53.548 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:53 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3493483684' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "774b752e-42fc-4201-9051-9a9cdb1f495c"}]': finished 2024-07-26T07:44:53.548 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:53 smithi097 ceph-mon[32231]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T07:44:53.548 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:44:53.548 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:53 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/54935857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:44:53.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:53 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3493483684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "774b752e-42fc-4201-9051-9a9cdb1f495c"}]: dispatch 2024-07-26T07:44:53.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:53 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3493483684' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "774b752e-42fc-4201-9051-9a9cdb1f495c"}]': finished 2024-07-26T07:44:53.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:53 smithi181 ceph-mon[35869]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T07:44:53.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:44:53.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:53 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/54935857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:44:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:54 smithi115 ceph-mon[35364]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-26T07:44:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:54 smithi181 ceph-mon[35869]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-26T07:44:54.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:54 smithi097 ceph-mon[32231]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-26T07:44:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:56 smithi115 ceph-mon[35364]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:56 smithi181 ceph-mon[35869]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:56.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:56 smithi097 ceph-mon[32231]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:44:58 smithi115 ceph-mon[35364]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:44:58 smithi181 ceph-mon[35869]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:44:58.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:44:58 smithi097 ceph-mon[32231]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:45:00.444 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:00 smithi097 ceph-mon[32231]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:00.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:00 smithi115 ceph-mon[35364]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:00 smithi181 ceph-mon[35869]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T07:45:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T07:45:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:02.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T07:45:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:02 smithi115 ceph-mon[35364]: Deploying daemon osd.9 on smithi097 2024-07-26T07:45:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:02 smithi115 ceph-mon[35364]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:02 smithi181 ceph-mon[35869]: Deploying daemon osd.9 on smithi097 2024-07-26T07:45:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:02 smithi181 ceph-mon[35869]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:03.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:02 smithi097 ceph-mon[32231]: Deploying daemon osd.9 on smithi097 2024-07-26T07:45:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:02 smithi097 ceph-mon[32231]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:04.433 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:04 smithi097 ceph-mon[32231]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:04.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:04 smithi115 ceph-mon[35364]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:04 smithi181 ceph-mon[35869]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:06.426 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:06 smithi097 ceph-mon[32231]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:06 smithi115 ceph-mon[35364]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:06 smithi181 ceph-mon[35869]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:08.454 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:08 smithi097 ceph-mon[32231]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:08.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:08 smithi115 ceph-mon[35364]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:08 smithi181 ceph-mon[35869]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:10 smithi181 ceph-mon[35869]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:10.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:10 smithi115 ceph-mon[35364]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:10.543 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:10 smithi097 ceph-mon[32231]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:11.283 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:11.283 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:11.283 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:11.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:12.465 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:12 smithi097 ceph-mon[32231]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:12.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:12 smithi115 ceph-mon[35364]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:12 smithi181 ceph-mon[35869]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:14 smithi115 ceph-mon[35364]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:14 smithi115 ceph-mon[35364]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T07:45:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:14 smithi181 ceph-mon[35869]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:14 smithi181 ceph-mon[35869]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T07:45:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:14 smithi097 ceph-mon[32231]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:14 smithi097 ceph-mon[32231]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T07:45:15.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.135+0000 7f5837e00640 1 -- 172.21.15.97:0/335191909 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5840063230 con 0x7f58240b3bd0 2024-07-26T07:45:15.138 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 9 on host 'smithi097' 2024-07-26T07:45:15.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58240b3bd0 msgr2=0x7f58240b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:15.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58240b3bd0 0x7f58240b60e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f583000abe0 tx=0x7f583000a2d0 comp rx=0 tx=0).stop 2024-07-26T07:45:15.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010b000 msgr2=0x7f58401a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:15.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010b000 0x7f58401a1010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f583800cad0 tx=0x7f583800ced0 comp rx=0 tx=0).stop 2024-07-26T07:45:15.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 shutdown_connections 2024-07-26T07:45:15.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58240b3bd0 0x7f58240b60e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:15.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f584010b000 0x7f58401a1010 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:15.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f584010a660 0x7f58401a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:15.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 --2- 172.21.15.97:0/335191909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58401095d0 0x7f584019fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:15.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 >> 172.21.15.97:0/335191909 conn(0x7f5840078970 msgr2=0x7f584010d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:45:15.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 shutdown_connections 2024-07-26T07:45:15.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:45:15.139+0000 7f5845200640 1 -- 172.21.15.97:0/335191909 wait complete. 2024-07-26T07:45:15.835 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:15 smithi097 ceph-mon[32231]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T07:45:15.835 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:15 smithi097 ceph-mon[32231]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T07:45:15.835 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:15 smithi097 ceph-mon[32231]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:45:15.835 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:15.835 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:15.835 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:15.835 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:15.893 DEBUG:teuthology.orchestra.run.smithi097:osd.9> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.9.service 2024-07-26T07:45:15.899 INFO:tasks.cephadm:Deploying osd.10 on smithi115 with /dev/vg_nvme/lv_1... 2024-07-26T07:45:15.899 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T07:45:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:15 smithi115 ceph-mon[35364]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T07:45:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:15 smithi115 ceph-mon[35364]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T07:45:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:15 smithi115 ceph-mon[35364]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:45:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:15 smithi181 ceph-mon[35869]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T07:45:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:15 smithi181 ceph-mon[35869]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T07:45:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:15 smithi181 ceph-mon[35869]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-07-26T07:45:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:16.358 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T07:45:16.696 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:16 smithi097 ceph-mon[32231]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:45:16.696 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:16 smithi097 ceph-mon[32231]: osdmap e61: 10 total, 9 up, 10 in 2024-07-26T07:45:16.696 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:16.696 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:16.696 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:16 smithi097 ceph-mon[32231]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:16.848 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:16 smithi115 ceph-mon[35364]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:45:16.848 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:16 smithi115 ceph-mon[35364]: osdmap e61: 10 total, 9 up, 10 in 2024-07-26T07:45:16.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:16.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:16.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:16 smithi115 ceph-mon[35364]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:16 smithi181 ceph-mon[35869]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-07-26T07:45:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:16 smithi181 ceph-mon[35869]: osdmap e61: 10 total, 9 up, 10 in 2024-07-26T07:45:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:16 smithi181 ceph-mon[35869]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:17.001 INFO:journalctl@ceph.osd.9.smithi097.stdout:Jul 26 07:45:16 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-9[70056]: 2024-07-26T07:45:16.837+0000 7fb785000640 -1 osd.9 0 waiting for initial osdmap 2024-07-26T07:45:17.001 INFO:journalctl@ceph.osd.9.smithi097.stdout:Jul 26 07:45:16 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-9[70056]: 2024-07-26T07:45:16.878+0000 7fb77b000640 -1 osd.9 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T07:45:17.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:17 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:45:17.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:17 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:45:17.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:17.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:17 smithi097 ceph-mon[32231]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' 2024-07-26T07:45:17.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:17 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:45:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:17 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:45:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:17 smithi115 ceph-mon[35364]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' 2024-07-26T07:45:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:17 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:45:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:17 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:45:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:17 smithi181 ceph-mon[35869]: from='osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967]' entity='osd.9' 2024-07-26T07:45:18.923 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T07:45:18.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:18 smithi115 ceph-mon[35364]: osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] boot 2024-07-26T07:45:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:18 smithi115 ceph-mon[35364]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T07:45:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:18 smithi115 ceph-mon[35364]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:18 smithi181 ceph-mon[35869]: osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] boot 2024-07-26T07:45:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:18 smithi181 ceph-mon[35869]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T07:45:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:18 smithi181 ceph-mon[35869]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:19.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:19.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:18 smithi097 ceph-mon[32231]: osd.9 [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] boot 2024-07-26T07:45:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:18 smithi097 ceph-mon[32231]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T07:45:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T07:45:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:18 smithi097 ceph-mon[32231]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-07-26T07:45:19.112 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T07:45:19.154 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-07-26T07:45:19.154 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-07-26T07:45:19.154 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0351615 s, 298 MB/s 2024-07-26T07:45:19.155 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-07-26T07:45:19.934 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi115:vg_nvme/lv_1 2024-07-26T07:45:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:19 smithi097 ceph-mon[32231]: osdmap e63: 10 total, 10 up, 10 in 2024-07-26T07:45:20.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:19 smithi115 ceph-mon[35364]: osdmap e63: 10 total, 10 up, 10 in 2024-07-26T07:45:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:19 smithi181 ceph-mon[35869]: osdmap e63: 10 total, 10 up, 10 in 2024-07-26T07:45:20.378 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T07:45:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:22.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:21 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi097 2024-07-26T07:45:22.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:21 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 4272M 2024-07-26T07:45:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:21 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi097 2024-07-26T07:45:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:21 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 4272M 2024-07-26T07:45:22.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:21 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi097 2024-07-26T07:45:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:21 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 4272M 2024-07-26T07:45:22.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 -- 172.21.15.115:0/988124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec010a660 msgr2=0x7fdec010aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:22.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 --2- 172.21.15.115:0/988124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec010a660 0x7fdec010aa70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009900 tx=0x7fdea802a480 comp rx=0 tx=0).stop 2024-07-26T07:45:22.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 -- 172.21.15.115:0/988124 shutdown_connections 2024-07-26T07:45:22.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 --2- 172.21.15.115:0/988124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec010b000 0x7fdec01119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:22.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 --2- 172.21.15.115:0/988124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec010a660 0x7fdec010aa70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:22.744 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 --2- 172.21.15.115:0/988124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec01095d0 0x7fdec0109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:22.744 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 -- 172.21.15.115:0/988124 >> 172.21.15.115:0/988124 conn(0x7fdec0078970 msgr2=0x7fdec007ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:45:22.744 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 -- 172.21.15.115:0/988124 shutdown_connections 2024-07-26T07:45:22.744 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.741+0000 7fdec5200640 1 -- 172.21.15.115:0/988124 wait complete. 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.742+0000 7fdec5200640 1 Processor -- start 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdec5200640 1 -- start start 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdec5200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec01095d0 0x7fdec019fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdec5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec010a660 0x7fdec01a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdec5200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec010b000 0x7fdec01a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdec5200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdec0115120 con 0x7fdec010a660 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdec5200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdec0114fa0 con 0x7fdec01095d0 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdec5200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdec01152a0 con 0x7fdec010b000 2024-07-26T07:45:22.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdebf400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec01095d0 0x7fdec019fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdebf400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec01095d0 0x7fdec019fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:58516/0 (socket says 172.21.15.115:58516) 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.743+0000 7fdebf400640 1 -- 172.21.15.115:0/4242057677 learned_addr learned my addr 172.21.15.115:0/4242057677 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.744+0000 7fdeb6a00640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec010a660 0x7fdec01a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.744+0000 7fdebfe00640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec010b000 0x7fdec01a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.744+0000 7fdebf400640 1 -- 172.21.15.115:0/4242057677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec010b000 msgr2=0x7fdec01a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.744+0000 7fdebf400640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec010b000 0x7fdec01a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.744+0000 7fdebf400640 1 -- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec010a660 msgr2=0x7fdec01a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.744+0000 7fdebf400640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec010a660 0x7fdec01a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.744+0000 7fdebf400640 1 -- 172.21.15.115:0/4242057677 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec01a17e0 con 0x7fdec01095d0 2024-07-26T07:45:22.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.744+0000 7fdebf400640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec01095d0 0x7fdec019fdb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000d950 tx=0x7fdeb000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:45:22.748 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.745+0000 7fdebcc00640 1 -- 172.21.15.115:0/4242057677 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0017070 con 0x7fdec01095d0 2024-07-26T07:45:22.748 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.745+0000 7fdebcc00640 1 -- 172.21.15.115:0/4242057677 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeb0009c30 con 0x7fdec01095d0 2024-07-26T07:45:22.748 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.745+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec01a1ad0 con 0x7fdec01095d0 2024-07-26T07:45:22.748 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.745+0000 7fdebcc00640 1 -- 172.21.15.115:0/4242057677 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000f620 con 0x7fdec01095d0 2024-07-26T07:45:22.748 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.745+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec01a81d0 con 0x7fdec01095d0 2024-07-26T07:45:22.748 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.746+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec0110790 con 0x7fdec01095d0 2024-07-26T07:45:22.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.748+0000 7fdebcc00640 1 -- 172.21.15.115:0/4242057677 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fdeb0010460 con 0x7fdec01095d0 2024-07-26T07:45:22.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.750+0000 7fdebcc00640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdea40b3bd0 0x7fdea40b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:45:22.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.750+0000 7fdebcc00640 1 -- 172.21.15.115:0/4242057677 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6261+0+0 (secure 0 0 0) 0x7fdeb00e30c0 con 0x7fdec01095d0 2024-07-26T07:45:22.752 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.750+0000 7fdeb6a00640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdea40b3bd0 0x7fdea40b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:45:22.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.752+0000 7fdeb6a00640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdea40b3bd0 0x7fdea40b60e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdea8006010 tx=0x7fdea8005720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:45:22.755 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.754+0000 7fdebcc00640 1 -- 172.21.15.115:0/4242057677 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdeb00aaf10 con 0x7fdec01095d0 2024-07-26T07:45:22.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:22.965+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdec0063230 con 0x7fdea40b3bd0 2024-07-26T07:45:23.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:22 smithi115 ceph-mon[35364]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:22 smithi181 ceph-mon[35869]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:23.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:22 smithi097 ceph-mon[32231]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:23 smithi115 ceph-mon[35364]: from='client.34451 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:45:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:45:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:45:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:23 smithi181 ceph-mon[35869]: from='client.34451 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:45:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:45:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:45:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:24.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:23 smithi097 ceph-mon[32231]: from='client.34451 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:45:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:45:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:45:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:25.189 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:24 smithi115 ceph-mon[35364]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:24 smithi181 ceph-mon[35869]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:25.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:24 smithi097 ceph-mon[32231]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:26.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:26 smithi115 ceph-mon[35364]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:26 smithi181 ceph-mon[35869]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:26.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:26 smithi097 ceph-mon[32231]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:27.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:27 smithi097 ceph-mon[32231]: from='client.? 172.21.15.115:0/3530130581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]: dispatch 2024-07-26T07:45:27.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:27 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]: dispatch 2024-07-26T07:45:27.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:27 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]': finished 2024-07-26T07:45:27.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:27 smithi097 ceph-mon[32231]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T07:45:27.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:27 smithi115 ceph-mon[35364]: from='client.? 172.21.15.115:0/3530130581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]: dispatch 2024-07-26T07:45:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:27 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]: dispatch 2024-07-26T07:45:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:27 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]': finished 2024-07-26T07:45:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:27 smithi115 ceph-mon[35364]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T07:45:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:27 smithi181 ceph-mon[35869]: from='client.? 172.21.15.115:0/3530130581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]: dispatch 2024-07-26T07:45:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:27 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]: dispatch 2024-07-26T07:45:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:27 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2"}]': finished 2024-07-26T07:45:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:27 smithi181 ceph-mon[35869]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T07:45:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:28 smithi181 ceph-mon[35869]: from='client.? 172.21.15.115:0/2338869793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:45:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:28 smithi181 ceph-mon[35869]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:28.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:28 smithi097 ceph-mon[32231]: from='client.? 172.21.15.115:0/2338869793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:45:28.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:28 smithi097 ceph-mon[32231]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:28.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:28 smithi115 ceph-mon[35364]: from='client.? 172.21.15.115:0/2338869793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:45:28.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:28 smithi115 ceph-mon[35364]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:30 smithi181 ceph-mon[35869]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:30 smithi115 ceph-mon[35364]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:30.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:30 smithi097 ceph-mon[32231]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:32.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:32 smithi115 ceph-mon[35364]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:32 smithi181 ceph-mon[35869]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:32 smithi097 ceph-mon[32231]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:34 smithi115 ceph-mon[35364]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:34 smithi181 ceph-mon[35869]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:34.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:34 smithi097 ceph-mon[32231]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:36 smithi115 ceph-mon[35364]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:36.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:36 smithi097 ceph-mon[32231]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:36.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:36 smithi181 ceph-mon[35869]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T07:45:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:37 smithi115 ceph-mon[35364]: Deploying daemon osd.10 on smithi115 2024-07-26T07:45:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T07:45:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:37 smithi181 ceph-mon[35869]: Deploying daemon osd.10 on smithi115 2024-07-26T07:45:37.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T07:45:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:37 smithi097 ceph-mon[32231]: Deploying daemon osd.10 on smithi115 2024-07-26T07:45:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:38 smithi115 ceph-mon[35364]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:38 smithi181 ceph-mon[35869]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:38.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:38 smithi097 ceph-mon[32231]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:40.362 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:40 smithi115 ceph-mon[35364]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:40 smithi181 ceph-mon[35869]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:40.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:40 smithi097 ceph-mon[32231]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:42 smithi097 ceph-mon[32231]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:42 smithi115 ceph-mon[35364]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:42 smithi181 ceph-mon[35869]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:44 smithi115 ceph-mon[35364]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:44 smithi181 ceph-mon[35869]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:44.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:44 smithi097 ceph-mon[32231]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:45.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:45.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:45.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:45.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:46.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:46 smithi115 ceph-mon[35364]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:46 smithi181 ceph-mon[35869]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:46.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:46 smithi097 ceph-mon[32231]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:48.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:48 smithi115 ceph-mon[35364]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:48 smithi181 ceph-mon[35869]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:48.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:48 smithi097 ceph-mon[32231]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:49.341 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.339+0000 7fdebcc00640 1 -- 172.21.15.115:0/4242057677 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fdec0063230 con 0x7fdea40b3bd0 2024-07-26T07:45:49.341 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 10 on host 'smithi115' 2024-07-26T07:45:49.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.342+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdea40b3bd0 msgr2=0x7fdea40b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:49.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.342+0000 7fdec5200640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdea40b3bd0 0x7fdea40b60e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdea8006010 tx=0x7fdea8005720 comp rx=0 tx=0).stop 2024-07-26T07:45:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec01095d0 msgr2=0x7fdec019fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec01095d0 0x7fdec019fdb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000d950 tx=0x7fdeb000de10 comp rx=0 tx=0).stop 2024-07-26T07:45:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 shutdown_connections 2024-07-26T07:45:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdea40b3bd0 0x7fdea40b60e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec010b000 0x7fdec01a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdec010a660 0x7fdec01a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 --2- 172.21.15.115:0/4242057677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdec01095d0 0x7fdec019fdb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:49.345 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 >> 172.21.15.115:0/4242057677 conn(0x7fdec0078970 msgr2=0x7fdec010d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:45:49.345 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.343+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 shutdown_connections 2024-07-26T07:45:49.345 INFO:teuthology.orchestra.run.smithi115.stderr:2024-07-26T07:45:49.344+0000 7fdec5200640 1 -- 172.21.15.115:0/4242057677 wait complete. 2024-07-26T07:45:49.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:49.483 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:49.483 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:49 smithi115 ceph-mon[35364]: from='osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T07:45:49.484 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:49 smithi115 ceph-mon[35364]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T07:45:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:49 smithi181 ceph-mon[35869]: from='osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T07:45:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:49 smithi181 ceph-mon[35869]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T07:45:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:49.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:49 smithi097 ceph-mon[32231]: from='osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T07:45:49.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:49 smithi097 ceph-mon[32231]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T07:45:50.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:50 smithi115 ceph-mon[35364]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T07:45:50.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:50 smithi115 ceph-mon[35364]: from='osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:45:50.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:50 smithi115 ceph-mon[35364]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T07:45:50.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:50.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:50 smithi115 ceph-mon[35364]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:45:50.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:50.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:50.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:50 smithi115 ceph-mon[35364]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:50.443 DEBUG:teuthology.orchestra.run.smithi115:osd.10> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.10.service 2024-07-26T07:45:50.447 INFO:tasks.cephadm:Deploying osd.11 on smithi181 with /dev/vg_nvme/lv_1... 2024-07-26T07:45:50.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T07:45:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:50 smithi097 ceph-mon[32231]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T07:45:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:50 smithi097 ceph-mon[32231]: from='osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:45:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:50 smithi097 ceph-mon[32231]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T07:45:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:50 smithi097 ceph-mon[32231]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:45:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:50 smithi097 ceph-mon[32231]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:50.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:50 smithi181 ceph-mon[35869]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T07:45:50.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:50 smithi181 ceph-mon[35869]: from='osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:45:50.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:50 smithi181 ceph-mon[35869]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T07:45:50.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:50.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:50 smithi181 ceph-mon[35869]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-07-26T07:45:50.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:50.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:50.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:50 smithi181 ceph-mon[35869]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:50.902 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:45:51.308 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:51 smithi181 ceph-mon[35869]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:45:51.308 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:51 smithi181 ceph-mon[35869]: osdmap e66: 11 total, 10 up, 11 in 2024-07-26T07:45:51.308 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.308 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.308 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:51 smithi181 ceph-mon[35869]: osdmap e67: 11 total, 10 up, 11 in 2024-07-26T07:45:51.308 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:51 smithi097 ceph-mon[32231]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:45:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:51 smithi097 ceph-mon[32231]: osdmap e66: 11 total, 10 up, 11 in 2024-07-26T07:45:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:51 smithi097 ceph-mon[32231]: osdmap e67: 11 total, 10 up, 11 in 2024-07-26T07:45:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:51 smithi115 ceph-mon[35364]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-07-26T07:45:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:51 smithi115 ceph-mon[35364]: osdmap e66: 11 total, 10 up, 11 in 2024-07-26T07:45:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:51 smithi115 ceph-mon[35364]: osdmap e67: 11 total, 10 up, 11 in 2024-07-26T07:45:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:51.725 INFO:journalctl@ceph.osd.10.smithi115.stdout:Jul 26 07:45:51 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-10[60682]: 2024-07-26T07:45:51.348+0000 7fc4dfa00640 -1 osd.10 0 waiting for initial osdmap 2024-07-26T07:45:51.725 INFO:journalctl@ceph.osd.10.smithi115.stdout:Jul 26 07:45:51 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-10[60682]: 2024-07-26T07:45:51.383+0000 7fc4d5a00640 -1 osd.10 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T07:45:52.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:52 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:45:52.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:52 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:45:52.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:52.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:52 smithi115 ceph-mon[35364]: from='osd.10 ' entity='osd.10' 2024-07-26T07:45:52.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:52 smithi115 ceph-mon[35364]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:52.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:52 smithi115 ceph-mon[35364]: osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] boot 2024-07-26T07:45:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:52 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:45:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:52 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:45:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:52 smithi181 ceph-mon[35869]: from='osd.10 ' entity='osd.10' 2024-07-26T07:45:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:52 smithi181 ceph-mon[35869]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:52 smithi181 ceph-mon[35869]: osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] boot 2024-07-26T07:45:52.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:52 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:45:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:52 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:45:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:52 smithi097 ceph-mon[32231]: from='osd.10 ' entity='osd.10' 2024-07-26T07:45:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:52 smithi097 ceph-mon[32231]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-07-26T07:45:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:52 smithi097 ceph-mon[32231]: osd.10 [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] boot 2024-07-26T07:45:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:53 smithi181 ceph-mon[35869]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T07:45:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:53 smithi181 ceph-mon[35869]: osdmap e69: 11 total, 11 up, 11 in 2024-07-26T07:45:53.575 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T07:45:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:53 smithi115 ceph-mon[35364]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T07:45:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:53 smithi115 ceph-mon[35364]: osdmap e69: 11 total, 11 up, 11 in 2024-07-26T07:45:53.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:53 smithi097 ceph-mon[32231]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T07:45:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T07:45:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:53 smithi097 ceph-mon[32231]: osdmap e69: 11 total, 11 up, 11 in 2024-07-26T07:45:53.793 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T07:45:53.836 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-07-26T07:45:53.836 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-07-26T07:45:53.836 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0332397 s, 315 MB/s 2024-07-26T07:45:53.836 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-07-26T07:45:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:54 smithi115 ceph-mon[35364]: pgmap v317: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:45:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:54 smithi181 ceph-mon[35869]: pgmap v317: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:45:54.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2024-07-26T07:45:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:54 smithi097 ceph-mon[32231]: pgmap v317: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:45:55.107 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:45:55.585 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: osdmap e70: 11 total, 11 up, 11 in 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi115 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 4272M 2024-07-26T07:45:55.586 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.587 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:55.587 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:55.587 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.587 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:55.587 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:55.587 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: osdmap e70: 11 total, 11 up, 11 in 2024-07-26T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi115 2024-07-26T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 4272M 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:55.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: osdmap e70: 11 total, 11 up, 11 in 2024-07-26T07:45:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi115 2024-07-26T07:45:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 4272M 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:56 smithi181 ceph-mon[35869]: pgmap v319: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:45:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:56 smithi115 ceph-mon[35364]: pgmap v319: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:45:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:45:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:56 smithi097 ceph-mon[32231]: pgmap v319: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:45:57.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 -- 172.21.15.181:0/1432035927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc106260 msgr2=0x7f18cc106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:57.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1432035927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc106260 0x7f18cc106670 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009900 tx=0x7f18b802a480 comp rx=0 tx=0).stop 2024-07-26T07:45:57.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 -- 172.21.15.181:0/1432035927 shutdown_connections 2024-07-26T07:45:57.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1432035927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18cc106c00 0x7f18cc10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:57.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1432035927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc106260 0x7f18cc106670 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:57.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1432035927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18cc1051d0 0x7f18cc105640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:57.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 -- 172.21.15.181:0/1432035927 >> 172.21.15.181:0/1432035927 conn(0x7f18cc100b60 msgr2=0x7f18cc102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:45:57.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 -- 172.21.15.181:0/1432035927 shutdown_connections 2024-07-26T07:45:57.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.474+0000 7f18d2c00640 1 -- 172.21.15.181:0/1432035927 wait complete. 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.475+0000 7f18d2c00640 1 Processor -- start 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.475+0000 7f18d2c00640 1 -- start start 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d2c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc1051d0 0x7f18cc0775e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d2c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18cc106260 0x7f18cc077c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d2c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18cc106c00 0x7f18cc0787f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d2c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc110d30 con 0x7f18cc106c00 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d2c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc110bb0 con 0x7f18cc106260 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d2c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc110eb0 con 0x7f18cc1051d0 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d1800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc1051d0 0x7f18cc0775e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d1800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc1051d0 0x7f18cc0775e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:42340/0 (socket says 172.21.15.181:42340) 2024-07-26T07:45:57.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d1800640 1 -- 172.21.15.181:0/1949613858 learned_addr learned my addr 172.21.15.181:0/1949613858 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-07-26T07:45:57.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d0e00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18cc106260 0x7f18cc077c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:45:57.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d1800640 1 -- 172.21.15.181:0/1949613858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18cc106260 msgr2=0x7f18cc077c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:57.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d1800640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18cc106260 0x7f18cc077c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:57.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d1800640 1 -- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18cc106c00 msgr2=0x7f18cc0787f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:45:57.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d1800640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18cc106c00 0x7f18cc0787f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:45:57.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d1800640 1 -- 172.21.15.181:0/1949613858 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18cc078f80 con 0x7f18cc1051d0 2024-07-26T07:45:57.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.476+0000 7f18d0e00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18cc106260 0x7f18cc077c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:45:57.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.477+0000 7f18d1800640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc1051d0 0x7f18cc0775e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f18c000d950 tx=0x7f18c000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:45:57.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.477+0000 7f18ca000640 1 -- 172.21.15.181:0/1949613858 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c0017070 con 0x7f18cc1051d0 2024-07-26T07:45:57.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.477+0000 7f18ca000640 1 -- 172.21.15.181:0/1949613858 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f18c0009c30 con 0x7f18cc1051d0 2024-07-26T07:45:57.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.477+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18cc079270 con 0x7f18cc1051d0 2024-07-26T07:45:57.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.477+0000 7f18ca000640 1 -- 172.21.15.181:0/1949613858 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c000f620 con 0x7f18cc1051d0 2024-07-26T07:45:57.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.477+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18cc0799d0 con 0x7f18cc1051d0 2024-07-26T07:45:57.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.479+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18cc105640 con 0x7f18cc1051d0 2024-07-26T07:45:57.482 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.481+0000 7f18ca000640 1 -- 172.21.15.181:0/1949613858 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f18c0010460 con 0x7f18cc1051d0 2024-07-26T07:45:57.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.482+0000 7f18ca000640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18b40b3c50 0x7f18b40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:45:57.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.483+0000 7f18ca000640 1 -- 172.21.15.181:0/1949613858 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6694+0+0 (secure 0 0 0) 0x7f18c00e32b0 con 0x7f18cc1051d0 2024-07-26T07:45:57.484 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.483+0000 7f18d0e00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18b40b3c50 0x7f18b40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:45:57.485 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.484+0000 7f18d0e00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18b40b3c50 0x7f18b40b6160 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f18b802a9a0 tx=0x7f18b8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:45:57.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.485+0000 7f18ca000640 1 -- 172.21.15.181:0/1949613858 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f18c00aaf30 con 0x7f18cc1051d0 2024-07-26T07:45:57.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:45:57.690+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f18cc063230 con 0x7f18b40b3c50 2024-07-26T07:45:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:45:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:45:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:45:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:45:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:58.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T07:45:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T07:45:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:58 smithi181 ceph-mon[35869]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:45:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:45:58 smithi181 ceph-mon[35869]: pgmap v320: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:45:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:58 smithi115 ceph-mon[35364]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:45:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:45:58 smithi115 ceph-mon[35364]: pgmap v320: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:45:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:58 smithi097 ceph-mon[32231]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:45:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:45:58 smithi097 ceph-mon[32231]: pgmap v320: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:00.332 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:00 smithi181 ceph-mon[35869]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-07-26T07:46:00.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:00 smithi115 ceph-mon[35364]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-07-26T07:46:00.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:00 smithi097 ceph-mon[32231]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-07-26T07:46:01.862 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:01 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/1291073041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]: dispatch 2024-07-26T07:46:01.865 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:01 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]: dispatch 2024-07-26T07:46:01.865 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:01 smithi097 ceph-mon[32231]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]': finished 2024-07-26T07:46:01.865 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:01 smithi097 ceph-mon[32231]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T07:46:01.865 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:01.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:01 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/1291073041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]: dispatch 2024-07-26T07:46:01.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:01 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]: dispatch 2024-07-26T07:46:01.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:01 smithi181 ceph-mon[35869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]': finished 2024-07-26T07:46:01.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:01 smithi181 ceph-mon[35869]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T07:46:01.866 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:01 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/1291073041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]: dispatch 2024-07-26T07:46:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:01 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]: dispatch 2024-07-26T07:46:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:01 smithi115 ceph-mon[35364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7"}]': finished 2024-07-26T07:46:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:01 smithi115 ceph-mon[35364]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T07:46:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:02 smithi181 ceph-mon[35869]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:46:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:02 smithi181 ceph-mon[35869]: from='client.? 172.21.15.181:0/1137609549' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:46:02.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:02 smithi115 ceph-mon[35364]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:46:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:02 smithi115 ceph-mon[35364]: from='client.? 172.21.15.181:0/1137609549' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:46:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:02 smithi097 ceph-mon[32231]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T07:46:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:02 smithi097 ceph-mon[32231]: from='client.? 172.21.15.181:0/1137609549' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T07:46:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:04 smithi115 ceph-mon[35364]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-07-26T07:46:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:04 smithi181 ceph-mon[35869]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-07-26T07:46:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:04 smithi097 ceph-mon[32231]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-07-26T07:46:06.400 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:06 smithi181 ceph-mon[35869]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:46:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:06 smithi115 ceph-mon[35364]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:46:06.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:06 smithi097 ceph-mon[32231]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:46:08.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:08 smithi181 ceph-mon[35869]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:46:08.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:08 smithi115 ceph-mon[35364]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:46:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:08 smithi097 ceph-mon[32231]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T07:46:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:10 smithi115 ceph-mon[35364]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:10 smithi181 ceph-mon[35869]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:10 smithi097 ceph-mon[32231]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:11.664 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T07:46:11.664 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T07:46:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T07:46:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:12 smithi097 ceph-mon[32231]: Deploying daemon osd.11 on smithi181 2024-07-26T07:46:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:12 smithi097 ceph-mon[32231]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:12.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:12 smithi115 ceph-mon[35364]: Deploying daemon osd.11 on smithi181 2024-07-26T07:46:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:12 smithi115 ceph-mon[35364]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:12 smithi181 ceph-mon[35869]: Deploying daemon osd.11 on smithi181 2024-07-26T07:46:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:12 smithi181 ceph-mon[35869]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:14 smithi115 ceph-mon[35364]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:14 smithi181 ceph-mon[35869]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:15.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:14 smithi097 ceph-mon[32231]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:16.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:16 smithi115 ceph-mon[35364]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:16 smithi181 ceph-mon[35869]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:16 smithi097 ceph-mon[32231]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:18.399 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:18 smithi181 ceph-mon[35869]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:18 smithi115 ceph-mon[35364]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:18.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:18 smithi097 ceph-mon[32231]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:20 smithi115 ceph-mon[35364]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:20 smithi181 ceph-mon[35869]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:20 smithi097 ceph-mon[32231]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:22 smithi115 ceph-mon[35364]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:22 smithi181 ceph-mon[35869]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:22.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:22 smithi097 ceph-mon[32231]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:24 smithi181 ceph-mon[35869]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:24 smithi181 ceph-mon[35869]: from='osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T07:46:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:24 smithi181 ceph-mon[35869]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T07:46:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:24 smithi115 ceph-mon[35364]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:24 smithi115 ceph-mon[35364]: from='osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T07:46:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:24 smithi115 ceph-mon[35364]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T07:46:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:24 smithi097 ceph-mon[32231]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:24 smithi097 ceph-mon[32231]: from='osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T07:46:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:24 smithi097 ceph-mon[32231]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T07:46:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:25.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.337+0000 7f18ca000640 1 -- 172.21.15.181:0/1949613858 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f18cc063230 con 0x7f18b40b3c50 2024-07-26T07:46:25.339 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 11 on host 'smithi181' 2024-07-26T07:46:25.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.341+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18b40b3c50 msgr2=0x7f18b40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:25.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.341+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18b40b3c50 0x7f18b40b6160 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f18b802a9a0 tx=0x7f18b8005fd0 comp rx=0 tx=0).stop 2024-07-26T07:46:25.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.341+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc1051d0 msgr2=0x7f18cc0775e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:25.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.341+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc1051d0 0x7f18cc0775e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f18c000d950 tx=0x7f18c000de10 comp rx=0 tx=0).stop 2024-07-26T07:46:25.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.342+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 shutdown_connections 2024-07-26T07:46:25.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.342+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18b40b3c50 0x7f18b40b6160 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:25.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.342+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18cc106c00 0x7f18cc0787f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:25.342 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.342+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18cc106260 0x7f18cc077c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:25.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.342+0000 7f18d2c00640 1 --2- 172.21.15.181:0/1949613858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18cc1051d0 0x7f18cc0775e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:25.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.342+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 >> 172.21.15.181:0/1949613858 conn(0x7f18cc100b60 msgr2=0x7f18cc109560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:25.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.342+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 shutdown_connections 2024-07-26T07:46:25.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:25.342+0000 7f18d2c00640 1 -- 172.21.15.181:0/1949613858 wait complete. 2024-07-26T07:46:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:25 smithi115 ceph-mon[35364]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T07:46:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:25 smithi115 ceph-mon[35364]: osdmap e72: 12 total, 11 up, 12 in 2024-07-26T07:46:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:25 smithi115 ceph-mon[35364]: from='osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:46:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:25 smithi115 ceph-mon[35364]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:46:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:25 smithi181 ceph-mon[35869]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T07:46:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:25 smithi181 ceph-mon[35869]: osdmap e72: 12 total, 11 up, 12 in 2024-07-26T07:46:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:25 smithi181 ceph-mon[35869]: from='osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:46:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:25 smithi181 ceph-mon[35869]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:46:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:25 smithi097 ceph-mon[32231]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T07:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:25 smithi097 ceph-mon[32231]: osdmap e72: 12 total, 11 up, 12 in 2024-07-26T07:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:25 smithi097 ceph-mon[32231]: from='osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:25 smithi097 ceph-mon[32231]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-07-26T07:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:26.151 DEBUG:teuthology.orchestra.run.smithi181:osd.11> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@osd.11.service 2024-07-26T07:46:26.156 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-07-26T07:46:26.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd stat -f json 2024-07-26T07:46:26.605 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:46:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:26 smithi115 ceph-mon[35364]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:46:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:26 smithi115 ceph-mon[35364]: osdmap e73: 12 total, 11 up, 12 in 2024-07-26T07:46:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:26 smithi115 ceph-mon[35364]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:26.975 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:26 smithi097 ceph-mon[32231]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:46:26.976 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:26 smithi097 ceph-mon[32231]: osdmap e73: 12 total, 11 up, 12 in 2024-07-26T07:46:26.976 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:26.976 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:26.976 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:26 smithi097 ceph-mon[32231]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:26 smithi181 ceph-mon[35869]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-07-26T07:46:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:26 smithi181 ceph-mon[35869]: osdmap e73: 12 total, 11 up, 12 in 2024-07-26T07:46:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:26 smithi181 ceph-mon[35869]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:26.983 INFO:journalctl@ceph.osd.11.smithi181.stdout:Jul 26 07:46:26 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-11[60965]: 2024-07-26T07:46:26.781+0000 7f6fe3800640 -1 osd.11 0 waiting for initial osdmap 2024-07-26T07:46:26.983 INFO:journalctl@ceph.osd.11.smithi181.stdout:Jul 26 07:46:26 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-osd-11[60965]: 2024-07-26T07:46:26.840+0000 7f6fd8e00640 -1 osd.11 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T07:46:27.855 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:27 smithi181 ceph-mon[35869]: purged_snaps scrub starts 2024-07-26T07:46:27.855 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:27 smithi181 ceph-mon[35869]: purged_snaps scrub ok 2024-07-26T07:46:27.855 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:27.855 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:27 smithi181 ceph-mon[35869]: from='osd.11 ' entity='osd.11' 2024-07-26T07:46:27.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:27 smithi115 ceph-mon[35364]: purged_snaps scrub starts 2024-07-26T07:46:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:27 smithi115 ceph-mon[35364]: purged_snaps scrub ok 2024-07-26T07:46:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:27 smithi115 ceph-mon[35364]: from='osd.11 ' entity='osd.11' 2024-07-26T07:46:28.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:27 smithi097 ceph-mon[32231]: purged_snaps scrub starts 2024-07-26T07:46:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:27 smithi097 ceph-mon[32231]: purged_snaps scrub ok 2024-07-26T07:46:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:27 smithi097 ceph-mon[32231]: from='osd.11 ' entity='osd.11' 2024-07-26T07:46:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:28 smithi115 ceph-mon[35364]: osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] boot 2024-07-26T07:46:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:28 smithi115 ceph-mon[35364]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T07:46:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:28 smithi115 ceph-mon[35364]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:28 smithi181 ceph-mon[35869]: osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] boot 2024-07-26T07:46:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:28 smithi181 ceph-mon[35869]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T07:46:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:28 smithi181 ceph-mon[35869]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:28.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.995+0000 7f7653200640 1 -- 172.21.15.97:0/1396523448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76541095d0 msgr2=0x7f7654109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:28.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.995+0000 7f7653200640 1 --2- 172.21.15.97:0/1396523448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76541095d0 0x7f7654109a40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7644009900 tx=0x7f764402a480 comp rx=0 tx=0).stop 2024-07-26T07:46:28.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.996+0000 7f7653200640 1 -- 172.21.15.97:0/1396523448 shutdown_connections 2024-07-26T07:46:28.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.996+0000 7f7653200640 1 --2- 172.21.15.97:0/1396523448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f765410b000 0x7f76541119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:28.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.996+0000 7f7653200640 1 --2- 172.21.15.97:0/1396523448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f765410a660 0x7f765410aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:28.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.996+0000 7f7653200640 1 --2- 172.21.15.97:0/1396523448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76541095d0 0x7f7654109a40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:28.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.996+0000 7f7653200640 1 -- 172.21.15.97:0/1396523448 >> 172.21.15.97:0/1396523448 conn(0x7f7654078970 msgr2=0x7f765407ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:28.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.996+0000 7f7653200640 1 -- 172.21.15.97:0/1396523448 shutdown_connections 2024-07-26T07:46:28.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.996+0000 7f7653200640 1 -- 172.21.15.97:0/1396523448 wait complete. 2024-07-26T07:46:28.998 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:28.998 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:28 smithi097 ceph-mon[32231]: osd.11 [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] boot 2024-07-26T07:46:28.999 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:28 smithi097 ceph-mon[32231]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T07:46:28.999 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T07:46:28.999 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:28 smithi097 ceph-mon[32231]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-07-26T07:46:29.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.997+0000 7f7653200640 1 Processor -- start 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.997+0000 7f7653200640 1 -- start start 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.997+0000 7f7653200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76541095d0 0x7f765419fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.997+0000 7f7653200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f765410a660 0x7f76541a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7653200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f765410b000 0x7f76541a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7653200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f76541150d0 con 0x7f765410b000 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7653200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7654114f50 con 0x7f76541095d0 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7653200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7654115250 con 0x7f765410a660 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7652800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f765410b000 0x7f76541a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:29.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7652800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f765410b000 0x7f76541a0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36704/0 (socket says 172.21.15.97:36704) 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7652800640 1 -- 172.21.15.97:0/95788043 learned_addr learned my addr 172.21.15.97:0/95788043 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7651e00640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76541095d0 0x7f765419fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.998+0000 7f7651400640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f765410a660 0x7f76541a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f7652800640 1 -- 172.21.15.97:0/95788043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f765410a660 msgr2=0x7f76541a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f7652800640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f765410a660 0x7f76541a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f7652800640 1 -- 172.21.15.97:0/95788043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76541095d0 msgr2=0x7f765419fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f7652800640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76541095d0 0x7f765419fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f7652800640 1 -- 172.21.15.97:0/95788043 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76541a1750 con 0x7f765410b000 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f7652800640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f765410b000 0x7f76541a0fc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f764c00cad0 tx=0x7f764c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:29.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f764aa00640 1 -- 172.21.15.97:0/95788043 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c00ad60 con 0x7f765410b000 2024-07-26T07:46:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f764aa00640 1 -- 172.21.15.97:0/95788043 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f764c005020 con 0x7f765410b000 2024-07-26T07:46:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f764aa00640 1 -- 172.21.15.97:0/95788043 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c00f6b0 con 0x7f765410b000 2024-07-26T07:46:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:28.999+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76541a1a40 con 0x7f765410b000 2024-07-26T07:46:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.000+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76541a8000 con 0x7f765410b000 2024-07-26T07:46:29.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.001+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76541107b0 con 0x7f765410b000 2024-07-26T07:46:29.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.003+0000 7f764aa00640 1 -- 172.21.15.97:0/95788043 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f764c0051d0 con 0x7f765410b000 2024-07-26T07:46:29.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.005+0000 7f764aa00640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76380b3bd0 0x7f76380b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:29.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.005+0000 7f764aa00640 1 -- 172.21.15.97:0/95788043 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f764c0e3b60 con 0x7f765410b000 2024-07-26T07:46:29.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.005+0000 7f7651e00640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76380b3bd0 0x7f76380b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:29.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.006+0000 7f7651e00640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76380b3bd0 0x7f76380b60e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f76440057c0 tx=0x7f7644000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:29.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.007+0000 7f764aa00640 1 -- 172.21.15.97:0/95788043 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f764c014030 con 0x7f765410b000 2024-07-26T07:46:29.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.206+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f76540639f0 con 0x7f765410b000 2024-07-26T07:46:29.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.207+0000 7f764aa00640 1 -- 172.21.15.97:0/95788043 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) ==== 74+0+133 (secure 0 0 0) 0x7f764c0ab5b0 con 0x7f765410b000 2024-07-26T07:46:29.208 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:46:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.209+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76380b3bd0 msgr2=0x7f76380b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76380b3bd0 0x7f76380b60e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f76440057c0 tx=0x7f7644000bc0 comp rx=0 tx=0).stop 2024-07-26T07:46:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f765410b000 msgr2=0x7f76541a0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f765410b000 0x7f76541a0fc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f764c00cad0 tx=0x7f764c00ced0 comp rx=0 tx=0).stop 2024-07-26T07:46:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 shutdown_connections 2024-07-26T07:46:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76380b3bd0 0x7f76380b60e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f765410b000 0x7f76541a0fc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f765410a660 0x7f76541a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 --2- 172.21.15.97:0/95788043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76541095d0 0x7f765419fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 >> 172.21.15.97:0/95788043 conn(0x7f7654078970 msgr2=0x7f765410d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 shutdown_connections 2024-07-26T07:46:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:29.210+0000 7f7653200640 1 -- 172.21.15.97:0/95788043 wait complete. 2024-07-26T07:46:29.972 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:29 smithi097 ceph-mon[32231]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T07:46:29.972 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:29 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/95788043' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T07:46:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:29 smithi181 ceph-mon[35869]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T07:46:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:29 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/95788043' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T07:46:30.046 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":75,"num_osds":12,"num_up_osds":12,"osd_up_since":1721979987,"num_in_osds":12,"osd_in_since":1721979961,"num_remapped_pgs":0} 2024-07-26T07:46:30.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:46:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:29 smithi115 ceph-mon[35364]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T07:46:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:29 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/95788043' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T07:46:30.504 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:46:30.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:30.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:30.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:31.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:31.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:31.981 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:31 smithi097 ceph-mon[32231]: Detected new or changed devices on smithi181 2024-07-26T07:46:31.981 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:31 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 4272M 2024-07-26T07:46:32.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:31 smithi115 ceph-mon[35364]: Detected new or changed devices on smithi181 2024-07-26T07:46:32.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:31 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 4272M 2024-07-26T07:46:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:31 smithi181 ceph-mon[35869]: Detected new or changed devices on smithi181 2024-07-26T07:46:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:31 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 4272M 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 -- 172.21.15.97:0/1152661357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f810af10 msgr2=0x7fb2f810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 --2- 172.21.15.97:0/1152661357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f810af10 0x7fb2f810b320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009900 tx=0x7fb2e402a410 comp rx=0 tx=0).stop 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 -- 172.21.15.97:0/1152661357 shutdown_connections 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 --2- 172.21.15.97:0/1152661357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2f810b860 0x7fb2f8112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 --2- 172.21.15.97:0/1152661357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f810af10 0x7fb2f810b320 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 --2- 172.21.15.97:0/1152661357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f81055d0 0x7fb2f81059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 -- 172.21.15.97:0/1152661357 >> 172.21.15.97:0/1152661357 conn(0x7fb2f8101040 msgr2=0x7fb2f8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 -- 172.21.15.97:0/1152661357 shutdown_connections 2024-07-26T07:46:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.741+0000 7fb2fe600640 1 -- 172.21.15.97:0/1152661357 wait complete. 2024-07-26T07:46:32.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.743+0000 7fb2fe600640 1 Processor -- start 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.743+0000 7fb2fe600640 1 -- start start 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.743+0000 7fb2fe600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f81055d0 0x7fb2f819be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.743+0000 7fb2fe600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f810af10 0x7fb2f819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.743+0000 7fb2fe600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2f810b860 0x7fb2f81a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.743+0000 7fb2fe600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f81153a0 con 0x7fb2f81055d0 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.743+0000 7fb2fe600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f8115220 con 0x7fb2f810af10 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.743+0000 7fb2fe600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f8115520 con 0x7fb2f810b860 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f81055d0 0x7fb2f819be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f81055d0 0x7fb2f819be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36716/0 (socket says 172.21.15.97:36716) 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 -- 172.21.15.97:0/3181917925 learned_addr learned my addr 172.21.15.97:0/3181917925 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:46:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2f7e00640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f810af10 0x7fb2f819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fdc00640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2f810b860 0x7fb2f81a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 -- 172.21.15.97:0/3181917925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2f810b860 msgr2=0x7fb2f81a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2f810b860 0x7fb2f81a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 -- 172.21.15.97:0/3181917925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f810af10 msgr2=0x7fb2f819c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f810af10 0x7fb2f819c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 -- 172.21.15.97:0/3181917925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f81a1de0 con 0x7fb2f81055d0 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2f7e00640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f810af10 0x7fb2f819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.744+0000 7fb2fd200640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f81055d0 0x7fb2f819be70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00d950 tx=0x7fb2ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.745+0000 7fb2f5600640 1 -- 172.21.15.97:0/3181917925 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec017070 con 0x7fb2f81055d0 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.745+0000 7fb2f5600640 1 -- 172.21.15.97:0/3181917925 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb2ec009c30 con 0x7fb2f81055d0 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.745+0000 7fb2f5600640 1 -- 172.21.15.97:0/3181917925 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00f620 con 0x7fb2f81055d0 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.745+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f81a20d0 con 0x7fb2f81055d0 2024-07-26T07:46:32.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.745+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f81a8830 con 0x7fb2f81055d0 2024-07-26T07:46:32.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.746+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f81059e0 con 0x7fb2f81055d0 2024-07-26T07:46:32.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.748+0000 7fb2f5600640 1 -- 172.21.15.97:0/3181917925 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fb2ec005030 con 0x7fb2f81055d0 2024-07-26T07:46:32.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.750+0000 7fb2f5600640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb2e00b3bd0 0x7fb2e00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.750+0000 7fb2f5600640 1 -- 172.21.15.97:0/3181917925 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fb2ec0e34b0 con 0x7fb2f81055d0 2024-07-26T07:46:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.750+0000 7fb2f7e00640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb2e00b3bd0 0x7fb2e00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.751+0000 7fb2f7e00640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb2e00b3bd0 0x7fb2e00b60e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40057c0 tx=0x7fb2e4009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:32.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.752+0000 7fb2f5600640 1 -- 172.21.15.97:0/3181917925 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb2ec0aaf00 con 0x7fb2f81055d0 2024-07-26T07:46:32.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.953+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2f810d080 con 0x7fb2f81055d0 2024-07-26T07:46:32.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.955+0000 7fb2f5600640 1 -- 172.21.15.97:0/3181917925 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+18470 (secure 0 0 0) 0x7fb2ec009de0 con 0x7fb2f81055d0 2024-07-26T07:46:32.956 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:46:32.957 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":75,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:46:28.791564+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:46:32.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.957+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb2e00b3bd0 msgr2=0x7fb2e00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:32.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.957+0000 7fb2fe600640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb2e00b3bd0 0x7fb2e00b60e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40057c0 tx=0x7fb2e4009680 comp rx=0 tx=0).stop 2024-07-26T07:46:32.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f81055d0 msgr2=0x7fb2f819be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:32.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f81055d0 0x7fb2f819be70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00d950 tx=0x7fb2ec00de10 comp rx=0 tx=0).stop 2024-07-26T07:46:32.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 shutdown_connections 2024-07-26T07:46:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb2e00b3bd0 0x7fb2e00b60e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2f810b860 0x7fb2f81a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f810af10 0x7fb2f819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 --2- 172.21.15.97:0/3181917925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2f81055d0 0x7fb2f819be70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 >> 172.21.15.97:0/3181917925 conn(0x7fb2f8101040 msgr2=0x7fb2f8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 shutdown_connections 2024-07-26T07:46:32.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:32.958+0000 7fb2fe600640 1 -- 172.21.15.97:0/3181917925 wait complete. 2024-07-26T07:46:33.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:32 smithi115 ceph-mon[35364]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:32 smithi181 ceph-mon[35869]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:33.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:32 smithi097 ceph-mon[32231]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:33.776 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T07:41:16.123688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T07:46:33.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-07-26T07:46:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:33 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3181917925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:46:34.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:33 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3181917925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:46:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:33 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3181917925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:46:34.247 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:46:35.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:34 smithi097 ceph-mon[32231]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:34 smithi115 ceph-mon[35364]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:34 smithi181 ceph-mon[35869]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:36 smithi097 ceph-mon[32231]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:36 smithi115 ceph-mon[35364]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:36.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 -- 172.21.15.97:0/4282592649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 msgr2=0x7f7f6c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:36.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 --2- 172.21.15.97:0/4282592649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 0x7f7f6c10f740 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009900 tx=0x7f7f5402a410 comp rx=0 tx=0).stop 2024-07-26T07:46:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 -- 172.21.15.97:0/4282592649 shutdown_connections 2024-07-26T07:46:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 --2- 172.21.15.97:0/4282592649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f6c10fc80 0x7f7f6c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 --2- 172.21.15.97:0/4282592649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 0x7f7f6c10f740 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 --2- 172.21.15.97:0/4282592649 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f6c1099f0 0x7f7f6c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 -- 172.21.15.97:0/4282592649 >> 172.21.15.97:0/4282592649 conn(0x7f7f6c078ff0 msgr2=0x7f7f6c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 -- 172.21.15.97:0/4282592649 shutdown_connections 2024-07-26T07:46:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.479+0000 7f7f71c00640 1 -- 172.21.15.97:0/4282592649 wait complete. 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.480+0000 7f7f71c00640 1 Processor -- start 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.480+0000 7f7f71c00640 1 -- start start 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f71c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f6c1099f0 0x7f7f6c1a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f71c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 0x7f7f6c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f71c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f6c10fc80 0x7f7f6c1a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f71c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6c1197f0 con 0x7f7f6c10f330 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f71c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6c119670 con 0x7f7f6c1099f0 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f71c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6c119970 con 0x7f7f6c10fc80 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f6b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 0x7f7f6c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:36.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f6b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 0x7f7f6c1a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51834/0 (socket says 172.21.15.97:51834) 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f6b400640 1 -- 172.21.15.97:0/96889941 learned_addr learned my addr 172.21.15.97:0/96889941 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.481+0000 7f7f6be00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f6c1099f0 0x7f7f6c1a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f71200640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f6c10fc80 0x7f7f6c1a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f6b400640 1 -- 172.21.15.97:0/96889941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f6c10fc80 msgr2=0x7f7f6c1a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f6b400640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f6c10fc80 0x7f7f6c1a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f6b400640 1 -- 172.21.15.97:0/96889941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f6c1099f0 msgr2=0x7f7f6c1a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f6b400640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f6c1099f0 0x7f7f6c1a02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f6b400640 1 -- 172.21.15.97:0/96889941 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f6c1a6150 con 0x7f7f6c10f330 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f71200640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f6c10fc80 0x7f7f6c1a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:46:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f6b400640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 0x7f7f6c1a0840 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7f54005940 tx=0x7f7f540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:36.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:36 smithi181 ceph-mon[35869]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:36.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f68c00640 1 -- 172.21.15.97:0/96889941 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5403f030 con 0x7f7f6c10f330 2024-07-26T07:46:36.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.482+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f6c1a63e0 con 0x7f7f6c10f330 2024-07-26T07:46:36.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.483+0000 7f7f68c00640 1 -- 172.21.15.97:0/96889941 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f54003c90 con 0x7f7f6c10f330 2024-07-26T07:46:36.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.483+0000 7f7f68c00640 1 -- 172.21.15.97:0/96889941 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54035070 con 0x7f7f6c10f330 2024-07-26T07:46:36.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.483+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f6c1a6910 con 0x7f7f6c10f330 2024-07-26T07:46:36.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.484+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f6c109e00 con 0x7f7f6c10f330 2024-07-26T07:46:36.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.486+0000 7f7f68c00640 1 -- 172.21.15.97:0/96889941 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f7f54042030 con 0x7f7f6c10f330 2024-07-26T07:46:36.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.488+0000 7f7f68c00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7f500b3c80 0x7f7f500b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:36.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.488+0000 7f7f68c00640 1 -- 172.21.15.97:0/96889941 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f7f541041c0 con 0x7f7f6c10f330 2024-07-26T07:46:36.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.488+0000 7f7f6be00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7f500b3c80 0x7f7f500b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:36.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.489+0000 7f7f6be00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7f500b3c80 0x7f7f500b6190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c007350 tx=0x7f7f5c0072e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:36.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.490+0000 7f7f68c00640 1 -- 172.21.15.97:0/96889941 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7f6c1a6910 con 0x7f7f6c10f330 2024-07-26T07:46:36.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.686+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7f6c111ad0 con 0x7f7f6c10f330 2024-07-26T07:46:36.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.687+0000 7f7f68c00640 1 -- 172.21.15.97:0/96889941 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) ==== 93+0+10 (secure 0 0 0) 0x7f7f540cbc10 con 0x7f7f6c10f330 2024-07-26T07:46:36.689 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-07-26T07:46:36.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.691+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7f500b3c80 msgr2=0x7f7f500b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:36.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.691+0000 7f7f71c00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7f500b3c80 0x7f7f500b6190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c007350 tx=0x7f7f5c0072e0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.691+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 msgr2=0x7f7f6c1a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:36.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.691+0000 7f7f71c00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 0x7f7f6c1a0840 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7f54005940 tx=0x7f7f540042f0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.692+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 shutdown_connections 2024-07-26T07:46:36.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.692+0000 7f7f71c00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7f500b3c80 0x7f7f500b6190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.692+0000 7f7f71c00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f6c10fc80 0x7f7f6c1a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.692+0000 7f7f71c00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f6c10f330 0x7f7f6c1a0840 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.692+0000 7f7f71c00640 1 --2- 172.21.15.97:0/96889941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f6c1099f0 0x7f7f6c1a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:36.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.692+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 >> 172.21.15.97:0/96889941 conn(0x7f7f6c078ff0 msgr2=0x7f7f6c079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:36.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.692+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 shutdown_connections 2024-07-26T07:46:36.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:36.692+0000 7f7f71c00640 1 -- 172.21.15.97:0/96889941 wait complete. 2024-07-26T07:46:37.468 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:37 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/96889941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T07:46:37.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:37 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/96889941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T07:46:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:37 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/96889941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T07:46:37.543 INFO:tasks.cephadm:Adding mds.a on smithi097 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.d on smithi097 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.g on smithi097 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.j on smithi097 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.b on smithi115 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.e on smithi115 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.h on smithi115 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.k on smithi115 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.c on smithi181 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.f on smithi181 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.i on smithi181 2024-07-26T07:46:37.544 INFO:tasks.cephadm:Adding mds.l on smithi181 2024-07-26T07:46:37.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph orch apply mds all '12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l' 2024-07-26T07:46:38.001 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T07:46:38.456 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:38 smithi181 ceph-mon[35869]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:38.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:38 smithi115 ceph-mon[35364]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:38.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:38 smithi097 ceph-mon[32231]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:40.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 -- 172.21.15.181:0/2722471924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106c00 msgr2=0x7ff36010d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:40.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 --2- 172.21.15.181:0/2722471924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106c00 0x7ff36010d5a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff358009ca0 tx=0x7ff35802a950 comp rx=0 tx=0).stop 2024-07-26T07:46:40.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 -- 172.21.15.181:0/2722471924 shutdown_connections 2024-07-26T07:46:40.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 --2- 172.21.15.181:0/2722471924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106c00 0x7ff36010d5a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 --2- 172.21.15.181:0/2722471924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff360106260 0x7ff360106670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 --2- 172.21.15.181:0/2722471924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3601051d0 0x7ff360105640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 -- 172.21.15.181:0/2722471924 >> 172.21.15.181:0/2722471924 conn(0x7ff360100b60 msgr2=0x7ff360102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:40.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 -- 172.21.15.181:0/2722471924 shutdown_connections 2024-07-26T07:46:40.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.200+0000 7ff364a00640 1 -- 172.21.15.181:0/2722471924 wait complete. 2024-07-26T07:46:40.213 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.201+0000 7ff364a00640 1 Processor -- start 2024-07-26T07:46:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.201+0000 7ff364a00640 1 -- start start 2024-07-26T07:46:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff364a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3601051d0 0x7ff36019b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff364a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106260 0x7ff36019bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff364a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff360106c00 0x7ff36019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff364a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff360110d20 con 0x7ff3601051d0 2024-07-26T07:46:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff364a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff360110ba0 con 0x7ff360106c00 2024-07-26T07:46:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff364a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff360110ea0 con 0x7ff360106260 2024-07-26T07:46:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff356a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106260 0x7ff36019bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff356a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106260 0x7ff36019bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48286/0 (socket says 172.21.15.181:48286) 2024-07-26T07:46:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff356a00640 1 -- 172.21.15.181:0/1695956689 learned_addr learned my addr 172.21.15.181:0/1695956689 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-07-26T07:46:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff35f400640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3601051d0 0x7ff36019b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.202+0000 7ff35fe00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff360106c00 0x7ff36019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff356a00640 1 -- 172.21.15.181:0/1695956689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff360106c00 msgr2=0x7ff36019cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff356a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff360106c00 0x7ff36019cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff356a00640 1 -- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3601051d0 msgr2=0x7ff36019b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:40.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff356a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3601051d0 0x7ff36019b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff356a00640 1 -- 172.21.15.181:0/1695956689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36019d3b0 con 0x7ff360106260 2024-07-26T07:46:40.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff356a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106260 0x7ff36019bff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff34800b4a0 tx=0x7ff34800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:40.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff35cc00640 1 -- 172.21.15.181:0/1695956689 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348014070 con 0x7ff360106260 2024-07-26T07:46:40.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff35cc00640 1 -- 172.21.15.181:0/1695956689 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff348010aa0 con 0x7ff360106260 2024-07-26T07:46:40.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36019d6a0 con 0x7ff360106260 2024-07-26T07:46:40.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.203+0000 7ff35cc00640 1 -- 172.21.15.181:0/1695956689 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348017b70 con 0x7ff360106260 2024-07-26T07:46:40.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.204+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3601a3da0 con 0x7ff360106260 2024-07-26T07:46:40.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.205+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36010c390 con 0x7ff360106260 2024-07-26T07:46:40.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.207+0000 7ff35cc00640 1 -- 172.21.15.181:0/1695956689 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7ff348017070 con 0x7ff360106260 2024-07-26T07:46:40.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.209+0000 7ff35cc00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3440b3c50 0x7ff3440b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:40.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.209+0000 7ff35cc00640 1 -- 172.21.15.181:0/1695956689 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7ff3480e2f30 con 0x7ff360106260 2024-07-26T07:46:40.220 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.209+0000 7ff35f400640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3440b3c50 0x7ff3440b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:40.220 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.211+0000 7ff35f400640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3440b3c50 0x7ff3440b6160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff3500058c0 tx=0x7ff35000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:40.221 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.213+0000 7ff35cc00640 1 -- 172.21.15.181:0/1695956689 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff3480aa980 con 0x7ff360106260 2024-07-26T07:46:40.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.424+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}) -- 0x7ff360063430 con 0x7ff3440b3c50 2024-07-26T07:46:40.426 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:40 smithi181 ceph-mon[35869]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:40.444 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mds.all update... 2024-07-26T07:46:40.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.443+0000 7ff35cc00640 1 -- 172.21.15.181:0/1695956689 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7ff360063430 con 0x7ff3440b3c50 2024-07-26T07:46:40.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3440b3c50 msgr2=0x7ff3440b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:40.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3440b3c50 0x7ff3440b6160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff3500058c0 tx=0x7ff35000a4c0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106260 msgr2=0x7ff36019bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:40.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106260 0x7ff36019bff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff34800b4a0 tx=0x7ff34800b960 comp rx=0 tx=0).stop 2024-07-26T07:46:40.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 shutdown_connections 2024-07-26T07:46:40.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3440b3c50 0x7ff3440b6160 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff360106c00 0x7ff36019cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff360106260 0x7ff36019bff0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 --2- 172.21.15.181:0/1695956689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3601051d0 0x7ff36019b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:40.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.446+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 >> 172.21.15.181:0/1695956689 conn(0x7ff360100b60 msgr2=0x7ff360109560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:40.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.447+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 shutdown_connections 2024-07-26T07:46:40.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-07-26T07:46:40.447+0000 7ff364a00640 1 -- 172.21.15.181:0/1695956689 wait complete. 2024-07-26T07:46:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:40 smithi115 ceph-mon[35364]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:40 smithi097 ceph-mon[32231]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:41.217 DEBUG:teuthology.orchestra.run.smithi097:mds.a> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.a.service 2024-07-26T07:46:41.226 DEBUG:teuthology.orchestra.run.smithi097:mds.d> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service 2024-07-26T07:46:41.271 DEBUG:teuthology.orchestra.run.smithi097:mds.g> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.g.service 2024-07-26T07:46:41.318 DEBUG:teuthology.orchestra.run.smithi097:mds.j> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service 2024-07-26T07:46:41.363 DEBUG:teuthology.orchestra.run.smithi115:mds.b> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.b.service 2024-07-26T07:46:41.372 DEBUG:teuthology.orchestra.run.smithi115:mds.e> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service 2024-07-26T07:46:41.418 DEBUG:teuthology.orchestra.run.smithi115:mds.h> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.h.service 2024-07-26T07:46:41.466 DEBUG:teuthology.orchestra.run.smithi115:mds.k> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.k.service 2024-07-26T07:46:41.511 DEBUG:teuthology.orchestra.run.smithi181:mds.c> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.c.service 2024-07-26T07:46:41.517 DEBUG:teuthology.orchestra.run.smithi181:mds.f> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service 2024-07-26T07:46:41.564 DEBUG:teuthology.orchestra.run.smithi181:mds.i> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.i.service 2024-07-26T07:46:41.610 DEBUG:teuthology.orchestra.run.smithi181:mds.l> sudo journalctl -f -n 0 -u ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.l.service 2024-07-26T07:46:41.655 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-07-26T07:46:41.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:46:41.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:46:41.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: Saving service mds.all spec with placement smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-07-26T07:46:41.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:41.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:41.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:41.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:41.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:41.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:41.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:41.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:41.706 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:41 smithi181 ceph-mon[35869]: Deploying daemon mds.l on smithi181 2024-07-26T07:46:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:46:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: Saving service mds.all spec with placement smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-07-26T07:46:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:41 smithi115 ceph-mon[35364]: Deploying daemon mds.l on smithi181 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: Saving service mds.all spec with placement smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:41.734 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:41 smithi097 ceph-mon[32231]: Deploying daemon mds.l on smithi181 2024-07-26T07:46:42.113 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:46:42.704 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:42 smithi097 ceph-mon[32231]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:42.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:42 smithi115 ceph-mon[35364]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:42.724 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:42 smithi181 ceph-mon[35869]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:43.892 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:43 smithi181 systemd[1]: Starting Ceph mds.l for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:46:44.159 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:43 smithi181 podman[65272]: 2024-07-26 07:46:43.795221631 +0000 UTC m=+0.030973111 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:46:44.159 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:43 smithi181 podman[65272]: 2024-07-26 07:46:43.900694525 +0000 UTC m=+0.136445995 container create 4af81ce494c911f71f228810b462b872eea01304e764a091a07a2b27bd8b9485 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-l, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:46:44.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.398+0000 7f2a43200640 1 -- 172.21.15.97:0/1959087304 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 msgr2=0x7f2a44109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:44.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.398+0000 7f2a43200640 1 --2- 172.21.15.97:0/1959087304 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 0x7f2a44109e00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2a34009900 tx=0x7f2a3402a3f0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.398+0000 7f2a43200640 1 -- 172.21.15.97:0/1959087304 shutdown_connections 2024-07-26T07:46:44.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.398+0000 7f2a43200640 1 --2- 172.21.15.97:0/1959087304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a4410fc80 0x7f2a441164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.398+0000 7f2a43200640 1 --2- 172.21.15.97:0/1959087304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a4410f330 0x7f2a4410f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.398+0000 7f2a43200640 1 --2- 172.21.15.97:0/1959087304 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 0x7f2a44109e00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.398+0000 7f2a43200640 1 -- 172.21.15.97:0/1959087304 >> 172.21.15.97:0/1959087304 conn(0x7f2a44078ff0 msgr2=0x7f2a44107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:44.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.399+0000 7f2a43200640 1 -- 172.21.15.97:0/1959087304 shutdown_connections 2024-07-26T07:46:44.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.399+0000 7f2a43200640 1 -- 172.21.15.97:0/1959087304 wait complete. 2024-07-26T07:46:44.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.400+0000 7f2a43200640 1 Processor -- start 2024-07-26T07:46:44.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a43200640 1 -- start start 2024-07-26T07:46:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a43200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 0x7f2a441a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a43200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a4410f330 0x7f2a441a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a43200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a4410fc80 0x7f2a441a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a43200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2a44119840 con 0x7f2a441099f0 2024-07-26T07:46:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a43200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2a441196c0 con 0x7f2a4410fc80 2024-07-26T07:46:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a43200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2a441199c0 con 0x7f2a4410f330 2024-07-26T07:46:44.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a41e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 0x7f2a441a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:44.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a41e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 0x7f2a441a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51866/0 (socket says 172.21.15.97:51866) 2024-07-26T07:46:44.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.401+0000 7f2a41e00640 1 -- 172.21.15.97:0/4168040125 learned_addr learned my addr 172.21.15.97:0/4168040125 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:46:44.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.402+0000 7f2a42800640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a4410fc80 0x7f2a441a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:44.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.402+0000 7f2a41e00640 1 -- 172.21.15.97:0/4168040125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a4410f330 msgr2=0x7f2a441a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:44.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.402+0000 7f2a41e00640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a4410f330 0x7f2a441a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.402+0000 7f2a41e00640 1 -- 172.21.15.97:0/4168040125 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a4410fc80 msgr2=0x7f2a441a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:44.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.402+0000 7f2a41e00640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a4410fc80 0x7f2a441a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.402+0000 7f2a41e00640 1 -- 172.21.15.97:0/4168040125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a441a6140 con 0x7f2a441099f0 2024-07-26T07:46:44.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.402+0000 7f2a42800640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a4410fc80 0x7f2a441a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:46:44.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.403+0000 7f2a41e00640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 0x7f2a441a0290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a3402a910 tx=0x7f2a34003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:44.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.403+0000 7f2a3b400640 1 -- 172.21.15.97:0/4168040125 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3403f070 con 0x7f2a441099f0 2024-07-26T07:46:44.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.403+0000 7f2a3b400640 1 -- 172.21.15.97:0/4168040125 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a34003e40 con 0x7f2a441099f0 2024-07-26T07:46:44.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.403+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a441a63d0 con 0x7f2a441099f0 2024-07-26T07:46:44.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.403+0000 7f2a3b400640 1 -- 172.21.15.97:0/4168040125 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a34035030 con 0x7f2a441099f0 2024-07-26T07:46:44.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.404+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a441a6900 con 0x7f2a441099f0 2024-07-26T07:46:44.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.405+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a44109e00 con 0x7f2a441099f0 2024-07-26T07:46:44.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:44 smithi097 ceph-mon[32231]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:44.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.408+0000 7f2a3b400640 1 -- 172.21.15.97:0/4168040125 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f2a34042070 con 0x7f2a441099f0 2024-07-26T07:46:44.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.409+0000 7f2a3b400640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a280b3c80 0x7f2a280b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:44.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.410+0000 7f2a3b400640 1 -- 172.21.15.97:0/4168040125 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f2a34103fe0 con 0x7f2a441099f0 2024-07-26T07:46:44.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.410+0000 7f2a39400640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a280b3c80 0x7f2a280b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:44.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.411+0000 7f2a39400640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a280b3c80 0x7f2a280b6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2a4410fb10 tx=0x7f2a2c005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:44.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.416+0000 7f2a3b400640 1 -- 172.21.15.97:0/4168040125 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2a441a6900 con 0x7f2a441099f0 2024-07-26T07:46:44.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:44 smithi115 ceph-mon[35364]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:44 smithi181 ceph-mon[35869]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:44.482 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:44 smithi181 podman[65272]: 2024-07-26 07:46:44.197002487 +0000 UTC m=+0.432753995 container init 4af81ce494c911f71f228810b462b872eea01304e764a091a07a2b27bd8b9485 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T07:46:44.483 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:44 smithi181 podman[65272]: 2024-07-26 07:46:44.210259332 +0000 UTC m=+0.446010870 container start 4af81ce494c911f71f228810b462b872eea01304e764a091a07a2b27bd8b9485 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-l, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-07-26T07:46:44.483 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:44 smithi181 bash[65272]: 4af81ce494c911f71f228810b462b872eea01304e764a091a07a2b27bd8b9485 2024-07-26T07:46:44.483 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:44 smithi181 systemd[1]: Started Ceph mds.l for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:46:44.483 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:44 smithi181 ceph-mds[65291]: -- 172.21.15.181:0/3752709656 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c7a29b860 con 0x559c7e43da80 2024-07-26T07:46:44.483 INFO:journalctl@ceph.mds.l.smithi181.stdout:Jul 26 07:46:44 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-l[65287]: starting mds.l at 2024-07-26T07:46:44.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.649+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a441a13c0 con 0x7f2a441099f0 2024-07-26T07:46:44.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.651+0000 7f2a3b400640 1 -- 172.21.15.97:0/4168040125 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7f2a340cba50 con 0x7f2a441099f0 2024-07-26T07:46:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:46:44.652 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":2,"btime":"2024-07-26T07:46:44:338121+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-26T07:46:44.652 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 2 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.653+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a280b3c80 msgr2=0x7f2a280b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.653+0000 7f2a43200640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a280b3c80 0x7f2a280b6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2a4410fb10 tx=0x7f2a2c005c70 comp rx=0 tx=0).stop 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.653+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 msgr2=0x7f2a441a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.653+0000 7f2a43200640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 0x7f2a441a0290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a3402a910 tx=0x7f2a34003c20 comp rx=0 tx=0).stop 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.654+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 shutdown_connections 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.654+0000 7f2a43200640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a280b3c80 0x7f2a280b6190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.654+0000 7f2a43200640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a4410fc80 0x7f2a441a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.654+0000 7f2a43200640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a4410f330 0x7f2a441a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.654+0000 7f2a43200640 1 --2- 172.21.15.97:0/4168040125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a441099f0 0x7f2a441a0290 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.654+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 >> 172.21.15.97:0/4168040125 conn(0x7f2a44078ff0 msgr2=0x7f2a44079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.654+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 shutdown_connections 2024-07-26T07:46:44.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:44.654+0000 7f2a43200640 1 -- 172.21.15.97:0/4168040125 wait complete. 2024-07-26T07:46:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:boot 2024-07-26T07:46:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: fsmap 1 up:standby 2024-07-26T07:46:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T07:46:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:45.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:45.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:45.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: Deploying daemon mds.k on smithi115 2024-07-26T07:46:45.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:45 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4168040125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:boot 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: fsmap 1 up:standby 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: Deploying daemon mds.k on smithi115 2024-07-26T07:46:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:45 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4168040125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:boot 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: fsmap 1 up:standby 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: Deploying daemon mds.k on smithi115 2024-07-26T07:46:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:45 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4168040125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:46.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:46 smithi115 ceph-mon[35364]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:46 smithi181 ceph-mon[35869]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:46.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:46 smithi097 ceph-mon[32231]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:47.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:46:47.724 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:47 smithi115 systemd[1]: Starting Ceph mds.k for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:46:47.820 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:46:48.049 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:47 smithi115 podman[65193]: 2024-07-26 07:46:47.63694547 +0000 UTC m=+0.031158772 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:46:48.049 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:47 smithi115 podman[65193]: 2024-07-26 07:46:47.76048706 +0000 UTC m=+0.154700380 container create a7747372ca68603d4423886415a8244c89f9509a3364d39dd5a254168924c1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0) 2024-07-26T07:46:48.049 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:48 smithi115 podman[65193]: 2024-07-26 07:46:48.047561022 +0000 UTC m=+0.441774415 container init a7747372ca68603d4423886415a8244c89f9509a3364d39dd5a254168924c1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-07-26T07:46:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:48 smithi097 ceph-mon[32231]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:48 smithi115 ceph-mon[35364]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:48.474 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:48 smithi115 podman[65193]: 2024-07-26 07:46:48.060840116 +0000 UTC m=+0.455053410 container start a7747372ca68603d4423886415a8244c89f9509a3364d39dd5a254168924c1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:46:48.474 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:48 smithi115 bash[65193]: a7747372ca68603d4423886415a8244c89f9509a3364d39dd5a254168924c1fd 2024-07-26T07:46:48.474 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:48 smithi115 systemd[1]: Started Ceph mds.k for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:46:48.474 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:48 smithi115 ceph-mds[65213]: -- 172.21.15.115:0/2293616147 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c467661860 con 0x55c46b803180 2024-07-26T07:46:48.475 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:46:48 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: starting mds.k at 2024-07-26T07:46:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:48 smithi181 ceph-mon[35869]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:49.440 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: Deploying daemon mds.j on smithi097 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] up:boot 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: fsmap 2 up:standby 2024-07-26T07:46:49.441 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T07:46:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: Deploying daemon mds.j on smithi097 2024-07-26T07:46:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] up:boot 2024-07-26T07:46:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: fsmap 2 up:standby 2024-07-26T07:46:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T07:46:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: Deploying daemon mds.j on smithi097 2024-07-26T07:46:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] up:boot 2024-07-26T07:46:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: fsmap 2 up:standby 2024-07-26T07:46:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T07:46:50.514 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:50 smithi097 ceph-mon[32231]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:50.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 -- 172.21.15.97:0/2991583479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634107bf0 msgr2=0x7fb634108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:50.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 --2- 172.21.15.97:0/2991583479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634107bf0 0x7fb634108000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009ca0 tx=0x7fb61c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 -- 172.21.15.97:0/2991583479 shutdown_connections 2024-07-26T07:46:50.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 --2- 172.21.15.97:0/2991583479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb634108540 0x7fb63410ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 --2- 172.21.15.97:0/2991583479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634107bf0 0x7fb634108000 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 --2- 172.21.15.97:0/2991583479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6341022b0 0x7fb6341026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 -- 172.21.15.97:0/2991583479 >> 172.21.15.97:0/2991583479 conn(0x7fb6340fdce0 msgr2=0x7fb634100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:50.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 -- 172.21.15.97:0/2991583479 shutdown_connections 2024-07-26T07:46:50.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.610+0000 7fb639c00640 1 -- 172.21.15.97:0/2991583479 wait complete. 2024-07-26T07:46:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.611+0000 7fb639c00640 1 Processor -- start 2024-07-26T07:46:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639c00640 1 -- start start 2024-07-26T07:46:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6341022b0 0x7fb63419bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb634107bf0 0x7fb63419c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634108540 0x7fb6341a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:50.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb634112120 con 0x7fb634108540 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb634111fa0 con 0x7fb634107bf0 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb6341122a0 con 0x7fb6341022b0 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634108540 0x7fb6341a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634108540 0x7fb6341a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60938/0 (socket says 172.21.15.97:60938) 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.612+0000 7fb639200640 1 -- 172.21.15.97:0/3222754772 learned_addr learned my addr 172.21.15.97:0/3222754772 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb633e00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6341022b0 0x7fb63419bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb633400640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb634107bf0 0x7fb63419c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb639200640 1 -- 172.21.15.97:0/3222754772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6341022b0 msgr2=0x7fb63419bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb639200640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6341022b0 0x7fb63419bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb639200640 1 -- 172.21.15.97:0/3222754772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb634107bf0 msgr2=0x7fb63419c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:50.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb639200640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb634107bf0 0x7fb63419c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb639200640 1 -- 172.21.15.97:0/3222754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6341a1d80 con 0x7fb634108540 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb633e00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6341022b0 0x7fb63419bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.613+0000 7fb639200640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634108540 0x7fb6341a1710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb62400c970 tx=0x7fb62400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.614+0000 7fb630c00640 1 -- 172.21.15.97:0/3222754772 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb624007bb0 con 0x7fb634108540 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.614+0000 7fb630c00640 1 -- 172.21.15.97:0/3222754772 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb624007d50 con 0x7fb634108540 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.614+0000 7fb630c00640 1 -- 172.21.15.97:0/3222754772 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb624018660 con 0x7fb634108540 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.614+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6341a2070 con 0x7fb634108540 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.614+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6341a2530 con 0x7fb634108540 2024-07-26T07:46:50.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.615+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb634103530 con 0x7fb634108540 2024-07-26T07:46:50.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.617+0000 7fb630c00640 1 -- 172.21.15.97:0/3222754772 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fb62400f970 con 0x7fb634108540 2024-07-26T07:46:50.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.619+0000 7fb630c00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6180b3c80 0x7fb6180b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.619+0000 7fb630c00640 1 -- 172.21.15.97:0/3222754772 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fb6240e3810 con 0x7fb634108540 2024-07-26T07:46:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.619+0000 7fb633e00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6180b3c80 0x7fb6180b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.620+0000 7fb633e00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6180b3c80 0x7fb6180b6190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009930 tx=0x7fb62c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.623+0000 7fb630c00640 1 -- 172.21.15.97:0/3222754772 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb624014030 con 0x7fb634108540 2024-07-26T07:46:50.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:50 smithi115 ceph-mon[35364]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:50 smithi181 ceph-mon[35869]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:50.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.864+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb63402e0c0 con 0x7fb634108540 2024-07-26T07:46:50.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.865+0000 7fb630c00640 1 -- 172.21.15.97:0/3222754772 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2266 (secure 0 0 0) 0x7fb6240ab260 con 0x7fb634108540 2024-07-26T07:46:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:46:50.867 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":3,"btime":"2024-07-26T07:46:49:156773+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34517,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-07-26T07:46:50.867 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 3 2024-07-26T07:46:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6180b3c80 msgr2=0x7fb6180b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:50.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6180b3c80 0x7fb6180b6190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009930 tx=0x7fb62c008040 comp rx=0 tx=0).stop 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634108540 msgr2=0x7fb6341a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634108540 0x7fb6341a1710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb62400c970 tx=0x7fb62400ce30 comp rx=0 tx=0).stop 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 shutdown_connections 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6180b3c80 0x7fb6180b6190 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb634108540 0x7fb6341a1710 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb634107bf0 0x7fb63419c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 --2- 172.21.15.97:0/3222754772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6341022b0 0x7fb63419bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.868+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 >> 172.21.15.97:0/3222754772 conn(0x7fb6340fdce0 msgr2=0x7fb634107240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.869+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 shutdown_connections 2024-07-26T07:46:50.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:50.869+0000 7fb639c00640 1 -- 172.21.15.97:0/3222754772 wait complete. 2024-07-26T07:46:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:51 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3222754772' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:51 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3222754772' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:51 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3222754772' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:52.522 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:52 smithi097 ceph-mon[32231]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:52.523 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 07:46:52 smithi097 podman[75151]: 2024-07-26 07:46:52.37280074 +0000 UTC m=+0.165303969 container create e1efe33a3a8a0bd1a0f8b95ef23912bc23661ea65d8626d9ae2ab2be75708d82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0) 2024-07-26T07:46:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:52 smithi115 ceph-mon[35364]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:52 smithi181 ceph-mon[35869]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:53.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 07:46:52 smithi097 podman[75151]: 2024-07-26 07:46:52.601284214 +0000 UTC m=+0.393787449 container init e1efe33a3a8a0bd1a0f8b95ef23912bc23661ea65d8626d9ae2ab2be75708d82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T07:46:53.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 07:46:52 smithi097 podman[75151]: 2024-07-26 07:46:52.611075634 +0000 UTC m=+0.403578890 container start e1efe33a3a8a0bd1a0f8b95ef23912bc23661ea65d8626d9ae2ab2be75708d82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-07-26T07:46:53.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 07:46:52 smithi097 bash[75151]: e1efe33a3a8a0bd1a0f8b95ef23912bc23661ea65d8626d9ae2ab2be75708d82 2024-07-26T07:46:53.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 07:46:52 smithi097 systemd[1]: Started Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:46:53.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 07:46:52 smithi097 ceph-mds[75169]: -- 172.21.15.97:0/3338951696 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cd52b0fc20 con 0x55cd56cb1600 2024-07-26T07:46:53.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 07:46:52 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: starting mds.j at 2024-07-26T07:46:53.906 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: Deploying daemon mds.d on smithi097 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:boot 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: fsmap 3 up:standby 2024-07-26T07:46:53.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T07:46:54.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:46:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: Deploying daemon mds.d on smithi097 2024-07-26T07:46:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:boot 2024-07-26T07:46:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: fsmap 3 up:standby 2024-07-26T07:46:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T07:46:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: Deploying daemon mds.d on smithi097 2024-07-26T07:46:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:boot 2024-07-26T07:46:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: fsmap 3 up:standby 2024-07-26T07:46:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T07:46:54.900 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:46:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:54 smithi097 ceph-mon[32231]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:55.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:54 smithi115 ceph-mon[35364]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:54 smithi181 ceph-mon[35869]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:56 smithi097 ceph-mon[32231]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:56 smithi115 ceph-mon[35364]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:56 smithi181 ceph-mon[35869]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:56.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:46:56 smithi097 podman[75547]: 2024-07-26 07:46:56.420946672 +0000 UTC m=+0.032531066 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:46:57.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:46:56 smithi097 podman[75547]: 2024-07-26 07:46:56.769934771 +0000 UTC m=+0.381519204 container init c053db36be3d8ab03a75401de14c75e977d1448db2687db5924d888ec9b2a3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T07:46:57.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:46:56 smithi097 podman[75547]: 2024-07-26 07:46:56.78150242 +0000 UTC m=+0.393086823 container start c053db36be3d8ab03a75401de14c75e977d1448db2687db5924d888ec9b2a3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T07:46:57.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:46:56 smithi097 bash[75547]: c053db36be3d8ab03a75401de14c75e977d1448db2687db5924d888ec9b2a3ec 2024-07-26T07:46:57.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:46:56 smithi097 systemd[1]: Started Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:46:57.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:46:56 smithi097 ceph-mds[75584]: -- 172.21.15.97:0/1553577274 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eca5cb7860 con 0x55eca9e59180 2024-07-26T07:46:57.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:46:56 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: starting mds.d at 2024-07-26T07:46:57.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 -- 172.21.15.97:0/3945494709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 msgr2=0x7fb7d01164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:57.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3945494709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 0x7fb7d01164b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009ca0 tx=0x7fb7c802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 -- 172.21.15.97:0/3945494709 shutdown_connections 2024-07-26T07:46:57.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3945494709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 0x7fb7d01164b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3945494709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb7d010f330 0x7fb7d010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3945494709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d01099f0 0x7fb7d0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 -- 172.21.15.97:0/3945494709 >> 172.21.15.97:0/3945494709 conn(0x7fb7d0078ff0 msgr2=0x7fb7d0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:57.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 -- 172.21.15.97:0/3945494709 shutdown_connections 2024-07-26T07:46:57.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.526+0000 7fb7d6800640 1 -- 172.21.15.97:0/3945494709 wait complete. 2024-07-26T07:46:57.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.527+0000 7fb7d6800640 1 Processor -- start 2024-07-26T07:46:57.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.528+0000 7fb7d6800640 1 -- start start 2024-07-26T07:46:57.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.528+0000 7fb7d6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d01099f0 0x7fb7d01a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:57.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.528+0000 7fb7d6800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb7d010f330 0x7fb7d01a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:57.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.528+0000 7fb7d6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 0x7fb7d01a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:57.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.528+0000 7fb7d6800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d0119840 con 0x7fb7d010fc80 2024-07-26T07:46:57.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.528+0000 7fb7d6800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d01196c0 con 0x7fb7d010f330 2024-07-26T07:46:57.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.528+0000 7fb7d6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d01199c0 con 0x7fb7d01099f0 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.528+0000 7fb7d5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 0x7fb7d01a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 0x7fb7d01a5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37684/0 (socket says 172.21.15.97:37684) 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d5e00640 1 -- 172.21.15.97:0/3736533075 learned_addr learned my addr 172.21.15.97:0/3736533075 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d4a00640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb7d010f330 0x7fb7d01a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d5400640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d01099f0 0x7fb7d01a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d5e00640 1 -- 172.21.15.97:0/3736533075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d01099f0 msgr2=0x7fb7d01a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d5e00640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d01099f0 0x7fb7d01a0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d5e00640 1 -- 172.21.15.97:0/3736533075 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb7d010f330 msgr2=0x7fb7d01a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d5e00640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb7d010f330 0x7fb7d01a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.529+0000 7fb7d5e00640 1 -- 172.21.15.97:0/3736533075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d01a61a0 con 0x7fb7d010fc80 2024-07-26T07:46:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.530+0000 7fb7d5e00640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 0x7fb7d01a5a70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005e30 tx=0x7fb7c80093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.530+0000 7fb7ce000640 1 -- 172.21.15.97:0/3736533075 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8032d10 con 0x7fb7d010fc80 2024-07-26T07:46:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.530+0000 7fb7ce000640 1 -- 172.21.15.97:0/3736533075 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb7c8032eb0 con 0x7fb7d010fc80 2024-07-26T07:46:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.530+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d01a6430 con 0x7fb7d010fc80 2024-07-26T07:46:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.530+0000 7fb7ce000640 1 -- 172.21.15.97:0/3736533075 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8030b20 con 0x7fb7d010fc80 2024-07-26T07:46:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.530+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d01acbf0 con 0x7fb7d010fc80 2024-07-26T07:46:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.531+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d0109e00 con 0x7fb7d010fc80 2024-07-26T07:46:57.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.534+0000 7fb7ce000640 1 -- 172.21.15.97:0/3736533075 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fb7c8030ce0 con 0x7fb7d010fc80 2024-07-26T07:46:57.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.535+0000 7fb7ce000640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb7b80b3bd0 0x7fb7b80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:46:57.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.535+0000 7fb7ce000640 1 -- 172.21.15.97:0/3736533075 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fb7c8104c10 con 0x7fb7d010fc80 2024-07-26T07:46:57.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.535+0000 7fb7d5400640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb7b80b3bd0 0x7fb7b80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:46:57.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.536+0000 7fb7d5400640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb7b80b3bd0 0x7fb7b80b60e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40058c0 tx=0x7fb7c400a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:46:57.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.540+0000 7fb7ce000640 1 -- 172.21.15.97:0/3736533075 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb7c80cc660 con 0x7fb7d010fc80 2024-07-26T07:46:57.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.777+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7d0111ad0 con 0x7fb7d010fc80 2024-07-26T07:46:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.778+0000 7fb7ce000640 1 -- 172.21.15.97:0/3736533075 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3916 (secure 0 0 0) 0x7fb7c80cbf60 con 0x7fb7d010fc80 2024-07-26T07:46:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:46:57.806 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":5,"btime":"2024-07-26T07:46:57:159785+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34517,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 5 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb7b80b3bd0 msgr2=0x7fb7b80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb7b80b3bd0 0x7fb7b80b60e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40058c0 tx=0x7fb7c400a4c0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 msgr2=0x7fb7d01a5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 0x7fb7d01a5a70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005e30 tx=0x7fb7c80093d0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 shutdown_connections 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb7b80b3bd0 0x7fb7b80b60e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7d010fc80 0x7fb7d01a5a70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb7d010f330 0x7fb7d01a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 --2- 172.21.15.97:0/3736533075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d01099f0 0x7fb7d01a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 >> 172.21.15.97:0/3736533075 conn(0x7fb7d0078ff0 msgr2=0x7fb7d0079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 shutdown_connections 2024-07-26T07:46:57.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:46:57.781+0000 7fb7d6800640 1 -- 172.21.15.97:0/3736533075 wait complete. 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: Deploying daemon mds.e on smithi115 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2971688113,v1:172.21.15.97:6837/2971688113] up:boot 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: fsmap 4 up:standby 2024-07-26T07:46:58.167 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T07:46:58.168 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:57 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3736533075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:58.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: Deploying daemon mds.e on smithi115 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2971688113,v1:172.21.15.97:6837/2971688113] up:boot 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: fsmap 4 up:standby 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T07:46:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:57 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3736533075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: Deploying daemon mds.e on smithi115 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2971688113,v1:172.21.15.97:6837/2971688113] up:boot 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: fsmap 4 up:standby 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T07:46:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:57 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3736533075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:46:59.241 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:46:58 smithi115 ceph-mon[35364]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:59.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:46:58 smithi097 ceph-mon[32231]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:46:59.262 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:46:58 smithi181 ceph-mon[35869]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:00 smithi115 ceph-mon[35364]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:00 smithi181 ceph-mon[35869]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:00.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:00 smithi097 ceph-mon[32231]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:00.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:47:00.620 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:47:00 smithi115 podman[65491]: 2024-07-26 07:47:00.328265299 +0000 UTC m=+0.134348634 container create 0940ab2ad1c41a58e461c71ee7a35b17ce5ab826838f35aa44c0ec47ed9cfd0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-07-26T07:47:00.621 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:47:00 smithi115 podman[65491]: 2024-07-26 07:47:00.618829982 +0000 UTC m=+0.424913360 container init 0940ab2ad1c41a58e461c71ee7a35b17ce5ab826838f35aa44c0ec47ed9cfd0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-26T07:47:00.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:47:00 smithi115 podman[65491]: 2024-07-26 07:47:00.632401905 +0000 UTC m=+0.438485243 container start 0940ab2ad1c41a58e461c71ee7a35b17ce5ab826838f35aa44c0ec47ed9cfd0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-07-26T07:47:00.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:47:00 smithi115 bash[65491]: 0940ab2ad1c41a58e461c71ee7a35b17ce5ab826838f35aa44c0ec47ed9cfd0b 2024-07-26T07:47:00.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:47:00 smithi115 systemd[1]: Started Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:47:00.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:47:00 smithi115 ceph-mds[65511]: -- 172.21.15.115:0/1983097853 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555a32151c20 con 0x555a362f3180 2024-07-26T07:47:00.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:47:00 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: starting mds.e at 2024-07-26T07:47:01.019 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:01.850 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:01.850 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:01.850 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:01.850 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:01.850 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:01.850 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:01.851 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: Deploying daemon mds.f on smithi181 2024-07-26T07:47:01.851 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] up:boot 2024-07-26T07:47:01.851 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: fsmap 5 up:standby 2024-07-26T07:47:01.851 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T07:47:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: Deploying daemon mds.f on smithi181 2024-07-26T07:47:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] up:boot 2024-07-26T07:47:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: fsmap 5 up:standby 2024-07-26T07:47:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T07:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: Deploying daemon mds.f on smithi181 2024-07-26T07:47:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] up:boot 2024-07-26T07:47:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: fsmap 5 up:standby 2024-07-26T07:47:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T07:47:02.930 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:02 smithi097 ceph-mon[32231]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:03.081 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:02 smithi181 ceph-mon[35869]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:03.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 -- 172.21.15.97:0/1205486457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb1010fc80 msgr2=0x7feb101164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:03.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 --2- 172.21.15.97:0/1205486457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb1010fc80 0x7feb101164b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feb08009ca0 tx=0x7feb0802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 -- 172.21.15.97:0/1205486457 shutdown_connections 2024-07-26T07:47:03.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 --2- 172.21.15.97:0/1205486457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb1010fc80 0x7feb101164b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 --2- 172.21.15.97:0/1205486457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feb1010f330 0x7feb1010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 --2- 172.21.15.97:0/1205486457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb101099f0 0x7feb10109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 -- 172.21.15.97:0/1205486457 >> 172.21.15.97:0/1205486457 conn(0x7feb10078ff0 msgr2=0x7feb10107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:03.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 -- 172.21.15.97:0/1205486457 shutdown_connections 2024-07-26T07:47:03.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.132+0000 7feb17000640 1 -- 172.21.15.97:0/1205486457 wait complete. 2024-07-26T07:47:03.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.134+0000 7feb17000640 1 Processor -- start 2024-07-26T07:47:03.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.134+0000 7feb17000640 1 -- start start 2024-07-26T07:47:03.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.134+0000 7feb17000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb101099f0 0x7feb101a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:03.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.134+0000 7feb17000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1010f330 0x7feb101a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:03.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.134+0000 7feb17000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feb1010fc80 0x7feb101a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:03.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.134+0000 7feb17000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7feb10119840 con 0x7feb101099f0 2024-07-26T07:47:03.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.134+0000 7feb17000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7feb101196c0 con 0x7feb1010fc80 2024-07-26T07:47:03.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.134+0000 7feb17000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feb101199c0 con 0x7feb1010f330 2024-07-26T07:47:03.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb101099f0 0x7feb101a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:03.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb101099f0 0x7feb101a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37698/0 (socket says 172.21.15.97:37698) 2024-07-26T07:47:03.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15c00640 1 -- 172.21.15.97:0/1023100095 learned_addr learned my addr 172.21.15.97:0/1023100095 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15200640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1010f330 0x7feb101a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb16600640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feb1010fc80 0x7feb101a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15c00640 1 -- 172.21.15.97:0/1023100095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1010f330 msgr2=0x7feb101a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15c00640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1010f330 0x7feb101a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15c00640 1 -- 172.21.15.97:0/1023100095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feb1010fc80 msgr2=0x7feb101a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15c00640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feb1010fc80 0x7feb101a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15c00640 1 -- 172.21.15.97:0/1023100095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb101a60e0 con 0x7feb101099f0 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb16600640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feb1010fc80 0x7feb101a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.135+0000 7feb15200640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1010f330 0x7feb101a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.136+0000 7feb15c00640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb101099f0 0x7feb101a0230 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feb0400d950 tx=0x7feb0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:03.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.136+0000 7feb0e000640 1 -- 172.21.15.97:0/1023100095 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb04017070 con 0x7feb101099f0 2024-07-26T07:47:03.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.136+0000 7feb0e000640 1 -- 172.21.15.97:0/1023100095 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb04009c30 con 0x7feb101099f0 2024-07-26T07:47:03.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.136+0000 7feb0e000640 1 -- 172.21.15.97:0/1023100095 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0400f620 con 0x7feb101099f0 2024-07-26T07:47:03.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.136+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb101a63d0 con 0x7feb101099f0 2024-07-26T07:47:03.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.136+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb101acbf0 con 0x7feb101099f0 2024-07-26T07:47:03.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.138+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb1010ac70 con 0x7feb101099f0 2024-07-26T07:47:03.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.140+0000 7feb0e000640 1 -- 172.21.15.97:0/1023100095 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7feb04010460 con 0x7feb101099f0 2024-07-26T07:47:03.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.141+0000 7feb0e000640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feaf80b3d00 0x7feaf80b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:03.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.142+0000 7feb0e000640 1 -- 172.21.15.97:0/1023100095 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7feb040e3460 con 0x7feb101099f0 2024-07-26T07:47:03.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.142+0000 7feb15200640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feaf80b3d00 0x7feaf80b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:03.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.142+0000 7feb15200640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feaf80b3d00 0x7feaf80b6210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feb1010fae0 tx=0x7feafc00a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:03.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.146+0000 7feb0e000640 1 -- 172.21.15.97:0/1023100095 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feb040aaeb0 con 0x7feb101099f0 2024-07-26T07:47:03.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:02 smithi115 ceph-mon[35364]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:03.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.383+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb10109e00 con 0x7feb101099f0 2024-07-26T07:47:03.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.384+0000 7feb0e000640 1 -- 172.21.15.97:0/1023100095 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4744 (secure 0 0 0) 0x7feb04009de0 con 0x7feb101099f0 2024-07-26T07:47:03.385 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:03.385 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":6,"btime":"2024-07-26T07:47:01:158616+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34517,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34535,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-07-26T07:47:03.386 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 6 2024-07-26T07:47:03.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feaf80b3d00 msgr2=0x7feaf80b6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:03.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feaf80b3d00 0x7feaf80b6210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feb1010fae0 tx=0x7feafc00a5c0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb101099f0 msgr2=0x7feb101a0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:03.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb101099f0 0x7feb101a0230 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feb0400d950 tx=0x7feb0400de10 comp rx=0 tx=0).stop 2024-07-26T07:47:03.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 shutdown_connections 2024-07-26T07:47:03.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feaf80b3d00 0x7feaf80b6210 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feb1010fc80 0x7feb101a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1010f330 0x7feb101a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 --2- 172.21.15.97:0/1023100095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb101099f0 0x7feb101a0230 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:03.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 >> 172.21.15.97:0/1023100095 conn(0x7feb10078ff0 msgr2=0x7feb10079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:03.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 shutdown_connections 2024-07-26T07:47:03.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:03.387+0000 7feb17000640 1 -- 172.21.15.97:0/1023100095 wait complete. 2024-07-26T07:47:03.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:03 smithi181 systemd[1]: Starting Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:47:04.017 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:03 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1023100095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:04.018 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:03 smithi181 podman[65585]: 2024-07-26 07:47:03.665392609 +0000 UTC m=+0.040184779 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:47:04.018 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:03 smithi181 podman[65585]: 2024-07-26 07:47:03.78748889 +0000 UTC m=+0.162281098 container create 15af4714afedd9e96861fee038e9989a13e2ca71decf3512280ec039b0e0ae57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-07-26T07:47:04.018 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:04 smithi181 podman[65585]: 2024-07-26 07:47:04.017254487 +0000 UTC m=+0.392046695 container init 15af4714afedd9e96861fee038e9989a13e2ca71decf3512280ec039b0e0ae57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:47:04.140 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:03 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1023100095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:04.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:03 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1023100095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:04.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:04 smithi181 podman[65585]: 2024-07-26 07:47:04.027448191 +0000 UTC m=+0.402240369 container start 15af4714afedd9e96861fee038e9989a13e2ca71decf3512280ec039b0e0ae57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T07:47:04.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-mds[65604]: -- 172.21.15.181:0/2115350512 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563d246edc20 con 0x563d2888fa80 2024-07-26T07:47:04.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: starting mds.f at 2024-07-26T07:47:04.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:04 smithi181 bash[65585]: 15af4714afedd9e96861fee038e9989a13e2ca71decf3512280ec039b0e0ae57 2024-07-26T07:47:04.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:47:04 smithi181 systemd[1]: Started Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:47:05.101 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-mon[35869]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:05.101 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.101 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.101 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.101 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:05.101 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:05.102 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:04 smithi115 ceph-mon[35364]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:04 smithi097 ceph-mon[32231]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:06.114 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:05 smithi181 ceph-mon[35869]: Deploying daemon mds.c on smithi181 2024-07-26T07:47:06.114 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:05 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:boot 2024-07-26T07:47:06.114 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:05 smithi181 ceph-mon[35869]: fsmap 6 up:standby 2024-07-26T07:47:06.114 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T07:47:06.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:47:06.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:05 smithi115 ceph-mon[35364]: Deploying daemon mds.c on smithi181 2024-07-26T07:47:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:05 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:boot 2024-07-26T07:47:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:05 smithi115 ceph-mon[35364]: fsmap 6 up:standby 2024-07-26T07:47:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T07:47:06.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:05 smithi097 ceph-mon[32231]: Deploying daemon mds.c on smithi181 2024-07-26T07:47:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:05 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:boot 2024-07-26T07:47:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:05 smithi097 ceph-mon[32231]: fsmap 6 up:standby 2024-07-26T07:47:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T07:47:06.670 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:06 smithi181 ceph-mon[35869]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:07.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:06 smithi097 ceph-mon[32231]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:06 smithi115 ceph-mon[35364]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:07.404 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 systemd[1]: Starting Ceph mds.c for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:47:07.657 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 podman[65888]: 2024-07-26 07:47:07.306883945 +0000 UTC m=+0.035706214 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:47:07.657 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 podman[65888]: 2024-07-26 07:47:07.425789296 +0000 UTC m=+0.154611600 container create 8970b3366a51d2a658e0785f82bacc26d588b2301f13631b09b0d80fea2c206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-07-26T07:47:07.658 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 podman[65888]: 2024-07-26 07:47:07.64641899 +0000 UTC m=+0.375241321 container init 8970b3366a51d2a658e0785f82bacc26d588b2301f13631b09b0d80fea2c206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T07:47:07.658 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 podman[65888]: 2024-07-26 07:47:07.656152799 +0000 UTC m=+0.384975115 container start 8970b3366a51d2a658e0785f82bacc26d588b2301f13631b09b0d80fea2c206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-07-26T07:47:07.982 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 bash[65888]: 8970b3366a51d2a658e0785f82bacc26d588b2301f13631b09b0d80fea2c206f 2024-07-26T07:47:07.983 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 systemd[1]: Started Ceph mds.c for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:47:07.983 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 ceph-mds[65907]: -- 172.21.15.181:0/2597915240 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5565cd339860 con 0x5565d14db180 2024-07-26T07:47:07.983 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:47:07 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[65903]: starting mds.c at 2024-07-26T07:47:08.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.774+0000 7fd163e00640 1 -- 172.21.15.97:0/3090456542 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd164108d20 msgr2=0x7fd164109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:08.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.774+0000 7fd163e00640 1 --2- 172.21.15.97:0/3090456542 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd164108d20 0x7fd164109130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009900 tx=0x7fd14c02a410 comp rx=0 tx=0).stop 2024-07-26T07:47:08.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.775+0000 7fd163e00640 1 -- 172.21.15.97:0/3090456542 shutdown_connections 2024-07-26T07:47:08.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.775+0000 7fd163e00640 1 --2- 172.21.15.97:0/3090456542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd164109670 0x7fd16410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:08.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.775+0000 7fd163e00640 1 --2- 172.21.15.97:0/3090456542 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd164108d20 0x7fd164109130 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:08.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.775+0000 7fd163e00640 1 --2- 172.21.15.97:0/3090456542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1641033e0 0x7fd1641037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:08.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.775+0000 7fd163e00640 1 -- 172.21.15.97:0/3090456542 >> 172.21.15.97:0/3090456542 conn(0x7fd1640fee70 msgr2=0x7fd1641012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:08.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.775+0000 7fd163e00640 1 -- 172.21.15.97:0/3090456542 shutdown_connections 2024-07-26T07:47:08.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.775+0000 7fd163e00640 1 -- 172.21.15.97:0/3090456542 wait complete. 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.776+0000 7fd163e00640 1 Processor -- start 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.776+0000 7fd163e00640 1 -- start start 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.776+0000 7fd163e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1641033e0 0x7fd164199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.776+0000 7fd163e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd164108d20 0x7fd16419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.776+0000 7fd163e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd164109670 0x7fd16419f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.776+0000 7fd163e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd1641132d0 con 0x7fd1641033e0 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.776+0000 7fd163e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd164113150 con 0x7fd164109670 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.776+0000 7fd163e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd164113450 con 0x7fd164108d20 2024-07-26T07:47:08.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1641033e0 0x7fd164199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1641033e0 0x7fd164199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54618/0 (socket says 172.21.15.97:54618) 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162a00640 1 -- 172.21.15.97:0/1747146319 learned_addr learned my addr 172.21.15.97:0/1747146319 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd163400640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd164109670 0x7fd16419f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162000640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd164108d20 0x7fd16419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162a00640 1 -- 172.21.15.97:0/1747146319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd164108d20 msgr2=0x7fd16419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162a00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd164108d20 0x7fd16419a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162a00640 1 -- 172.21.15.97:0/1747146319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd164109670 msgr2=0x7fd16419f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162a00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd164109670 0x7fd16419f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.777+0000 7fd162a00640 1 -- 172.21.15.97:0/1747146319 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16419fd30 con 0x7fd1641033e0 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.778+0000 7fd162a00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1641033e0 0x7fd164199dc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd15400ba50 tx=0x7fd15400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.778+0000 7fd15b400640 1 -- 172.21.15.97:0/1747146319 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154019070 con 0x7fd1641033e0 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.778+0000 7fd15b400640 1 -- 172.21.15.97:0/1747146319 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd15400cbe0 con 0x7fd1641033e0 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.778+0000 7fd15b400640 1 -- 172.21.15.97:0/1747146319 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1540075d0 con 0x7fd1641033e0 2024-07-26T07:47:08.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.778+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1641a0020 con 0x7fd1641033e0 2024-07-26T07:47:08.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.778+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1641a6780 con 0x7fd1641033e0 2024-07-26T07:47:08.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.779+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd164103870 con 0x7fd1641033e0 2024-07-26T07:47:08.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.782+0000 7fd15b400640 1 -- 172.21.15.97:0/1747146319 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fd15401c070 con 0x7fd1641033e0 2024-07-26T07:47:08.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.783+0000 7fd15b400640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd1480b3c50 0x7fd1480b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:08.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.784+0000 7fd15b400640 1 -- 172.21.15.97:0/1747146319 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fd1540e3690 con 0x7fd1641033e0 2024-07-26T07:47:08.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.784+0000 7fd162000640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd1480b3c50 0x7fd1480b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:08.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.785+0000 7fd162000640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd1480b3c50 0x7fd1480b6160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009230 tx=0x7fd14c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:08.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:08.786+0000 7fd15b400640 1 -- 172.21.15.97:0/1747146319 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd154023070 con 0x7fd1641033e0 2024-07-26T07:47:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6836/3053625422,v1:172.21.15.181:6837/3053625422] up:boot 2024-07-26T07:47:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: fsmap 7 up:standby 2024-07-26T07:47:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T07:47:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: Deploying daemon mds.h on smithi115 2024-07-26T07:47:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:08 smithi115 ceph-mon[35364]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:09.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.024+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd16419aef0 con 0x7fd1641033e0 2024-07-26T07:47:09.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.025+0000 7fd15b400640 1 -- 172.21.15.97:0/1747146319 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6400 (secure 0 0 0) 0x7fd16419aef0 con 0x7fd1641033e0 2024-07-26T07:47:09.028 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:09.028 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"btime":"2024-07-26T07:47:07:864095+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24538,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":34517,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34535,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-07-26T07:47:09.028 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-07-26T07:47:09.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd1480b3c50 msgr2=0x7fd1480b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:09.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd1480b3c50 0x7fd1480b6160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009230 tx=0x7fd14c005fd0 comp rx=0 tx=0).stop 2024-07-26T07:47:09.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1641033e0 msgr2=0x7fd164199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:09.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1641033e0 0x7fd164199dc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd15400ba50 tx=0x7fd15400bf10 comp rx=0 tx=0).stop 2024-07-26T07:47:09.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 shutdown_connections 2024-07-26T07:47:09.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd1480b3c50 0x7fd1480b6160 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:09.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd164109670 0x7fd16419f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:09.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd164108d20 0x7fd16419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:09.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.028+0000 7fd163e00640 1 --2- 172.21.15.97:0/1747146319 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1641033e0 0x7fd164199dc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:09.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.029+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 >> 172.21.15.97:0/1747146319 conn(0x7fd1640fee70 msgr2=0x7fd164101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:09.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.029+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 shutdown_connections 2024-07-26T07:47:09.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:09.029+0000 7fd163e00640 1 -- 172.21.15.97:0/1747146319 wait complete. 2024-07-26T07:47:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6836/3053625422,v1:172.21.15.181:6837/3053625422] up:boot 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: fsmap 7 up:standby 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: Deploying daemon mds.h on smithi115 2024-07-26T07:47:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:08 smithi181 ceph-mon[35869]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6836/3053625422,v1:172.21.15.181:6837/3053625422] up:boot 2024-07-26T07:47:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: fsmap 7 up:standby 2024-07-26T07:47:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T07:47:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: Deploying daemon mds.h on smithi115 2024-07-26T07:47:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:08 smithi097 ceph-mon[32231]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:09 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1747146319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:10.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:09 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1747146319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:09 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1747146319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:11.056 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:10 smithi115 ceph-mon[35364]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:10 smithi181 ceph-mon[35869]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:10 smithi097 ceph-mon[32231]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:11.425 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 systemd[1]: Starting Ceph mds.h for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:47:11.687 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 podman[65806]: 2024-07-26 07:47:11.422781891 +0000 UTC m=+0.121383402 container create fa4447fff829e0e16149fb4953e54f689c5cd329108b86761aa4248de8c8e652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:47:11.687 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 podman[65806]: 2024-07-26 07:47:11.336157814 +0000 UTC m=+0.034759252 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:47:11.687 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 podman[65806]: 2024-07-26 07:47:11.685260102 +0000 UTC m=+0.383861579 container init fa4447fff829e0e16149fb4953e54f689c5cd329108b86761aa4248de8c8e652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T07:47:11.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:47:11.974 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 podman[65806]: 2024-07-26 07:47:11.6946732 +0000 UTC m=+0.393274645 container start fa4447fff829e0e16149fb4953e54f689c5cd329108b86761aa4248de8c8e652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T07:47:11.974 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 bash[65806]: fa4447fff829e0e16149fb4953e54f689c5cd329108b86761aa4248de8c8e652 2024-07-26T07:47:11.974 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 systemd[1]: Started Ceph mds.h for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:47:11.975 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 ceph-mds[65825]: -- 172.21.15.115:0/3570403369 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55856f8a3860 con 0x558573a45600 2024-07-26T07:47:11.975 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:47:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: starting mds.h at 2024-07-26T07:47:12.253 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] up:boot 2024-07-26T07:47:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: fsmap 8 up:standby 2024-07-26T07:47:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T07:47:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: Deploying daemon mds.i on smithi181 2024-07-26T07:47:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:12 smithi181 ceph-mon[35869]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:13.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] up:boot 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: fsmap 8 up:standby 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: Deploying daemon mds.i on smithi181 2024-07-26T07:47:13.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:12 smithi097 ceph-mon[32231]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] up:boot 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: fsmap 8 up:standby 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: Deploying daemon mds.i on smithi181 2024-07-26T07:47:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:12 smithi115 ceph-mon[35364]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:14 smithi115 ceph-mon[35364]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:14 smithi097 ceph-mon[32231]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:14.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 -- 172.21.15.97:0/1059440993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 msgr2=0x7f00ac1122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:14.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 --2- 172.21.15.97:0/1059440993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 0x7f00ac1122a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009ca0 tx=0x7f00a402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 -- 172.21.15.97:0/1059440993 shutdown_connections 2024-07-26T07:47:14.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 --2- 172.21.15.97:0/1059440993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 0x7f00ac1122a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 --2- 172.21.15.97:0/1059440993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac10b120 0x7f00ac10b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 --2- 172.21.15.97:0/1059440993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00ac1057e0 0x7f00ac105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 -- 172.21.15.97:0/1059440993 >> 172.21.15.97:0/1059440993 conn(0x7f00ac1011f0 msgr2=0x7f00ac103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:14.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 -- 172.21.15.97:0/1059440993 shutdown_connections 2024-07-26T07:47:14.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.473+0000 7f00b2600640 1 -- 172.21.15.97:0/1059440993 wait complete. 2024-07-26T07:47:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.474+0000 7f00b2600640 1 Processor -- start 2024-07-26T07:47:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00b2600640 1 -- start start 2024-07-26T07:47:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00b2600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1057e0 0x7f00ac19bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00b2600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00ac10b120 0x7f00ac19c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00b2600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 0x7f00ac1a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00b2600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac115640 con 0x7f00ac10ba70 2024-07-26T07:47:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00b2600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac1154c0 con 0x7f00ac10b120 2024-07-26T07:47:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00b2600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac1157c0 con 0x7f00ac1057e0 2024-07-26T07:47:14.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 0x7f00ac1a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:14.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 0x7f00ac1a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54644/0 (socket says 172.21.15.97:54644) 2024-07-26T07:47:14.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.475+0000 7f00abe00640 1 -- 172.21.15.97:0/1182688832 learned_addr learned my addr 172.21.15.97:0/1182688832 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:14.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.476+0000 7f00ab400640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1057e0 0x7f00ac19bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:14.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.476+0000 7f00a3e00640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00ac10b120 0x7f00ac19c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:14.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.476+0000 7f00abe00640 1 -- 172.21.15.97:0/1182688832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1057e0 msgr2=0x7f00ac19bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:14.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.476+0000 7f00abe00640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1057e0 0x7f00ac19bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.476+0000 7f00abe00640 1 -- 172.21.15.97:0/1182688832 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00ac10b120 msgr2=0x7f00ac19c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:14.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.476+0000 7f00abe00640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00ac10b120 0x7f00ac19c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.476+0000 7f00abe00640 1 -- 172.21.15.97:0/1182688832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ac1a1f60 con 0x7f00ac10ba70 2024-07-26T07:47:14.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.477+0000 7f00abe00640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 0x7f00ac1a1830 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f00a4005cc0 tx=0x7f00a4032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.477+0000 7f00a8c00640 1 -- 172.21.15.97:0/1182688832 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4032e00 con 0x7f00ac10ba70 2024-07-26T07:47:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.477+0000 7f00a8c00640 1 -- 172.21.15.97:0/1182688832 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f00a40044d0 con 0x7f00ac10ba70 2024-07-26T07:47:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.477+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ac1a21f0 con 0x7f00ac10ba70 2024-07-26T07:47:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.477+0000 7f00a8c00640 1 -- 172.21.15.97:0/1182688832 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4030b70 con 0x7f00ac10ba70 2024-07-26T07:47:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.477+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ac1a89b0 con 0x7f00ac10ba70 2024-07-26T07:47:14.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.478+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00ac105bf0 con 0x7f00ac10ba70 2024-07-26T07:47:14.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.481+0000 7f00a8c00640 1 -- 172.21.15.97:0/1182688832 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f00a4030d10 con 0x7f00ac10ba70 2024-07-26T07:47:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:14 smithi181 ceph-mon[35869]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:14.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.482+0000 7f00a8c00640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f00900b3c50 0x7f00900b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:14.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.483+0000 7f00a8c00640 1 -- 172.21.15.97:0/1182688832 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f00a4104d90 con 0x7f00ac10ba70 2024-07-26T07:47:14.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.483+0000 7f00ab400640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f00900b3c50 0x7f00900b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:14.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.484+0000 7f00ab400640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f00900b3c50 0x7f00900b6160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f009c00bbe0 tx=0x7f009c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:14.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.485+0000 7f00a8c00640 1 -- 172.21.15.97:0/1182688832 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f00a40cc7e0 con 0x7f00ac10ba70 2024-07-26T07:47:14.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.719+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00ac19d120 con 0x7f00ac10ba70 2024-07-26T07:47:14.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.721+0000 7f00a8c00640 1 -- 172.21.15.97:0/1182688832 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7228 (secure 0 0 0) 0x7f00ac19d120 con 0x7f00ac10ba70 2024-07-26T07:47:14.722 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:14.722 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":9,"btime":"2024-07-26T07:47:11:907476+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24538,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":34517,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34535,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34556,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-07-26T07:47:14.722 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 9 2024-07-26T07:47:14.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f00900b3c50 msgr2=0x7f00900b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:14.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f00900b3c50 0x7f00900b6160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f009c00bbe0 tx=0x7f009c00b390 comp rx=0 tx=0).stop 2024-07-26T07:47:14.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 msgr2=0x7f00ac1a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 0x7f00ac1a1830 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f00a4005cc0 tx=0x7f00a4032a10 comp rx=0 tx=0).stop 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 shutdown_connections 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f00900b3c50 0x7f00900b6160 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00ac10ba70 0x7f00ac1a1830 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f00ac10b120 0x7f00ac19c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 --2- 172.21.15.97:0/1182688832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1057e0 0x7f00ac19bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.724+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 >> 172.21.15.97:0/1182688832 conn(0x7f00ac1011f0 msgr2=0x7f00ac104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.725+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 shutdown_connections 2024-07-26T07:47:14.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:14.725+0000 7f00b2600640 1 -- 172.21.15.97:0/1182688832 wait complete. 2024-07-26T07:47:14.969 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:14 smithi181 systemd[1]: Starting Ceph mds.i for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:47:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:15 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1182688832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:15.233 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:14 smithi181 podman[66187]: 2024-07-26 07:47:14.871742495 +0000 UTC m=+0.033704427 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:47:15.233 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:15 smithi181 podman[66187]: 2024-07-26 07:47:15.005672796 +0000 UTC m=+0.167634762 container create 51179b6e8ec8559a701dfd6fbebec0fc06982a130f25e8729c8fa995e79a7d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T07:47:15.435 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:15 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1182688832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:15.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:15 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1182688832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:15.733 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:15 smithi181 podman[66187]: 2024-07-26 07:47:15.276407268 +0000 UTC m=+0.438369252 container init 51179b6e8ec8559a701dfd6fbebec0fc06982a130f25e8729c8fa995e79a7d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-07-26T07:47:15.733 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:15 smithi181 podman[66187]: 2024-07-26 07:47:15.287319341 +0000 UTC m=+0.449281284 container start 51179b6e8ec8559a701dfd6fbebec0fc06982a130f25e8729c8fa995e79a7d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-07-26T07:47:15.733 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:15 smithi181 ceph-mds[66205]: -- 172.21.15.181:0/3297175764 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aaf89cf860 con 0x55aafcb71600 2024-07-26T07:47:15.733 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:15 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: starting mds.i at 2024-07-26T07:47:15.733 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:15 smithi181 bash[66187]: 51179b6e8ec8559a701dfd6fbebec0fc06982a130f25e8729c8fa995e79a7d6d 2024-07-26T07:47:15.733 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:47:15 smithi181 systemd[1]: Started Ceph mds.i for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:47:16.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.631 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: Deploying daemon mds.g on smithi097 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] up:boot 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: fsmap 9 up:standby 2024-07-26T07:47:16.632 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T07:47:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: Deploying daemon mds.g on smithi097 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] up:boot 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: fsmap 9 up:standby 2024-07-26T07:47:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T07:47:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: Deploying daemon mds.g on smithi097 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] up:boot 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: fsmap 9 up:standby 2024-07-26T07:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T07:47:17.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:47:18.222 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:18 smithi115 ceph-mon[35364]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:18 smithi181 ceph-mon[35869]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:18.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:18 smithi097 ceph-mon[32231]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:20.087 INFO:journalctl@ceph.mds.g.smithi097.stdout:Jul 26 07:47:20 smithi097 podman[76473]: 2024-07-26 07:47:20.005944831 +0000 UTC m=+0.941719038 container start 023a8794e0a10296a99465b1d7754f23826d59d0838f4451041d2fdaaf50063c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-26T07:47:20.405 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:20 smithi097 ceph-mon[32231]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:20.405 INFO:journalctl@ceph.mds.g.smithi097.stdout:Jul 26 07:47:20 smithi097 bash[76473]: 023a8794e0a10296a99465b1d7754f23826d59d0838f4451041d2fdaaf50063c 2024-07-26T07:47:20.405 INFO:journalctl@ceph.mds.g.smithi097.stdout:Jul 26 07:47:20 smithi097 systemd[1]: Started Ceph mds.g for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:47:20.406 INFO:journalctl@ceph.mds.g.smithi097.stdout:Jul 26 07:47:20 smithi097 ceph-mds[76508]: -- 172.21.15.97:0/3327096933 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b33854f860 con 0x55b33c6f1600 2024-07-26T07:47:20.406 INFO:journalctl@ceph.mds.g.smithi097.stdout:Jul 26 07:47:20 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-g[76491]: starting mds.g at 2024-07-26T07:47:20.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:20 smithi115 ceph-mon[35364]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:20 smithi181 ceph-mon[35869]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:20.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.766+0000 7f2a3be00640 1 -- 172.21.15.97:0/418472338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c1033e0 msgr2=0x7f2a3c1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:20.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.766+0000 7f2a3be00640 1 --2- 172.21.15.97:0/418472338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c1033e0 0x7f2a3c1037f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009900 tx=0x7f2a2c02a410 comp rx=0 tx=0).stop 2024-07-26T07:47:20.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.767+0000 7f2a3be00640 1 -- 172.21.15.97:0/418472338 shutdown_connections 2024-07-26T07:47:20.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.767+0000 7f2a3be00640 1 --2- 172.21.15.97:0/418472338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a3c109670 0x7f2a3c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:20.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.767+0000 7f2a3be00640 1 --2- 172.21.15.97:0/418472338 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a3c108d20 0x7f2a3c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:20.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.767+0000 7f2a3be00640 1 --2- 172.21.15.97:0/418472338 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c1033e0 0x7f2a3c1037f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:20.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.767+0000 7f2a3be00640 1 -- 172.21.15.97:0/418472338 >> 172.21.15.97:0/418472338 conn(0x7f2a3c0fee70 msgr2=0x7f2a3c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:20.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.767+0000 7f2a3be00640 1 -- 172.21.15.97:0/418472338 shutdown_connections 2024-07-26T07:47:20.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.767+0000 7f2a3be00640 1 -- 172.21.15.97:0/418472338 wait complete. 2024-07-26T07:47:20.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.768+0000 7f2a3be00640 1 Processor -- start 2024-07-26T07:47:20.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.769+0000 7f2a3be00640 1 -- start start 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.769+0000 7f2a3be00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a3c1033e0 0x7f2a3c075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.769+0000 7f2a3be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c108d20 0x7f2a3c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.769+0000 7f2a3be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a3c109670 0x7f2a3c075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.769+0000 7f2a3be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c113240 con 0x7f2a3c108d20 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.769+0000 7f2a3be00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c1130c0 con 0x7f2a3c1033e0 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.769+0000 7f2a3be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c1133c0 con 0x7f2a3c109670 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c108d20 0x7f2a3c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c108d20 0x7f2a3c075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55646/0 (socket says 172.21.15.97:55646) 2024-07-26T07:47:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3a000640 1 -- 172.21.15.97:0/2160738437 learned_addr learned my addr 172.21.15.97:0/2160738437 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:20.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3aa00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a3c1033e0 0x7f2a3c075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:20.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3a000640 1 -- 172.21.15.97:0/2160738437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a3c109670 msgr2=0x7f2a3c075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:20.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3a000640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a3c109670 0x7f2a3c075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:20.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3a000640 1 -- 172.21.15.97:0/2160738437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a3c1033e0 msgr2=0x7f2a3c075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:20.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3a000640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a3c1033e0 0x7f2a3c075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:20.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.770+0000 7f2a3a000640 1 -- 172.21.15.97:0/2160738437 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c1a7ee0 con 0x7f2a3c108d20 2024-07-26T07:47:20.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.771+0000 7f2a3a000640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c108d20 0x7f2a3c075680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800c8c0 tx=0x7f2a2800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:20.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.771+0000 7f2a33400640 1 -- 172.21.15.97:0/2160738437 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28007bb0 con 0x7f2a3c108d20 2024-07-26T07:47:20.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.771+0000 7f2a33400640 1 -- 172.21.15.97:0/2160738437 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a28007d50 con 0x7f2a3c108d20 2024-07-26T07:47:20.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.771+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3c1a81d0 con 0x7f2a3c108d20 2024-07-26T07:47:20.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.772+0000 7f2a33400640 1 -- 172.21.15.97:0/2160738437 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a280186a0 con 0x7f2a3c108d20 2024-07-26T07:47:20.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.772+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3c1a87e0 con 0x7f2a3c108d20 2024-07-26T07:47:20.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.773+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c1037f0 con 0x7f2a3c108d20 2024-07-26T07:47:20.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.776+0000 7f2a33400640 1 -- 172.21.15.97:0/2160738437 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f2a2800fa00 con 0x7f2a3c108d20 2024-07-26T07:47:20.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.778+0000 7f2a33400640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a200b3c50 0x7f2a200b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:20.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.778+0000 7f2a33400640 1 -- 172.21.15.97:0/2160738437 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f2a280e3ad0 con 0x7f2a3c108d20 2024-07-26T07:47:20.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.778+0000 7f2a3aa00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a200b3c50 0x7f2a200b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:20.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.779+0000 7f2a3aa00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a200b3c50 0x7f2a200b6160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c076b90 tx=0x7f2a2c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:20.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:20.779+0000 7f2a33400640 1 -- 172.21.15.97:0/2160738437 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2a28014030 con 0x7f2a3c108d20 2024-07-26T07:47:21.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.031+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a3c076440 con 0x7f2a3c108d20 2024-07-26T07:47:21.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.033+0000 7f2a33400640 1 -- 172.21.15.97:0/2160738437 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8884 (secure 0 0 0) 0x7f2a280ab630 con 0x7f2a3c108d20 2024-07-26T07:47:21.034 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:21.034 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":11,"btime":"2024-07-26T07:47:20:180980+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14748,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24538,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34517,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34535,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34556,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-07-26T07:47:21.035 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 11 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.035+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a200b3c50 msgr2=0x7f2a200b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.035+0000 7f2a3be00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a200b3c50 0x7f2a200b6160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c076b90 tx=0x7f2a2c0058d0 comp rx=0 tx=0).stop 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.035+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c108d20 msgr2=0x7f2a3c075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.035+0000 7f2a3be00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c108d20 0x7f2a3c075680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800c8c0 tx=0x7f2a2800cd80 comp rx=0 tx=0).stop 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.036+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 shutdown_connections 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.036+0000 7f2a3be00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2a200b3c50 0x7f2a200b6160 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.036+0000 7f2a3be00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a3c109670 0x7f2a3c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.036+0000 7f2a3be00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a3c108d20 0x7f2a3c075680 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.036+0000 7f2a3be00640 1 --2- 172.21.15.97:0/2160738437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a3c1033e0 0x7f2a3c075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.036+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 >> 172.21.15.97:0/2160738437 conn(0x7f2a3c0fee70 msgr2=0x7f2a3c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:21.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.036+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 shutdown_connections 2024-07-26T07:47:21.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:21.036+0000 7f2a3be00640 1 -- 172.21.15.97:0/2160738437 wait complete. 2024-07-26T07:47:21.470 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:boot 2024-07-26T07:47:21.470 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: fsmap 10 up:standby 2024-07-26T07:47:21.470 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T07:47:21.470 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.470 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.470 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.470 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:21.471 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:21.471 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:21.471 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: Deploying daemon mds.a on smithi097 2024-07-26T07:47:21.471 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:21 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2160738437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:boot 2024-07-26T07:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: fsmap 10 up:standby 2024-07-26T07:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T07:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: Deploying daemon mds.a on smithi097 2024-07-26T07:47:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:21 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2160738437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:boot 2024-07-26T07:47:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: fsmap 10 up:standby 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: Deploying daemon mds.a on smithi097 2024-07-26T07:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:21 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2160738437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:22.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:22 smithi097 ceph-mon[32231]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:22.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:22 smithi115 ceph-mon[35364]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:22 smithi181 ceph-mon[35869]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:23.904 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:23 smithi097 systemd[1]: Starting Ceph mds.a for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:47:24.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:47:24.225 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:23 smithi097 podman[76835]: 2024-07-26 07:47:23.805675134 +0000 UTC m=+0.033986030 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:47:24.225 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:23 smithi097 podman[76835]: 2024-07-26 07:47:23.913383016 +0000 UTC m=+0.141693932 container create 53a37cf587b46a6a29f31a9fe910259add52ae217c09125eea4fcaf73697d57e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-07-26T07:47:24.226 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:24 smithi097 podman[76835]: 2024-07-26 07:47:24.224818593 +0000 UTC m=+0.453129510 container init 53a37cf587b46a6a29f31a9fe910259add52ae217c09125eea4fcaf73697d57e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:47:24.501 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:24 smithi097 podman[76835]: 2024-07-26 07:47:24.233861857 +0000 UTC m=+0.462172751 container start 53a37cf587b46a6a29f31a9fe910259add52ae217c09125eea4fcaf73697d57e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-07-26T07:47:24.501 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:24 smithi097 bash[76835]: 53a37cf587b46a6a29f31a9fe910259add52ae217c09125eea4fcaf73697d57e 2024-07-26T07:47:24.501 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:24 smithi097 systemd[1]: Started Ceph mds.a for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:47:24.501 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:24 smithi097 ceph-mds[76871]: -- 172.21.15.97:0/4023557590 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560f1f27860 con 0x5560f60c9180 2024-07-26T07:47:24.501 INFO:journalctl@ceph.mds.a.smithi097.stdout:Jul 26 07:47:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-a[76851]: starting mds.a at 2024-07-26T07:47:24.680 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: Deploying daemon mds.b on smithi115 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:boot 2024-07-26T07:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: fsmap 11 up:standby 2024-07-26T07:47:25.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: Deploying daemon mds.b on smithi115 2024-07-26T07:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:boot 2024-07-26T07:47:25.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: fsmap 11 up:standby 2024-07-26T07:47:25.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T07:47:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T07:47:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T07:47:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: Deploying daemon mds.b on smithi115 2024-07-26T07:47:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:boot 2024-07-26T07:47:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: fsmap 11 up:standby 2024-07-26T07:47:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.903+0000 7f3cb1e00640 1 -- 172.21.15.97:0/3323905156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 msgr2=0x7f3cac109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.903+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/3323905156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 0x7f3cac109e00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c009900 tx=0x7f3c9c02a410 comp rx=0 tx=0).stop 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.903+0000 7f3cb1e00640 1 -- 172.21.15.97:0/3323905156 shutdown_connections 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.903+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/3323905156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cac10fc80 0x7f3cac1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.903+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/3323905156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac10f330 0x7f3cac10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.903+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/3323905156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 0x7f3cac109e00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.903+0000 7f3cb1e00640 1 -- 172.21.15.97:0/3323905156 >> 172.21.15.97:0/3323905156 conn(0x7f3cac078ff0 msgr2=0x7f3cac107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.903+0000 7f3cb1e00640 1 -- 172.21.15.97:0/3323905156 shutdown_connections 2024-07-26T07:47:26.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.904+0000 7f3cb1e00640 1 -- 172.21.15.97:0/3323905156 wait complete. 2024-07-26T07:47:26.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.905+0000 7f3cb1e00640 1 Processor -- start 2024-07-26T07:47:26.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.905+0000 7f3cb1e00640 1 -- start start 2024-07-26T07:47:26.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.905+0000 7f3cb1e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 0x7f3cac1a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:26.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.905+0000 7f3cb1e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac10f330 0x7f3cac1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:26.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.905+0000 7f3cb1e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cac10fc80 0x7f3cac1a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:26.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.905+0000 7f3cb1e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac1197f0 con 0x7f3cac1099f0 2024-07-26T07:47:26.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.905+0000 7f3cb1e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac119670 con 0x7f3cac10f330 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.905+0000 7f3cb1e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac119970 con 0x7f3cac10fc80 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 0x7f3cac1a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 0x7f3cac1a02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48110/0 (socket says 172.21.15.97:48110) 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb0a00640 1 -- 172.21.15.97:0/2171175445 learned_addr learned my addr 172.21.15.97:0/2171175445 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3ca3e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac10f330 0x7f3cac1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb1400640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cac10fc80 0x7f3cac1a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb0a00640 1 -- 172.21.15.97:0/2171175445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cac10fc80 msgr2=0x7f3cac1a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb0a00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cac10fc80 0x7f3cac1a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb0a00640 1 -- 172.21.15.97:0/2171175445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac10f330 msgr2=0x7f3cac1a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb0a00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac10f330 0x7f3cac1a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:26.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.906+0000 7f3cb0a00640 1 -- 172.21.15.97:0/2171175445 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cac1a6210 con 0x7f3cac1099f0 2024-07-26T07:47:26.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.907+0000 7f3cb0a00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 0x7f3cac1a02a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c009230 tx=0x7f3c9c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:26.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.907+0000 7f3caa000640 1 -- 172.21.15.97:0/2171175445 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c03f030 con 0x7f3cac1099f0 2024-07-26T07:47:26.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.907+0000 7f3caa000640 1 -- 172.21.15.97:0/2171175445 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c9c003a80 con 0x7f3cac1099f0 2024-07-26T07:47:26.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.907+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cac1a64a0 con 0x7f3cac1099f0 2024-07-26T07:47:26.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.907+0000 7f3caa000640 1 -- 172.21.15.97:0/2171175445 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c035070 con 0x7f3cac1099f0 2024-07-26T07:47:26.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.907+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cac1acc60 con 0x7f3cac1099f0 2024-07-26T07:47:26.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.908+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cac109e00 con 0x7f3cac1099f0 2024-07-26T07:47:26.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.911+0000 7f3caa000640 1 -- 172.21.15.97:0/2171175445 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f3c9c042030 con 0x7f3cac1099f0 2024-07-26T07:47:26.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.912+0000 7f3caa000640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3c900b3cd0 0x7f3c900b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:26.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.913+0000 7f3caa000640 1 -- 172.21.15.97:0/2171175445 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f3c9c1042b0 con 0x7f3cac1099f0 2024-07-26T07:47:26.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.913+0000 7f3ca3e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3c900b3cd0 0x7f3c900b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:26.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.913+0000 7f3ca3e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3c900b3cd0 0x7f3c900b61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3cac10fb50 tx=0x7f3c9400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:26.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:26.915+0000 7f3caa000640 1 -- 172.21.15.97:0/2171175445 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3c9c0cbd00 con 0x7f3cac1099f0 2024-07-26T07:47:27.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.152+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cac1a13d0 con 0x7f3cac1099f0 2024-07-26T07:47:27.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.153+0000 7f3caa000640 1 -- 172.21.15.97:0/2171175445 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9710 (secure 0 0 0) 0x7f3cac1a13d0 con 0x7f3cac1099f0 2024-07-26T07:47:27.155 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:27.155 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":12,"btime":"2024-07-26T07:47:25:159249+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14748,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24538,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34517,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34535,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34556,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-07-26T07:47:27.155 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 12 2024-07-26T07:47:27.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3c900b3cd0 msgr2=0x7f3c900b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:27.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3c900b3cd0 0x7f3c900b61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3cac10fb50 tx=0x7f3c9400b410 comp rx=0 tx=0).stop 2024-07-26T07:47:27.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 msgr2=0x7f3cac1a02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:27.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 0x7f3cac1a02a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c009230 tx=0x7f3c9c004270 comp rx=0 tx=0).stop 2024-07-26T07:47:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 shutdown_connections 2024-07-26T07:47:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3c900b3cd0 0x7f3c900b61e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cac10fc80 0x7f3cac1a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac10f330 0x7f3cac1a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 --2- 172.21.15.97:0/2171175445 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cac1099f0 0x7f3cac1a02a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.156+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 >> 172.21.15.97:0/2171175445 conn(0x7f3cac078ff0 msgr2=0x7f3cac079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.157+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 shutdown_connections 2024-07-26T07:47:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:27.157+0000 7f3cb1e00640 1 -- 172.21.15.97:0/2171175445 wait complete. 2024-07-26T07:47:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:27 smithi115 ceph-mon[35364]: pgmap v369: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:27 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2171175445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:27 smithi181 ceph-mon[35869]: pgmap v369: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:27 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2171175445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:27.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:27 smithi097 ceph-mon[32231]: pgmap v369: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:27 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2171175445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:27.992 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:47:27 smithi115 podman[66107]: 2024-07-26 07:47:27.799558904 +0000 UTC m=+0.188375564 container create ec7f3da47c46fb79b7a26167dd751bf927c455d2904ae81720c431825b807cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-07-26T07:47:28.474 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:47:28 smithi115 podman[66107]: 2024-07-26 07:47:28.086579774 +0000 UTC m=+0.475396442 container init ec7f3da47c46fb79b7a26167dd751bf927c455d2904ae81720c431825b807cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T07:47:28.475 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:47:28 smithi115 podman[66107]: 2024-07-26 07:47:28.097477264 +0000 UTC m=+0.486293928 container start ec7f3da47c46fb79b7a26167dd751bf927c455d2904ae81720c431825b807cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:47:28.475 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:47:28 smithi115 ceph-mds[66134]: -- 172.21.15.115:0/1308947307 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555af0b0f860 con 0x555af4cb1a80 2024-07-26T07:47:28.475 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:47:28 smithi115 bash[66107]: ec7f3da47c46fb79b7a26167dd751bf927c455d2904ae81720c431825b807cda 2024-07-26T07:47:28.475 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:47:28 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: starting mds.b at 2024-07-26T07:47:28.475 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:47:28 smithi115 systemd[1]: Started Ceph mds.b for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:47:29.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: pgmap v370: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:29.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.565 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] up:boot 2024-07-26T07:47:29.565 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: fsmap 12 up:standby 2024-07-26T07:47:29.565 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T07:47:29.565 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:29.565 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: pgmap v370: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] up:boot 2024-07-26T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: fsmap 12 up:standby 2024-07-26T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: pgmap v370: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] up:boot 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: fsmap 12 up:standby 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:29.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:47:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:30.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:30.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:30.804 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:31 smithi181 ceph-mon[35869]: pgmap v371: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.512 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:31 smithi097 ceph-mon[32231]: pgmap v371: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:31.512 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.512 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.512 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.513 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:31 smithi115 ceph-mon[35364]: pgmap v371: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:31.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:31.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:32.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 -- 172.21.15.97:0/3021476189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 msgr2=0x7fa0d41046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 --2- 172.21.15.97:0/3021476189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 0x7fa0d41046c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009900 tx=0x7fa0c802a410 comp rx=0 tx=0).stop 2024-07-26T07:47:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 -- 172.21.15.97:0/3021476189 shutdown_connections 2024-07-26T07:47:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 --2- 172.21.15.97:0/3021476189 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d410d860 0x7fa0d410fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 --2- 172.21.15.97:0/3021476189 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0d4104ce0 0x7fa0d410d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 --2- 172.21.15.97:0/3021476189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 0x7fa0d41046c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 -- 172.21.15.97:0/3021476189 >> 172.21.15.97:0/3021476189 conn(0x7fa0d4079160 msgr2=0x7fa0d4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 -- 172.21.15.97:0/3021476189 shutdown_connections 2024-07-26T07:47:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.923+0000 7fa0dc800640 1 -- 172.21.15.97:0/3021476189 wait complete. 2024-07-26T07:47:32.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.924+0000 7fa0dc800640 1 Processor -- start 2024-07-26T07:47:32.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.924+0000 7fa0dc800640 1 -- start start 2024-07-26T07:47:32.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0dc800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 0x7fa0d41021c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0dc800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0d4104ce0 0x7fa0d4102700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0dc800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d410d860 0x7fa0d4102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0dc800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d41133b0 con 0x7fa0d41042b0 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0dc800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d4113230 con 0x7fa0d4104ce0 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0dc800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d4113530 con 0x7fa0d410d860 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0d9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 0x7fa0d41021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0d9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 0x7fa0d41021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48132/0 (socket says 172.21.15.97:48132) 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0d9e00640 1 -- 172.21.15.97:0/58304498 learned_addr learned my addr 172.21.15.97:0/58304498 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0da800640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d410d860 0x7fa0d4102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.925+0000 7fa0d9400640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0d4104ce0 0x7fa0d4102700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0d9e00640 1 -- 172.21.15.97:0/58304498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d410d860 msgr2=0x7fa0d4102c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0d9e00640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d410d860 0x7fa0d4102c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0d9e00640 1 -- 172.21.15.97:0/58304498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0d4104ce0 msgr2=0x7fa0d4102700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0d9e00640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0d4104ce0 0x7fa0d4102700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:32.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0d9e00640 1 -- 172.21.15.97:0/58304498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d41a80c0 con 0x7fa0d41042b0 2024-07-26T07:47:32.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0d9e00640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 0x7fa0d41021c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0c802a930 tx=0x7fa0c8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:32.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0cea00640 1 -- 172.21.15.97:0/58304498 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c803f070 con 0x7fa0d41042b0 2024-07-26T07:47:32.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0cea00640 1 -- 172.21.15.97:0/58304498 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0c8003e10 con 0x7fa0d41042b0 2024-07-26T07:47:32.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0cea00640 1 -- 172.21.15.97:0/58304498 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8035030 con 0x7fa0d41042b0 2024-07-26T07:47:32.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.926+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d41a8350 con 0x7fa0d41042b0 2024-07-26T07:47:32.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.927+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d41a8830 con 0x7fa0d41042b0 2024-07-26T07:47:32.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.928+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d41046c0 con 0x7fa0d41042b0 2024-07-26T07:47:32.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.931+0000 7fa0cea00640 1 -- 172.21.15.97:0/58304498 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fa0c8042070 con 0x7fa0d41042b0 2024-07-26T07:47:32.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.932+0000 7fa0cea00640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa0bc0b3bd0 0x7fa0bc0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:32.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.932+0000 7fa0cea00640 1 -- 172.21.15.97:0/58304498 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fa0c81044c0 con 0x7fa0d41042b0 2024-07-26T07:47:32.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.933+0000 7fa0d9400640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa0bc0b3bd0 0x7fa0bc0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:32.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.933+0000 7fa0d9400640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa0bc0b3bd0 0x7fa0bc0b60e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4005fd0 tx=0x7fa0c4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:32.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:32.934+0000 7fa0cea00640 1 -- 172.21.15.97:0/58304498 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa0c80cbf10 con 0x7fa0d41042b0 2024-07-26T07:47:33.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.168+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0d402e0c0 con 0x7fa0d41042b0 2024-07-26T07:47:33.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.170+0000 7fa0cea00640 1 -- 172.21.15.97:0/58304498 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10539 (secure 0 0 0) 0x7fa0c80cb810 con 0x7fa0d41042b0 2024-07-26T07:47:33.171 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:33.172 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 13 2024-07-26T07:47:33.172 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":13,"btime":"2024-07-26T07:47:28:430933+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14748,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24505,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24538,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34517,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34535,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34556,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":34583,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-07-26T07:47:33.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa0bc0b3bd0 msgr2=0x7fa0bc0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:33.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa0bc0b3bd0 0x7fa0bc0b60e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4005fd0 tx=0x7fa0c4005850 comp rx=0 tx=0).stop 2024-07-26T07:47:33.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 msgr2=0x7fa0d41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:33.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 0x7fa0d41021c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0c802a930 tx=0x7fa0c8003c20 comp rx=0 tx=0).stop 2024-07-26T07:47:33.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 shutdown_connections 2024-07-26T07:47:33.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa0bc0b3bd0 0x7fa0bc0b60e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:33.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d410d860 0x7fa0d4102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:33.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0d4104ce0 0x7fa0d4102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:33.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 --2- 172.21.15.97:0/58304498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d41042b0 0x7fa0d41021c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:33.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 >> 172.21.15.97:0/58304498 conn(0x7fa0d4079160 msgr2=0x7fa0d4100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:33.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 shutdown_connections 2024-07-26T07:47:33.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:33.173+0000 7fa0dc800640 1 -- 172.21.15.97:0/58304498 wait complete. 2024-07-26T07:47:33.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-07-26T07:47:34.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:33 smithi115 ceph-mon[35364]: pgmap v372: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:33 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/58304498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:33 smithi181 ceph-mon[35869]: pgmap v372: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:33 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/58304498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:34.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:33 smithi097 ceph-mon[32231]: pgmap v372: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:34.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:33 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/58304498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:47:34.367 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:35.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:36 smithi097 ceph-mon[32231]: pgmap v373: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:36.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:36 smithi115 ceph-mon[35364]: pgmap v373: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:36 smithi181 ceph-mon[35869]: pgmap v373: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:36.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.549+0000 7fb88be00640 1 -- 172.21.15.97:0/1002030923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10af10 msgr2=0x7fb88c10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:36.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.549+0000 7fb88be00640 1 --2- 172.21.15.97:0/1002030923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10af10 0x7fb88c10b320 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb878009900 tx=0x7fb87802a410 comp rx=0 tx=0).stop 2024-07-26T07:47:36.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.549+0000 7fb88be00640 1 -- 172.21.15.97:0/1002030923 shutdown_connections 2024-07-26T07:47:36.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.549+0000 7fb88be00640 1 --2- 172.21.15.97:0/1002030923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb88c10b860 0x7fb88c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.549+0000 7fb88be00640 1 --2- 172.21.15.97:0/1002030923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10af10 0x7fb88c10b320 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.549+0000 7fb88be00640 1 --2- 172.21.15.97:0/1002030923 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb88c1055d0 0x7fb88c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.549+0000 7fb88be00640 1 -- 172.21.15.97:0/1002030923 >> 172.21.15.97:0/1002030923 conn(0x7fb88c101040 msgr2=0x7fb88c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:36.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.550+0000 7fb88be00640 1 -- 172.21.15.97:0/1002030923 shutdown_connections 2024-07-26T07:47:36.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.550+0000 7fb88be00640 1 -- 172.21.15.97:0/1002030923 wait complete. 2024-07-26T07:47:36.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.551+0000 7fb88be00640 1 Processor -- start 2024-07-26T07:47:36.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.551+0000 7fb88be00640 1 -- start start 2024-07-26T07:47:36.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.551+0000 7fb88be00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb88c1055d0 0x7fb88c19be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.551+0000 7fb88be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb88c10af10 0x7fb88c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.551+0000 7fb88be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10b860 0x7fb88c1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.551+0000 7fb88be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1153a0 con 0x7fb88c10b860 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.551+0000 7fb88be00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c115220 con 0x7fb88c1055d0 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.551+0000 7fb88be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c115520 con 0x7fb88c10af10 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10b860 0x7fb88c1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10b860 0x7fb88c1a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46498/0 (socket says 172.21.15.97:46498) 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 -- 172.21.15.97:0/2482460106 learned_addr learned my addr 172.21.15.97:0/2482460106 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88a000640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb88c10af10 0x7fb88c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:36.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88aa00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb88c1055d0 0x7fb88c19be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:36.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 -- 172.21.15.97:0/2482460106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb88c10af10 msgr2=0x7fb88c19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:36.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb88c10af10 0x7fb88c19c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 -- 172.21.15.97:0/2482460106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb88c1055d0 msgr2=0x7fb88c19be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:36.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb88c1055d0 0x7fb88c19be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 -- 172.21.15.97:0/2482460106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88c1a1de0 con 0x7fb88c10b860 2024-07-26T07:47:36.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.552+0000 7fb88b400640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10b860 0x7fb88c1a16b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb88400eb10 tx=0x7fb88400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:36.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.553+0000 7fb883400640 1 -- 172.21.15.97:0/2482460106 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884004b20 con 0x7fb88c10b860 2024-07-26T07:47:36.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.553+0000 7fb883400640 1 -- 172.21.15.97:0/2482460106 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb884004cc0 con 0x7fb88c10b860 2024-07-26T07:47:36.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.553+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb88c1a20d0 con 0x7fb88c10b860 2024-07-26T07:47:36.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.553+0000 7fb883400640 1 -- 172.21.15.97:0/2482460106 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884010690 con 0x7fb88c10b860 2024-07-26T07:47:36.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.553+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb88c1a8830 con 0x7fb88c10b860 2024-07-26T07:47:36.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.554+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb88c1059e0 con 0x7fb88c10b860 2024-07-26T07:47:36.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.556+0000 7fb883400640 1 -- 172.21.15.97:0/2482460106 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fb884005000 con 0x7fb88c10b860 2024-07-26T07:47:36.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.558+0000 7fb883400640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb8700b3bd0 0x7fb8700b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:36.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.558+0000 7fb883400640 1 -- 172.21.15.97:0/2482460106 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fb8840e3f70 con 0x7fb88c10b860 2024-07-26T07:47:36.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.558+0000 7fb88aa00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb8700b3bd0 0x7fb8700b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:36.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.559+0000 7fb88aa00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb8700b3bd0 0x7fb8700b60e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009bd0 tx=0x7fb87c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:36.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.560+0000 7fb883400640 1 -- 172.21.15.97:0/2482460106 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb884055600 con 0x7fb88c10b860 2024-07-26T07:47:36.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.769+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb88c19cfa0 con 0x7fb88c10b860 2024-07-26T07:47:36.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.770+0000 7fb883400640 1 -- 172.21.15.97:0/2482460106 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) ==== 84+0+65 (secure 0 0 0) 0x7fb88c19cfa0 con 0x7fb88c10b860 2024-07-26T07:47:36.771 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:36.771 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-07-26T07:47:36.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-07-26T07:47:36.771 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-07-26T07:47:36.771 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-07-26T07:47:36.771 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:47:36.771 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-07-26T07:47:36.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb8700b3bd0 msgr2=0x7fb8700b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:36.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb8700b3bd0 0x7fb8700b60e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009bd0 tx=0x7fb87c005f90 comp rx=0 tx=0).stop 2024-07-26T07:47:36.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10b860 msgr2=0x7fb88c1a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:36.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10b860 0x7fb88c1a16b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb88400eb10 tx=0x7fb88400c2d0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 shutdown_connections 2024-07-26T07:47:36.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb8700b3bd0 0x7fb8700b60e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb88c10b860 0x7fb88c1a16b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb88c10af10 0x7fb88c19c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 --2- 172.21.15.97:0/2482460106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb88c1055d0 0x7fb88c19be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:36.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 >> 172.21.15.97:0/2482460106 conn(0x7fb88c101040 msgr2=0x7fb88c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:36.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 shutdown_connections 2024-07-26T07:47:36.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:36.773+0000 7fb88be00640 1 -- 172.21.15.97:0/2482460106 wait complete. 2024-07-26T07:47:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:37 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2482460106' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T07:47:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:37 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2482460106' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T07:47:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.505 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:37 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2482460106' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T07:47:37.505 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.505 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.506 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.506 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:37.882 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T07:47:37.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T07:47:38.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:38 smithi097 ceph-mon[32231]: pgmap v374: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:38.326 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:38.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:38 smithi115 ceph-mon[35364]: pgmap v374: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:38 smithi181 ceph-mon[35869]: pgmap v374: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:39.867 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:39 smithi097 ceph-mon[32231]: pgmap v375: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:39.868 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:39.868 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:39.868 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:39.868 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:39.868 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:39 smithi115 ceph-mon[35364]: pgmap v375: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:39 smithi181 ceph-mon[35869]: pgmap v375: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:47:40.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 -- 172.21.15.97:0/654612287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 msgr2=0x7fc5e010b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:40.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/654612287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 0x7fc5e010b320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009900 tx=0x7fc5c802a410 comp rx=0 tx=0).stop 2024-07-26T07:47:40.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 -- 172.21.15.97:0/654612287 shutdown_connections 2024-07-26T07:47:40.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/654612287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5e010b860 0x7fc5e0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/654612287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 0x7fc5e010b320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/654612287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5e01055d0 0x7fc5e01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 -- 172.21.15.97:0/654612287 >> 172.21.15.97:0/654612287 conn(0x7fc5e0101040 msgr2=0x7fc5e0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:40.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 -- 172.21.15.97:0/654612287 shutdown_connections 2024-07-26T07:47:40.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.410+0000 7fc5e5c00640 1 -- 172.21.15.97:0/654612287 wait complete. 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.411+0000 7fc5e5c00640 1 Processor -- start 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.411+0000 7fc5e5c00640 1 -- start start 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5e5c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5e01055d0 0x7fc5e019be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5e5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 0x7fc5e019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5e5c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5e010b860 0x7fc5e01a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5e5c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e0115380 con 0x7fc5e010af10 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5e5c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e0115200 con 0x7fc5e010b860 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5e5c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e0115500 con 0x7fc5e01055d0 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 0x7fc5e019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 0x7fc5e019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46518/0 (socket says 172.21.15.97:46518) 2024-07-26T07:47:40.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5d7400640 1 -- 172.21.15.97:0/1020933783 learned_addr learned my addr 172.21.15.97:0/1020933783 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:40.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5dfe00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5e01055d0 0x7fc5e019be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:40.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.412+0000 7fc5e5200640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5e010b860 0x7fc5e01a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:40.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5d7400640 1 -- 172.21.15.97:0/1020933783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5e01055d0 msgr2=0x7fc5e019be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5d7400640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5e01055d0 0x7fc5e019be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5d7400640 1 -- 172.21.15.97:0/1020933783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5e010b860 msgr2=0x7fc5e01a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5d7400640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5e010b860 0x7fc5e01a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5d7400640 1 -- 172.21.15.97:0/1020933783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e01a1e00 con 0x7fc5e010af10 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5dfe00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5e01055d0 0x7fc5e019be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5d7400640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 0x7fc5e019c430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8005940 tx=0x7fc5c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5dd600640 1 -- 172.21.15.97:0/1020933783 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c803f030 con 0x7fc5e010af10 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5dd600640 1 -- 172.21.15.97:0/1020933783 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc5c8003c90 con 0x7fc5e010af10 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5dd600640 1 -- 172.21.15.97:0/1020933783 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8035070 con 0x7fc5e010af10 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.413+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e01a2090 con 0x7fc5e010af10 2024-07-26T07:47:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.414+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e01a8850 con 0x7fc5e010af10 2024-07-26T07:47:40.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.415+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e0106850 con 0x7fc5e010af10 2024-07-26T07:47:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.419+0000 7fc5dd600640 1 -- 172.21.15.97:0/1020933783 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fc5c8042030 con 0x7fc5e010af10 2024-07-26T07:47:40.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.420+0000 7fc5dd600640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc5c40b3bd0 0x7fc5c40b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:40.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.420+0000 7fc5dd600640 1 -- 172.21.15.97:0/1020933783 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fc5c8104120 con 0x7fc5e010af10 2024-07-26T07:47:40.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.421+0000 7fc5dfe00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc5c40b3bd0 0x7fc5c40b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:40.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.421+0000 7fc5dfe00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc5c40b3bd0 0x7fc5c40b60e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00058c0 tx=0x7fc5d0005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:40.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.423+0000 7fc5dd600640 1 -- 172.21.15.97:0/1020933783 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc5c80cbc10 con 0x7fc5e010af10 2024-07-26T07:47:40.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.620+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc5e01059e0 con 0x7fc5e010af10 2024-07-26T07:47:40.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.738+0000 7fc5dd600640 1 -- 172.21.15.97:0/1020933783 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v76) ==== 138+0+0 (secure 0 0 0) 0x7fc5c80cb510 con 0x7fc5e010af10 2024-07-26T07:47:40.740 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-07-26T07:47:40.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.743+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc5c40b3bd0 msgr2=0x7fc5c40b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:40.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.743+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc5c40b3bd0 0x7fc5c40b60e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00058c0 tx=0x7fc5d0005ec0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.743+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 msgr2=0x7fc5e019c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:40.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.743+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 0x7fc5e019c430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8005940 tx=0x7fc5c80042f0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.744+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 shutdown_connections 2024-07-26T07:47:40.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.744+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc5c40b3bd0 0x7fc5c40b60e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.744+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5e010b860 0x7fc5e01a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.744+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc5e010af10 0x7fc5e019c430 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.744+0000 7fc5e5c00640 1 --2- 172.21.15.97:0/1020933783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5e01055d0 0x7fc5e019be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:40.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.744+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 >> 172.21.15.97:0/1020933783 conn(0x7fc5e0101040 msgr2=0x7fc5e0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:40.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.744+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 shutdown_connections 2024-07-26T07:47:40.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:40.744+0000 7fc5e5c00640 1 -- 172.21.15.97:0/1020933783 wait complete. 2024-07-26T07:47:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:40 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1020933783' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T07:47:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:40 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1020933783' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T07:47:41.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:40 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1020933783' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T07:47:41.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T07:47:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:41 smithi115 ceph-mon[35364]: pgmap v376: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:41 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1020933783' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T07:47:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:41 smithi115 ceph-mon[35364]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T07:47:42.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:41 smithi097 ceph-mon[32231]: pgmap v376: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:42.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:41 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1020933783' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T07:47:42.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:41 smithi097 ceph-mon[32231]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T07:47:42.043 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:41 smithi181 ceph-mon[35869]: pgmap v376: 1 pgs: 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:41 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1020933783' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T07:47:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:41 smithi181 ceph-mon[35869]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T07:47:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:42 smithi097 ceph-mon[32231]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T07:47:43.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:42 smithi115 ceph-mon[35364]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T07:47:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:42 smithi181 ceph-mon[35869]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.121+0000 7f0a7ea00640 1 -- 172.21.15.97:0/1543692476 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a7810b120 msgr2=0x7f0a7810b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.121+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/1543692476 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a7810b120 0x7f0a7810b530 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009900 tx=0x7f0a6402a3f0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.121+0000 7f0a7ea00640 1 -- 172.21.15.97:0/1543692476 shutdown_connections 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.121+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/1543692476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7810ba70 0x7f0a781122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.121+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/1543692476 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a7810b120 0x7f0a7810b530 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.121+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/1543692476 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a781057e0 0x7f0a78105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.121+0000 7f0a7ea00640 1 -- 172.21.15.97:0/1543692476 >> 172.21.15.97:0/1543692476 conn(0x7f0a781011f0 msgr2=0x7f0a78103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.122+0000 7f0a7ea00640 1 -- 172.21.15.97:0/1543692476 shutdown_connections 2024-07-26T07:47:44.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.122+0000 7f0a7ea00640 1 -- 172.21.15.97:0/1543692476 wait complete. 2024-07-26T07:47:44.124 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:43 smithi097 ceph-mon[32231]: pgmap v379: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:44.124 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:43 smithi097 ceph-mon[32231]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T07:47:44.124 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:43 smithi097 ceph-mon[32231]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T07:47:44.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.123+0000 7f0a7ea00640 1 Processor -- start 2024-07-26T07:47:44.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.123+0000 7f0a7ea00640 1 -- start start 2024-07-26T07:47:44.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a7ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a781057e0 0x7f0a7819c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:44.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a7ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7810b120 0x7f0a7819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a7ea00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a7810ba70 0x7f0a781a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a7ea00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781155e0 con 0x7f0a781057e0 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a7ea00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0a78115460 con 0x7f0a7810ba70 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a7ea00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0a78115760 con 0x7f0a7810b120 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a77e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a781057e0 0x7f0a7819c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a77e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a781057e0 0x7f0a7819c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46550/0 (socket says 172.21.15.97:46550) 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a77e00640 1 -- 172.21.15.97:0/2929057674 learned_addr learned my addr 172.21.15.97:0/2929057674 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a77400640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7810b120 0x7f0a7819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.124+0000 7f0a7ca00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a7810ba70 0x7f0a781a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a77e00640 1 -- 172.21.15.97:0/2929057674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7810b120 msgr2=0x7f0a7819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:44.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a77e00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7810b120 0x7f0a7819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a77e00640 1 -- 172.21.15.97:0/2929057674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a7810ba70 msgr2=0x7f0a781a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:44.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a77e00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a7810ba70 0x7f0a781a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a77e00640 1 -- 172.21.15.97:0/2929057674 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a781a1f00 con 0x7f0a781057e0 2024-07-26T07:47:44.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a77e00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a781057e0 0x7f0a7819c050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800d950 tx=0x7f0a6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:44.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a74c00640 1 -- 172.21.15.97:0/2929057674 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68017070 con 0x7f0a781057e0 2024-07-26T07:47:44.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a74c00640 1 -- 172.21.15.97:0/2929057674 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a68009c30 con 0x7f0a781057e0 2024-07-26T07:47:44.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a781a21f0 con 0x7f0a781057e0 2024-07-26T07:47:44.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a74c00640 1 -- 172.21.15.97:0/2929057674 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6800f620 con 0x7f0a781057e0 2024-07-26T07:47:44.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.125+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a781a8a10 con 0x7f0a781057e0 2024-07-26T07:47:44.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.127+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a78105bf0 con 0x7f0a781057e0 2024-07-26T07:47:44.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.131+0000 7f0a74c00640 1 -- 172.21.15.97:0/2929057674 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f0a68010460 con 0x7f0a781057e0 2024-07-26T07:47:44.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.133+0000 7f0a74c00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0a5c0b3c00 0x7f0a5c0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:44.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.133+0000 7f0a74c00640 1 -- 172.21.15.97:0/2929057674 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7497+0+0 (secure 0 0 0) 0x7f0a680e3620 con 0x7f0a781057e0 2024-07-26T07:47:44.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.133+0000 7f0a77400640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0a5c0b3c00 0x7f0a5c0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:44.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.134+0000 7f0a77400640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0a5c0b3c00 0x7f0a5c0b6110 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0a6402a910 tx=0x7f0a64000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:44.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.135+0000 7f0a74c00640 1 -- 172.21.15.97:0/2929057674 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0a680aaf00 con 0x7f0a781057e0 2024-07-26T07:47:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:43 smithi115 ceph-mon[35364]: pgmap v379: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:43 smithi115 ceph-mon[35364]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T07:47:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:43 smithi115 ceph-mon[35364]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T07:47:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:43 smithi181 ceph-mon[35869]: pgmap v379: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:43 smithi181 ceph-mon[35869]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T07:47:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:43 smithi181 ceph-mon[35869]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T07:47:44.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.335+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0a781a8cb0 con 0x7f0a781057e0 2024-07-26T07:47:44.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.790+0000 7f0a74c00640 1 -- 172.21.15.97:0/2929057674 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) ==== 170+0+0 (secure 0 0 0) 0x7f0a68009de0 con 0x7f0a781057e0 2024-07-26T07:47:44.792 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-07-26T07:47:44.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.795+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0a5c0b3c00 msgr2=0x7f0a5c0b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.795+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0a5c0b3c00 0x7f0a5c0b6110 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0a6402a910 tx=0x7f0a64000bc0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.795+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a781057e0 msgr2=0x7f0a7819c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.795+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a781057e0 0x7f0a7819c050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800d950 tx=0x7f0a6800de10 comp rx=0 tx=0).stop 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.796+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 shutdown_connections 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.796+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0a5c0b3c00 0x7f0a5c0b6110 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.796+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0a7810ba70 0x7f0a781a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.796+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7810b120 0x7f0a7819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.796+0000 7f0a7ea00640 1 --2- 172.21.15.97:0/2929057674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a781057e0 0x7f0a7819c050 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.796+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 >> 172.21.15.97:0/2929057674 conn(0x7f0a781011f0 msgr2=0x7f0a781098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.796+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 shutdown_connections 2024-07-26T07:47:44.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:44.796+0000 7f0a7ea00640 1 -- 172.21.15.97:0/2929057674 wait complete. 2024-07-26T07:47:45.149 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:44 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2929057674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T07:47:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:44 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2929057674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T07:47:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:44 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2929057674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T07:47:45.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T07:47:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:45 smithi115 ceph-mon[35364]: pgmap v381: 33 pgs: 22 unknown, 11 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:45 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2929057674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T07:47:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:45 smithi115 ceph-mon[35364]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T07:47:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:45 smithi181 ceph-mon[35869]: pgmap v381: 33 pgs: 22 unknown, 11 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:45 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2929057674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T07:47:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:45 smithi181 ceph-mon[35869]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T07:47:46.236 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:46.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:45 smithi097 ceph-mon[32231]: pgmap v381: 33 pgs: 22 unknown, 11 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:46.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:45 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2929057674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T07:47:46.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:45 smithi097 ceph-mon[32231]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T07:47:47.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:46 smithi097 ceph-mon[32231]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T07:47:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:46 smithi115 ceph-mon[35364]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T07:47:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:46 smithi181 ceph-mon[35869]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T07:47:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:47 smithi097 ceph-mon[32231]: pgmap v384: 97 pgs: 11 creating+peering, 53 unknown, 33 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:47 smithi097 ceph-mon[32231]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T07:47:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:47 smithi115 ceph-mon[35364]: pgmap v384: 97 pgs: 11 creating+peering, 53 unknown, 33 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:47 smithi115 ceph-mon[35364]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T07:47:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 -- 172.21.15.97:0/4199032519 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 msgr2=0x7f0530111de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 --2- 172.21.15.97:0/4199032519 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 0x7f0530111de0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0528009ca0 tx=0x7f052802a950 comp rx=0 tx=0).stop 2024-07-26T07:47:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 -- 172.21.15.97:0/4199032519 shutdown_connections 2024-07-26T07:47:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 --2- 172.21.15.97:0/4199032519 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 0x7f0530111de0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 --2- 172.21.15.97:0/4199032519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0530102760 0x7f053010f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 --2- 172.21.15.97:0/4199032519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0530101e10 0x7f0530102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 -- 172.21.15.97:0/4199032519 >> 172.21.15.97:0/4199032519 conn(0x7f05300fdcf0 msgr2=0x7f0530100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:48.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 -- 172.21.15.97:0/4199032519 shutdown_connections 2024-07-26T07:47:48.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.226+0000 7f0536400640 1 -- 172.21.15.97:0/4199032519 wait complete. 2024-07-26T07:47:48.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.227+0000 7f0536400640 1 Processor -- start 2024-07-26T07:47:48.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.228+0000 7f0536400640 1 -- start start 2024-07-26T07:47:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.228+0000 7f0536400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0530101e10 0x7f053019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.228+0000 7f0536400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0530102760 0x7f053019c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.228+0000 7f0536400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 0x7f05301a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.228+0000 7f0536400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0530115350 con 0x7f053010f9c0 2024-07-26T07:47:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.228+0000 7f0536400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f05301151d0 con 0x7f0530102760 2024-07-26T07:47:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.228+0000 7f0536400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f05301154d0 con 0x7f0530101e10 2024-07-26T07:47:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 0x7f05301a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 0x7f05301a15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50840/0 (socket says 172.21.15.97:50840) 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 -- 172.21.15.97:0/867574145 learned_addr learned my addr 172.21.15.97:0/867574145 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535000640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0530101e10 0x7f053019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f052fe00640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0530102760 0x7f053019c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 -- 172.21.15.97:0/867574145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0530101e10 msgr2=0x7f053019bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0530101e10 0x7f053019bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 -- 172.21.15.97:0/867574145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0530102760 msgr2=0x7f053019c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0530102760 0x7f053019c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 -- 172.21.15.97:0/867574145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05301a1c50 con 0x7f053010f9c0 2024-07-26T07:47:48.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.229+0000 7f0535a00640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 0x7f05301a15e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f052802ae70 tx=0x7f0528003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:48.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:47 smithi181 ceph-mon[35869]: pgmap v384: 97 pgs: 11 creating+peering, 53 unknown, 33 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:48.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:47 smithi181 ceph-mon[35869]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T07:47:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.230+0000 7f052d600640 1 -- 172.21.15.97:0/867574145 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528034040 con 0x7f053010f9c0 2024-07-26T07:47:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.230+0000 7f052d600640 1 -- 172.21.15.97:0/867574145 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0528030850 con 0x7f053010f9c0 2024-07-26T07:47:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.230+0000 7f052d600640 1 -- 172.21.15.97:0/867574145 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528004670 con 0x7f053010f9c0 2024-07-26T07:47:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.230+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05301a1ee0 con 0x7f053010f9c0 2024-07-26T07:47:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.230+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05301a23c0 con 0x7f053010f9c0 2024-07-26T07:47:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.231+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0530102220 con 0x7f053010f9c0 2024-07-26T07:47:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.234+0000 7f052d600640 1 -- 172.21.15.97:0/867574145 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f0528037020 con 0x7f053010f9c0 2024-07-26T07:47:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.236+0000 7f052d600640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05180b3c00 0x7f05180b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.237+0000 7f052d600640 1 -- 172.21.15.97:0/867574145 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7879+0+0 (secure 0 0 0) 0x7f0528104b20 con 0x7f053010f9c0 2024-07-26T07:47:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.237+0000 7f0535000640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05180b3c00 0x7f05180b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:48.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.237+0000 7f0535000640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05180b3c00 0x7f05180b6110 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0524005e10 tx=0x7f0524005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:48.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.241+0000 7f052d600640 1 -- 172.21.15.97:0/867574145 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f05280cc310 con 0x7f053010f9c0 2024-07-26T07:47:48.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.480+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f053002e0c0 con 0x7f053010f9c0 2024-07-26T07:47:48.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.840+0000 7f052d600640 1 -- 172.21.15.97:0/867574145 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f05280cbc10 con 0x7f053010f9c0 2024-07-26T07:47:48.841 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 2 and data pool 3 2024-07-26T07:47:48.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.843+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05180b3c00 msgr2=0x7f05180b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:48.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.843+0000 7f0536400640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05180b3c00 0x7f05180b6110 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0524005e10 tx=0x7f0524005da0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.843+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 msgr2=0x7f05301a15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.843+0000 7f0536400640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 0x7f05301a15e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f052802ae70 tx=0x7f0528003f10 comp rx=0 tx=0).stop 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.844+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 shutdown_connections 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.844+0000 7f0536400640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05180b3c00 0x7f05180b6110 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.844+0000 7f0536400640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f053010f9c0 0x7f05301a15e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.844+0000 7f0536400640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0530102760 0x7f053019c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.844+0000 7f0536400640 1 --2- 172.21.15.97:0/867574145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0530101e10 0x7f053019bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.844+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 >> 172.21.15.97:0/867574145 conn(0x7f05300fdcf0 msgr2=0x7f0530111b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.844+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 shutdown_connections 2024-07-26T07:47:48.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:48.844+0000 7f0536400640 1 -- 172.21.15.97:0/867574145 wait complete. 2024-07-26T07:47:49.222 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:48 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/867574145' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T07:47:49.222 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:48 smithi097 ceph-mon[32231]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T07:47:49.222 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:48 smithi097 ceph-mon[32231]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T07:47:49.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:48 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/867574145' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T07:47:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:48 smithi115 ceph-mon[35364]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T07:47:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:48 smithi115 ceph-mon[35364]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T07:47:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:48 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/867574145' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T07:47:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:48 smithi181 ceph-mon[35869]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T07:47:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:48 smithi181 ceph-mon[35869]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T07:47:49.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:47:50.097 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:49 smithi115 ceph-mon[35364]: pgmap v386: 97 pgs: 11 creating+peering, 49 unknown, 37 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:49 smithi115 ceph-mon[35364]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T07:47:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:49 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/867574145' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T07:47:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:49 smithi115 ceph-mon[35364]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T07:47:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:49 smithi115 ceph-mon[35364]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T07:47:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:49 smithi181 ceph-mon[35869]: pgmap v386: 97 pgs: 11 creating+peering, 49 unknown, 37 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:49 smithi181 ceph-mon[35869]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T07:47:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:49 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/867574145' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T07:47:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:49 smithi181 ceph-mon[35869]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T07:47:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:49 smithi181 ceph-mon[35869]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T07:47:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:49 smithi097 ceph-mon[32231]: pgmap v386: 97 pgs: 11 creating+peering, 49 unknown, 37 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T07:47:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:49 smithi097 ceph-mon[32231]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T07:47:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:49 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/867574145' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T07:47:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:49 smithi097 ceph-mon[32231]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T07:47:50.243 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:49 smithi097 ceph-mon[32231]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T07:47:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:50 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] up:active 2024-07-26T07:47:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:50 smithi115 ceph-mon[35364]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:47:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:50 smithi115 ceph-mon[35364]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:47:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:50 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] up:active 2024-07-26T07:47:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:50 smithi181 ceph-mon[35869]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:47:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:50 smithi181 ceph-mon[35869]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:47:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:50 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] up:active 2024-07-26T07:47:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:50 smithi097 ceph-mon[32231]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:47:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:50 smithi097 ceph-mon[32231]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:47:52.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:51 smithi097 ceph-mon[32231]: pgmap v388: 97 pgs: 9 creating+peering, 88 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 192 B/s wr, 4 op/s 2024-07-26T07:47:52.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:51 smithi097 ceph-mon[32231]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T07:47:52.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:51 smithi097 ceph-mon[32231]: Cluster is now healthy 2024-07-26T07:47:52.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 -- 172.21.15.97:0/3970713533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810fc80 msgr2=0x7f46c81164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:52.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 --2- 172.21.15.97:0/3970713533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810fc80 0x7f46c81164b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009ca0 tx=0x7f46c002a8c0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 -- 172.21.15.97:0/3970713533 shutdown_connections 2024-07-26T07:47:52.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 --2- 172.21.15.97:0/3970713533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810fc80 0x7f46c81164b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 --2- 172.21.15.97:0/3970713533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46c810f330 0x7f46c810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 --2- 172.21.15.97:0/3970713533 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c81099f0 0x7f46c8109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 -- 172.21.15.97:0/3970713533 >> 172.21.15.97:0/3970713533 conn(0x7f46c8078ff0 msgr2=0x7f46c8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:52.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 -- 172.21.15.97:0/3970713533 shutdown_connections 2024-07-26T07:47:52.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.205+0000 7f46cee00640 1 -- 172.21.15.97:0/3970713533 wait complete. 2024-07-26T07:47:52.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.206+0000 7f46cee00640 1 Processor -- start 2024-07-26T07:47:52.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.206+0000 7f46cee00640 1 -- start start 2024-07-26T07:47:52.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46c81099f0 0x7f46c81a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:52.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810f330 0x7f46c81a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cee00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c810fc80 0x7f46c81a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f46c8119840 con 0x7f46c810f330 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cee00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f46c81196c0 con 0x7f46c810fc80 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46c81199c0 con 0x7f46c81099f0 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810f330 0x7f46c81a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810f330 0x7f46c81a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50858/0 (socket says 172.21.15.97:50858) 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cd000640 1 -- 172.21.15.97:0/766562648 learned_addr learned my addr 172.21.15.97:0/766562648 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46cda00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46c81099f0 0x7f46c81a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.207+0000 7f46ce400640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c810fc80 0x7f46c81a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46cd000640 1 -- 172.21.15.97:0/766562648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46c81099f0 msgr2=0x7f46c81a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:52.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46cd000640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46c81099f0 0x7f46c81a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46cd000640 1 -- 172.21.15.97:0/766562648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c810fc80 msgr2=0x7f46c81a5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46cd000640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c810fc80 0x7f46c81a5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46cd000640 1 -- 172.21.15.97:0/766562648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c81a6080 con 0x7f46c810f330 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46cd000640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810f330 0x7f46c81a0770 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f46b400c970 tx=0x7f46b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46c6000640 1 -- 172.21.15.97:0/766562648 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4004220 con 0x7f46c810f330 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46c6000640 1 -- 172.21.15.97:0/766562648 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46b4007e40 con 0x7f46c810f330 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46c6000640 1 -- 172.21.15.97:0/766562648 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b40047a0 con 0x7f46c810f330 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.208+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c81a6370 con 0x7f46c810f330 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.209+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c81acb90 con 0x7f46c810f330 2024-07-26T07:47:52.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.210+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c8109e00 con 0x7f46c810f330 2024-07-26T07:47:52.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.212+0000 7f46c6000640 1 -- 172.21.15.97:0/766562648 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f46b40079b0 con 0x7f46c810f330 2024-07-26T07:47:52.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.214+0000 7f46c6000640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f46b00b3c80 0x7f46b00b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:52.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.214+0000 7f46c6000640 1 -- 172.21.15.97:0/766562648 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f46b40e3c40 con 0x7f46c810f330 2024-07-26T07:47:52.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.215+0000 7f46cda00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f46b00b3c80 0x7f46b00b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:52.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.215+0000 7f46cda00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f46b00b3c80 0x7f46b00b6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46bc005e10 tx=0x7f46bc005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:52.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.216+0000 7f46c6000640 1 -- 172.21.15.97:0/766562648 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f46b4014030 con 0x7f46c810f330 2024-07-26T07:47:52.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:51 smithi115 ceph-mon[35364]: pgmap v388: 97 pgs: 9 creating+peering, 88 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 192 B/s wr, 4 op/s 2024-07-26T07:47:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:51 smithi115 ceph-mon[35364]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T07:47:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:51 smithi115 ceph-mon[35364]: Cluster is now healthy 2024-07-26T07:47:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:51 smithi181 ceph-mon[35869]: pgmap v388: 97 pgs: 9 creating+peering, 88 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 192 B/s wr, 4 op/s 2024-07-26T07:47:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:51 smithi181 ceph-mon[35869]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T07:47:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:51 smithi181 ceph-mon[35869]: Cluster is now healthy 2024-07-26T07:47:52.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.419+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46c8111ad0 con 0x7f46c810f330 2024-07-26T07:47:52.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.423+0000 7f46c6000640 1 -- 172.21.15.97:0/766562648 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+22233 (secure 0 0 0) 0x7f46b40ab3c0 con 0x7f46c810f330 2024-07-26T07:47:52.425 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:52.425 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":82,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:47:48.483140+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T07:47:40.641206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T07:47:44.357412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:47:52.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f46b00b3c80 msgr2=0x7f46b00b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f46b00b3c80 0x7f46b00b6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46bc005e10 tx=0x7f46bc005da0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810f330 msgr2=0x7f46c81a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810f330 0x7f46c81a0770 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f46b400c970 tx=0x7f46b400ce30 comp rx=0 tx=0).stop 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 shutdown_connections 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f46b00b3c80 0x7f46b00b6190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c810fc80 0x7f46c81a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46c810f330 0x7f46c81a0770 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 --2- 172.21.15.97:0/766562648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46c81099f0 0x7f46c81a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.426+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 >> 172.21.15.97:0/766562648 conn(0x7f46c8078ff0 msgr2=0x7f46c80796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.427+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 shutdown_connections 2024-07-26T07:47:52.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:52.427+0000 7f46cee00640 1 -- 172.21.15.97:0/766562648 wait complete. 2024-07-26T07:47:53.138 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:52 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/766562648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:47:53.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:47:53.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:52 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/766562648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:47:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:52 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/766562648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:47:53.670 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:53.724 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:47:53 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:47:53.462+0000 7f6981c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T07:47:54.126 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:53 smithi097 ceph-mon[32231]: pgmap v389: 97 pgs: 97 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-07-26T07:47:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:53 smithi115 ceph-mon[35364]: pgmap v389: 97 pgs: 97 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-07-26T07:47:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:53 smithi181 ceph-mon[35869]: pgmap v389: 97 pgs: 97 active+clean; 577 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-07-26T07:47:55.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.861+0000 7f27da600640 1 -- 172.21.15.97:0/3944296547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 msgr2=0x7f27d410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.861+0000 7f27da600640 1 --2- 172.21.15.97:0/3944296547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 0x7f27d410b320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009900 tx=0x7f27bc02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:47:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.862+0000 7f27da600640 1 -- 172.21.15.97:0/3944296547 shutdown_connections 2024-07-26T07:47:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.862+0000 7f27da600640 1 --2- 172.21.15.97:0/3944296547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d410b860 0x7f27d4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.862+0000 7f27da600640 1 --2- 172.21.15.97:0/3944296547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 0x7f27d410b320 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.862+0000 7f27da600640 1 --2- 172.21.15.97:0/3944296547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d41055d0 0x7f27d41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.862+0000 7f27da600640 1 -- 172.21.15.97:0/3944296547 >> 172.21.15.97:0/3944296547 conn(0x7f27d4101040 msgr2=0x7f27d4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.862+0000 7f27da600640 1 -- 172.21.15.97:0/3944296547 shutdown_connections 2024-07-26T07:47:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.862+0000 7f27da600640 1 -- 172.21.15.97:0/3944296547 wait complete. 2024-07-26T07:47:55.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.863+0000 7f27da600640 1 Processor -- start 2024-07-26T07:47:55.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.863+0000 7f27da600640 1 -- start start 2024-07-26T07:47:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27da600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d41055d0 0x7f27d419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27da600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 0x7f27d419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27da600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d410b860 0x7f27d41a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27da600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4115380 con 0x7f27d410af10 2024-07-26T07:47:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27da600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4115200 con 0x7f27d410b860 2024-07-26T07:47:55.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27da600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4115500 con 0x7f27d41055d0 2024-07-26T07:47:55.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27d3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 0x7f27d419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:55.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27d3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 0x7f27d419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35216/0 (socket says 172.21.15.97:35216) 2024-07-26T07:47:55.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27d3e00640 1 -- 172.21.15.97:0/4116798048 learned_addr learned my addr 172.21.15.97:0/4116798048 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:55.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.864+0000 7f27d9200640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d41055d0 0x7f27d419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d9c00640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d410b860 0x7f27d41a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d3e00640 1 -- 172.21.15.97:0/4116798048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d41055d0 msgr2=0x7f27d419be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d3e00640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d41055d0 0x7f27d419be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d3e00640 1 -- 172.21.15.97:0/4116798048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d410b860 msgr2=0x7f27d41a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d3e00640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d410b860 0x7f27d41a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d3e00640 1 -- 172.21.15.97:0/4116798048 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d41a1d40 con 0x7f27d410af10 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d9c00640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d410b860 0x7f27d41a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d3e00640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 0x7f27d419c430 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f27bc005940 tx=0x7f27bc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d1600640 1 -- 172.21.15.97:0/4116798048 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc03f030 con 0x7f27d410af10 2024-07-26T07:47:55.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d1600640 1 -- 172.21.15.97:0/4116798048 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27bc003c90 con 0x7f27d410af10 2024-07-26T07:47:55.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.865+0000 7f27d1600640 1 -- 172.21.15.97:0/4116798048 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc035070 con 0x7f27d410af10 2024-07-26T07:47:55.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.866+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d41a1fd0 con 0x7f27d410af10 2024-07-26T07:47:55.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.866+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d41a2460 con 0x7f27d410af10 2024-07-26T07:47:55.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.867+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27d41059e0 con 0x7f27d410af10 2024-07-26T07:47:55.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.870+0000 7f27d1600640 1 -- 172.21.15.97:0/4116798048 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f27bc042030 con 0x7f27d410af10 2024-07-26T07:47:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.872+0000 7f27d1600640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27b80b3c00 0x7f27b80b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.873+0000 7f27d1600640 1 -- 172.21.15.97:0/4116798048 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f27bc104600 con 0x7f27d410af10 2024-07-26T07:47:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.873+0000 7f27d9200640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27b80b3c00 0x7f27b80b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:55.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.873+0000 7f27d9200640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27b80b3c00 0x7f27b80b6110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f27c4007350 tx=0x7f27c40072e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:55.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:55.875+0000 7f27d1600640 1 -- 172.21.15.97:0/4116798048 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f27bc0cbd00 con 0x7f27d410af10 2024-07-26T07:47:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:55 smithi097 ceph-mon[32231]: pgmap v390: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T07:47:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.073+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27d410d6a0 con 0x7f27d410af10 2024-07-26T07:47:56.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.077+0000 7f27d1600640 1 -- 172.21.15.97:0/4116798048 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+22233 (secure 0 0 0) 0x7f27d410d6a0 con 0x7f27d410af10 2024-07-26T07:47:56.079 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:47:56.079 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":82,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:47:48.483140+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T07:47:40.641206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T07:47:44.357412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:47:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.079+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27b80b3c00 msgr2=0x7f27b80b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.079+0000 7f27da600640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27b80b3c00 0x7f27b80b6110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f27c4007350 tx=0x7f27c40072e0 comp rx=0 tx=0).stop 2024-07-26T07:47:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.079+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 msgr2=0x7f27d419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.079+0000 7f27da600640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 0x7f27d419c430 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f27bc005940 tx=0x7f27bc0042f0 comp rx=0 tx=0).stop 2024-07-26T07:47:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.080+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 shutdown_connections 2024-07-26T07:47:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.080+0000 7f27da600640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f27b80b3c00 0x7f27b80b6110 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.080+0000 7f27da600640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27d410b860 0x7f27d41a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.080+0000 7f27da600640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27d410af10 0x7f27d419c430 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.080+0000 7f27da600640 1 --2- 172.21.15.97:0/4116798048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d41055d0 0x7f27d419be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.080+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 >> 172.21.15.97:0/4116798048 conn(0x7f27d4101040 msgr2=0x7f27d4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.080+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 shutdown_connections 2024-07-26T07:47:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:56.080+0000 7f27da600640 1 -- 172.21.15.97:0/4116798048 wait complete. 2024-07-26T07:47:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:55 smithi115 ceph-mon[35364]: pgmap v390: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T07:47:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:55 smithi181 ceph-mon[35869]: pgmap v390: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T07:47:56.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-07-26T07:47:57.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:56 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4116798048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:47:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:56 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4116798048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:47:57.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:56 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4116798048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:47:57.349 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:47:58.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:57 smithi097 ceph-mon[32231]: pgmap v391: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-07-26T07:47:58.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:57 smithi115 ceph-mon[35364]: pgmap v391: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-07-26T07:47:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:57 smithi181 ceph-mon[35869]: pgmap v391: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-07-26T07:47:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 -- 172.21.15.97:0/1187975412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c1033e0 msgr2=0x7f417c1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 --2- 172.21.15.97:0/1187975412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c1033e0 0x7f417c1037f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4170009900 tx=0x7f417002a410 comp rx=0 tx=0).stop 2024-07-26T07:47:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 -- 172.21.15.97:0/1187975412 shutdown_connections 2024-07-26T07:47:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 --2- 172.21.15.97:0/1187975412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417c109670 0x7f417c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 --2- 172.21.15.97:0/1187975412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f417c108d20 0x7f417c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 --2- 172.21.15.97:0/1187975412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c1033e0 0x7f417c1037f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 -- 172.21.15.97:0/1187975412 >> 172.21.15.97:0/1187975412 conn(0x7f417c0fee30 msgr2=0x7f417c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 -- 172.21.15.97:0/1187975412 shutdown_connections 2024-07-26T07:47:59.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.434+0000 7f4182800640 1 -- 172.21.15.97:0/1187975412 wait complete. 2024-07-26T07:47:59.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.435+0000 7f4182800640 1 Processor -- start 2024-07-26T07:47:59.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4182800640 1 -- start start 2024-07-26T07:47:59.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4182800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417c1033e0 0x7f417c19be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4182800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c108d20 0x7f417c19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4182800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f417c109670 0x7f417c1a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4182800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f417c113240 con 0x7f417c108d20 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4182800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f417c1130c0 con 0x7f417c109670 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4182800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f417c1133c0 con 0x7f417c1033e0 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4180a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c108d20 0x7f417c19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4180a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c108d20 0x7f417c19c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35222/0 (socket says 172.21.15.97:35222) 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.436+0000 7f4180a00640 1 -- 172.21.15.97:0/881338915 learned_addr learned my addr 172.21.15.97:0/881338915 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4181400640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417c1033e0 0x7f417c19be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:59.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4181e00640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f417c109670 0x7f417c1a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:59.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4180a00640 1 -- 172.21.15.97:0/881338915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417c1033e0 msgr2=0x7f417c19be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:59.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4180a00640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417c1033e0 0x7f417c19be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4180a00640 1 -- 172.21.15.97:0/881338915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f417c109670 msgr2=0x7f417c1a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:59.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4180a00640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f417c109670 0x7f417c1a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4180a00640 1 -- 172.21.15.97:0/881338915 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417c1a1dc0 con 0x7f417c108d20 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4181e00640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f417c109670 0x7f417c1a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4181400640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417c1033e0 0x7f417c19be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.437+0000 7f4180a00640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c108d20 0x7f417c19c3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f416800d740 tx=0x7f416800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.438+0000 7f417a000640 1 -- 172.21.15.97:0/881338915 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168017070 con 0x7f417c108d20 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.438+0000 7f417a000640 1 -- 172.21.15.97:0/881338915 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f416800de50 con 0x7f417c108d20 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.438+0000 7f417a000640 1 -- 172.21.15.97:0/881338915 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416800f620 con 0x7f417c108d20 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.438+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417c1a20b0 con 0x7f417c108d20 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.438+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f417c1a8810 con 0x7f417c108d20 2024-07-26T07:47:59.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.439+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f417c1037f0 con 0x7f417c108d20 2024-07-26T07:47:59.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.442+0000 7f417a000640 1 -- 172.21.15.97:0/881338915 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f416801a070 con 0x7f417c108d20 2024-07-26T07:47:59.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.444+0000 7f417a000640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f41640b3cd0 0x7f41640b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:47:59.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.444+0000 7f417a000640 1 -- 172.21.15.97:0/881338915 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f41680e3500 con 0x7f417c108d20 2024-07-26T07:47:59.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.444+0000 7f4181400640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f41640b3cd0 0x7f41640b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:47:59.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.445+0000 7f4181400640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f41640b3cd0 0x7f41640b61e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4170006010 tx=0x7f4170005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:47:59.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.445+0000 7f417a000640 1 -- 172.21.15.97:0/881338915 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f41680aac80 con 0x7f417c108d20 2024-07-26T07:47:59.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.684+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f417c19cf80 con 0x7f417c108d20 2024-07-26T07:47:59.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.911+0000 7f417a000640 1 -- 172.21.15.97:0/881338915 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f417c19cf80 con 0x7f417c108d20 2024-07-26T07:47:59.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.915+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f41640b3cd0 msgr2=0x7f41640b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:59.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f41640b3cd0 0x7f41640b61e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4170006010 tx=0x7f4170005a20 comp rx=0 tx=0).stop 2024-07-26T07:47:59.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c108d20 msgr2=0x7f417c19c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:47:59.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c108d20 0x7f417c19c3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f416800d740 tx=0x7f416800dc00 comp rx=0 tx=0).stop 2024-07-26T07:47:59.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 shutdown_connections 2024-07-26T07:47:59.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f41640b3cd0 0x7f41640b61e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f417c109670 0x7f417c1a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f417c108d20 0x7f417c19c3f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 --2- 172.21.15.97:0/881338915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417c1033e0 0x7f417c19be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:47:59.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 >> 172.21.15.97:0/881338915 conn(0x7f417c0fee30 msgr2=0x7f417c10e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:47:59.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 shutdown_connections 2024-07-26T07:47:59.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:47:59.916+0000 7f4182800640 1 -- 172.21.15.97:0/881338915 wait complete. 2024-07-26T07:47:59.945 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:59 smithi097 ceph-mon[32231]: pgmap v392: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-26T07:47:59.946 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:47:59 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/881338915' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T07:48:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:59 smithi115 ceph-mon[35364]: pgmap v392: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-26T07:48:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:47:59 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/881338915' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T07:48:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:59 smithi181 ceph-mon[35869]: pgmap v392: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-26T07:48:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:47:59 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/881338915' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T07:48:00.685 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-07-26T07:48:00.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-07-26T07:48:01.153 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:00 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/881338915' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T07:48:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:00 smithi115 ceph-mon[35364]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:48:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:00 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/881338915' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T07:48:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:00 smithi181 ceph-mon[35869]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:48:01.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:00 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/881338915' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T07:48:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:00 smithi097 ceph-mon[32231]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T07:48:02.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:01 smithi097 ceph-mon[32231]: pgmap v393: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T07:48:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:01 smithi115 ceph-mon[35364]: pgmap v393: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T07:48:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:01 smithi181 ceph-mon[35869]: pgmap v393: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T07:48:03.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 -- 172.21.15.97:0/159534088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd141099f0 msgr2=0x7fcd14109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:03.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 --2- 172.21.15.97:0/159534088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd141099f0 0x7fcd14109e00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009900 tx=0x7fcd0802a410 comp rx=0 tx=0).stop 2024-07-26T07:48:03.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 -- 172.21.15.97:0/159534088 shutdown_connections 2024-07-26T07:48:03.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 --2- 172.21.15.97:0/159534088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1410fc80 0x7fcd141164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 --2- 172.21.15.97:0/159534088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd1410f330 0x7fcd1410f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 --2- 172.21.15.97:0/159534088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd141099f0 0x7fcd14109e00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 -- 172.21.15.97:0/159534088 >> 172.21.15.97:0/159534088 conn(0x7fcd14078ff0 msgr2=0x7fcd14107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:03.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 -- 172.21.15.97:0/159534088 shutdown_connections 2024-07-26T07:48:03.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.438+0000 7fcd1a000640 1 -- 172.21.15.97:0/159534088 wait complete. 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.439+0000 7fcd1a000640 1 Processor -- start 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd1a000640 1 -- start start 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd1a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd141099f0 0x7fcd141a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd1a000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd1410f330 0x7fcd141a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd1a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd1410fc80 0x7fcd141a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd1a000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcd141197f0 con 0x7fcd1410fc80 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd1a000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcd14119670 con 0x7fcd1410f330 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd1a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcd14119970 con 0x7fcd141099f0 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd19600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd1410fc80 0x7fcd141a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd19600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd1410fc80 0x7fcd141a5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35246/0 (socket says 172.21.15.97:35246) 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.440+0000 7fcd19600640 1 -- 172.21.15.97:0/3120551741 learned_addr learned my addr 172.21.15.97:0/3120551741 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd18c00640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd141099f0 0x7fcd141a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd13e00640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd1410f330 0x7fcd141a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd19600640 1 -- 172.21.15.97:0/3120551741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd141099f0 msgr2=0x7fcd141a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd19600640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd141099f0 0x7fcd141a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd19600640 1 -- 172.21.15.97:0/3120551741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd1410f330 msgr2=0x7fcd141a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd19600640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd1410f330 0x7fcd141a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd19600640 1 -- 172.21.15.97:0/3120551741 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd141a6170 con 0x7fcd1410fc80 2024-07-26T07:48:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd13e00640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd1410f330 0x7fcd141a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:48:03.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.441+0000 7fcd19600640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd1410fc80 0x7fcd141a5b00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00eaa0 tx=0x7fcd0c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:03.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.442+0000 7fcd11600640 1 -- 172.21.15.97:0/3120551741 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c00cdb0 con 0x7fcd1410fc80 2024-07-26T07:48:03.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.442+0000 7fcd11600640 1 -- 172.21.15.97:0/3120551741 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcd0c005020 con 0x7fcd1410fc80 2024-07-26T07:48:03.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.442+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd141a6460 con 0x7fcd1410fc80 2024-07-26T07:48:03.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.442+0000 7fcd11600640 1 -- 172.21.15.97:0/3120551741 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c0116d0 con 0x7fcd1410fc80 2024-07-26T07:48:03.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.442+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd141acc80 con 0x7fcd1410fc80 2024-07-26T07:48:03.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.443+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd14109e00 con 0x7fcd1410fc80 2024-07-26T07:48:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.445+0000 7fcd11600640 1 -- 172.21.15.97:0/3120551741 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fcd0c0051d0 con 0x7fcd1410fc80 2024-07-26T07:48:03.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.447+0000 7fcd11600640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fccfc0b3c00 0x7fccfc0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:03.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.447+0000 7fcd11600640 1 -- 172.21.15.97:0/3120551741 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fcd0c0e3d70 con 0x7fcd1410fc80 2024-07-26T07:48:03.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.447+0000 7fcd18c00640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fccfc0b3c00 0x7fccfc0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:03.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.448+0000 7fcd18c00640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fccfc0b3c00 0x7fccfc0b6110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcd08000c00 tx=0x7fcd08009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:03.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.451+0000 7fcd11600640 1 -- 172.21.15.97:0/3120551741 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcd0c016030 con 0x7fcd1410fc80 2024-07-26T07:48:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.688+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fcd1402e0c0 con 0x7fcd1410fc80 2024-07-26T07:48:03.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.739+0000 7fcd11600640 1 -- 172.21.15.97:0/3120551741 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fcd0c0ab470 con 0x7fcd1410fc80 2024-07-26T07:48:03.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fccfc0b3c00 msgr2=0x7fccfc0b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:03.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fccfc0b3c00 0x7fccfc0b6110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcd08000c00 tx=0x7fcd08009680 comp rx=0 tx=0).stop 2024-07-26T07:48:03.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd1410fc80 msgr2=0x7fcd141a5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:03.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd1410fc80 0x7fcd141a5b00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00eaa0 tx=0x7fcd0c00ef60 comp rx=0 tx=0).stop 2024-07-26T07:48:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 shutdown_connections 2024-07-26T07:48:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fccfc0b3c00 0x7fccfc0b6110 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd1410fc80 0x7fcd141a5b00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd1410f330 0x7fcd141a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 --2- 172.21.15.97:0/3120551741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd141099f0 0x7fcd141a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 >> 172.21.15.97:0/3120551741 conn(0x7fcd14078ff0 msgr2=0x7fcd140798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.742+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 shutdown_connections 2024-07-26T07:48:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:03.743+0000 7fcd1a000640 1 -- 172.21.15.97:0/3120551741 wait complete. 2024-07-26T07:48:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:03 smithi115 ceph-mon[35364]: pgmap v394: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-07-26T07:48:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:03 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3120551741' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T07:48:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:03 smithi115 ceph-mon[35364]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T07:48:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:03 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T07:48:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:03 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3120551741' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T07:48:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:03 smithi115 ceph-mon[35364]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-07-26T07:48:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:03 smithi115 ceph-mon[35364]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-07-26T07:48:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:03 smithi181 ceph-mon[35869]: pgmap v394: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-07-26T07:48:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:03 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3120551741' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T07:48:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:03 smithi181 ceph-mon[35869]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T07:48:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:03 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T07:48:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:03 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3120551741' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T07:48:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:03 smithi181 ceph-mon[35869]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-07-26T07:48:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:03 smithi181 ceph-mon[35869]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-07-26T07:48:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:03 smithi097 ceph-mon[32231]: pgmap v394: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-07-26T07:48:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:03 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3120551741' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T07:48:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:03 smithi097 ceph-mon[32231]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T07:48:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:03 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T07:48:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:03 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3120551741' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T07:48:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:03 smithi097 ceph-mon[32231]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-07-26T07:48:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:03 smithi097 ceph-mon[32231]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-07-26T07:48:04.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-07-26T07:48:04.962 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:05 smithi097 ceph-mon[32231]: pgmap v395: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T07:48:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:05 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] up:active 2024-07-26T07:48:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:05 smithi097 ceph-mon[32231]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T07:48:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:05 smithi097 ceph-mon[32231]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-07-26T07:48:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:05 smithi097 ceph-mon[32231]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-07-26T07:48:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:05 smithi097 ceph-mon[32231]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-07-26T07:48:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:05 smithi115 ceph-mon[35364]: pgmap v395: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T07:48:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:05 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] up:active 2024-07-26T07:48:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:05 smithi115 ceph-mon[35364]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T07:48:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:05 smithi115 ceph-mon[35364]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-07-26T07:48:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:05 smithi115 ceph-mon[35364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-07-26T07:48:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:05 smithi115 ceph-mon[35364]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-07-26T07:48:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:05 smithi181 ceph-mon[35869]: pgmap v395: 97 pgs: 97 active+clean; 579 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T07:48:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:05 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] up:active 2024-07-26T07:48:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:05 smithi181 ceph-mon[35869]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T07:48:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:05 smithi181 ceph-mon[35869]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-07-26T07:48:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:05 smithi181 ceph-mon[35869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-07-26T07:48:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:05 smithi181 ceph-mon[35869]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-07-26T07:48:06.937 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:06 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] up:active 2024-07-26T07:48:06.937 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:06 smithi097 ceph-mon[32231]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T07:48:06.937 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:06 smithi097 ceph-mon[32231]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-07-26T07:48:06.937 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:06 smithi097 ceph-mon[32231]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:creating} 8 up:standby 2024-07-26T07:48:06.938 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:06 smithi097 ceph-mon[32231]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-07-26T07:48:07.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.140+0000 7f474be00640 1 -- 172.21.15.97:0/1404966374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c10b860 msgr2=0x7f474c112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:07.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.140+0000 7f474be00640 1 --2- 172.21.15.97:0/1404966374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c10b860 0x7f474c112090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4744009ca0 tx=0x7f474402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.141+0000 7f474be00640 1 -- 172.21.15.97:0/1404966374 shutdown_connections 2024-07-26T07:48:07.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.141+0000 7f474be00640 1 --2- 172.21.15.97:0/1404966374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c10b860 0x7f474c112090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.141+0000 7f474be00640 1 --2- 172.21.15.97:0/1404966374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474c10af10 0x7f474c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.141+0000 7f474be00640 1 --2- 172.21.15.97:0/1404966374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f474c1055d0 0x7f474c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.141+0000 7f474be00640 1 -- 172.21.15.97:0/1404966374 >> 172.21.15.97:0/1404966374 conn(0x7f474c101040 msgr2=0x7f474c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:07.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.141+0000 7f474be00640 1 -- 172.21.15.97:0/1404966374 shutdown_connections 2024-07-26T07:48:07.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.141+0000 7f474be00640 1 -- 172.21.15.97:0/1404966374 wait complete. 2024-07-26T07:48:07.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.142+0000 7f474be00640 1 Processor -- start 2024-07-26T07:48:07.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.142+0000 7f474be00640 1 -- start start 2024-07-26T07:48:07.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.142+0000 7f474be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c1055d0 0x7f474c19bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:07.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474c10af10 0x7f474c19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:07.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474be00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f474c10b860 0x7f474c1a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f474c115430 con 0x7f474c1055d0 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474be00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1152b0 con 0x7f474c10b860 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1155b0 con 0x7f474c10af10 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c1055d0 0x7f474c19bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c1055d0 0x7f474c19bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52976/0 (socket says 172.21.15.97:52976) 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474aa00640 1 -- 172.21.15.97:0/404257161 learned_addr learned my addr 172.21.15.97:0/404257161 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474a000640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474c10af10 0x7f474c19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474b400640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f474c10b860 0x7f474c1a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474aa00640 1 -- 172.21.15.97:0/404257161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474c10af10 msgr2=0x7f474c19c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474aa00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474c10af10 0x7f474c19c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474aa00640 1 -- 172.21.15.97:0/404257161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f474c10b860 msgr2=0x7f474c1a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474aa00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f474c10b860 0x7f474c1a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.143+0000 7f474aa00640 1 -- 172.21.15.97:0/404257161 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474c1a1d50 con 0x7f474c1055d0 2024-07-26T07:48:07.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.144+0000 7f474aa00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c1055d0 0x7f474c19bde0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f473c00d950 tx=0x7f473c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:07.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.144+0000 7f4743400640 1 -- 172.21.15.97:0/404257161 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c017070 con 0x7f474c1055d0 2024-07-26T07:48:07.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.144+0000 7f4743400640 1 -- 172.21.15.97:0/404257161 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f473c009c30 con 0x7f474c1055d0 2024-07-26T07:48:07.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.144+0000 7f4743400640 1 -- 172.21.15.97:0/404257161 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c00f620 con 0x7f474c1055d0 2024-07-26T07:48:07.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.144+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474c1a2040 con 0x7f474c1055d0 2024-07-26T07:48:07.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.144+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f474c1a87a0 con 0x7f474c1055d0 2024-07-26T07:48:07.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.145+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f474c1059e0 con 0x7f474c1055d0 2024-07-26T07:48:07.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.148+0000 7f4743400640 1 -- 172.21.15.97:0/404257161 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f473c005030 con 0x7f474c1055d0 2024-07-26T07:48:07.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.149+0000 7f4743400640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f47300b3c50 0x7f47300b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:07.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.149+0000 7f4743400640 1 -- 172.21.15.97:0/404257161 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f473c0e3740 con 0x7f474c1055d0 2024-07-26T07:48:07.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.150+0000 7f474a000640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f47300b3c50 0x7f47300b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:07.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.150+0000 7f474a000640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f47300b3c50 0x7f47300b6160 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4738005fd0 tx=0x7f4738005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:07.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.151+0000 7f4743400640 1 -- 172.21.15.97:0/404257161 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f473c0aaec0 con 0x7f474c1055d0 2024-07-26T07:48:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:06 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] up:active 2024-07-26T07:48:07.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:06 smithi115 ceph-mon[35364]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T07:48:07.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:06 smithi115 ceph-mon[35364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-07-26T07:48:07.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:06 smithi115 ceph-mon[35364]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:creating} 8 up:standby 2024-07-26T07:48:07.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:06 smithi115 ceph-mon[35364]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-07-26T07:48:07.262 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:06 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] up:active 2024-07-26T07:48:07.262 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:06 smithi181 ceph-mon[35869]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T07:48:07.262 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:06 smithi181 ceph-mon[35869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-07-26T07:48:07.262 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:06 smithi181 ceph-mon[35869]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:creating} 8 up:standby 2024-07-26T07:48:07.262 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:06 smithi181 ceph-mon[35869]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-07-26T07:48:07.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.389+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f474c02e0c0 con 0x7f474c1055d0 2024-07-26T07:48:07.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.808+0000 7f4743400640 1 -- 172.21.15.97:0/404257161 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7f473c009de0 con 0x7f474c1055d0 2024-07-26T07:48:07.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f47300b3c50 msgr2=0x7f47300b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:07.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f47300b3c50 0x7f47300b6160 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4738005fd0 tx=0x7f4738005e20 comp rx=0 tx=0).stop 2024-07-26T07:48:07.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c1055d0 msgr2=0x7f474c19bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:07.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c1055d0 0x7f474c19bde0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f473c00d950 tx=0x7f473c00de10 comp rx=0 tx=0).stop 2024-07-26T07:48:07.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 shutdown_connections 2024-07-26T07:48:07.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f47300b3c50 0x7f47300b6160 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f474c10b860 0x7f474c1a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474c10af10 0x7f474c19c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 --2- 172.21.15.97:0/404257161 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f474c1055d0 0x7f474c19bde0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:07.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 >> 172.21.15.97:0/404257161 conn(0x7f474c101040 msgr2=0x7f474c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:07.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 shutdown_connections 2024-07-26T07:48:07.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:07.812+0000 7f474be00640 1 -- 172.21.15.97:0/404257161 wait complete. 2024-07-26T07:48:08.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: pgmap v396: 97 pgs: 97 active+clean; 581 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T07:48:08.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] up:active 2024-07-26T07:48:08.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: daemon mds.i assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T07:48:08.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T07:48:08.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: Cluster is now healthy 2024-07-26T07:48:08.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:active} 8 up:standby 2024-07-26T07:48:08.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T07:48:08.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: daemon mds.i is now active in filesystem cephfs as rank 4 2024-07-26T07:48:08.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:07 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/404257161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T07:48:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: pgmap v396: 97 pgs: 97 active+clean; 581 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T07:48:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] up:active 2024-07-26T07:48:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: daemon mds.i assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T07:48:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T07:48:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: Cluster is now healthy 2024-07-26T07:48:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:active} 8 up:standby 2024-07-26T07:48:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T07:48:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: daemon mds.i is now active in filesystem cephfs as rank 4 2024-07-26T07:48:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:07 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/404257161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T07:48:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: pgmap v396: 97 pgs: 97 active+clean; 581 KiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T07:48:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] up:active 2024-07-26T07:48:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: daemon mds.i assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T07:48:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T07:48:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: Cluster is now healthy 2024-07-26T07:48:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:active} 8 up:standby 2024-07-26T07:48:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T07:48:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: daemon mds.i is now active in filesystem cephfs as rank 4 2024-07-26T07:48:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:07 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/404257161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T07:48:08.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-07-26T07:48:08.724 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:48:08 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:48:08.464+0000 7f6981c00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T07:48:08.724 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:48:08 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:48:08.464+0000 7f6981c00640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T07:48:08.724 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:48:08 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:48:08.464+0000 7f6981c00640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T07:48:09.027 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:09.174 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:08 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/404257161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T07:48:09.175 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:08 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] up:active 2024-07-26T07:48:09.175 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:08 smithi097 ceph-mon[32231]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T07:48:09.175 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:08 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:48:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:08 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/404257161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T07:48:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:08 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] up:active 2024-07-26T07:48:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:08 smithi115 ceph-mon[35364]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T07:48:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:08 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:48:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:08 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/404257161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T07:48:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:08 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] up:active 2024-07-26T07:48:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:08 smithi181 ceph-mon[35869]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T07:48:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:08 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:48:10.216 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:09 smithi097 ceph-mon[32231]: pgmap v397: 97 pgs: 97 active+clean; 582 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-07-26T07:48:10.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:09 smithi115 ceph-mon[35364]: pgmap v397: 97 pgs: 97 active+clean; 582 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-07-26T07:48:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:09 smithi181 ceph-mon[35869]: pgmap v397: 97 pgs: 97 active+clean; 582 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-07-26T07:48:11.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 -- 172.21.15.97:0/1400679782 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 msgr2=0x7fbbcc1163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:11.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/1400679782 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 0x7fbbcc1163d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009ca0 tx=0x7fbbc402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 -- 172.21.15.97:0/1400679782 shutdown_connections 2024-07-26T07:48:11.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/1400679782 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 0x7fbbcc1163d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/1400679782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbcc077cc0 0x7fbbcc113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/1400679782 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbcc077370 0x7fbbcc077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 -- 172.21.15.97:0/1400679782 >> 172.21.15.97:0/1400679782 conn(0x7fbbcc101060 msgr2=0x7fbbcc1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:11.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 -- 172.21.15.97:0/1400679782 shutdown_connections 2024-07-26T07:48:11.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.254+0000 7fbbd1c00640 1 -- 172.21.15.97:0/1400679782 wait complete. 2024-07-26T07:48:11.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.255+0000 7fbbd1c00640 1 Processor -- start 2024-07-26T07:48:11.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.256+0000 7fbbd1c00640 1 -- start start 2024-07-26T07:48:11.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.256+0000 7fbbd1c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbcc077370 0x7fbbcc1a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.256+0000 7fbbd1c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbcc077cc0 0x7fbbcc1a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.256+0000 7fbbd1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 0x7fbbcc1a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.256+0000 7fbbd1c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc119830 con 0x7fbbcc113fb0 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.256+0000 7fbbd1c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc1196b0 con 0x7fbbcc077cc0 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.256+0000 7fbbd1c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc1199b0 con 0x7fbbcc077370 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.256+0000 7fbbd1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 0x7fbbcc1a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbd1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 0x7fbbcc1a5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52996/0 (socket says 172.21.15.97:52996) 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbd1200640 1 -- 172.21.15.97:0/937477483 learned_addr learned my addr 172.21.15.97:0/937477483 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbcbe00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbcc077370 0x7fbbcc1a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbcb400640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbcc077cc0 0x7fbbcc1a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:11.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbd1200640 1 -- 172.21.15.97:0/937477483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbcc077370 msgr2=0x7fbbcc1a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:11.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbd1200640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbcc077370 0x7fbbcc1a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbd1200640 1 -- 172.21.15.97:0/937477483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbcc077cc0 msgr2=0x7fbbcc1a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:11.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbd1200640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbcc077cc0 0x7fbbcc1a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.257+0000 7fbbd1200640 1 -- 172.21.15.97:0/937477483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc1a6130 con 0x7fbbcc113fb0 2024-07-26T07:48:11.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.258+0000 7fbbd1200640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 0x7fbbcc1a5a00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4005cc0 tx=0x7fbbc4032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:11.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.258+0000 7fbbc8c00640 1 -- 172.21.15.97:0/937477483 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4032e00 con 0x7fbbcc113fb0 2024-07-26T07:48:11.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.258+0000 7fbbc8c00640 1 -- 172.21.15.97:0/937477483 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbbc40044d0 con 0x7fbbcc113fb0 2024-07-26T07:48:11.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.258+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbcc1a63c0 con 0x7fbbcc113fb0 2024-07-26T07:48:11.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.258+0000 7fbbc8c00640 1 -- 172.21.15.97:0/937477483 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4030b70 con 0x7fbbcc113fb0 2024-07-26T07:48:11.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.258+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbcc1acb80 con 0x7fbbcc113fb0 2024-07-26T07:48:11.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.260+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbcc077780 con 0x7fbbcc113fb0 2024-07-26T07:48:11.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.262+0000 7fbbc8c00640 1 -- 172.21.15.97:0/937477483 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fbbc4004680 con 0x7fbbcc113fb0 2024-07-26T07:48:11.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.263+0000 7fbbc8c00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbbac0b3bd0 0x7fbbac0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:11.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.263+0000 7fbbc8c00640 1 -- 172.21.15.97:0/937477483 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fbbc4104f90 con 0x7fbbcc113fb0 2024-07-26T07:48:11.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.264+0000 7fbbcbe00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbbac0b3bd0 0x7fbbac0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:11.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.264+0000 7fbbcbe00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbbac0b3bd0 0x7fbbac0b60e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00bbe0 tx=0x7fbbbc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:11.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.268+0000 7fbbc8c00640 1 -- 172.21.15.97:0/937477483 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbbc40cc690 con 0x7fbbcc113fb0 2024-07-26T07:48:11.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.502+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fbbcc02e0c0 con 0x7fbbcc113fb0 2024-07-26T07:48:11.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.833+0000 7fbbc8c00640 1 -- 172.21.15.97:0/937477483 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7fbbc40cbf90 con 0x7fbbcc113fb0 2024-07-26T07:48:11.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbbac0b3bd0 msgr2=0x7fbbac0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:11.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbbac0b3bd0 0x7fbbac0b60e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00bbe0 tx=0x7fbbbc00b390 comp rx=0 tx=0).stop 2024-07-26T07:48:11.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 msgr2=0x7fbbcc1a5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:11.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 0x7fbbcc1a5a00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4005cc0 tx=0x7fbbc4032a10 comp rx=0 tx=0).stop 2024-07-26T07:48:11.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 shutdown_connections 2024-07-26T07:48:11.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbbac0b3bd0 0x7fbbac0b60e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbbcc113fb0 0x7fbbcc1a5a00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbcc077cc0 0x7fbbcc1a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 --2- 172.21.15.97:0/937477483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbcc077370 0x7fbbcc1a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:11.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 >> 172.21.15.97:0/937477483 conn(0x7fbbcc101060 msgr2=0x7fbbcc116020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:11.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 shutdown_connections 2024-07-26T07:48:11.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:11.838+0000 7fbbd1c00640 1 -- 172.21.15.97:0/937477483 wait complete. 2024-07-26T07:48:12.111 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:11 smithi181 ceph-mon[35869]: pgmap v398: 97 pgs: 97 active+clean; 583 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-07-26T07:48:12.111 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:11 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/937477483' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T07:48:12.207 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:11 smithi097 ceph-mon[32231]: pgmap v398: 97 pgs: 97 active+clean; 583 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-07-26T07:48:12.207 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:11 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/937477483' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T07:48:12.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:11 smithi115 ceph-mon[35364]: pgmap v398: 97 pgs: 97 active+clean; 583 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-07-26T07:48:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:11 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/937477483' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T07:48:12.553 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-07-26T07:48:12.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-07-26T07:48:13.022 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:13.174 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:12 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/937477483' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T07:48:13.174 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:12 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:48:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:12 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/937477483' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T07:48:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:12 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:48:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:12 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/937477483' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T07:48:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:12 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:48:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:13 smithi115 ceph-mon[35364]: pgmap v399: 97 pgs: 97 active+clean; 584 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.0 KiB/s wr, 16 op/s 2024-07-26T07:48:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:13 smithi181 ceph-mon[35869]: pgmap v399: 97 pgs: 97 active+clean; 584 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.0 KiB/s wr, 16 op/s 2024-07-26T07:48:14.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:13 smithi097 ceph-mon[32231]: pgmap v399: 97 pgs: 97 active+clean; 584 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.0 KiB/s wr, 16 op/s 2024-07-26T07:48:15.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3000212614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd81099f0 msgr2=0x7f4cd8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:15.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3000212614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd81099f0 0x7f4cd8109e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009900 tx=0x7f4cc802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3000212614 shutdown_connections 2024-07-26T07:48:15.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3000212614 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4cd810fc80 0x7f4cd81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3000212614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd810f330 0x7f4cd810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3000212614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd81099f0 0x7f4cd8109e00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3000212614 >> 172.21.15.97:0/3000212614 conn(0x7f4cd8078ff0 msgr2=0x7f4cd8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:15.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3000212614 shutdown_connections 2024-07-26T07:48:15.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.359+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3000212614 wait complete. 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.360+0000 7f4cdcc00640 1 Processor -- start 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cdcc00640 1 -- start start 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cdcc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd81099f0 0x7f4cd81a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cdcc00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4cd810f330 0x7f4cd81a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cdcc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd810fc80 0x7f4cd81a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cdcc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81197f0 con 0x7f4cd810fc80 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cdcc00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd8119670 con 0x7f4cd810f330 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cdcc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd8119970 con 0x7f4cd81099f0 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cd7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd810fc80 0x7f4cd81a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cd7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd810fc80 0x7f4cd81a5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53018/0 (socket says 172.21.15.97:53018) 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.361+0000 7f4cd7e00640 1 -- 172.21.15.97:0/3362606363 learned_addr learned my addr 172.21.15.97:0/3362606363 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:15.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.362+0000 7f4cd7400640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd81099f0 0x7f4cd81a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:15.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.362+0000 7f4cd6a00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4cd810f330 0x7f4cd81a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:15.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.362+0000 7f4cd7e00640 1 -- 172.21.15.97:0/3362606363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd81099f0 msgr2=0x7f4cd81a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:15.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.362+0000 7f4cd7e00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd81099f0 0x7f4cd81a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.362+0000 7f4cd7e00640 1 -- 172.21.15.97:0/3362606363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4cd810f330 msgr2=0x7f4cd81a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:15.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.362+0000 7f4cd7e00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4cd810f330 0x7f4cd81a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.362+0000 7f4cd7e00640 1 -- 172.21.15.97:0/3362606363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd81a61b0 con 0x7f4cd810fc80 2024-07-26T07:48:15.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.362+0000 7f4cd7e00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd810fc80 0x7f4cd81a5b40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000cce0 tx=0x7f4cd000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:15.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.363+0000 7f4ccfe00640 1 -- 172.21.15.97:0/3362606363 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd000f450 con 0x7f4cd810fc80 2024-07-26T07:48:15.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.363+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd81a64a0 con 0x7f4cd810fc80 2024-07-26T07:48:15.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.363+0000 7f4ccfe00640 1 -- 172.21.15.97:0/3362606363 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4cd000fa80 con 0x7f4cd810fc80 2024-07-26T07:48:15.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.363+0000 7f4ccfe00640 1 -- 172.21.15.97:0/3362606363 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0018ad0 con 0x7f4cd810fc80 2024-07-26T07:48:15.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.363+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd81accc0 con 0x7f4cd810fc80 2024-07-26T07:48:15.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.364+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd8109e00 con 0x7f4cd810fc80 2024-07-26T07:48:15.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.366+0000 7f4ccfe00640 1 -- 172.21.15.97:0/3362606363 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f4cd000f5f0 con 0x7f4cd810fc80 2024-07-26T07:48:15.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.368+0000 7f4ccfe00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4cbc0b3c80 0x7f4cbc0b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:15.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.368+0000 7f4ccfe00640 1 -- 172.21.15.97:0/3362606363 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f4cd00e4280 con 0x7f4cd810fc80 2024-07-26T07:48:15.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.368+0000 7f4cd7400640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4cbc0b3c80 0x7f4cbc0b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:15.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.369+0000 7f4cd7400640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4cbc0b3c80 0x7f4cbc0b6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80057c0 tx=0x7f4cc80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:15.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.372+0000 7f4ccfe00640 1 -- 172.21.15.97:0/3362606363 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4cd0014030 con 0x7f4cd810fc80 2024-07-26T07:48:15.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.595+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4cd8063230 con 0x7f4cbc0b3c80 2024-07-26T07:48:15.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.662+0000 7f4ccfe00640 1 -- 172.21.15.97:0/3362606363 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4cd8063230 con 0x7f4cbc0b3c80 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4cbc0b3c80 msgr2=0x7f4cbc0b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4cbc0b3c80 0x7f4cbc0b6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80057c0 tx=0x7f4cc80058d0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd810fc80 msgr2=0x7f4cd81a5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd810fc80 0x7f4cd81a5b40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000cce0 tx=0x7f4cd000a620 comp rx=0 tx=0).stop 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 shutdown_connections 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4cbc0b3c80 0x7f4cbc0b6190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cd810fc80 0x7f4cd81a5b40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4cd810f330 0x7f4cd81a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 --2- 172.21.15.97:0/3362606363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd81099f0 0x7f4cd81a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:15.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 >> 172.21.15.97:0/3362606363 conn(0x7f4cd8078ff0 msgr2=0x7f4cd8079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:15.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 shutdown_connections 2024-07-26T07:48:15.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:15.666+0000 7f4cdcc00640 1 -- 172.21.15.97:0/3362606363 wait complete. 2024-07-26T07:48:15.870 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:48:15 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:48:15.604+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:48:15.870 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:48:15 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:48:15.604+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:48:15.871 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:48:15 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:48:15.604+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:48:15.871 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:48:15 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:48:15.604+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:48:15.871 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:48:15 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:48:15.604+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:48:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:15 smithi115 ceph-mon[35364]: pgmap v400: 97 pgs: 97 active+clean; 584 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.0 KiB/s wr, 19 op/s 2024-07-26T07:48:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:15 smithi181 ceph-mon[35869]: pgmap v400: 97 pgs: 97 active+clean; 584 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.0 KiB/s wr, 19 op/s 2024-07-26T07:48:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:15 smithi097 ceph-mon[32231]: pgmap v400: 97 pgs: 97 active+clean; 584 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.0 KiB/s wr, 19 op/s 2024-07-26T07:48:16.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-07-26T07:48:16.993 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:17.138 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:16 smithi097 ceph-mon[32231]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:17.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:16 smithi115 ceph-mon[35364]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:16 smithi181 ceph-mon[35869]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:17 smithi115 ceph-mon[35364]: pgmap v401: 97 pgs: 97 active+clean; 585 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.1 KiB/s wr, 21 op/s 2024-07-26T07:48:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:17 smithi115 ceph-mon[35364]: mgrmap e17: x(active, since 10m), standbys: z, y 2024-07-26T07:48:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:17 smithi181 ceph-mon[35869]: pgmap v401: 97 pgs: 97 active+clean; 585 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.1 KiB/s wr, 21 op/s 2024-07-26T07:48:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:17 smithi181 ceph-mon[35869]: mgrmap e17: x(active, since 10m), standbys: z, y 2024-07-26T07:48:18.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:17 smithi097 ceph-mon[32231]: pgmap v401: 97 pgs: 97 active+clean; 585 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.1 KiB/s wr, 21 op/s 2024-07-26T07:48:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:17 smithi097 ceph-mon[32231]: mgrmap e17: x(active, since 10m), standbys: z, y 2024-07-26T07:48:19.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 -- 172.21.15.97:0/1336954129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c810fb50 msgr2=0x7f59c8111f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:19.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 --2- 172.21.15.97:0/1336954129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c810fb50 0x7f59c8111f70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009ca0 tx=0x7f59c002a8c0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 -- 172.21.15.97:0/1336954129 shutdown_connections 2024-07-26T07:48:19.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 --2- 172.21.15.97:0/1336954129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c810fb50 0x7f59c8111f70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 --2- 172.21.15.97:0/1336954129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c8075ad0 0x7f59c810f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 --2- 172.21.15.97:0/1336954129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c8075180 0x7f59c8075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 -- 172.21.15.97:0/1336954129 >> 172.21.15.97:0/1336954129 conn(0x7f59c807b280 msgr2=0x7f59c807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 -- 172.21.15.97:0/1336954129 shutdown_connections 2024-07-26T07:48:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.127+0000 7f59cd000640 1 -- 172.21.15.97:0/1336954129 wait complete. 2024-07-26T07:48:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.128+0000 7f59cd000640 1 Processor -- start 2024-07-26T07:48:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.128+0000 7f59cd000640 1 -- start start 2024-07-26T07:48:19.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59cd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8075180 0x7f59c819bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59cd000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c8075ad0 0x7f59c819c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59cd000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c810fb50 0x7f59c81a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59cd000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f59c81155c0 con 0x7f59c8075180 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59cd000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f59c8115440 con 0x7f59c8075ad0 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59cd000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f59c8115740 con 0x7f59c810fb50 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59c5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8075180 0x7f59c819bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59c5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8075180 0x7f59c819bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53238/0 (socket says 172.21.15.97:53238) 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59c5e00640 1 -- 172.21.15.97:0/1245875160 learned_addr learned my addr 172.21.15.97:0/1245875160 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59c5400640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c8075ad0 0x7f59c819c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.129+0000 7f59c6800640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c810fb50 0x7f59c81a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59c5e00640 1 -- 172.21.15.97:0/1245875160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c810fb50 msgr2=0x7f59c81a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:19.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59c5e00640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c810fb50 0x7f59c81a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59c5e00640 1 -- 172.21.15.97:0/1245875160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c8075ad0 msgr2=0x7f59c819c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59c5e00640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c8075ad0 0x7f59c819c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59c5e00640 1 -- 172.21.15.97:0/1245875160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c81a1e80 con 0x7f59c8075180 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59c6800640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c810fb50 0x7f59c81a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59c5e00640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8075180 0x7f59c819bfd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f59b800d770 tx=0x7f59b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59bea00640 1 -- 172.21.15.97:0/1245875160 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8017070 con 0x7f59c8075180 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59bea00640 1 -- 172.21.15.97:0/1245875160 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59b80054b0 con 0x7f59c8075180 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59bea00640 1 -- 172.21.15.97:0/1245875160 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8004c60 con 0x7f59c8075180 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c81a2170 con 0x7f59c8075180 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.130+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c81a8990 con 0x7f59c8075180 2024-07-26T07:48:19.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.132+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c8076690 con 0x7f59c8075180 2024-07-26T07:48:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.134+0000 7f59bea00640 1 -- 172.21.15.97:0/1245875160 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f59b8004e00 con 0x7f59c8075180 2024-07-26T07:48:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.135+0000 7f59bea00640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59ac0b3d10 0x7f59ac0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.136+0000 7f59bea00640 1 -- 172.21.15.97:0/1245875160 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8269+0+0 (secure 0 0 0) 0x7f59b80e3800 con 0x7f59c8075180 2024-07-26T07:48:19.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.136+0000 7f59c5400640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59ac0b3d10 0x7f59ac0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.136+0000 7f59c5400640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59ac0b3d10 0x7f59ac0b6220 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f59b4005e10 tx=0x7f59b4005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:19.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.140+0000 7f59bea00640 1 -- 172.21.15.97:0/1245875160 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f59b80aacd0 con 0x7f59c8075180 2024-07-26T07:48:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-07-26T07:48:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T07:48:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-07-26T07:48:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-07-26T07:48:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-07-26T07:48:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-07-26T07:48:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-07-26T07:48:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T07:48:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-07-26T07:48:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-07-26T07:48:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-07-26T07:48:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T07:48:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-07-26T07:48:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-07-26T07:48:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-07-26T07:48:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-07-26T07:48:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-07-26T07:48:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T07:48:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-07-26T07:48:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-07-26T07:48:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-07-26T07:48:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T07:48:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-07-26T07:48:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-07-26T07:48:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-07-26T07:48:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-07-26T07:48:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-07-26T07:48:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T07:48:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-07-26T07:48:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-07-26T07:48:19.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.368+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f59c8102910 con 0x7f59ac0b3d10 2024-07-26T07:48:19.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.439+0000 7f59bea00640 1 -- 172.21.15.97:0/1245875160 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f59c8102910 con 0x7f59ac0b3d10 2024-07-26T07:48:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59ac0b3d10 msgr2=0x7f59ac0b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:19.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59ac0b3d10 0x7f59ac0b6220 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f59b4005e10 tx=0x7f59b4005da0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8075180 msgr2=0x7f59c819bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8075180 0x7f59c819bfd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f59b800d770 tx=0x7f59b800dc30 comp rx=0 tx=0).stop 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 shutdown_connections 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59ac0b3d10 0x7f59ac0b6220 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c810fb50 0x7f59c81a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c8075ad0 0x7f59c819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 --2- 172.21.15.97:0/1245875160 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8075180 0x7f59c819bfd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.443+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 >> 172.21.15.97:0/1245875160 conn(0x7f59c807b280 msgr2=0x7f59c81011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.444+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 shutdown_connections 2024-07-26T07:48:19.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:19.444+0000 7f59cd000640 1 -- 172.21.15.97:0/1245875160 wait complete. 2024-07-26T07:48:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: pgmap v402: 97 pgs: 97 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-07-26T07:48:20.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-07-26T07:48:20.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-07-26T07:48:20.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-07-26T07:48:20.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-07-26T07:48:20.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-07-26T07:48:20.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-07-26T07:48:20.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-07-26T07:48:20.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-07-26T07:48:20.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-07-26T07:48:20.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-07-26T07:48:20.005 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:19 smithi097 ceph-mon[32231]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T07:48:20.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-07-26T07:48:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: pgmap v402: 97 pgs: 97 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-07-26T07:48:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-07-26T07:48:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-07-26T07:48:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-07-26T07:48:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-07-26T07:48:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-07-26T07:48:20.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-07-26T07:48:20.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-07-26T07:48:20.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-07-26T07:48:20.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-07-26T07:48:20.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-07-26T07:48:20.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:19 smithi115 ceph-mon[35364]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T07:48:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: pgmap v402: 97 pgs: 97 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-07-26T07:48:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-07-26T07:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:19 smithi181 ceph-mon[35869]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T07:48:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:21.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:20 smithi097 ceph-mon[32231]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:21.059 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:20 smithi097 ceph-mon[32231]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T07:48:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:20 smithi115 ceph-mon[35364]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:20 smithi115 ceph-mon[35364]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T07:48:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:20 smithi181 ceph-mon[35869]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:20 smithi181 ceph-mon[35869]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T07:48:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:21 smithi115 ceph-mon[35364]: pgmap v405: 97 pgs: 5 peering, 92 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-07-26T07:48:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:21 smithi181 ceph-mon[35869]: pgmap v405: 97 pgs: 5 peering, 92 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-07-26T07:48:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:21 smithi097 ceph-mon[32231]: pgmap v405: 97 pgs: 5 peering, 92 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-07-26T07:48:22.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 -- 172.21.15.97:0/2004713358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c010a910 msgr2=0x7fd4c0115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:22.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/2004713358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c010a910 0x7fd4c0115a00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009900 tx=0x7fd4a802a410 comp rx=0 tx=0).stop 2024-07-26T07:48:22.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 -- 172.21.15.97:0/2004713358 shutdown_connections 2024-07-26T07:48:22.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/2004713358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4c0116070 0x7fd4c0118580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:22.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/2004713358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c010a910 0x7fd4c0115a00 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:22.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/2004713358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c0109fc0 0x7fd4c010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:22.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 -- 172.21.15.97:0/2004713358 >> 172.21.15.97:0/2004713358 conn(0x7fd4c0070d20 msgr2=0x7fd4c0073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:22.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 -- 172.21.15.97:0/2004713358 shutdown_connections 2024-07-26T07:48:22.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.861+0000 7fd4c4e00640 1 -- 172.21.15.97:0/2004713358 wait complete. 2024-07-26T07:48:22.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.863+0000 7fd4c4e00640 1 Processor -- start 2024-07-26T07:48:22.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.863+0000 7fd4c4e00640 1 -- start start 2024-07-26T07:48:22.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.863+0000 7fd4c4e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c0109fc0 0x7fd4c01a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.863+0000 7fd4c4e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4c010a910 0x7fd4c01a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.863+0000 7fd4c4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c0116070 0x7fd4c01a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.863+0000 7fd4c4e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c011bbf0 con 0x7fd4c0116070 2024-07-26T07:48:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.863+0000 7fd4c4e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c011ba70 con 0x7fd4c0109fc0 2024-07-26T07:48:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.863+0000 7fd4c4e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c011bd70 con 0x7fd4c010a910 2024-07-26T07:48:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4be800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c0116070 0x7fd4c01a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:22.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4be800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c0116070 0x7fd4c01a7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53252/0 (socket says 172.21.15.97:53252) 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4be800640 1 -- 172.21.15.97:0/1751406760 learned_addr learned my addr 172.21.15.97:0/1751406760 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4bde00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c0109fc0 0x7fd4c01a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4b5400640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4c010a910 0x7fd4c01a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4be800640 1 -- 172.21.15.97:0/1751406760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4c010a910 msgr2=0x7fd4c01a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4be800640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4c010a910 0x7fd4c01a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4be800640 1 -- 172.21.15.97:0/1751406760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c0109fc0 msgr2=0x7fd4c01a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4be800640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c0109fc0 0x7fd4c01a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.864+0000 7fd4be800640 1 -- 172.21.15.97:0/1751406760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c01a8570 con 0x7fd4c0116070 2024-07-26T07:48:22.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.865+0000 7fd4be800640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c0116070 0x7fd4c01a7e40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800eaa0 tx=0x7fd4b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.865+0000 7fd4b7400640 1 -- 172.21.15.97:0/1751406760 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b800cdb0 con 0x7fd4c0116070 2024-07-26T07:48:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.865+0000 7fd4b7400640 1 -- 172.21.15.97:0/1751406760 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4b8005020 con 0x7fd4c0116070 2024-07-26T07:48:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.865+0000 7fd4b7400640 1 -- 172.21.15.97:0/1751406760 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80116d0 con 0x7fd4c0116070 2024-07-26T07:48:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.865+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c01a8860 con 0x7fd4c0116070 2024-07-26T07:48:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.865+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c01aefc0 con 0x7fd4c0116070 2024-07-26T07:48:22.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.867+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c010a3d0 con 0x7fd4c0116070 2024-07-26T07:48:22.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.869+0000 7fd4b7400640 1 -- 172.21.15.97:0/1751406760 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fd4b80051d0 con 0x7fd4c0116070 2024-07-26T07:48:22.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.870+0000 7fd4b7400640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4a40b3ce0 0x7fd4a40b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:22.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.871+0000 7fd4b7400640 1 -- 172.21.15.97:0/1751406760 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7fd4b80e3fe0 con 0x7fd4c0116070 2024-07-26T07:48:22.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.871+0000 7fd4bde00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4a40b3ce0 0x7fd4a40b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:22.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.871+0000 7fd4bde00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4a40b3ce0 0x7fd4a40b61f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00058c0 tx=0x7fd4b000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:22.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:22.874+0000 7fd4b7400640 1 -- 172.21.15.97:0/1751406760 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd4b8016030 con 0x7fd4c0116070 2024-07-26T07:48:23.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.106+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fd4c0063230 con 0x7fd4a40b3ce0 2024-07-26T07:48:23.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.113+0000 7fd4b7400640 1 -- 172.21.15.97:0/1751406760 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd4c0063230 con 0x7fd4a40b3ce0 2024-07-26T07:48:23.161 INFO:teuthology.orchestra.run.smithi097.stdout:/volumes/qa/sv_0/4e92e127-3c83-480e-8968-682139f34bdb 2024-07-26T07:48:23.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4a40b3ce0 msgr2=0x7fd4a40b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4a40b3ce0 0x7fd4a40b61f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00058c0 tx=0x7fd4b000a4c0 comp rx=0 tx=0).stop 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c0116070 msgr2=0x7fd4c01a7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c0116070 0x7fd4c01a7e40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800eaa0 tx=0x7fd4b800ef60 comp rx=0 tx=0).stop 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 shutdown_connections 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4a40b3ce0 0x7fd4a40b61f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4c0116070 0x7fd4c01a7e40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4c010a910 0x7fd4c01a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.116+0000 7fd4c4e00640 1 --2- 172.21.15.97:0/1751406760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c0109fc0 0x7fd4c01a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.117+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 >> 172.21.15.97:0/1751406760 conn(0x7fd4c0070d20 msgr2=0x7fd4c00731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.117+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 shutdown_connections 2024-07-26T07:48:23.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:23.117+0000 7fd4c4e00640 1 -- 172.21.15.97:0/1751406760 wait complete. 2024-07-26T07:48:23.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-07-26T07:48:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:23 smithi097 ceph-mon[32231]: pgmap v406: 97 pgs: 5 peering, 92 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-07-26T07:48:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:23 smithi097 ceph-mon[32231]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:23 smithi115 ceph-mon[35364]: pgmap v406: 97 pgs: 5 peering, 92 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-07-26T07:48:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:23 smithi115 ceph-mon[35364]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:23 smithi181 ceph-mon[35869]: pgmap v406: 97 pgs: 5 peering, 92 active+clean; 591 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-07-26T07:48:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:23 smithi181 ceph-mon[35869]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:24.311 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:26.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:25 smithi097 ceph-mon[32231]: pgmap v407: 97 pgs: 3 peering, 94 active+clean; 605 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 83 B/s, 0 objects/s recovering 2024-07-26T07:48:26.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:25 smithi115 ceph-mon[35364]: pgmap v407: 97 pgs: 3 peering, 94 active+clean; 605 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 83 B/s, 0 objects/s recovering 2024-07-26T07:48:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:25 smithi181 ceph-mon[35869]: pgmap v407: 97 pgs: 3 peering, 94 active+clean; 605 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 83 B/s, 0 objects/s recovering 2024-07-26T07:48:26.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.499+0000 7f50f6800640 1 -- 172.21.15.97:0/3513976426 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f010f330 msgr2=0x7f50f010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:26.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.499+0000 7f50f6800640 1 --2- 172.21.15.97:0/3513976426 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f010f330 0x7f50f010f740 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009900 tx=0x7f50dc02a410 comp rx=0 tx=0).stop 2024-07-26T07:48:26.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.499+0000 7f50f6800640 1 -- 172.21.15.97:0/3513976426 shutdown_connections 2024-07-26T07:48:26.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.499+0000 7f50f6800640 1 --2- 172.21.15.97:0/3513976426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50f010fc80 0x7f50f01164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.499+0000 7f50f6800640 1 --2- 172.21.15.97:0/3513976426 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f010f330 0x7f50f010f740 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.499+0000 7f50f6800640 1 --2- 172.21.15.97:0/3513976426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f01099f0 0x7f50f0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.499+0000 7f50f6800640 1 -- 172.21.15.97:0/3513976426 >> 172.21.15.97:0/3513976426 conn(0x7f50f0078ff0 msgr2=0x7f50f0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:26.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.499+0000 7f50f6800640 1 -- 172.21.15.97:0/3513976426 shutdown_connections 2024-07-26T07:48:26.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.500+0000 7f50f6800640 1 -- 172.21.15.97:0/3513976426 wait complete. 2024-07-26T07:48:26.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.501+0000 7f50f6800640 1 Processor -- start 2024-07-26T07:48:26.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.501+0000 7f50f6800640 1 -- start start 2024-07-26T07:48:26.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.501+0000 7f50f6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f01099f0 0x7f50f01a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:26.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.501+0000 7f50f6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f010f330 0x7f50f01a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:26.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.501+0000 7f50f6800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50f010fc80 0x7f50f01a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:26.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.501+0000 7f50f6800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f50f01197f0 con 0x7f50f01099f0 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.501+0000 7f50f6800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f50f0119670 con 0x7f50f010fc80 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.501+0000 7f50f6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f50f0119970 con 0x7f50f010f330 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f01099f0 0x7f50f01a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f01099f0 0x7f50f01a02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47452/0 (socket says 172.21.15.97:47452) 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 -- 172.21.15.97:0/4189887219 learned_addr learned my addr 172.21.15.97:0/4189887219 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5e00640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50f010fc80 0x7f50f01a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f4a00640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f010f330 0x7f50f01a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 -- 172.21.15.97:0/4189887219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f010f330 msgr2=0x7f50f01a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f010f330 0x7f50f01a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 -- 172.21.15.97:0/4189887219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50f010fc80 msgr2=0x7f50f01a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50f010fc80 0x7f50f01a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 -- 172.21.15.97:0/4189887219 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f01a6210 con 0x7f50f01099f0 2024-07-26T07:48:26.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.502+0000 7f50f5400640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f01099f0 0x7f50f01a02a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f50e400d770 tx=0x7f50e400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:26.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.503+0000 7f50ee000640 1 -- 172.21.15.97:0/4189887219 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4017070 con 0x7f50f01099f0 2024-07-26T07:48:26.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.503+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f01a6500 con 0x7f50f01099f0 2024-07-26T07:48:26.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.503+0000 7f50ee000640 1 -- 172.21.15.97:0/4189887219 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50e40054b0 con 0x7f50f01099f0 2024-07-26T07:48:26.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.503+0000 7f50ee000640 1 -- 172.21.15.97:0/4189887219 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4004c60 con 0x7f50f01099f0 2024-07-26T07:48:26.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.503+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f01acc60 con 0x7f50f01099f0 2024-07-26T07:48:26.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.504+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50f0109e00 con 0x7f50f01099f0 2024-07-26T07:48:26.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.507+0000 7f50ee000640 1 -- 172.21.15.97:0/4189887219 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f50e4004e00 con 0x7f50f01099f0 2024-07-26T07:48:26.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.508+0000 7f50ee000640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f50d80b3c60 0x7f50d80b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:26.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.508+0000 7f50ee000640 1 -- 172.21.15.97:0/4189887219 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f50e40e3d20 con 0x7f50f01099f0 2024-07-26T07:48:26.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.509+0000 7f50f4a00640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f50d80b3c60 0x7f50d80b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:26.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.509+0000 7f50f4a00640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f50d80b3c60 0x7f50d80b6170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0057c0 tx=0x7f50dc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:26.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.511+0000 7f50ee000640 1 -- 172.21.15.97:0/4189887219 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f50e40ab190 con 0x7f50f01099f0 2024-07-26T07:48:26.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.735+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f50f0111ab0 con 0x7f50d80b3c60 2024-07-26T07:48:26.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.804+0000 7f50ee000640 1 -- 172.21.15.97:0/4189887219 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f50f0111ab0 con 0x7f50d80b3c60 2024-07-26T07:48:26.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.807+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f50d80b3c60 msgr2=0x7f50d80b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:26.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.807+0000 7f50f6800640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f50d80b3c60 0x7f50d80b6170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0057c0 tx=0x7f50dc000bc0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.807+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f01099f0 msgr2=0x7f50f01a02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:26.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.807+0000 7f50f6800640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f01099f0 0x7f50f01a02a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f50e400d770 tx=0x7f50e400dc30 comp rx=0 tx=0).stop 2024-07-26T07:48:26.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.808+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 shutdown_connections 2024-07-26T07:48:26.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.808+0000 7f50f6800640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f50d80b3c60 0x7f50d80b6170 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.808+0000 7f50f6800640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50f010fc80 0x7f50f01a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.808+0000 7f50f6800640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f010f330 0x7f50f01a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.808+0000 7f50f6800640 1 --2- 172.21.15.97:0/4189887219 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50f01099f0 0x7f50f01a02a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:26.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.808+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 >> 172.21.15.97:0/4189887219 conn(0x7f50f0078ff0 msgr2=0x7f50f0079950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:26.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.808+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 shutdown_connections 2024-07-26T07:48:26.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:26.808+0000 7f50f6800640 1 -- 172.21.15.97:0/4189887219 wait complete. 2024-07-26T07:48:27.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-07-26T07:48:28.125 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:27 smithi115 ceph-mon[35364]: pgmap v408: 97 pgs: 97 active+clean; 605 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s; 83 B/s, 1 objects/s recovering 2024-07-26T07:48:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:27 smithi115 ceph-mon[35364]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:27 smithi181 ceph-mon[35869]: pgmap v408: 97 pgs: 97 active+clean; 605 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s; 83 B/s, 1 objects/s recovering 2024-07-26T07:48:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:27 smithi181 ceph-mon[35869]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:28.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:27 smithi097 ceph-mon[32231]: pgmap v408: 97 pgs: 97 active+clean; 605 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s; 83 B/s, 1 objects/s recovering 2024-07-26T07:48:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:27 smithi097 ceph-mon[32231]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:29 smithi115 ceph-mon[35364]: pgmap v409: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.0 KiB/s wr, 10 op/s; 72 B/s, 1 objects/s recovering 2024-07-26T07:48:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:29 smithi181 ceph-mon[35869]: pgmap v409: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.0 KiB/s wr, 10 op/s; 72 B/s, 1 objects/s recovering 2024-07-26T07:48:30.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 -- 172.21.15.97:0/3829500163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4010f9c0 msgr2=0x7f0c40111de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:30.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 --2- 172.21.15.97:0/3829500163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4010f9c0 0x7f0c40111de0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0c38009ca0 tx=0x7f0c3802a950 comp rx=0 tx=0).stop 2024-07-26T07:48:30.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 -- 172.21.15.97:0/3829500163 shutdown_connections 2024-07-26T07:48:30.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 --2- 172.21.15.97:0/3829500163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c4010f9c0 0x7f0c40111de0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 --2- 172.21.15.97:0/3829500163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c40102760 0x7f0c4010f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 --2- 172.21.15.97:0/3829500163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c40101e10 0x7f0c40102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 -- 172.21.15.97:0/3829500163 >> 172.21.15.97:0/3829500163 conn(0x7f0c400fdcf0 msgr2=0x7f0c40100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:30.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 -- 172.21.15.97:0/3829500163 shutdown_connections 2024-07-26T07:48:30.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.237+0000 7f0c46200640 1 -- 172.21.15.97:0/3829500163 wait complete. 2024-07-26T07:48:30.239 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:29 smithi097 ceph-mon[32231]: pgmap v409: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.0 KiB/s wr, 10 op/s; 72 B/s, 1 objects/s recovering 2024-07-26T07:48:30.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.238+0000 7f0c46200640 1 Processor -- start 2024-07-26T07:48:30.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.239+0000 7f0c46200640 1 -- start start 2024-07-26T07:48:30.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.239+0000 7f0c46200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c40101e10 0x7f0c4019bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:30.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.239+0000 7f0c46200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c40102760 0x7f0c4019c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:30.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.239+0000 7f0c46200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c4010f9c0 0x7f0c401a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:30.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.239+0000 7f0c46200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0c40115330 con 0x7f0c40101e10 2024-07-26T07:48:30.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.239+0000 7f0c46200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401151b0 con 0x7f0c40102760 2024-07-26T07:48:30.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.239+0000 7f0c46200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401154b0 con 0x7f0c4010f9c0 2024-07-26T07:48:30.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.239+0000 7f0c44e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c40101e10 0x7f0c4019bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:30.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c44e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c40101e10 0x7f0c4019bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47476/0 (socket says 172.21.15.97:47476) 2024-07-26T07:48:30.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c44e00640 1 -- 172.21.15.97:0/2668745144 learned_addr learned my addr 172.21.15.97:0/2668745144 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:30.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c3fe00640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c40102760 0x7f0c4019c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:30.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c45800640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c4010f9c0 0x7f0c401a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:30.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c44e00640 1 -- 172.21.15.97:0/2668745144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c4010f9c0 msgr2=0x7f0c401a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:30.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c44e00640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c4010f9c0 0x7f0c401a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c44e00640 1 -- 172.21.15.97:0/2668745144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c40102760 msgr2=0x7f0c4019c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:30.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c44e00640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c40102760 0x7f0c4019c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c44e00640 1 -- 172.21.15.97:0/2668745144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c401a1c30 con 0x7f0c40101e10 2024-07-26T07:48:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c45800640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c4010f9c0 0x7f0c401a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:48:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.240+0000 7f0c3fe00640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c40102760 0x7f0c4019c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:48:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.241+0000 7f0c44e00640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c40101e10 0x7f0c4019bd80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c920 tx=0x7f0c3400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.241+0000 7f0c3d600640 1 -- 172.21.15.97:0/2668745144 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34007c70 con 0x7f0c40101e10 2024-07-26T07:48:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.241+0000 7f0c3d600640 1 -- 172.21.15.97:0/2668745144 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c34007e10 con 0x7f0c40101e10 2024-07-26T07:48:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.241+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c401a1f20 con 0x7f0c40101e10 2024-07-26T07:48:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.241+0000 7f0c3d600640 1 -- 172.21.15.97:0/2668745144 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34018710 con 0x7f0c40101e10 2024-07-26T07:48:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.241+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c401a8740 con 0x7f0c40101e10 2024-07-26T07:48:30.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.243+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c200052c0 con 0x7f0c40101e10 2024-07-26T07:48:30.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.245+0000 7f0c3d600640 1 -- 172.21.15.97:0/2668745144 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f0c340188b0 con 0x7f0c40101e10 2024-07-26T07:48:30.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.246+0000 7f0c3d600640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c280b3d10 0x7f0c280b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:30.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.247+0000 7f0c3d600640 1 -- 172.21.15.97:0/2668745144 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f0c340e3ea0 con 0x7f0c40101e10 2024-07-26T07:48:30.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.247+0000 7f0c3fe00640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c280b3d10 0x7f0c280b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:30.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.248+0000 7f0c3fe00640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c280b3d10 0x7f0c280b6220 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0058c0 tx=0x7f0c2c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:30.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.251+0000 7f0c3d600640 1 -- 172.21.15.97:0/2668745144 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0c34014030 con 0x7f0c40101e10 2024-07-26T07:48:30.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.481+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f0c20000bf0 con 0x7f0c280b3d10 2024-07-26T07:48:30.490 INFO:teuthology.orchestra.run.smithi097.stdout:/volumes/qa/sv_1/731ab551-2987-41ca-823b-3fa815fae64b 2024-07-26T07:48:30.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.489+0000 7f0c3d600640 1 -- 172.21.15.97:0/2668745144 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0c20000bf0 con 0x7f0c280b3d10 2024-07-26T07:48:30.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c280b3d10 msgr2=0x7f0c280b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c280b3d10 0x7f0c280b6220 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0058c0 tx=0x7f0c2c005850 comp rx=0 tx=0).stop 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c40101e10 msgr2=0x7f0c4019bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c40101e10 0x7f0c4019bd80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c920 tx=0x7f0c3400cde0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 shutdown_connections 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c280b3d10 0x7f0c280b6220 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c4010f9c0 0x7f0c401a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c40102760 0x7f0c4019c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 --2- 172.21.15.97:0/2668745144 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c40101e10 0x7f0c4019bd80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.491+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 >> 172.21.15.97:0/2668745144 conn(0x7f0c400fdcf0 msgr2=0x7f0c40111b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:30.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.492+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 shutdown_connections 2024-07-26T07:48:30.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:30.492+0000 7f0c46200640 1 -- 172.21.15.97:0/2668745144 wait complete. 2024-07-26T07:48:31.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:48:31.763 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:32.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:31 smithi097 ceph-mon[32231]: pgmap v410: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 12 op/s; 65 B/s, 0 objects/s recovering 2024-07-26T07:48:32.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:31 smithi097 ceph-mon[32231]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:31 smithi181 ceph-mon[35869]: pgmap v410: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 12 op/s; 65 B/s, 0 objects/s recovering 2024-07-26T07:48:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:31 smithi181 ceph-mon[35869]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:31 smithi115 ceph-mon[35364]: pgmap v410: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 12 op/s; 65 B/s, 0 objects/s recovering 2024-07-26T07:48:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:31 smithi115 ceph-mon[35364]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:48:33.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.985+0000 7feaaec00640 1 -- 172.21.15.97:0/2409784825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa81099f0 msgr2=0x7feaa8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:33.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.985+0000 7feaaec00640 1 --2- 172.21.15.97:0/2409784825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa81099f0 0x7feaa8109e00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fea9c009900 tx=0x7fea9c02a410 comp rx=0 tx=0).stop 2024-07-26T07:48:33.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.985+0000 7feaaec00640 1 -- 172.21.15.97:0/2409784825 shutdown_connections 2024-07-26T07:48:33.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.985+0000 7feaaec00640 1 --2- 172.21.15.97:0/2409784825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaa810fc80 0x7feaa81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:33.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.985+0000 7feaaec00640 1 --2- 172.21.15.97:0/2409784825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaa810f330 0x7feaa810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:33.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.985+0000 7feaaec00640 1 --2- 172.21.15.97:0/2409784825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa81099f0 0x7feaa8109e00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:33.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.985+0000 7feaaec00640 1 -- 172.21.15.97:0/2409784825 >> 172.21.15.97:0/2409784825 conn(0x7feaa8078ff0 msgr2=0x7feaa8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:33.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.986+0000 7feaaec00640 1 -- 172.21.15.97:0/2409784825 shutdown_connections 2024-07-26T07:48:33.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.986+0000 7feaaec00640 1 -- 172.21.15.97:0/2409784825 wait complete. 2024-07-26T07:48:33.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.987+0000 7feaaec00640 1 Processor -- start 2024-07-26T07:48:33.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.987+0000 7feaaec00640 1 -- start start 2024-07-26T07:48:33.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaaec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaa81099f0 0x7feaa81a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaaec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa810f330 0x7feaa81a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaaec00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaa810fc80 0x7feaa81a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaaec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7feaa81198a0 con 0x7feaa810f330 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaaec00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7feaa8119720 con 0x7feaa810fc80 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaaec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feaa8119a20 con 0x7feaa81099f0 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaace00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa810f330 0x7feaa81a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaace00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa810f330 0x7feaa81a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47494/0 (socket says 172.21.15.97:47494) 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.988+0000 7feaace00640 1 -- 172.21.15.97:0/3392617889 learned_addr learned my addr 172.21.15.97:0/3392617889 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.989+0000 7feaae200640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaa810fc80 0x7feaa81a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:33.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.989+0000 7feaad800640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaa81099f0 0x7feaa81a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:33.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.989+0000 7feaace00640 1 -- 172.21.15.97:0/3392617889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaa81099f0 msgr2=0x7feaa81a02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:33.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.989+0000 7feaace00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaa81099f0 0x7feaa81a02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:33.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.989+0000 7feaace00640 1 -- 172.21.15.97:0/3392617889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaa810fc80 msgr2=0x7feaa81a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:33.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.989+0000 7feaace00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaa810fc80 0x7feaa81a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.989+0000 7feaace00640 1 -- 172.21.15.97:0/3392617889 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa81a6160 con 0x7feaa810f330 2024-07-26T07:48:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.989+0000 7feaace00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa810f330 0x7feaa81a0850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fea9400c8c0 tx=0x7fea9400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.990+0000 7feaa6000640 1 -- 172.21.15.97:0/3392617889 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94007bb0 con 0x7feaa810f330 2024-07-26T07:48:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.990+0000 7feaa6000640 1 -- 172.21.15.97:0/3392617889 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea94007d50 con 0x7feaa810f330 2024-07-26T07:48:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.990+0000 7feaa6000640 1 -- 172.21.15.97:0/3392617889 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea940186a0 con 0x7feaa810f330 2024-07-26T07:48:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.990+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa81a6450 con 0x7feaa810f330 2024-07-26T07:48:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.990+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa81acc70 con 0x7feaa810f330 2024-07-26T07:48:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.991+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaa8109e00 con 0x7feaa810f330 2024-07-26T07:48:33.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.995+0000 7feaa6000640 1 -- 172.21.15.97:0/3392617889 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fea9400fa00 con 0x7feaa810f330 2024-07-26T07:48:33.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.997+0000 7feaa6000640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fea900b3c90 0x7fea900b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:33.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.997+0000 7feaa6000640 1 -- 172.21.15.97:0/3392617889 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7fea940e4130 con 0x7feaa810f330 2024-07-26T07:48:33.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.997+0000 7feaad800640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fea900b3c90 0x7fea900b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:33.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.998+0000 7feaad800640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fea900b3c90 0x7fea900b61a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea9c006010 tx=0x7fea9c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:33.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:33.999+0000 7feaa6000640 1 -- 172.21.15.97:0/3392617889 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fea94014030 con 0x7feaa810f330 2024-07-26T07:48:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:33 smithi181 ceph-mon[35869]: pgmap v411: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.3 KiB/s wr, 10 op/s; 55 B/s, 0 objects/s recovering 2024-07-26T07:48:34.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.235+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feaa81a13e0 con 0x7feaa810f330 2024-07-26T07:48:34.236 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:33 smithi097 ceph-mon[32231]: pgmap v411: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.3 KiB/s wr, 10 op/s; 55 B/s, 0 objects/s recovering 2024-07-26T07:48:34.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.236+0000 7feaa6000640 1 -- 172.21.15.97:0/3392617889 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11990 (secure 0 0 0) 0x7feaa81a13e0 con 0x7feaa810f330 2024-07-26T07:48:34.237 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:48:34.238 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":27,"btime":"2024-07-26T07:48:11:830543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:48:11.504745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:48:34.238 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 27 2024-07-26T07:48:34.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fea900b3c90 msgr2=0x7fea900b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:34.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fea900b3c90 0x7fea900b61a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea9c006010 tx=0x7fea9c000bc0 comp rx=0 tx=0).stop 2024-07-26T07:48:34.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa810f330 msgr2=0x7feaa81a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:34.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa810f330 0x7feaa81a0850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fea9400c8c0 tx=0x7fea9400cd80 comp rx=0 tx=0).stop 2024-07-26T07:48:34.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 shutdown_connections 2024-07-26T07:48:34.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fea900b3c90 0x7fea900b61a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:34.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaa810fc80 0x7feaa81a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:34.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaa810f330 0x7feaa81a0850 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:34.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 --2- 172.21.15.97:0/3392617889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaa81099f0 0x7feaa81a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:34.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 >> 172.21.15.97:0/3392617889 conn(0x7feaa8078ff0 msgr2=0x7feaa80797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:34.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 shutdown_connections 2024-07-26T07:48:34.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:34.239+0000 7feaaec00640 1 -- 172.21.15.97:0/3392617889 wait complete. 2024-07-26T07:48:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:33 smithi115 ceph-mon[35364]: pgmap v411: 97 pgs: 97 active+clean; 625 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.3 KiB/s wr, 10 op/s; 55 B/s, 0 objects/s recovering 2024-07-26T07:48:35.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:48:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:34 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3392617889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:48:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:34 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3392617889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:48:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:34 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3392617889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:48:35.491 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:36.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:35 smithi097 ceph-mon[32231]: pgmap v412: 97 pgs: 97 active+clean; 633 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.2 KiB/s wr, 12 op/s; 55 B/s, 0 objects/s recovering 2024-07-26T07:48:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:36 smithi115 ceph-mon[35364]: pgmap v412: 97 pgs: 97 active+clean; 633 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.2 KiB/s wr, 12 op/s; 55 B/s, 0 objects/s recovering 2024-07-26T07:48:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:36 smithi181 ceph-mon[35869]: pgmap v412: 97 pgs: 97 active+clean; 633 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.2 KiB/s wr, 12 op/s; 55 B/s, 0 objects/s recovering 2024-07-26T07:48:37.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 -- 172.21.15.97:0/4282421182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 msgr2=0x7fae8010fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:37.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 --2- 172.21.15.97:0/4282421182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 0x7fae8010fe80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fae78009ca0 tx=0x7fae7802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:48:37.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 -- 172.21.15.97:0/4282421182 shutdown_connections 2024-07-26T07:48:37.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 --2- 172.21.15.97:0/4282421182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 0x7fae8010fe80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:37.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 --2- 172.21.15.97:0/4282421182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae80108d00 0x7fae80109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:37.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 --2- 172.21.15.97:0/4282421182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae801033c0 0x7fae801037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:37.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 -- 172.21.15.97:0/4282421182 >> 172.21.15.97:0/4282421182 conn(0x7fae800fee10 msgr2=0x7fae80101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:37.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 -- 172.21.15.97:0/4282421182 shutdown_connections 2024-07-26T07:48:37.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.902+0000 7fae7f200640 1 -- 172.21.15.97:0/4282421182 wait complete. 2024-07-26T07:48:37.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.903+0000 7fae7f200640 1 Processor -- start 2024-07-26T07:48:37.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.903+0000 7fae7f200640 1 -- start start 2024-07-26T07:48:37.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae801033c0 0x7fae8019be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:37.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7f200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae80108d00 0x7fae8019c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:37.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 0x7fae801a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:37.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fae80113220 con 0x7fae80109650 2024-07-26T07:48:37.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7f200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fae801130a0 con 0x7fae80108d00 2024-07-26T07:48:37.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae801133a0 con 0x7fae801033c0 2024-07-26T07:48:37.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 0x7fae801a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:37.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 0x7fae801a1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39892/0 (socket says 172.21.15.97:39892) 2024-07-26T07:48:37.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7e800640 1 -- 172.21.15.97:0/2099216245 learned_addr learned my addr 172.21.15.97:0/2099216245 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:37.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae7de00640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae801033c0 0x7fae8019be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:37.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.904+0000 7fae77e00640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae80108d00 0x7fae8019c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:37.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae7e800640 1 -- 172.21.15.97:0/2099216245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae801033c0 msgr2=0x7fae8019be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:37.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae7e800640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae801033c0 0x7fae8019be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:37.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae7e800640 1 -- 172.21.15.97:0/2099216245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae80108d00 msgr2=0x7fae8019c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:37.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae7e800640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae80108d00 0x7fae8019c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae7e800640 1 -- 172.21.15.97:0/2099216245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae801a1ce0 con 0x7fae80109650 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae7de00640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae801033c0 0x7fae8019be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae77e00640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae80108d00 0x7fae8019c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae7e800640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 0x7fae801a1670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fae78031040 tx=0x7fae78032bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae76a00640 1 -- 172.21.15.97:0/2099216245 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78004040 con 0x7fae80109650 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae76a00640 1 -- 172.21.15.97:0/2099216245 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fae78004670 con 0x7fae80109650 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae76a00640 1 -- 172.21.15.97:0/2099216245 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78030d50 con 0x7fae80109650 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.905+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae801a1f70 con 0x7fae80109650 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.906+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae801a87f0 con 0x7fae80109650 2024-07-26T07:48:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.907+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae801037d0 con 0x7fae80109650 2024-07-26T07:48:37.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.909+0000 7fae76a00640 1 -- 172.21.15.97:0/2099216245 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fae780305c0 con 0x7fae80109650 2024-07-26T07:48:37.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.911+0000 7fae76a00640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae640b3d10 0x7fae640b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:37.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.911+0000 7fae76a00640 1 -- 172.21.15.97:0/2099216245 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7fae78105490 con 0x7fae80109650 2024-07-26T07:48:37.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.911+0000 7fae7de00640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae640b3d10 0x7fae640b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:37.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.912+0000 7fae7de00640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae640b3d10 0x7fae640b6220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae7000abe0 tx=0x7fae7000a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:37.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:37.913+0000 7fae76a00640 1 -- 172.21.15.97:0/2099216245 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fae780ccaa0 con 0x7fae80109650 2024-07-26T07:48:38.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.116+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae801a8a20 con 0x7fae80109650 2024-07-26T07:48:38.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.118+0000 7fae76a00640 1 -- 172.21.15.97:0/2099216245 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22701 (secure 0 0 0) 0x7fae780cc3a0 con 0x7fae80109650 2024-07-26T07:48:38.119 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:48:38.120 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":84,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:48:19.900531+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T07:47:40.641206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T07:47:44.357412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:48:38.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.120+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae640b3d10 msgr2=0x7fae640b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:38.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.120+0000 7fae7f200640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae640b3d10 0x7fae640b6220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae7000abe0 tx=0x7fae7000a5c0 comp rx=0 tx=0).stop 2024-07-26T07:48:38.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.120+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 msgr2=0x7fae801a1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:38.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.120+0000 7fae7f200640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 0x7fae801a1670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fae78031040 tx=0x7fae78032bc0 comp rx=0 tx=0).stop 2024-07-26T07:48:38.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.121+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 shutdown_connections 2024-07-26T07:48:38.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.121+0000 7fae7f200640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fae640b3d10 0x7fae640b6220 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:38.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.121+0000 7fae7f200640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae80109650 0x7fae801a1670 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:38.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.121+0000 7fae7f200640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fae80108d00 0x7fae8019c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:38.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.121+0000 7fae7f200640 1 --2- 172.21.15.97:0/2099216245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae801033c0 0x7fae8019be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:38.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.121+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 >> 172.21.15.97:0/2099216245 conn(0x7fae800fee10 msgr2=0x7fae801083a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:38.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.121+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 shutdown_connections 2024-07-26T07:48:38.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:38.121+0000 7fae7f200640 1 -- 172.21.15.97:0/2099216245 wait complete. 2024-07-26T07:48:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:38 smithi115 ceph-mon[35364]: pgmap v413: 97 pgs: 97 active+clean; 633 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.9 KiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-26T07:48:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:38 smithi181 ceph-mon[35869]: pgmap v413: 97 pgs: 97 active+clean; 633 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.9 KiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-26T07:48:38.490 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:38 smithi097 ceph-mon[32231]: pgmap v413: 97 pgs: 97 active+clean; 633 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.9 KiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-26T07:48:38.924 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T07:48:38.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T07:48:39.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:39 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2099216245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:48:39.388 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:39 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2099216245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:48:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:39 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2099216245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:48:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:40 smithi097 ceph-mon[32231]: pgmap v414: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-07-26T07:48:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:48:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:40 smithi115 ceph-mon[35364]: pgmap v414: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-07-26T07:48:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:48:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:40 smithi181 ceph-mon[35869]: pgmap v414: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-07-26T07:48:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:48:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:41.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 -- 172.21.15.97:0/3532377643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077370 msgr2=0x7f5e84077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:41.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 --2- 172.21.15.97:0/3532377643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077370 0x7f5e84077780 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009900 tx=0x7f5e7402a410 comp rx=0 tx=0).stop 2024-07-26T07:48:41.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 -- 172.21.15.97:0/3532377643 shutdown_connections 2024-07-26T07:48:41.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 --2- 172.21.15.97:0/3532377643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e84113f80 0x7f5e841163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 --2- 172.21.15.97:0/3532377643 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e84077cc0 0x7f5e84113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 --2- 172.21.15.97:0/3532377643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077370 0x7f5e84077780 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 -- 172.21.15.97:0/3532377643 >> 172.21.15.97:0/3532377643 conn(0x7f5e84101010 msgr2=0x7f5e84103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:41.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 -- 172.21.15.97:0/3532377643 shutdown_connections 2024-07-26T07:48:41.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.745+0000 7f5e89800640 1 -- 172.21.15.97:0/3532377643 wait complete. 2024-07-26T07:48:41.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.747+0000 7f5e89800640 1 Processor -- start 2024-07-26T07:48:41.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.747+0000 7f5e89800640 1 -- start start 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.747+0000 7f5e89800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e84077370 0x7f5e841a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.747+0000 7f5e89800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077cc0 0x7f5e841a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.747+0000 7f5e89800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e84113f80 0x7f5e841a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.747+0000 7f5e89800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5e841197b0 con 0x7f5e84077cc0 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.747+0000 7f5e89800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5e84119630 con 0x7f5e84077370 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.747+0000 7f5e89800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5e84119930 con 0x7f5e84113f80 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077cc0 0x7f5e841a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e83e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e84077370 0x7f5e841a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e83e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e84077370 0x7f5e841a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:46758/0 (socket says 172.21.15.97:46758) 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e7be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077cc0 0x7f5e841a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39902/0 (socket says 172.21.15.97:39902) 2024-07-26T07:48:41.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e83e00640 1 -- 172.21.15.97:0/3167694179 learned_addr learned my addr 172.21.15.97:0/3167694179 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:41.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e88e00640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e84113f80 0x7f5e841a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e7be00640 1 -- 172.21.15.97:0/3167694179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e84113f80 msgr2=0x7f5e841a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e7be00640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e84113f80 0x7f5e841a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e7be00640 1 -- 172.21.15.97:0/3167694179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e84077370 msgr2=0x7f5e841a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e7be00640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e84077370 0x7f5e841a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e7be00640 1 -- 172.21.15.97:0/3167694179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e841a61b0 con 0x7f5e84077cc0 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.748+0000 7f5e83e00640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e84077370 0x7f5e841a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.749+0000 7f5e7be00640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077cc0 0x7f5e841a07e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5e8406c310 tx=0x7f5e6c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.749+0000 7f5e81600640 1 -- 172.21.15.97:0/3167694179 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c00cc40 con 0x7f5e84077cc0 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.749+0000 7f5e81600640 1 -- 172.21.15.97:0/3167694179 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e6c00cde0 con 0x7f5e84077cc0 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.749+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e841a64a0 con 0x7f5e84077cc0 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.749+0000 7f5e81600640 1 -- 172.21.15.97:0/3167694179 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c0186f0 con 0x7f5e84077cc0 2024-07-26T07:48:41.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.749+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e841acc00 con 0x7f5e84077cc0 2024-07-26T07:48:41.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.750+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e84077780 con 0x7f5e84077cc0 2024-07-26T07:48:41.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.754+0000 7f5e81600640 1 -- 172.21.15.97:0/3167694179 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f5e6c010a70 con 0x7f5e84077cc0 2024-07-26T07:48:41.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.756+0000 7f5e81600640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5e680b3ce0 0x7f5e680b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:41.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.756+0000 7f5e81600640 1 -- 172.21.15.97:0/3167694179 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f5e6c0e45a0 con 0x7f5e84077cc0 2024-07-26T07:48:41.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.756+0000 7f5e83e00640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5e680b3ce0 0x7f5e680b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:41.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.757+0000 7f5e83e00640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5e680b3ce0 0x7f5e680b61f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5e74032fd0 tx=0x7f5e740095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:41.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.758+0000 7f5e81600640 1 -- 172.21.15.97:0/3167694179 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5e6c0aba10 con 0x7f5e84077cc0 2024-07-26T07:48:41.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.968+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e84063230 con 0x7f5e680b3ce0 2024-07-26T07:48:41.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.984+0000 7f5e81600640 1 -- 172.21.15.97:0/3167694179 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258821 (secure 0 0 0) 0x7f5e84063230 con 0x7f5e680b3ce0 2024-07-26T07:48:41.989 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:48:41.991 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-07-26T07:48:41.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5e680b3ce0 msgr2=0x7f5e680b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:41.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5e680b3ce0 0x7f5e680b61f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5e74032fd0 tx=0x7f5e740095c0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077cc0 msgr2=0x7f5e841a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:41.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077cc0 0x7f5e841a07e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5e8406c310 tx=0x7f5e6c00ee50 comp rx=0 tx=0).stop 2024-07-26T07:48:41.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 shutdown_connections 2024-07-26T07:48:41.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5e680b3ce0 0x7f5e680b61f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e84113f80 0x7f5e841a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5e84077cc0 0x7f5e841a07e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 --2- 172.21.15.97:0/3167694179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5e84077370 0x7f5e841a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:41.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 >> 172.21.15.97:0/3167694179 conn(0x7f5e84101010 msgr2=0x7f5e84103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:41.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 shutdown_connections 2024-07-26T07:48:41.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:41.991+0000 7f5e89800640 1 -- 172.21.15.97:0/3167694179 wait complete. 2024-07-26T07:48:42.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:42 smithi097 ceph-mon[32231]: pgmap v415: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:48:42.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:42 smithi115 ceph-mon[35364]: pgmap v415: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:48:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:42 smithi181 ceph-mon[35869]: pgmap v415: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:48:42.785 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-07-26T07:48:40.136738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652920,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":412,"num_read_kb":299,"num_write":368,"num_write_kb":2281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403792,"kb_used_data":13012,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538032,"statfs":{"total":1151940427776,"available":1151526944768,"internally_reserved":0,"allocated":13324288,"data_stored":7299561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13421,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":74,"num_write":20,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005964"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921680+0000","last_change":"2024-07-26T07:47:45.843670+0000","last_active":"2024-07-26T07:48:19.921680+0000","last_peered":"2024-07-26T07:48:19.921680+0000","last_clean":"2024-07-26T07:48:19.921680+0000","last_became_active":"2024-07-26T07:47:45.841137+0000","last_became_peered":"2024-07-26T07:47:45.841137+0000","last_unstale":"2024-07-26T07:48:19.921680+0000","last_undegraded":"2024-07-26T07:48:19.921680+0000","last_fullsized":"2024-07-26T07:48:19.921680+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:38.063503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923347+0000","last_change":"2024-07-26T07:47:45.849326+0000","last_active":"2024-07-26T07:48:19.923347+0000","last_peered":"2024-07-26T07:48:19.923347+0000","last_clean":"2024-07-26T07:48:19.923347+0000","last_became_active":"2024-07-26T07:47:45.847761+0000","last_became_peered":"2024-07-26T07:47:45.847761+0000","last_unstale":"2024-07-26T07:48:19.923347+0000","last_undegraded":"2024-07-26T07:48:19.923347+0000","last_fullsized":"2024-07-26T07:48:19.923347+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:31.098794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.919179+0000","last_change":"2024-07-26T07:47:45.852343+0000","last_active":"2024-07-26T07:48:19.919179+0000","last_peered":"2024-07-26T07:48:19.919179+0000","last_clean":"2024-07-26T07:48:19.919179+0000","last_became_active":"2024-07-26T07:47:45.851950+0000","last_became_peered":"2024-07-26T07:47:45.851950+0000","last_unstale":"2024-07-26T07:48:19.919179+0000","last_undegraded":"2024-07-26T07:48:19.919179+0000","last_fullsized":"2024-07-26T07:48:19.919179+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:41.030903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.999037+0000","last_change":"2024-07-26T07:47:45.857042+0000","last_active":"2024-07-26T07:48:19.999037+0000","last_peered":"2024-07-26T07:48:19.999037+0000","last_clean":"2024-07-26T07:48:19.999037+0000","last_became_active":"2024-07-26T07:47:45.855145+0000","last_became_peered":"2024-07-26T07:47:45.855145+0000","last_unstale":"2024-07-26T07:48:19.999037+0000","last_undegraded":"2024-07-26T07:48:19.999037+0000","last_fullsized":"2024-07-26T07:48:19.999037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:32.233019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924168+0000","last_change":"2024-07-26T07:47:45.838560+0000","last_active":"2024-07-26T07:48:19.924168+0000","last_peered":"2024-07-26T07:48:19.924168+0000","last_clean":"2024-07-26T07:48:19.924168+0000","last_became_active":"2024-07-26T07:47:45.835578+0000","last_became_peered":"2024-07-26T07:47:45.835578+0000","last_unstale":"2024-07-26T07:48:19.924168+0000","last_undegraded":"2024-07-26T07:48:19.924168+0000","last_fullsized":"2024-07-26T07:48:19.924168+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:09:40.445625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.995589+0000","last_change":"2024-07-26T07:47:45.843980+0000","last_active":"2024-07-26T07:48:19.995589+0000","last_peered":"2024-07-26T07:48:19.995589+0000","last_clean":"2024-07-26T07:48:19.995589+0000","last_became_active":"2024-07-26T07:47:45.843546+0000","last_became_peered":"2024-07-26T07:47:45.843546+0000","last_unstale":"2024-07-26T07:48:19.995589+0000","last_undegraded":"2024-07-26T07:48:19.995589+0000","last_fullsized":"2024-07-26T07:48:19.995589+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:06.839136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.932049+0000","last_change":"2024-07-26T07:47:45.854427+0000","last_active":"2024-07-26T07:48:19.932049+0000","last_peered":"2024-07-26T07:48:19.932049+0000","last_clean":"2024-07-26T07:48:19.932049+0000","last_became_active":"2024-07-26T07:47:45.853495+0000","last_became_peered":"2024-07-26T07:47:45.853495+0000","last_unstale":"2024-07-26T07:48:19.932049+0000","last_undegraded":"2024-07-26T07:48:19.932049+0000","last_fullsized":"2024-07-26T07:48:19.932049+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:34:34.110581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920807+0000","last_change":"2024-07-26T07:47:45.847732+0000","last_active":"2024-07-26T07:48:19.920807+0000","last_peered":"2024-07-26T07:48:19.920807+0000","last_clean":"2024-07-26T07:48:19.920807+0000","last_became_active":"2024-07-26T07:47:45.845583+0000","last_became_peered":"2024-07-26T07:47:45.845583+0000","last_unstale":"2024-07-26T07:48:19.920807+0000","last_undegraded":"2024-07-26T07:48:19.920807+0000","last_fullsized":"2024-07-26T07:48:19.920807+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:07:09.370076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.913159+0000","last_change":"2024-07-26T07:47:45.849962+0000","last_active":"2024-07-26T07:48:19.913159+0000","last_peered":"2024-07-26T07:48:19.913159+0000","last_clean":"2024-07-26T07:48:19.913159+0000","last_became_active":"2024-07-26T07:47:45.845766+0000","last_became_peered":"2024-07-26T07:47:45.845766+0000","last_unstale":"2024-07-26T07:48:19.913159+0000","last_undegraded":"2024-07-26T07:48:19.913159+0000","last_fullsized":"2024-07-26T07:48:19.913159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:56.602825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.996633+0000","last_change":"2024-07-26T07:47:45.857287+0000","last_active":"2024-07-26T07:48:19.996633+0000","last_peered":"2024-07-26T07:48:19.996633+0000","last_clean":"2024-07-26T07:48:19.996633+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:48:19.996633+0000","last_undegraded":"2024-07-26T07:48:19.996633+0000","last_fullsized":"2024-07-26T07:48:19.996633+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:38:24.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933958+0000","last_change":"2024-07-26T07:47:45.858369+0000","last_active":"2024-07-26T07:48:19.933958+0000","last_peered":"2024-07-26T07:48:19.933958+0000","last_clean":"2024-07-26T07:48:19.933958+0000","last_became_active":"2024-07-26T07:47:45.856685+0000","last_became_peered":"2024-07-26T07:47:45.856685+0000","last_unstale":"2024-07-26T07:48:19.933958+0000","last_undegraded":"2024-07-26T07:48:19.933958+0000","last_fullsized":"2024-07-26T07:48:19.933958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:44.877953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:20.002675+0000","last_change":"2024-07-26T07:48:20.001394+0000","last_active":"2024-07-26T07:48:20.002675+0000","last_peered":"2024-07-26T07:48:20.002675+0000","last_clean":"2024-07-26T07:48:20.002675+0000","last_became_active":"2024-07-26T07:48:20.000751+0000","last_became_peered":"2024-07-26T07:48:20.000751+0000","last_unstale":"2024-07-26T07:48:20.002675+0000","last_undegraded":"2024-07-26T07:48:20.002675+0000","last_fullsized":"2024-07-26T07:48:20.002675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:26:24.513652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911865+0000","last_change":"2024-07-26T07:47:45.847931+0000","last_active":"2024-07-26T07:48:19.911865+0000","last_peered":"2024-07-26T07:48:19.911865+0000","last_clean":"2024-07-26T07:48:19.911865+0000","last_became_active":"2024-07-26T07:47:45.847484+0000","last_became_peered":"2024-07-26T07:47:45.847484+0000","last_unstale":"2024-07-26T07:48:19.911865+0000","last_undegraded":"2024-07-26T07:48:19.911865+0000","last_fullsized":"2024-07-26T07:48:19.911865+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:04.498360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922963+0000","last_change":"2024-07-26T07:47:45.846484+0000","last_active":"2024-07-26T07:48:19.922963+0000","last_peered":"2024-07-26T07:48:19.922963+0000","last_clean":"2024-07-26T07:48:19.922963+0000","last_became_active":"2024-07-26T07:47:45.844772+0000","last_became_peered":"2024-07-26T07:47:45.844772+0000","last_unstale":"2024-07-26T07:48:19.922963+0000","last_undegraded":"2024-07-26T07:48:19.922963+0000","last_fullsized":"2024-07-26T07:48:19.922963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:54.318026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.918081+0000","last_change":"2024-07-26T07:47:45.849840+0000","last_active":"2024-07-26T07:48:19.918081+0000","last_peered":"2024-07-26T07:48:19.918081+0000","last_clean":"2024-07-26T07:48:19.918081+0000","last_became_active":"2024-07-26T07:47:45.848744+0000","last_became_peered":"2024-07-26T07:47:45.848744+0000","last_unstale":"2024-07-26T07:48:19.918081+0000","last_undegraded":"2024-07-26T07:48:19.918081+0000","last_fullsized":"2024-07-26T07:48:19.918081+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:44.675382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911439+0000","last_change":"2024-07-26T07:47:45.847176+0000","last_active":"2024-07-26T07:48:19.911439+0000","last_peered":"2024-07-26T07:48:19.911439+0000","last_clean":"2024-07-26T07:48:19.911439+0000","last_became_active":"2024-07-26T07:47:45.845936+0000","last_became_peered":"2024-07-26T07:47:45.845936+0000","last_unstale":"2024-07-26T07:48:19.911439+0000","last_undegraded":"2024-07-26T07:48:19.911439+0000","last_fullsized":"2024-07-26T07:48:19.911439+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:02.245098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921801+0000","last_change":"2024-07-26T07:47:45.853210+0000","last_active":"2024-07-26T07:48:19.921801+0000","last_peered":"2024-07-26T07:48:19.921801+0000","last_clean":"2024-07-26T07:48:19.921801+0000","last_became_active":"2024-07-26T07:47:45.852766+0000","last_became_peered":"2024-07-26T07:47:45.852766+0000","last_unstale":"2024-07-26T07:48:19.921801+0000","last_undegraded":"2024-07-26T07:48:19.921801+0000","last_fullsized":"2024-07-26T07:48:19.921801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:57.406748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924715+0000","last_change":"2024-07-26T07:47:45.850699+0000","last_active":"2024-07-26T07:48:19.924715+0000","last_peered":"2024-07-26T07:48:19.924715+0000","last_clean":"2024-07-26T07:48:19.924715+0000","last_became_active":"2024-07-26T07:47:45.850314+0000","last_became_peered":"2024-07-26T07:47:45.850314+0000","last_unstale":"2024-07-26T07:48:19.924715+0000","last_undegraded":"2024-07-26T07:48:19.924715+0000","last_fullsized":"2024-07-26T07:48:19.924715+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:09.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920050+0000","last_change":"2024-07-26T07:47:45.851063+0000","last_active":"2024-07-26T07:48:19.920050+0000","last_peered":"2024-07-26T07:48:19.920050+0000","last_clean":"2024-07-26T07:48:19.920050+0000","last_became_active":"2024-07-26T07:47:45.850164+0000","last_became_peered":"2024-07-26T07:47:45.850164+0000","last_unstale":"2024-07-26T07:48:19.920050+0000","last_undegraded":"2024-07-26T07:48:19.920050+0000","last_fullsized":"2024-07-26T07:48:19.920050+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:31.597551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920032+0000","last_change":"2024-07-26T07:47:45.861207+0000","last_active":"2024-07-26T07:48:19.920032+0000","last_peered":"2024-07-26T07:48:19.920032+0000","last_clean":"2024-07-26T07:48:19.920032+0000","last_became_active":"2024-07-26T07:47:45.860179+0000","last_became_peered":"2024-07-26T07:47:45.860179+0000","last_unstale":"2024-07-26T07:48:19.920032+0000","last_undegraded":"2024-07-26T07:48:19.920032+0000","last_fullsized":"2024-07-26T07:48:19.920032+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:12:51.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911971+0000","last_change":"2024-07-26T07:47:45.846197+0000","last_active":"2024-07-26T07:48:19.911971+0000","last_peered":"2024-07-26T07:48:19.911971+0000","last_clean":"2024-07-26T07:48:19.911971+0000","last_became_active":"2024-07-26T07:47:45.845238+0000","last_became_peered":"2024-07-26T07:47:45.845238+0000","last_unstale":"2024-07-26T07:48:19.911971+0000","last_undegraded":"2024-07-26T07:48:19.911971+0000","last_fullsized":"2024-07-26T07:48:19.911971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:29.900684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911834+0000","last_change":"2024-07-26T07:47:45.847963+0000","last_active":"2024-07-26T07:48:19.911834+0000","last_peered":"2024-07-26T07:48:19.911834+0000","last_clean":"2024-07-26T07:48:19.911834+0000","last_became_active":"2024-07-26T07:47:45.847513+0000","last_became_peered":"2024-07-26T07:47:45.847513+0000","last_unstale":"2024-07-26T07:48:19.911834+0000","last_undegraded":"2024-07-26T07:48:19.911834+0000","last_fullsized":"2024-07-26T07:48:19.911834+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:51.082816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921823+0000","last_change":"2024-07-26T07:47:45.853273+0000","last_active":"2024-07-26T07:48:19.921823+0000","last_peered":"2024-07-26T07:48:19.921823+0000","last_clean":"2024-07-26T07:48:19.921823+0000","last_became_active":"2024-07-26T07:47:45.852894+0000","last_became_peered":"2024-07-26T07:47:45.852894+0000","last_unstale":"2024-07-26T07:48:19.921823+0000","last_undegraded":"2024-07-26T07:48:19.921823+0000","last_fullsized":"2024-07-26T07:48:19.921823+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:57:12.945228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.932104+0000","last_change":"2024-07-26T07:48:19.931938+0000","last_active":"2024-07-26T07:48:19.932104+0000","last_peered":"2024-07-26T07:48:19.932104+0000","last_clean":"2024-07-26T07:48:19.932104+0000","last_became_active":"2024-07-26T07:48:19.930976+0000","last_became_peered":"2024-07-26T07:48:19.930976+0000","last_unstale":"2024-07-26T07:48:19.932104+0000","last_undegraded":"2024-07-26T07:48:19.932104+0000","last_fullsized":"2024-07-26T07:48:19.932104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:59.610464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923540+0000","last_change":"2024-07-26T07:47:45.849736+0000","last_active":"2024-07-26T07:48:19.923540+0000","last_peered":"2024-07-26T07:48:19.923540+0000","last_clean":"2024-07-26T07:48:19.923540+0000","last_became_active":"2024-07-26T07:47:45.849335+0000","last_became_peered":"2024-07-26T07:47:45.849335+0000","last_unstale":"2024-07-26T07:48:19.923540+0000","last_undegraded":"2024-07-26T07:48:19.923540+0000","last_fullsized":"2024-07-26T07:48:19.923540+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:20:12.758331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922915+0000","last_change":"2024-07-26T07:47:45.847250+0000","last_active":"2024-07-26T07:48:19.922915+0000","last_peered":"2024-07-26T07:48:19.922915+0000","last_clean":"2024-07-26T07:48:19.922915+0000","last_became_active":"2024-07-26T07:47:45.845564+0000","last_became_peered":"2024-07-26T07:47:45.845564+0000","last_unstale":"2024-07-26T07:48:19.922915+0000","last_undegraded":"2024-07-26T07:48:19.922915+0000","last_fullsized":"2024-07-26T07:48:19.922915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:03.616782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924228+0000","last_change":"2024-07-26T07:47:45.850462+0000","last_active":"2024-07-26T07:48:19.924228+0000","last_peered":"2024-07-26T07:48:19.924228+0000","last_clean":"2024-07-26T07:48:19.924228+0000","last_became_active":"2024-07-26T07:47:45.850036+0000","last_became_peered":"2024-07-26T07:47:45.850036+0000","last_unstale":"2024-07-26T07:48:19.924228+0000","last_undegraded":"2024-07-26T07:48:19.924228+0000","last_fullsized":"2024-07-26T07:48:19.924228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:38.374782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911747+0000","last_change":"2024-07-26T07:47:45.859294+0000","last_active":"2024-07-26T07:48:19.911747+0000","last_peered":"2024-07-26T07:48:19.911747+0000","last_clean":"2024-07-26T07:48:19.911747+0000","last_became_active":"2024-07-26T07:47:45.858812+0000","last_became_peered":"2024-07-26T07:47:45.858812+0000","last_unstale":"2024-07-26T07:48:19.911747+0000","last_undegraded":"2024-07-26T07:48:19.911747+0000","last_fullsized":"2024-07-26T07:48:19.911747+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:11.689717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.931110+0000","last_change":"2024-07-26T07:48:19.930943+0000","last_active":"2024-07-26T07:48:19.931110+0000","last_peered":"2024-07-26T07:48:19.931110+0000","last_clean":"2024-07-26T07:48:19.931110+0000","last_became_active":"2024-07-26T07:48:19.929758+0000","last_became_peered":"2024-07-26T07:48:19.929758+0000","last_unstale":"2024-07-26T07:48:19.931110+0000","last_undegraded":"2024-07-26T07:48:19.931110+0000","last_fullsized":"2024-07-26T07:48:19.931110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:29:27.657376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920446+0000","last_change":"2024-07-26T07:47:45.846867+0000","last_active":"2024-07-26T07:48:19.920446+0000","last_peered":"2024-07-26T07:48:19.920446+0000","last_clean":"2024-07-26T07:48:19.920446+0000","last_became_active":"2024-07-26T07:47:45.846455+0000","last_became_peered":"2024-07-26T07:47:45.846455+0000","last_unstale":"2024-07-26T07:48:19.920446+0000","last_undegraded":"2024-07-26T07:48:19.920446+0000","last_fullsized":"2024-07-26T07:48:19.920446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:14.997452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922151+0000","last_change":"2024-07-26T07:47:45.846179+0000","last_active":"2024-07-26T07:48:19.922151+0000","last_peered":"2024-07-26T07:48:19.922151+0000","last_clean":"2024-07-26T07:48:19.922151+0000","last_became_active":"2024-07-26T07:47:45.844880+0000","last_became_peered":"2024-07-26T07:47:45.844880+0000","last_unstale":"2024-07-26T07:48:19.922151+0000","last_undegraded":"2024-07-26T07:48:19.922151+0000","last_fullsized":"2024-07-26T07:48:19.922151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:00:50.899573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.935350+0000","last_change":"2024-07-26T07:48:19.935196+0000","last_active":"2024-07-26T07:48:19.935350+0000","last_peered":"2024-07-26T07:48:19.935350+0000","last_clean":"2024-07-26T07:48:19.935350+0000","last_became_active":"2024-07-26T07:48:19.933730+0000","last_became_peered":"2024-07-26T07:48:19.933730+0000","last_unstale":"2024-07-26T07:48:19.935350+0000","last_undegraded":"2024-07-26T07:48:19.935350+0000","last_fullsized":"2024-07-26T07:48:19.935350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:10.305122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.919717+0000","last_change":"2024-07-26T07:47:45.843360+0000","last_active":"2024-07-26T07:48:19.919717+0000","last_peered":"2024-07-26T07:48:19.919717+0000","last_clean":"2024-07-26T07:48:19.919717+0000","last_became_active":"2024-07-26T07:47:45.842971+0000","last_became_peered":"2024-07-26T07:47:45.842971+0000","last_unstale":"2024-07-26T07:48:19.919717+0000","last_undegraded":"2024-07-26T07:48:19.919717+0000","last_fullsized":"2024-07-26T07:48:19.919717+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:58.687638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"82'5","reported_seq":62,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:33.922748+0000","last_change":"2024-07-26T07:47:41.787760+0000","last_active":"2024-07-26T07:48:33.922748+0000","last_peered":"2024-07-26T07:48:33.922748+0000","last_clean":"2024-07-26T07:48:33.922748+0000","last_became_active":"2024-07-26T07:47:41.787197+0000","last_became_peered":"2024-07-26T07:47:41.787197+0000","last_unstale":"2024-07-26T07:48:33.922748+0000","last_undegraded":"2024-07-26T07:48:33.922748+0000","last_fullsized":"2024-07-26T07:48:33.922748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:14.597759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.913354+0000","last_change":"2024-07-26T07:47:45.846822+0000","last_active":"2024-07-26T07:48:19.913354+0000","last_peered":"2024-07-26T07:48:19.913354+0000","last_clean":"2024-07-26T07:48:19.913354+0000","last_became_active":"2024-07-26T07:47:45.846398+0000","last_became_peered":"2024-07-26T07:47:45.846398+0000","last_unstale":"2024-07-26T07:48:19.913354+0000","last_undegraded":"2024-07-26T07:48:19.913354+0000","last_fullsized":"2024-07-26T07:48:19.913354+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:33:34.727843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.995470+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:48:19.995470+0000","last_peered":"2024-07-26T07:48:19.995470+0000","last_clean":"2024-07-26T07:48:19.995470+0000","last_became_active":"2024-07-26T07:47:41.782077+0000","last_became_peered":"2024-07-26T07:47:41.782077+0000","last_unstale":"2024-07-26T07:48:19.995470+0000","last_undegraded":"2024-07-26T07:48:19.995470+0000","last_fullsized":"2024-07-26T07:48:19.995470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:42.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911008+0000","last_change":"2024-07-26T07:47:41.784615+0000","last_active":"2024-07-26T07:48:19.911008+0000","last_peered":"2024-07-26T07:48:19.911008+0000","last_clean":"2024-07-26T07:48:19.911008+0000","last_became_active":"2024-07-26T07:47:41.783945+0000","last_became_peered":"2024-07-26T07:47:41.783945+0000","last_unstale":"2024-07-26T07:48:19.911008+0000","last_undegraded":"2024-07-26T07:48:19.911008+0000","last_fullsized":"2024-07-26T07:48:19.911008+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:42.618524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921175+0000","last_change":"2024-07-26T07:47:45.838774+0000","last_active":"2024-07-26T07:48:19.921175+0000","last_peered":"2024-07-26T07:48:19.921175+0000","last_clean":"2024-07-26T07:48:19.921175+0000","last_became_active":"2024-07-26T07:47:45.837707+0000","last_became_peered":"2024-07-26T07:47:45.837707+0000","last_unstale":"2024-07-26T07:48:19.921175+0000","last_undegraded":"2024-07-26T07:48:19.921175+0000","last_fullsized":"2024-07-26T07:48:19.921175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:59.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.913292+0000","last_change":"2024-07-26T07:47:41.792031+0000","last_active":"2024-07-26T07:48:19.913292+0000","last_peered":"2024-07-26T07:48:19.913292+0000","last_clean":"2024-07-26T07:48:19.913292+0000","last_became_active":"2024-07-26T07:47:41.785955+0000","last_became_peered":"2024-07-26T07:47:41.785955+0000","last_unstale":"2024-07-26T07:48:19.913292+0000","last_undegraded":"2024-07-26T07:48:19.913292+0000","last_fullsized":"2024-07-26T07:48:19.913292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:01:47.073970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922983+0000","last_change":"2024-07-26T07:47:45.844595+0000","last_active":"2024-07-26T07:48:19.922983+0000","last_peered":"2024-07-26T07:48:19.922983+0000","last_clean":"2024-07-26T07:48:19.922983+0000","last_became_active":"2024-07-26T07:47:45.843381+0000","last_became_peered":"2024-07-26T07:47:45.843381+0000","last_unstale":"2024-07-26T07:48:19.922983+0000","last_undegraded":"2024-07-26T07:48:19.922983+0000","last_fullsized":"2024-07-26T07:48:19.922983+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:39.692463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'4","reported_seq":91,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:36.931084+0000","last_change":"2024-07-26T07:47:41.788239+0000","last_active":"2024-07-26T07:48:36.931084+0000","last_peered":"2024-07-26T07:48:36.931084+0000","last_clean":"2024-07-26T07:48:36.931084+0000","last_became_active":"2024-07-26T07:47:41.787470+0000","last_became_peered":"2024-07-26T07:47:41.787470+0000","last_unstale":"2024-07-26T07:48:36.931084+0000","last_undegraded":"2024-07-26T07:48:36.931084+0000","last_fullsized":"2024-07-26T07:48:36.931084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:03.129563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":60,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.912774+0000","last_change":"2024-07-26T07:47:45.852794+0000","last_active":"2024-07-26T07:48:19.912774+0000","last_peered":"2024-07-26T07:48:19.912774+0000","last_clean":"2024-07-26T07:48:19.912774+0000","last_became_active":"2024-07-26T07:47:45.849699+0000","last_became_peered":"2024-07-26T07:47:45.849699+0000","last_unstale":"2024-07-26T07:48:19.912774+0000","last_undegraded":"2024-07-26T07:48:19.912774+0000","last_fullsized":"2024-07-26T07:48:19.912774+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:06:31.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933479+0000","last_change":"2024-07-26T07:47:41.789414+0000","last_active":"2024-07-26T07:48:19.933479+0000","last_peered":"2024-07-26T07:48:19.933479+0000","last_clean":"2024-07-26T07:48:19.933479+0000","last_became_active":"2024-07-26T07:47:41.788679+0000","last_became_peered":"2024-07-26T07:47:41.788679+0000","last_unstale":"2024-07-26T07:48:19.933479+0000","last_undegraded":"2024-07-26T07:48:19.933479+0000","last_fullsized":"2024-07-26T07:48:19.933479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:51.809997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.937784+0000","last_change":"2024-07-26T07:48:19.937603+0000","last_active":"2024-07-26T07:48:19.937784+0000","last_peered":"2024-07-26T07:48:19.937784+0000","last_clean":"2024-07-26T07:48:19.937784+0000","last_became_active":"2024-07-26T07:48:19.936889+0000","last_became_peered":"2024-07-26T07:48:19.936889+0000","last_unstale":"2024-07-26T07:48:19.937784+0000","last_undegraded":"2024-07-26T07:48:19.937784+0000","last_fullsized":"2024-07-26T07:48:19.937784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:46:35.138825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":33,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.914560+0000","last_change":"2024-07-26T07:47:42.564288+0000","last_active":"2024-07-26T07:48:19.914560+0000","last_peered":"2024-07-26T07:48:19.914560+0000","last_clean":"2024-07-26T07:48:19.914560+0000","last_became_active":"2024-07-26T07:47:42.561766+0000","last_became_peered":"2024-07-26T07:47:42.561766+0000","last_unstale":"2024-07-26T07:48:19.914560+0000","last_undegraded":"2024-07-26T07:48:19.914560+0000","last_fullsized":"2024-07-26T07:48:19.914560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:16.414030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.912074+0000","last_change":"2024-07-26T07:47:45.861895+0000","last_active":"2024-07-26T07:48:19.912074+0000","last_peered":"2024-07-26T07:48:19.912074+0000","last_clean":"2024-07-26T07:48:19.912074+0000","last_became_active":"2024-07-26T07:47:45.861439+0000","last_became_peered":"2024-07-26T07:47:45.861439+0000","last_unstale":"2024-07-26T07:48:19.912074+0000","last_undegraded":"2024-07-26T07:48:19.912074+0000","last_fullsized":"2024-07-26T07:48:19.912074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:26.276453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"84'30","reported_seq":125,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:36.953868+0000","last_change":"2024-07-26T07:47:41.789977+0000","last_active":"2024-07-26T07:48:36.953868+0000","last_peered":"2024-07-26T07:48:36.953868+0000","last_clean":"2024-07-26T07:48:36.953868+0000","last_became_active":"2024-07-26T07:47:41.789488+0000","last_became_peered":"2024-07-26T07:47:41.789488+0000","last_unstale":"2024-07-26T07:48:36.953868+0000","last_undegraded":"2024-07-26T07:48:36.953868+0000","last_fullsized":"2024-07-26T07:48:36.953868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:33.059279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55505,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":87,"num_write":32,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.925719+0000","last_change":"2024-07-26T07:47:45.854772+0000","last_active":"2024-07-26T07:48:19.925719+0000","last_peered":"2024-07-26T07:48:19.925719+0000","last_clean":"2024-07-26T07:48:19.925719+0000","last_became_active":"2024-07-26T07:47:45.854354+0000","last_became_peered":"2024-07-26T07:47:45.854354+0000","last_unstale":"2024-07-26T07:48:19.925719+0000","last_undegraded":"2024-07-26T07:48:19.925719+0000","last_fullsized":"2024-07-26T07:48:19.925719+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:31:54.856407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923492+0000","last_change":"2024-07-26T07:47:41.789752+0000","last_active":"2024-07-26T07:48:19.923492+0000","last_peered":"2024-07-26T07:48:19.923492+0000","last_clean":"2024-07-26T07:48:19.923492+0000","last_became_active":"2024-07-26T07:47:41.788952+0000","last_became_peered":"2024-07-26T07:47:41.788952+0000","last_unstale":"2024-07-26T07:48:19.923492+0000","last_undegraded":"2024-07-26T07:48:19.923492+0000","last_fullsized":"2024-07-26T07:48:19.923492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:45.141693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911113+0000","last_change":"2024-07-26T07:47:45.845120+0000","last_active":"2024-07-26T07:48:19.911113+0000","last_peered":"2024-07-26T07:48:19.911113+0000","last_clean":"2024-07-26T07:48:19.911113+0000","last_became_active":"2024-07-26T07:47:45.844699+0000","last_became_peered":"2024-07-26T07:47:45.844699+0000","last_unstale":"2024-07-26T07:48:19.911113+0000","last_undegraded":"2024-07-26T07:48:19.911113+0000","last_fullsized":"2024-07-26T07:48:19.911113+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:23:42.130846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":33,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.913543+0000","last_change":"2024-07-26T07:47:41.790765+0000","last_active":"2024-07-26T07:48:19.913543+0000","last_peered":"2024-07-26T07:48:19.913543+0000","last_clean":"2024-07-26T07:48:19.913543+0000","last_became_active":"2024-07-26T07:47:41.790288+0000","last_became_peered":"2024-07-26T07:47:41.790288+0000","last_unstale":"2024-07-26T07:48:19.913543+0000","last_undegraded":"2024-07-26T07:48:19.913543+0000","last_fullsized":"2024-07-26T07:48:19.913543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:05.734468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.932830+0000","last_change":"2024-07-26T07:47:45.854648+0000","last_active":"2024-07-26T07:48:19.932830+0000","last_peered":"2024-07-26T07:48:19.932830+0000","last_clean":"2024-07-26T07:48:19.932830+0000","last_became_active":"2024-07-26T07:47:45.854240+0000","last_became_peered":"2024-07-26T07:47:45.854240+0000","last_unstale":"2024-07-26T07:48:19.932830+0000","last_undegraded":"2024-07-26T07:48:19.932830+0000","last_fullsized":"2024-07-26T07:48:19.932830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:41.775760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:38.938127+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:48:38.938127+0000","last_peered":"2024-07-26T07:48:38.938127+0000","last_clean":"2024-07-26T07:48:38.938127+0000","last_became_active":"2024-07-26T07:47:41.781324+0000","last_became_peered":"2024-07-26T07:47:41.781324+0000","last_unstale":"2024-07-26T07:48:38.938127+0000","last_undegraded":"2024-07-26T07:48:38.938127+0000","last_fullsized":"2024-07-26T07:48:38.938127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:45.154354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.931890+0000","last_change":"2024-07-26T07:47:45.848731+0000","last_active":"2024-07-26T07:48:19.931890+0000","last_peered":"2024-07-26T07:48:19.931890+0000","last_clean":"2024-07-26T07:48:19.931890+0000","last_became_active":"2024-07-26T07:47:45.846663+0000","last_became_peered":"2024-07-26T07:47:45.846663+0000","last_unstale":"2024-07-26T07:48:19.931890+0000","last_undegraded":"2024-07-26T07:48:19.931890+0000","last_fullsized":"2024-07-26T07:48:19.931890+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:05.251423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922521+0000","last_change":"2024-07-26T07:47:42.561839+0000","last_active":"2024-07-26T07:48:19.922521+0000","last_peered":"2024-07-26T07:48:19.922521+0000","last_clean":"2024-07-26T07:48:19.922521+0000","last_became_active":"2024-07-26T07:47:42.560417+0000","last_became_peered":"2024-07-26T07:47:42.560417+0000","last_unstale":"2024-07-26T07:48:19.922521+0000","last_undegraded":"2024-07-26T07:48:19.922521+0000","last_fullsized":"2024-07-26T07:48:19.922521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:16.717573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.912844+0000","last_change":"2024-07-26T07:47:45.842572+0000","last_active":"2024-07-26T07:48:19.912844+0000","last_peered":"2024-07-26T07:48:19.912844+0000","last_clean":"2024-07-26T07:48:19.912844+0000","last_became_active":"2024-07-26T07:47:45.842116+0000","last_became_peered":"2024-07-26T07:47:45.842116+0000","last_unstale":"2024-07-26T07:48:19.912844+0000","last_undegraded":"2024-07-26T07:48:19.912844+0000","last_fullsized":"2024-07-26T07:48:19.912844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:06.913737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923293+0000","last_change":"2024-07-26T07:47:41.790215+0000","last_active":"2024-07-26T07:48:19.923293+0000","last_peered":"2024-07-26T07:48:19.923293+0000","last_clean":"2024-07-26T07:48:19.923293+0000","last_became_active":"2024-07-26T07:47:41.789691+0000","last_became_peered":"2024-07-26T07:47:41.789691+0000","last_unstale":"2024-07-26T07:48:19.923293+0000","last_undegraded":"2024-07-26T07:48:19.923293+0000","last_fullsized":"2024-07-26T07:48:19.923293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:01:47.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921097+0000","last_change":"2024-07-26T07:47:45.860887+0000","last_active":"2024-07-26T07:48:19.921097+0000","last_peered":"2024-07-26T07:48:19.921097+0000","last_clean":"2024-07-26T07:48:19.921097+0000","last_became_active":"2024-07-26T07:47:45.860315+0000","last_became_peered":"2024-07-26T07:47:45.860315+0000","last_unstale":"2024-07-26T07:48:19.921097+0000","last_undegraded":"2024-07-26T07:48:19.921097+0000","last_fullsized":"2024-07-26T07:48:19.921097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:17:02.231206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:20.023019+0000","last_change":"2024-07-26T07:48:20.022885+0000","last_active":"2024-07-26T07:48:20.023019+0000","last_peered":"2024-07-26T07:48:20.023019+0000","last_clean":"2024-07-26T07:48:20.023019+0000","last_became_active":"2024-07-26T07:48:20.001946+0000","last_became_peered":"2024-07-26T07:48:20.001946+0000","last_unstale":"2024-07-26T07:48:20.023019+0000","last_undegraded":"2024-07-26T07:48:20.023019+0000","last_fullsized":"2024-07-26T07:48:20.023019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:45.892886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.934773+0000","last_change":"2024-07-26T07:47:45.858267+0000","last_active":"2024-07-26T07:48:19.934773+0000","last_peered":"2024-07-26T07:48:19.934773+0000","last_clean":"2024-07-26T07:48:19.934773+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:48:19.934773+0000","last_undegraded":"2024-07-26T07:48:19.934773+0000","last_fullsized":"2024-07-26T07:48:19.934773+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:32.055977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.914861+0000","last_change":"2024-07-26T07:47:41.789846+0000","last_active":"2024-07-26T07:48:19.914861+0000","last_peered":"2024-07-26T07:48:19.914861+0000","last_clean":"2024-07-26T07:48:19.914861+0000","last_became_active":"2024-07-26T07:47:41.789237+0000","last_became_peered":"2024-07-26T07:47:41.789237+0000","last_unstale":"2024-07-26T07:48:19.914861+0000","last_undegraded":"2024-07-26T07:48:19.914861+0000","last_fullsized":"2024-07-26T07:48:19.914861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:39.031047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"84'1","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.932650+0000","last_change":"2024-07-26T07:47:45.846294+0000","last_active":"2024-07-26T07:48:19.932650+0000","last_peered":"2024-07-26T07:48:19.932650+0000","last_clean":"2024-07-26T07:48:19.932650+0000","last_became_active":"2024-07-26T07:47:45.845861+0000","last_became_peered":"2024-07-26T07:47:45.845861+0000","last_unstale":"2024-07-26T07:48:19.932650+0000","last_undegraded":"2024-07-26T07:48:19.932650+0000","last_fullsized":"2024-07-26T07:48:19.932650+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:56:10.594430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'6","reported_seq":64,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:33.922072+0000","last_change":"2024-07-26T07:47:42.563412+0000","last_active":"2024-07-26T07:48:33.922072+0000","last_peered":"2024-07-26T07:48:33.922072+0000","last_clean":"2024-07-26T07:48:33.922072+0000","last_became_active":"2024-07-26T07:47:42.562364+0000","last_became_peered":"2024-07-26T07:47:42.562364+0000","last_unstale":"2024-07-26T07:48:33.922072+0000","last_undegraded":"2024-07-26T07:48:33.922072+0000","last_fullsized":"2024-07-26T07:48:33.922072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:14.796738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933936+0000","last_change":"2024-07-26T07:47:45.856170+0000","last_active":"2024-07-26T07:48:19.933936+0000","last_peered":"2024-07-26T07:48:19.933936+0000","last_clean":"2024-07-26T07:48:19.933936+0000","last_became_active":"2024-07-26T07:47:45.855782+0000","last_became_peered":"2024-07-26T07:47:45.855782+0000","last_unstale":"2024-07-26T07:48:19.933936+0000","last_undegraded":"2024-07-26T07:48:19.933936+0000","last_fullsized":"2024-07-26T07:48:19.933936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:22.690910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":106,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933763+0000","last_change":"2024-07-26T07:45:54.310557+0000","last_active":"2024-07-26T07:48:19.933763+0000","last_peered":"2024-07-26T07:48:19.933763+0000","last_clean":"2024-07-26T07:48:19.933763+0000","last_became_active":"2024-07-26T07:45:54.290906+0000","last_became_peered":"2024-07-26T07:45:54.290906+0000","last_unstale":"2024-07-26T07:48:19.933763+0000","last_undegraded":"2024-07-26T07:48:19.933763+0000","last_fullsized":"2024-07-26T07:48:19.933763+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_clean_scrub_stamp":"2024-07-26T07:41:16.689463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:51.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.999798+0000","last_change":"2024-07-26T07:47:41.776420+0000","last_active":"2024-07-26T07:48:19.999798+0000","last_peered":"2024-07-26T07:48:19.999798+0000","last_clean":"2024-07-26T07:48:19.999798+0000","last_became_active":"2024-07-26T07:47:41.775858+0000","last_became_peered":"2024-07-26T07:47:41.775858+0000","last_unstale":"2024-07-26T07:48:19.999798+0000","last_undegraded":"2024-07-26T07:48:19.999798+0000","last_fullsized":"2024-07-26T07:48:19.999798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:18.880331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.935467+0000","last_change":"2024-07-26T07:48:19.933318+0000","last_active":"2024-07-26T07:48:19.935467+0000","last_peered":"2024-07-26T07:48:19.935467+0000","last_clean":"2024-07-26T07:48:19.935467+0000","last_became_active":"2024-07-26T07:48:19.932683+0000","last_became_peered":"2024-07-26T07:48:19.932683+0000","last_unstale":"2024-07-26T07:48:19.935467+0000","last_undegraded":"2024-07-26T07:48:19.935467+0000","last_fullsized":"2024-07-26T07:48:19.935467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:25:23.387428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933375+0000","last_change":"2024-07-26T07:47:41.786698+0000","last_active":"2024-07-26T07:48:19.933375+0000","last_peered":"2024-07-26T07:48:19.933375+0000","last_clean":"2024-07-26T07:48:19.933375+0000","last_became_active":"2024-07-26T07:47:41.784491+0000","last_became_peered":"2024-07-26T07:47:41.784491+0000","last_unstale":"2024-07-26T07:48:19.933375+0000","last_undegraded":"2024-07-26T07:48:19.933375+0000","last_fullsized":"2024-07-26T07:48:19.933375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:09:08.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.910962+0000","last_change":"2024-07-26T07:47:45.849698+0000","last_active":"2024-07-26T07:48:19.910962+0000","last_peered":"2024-07-26T07:48:19.910962+0000","last_clean":"2024-07-26T07:48:19.910962+0000","last_became_active":"2024-07-26T07:47:45.849266+0000","last_became_peered":"2024-07-26T07:47:45.849266+0000","last_unstale":"2024-07-26T07:48:19.910962+0000","last_undegraded":"2024-07-26T07:48:19.910962+0000","last_fullsized":"2024-07-26T07:48:19.910962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:57.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:37.933322+0000","last_change":"2024-07-26T07:47:42.558680+0000","last_active":"2024-07-26T07:48:37.933322+0000","last_peered":"2024-07-26T07:48:37.933322+0000","last_clean":"2024-07-26T07:48:37.933322+0000","last_became_active":"2024-07-26T07:47:42.558197+0000","last_became_peered":"2024-07-26T07:47:42.558197+0000","last_unstale":"2024-07-26T07:48:37.933322+0000","last_undegraded":"2024-07-26T07:48:37.933322+0000","last_fullsized":"2024-07-26T07:48:37.933322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:52:52.505455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.930090+0000","last_change":"2024-07-26T07:48:19.929897+0000","last_active":"2024-07-26T07:48:19.930090+0000","last_peered":"2024-07-26T07:48:19.930090+0000","last_clean":"2024-07-26T07:48:19.930090+0000","last_became_active":"2024-07-26T07:48:19.927946+0000","last_became_peered":"2024-07-26T07:48:19.927946+0000","last_unstale":"2024-07-26T07:48:19.930090+0000","last_undegraded":"2024-07-26T07:48:19.930090+0000","last_fullsized":"2024-07-26T07:48:19.930090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:11.157824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"82'5","reported_seq":64,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:36.988278+0000","last_change":"2024-07-26T07:47:41.785513+0000","last_active":"2024-07-26T07:48:36.988278+0000","last_peered":"2024-07-26T07:48:36.988278+0000","last_clean":"2024-07-26T07:48:36.988278+0000","last_became_active":"2024-07-26T07:47:41.784783+0000","last_became_peered":"2024-07-26T07:47:41.784783+0000","last_unstale":"2024-07-26T07:48:36.988278+0000","last_undegraded":"2024-07-26T07:48:36.988278+0000","last_fullsized":"2024-07-26T07:48:36.988278+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:54.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.995470+0000","last_change":"2024-07-26T07:47:45.856580+0000","last_active":"2024-07-26T07:48:19.995470+0000","last_peered":"2024-07-26T07:48:19.995470+0000","last_clean":"2024-07-26T07:48:19.995470+0000","last_became_active":"2024-07-26T07:47:45.852330+0000","last_became_peered":"2024-07-26T07:47:45.852330+0000","last_unstale":"2024-07-26T07:48:19.995470+0000","last_undegraded":"2024-07-26T07:48:19.995470+0000","last_fullsized":"2024-07-26T07:48:19.995470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:09.479563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924407+0000","last_change":"2024-07-26T07:47:42.559255+0000","last_active":"2024-07-26T07:48:19.924407+0000","last_peered":"2024-07-26T07:48:19.924407+0000","last_clean":"2024-07-26T07:48:19.924407+0000","last_became_active":"2024-07-26T07:47:42.558123+0000","last_became_peered":"2024-07-26T07:47:42.558123+0000","last_unstale":"2024-07-26T07:48:19.924407+0000","last_undegraded":"2024-07-26T07:48:19.924407+0000","last_fullsized":"2024-07-26T07:48:19.924407+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:49:46.576674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'1","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.995941+0000","last_change":"2024-07-26T07:47:45.856256+0000","last_active":"2024-07-26T07:48:19.995941+0000","last_peered":"2024-07-26T07:48:19.995941+0000","last_clean":"2024-07-26T07:48:19.995941+0000","last_became_active":"2024-07-26T07:47:45.855696+0000","last_became_peered":"2024-07-26T07:47:45.855696+0000","last_unstale":"2024-07-26T07:48:19.995941+0000","last_undegraded":"2024-07-26T07:48:19.995941+0000","last_fullsized":"2024-07-26T07:48:19.995941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:24.451225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":32,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.925947+0000","last_change":"2024-07-26T07:47:42.562631+0000","last_active":"2024-07-26T07:48:19.925947+0000","last_peered":"2024-07-26T07:48:19.925947+0000","last_clean":"2024-07-26T07:48:19.925947+0000","last_became_active":"2024-07-26T07:47:42.561379+0000","last_became_peered":"2024-07-26T07:47:42.561379+0000","last_unstale":"2024-07-26T07:48:19.925947+0000","last_undegraded":"2024-07-26T07:48:19.925947+0000","last_fullsized":"2024-07-26T07:48:19.925947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:03:30.471694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.912133+0000","last_change":"2024-07-26T07:47:45.849580+0000","last_active":"2024-07-26T07:48:19.912133+0000","last_peered":"2024-07-26T07:48:19.912133+0000","last_clean":"2024-07-26T07:48:19.912133+0000","last_became_active":"2024-07-26T07:47:45.849107+0000","last_became_peered":"2024-07-26T07:47:45.849107+0000","last_unstale":"2024-07-26T07:48:19.912133+0000","last_undegraded":"2024-07-26T07:48:19.912133+0000","last_fullsized":"2024-07-26T07:48:19.912133+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:57:13.694059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"84'15","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:32.866485+0000","last_change":"2024-07-26T07:47:41.774083+0000","last_active":"2024-07-26T07:48:32.866485+0000","last_peered":"2024-07-26T07:48:32.866485+0000","last_clean":"2024-07-26T07:48:32.866485+0000","last_became_active":"2024-07-26T07:47:41.773502+0000","last_became_peered":"2024-07-26T07:47:41.773502+0000","last_unstale":"2024-07-26T07:48:32.866485+0000","last_undegraded":"2024-07-26T07:48:32.866485+0000","last_fullsized":"2024-07-26T07:48:32.866485+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:11:44.083944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923886+0000","last_change":"2024-07-26T07:47:45.845894+0000","last_active":"2024-07-26T07:48:19.923886+0000","last_peered":"2024-07-26T07:48:19.923886+0000","last_clean":"2024-07-26T07:48:19.923886+0000","last_became_active":"2024-07-26T07:47:45.845534+0000","last_became_peered":"2024-07-26T07:47:45.845534+0000","last_unstale":"2024-07-26T07:48:19.923886+0000","last_undegraded":"2024-07-26T07:48:19.923886+0000","last_fullsized":"2024-07-26T07:48:19.923886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:52:42.209561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.999884+0000","last_change":"2024-07-26T07:47:41.904220+0000","last_active":"2024-07-26T07:48:19.999884+0000","last_peered":"2024-07-26T07:48:19.999884+0000","last_clean":"2024-07-26T07:48:19.999884+0000","last_became_active":"2024-07-26T07:47:41.902349+0000","last_became_peered":"2024-07-26T07:47:41.902349+0000","last_unstale":"2024-07-26T07:48:19.999884+0000","last_undegraded":"2024-07-26T07:48:19.999884+0000","last_fullsized":"2024-07-26T07:48:19.999884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:08.520461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923525+0000","last_change":"2024-07-26T07:47:45.846920+0000","last_active":"2024-07-26T07:48:19.923525+0000","last_peered":"2024-07-26T07:48:19.923525+0000","last_clean":"2024-07-26T07:48:19.923525+0000","last_became_active":"2024-07-26T07:47:45.846458+0000","last_became_peered":"2024-07-26T07:47:45.846458+0000","last_unstale":"2024-07-26T07:48:19.923525+0000","last_undegraded":"2024-07-26T07:48:19.923525+0000","last_fullsized":"2024-07-26T07:48:19.923525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:06.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911839+0000","last_change":"2024-07-26T07:47:42.560346+0000","last_active":"2024-07-26T07:48:19.911839+0000","last_peered":"2024-07-26T07:48:19.911839+0000","last_clean":"2024-07-26T07:48:19.911839+0000","last_became_active":"2024-07-26T07:47:42.558937+0000","last_became_peered":"2024-07-26T07:47:42.558937+0000","last_unstale":"2024-07-26T07:48:19.911839+0000","last_undegraded":"2024-07-26T07:48:19.911839+0000","last_fullsized":"2024-07-26T07:48:19.911839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.676625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923945+0000","last_change":"2024-07-26T07:47:45.845553+0000","last_active":"2024-07-26T07:48:19.923945+0000","last_peered":"2024-07-26T07:48:19.923945+0000","last_clean":"2024-07-26T07:48:19.923945+0000","last_became_active":"2024-07-26T07:47:45.845125+0000","last_became_peered":"2024-07-26T07:47:45.845125+0000","last_unstale":"2024-07-26T07:48:19.923945+0000","last_undegraded":"2024-07-26T07:48:19.923945+0000","last_fullsized":"2024-07-26T07:48:19.923945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:10.896159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922224+0000","last_change":"2024-07-26T07:47:41.779688+0000","last_active":"2024-07-26T07:48:19.922224+0000","last_peered":"2024-07-26T07:48:19.922224+0000","last_clean":"2024-07-26T07:48:19.922224+0000","last_became_active":"2024-07-26T07:47:41.779115+0000","last_became_peered":"2024-07-26T07:47:41.779115+0000","last_unstale":"2024-07-26T07:48:19.922224+0000","last_undegraded":"2024-07-26T07:48:19.922224+0000","last_fullsized":"2024-07-26T07:48:19.922224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:20.708336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920383+0000","last_change":"2024-07-26T07:47:45.853869+0000","last_active":"2024-07-26T07:48:19.920383+0000","last_peered":"2024-07-26T07:48:19.920383+0000","last_clean":"2024-07-26T07:48:19.920383+0000","last_became_active":"2024-07-26T07:47:45.852509+0000","last_became_peered":"2024-07-26T07:47:45.852509+0000","last_unstale":"2024-07-26T07:48:19.920383+0000","last_undegraded":"2024-07-26T07:48:19.920383+0000","last_fullsized":"2024-07-26T07:48:19.920383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:01:31.764025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920208+0000","last_change":"2024-07-26T07:47:41.792331+0000","last_active":"2024-07-26T07:48:19.920208+0000","last_peered":"2024-07-26T07:48:19.920208+0000","last_clean":"2024-07-26T07:48:19.920208+0000","last_became_active":"2024-07-26T07:47:41.790668+0000","last_became_peered":"2024-07-26T07:47:41.790668+0000","last_unstale":"2024-07-26T07:48:19.920208+0000","last_undegraded":"2024-07-26T07:48:19.920208+0000","last_fullsized":"2024-07-26T07:48:19.920208+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:59.430692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922438+0000","last_change":"2024-07-26T07:47:45.843583+0000","last_active":"2024-07-26T07:48:19.922438+0000","last_peered":"2024-07-26T07:48:19.922438+0000","last_clean":"2024-07-26T07:48:19.922438+0000","last_became_active":"2024-07-26T07:47:45.843185+0000","last_became_peered":"2024-07-26T07:47:45.843185+0000","last_unstale":"2024-07-26T07:48:19.922438+0000","last_undegraded":"2024-07-26T07:48:19.922438+0000","last_fullsized":"2024-07-26T07:48:19.922438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:01.119065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:20.014568+0000","last_change":"2024-07-26T07:48:20.014411+0000","last_active":"2024-07-26T07:48:20.014568+0000","last_peered":"2024-07-26T07:48:20.014568+0000","last_clean":"2024-07-26T07:48:20.014568+0000","last_became_active":"2024-07-26T07:48:20.002085+0000","last_became_peered":"2024-07-26T07:48:20.002085+0000","last_unstale":"2024-07-26T07:48:20.014568+0000","last_undegraded":"2024-07-26T07:48:20.014568+0000","last_fullsized":"2024-07-26T07:48:20.014568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:46:59.335128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933049+0000","last_change":"2024-07-26T07:47:45.851963+0000","last_active":"2024-07-26T07:48:19.933049+0000","last_peered":"2024-07-26T07:48:19.933049+0000","last_clean":"2024-07-26T07:48:19.933049+0000","last_became_active":"2024-07-26T07:47:45.850459+0000","last_became_peered":"2024-07-26T07:47:45.850459+0000","last_unstale":"2024-07-26T07:48:19.933049+0000","last_undegraded":"2024-07-26T07:48:19.933049+0000","last_fullsized":"2024-07-26T07:48:19.933049+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:18.092186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.914232+0000","last_change":"2024-07-26T07:47:45.848744+0000","last_active":"2024-07-26T07:48:19.914232+0000","last_peered":"2024-07-26T07:48:19.914232+0000","last_clean":"2024-07-26T07:48:19.914232+0000","last_became_active":"2024-07-26T07:47:45.848292+0000","last_became_peered":"2024-07-26T07:47:45.848292+0000","last_unstale":"2024-07-26T07:48:19.914232+0000","last_undegraded":"2024-07-26T07:48:19.914232+0000","last_fullsized":"2024-07-26T07:48:19.914232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:54.549508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":33,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923455+0000","last_change":"2024-07-26T07:47:41.788875+0000","last_active":"2024-07-26T07:48:19.923455+0000","last_peered":"2024-07-26T07:48:19.923455+0000","last_clean":"2024-07-26T07:48:19.923455+0000","last_became_active":"2024-07-26T07:47:41.786942+0000","last_became_peered":"2024-07-26T07:47:41.786942+0000","last_unstale":"2024-07-26T07:48:19.923455+0000","last_undegraded":"2024-07-26T07:48:19.923455+0000","last_fullsized":"2024-07-26T07:48:19.923455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:11:31.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911805+0000","last_change":"2024-07-26T07:47:45.854832+0000","last_active":"2024-07-26T07:48:19.911805+0000","last_peered":"2024-07-26T07:48:19.911805+0000","last_clean":"2024-07-26T07:48:19.911805+0000","last_became_active":"2024-07-26T07:47:45.853756+0000","last_became_peered":"2024-07-26T07:47:45.853756+0000","last_unstale":"2024-07-26T07:48:19.911805+0000","last_undegraded":"2024-07-26T07:48:19.911805+0000","last_fullsized":"2024-07-26T07:48:19.911805+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:42:14.579177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.925948+0000","last_change":"2024-07-26T07:47:41.787839+0000","last_active":"2024-07-26T07:48:19.925948+0000","last_peered":"2024-07-26T07:48:19.925948+0000","last_clean":"2024-07-26T07:48:19.925948+0000","last_became_active":"2024-07-26T07:47:41.785951+0000","last_became_peered":"2024-07-26T07:47:41.785951+0000","last_unstale":"2024-07-26T07:48:19.925948+0000","last_undegraded":"2024-07-26T07:48:19.925948+0000","last_fullsized":"2024-07-26T07:48:19.925948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:58.699056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.918445+0000","last_change":"2024-07-26T07:47:45.841451+0000","last_active":"2024-07-26T07:48:19.918445+0000","last_peered":"2024-07-26T07:48:19.918445+0000","last_clean":"2024-07-26T07:48:19.918445+0000","last_became_active":"2024-07-26T07:47:45.840803+0000","last_became_peered":"2024-07-26T07:47:45.840803+0000","last_unstale":"2024-07-26T07:48:19.918445+0000","last_undegraded":"2024-07-26T07:48:19.918445+0000","last_fullsized":"2024-07-26T07:48:19.918445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:24:21.832317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924186+0000","last_change":"2024-07-26T07:47:42.558775+0000","last_active":"2024-07-26T07:48:19.924186+0000","last_peered":"2024-07-26T07:48:19.924186+0000","last_clean":"2024-07-26T07:48:19.924186+0000","last_became_active":"2024-07-26T07:47:42.558300+0000","last_became_peered":"2024-07-26T07:47:42.558300+0000","last_unstale":"2024-07-26T07:48:19.924186+0000","last_undegraded":"2024-07-26T07:48:19.924186+0000","last_fullsized":"2024-07-26T07:48:19.924186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:59.643709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923531+0000","last_change":"2024-07-26T07:47:45.852401+0000","last_active":"2024-07-26T07:48:19.923531+0000","last_peered":"2024-07-26T07:48:19.923531+0000","last_clean":"2024-07-26T07:48:19.923531+0000","last_became_active":"2024-07-26T07:47:45.852054+0000","last_became_peered":"2024-07-26T07:47:45.852054+0000","last_unstale":"2024-07-26T07:48:19.923531+0000","last_undegraded":"2024-07-26T07:48:19.923531+0000","last_fullsized":"2024-07-26T07:48:19.923531+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:25.347770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"82'13","reported_seq":93,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:36.986864+0000","last_change":"2024-07-26T07:48:19.961760+0000","last_active":"2024-07-26T07:48:36.986864+0000","last_peered":"2024-07-26T07:48:36.986864+0000","last_clean":"2024-07-26T07:48:36.986864+0000","last_became_active":"2024-07-26T07:48:19.937694+0000","last_became_peered":"2024-07-26T07:48:19.937694+0000","last_unstale":"2024-07-26T07:48:36.986864+0000","last_undegraded":"2024-07-26T07:48:36.986864+0000","last_fullsized":"2024-07-26T07:48:36.986864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:05:10.571086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62318,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":336,"num_read_kb":235,"num_write":243,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":191211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579934,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":925696,"data_stored":446295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776164,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1092438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972395,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":445834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168627,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":445540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397560,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1038339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561087,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36256,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708896,"statfs":{"total":95995035648,"available":95957909504,"internally_reserved":0,"allocated":933888,"data_stored":445551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36856,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708296,"statfs":{"total":95995035648,"available":95957295104,"internally_reserved":0,"allocated":1556480,"data_stored":1038017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":445412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":445376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":502580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":506898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":954368,"data_stored":447281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":57241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T07:48:42.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T07:48:43.260 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:43 smithi097 ceph-mon[32231]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:48:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:43 smithi115 ceph-mon[35364]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:48:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:43 smithi181 ceph-mon[35869]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:48:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:44.318 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:44 smithi097 ceph-mon[32231]: pgmap v416: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T07:48:44.318 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:44.318 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:44 smithi115 ceph-mon[35364]: pgmap v416: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T07:48:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:44 smithi181 ceph-mon[35869]: pgmap v416: 97 pgs: 97 active+clean; 638 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T07:48:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:45.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:48:45.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:48:45.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:48:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:48:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 -- 172.21.15.97:0/1409155814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d981055d0 msgr2=0x7f8d981059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/1409155814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d981055d0 0x7f8d981059e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009900 tx=0x7f8d8802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 -- 172.21.15.97:0/1409155814 shutdown_connections 2024-07-26T07:48:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/1409155814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d9810b860 0x7f8d98112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/1409155814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d9810af10 0x7f8d9810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/1409155814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d981055d0 0x7f8d981059e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 -- 172.21.15.97:0/1409155814 >> 172.21.15.97:0/1409155814 conn(0x7f8d98101040 msgr2=0x7f8d98103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 -- 172.21.15.97:0/1409155814 shutdown_connections 2024-07-26T07:48:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.476+0000 7f8d9ca00640 1 -- 172.21.15.97:0/1409155814 wait complete. 2024-07-26T07:48:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.477+0000 7f8d9ca00640 1 Processor -- start 2024-07-26T07:48:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.477+0000 7f8d9ca00640 1 -- start start 2024-07-26T07:48:45.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d9ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d981055d0 0x7f8d9819be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:45.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d9ca00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d9810af10 0x7f8d9819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:45.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d9ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d9810b860 0x7f8d981a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:45.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d9ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8d981153c0 con 0x7f8d9810b860 2024-07-26T07:48:45.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d9ca00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8d98115240 con 0x7f8d9810af10 2024-07-26T07:48:45.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d9ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8d98115540 con 0x7f8d981055d0 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d97200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d9810b860 0x7f8d981a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d97200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d9810b860 0x7f8d981a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39926/0 (socket says 172.21.15.97:39926) 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.478+0000 7f8d97200640 1 -- 172.21.15.97:0/3541920479 learned_addr learned my addr 172.21.15.97:0/3541920479 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.479+0000 7f8d96800640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d981055d0 0x7f8d9819be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.479+0000 7f8d8de00640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d9810af10 0x7f8d9819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.479+0000 7f8d97200640 1 -- 172.21.15.97:0/3541920479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d981055d0 msgr2=0x7f8d9819be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.479+0000 7f8d97200640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d981055d0 0x7f8d9819be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.479+0000 7f8d97200640 1 -- 172.21.15.97:0/3541920479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d9810af10 msgr2=0x7f8d9819c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.479+0000 7f8d97200640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d9810af10 0x7f8d9819c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.479+0000 7f8d97200640 1 -- 172.21.15.97:0/3541920479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d981a1d20 con 0x7f8d9810b860 2024-07-26T07:48:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.480+0000 7f8d97200640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d9810b860 0x7f8d981a16b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000cad0 tx=0x7f8d9000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.480+0000 7f8d8fe00640 1 -- 172.21.15.97:0/3541920479 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9000ad60 con 0x7f8d9810b860 2024-07-26T07:48:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.480+0000 7f8d8fe00640 1 -- 172.21.15.97:0/3541920479 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8d90005020 con 0x7f8d9810b860 2024-07-26T07:48:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.480+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d981a2010 con 0x7f8d9810b860 2024-07-26T07:48:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.480+0000 7f8d8fe00640 1 -- 172.21.15.97:0/3541920479 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9000f6b0 con 0x7f8d9810b860 2024-07-26T07:48:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.480+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d981a8830 con 0x7f8d9810b860 2024-07-26T07:48:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:48:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:48:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:48:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.481+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d981059e0 con 0x7f8d9810b860 2024-07-26T07:48:45.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.486+0000 7f8d8fe00640 1 -- 172.21.15.97:0/3541920479 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f8d900051d0 con 0x7f8d9810b860 2024-07-26T07:48:45.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.487+0000 7f8d8fe00640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8d7c0b3d10 0x7f8d7c0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:45.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.487+0000 7f8d8fe00640 1 -- 172.21.15.97:0/3541920479 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f8d900e3ff0 con 0x7f8d9810b860 2024-07-26T07:48:45.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.488+0000 7f8d96800640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8d7c0b3d10 0x7f8d7c0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:45.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.488+0000 7f8d96800640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8d7c0b3d10 0x7f8d7c0b6220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8d880057c0 tx=0x7f8d880058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:45.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.490+0000 7f8d8fe00640 1 -- 172.21.15.97:0/3541920479 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8d90014030 con 0x7f8d9810b860 2024-07-26T07:48:45.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.680+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8d98063230 con 0x7f8d7c0b3d10 2024-07-26T07:48:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.687+0000 7f8d8fe00640 1 -- 172.21.15.97:0/3541920479 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258799 (secure 0 0 0) 0x7f8d98063230 con 0x7f8d7c0b3d10 2024-07-26T07:48:45.689 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:48:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-07-26T07:48:45.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8d7c0b3d10 msgr2=0x7f8d7c0b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:45.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8d7c0b3d10 0x7f8d7c0b6220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8d880057c0 tx=0x7f8d880058d0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d9810b860 msgr2=0x7f8d981a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:45.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d9810b860 0x7f8d981a16b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000cad0 tx=0x7f8d9000ced0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 shutdown_connections 2024-07-26T07:48:45.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8d7c0b3d10 0x7f8d7c0b6220 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d9810b860 0x7f8d981a16b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d9810af10 0x7f8d9819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 --2- 172.21.15.97:0/3541920479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d981055d0 0x7f8d9819be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:45.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.694+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 >> 172.21.15.97:0/3541920479 conn(0x7f8d98101040 msgr2=0x7f8d98103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:45.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.695+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 shutdown_connections 2024-07-26T07:48:45.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:45.695+0000 7f8d9ca00640 1 -- 172.21.15.97:0/3541920479 wait complete. 2024-07-26T07:48:46.353 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:46 smithi097 ceph-mon[32231]: pgmap v417: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T07:48:46.408 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-07-26T07:48:44.138725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657177,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":453,"num_read_kb":324,"num_write":369,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2952512,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403800,"kb_used_data":13020,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538024,"statfs":{"total":1151940427776,"available":1151526936576,"internally_reserved":0,"allocated":13332480,"data_stored":7308075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17561,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":89,"num_write":14,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006123"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921680+0000","last_change":"2024-07-26T07:47:45.843670+0000","last_active":"2024-07-26T07:48:19.921680+0000","last_peered":"2024-07-26T07:48:19.921680+0000","last_clean":"2024-07-26T07:48:19.921680+0000","last_became_active":"2024-07-26T07:47:45.841137+0000","last_became_peered":"2024-07-26T07:47:45.841137+0000","last_unstale":"2024-07-26T07:48:19.921680+0000","last_undegraded":"2024-07-26T07:48:19.921680+0000","last_fullsized":"2024-07-26T07:48:19.921680+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:38.063503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923347+0000","last_change":"2024-07-26T07:47:45.849326+0000","last_active":"2024-07-26T07:48:19.923347+0000","last_peered":"2024-07-26T07:48:19.923347+0000","last_clean":"2024-07-26T07:48:19.923347+0000","last_became_active":"2024-07-26T07:47:45.847761+0000","last_became_peered":"2024-07-26T07:47:45.847761+0000","last_unstale":"2024-07-26T07:48:19.923347+0000","last_undegraded":"2024-07-26T07:48:19.923347+0000","last_fullsized":"2024-07-26T07:48:19.923347+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:31.098794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.919179+0000","last_change":"2024-07-26T07:47:45.852343+0000","last_active":"2024-07-26T07:48:19.919179+0000","last_peered":"2024-07-26T07:48:19.919179+0000","last_clean":"2024-07-26T07:48:19.919179+0000","last_became_active":"2024-07-26T07:47:45.851950+0000","last_became_peered":"2024-07-26T07:47:45.851950+0000","last_unstale":"2024-07-26T07:48:19.919179+0000","last_undegraded":"2024-07-26T07:48:19.919179+0000","last_fullsized":"2024-07-26T07:48:19.919179+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:41.030903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.999037+0000","last_change":"2024-07-26T07:47:45.857042+0000","last_active":"2024-07-26T07:48:19.999037+0000","last_peered":"2024-07-26T07:48:19.999037+0000","last_clean":"2024-07-26T07:48:19.999037+0000","last_became_active":"2024-07-26T07:47:45.855145+0000","last_became_peered":"2024-07-26T07:47:45.855145+0000","last_unstale":"2024-07-26T07:48:19.999037+0000","last_undegraded":"2024-07-26T07:48:19.999037+0000","last_fullsized":"2024-07-26T07:48:19.999037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:32.233019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924168+0000","last_change":"2024-07-26T07:47:45.838560+0000","last_active":"2024-07-26T07:48:19.924168+0000","last_peered":"2024-07-26T07:48:19.924168+0000","last_clean":"2024-07-26T07:48:19.924168+0000","last_became_active":"2024-07-26T07:47:45.835578+0000","last_became_peered":"2024-07-26T07:47:45.835578+0000","last_unstale":"2024-07-26T07:48:19.924168+0000","last_undegraded":"2024-07-26T07:48:19.924168+0000","last_fullsized":"2024-07-26T07:48:19.924168+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:09:40.445625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.995589+0000","last_change":"2024-07-26T07:47:45.843980+0000","last_active":"2024-07-26T07:48:19.995589+0000","last_peered":"2024-07-26T07:48:19.995589+0000","last_clean":"2024-07-26T07:48:19.995589+0000","last_became_active":"2024-07-26T07:47:45.843546+0000","last_became_peered":"2024-07-26T07:47:45.843546+0000","last_unstale":"2024-07-26T07:48:19.995589+0000","last_undegraded":"2024-07-26T07:48:19.995589+0000","last_fullsized":"2024-07-26T07:48:19.995589+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:06.839136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.932049+0000","last_change":"2024-07-26T07:47:45.854427+0000","last_active":"2024-07-26T07:48:19.932049+0000","last_peered":"2024-07-26T07:48:19.932049+0000","last_clean":"2024-07-26T07:48:19.932049+0000","last_became_active":"2024-07-26T07:47:45.853495+0000","last_became_peered":"2024-07-26T07:47:45.853495+0000","last_unstale":"2024-07-26T07:48:19.932049+0000","last_undegraded":"2024-07-26T07:48:19.932049+0000","last_fullsized":"2024-07-26T07:48:19.932049+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:34:34.110581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920807+0000","last_change":"2024-07-26T07:47:45.847732+0000","last_active":"2024-07-26T07:48:19.920807+0000","last_peered":"2024-07-26T07:48:19.920807+0000","last_clean":"2024-07-26T07:48:19.920807+0000","last_became_active":"2024-07-26T07:47:45.845583+0000","last_became_peered":"2024-07-26T07:47:45.845583+0000","last_unstale":"2024-07-26T07:48:19.920807+0000","last_undegraded":"2024-07-26T07:48:19.920807+0000","last_fullsized":"2024-07-26T07:48:19.920807+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:07:09.370076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.913159+0000","last_change":"2024-07-26T07:47:45.849962+0000","last_active":"2024-07-26T07:48:19.913159+0000","last_peered":"2024-07-26T07:48:19.913159+0000","last_clean":"2024-07-26T07:48:19.913159+0000","last_became_active":"2024-07-26T07:47:45.845766+0000","last_became_peered":"2024-07-26T07:47:45.845766+0000","last_unstale":"2024-07-26T07:48:19.913159+0000","last_undegraded":"2024-07-26T07:48:19.913159+0000","last_fullsized":"2024-07-26T07:48:19.913159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:56.602825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.996633+0000","last_change":"2024-07-26T07:47:45.857287+0000","last_active":"2024-07-26T07:48:19.996633+0000","last_peered":"2024-07-26T07:48:19.996633+0000","last_clean":"2024-07-26T07:48:19.996633+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:48:19.996633+0000","last_undegraded":"2024-07-26T07:48:19.996633+0000","last_fullsized":"2024-07-26T07:48:19.996633+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:38:24.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933958+0000","last_change":"2024-07-26T07:47:45.858369+0000","last_active":"2024-07-26T07:48:19.933958+0000","last_peered":"2024-07-26T07:48:19.933958+0000","last_clean":"2024-07-26T07:48:19.933958+0000","last_became_active":"2024-07-26T07:47:45.856685+0000","last_became_peered":"2024-07-26T07:47:45.856685+0000","last_unstale":"2024-07-26T07:48:19.933958+0000","last_undegraded":"2024-07-26T07:48:19.933958+0000","last_fullsized":"2024-07-26T07:48:19.933958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:44.877953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:20.002675+0000","last_change":"2024-07-26T07:48:20.001394+0000","last_active":"2024-07-26T07:48:20.002675+0000","last_peered":"2024-07-26T07:48:20.002675+0000","last_clean":"2024-07-26T07:48:20.002675+0000","last_became_active":"2024-07-26T07:48:20.000751+0000","last_became_peered":"2024-07-26T07:48:20.000751+0000","last_unstale":"2024-07-26T07:48:20.002675+0000","last_undegraded":"2024-07-26T07:48:20.002675+0000","last_fullsized":"2024-07-26T07:48:20.002675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:26:24.513652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911865+0000","last_change":"2024-07-26T07:47:45.847931+0000","last_active":"2024-07-26T07:48:19.911865+0000","last_peered":"2024-07-26T07:48:19.911865+0000","last_clean":"2024-07-26T07:48:19.911865+0000","last_became_active":"2024-07-26T07:47:45.847484+0000","last_became_peered":"2024-07-26T07:47:45.847484+0000","last_unstale":"2024-07-26T07:48:19.911865+0000","last_undegraded":"2024-07-26T07:48:19.911865+0000","last_fullsized":"2024-07-26T07:48:19.911865+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:04.498360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922963+0000","last_change":"2024-07-26T07:47:45.846484+0000","last_active":"2024-07-26T07:48:19.922963+0000","last_peered":"2024-07-26T07:48:19.922963+0000","last_clean":"2024-07-26T07:48:19.922963+0000","last_became_active":"2024-07-26T07:47:45.844772+0000","last_became_peered":"2024-07-26T07:47:45.844772+0000","last_unstale":"2024-07-26T07:48:19.922963+0000","last_undegraded":"2024-07-26T07:48:19.922963+0000","last_fullsized":"2024-07-26T07:48:19.922963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:54.318026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.918081+0000","last_change":"2024-07-26T07:47:45.849840+0000","last_active":"2024-07-26T07:48:19.918081+0000","last_peered":"2024-07-26T07:48:19.918081+0000","last_clean":"2024-07-26T07:48:19.918081+0000","last_became_active":"2024-07-26T07:47:45.848744+0000","last_became_peered":"2024-07-26T07:47:45.848744+0000","last_unstale":"2024-07-26T07:48:19.918081+0000","last_undegraded":"2024-07-26T07:48:19.918081+0000","last_fullsized":"2024-07-26T07:48:19.918081+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:44.675382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911439+0000","last_change":"2024-07-26T07:47:45.847176+0000","last_active":"2024-07-26T07:48:19.911439+0000","last_peered":"2024-07-26T07:48:19.911439+0000","last_clean":"2024-07-26T07:48:19.911439+0000","last_became_active":"2024-07-26T07:47:45.845936+0000","last_became_peered":"2024-07-26T07:47:45.845936+0000","last_unstale":"2024-07-26T07:48:19.911439+0000","last_undegraded":"2024-07-26T07:48:19.911439+0000","last_fullsized":"2024-07-26T07:48:19.911439+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:02.245098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921801+0000","last_change":"2024-07-26T07:47:45.853210+0000","last_active":"2024-07-26T07:48:19.921801+0000","last_peered":"2024-07-26T07:48:19.921801+0000","last_clean":"2024-07-26T07:48:19.921801+0000","last_became_active":"2024-07-26T07:47:45.852766+0000","last_became_peered":"2024-07-26T07:47:45.852766+0000","last_unstale":"2024-07-26T07:48:19.921801+0000","last_undegraded":"2024-07-26T07:48:19.921801+0000","last_fullsized":"2024-07-26T07:48:19.921801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:57.406748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924715+0000","last_change":"2024-07-26T07:47:45.850699+0000","last_active":"2024-07-26T07:48:19.924715+0000","last_peered":"2024-07-26T07:48:19.924715+0000","last_clean":"2024-07-26T07:48:19.924715+0000","last_became_active":"2024-07-26T07:47:45.850314+0000","last_became_peered":"2024-07-26T07:47:45.850314+0000","last_unstale":"2024-07-26T07:48:19.924715+0000","last_undegraded":"2024-07-26T07:48:19.924715+0000","last_fullsized":"2024-07-26T07:48:19.924715+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:09.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920050+0000","last_change":"2024-07-26T07:47:45.851063+0000","last_active":"2024-07-26T07:48:19.920050+0000","last_peered":"2024-07-26T07:48:19.920050+0000","last_clean":"2024-07-26T07:48:19.920050+0000","last_became_active":"2024-07-26T07:47:45.850164+0000","last_became_peered":"2024-07-26T07:47:45.850164+0000","last_unstale":"2024-07-26T07:48:19.920050+0000","last_undegraded":"2024-07-26T07:48:19.920050+0000","last_fullsized":"2024-07-26T07:48:19.920050+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:31.597551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920032+0000","last_change":"2024-07-26T07:47:45.861207+0000","last_active":"2024-07-26T07:48:19.920032+0000","last_peered":"2024-07-26T07:48:19.920032+0000","last_clean":"2024-07-26T07:48:19.920032+0000","last_became_active":"2024-07-26T07:47:45.860179+0000","last_became_peered":"2024-07-26T07:47:45.860179+0000","last_unstale":"2024-07-26T07:48:19.920032+0000","last_undegraded":"2024-07-26T07:48:19.920032+0000","last_fullsized":"2024-07-26T07:48:19.920032+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:12:51.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911971+0000","last_change":"2024-07-26T07:47:45.846197+0000","last_active":"2024-07-26T07:48:19.911971+0000","last_peered":"2024-07-26T07:48:19.911971+0000","last_clean":"2024-07-26T07:48:19.911971+0000","last_became_active":"2024-07-26T07:47:45.845238+0000","last_became_peered":"2024-07-26T07:47:45.845238+0000","last_unstale":"2024-07-26T07:48:19.911971+0000","last_undegraded":"2024-07-26T07:48:19.911971+0000","last_fullsized":"2024-07-26T07:48:19.911971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:29.900684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911834+0000","last_change":"2024-07-26T07:47:45.847963+0000","last_active":"2024-07-26T07:48:19.911834+0000","last_peered":"2024-07-26T07:48:19.911834+0000","last_clean":"2024-07-26T07:48:19.911834+0000","last_became_active":"2024-07-26T07:47:45.847513+0000","last_became_peered":"2024-07-26T07:47:45.847513+0000","last_unstale":"2024-07-26T07:48:19.911834+0000","last_undegraded":"2024-07-26T07:48:19.911834+0000","last_fullsized":"2024-07-26T07:48:19.911834+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:51.082816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921823+0000","last_change":"2024-07-26T07:47:45.853273+0000","last_active":"2024-07-26T07:48:19.921823+0000","last_peered":"2024-07-26T07:48:19.921823+0000","last_clean":"2024-07-26T07:48:19.921823+0000","last_became_active":"2024-07-26T07:47:45.852894+0000","last_became_peered":"2024-07-26T07:47:45.852894+0000","last_unstale":"2024-07-26T07:48:19.921823+0000","last_undegraded":"2024-07-26T07:48:19.921823+0000","last_fullsized":"2024-07-26T07:48:19.921823+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:57:12.945228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.932104+0000","last_change":"2024-07-26T07:48:19.931938+0000","last_active":"2024-07-26T07:48:19.932104+0000","last_peered":"2024-07-26T07:48:19.932104+0000","last_clean":"2024-07-26T07:48:19.932104+0000","last_became_active":"2024-07-26T07:48:19.930976+0000","last_became_peered":"2024-07-26T07:48:19.930976+0000","last_unstale":"2024-07-26T07:48:19.932104+0000","last_undegraded":"2024-07-26T07:48:19.932104+0000","last_fullsized":"2024-07-26T07:48:19.932104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:59.610464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923540+0000","last_change":"2024-07-26T07:47:45.849736+0000","last_active":"2024-07-26T07:48:19.923540+0000","last_peered":"2024-07-26T07:48:19.923540+0000","last_clean":"2024-07-26T07:48:19.923540+0000","last_became_active":"2024-07-26T07:47:45.849335+0000","last_became_peered":"2024-07-26T07:47:45.849335+0000","last_unstale":"2024-07-26T07:48:19.923540+0000","last_undegraded":"2024-07-26T07:48:19.923540+0000","last_fullsized":"2024-07-26T07:48:19.923540+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:20:12.758331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922915+0000","last_change":"2024-07-26T07:47:45.847250+0000","last_active":"2024-07-26T07:48:19.922915+0000","last_peered":"2024-07-26T07:48:19.922915+0000","last_clean":"2024-07-26T07:48:19.922915+0000","last_became_active":"2024-07-26T07:47:45.845564+0000","last_became_peered":"2024-07-26T07:47:45.845564+0000","last_unstale":"2024-07-26T07:48:19.922915+0000","last_undegraded":"2024-07-26T07:48:19.922915+0000","last_fullsized":"2024-07-26T07:48:19.922915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:03.616782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924228+0000","last_change":"2024-07-26T07:47:45.850462+0000","last_active":"2024-07-26T07:48:19.924228+0000","last_peered":"2024-07-26T07:48:19.924228+0000","last_clean":"2024-07-26T07:48:19.924228+0000","last_became_active":"2024-07-26T07:47:45.850036+0000","last_became_peered":"2024-07-26T07:47:45.850036+0000","last_unstale":"2024-07-26T07:48:19.924228+0000","last_undegraded":"2024-07-26T07:48:19.924228+0000","last_fullsized":"2024-07-26T07:48:19.924228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:38.374782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911747+0000","last_change":"2024-07-26T07:47:45.859294+0000","last_active":"2024-07-26T07:48:19.911747+0000","last_peered":"2024-07-26T07:48:19.911747+0000","last_clean":"2024-07-26T07:48:19.911747+0000","last_became_active":"2024-07-26T07:47:45.858812+0000","last_became_peered":"2024-07-26T07:47:45.858812+0000","last_unstale":"2024-07-26T07:48:19.911747+0000","last_undegraded":"2024-07-26T07:48:19.911747+0000","last_fullsized":"2024-07-26T07:48:19.911747+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:11.689717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.931110+0000","last_change":"2024-07-26T07:48:19.930943+0000","last_active":"2024-07-26T07:48:19.931110+0000","last_peered":"2024-07-26T07:48:19.931110+0000","last_clean":"2024-07-26T07:48:19.931110+0000","last_became_active":"2024-07-26T07:48:19.929758+0000","last_became_peered":"2024-07-26T07:48:19.929758+0000","last_unstale":"2024-07-26T07:48:19.931110+0000","last_undegraded":"2024-07-26T07:48:19.931110+0000","last_fullsized":"2024-07-26T07:48:19.931110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:29:27.657376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920446+0000","last_change":"2024-07-26T07:47:45.846867+0000","last_active":"2024-07-26T07:48:19.920446+0000","last_peered":"2024-07-26T07:48:19.920446+0000","last_clean":"2024-07-26T07:48:19.920446+0000","last_became_active":"2024-07-26T07:47:45.846455+0000","last_became_peered":"2024-07-26T07:47:45.846455+0000","last_unstale":"2024-07-26T07:48:19.920446+0000","last_undegraded":"2024-07-26T07:48:19.920446+0000","last_fullsized":"2024-07-26T07:48:19.920446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:14.997452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922151+0000","last_change":"2024-07-26T07:47:45.846179+0000","last_active":"2024-07-26T07:48:19.922151+0000","last_peered":"2024-07-26T07:48:19.922151+0000","last_clean":"2024-07-26T07:48:19.922151+0000","last_became_active":"2024-07-26T07:47:45.844880+0000","last_became_peered":"2024-07-26T07:47:45.844880+0000","last_unstale":"2024-07-26T07:48:19.922151+0000","last_undegraded":"2024-07-26T07:48:19.922151+0000","last_fullsized":"2024-07-26T07:48:19.922151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:00:50.899573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.935350+0000","last_change":"2024-07-26T07:48:19.935196+0000","last_active":"2024-07-26T07:48:19.935350+0000","last_peered":"2024-07-26T07:48:19.935350+0000","last_clean":"2024-07-26T07:48:19.935350+0000","last_became_active":"2024-07-26T07:48:19.933730+0000","last_became_peered":"2024-07-26T07:48:19.933730+0000","last_unstale":"2024-07-26T07:48:19.935350+0000","last_undegraded":"2024-07-26T07:48:19.935350+0000","last_fullsized":"2024-07-26T07:48:19.935350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:10.305122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.919717+0000","last_change":"2024-07-26T07:47:45.843360+0000","last_active":"2024-07-26T07:48:19.919717+0000","last_peered":"2024-07-26T07:48:19.919717+0000","last_clean":"2024-07-26T07:48:19.919717+0000","last_became_active":"2024-07-26T07:47:45.842971+0000","last_became_peered":"2024-07-26T07:47:45.842971+0000","last_unstale":"2024-07-26T07:48:19.919717+0000","last_undegraded":"2024-07-26T07:48:19.919717+0000","last_fullsized":"2024-07-26T07:48:19.919717+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:58.687638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"82'5","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:38.939786+0000","last_change":"2024-07-26T07:47:41.787760+0000","last_active":"2024-07-26T07:48:38.939786+0000","last_peered":"2024-07-26T07:48:38.939786+0000","last_clean":"2024-07-26T07:48:38.939786+0000","last_became_active":"2024-07-26T07:47:41.787197+0000","last_became_peered":"2024-07-26T07:47:41.787197+0000","last_unstale":"2024-07-26T07:48:38.939786+0000","last_undegraded":"2024-07-26T07:48:38.939786+0000","last_fullsized":"2024-07-26T07:48:38.939786+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:14.597759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.913354+0000","last_change":"2024-07-26T07:47:45.846822+0000","last_active":"2024-07-26T07:48:19.913354+0000","last_peered":"2024-07-26T07:48:19.913354+0000","last_clean":"2024-07-26T07:48:19.913354+0000","last_became_active":"2024-07-26T07:47:45.846398+0000","last_became_peered":"2024-07-26T07:47:45.846398+0000","last_unstale":"2024-07-26T07:48:19.913354+0000","last_undegraded":"2024-07-26T07:48:19.913354+0000","last_fullsized":"2024-07-26T07:48:19.913354+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:33:34.727843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.995470+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:48:19.995470+0000","last_peered":"2024-07-26T07:48:19.995470+0000","last_clean":"2024-07-26T07:48:19.995470+0000","last_became_active":"2024-07-26T07:47:41.782077+0000","last_became_peered":"2024-07-26T07:47:41.782077+0000","last_unstale":"2024-07-26T07:48:19.995470+0000","last_undegraded":"2024-07-26T07:48:19.995470+0000","last_fullsized":"2024-07-26T07:48:19.995470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:42.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911008+0000","last_change":"2024-07-26T07:47:41.784615+0000","last_active":"2024-07-26T07:48:19.911008+0000","last_peered":"2024-07-26T07:48:19.911008+0000","last_clean":"2024-07-26T07:48:19.911008+0000","last_became_active":"2024-07-26T07:47:41.783945+0000","last_became_peered":"2024-07-26T07:47:41.783945+0000","last_unstale":"2024-07-26T07:48:19.911008+0000","last_undegraded":"2024-07-26T07:48:19.911008+0000","last_fullsized":"2024-07-26T07:48:19.911008+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:42.618524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921175+0000","last_change":"2024-07-26T07:47:45.838774+0000","last_active":"2024-07-26T07:48:19.921175+0000","last_peered":"2024-07-26T07:48:19.921175+0000","last_clean":"2024-07-26T07:48:19.921175+0000","last_became_active":"2024-07-26T07:47:45.837707+0000","last_became_peered":"2024-07-26T07:47:45.837707+0000","last_unstale":"2024-07-26T07:48:19.921175+0000","last_undegraded":"2024-07-26T07:48:19.921175+0000","last_fullsized":"2024-07-26T07:48:19.921175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:59.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.913292+0000","last_change":"2024-07-26T07:47:41.792031+0000","last_active":"2024-07-26T07:48:19.913292+0000","last_peered":"2024-07-26T07:48:19.913292+0000","last_clean":"2024-07-26T07:48:19.913292+0000","last_became_active":"2024-07-26T07:47:41.785955+0000","last_became_peered":"2024-07-26T07:47:41.785955+0000","last_unstale":"2024-07-26T07:48:19.913292+0000","last_undegraded":"2024-07-26T07:48:19.913292+0000","last_fullsized":"2024-07-26T07:48:19.913292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:01:47.073970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922983+0000","last_change":"2024-07-26T07:47:45.844595+0000","last_active":"2024-07-26T07:48:19.922983+0000","last_peered":"2024-07-26T07:48:19.922983+0000","last_clean":"2024-07-26T07:48:19.922983+0000","last_became_active":"2024-07-26T07:47:45.843381+0000","last_became_peered":"2024-07-26T07:47:45.843381+0000","last_unstale":"2024-07-26T07:48:19.922983+0000","last_undegraded":"2024-07-26T07:48:19.922983+0000","last_fullsized":"2024-07-26T07:48:19.922983+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:39.692463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'4","reported_seq":101,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:41.945267+0000","last_change":"2024-07-26T07:47:41.788239+0000","last_active":"2024-07-26T07:48:41.945267+0000","last_peered":"2024-07-26T07:48:41.945267+0000","last_clean":"2024-07-26T07:48:41.945267+0000","last_became_active":"2024-07-26T07:47:41.787470+0000","last_became_peered":"2024-07-26T07:47:41.787470+0000","last_unstale":"2024-07-26T07:48:41.945267+0000","last_undegraded":"2024-07-26T07:48:41.945267+0000","last_fullsized":"2024-07-26T07:48:41.945267+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:03.129563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.912774+0000","last_change":"2024-07-26T07:47:45.852794+0000","last_active":"2024-07-26T07:48:19.912774+0000","last_peered":"2024-07-26T07:48:19.912774+0000","last_clean":"2024-07-26T07:48:19.912774+0000","last_became_active":"2024-07-26T07:47:45.849699+0000","last_became_peered":"2024-07-26T07:47:45.849699+0000","last_unstale":"2024-07-26T07:48:19.912774+0000","last_undegraded":"2024-07-26T07:48:19.912774+0000","last_fullsized":"2024-07-26T07:48:19.912774+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:06:31.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933479+0000","last_change":"2024-07-26T07:47:41.789414+0000","last_active":"2024-07-26T07:48:19.933479+0000","last_peered":"2024-07-26T07:48:19.933479+0000","last_clean":"2024-07-26T07:48:19.933479+0000","last_became_active":"2024-07-26T07:47:41.788679+0000","last_became_peered":"2024-07-26T07:47:41.788679+0000","last_unstale":"2024-07-26T07:48:19.933479+0000","last_undegraded":"2024-07-26T07:48:19.933479+0000","last_fullsized":"2024-07-26T07:48:19.933479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:51.809997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.937784+0000","last_change":"2024-07-26T07:48:19.937603+0000","last_active":"2024-07-26T07:48:19.937784+0000","last_peered":"2024-07-26T07:48:19.937784+0000","last_clean":"2024-07-26T07:48:19.937784+0000","last_became_active":"2024-07-26T07:48:19.936889+0000","last_became_peered":"2024-07-26T07:48:19.936889+0000","last_unstale":"2024-07-26T07:48:19.937784+0000","last_undegraded":"2024-07-26T07:48:19.937784+0000","last_fullsized":"2024-07-26T07:48:19.937784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:46:35.138825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":33,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.914560+0000","last_change":"2024-07-26T07:47:42.564288+0000","last_active":"2024-07-26T07:48:19.914560+0000","last_peered":"2024-07-26T07:48:19.914560+0000","last_clean":"2024-07-26T07:48:19.914560+0000","last_became_active":"2024-07-26T07:47:42.561766+0000","last_became_peered":"2024-07-26T07:47:42.561766+0000","last_unstale":"2024-07-26T07:48:19.914560+0000","last_undegraded":"2024-07-26T07:48:19.914560+0000","last_fullsized":"2024-07-26T07:48:19.914560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:16.414030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.912074+0000","last_change":"2024-07-26T07:47:45.861895+0000","last_active":"2024-07-26T07:48:19.912074+0000","last_peered":"2024-07-26T07:48:19.912074+0000","last_clean":"2024-07-26T07:48:19.912074+0000","last_became_active":"2024-07-26T07:47:45.861439+0000","last_became_peered":"2024-07-26T07:47:45.861439+0000","last_unstale":"2024-07-26T07:48:19.912074+0000","last_undegraded":"2024-07-26T07:48:19.912074+0000","last_fullsized":"2024-07-26T07:48:19.912074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:26.276453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"84'31","reported_seq":137,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:41.969987+0000","last_change":"2024-07-26T07:47:41.789977+0000","last_active":"2024-07-26T07:48:41.969987+0000","last_peered":"2024-07-26T07:48:41.969987+0000","last_clean":"2024-07-26T07:48:41.969987+0000","last_became_active":"2024-07-26T07:47:41.789488+0000","last_became_peered":"2024-07-26T07:47:41.789488+0000","last_unstale":"2024-07-26T07:48:41.969987+0000","last_undegraded":"2024-07-26T07:48:41.969987+0000","last_fullsized":"2024-07-26T07:48:41.969987+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:33.059279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59762,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":79,"num_read_kb":97,"num_write":33,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.925719+0000","last_change":"2024-07-26T07:47:45.854772+0000","last_active":"2024-07-26T07:48:19.925719+0000","last_peered":"2024-07-26T07:48:19.925719+0000","last_clean":"2024-07-26T07:48:19.925719+0000","last_became_active":"2024-07-26T07:47:45.854354+0000","last_became_peered":"2024-07-26T07:47:45.854354+0000","last_unstale":"2024-07-26T07:48:19.925719+0000","last_undegraded":"2024-07-26T07:48:19.925719+0000","last_fullsized":"2024-07-26T07:48:19.925719+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:31:54.856407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923492+0000","last_change":"2024-07-26T07:47:41.789752+0000","last_active":"2024-07-26T07:48:19.923492+0000","last_peered":"2024-07-26T07:48:19.923492+0000","last_clean":"2024-07-26T07:48:19.923492+0000","last_became_active":"2024-07-26T07:47:41.788952+0000","last_became_peered":"2024-07-26T07:47:41.788952+0000","last_unstale":"2024-07-26T07:48:19.923492+0000","last_undegraded":"2024-07-26T07:48:19.923492+0000","last_fullsized":"2024-07-26T07:48:19.923492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:45.141693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911113+0000","last_change":"2024-07-26T07:47:45.845120+0000","last_active":"2024-07-26T07:48:19.911113+0000","last_peered":"2024-07-26T07:48:19.911113+0000","last_clean":"2024-07-26T07:48:19.911113+0000","last_became_active":"2024-07-26T07:47:45.844699+0000","last_became_peered":"2024-07-26T07:47:45.844699+0000","last_unstale":"2024-07-26T07:48:19.911113+0000","last_undegraded":"2024-07-26T07:48:19.911113+0000","last_fullsized":"2024-07-26T07:48:19.911113+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:23:42.130846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":33,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.913543+0000","last_change":"2024-07-26T07:47:41.790765+0000","last_active":"2024-07-26T07:48:19.913543+0000","last_peered":"2024-07-26T07:48:19.913543+0000","last_clean":"2024-07-26T07:48:19.913543+0000","last_became_active":"2024-07-26T07:47:41.790288+0000","last_became_peered":"2024-07-26T07:47:41.790288+0000","last_unstale":"2024-07-26T07:48:19.913543+0000","last_undegraded":"2024-07-26T07:48:19.913543+0000","last_fullsized":"2024-07-26T07:48:19.913543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:05.734468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.932830+0000","last_change":"2024-07-26T07:47:45.854648+0000","last_active":"2024-07-26T07:48:19.932830+0000","last_peered":"2024-07-26T07:48:19.932830+0000","last_clean":"2024-07-26T07:48:19.932830+0000","last_became_active":"2024-07-26T07:47:45.854240+0000","last_became_peered":"2024-07-26T07:47:45.854240+0000","last_unstale":"2024-07-26T07:48:19.932830+0000","last_undegraded":"2024-07-26T07:48:19.932830+0000","last_fullsized":"2024-07-26T07:48:19.932830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:41.775760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:38.938127+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:48:38.938127+0000","last_peered":"2024-07-26T07:48:38.938127+0000","last_clean":"2024-07-26T07:48:38.938127+0000","last_became_active":"2024-07-26T07:47:41.781324+0000","last_became_peered":"2024-07-26T07:47:41.781324+0000","last_unstale":"2024-07-26T07:48:38.938127+0000","last_undegraded":"2024-07-26T07:48:38.938127+0000","last_fullsized":"2024-07-26T07:48:38.938127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:45.154354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.931890+0000","last_change":"2024-07-26T07:47:45.848731+0000","last_active":"2024-07-26T07:48:19.931890+0000","last_peered":"2024-07-26T07:48:19.931890+0000","last_clean":"2024-07-26T07:48:19.931890+0000","last_became_active":"2024-07-26T07:47:45.846663+0000","last_became_peered":"2024-07-26T07:47:45.846663+0000","last_unstale":"2024-07-26T07:48:19.931890+0000","last_undegraded":"2024-07-26T07:48:19.931890+0000","last_fullsized":"2024-07-26T07:48:19.931890+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:05.251423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922521+0000","last_change":"2024-07-26T07:47:42.561839+0000","last_active":"2024-07-26T07:48:19.922521+0000","last_peered":"2024-07-26T07:48:19.922521+0000","last_clean":"2024-07-26T07:48:19.922521+0000","last_became_active":"2024-07-26T07:47:42.560417+0000","last_became_peered":"2024-07-26T07:47:42.560417+0000","last_unstale":"2024-07-26T07:48:19.922521+0000","last_undegraded":"2024-07-26T07:48:19.922521+0000","last_fullsized":"2024-07-26T07:48:19.922521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:16.717573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.912844+0000","last_change":"2024-07-26T07:47:45.842572+0000","last_active":"2024-07-26T07:48:19.912844+0000","last_peered":"2024-07-26T07:48:19.912844+0000","last_clean":"2024-07-26T07:48:19.912844+0000","last_became_active":"2024-07-26T07:47:45.842116+0000","last_became_peered":"2024-07-26T07:47:45.842116+0000","last_unstale":"2024-07-26T07:48:19.912844+0000","last_undegraded":"2024-07-26T07:48:19.912844+0000","last_fullsized":"2024-07-26T07:48:19.912844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:06.913737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923293+0000","last_change":"2024-07-26T07:47:41.790215+0000","last_active":"2024-07-26T07:48:19.923293+0000","last_peered":"2024-07-26T07:48:19.923293+0000","last_clean":"2024-07-26T07:48:19.923293+0000","last_became_active":"2024-07-26T07:47:41.789691+0000","last_became_peered":"2024-07-26T07:47:41.789691+0000","last_unstale":"2024-07-26T07:48:19.923293+0000","last_undegraded":"2024-07-26T07:48:19.923293+0000","last_fullsized":"2024-07-26T07:48:19.923293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:01:47.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.921097+0000","last_change":"2024-07-26T07:47:45.860887+0000","last_active":"2024-07-26T07:48:19.921097+0000","last_peered":"2024-07-26T07:48:19.921097+0000","last_clean":"2024-07-26T07:48:19.921097+0000","last_became_active":"2024-07-26T07:47:45.860315+0000","last_became_peered":"2024-07-26T07:47:45.860315+0000","last_unstale":"2024-07-26T07:48:19.921097+0000","last_undegraded":"2024-07-26T07:48:19.921097+0000","last_fullsized":"2024-07-26T07:48:19.921097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:17:02.231206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:20.023019+0000","last_change":"2024-07-26T07:48:20.022885+0000","last_active":"2024-07-26T07:48:20.023019+0000","last_peered":"2024-07-26T07:48:20.023019+0000","last_clean":"2024-07-26T07:48:20.023019+0000","last_became_active":"2024-07-26T07:48:20.001946+0000","last_became_peered":"2024-07-26T07:48:20.001946+0000","last_unstale":"2024-07-26T07:48:20.023019+0000","last_undegraded":"2024-07-26T07:48:20.023019+0000","last_fullsized":"2024-07-26T07:48:20.023019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:45.892886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.934773+0000","last_change":"2024-07-26T07:47:45.858267+0000","last_active":"2024-07-26T07:48:19.934773+0000","last_peered":"2024-07-26T07:48:19.934773+0000","last_clean":"2024-07-26T07:48:19.934773+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:48:19.934773+0000","last_undegraded":"2024-07-26T07:48:19.934773+0000","last_fullsized":"2024-07-26T07:48:19.934773+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:32.055977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.914861+0000","last_change":"2024-07-26T07:47:41.789846+0000","last_active":"2024-07-26T07:48:19.914861+0000","last_peered":"2024-07-26T07:48:19.914861+0000","last_clean":"2024-07-26T07:48:19.914861+0000","last_became_active":"2024-07-26T07:47:41.789237+0000","last_became_peered":"2024-07-26T07:47:41.789237+0000","last_unstale":"2024-07-26T07:48:19.914861+0000","last_undegraded":"2024-07-26T07:48:19.914861+0000","last_fullsized":"2024-07-26T07:48:19.914861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:39.031047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"84'1","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.932650+0000","last_change":"2024-07-26T07:47:45.846294+0000","last_active":"2024-07-26T07:48:19.932650+0000","last_peered":"2024-07-26T07:48:19.932650+0000","last_clean":"2024-07-26T07:48:19.932650+0000","last_became_active":"2024-07-26T07:47:45.845861+0000","last_became_peered":"2024-07-26T07:47:45.845861+0000","last_unstale":"2024-07-26T07:48:19.932650+0000","last_undegraded":"2024-07-26T07:48:19.932650+0000","last_fullsized":"2024-07-26T07:48:19.932650+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:56:10.594430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'6","reported_seq":69,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:38.939001+0000","last_change":"2024-07-26T07:47:42.563412+0000","last_active":"2024-07-26T07:48:38.939001+0000","last_peered":"2024-07-26T07:48:38.939001+0000","last_clean":"2024-07-26T07:48:38.939001+0000","last_became_active":"2024-07-26T07:47:42.562364+0000","last_became_peered":"2024-07-26T07:47:42.562364+0000","last_unstale":"2024-07-26T07:48:38.939001+0000","last_undegraded":"2024-07-26T07:48:38.939001+0000","last_fullsized":"2024-07-26T07:48:38.939001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:14.796738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933936+0000","last_change":"2024-07-26T07:47:45.856170+0000","last_active":"2024-07-26T07:48:19.933936+0000","last_peered":"2024-07-26T07:48:19.933936+0000","last_clean":"2024-07-26T07:48:19.933936+0000","last_became_active":"2024-07-26T07:47:45.855782+0000","last_became_peered":"2024-07-26T07:47:45.855782+0000","last_unstale":"2024-07-26T07:48:19.933936+0000","last_undegraded":"2024-07-26T07:48:19.933936+0000","last_fullsized":"2024-07-26T07:48:19.933936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:22.690910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":106,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933763+0000","last_change":"2024-07-26T07:45:54.310557+0000","last_active":"2024-07-26T07:48:19.933763+0000","last_peered":"2024-07-26T07:48:19.933763+0000","last_clean":"2024-07-26T07:48:19.933763+0000","last_became_active":"2024-07-26T07:45:54.290906+0000","last_became_peered":"2024-07-26T07:45:54.290906+0000","last_unstale":"2024-07-26T07:48:19.933763+0000","last_undegraded":"2024-07-26T07:48:19.933763+0000","last_fullsized":"2024-07-26T07:48:19.933763+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_clean_scrub_stamp":"2024-07-26T07:41:16.689463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:51.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.999798+0000","last_change":"2024-07-26T07:47:41.776420+0000","last_active":"2024-07-26T07:48:19.999798+0000","last_peered":"2024-07-26T07:48:19.999798+0000","last_clean":"2024-07-26T07:48:19.999798+0000","last_became_active":"2024-07-26T07:47:41.775858+0000","last_became_peered":"2024-07-26T07:47:41.775858+0000","last_unstale":"2024-07-26T07:48:19.999798+0000","last_undegraded":"2024-07-26T07:48:19.999798+0000","last_fullsized":"2024-07-26T07:48:19.999798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:18.880331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.935467+0000","last_change":"2024-07-26T07:48:19.933318+0000","last_active":"2024-07-26T07:48:19.935467+0000","last_peered":"2024-07-26T07:48:19.935467+0000","last_clean":"2024-07-26T07:48:19.935467+0000","last_became_active":"2024-07-26T07:48:19.932683+0000","last_became_peered":"2024-07-26T07:48:19.932683+0000","last_unstale":"2024-07-26T07:48:19.935467+0000","last_undegraded":"2024-07-26T07:48:19.935467+0000","last_fullsized":"2024-07-26T07:48:19.935467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:25:23.387428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933375+0000","last_change":"2024-07-26T07:47:41.786698+0000","last_active":"2024-07-26T07:48:19.933375+0000","last_peered":"2024-07-26T07:48:19.933375+0000","last_clean":"2024-07-26T07:48:19.933375+0000","last_became_active":"2024-07-26T07:47:41.784491+0000","last_became_peered":"2024-07-26T07:47:41.784491+0000","last_unstale":"2024-07-26T07:48:19.933375+0000","last_undegraded":"2024-07-26T07:48:19.933375+0000","last_fullsized":"2024-07-26T07:48:19.933375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:09:08.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.910962+0000","last_change":"2024-07-26T07:47:45.849698+0000","last_active":"2024-07-26T07:48:19.910962+0000","last_peered":"2024-07-26T07:48:19.910962+0000","last_clean":"2024-07-26T07:48:19.910962+0000","last_became_active":"2024-07-26T07:47:45.849266+0000","last_became_peered":"2024-07-26T07:47:45.849266+0000","last_unstale":"2024-07-26T07:48:19.910962+0000","last_undegraded":"2024-07-26T07:48:19.910962+0000","last_fullsized":"2024-07-26T07:48:19.910962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:57.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:37.933322+0000","last_change":"2024-07-26T07:47:42.558680+0000","last_active":"2024-07-26T07:48:37.933322+0000","last_peered":"2024-07-26T07:48:37.933322+0000","last_clean":"2024-07-26T07:48:37.933322+0000","last_became_active":"2024-07-26T07:47:42.558197+0000","last_became_peered":"2024-07-26T07:47:42.558197+0000","last_unstale":"2024-07-26T07:48:37.933322+0000","last_undegraded":"2024-07-26T07:48:37.933322+0000","last_fullsized":"2024-07-26T07:48:37.933322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:52:52.505455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.930090+0000","last_change":"2024-07-26T07:48:19.929897+0000","last_active":"2024-07-26T07:48:19.930090+0000","last_peered":"2024-07-26T07:48:19.930090+0000","last_clean":"2024-07-26T07:48:19.930090+0000","last_became_active":"2024-07-26T07:48:19.927946+0000","last_became_peered":"2024-07-26T07:48:19.927946+0000","last_unstale":"2024-07-26T07:48:19.930090+0000","last_undegraded":"2024-07-26T07:48:19.930090+0000","last_fullsized":"2024-07-26T07:48:19.930090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:11.157824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"82'5","reported_seq":69,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:42.001944+0000","last_change":"2024-07-26T07:47:41.785513+0000","last_active":"2024-07-26T07:48:42.001944+0000","last_peered":"2024-07-26T07:48:42.001944+0000","last_clean":"2024-07-26T07:48:42.001944+0000","last_became_active":"2024-07-26T07:47:41.784783+0000","last_became_peered":"2024-07-26T07:47:41.784783+0000","last_unstale":"2024-07-26T07:48:42.001944+0000","last_undegraded":"2024-07-26T07:48:42.001944+0000","last_fullsized":"2024-07-26T07:48:42.001944+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:54.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.995470+0000","last_change":"2024-07-26T07:47:45.856580+0000","last_active":"2024-07-26T07:48:19.995470+0000","last_peered":"2024-07-26T07:48:19.995470+0000","last_clean":"2024-07-26T07:48:19.995470+0000","last_became_active":"2024-07-26T07:47:45.852330+0000","last_became_peered":"2024-07-26T07:47:45.852330+0000","last_unstale":"2024-07-26T07:48:19.995470+0000","last_undegraded":"2024-07-26T07:48:19.995470+0000","last_fullsized":"2024-07-26T07:48:19.995470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:09.479563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924407+0000","last_change":"2024-07-26T07:47:42.559255+0000","last_active":"2024-07-26T07:48:19.924407+0000","last_peered":"2024-07-26T07:48:19.924407+0000","last_clean":"2024-07-26T07:48:19.924407+0000","last_became_active":"2024-07-26T07:47:42.558123+0000","last_became_peered":"2024-07-26T07:47:42.558123+0000","last_unstale":"2024-07-26T07:48:19.924407+0000","last_undegraded":"2024-07-26T07:48:19.924407+0000","last_fullsized":"2024-07-26T07:48:19.924407+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:49:46.576674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'1","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.995941+0000","last_change":"2024-07-26T07:47:45.856256+0000","last_active":"2024-07-26T07:48:19.995941+0000","last_peered":"2024-07-26T07:48:19.995941+0000","last_clean":"2024-07-26T07:48:19.995941+0000","last_became_active":"2024-07-26T07:47:45.855696+0000","last_became_peered":"2024-07-26T07:47:45.855696+0000","last_unstale":"2024-07-26T07:48:19.995941+0000","last_undegraded":"2024-07-26T07:48:19.995941+0000","last_fullsized":"2024-07-26T07:48:19.995941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:24.451225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":32,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.925947+0000","last_change":"2024-07-26T07:47:42.562631+0000","last_active":"2024-07-26T07:48:19.925947+0000","last_peered":"2024-07-26T07:48:19.925947+0000","last_clean":"2024-07-26T07:48:19.925947+0000","last_became_active":"2024-07-26T07:47:42.561379+0000","last_became_peered":"2024-07-26T07:47:42.561379+0000","last_unstale":"2024-07-26T07:48:19.925947+0000","last_undegraded":"2024-07-26T07:48:19.925947+0000","last_fullsized":"2024-07-26T07:48:19.925947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:03:30.471694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.912133+0000","last_change":"2024-07-26T07:47:45.849580+0000","last_active":"2024-07-26T07:48:19.912133+0000","last_peered":"2024-07-26T07:48:19.912133+0000","last_clean":"2024-07-26T07:48:19.912133+0000","last_became_active":"2024-07-26T07:47:45.849107+0000","last_became_peered":"2024-07-26T07:47:45.849107+0000","last_unstale":"2024-07-26T07:48:19.912133+0000","last_undegraded":"2024-07-26T07:48:19.912133+0000","last_fullsized":"2024-07-26T07:48:19.912133+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:57:13.694059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"84'15","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:32.866485+0000","last_change":"2024-07-26T07:47:41.774083+0000","last_active":"2024-07-26T07:48:32.866485+0000","last_peered":"2024-07-26T07:48:32.866485+0000","last_clean":"2024-07-26T07:48:32.866485+0000","last_became_active":"2024-07-26T07:47:41.773502+0000","last_became_peered":"2024-07-26T07:47:41.773502+0000","last_unstale":"2024-07-26T07:48:32.866485+0000","last_undegraded":"2024-07-26T07:48:32.866485+0000","last_fullsized":"2024-07-26T07:48:32.866485+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:11:44.083944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923886+0000","last_change":"2024-07-26T07:47:45.845894+0000","last_active":"2024-07-26T07:48:19.923886+0000","last_peered":"2024-07-26T07:48:19.923886+0000","last_clean":"2024-07-26T07:48:19.923886+0000","last_became_active":"2024-07-26T07:47:45.845534+0000","last_became_peered":"2024-07-26T07:47:45.845534+0000","last_unstale":"2024-07-26T07:48:19.923886+0000","last_undegraded":"2024-07-26T07:48:19.923886+0000","last_fullsized":"2024-07-26T07:48:19.923886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:52:42.209561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.999884+0000","last_change":"2024-07-26T07:47:41.904220+0000","last_active":"2024-07-26T07:48:19.999884+0000","last_peered":"2024-07-26T07:48:19.999884+0000","last_clean":"2024-07-26T07:48:19.999884+0000","last_became_active":"2024-07-26T07:47:41.902349+0000","last_became_peered":"2024-07-26T07:47:41.902349+0000","last_unstale":"2024-07-26T07:48:19.999884+0000","last_undegraded":"2024-07-26T07:48:19.999884+0000","last_fullsized":"2024-07-26T07:48:19.999884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:08.520461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923525+0000","last_change":"2024-07-26T07:47:45.846920+0000","last_active":"2024-07-26T07:48:19.923525+0000","last_peered":"2024-07-26T07:48:19.923525+0000","last_clean":"2024-07-26T07:48:19.923525+0000","last_became_active":"2024-07-26T07:47:45.846458+0000","last_became_peered":"2024-07-26T07:47:45.846458+0000","last_unstale":"2024-07-26T07:48:19.923525+0000","last_undegraded":"2024-07-26T07:48:19.923525+0000","last_fullsized":"2024-07-26T07:48:19.923525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:06.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911839+0000","last_change":"2024-07-26T07:47:42.560346+0000","last_active":"2024-07-26T07:48:19.911839+0000","last_peered":"2024-07-26T07:48:19.911839+0000","last_clean":"2024-07-26T07:48:19.911839+0000","last_became_active":"2024-07-26T07:47:42.558937+0000","last_became_peered":"2024-07-26T07:47:42.558937+0000","last_unstale":"2024-07-26T07:48:19.911839+0000","last_undegraded":"2024-07-26T07:48:19.911839+0000","last_fullsized":"2024-07-26T07:48:19.911839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.676625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923945+0000","last_change":"2024-07-26T07:47:45.845553+0000","last_active":"2024-07-26T07:48:19.923945+0000","last_peered":"2024-07-26T07:48:19.923945+0000","last_clean":"2024-07-26T07:48:19.923945+0000","last_became_active":"2024-07-26T07:47:45.845125+0000","last_became_peered":"2024-07-26T07:47:45.845125+0000","last_unstale":"2024-07-26T07:48:19.923945+0000","last_undegraded":"2024-07-26T07:48:19.923945+0000","last_fullsized":"2024-07-26T07:48:19.923945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:10.896159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922224+0000","last_change":"2024-07-26T07:47:41.779688+0000","last_active":"2024-07-26T07:48:19.922224+0000","last_peered":"2024-07-26T07:48:19.922224+0000","last_clean":"2024-07-26T07:48:19.922224+0000","last_became_active":"2024-07-26T07:47:41.779115+0000","last_became_peered":"2024-07-26T07:47:41.779115+0000","last_unstale":"2024-07-26T07:48:19.922224+0000","last_undegraded":"2024-07-26T07:48:19.922224+0000","last_fullsized":"2024-07-26T07:48:19.922224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:20.708336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920383+0000","last_change":"2024-07-26T07:47:45.853869+0000","last_active":"2024-07-26T07:48:19.920383+0000","last_peered":"2024-07-26T07:48:19.920383+0000","last_clean":"2024-07-26T07:48:19.920383+0000","last_became_active":"2024-07-26T07:47:45.852509+0000","last_became_peered":"2024-07-26T07:47:45.852509+0000","last_unstale":"2024-07-26T07:48:19.920383+0000","last_undegraded":"2024-07-26T07:48:19.920383+0000","last_fullsized":"2024-07-26T07:48:19.920383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:01:31.764025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.920208+0000","last_change":"2024-07-26T07:47:41.792331+0000","last_active":"2024-07-26T07:48:19.920208+0000","last_peered":"2024-07-26T07:48:19.920208+0000","last_clean":"2024-07-26T07:48:19.920208+0000","last_became_active":"2024-07-26T07:47:41.790668+0000","last_became_peered":"2024-07-26T07:47:41.790668+0000","last_unstale":"2024-07-26T07:48:19.920208+0000","last_undegraded":"2024-07-26T07:48:19.920208+0000","last_fullsized":"2024-07-26T07:48:19.920208+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:59.430692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.922438+0000","last_change":"2024-07-26T07:47:45.843583+0000","last_active":"2024-07-26T07:48:19.922438+0000","last_peered":"2024-07-26T07:48:19.922438+0000","last_clean":"2024-07-26T07:48:19.922438+0000","last_became_active":"2024-07-26T07:47:45.843185+0000","last_became_peered":"2024-07-26T07:47:45.843185+0000","last_unstale":"2024-07-26T07:48:19.922438+0000","last_undegraded":"2024-07-26T07:48:19.922438+0000","last_fullsized":"2024-07-26T07:48:19.922438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:01.119065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:20.014568+0000","last_change":"2024-07-26T07:48:20.014411+0000","last_active":"2024-07-26T07:48:20.014568+0000","last_peered":"2024-07-26T07:48:20.014568+0000","last_clean":"2024-07-26T07:48:20.014568+0000","last_became_active":"2024-07-26T07:48:20.002085+0000","last_became_peered":"2024-07-26T07:48:20.002085+0000","last_unstale":"2024-07-26T07:48:20.014568+0000","last_undegraded":"2024-07-26T07:48:20.014568+0000","last_fullsized":"2024-07-26T07:48:20.014568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:46:59.335128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.933049+0000","last_change":"2024-07-26T07:47:45.851963+0000","last_active":"2024-07-26T07:48:19.933049+0000","last_peered":"2024-07-26T07:48:19.933049+0000","last_clean":"2024-07-26T07:48:19.933049+0000","last_became_active":"2024-07-26T07:47:45.850459+0000","last_became_peered":"2024-07-26T07:47:45.850459+0000","last_unstale":"2024-07-26T07:48:19.933049+0000","last_undegraded":"2024-07-26T07:48:19.933049+0000","last_fullsized":"2024-07-26T07:48:19.933049+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:18.092186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.914232+0000","last_change":"2024-07-26T07:47:45.848744+0000","last_active":"2024-07-26T07:48:19.914232+0000","last_peered":"2024-07-26T07:48:19.914232+0000","last_clean":"2024-07-26T07:48:19.914232+0000","last_became_active":"2024-07-26T07:47:45.848292+0000","last_became_peered":"2024-07-26T07:47:45.848292+0000","last_unstale":"2024-07-26T07:48:19.914232+0000","last_undegraded":"2024-07-26T07:48:19.914232+0000","last_fullsized":"2024-07-26T07:48:19.914232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:54.549508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":33,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923455+0000","last_change":"2024-07-26T07:47:41.788875+0000","last_active":"2024-07-26T07:48:19.923455+0000","last_peered":"2024-07-26T07:48:19.923455+0000","last_clean":"2024-07-26T07:48:19.923455+0000","last_became_active":"2024-07-26T07:47:41.786942+0000","last_became_peered":"2024-07-26T07:47:41.786942+0000","last_unstale":"2024-07-26T07:48:19.923455+0000","last_undegraded":"2024-07-26T07:48:19.923455+0000","last_fullsized":"2024-07-26T07:48:19.923455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:11:31.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.911805+0000","last_change":"2024-07-26T07:47:45.854832+0000","last_active":"2024-07-26T07:48:19.911805+0000","last_peered":"2024-07-26T07:48:19.911805+0000","last_clean":"2024-07-26T07:48:19.911805+0000","last_became_active":"2024-07-26T07:47:45.853756+0000","last_became_peered":"2024-07-26T07:47:45.853756+0000","last_unstale":"2024-07-26T07:48:19.911805+0000","last_undegraded":"2024-07-26T07:48:19.911805+0000","last_fullsized":"2024-07-26T07:48:19.911805+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:42:14.579177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.925948+0000","last_change":"2024-07-26T07:47:41.787839+0000","last_active":"2024-07-26T07:48:19.925948+0000","last_peered":"2024-07-26T07:48:19.925948+0000","last_clean":"2024-07-26T07:48:19.925948+0000","last_became_active":"2024-07-26T07:47:41.785951+0000","last_became_peered":"2024-07-26T07:47:41.785951+0000","last_unstale":"2024-07-26T07:48:19.925948+0000","last_undegraded":"2024-07-26T07:48:19.925948+0000","last_fullsized":"2024-07-26T07:48:19.925948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:58.699056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.918445+0000","last_change":"2024-07-26T07:47:45.841451+0000","last_active":"2024-07-26T07:48:19.918445+0000","last_peered":"2024-07-26T07:48:19.918445+0000","last_clean":"2024-07-26T07:48:19.918445+0000","last_became_active":"2024-07-26T07:47:45.840803+0000","last_became_peered":"2024-07-26T07:47:45.840803+0000","last_unstale":"2024-07-26T07:48:19.918445+0000","last_undegraded":"2024-07-26T07:48:19.918445+0000","last_fullsized":"2024-07-26T07:48:19.918445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:24:21.832317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.924186+0000","last_change":"2024-07-26T07:47:42.558775+0000","last_active":"2024-07-26T07:48:19.924186+0000","last_peered":"2024-07-26T07:48:19.924186+0000","last_clean":"2024-07-26T07:48:19.924186+0000","last_became_active":"2024-07-26T07:47:42.558300+0000","last_became_peered":"2024-07-26T07:47:42.558300+0000","last_unstale":"2024-07-26T07:48:19.924186+0000","last_undegraded":"2024-07-26T07:48:19.924186+0000","last_fullsized":"2024-07-26T07:48:19.924186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:59.643709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:19.923531+0000","last_change":"2024-07-26T07:47:45.852401+0000","last_active":"2024-07-26T07:48:19.923531+0000","last_peered":"2024-07-26T07:48:19.923531+0000","last_clean":"2024-07-26T07:48:19.923531+0000","last_became_active":"2024-07-26T07:47:45.852054+0000","last_became_peered":"2024-07-26T07:47:45.852054+0000","last_unstale":"2024-07-26T07:48:19.923531+0000","last_undegraded":"2024-07-26T07:48:19.923531+0000","last_fullsized":"2024-07-26T07:48:19.923531+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:25.347770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"82'13","reported_seq":98,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-07-26T07:48:42.001163+0000","last_change":"2024-07-26T07:48:19.961760+0000","last_active":"2024-07-26T07:48:42.001163+0000","last_peered":"2024-07-26T07:48:42.001163+0000","last_clean":"2024-07-26T07:48:42.001163+0000","last_became_active":"2024-07-26T07:48:19.937694+0000","last_became_peered":"2024-07-26T07:48:19.937694+0000","last_unstale":"2024-07-26T07:48:42.001163+0000","last_undegraded":"2024-07-26T07:48:42.001163+0000","last_fullsized":"2024-07-26T07:48:42.001163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:05:10.571086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":31,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66575,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":377,"num_read_kb":260,"num_write":244,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":199725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579935,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":925696,"data_stored":446295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776165,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1626112,"data_stored":1096695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972396,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":445834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168627,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":445540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397561,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1038339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561087,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36256,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708896,"statfs":{"total":95995035648,"available":95957909504,"internally_reserved":0,"allocated":933888,"data_stored":445551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36856,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708296,"statfs":{"total":95995035648,"available":95957295104,"internally_reserved":0,"allocated":1556480,"data_stored":1038017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":445412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":445376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":506837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":506898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":954368,"data_stored":447281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T07:48:46.412 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T07:48:46.412 INFO:tasks.cephadm:Setting up client nodes... 2024-07-26T07:48:46.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-07-26T07:48:46.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:46 smithi115 ceph-mon[35364]: pgmap v417: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T07:48:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:46 smithi181 ceph-mon[35869]: pgmap v417: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T07:48:46.925 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:47.382 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:47 smithi097 ceph-mon[32231]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:48:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:47 smithi115 ceph-mon[35364]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:48:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:47 smithi181 ceph-mon[35869]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:48:48.426 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:48 smithi097 ceph-mon[32231]: pgmap v418: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T07:48:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:48 smithi115 ceph-mon[35364]: pgmap v418: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T07:48:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:48 smithi181 ceph-mon[35869]: pgmap v418: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T07:48:49.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 -- 172.21.15.97:0/416113024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106250 msgr2=0x7f82a0106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:49.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 --2- 172.21.15.97:0/416113024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106250 0x7f82a0106660 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8288009900 tx=0x7f828802a490 comp rx=0 tx=0).stop 2024-07-26T07:48:49.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 -- 172.21.15.97:0/416113024 shutdown_connections 2024-07-26T07:48:49.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 --2- 172.21.15.97:0/416113024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a0106bf0 0x7f82a010d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 --2- 172.21.15.97:0/416113024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106250 0x7f82a0106660 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 --2- 172.21.15.97:0/416113024 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82a01051c0 0x7f82a0105630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 -- 172.21.15.97:0/416113024 >> 172.21.15.97:0/416113024 conn(0x7f82a0100b10 msgr2=0x7f82a0102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 -- 172.21.15.97:0/416113024 shutdown_connections 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.054+0000 7f829fe00640 1 -- 172.21.15.97:0/416113024 wait complete. 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.055+0000 7f829fe00640 1 Processor -- start 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.055+0000 7f829fe00640 1 -- start start 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a01051c0 0x7f82a019b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829fe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82a0106250 0x7f82a019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106bf0 0x7f82a019caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0110d20 con 0x7f82a0106bf0 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829fe00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0110ba0 con 0x7f82a0106250 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829fe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0110ea0 con 0x7f82a01051c0 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106bf0 0x7f82a019caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:49.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106bf0 0x7f82a019caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40504/0 (socket says 172.21.15.97:40504) 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829f400640 1 -- 172.21.15.97:0/124251150 learned_addr learned my addr 172.21.15.97:0/124251150 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829ea00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a01051c0 0x7f82a019b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.056+0000 7f829e000640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82a0106250 0x7f82a019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f829f400640 1 -- 172.21.15.97:0/124251150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a01051c0 msgr2=0x7f82a019b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f829f400640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a01051c0 0x7f82a019b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f829f400640 1 -- 172.21.15.97:0/124251150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82a0106250 msgr2=0x7f82a019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f829f400640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82a0106250 0x7f82a019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f829f400640 1 -- 172.21.15.97:0/124251150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a019d280 con 0x7f82a0106bf0 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f829e000640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82a0106250 0x7f82a019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f829f400640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106bf0 0x7f82a019caf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f829800cce0 tx=0x7f829800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f8297400640 1 -- 172.21.15.97:0/124251150 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829800f450 con 0x7f82a0106bf0 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f8297400640 1 -- 172.21.15.97:0/124251150 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f829800fa80 con 0x7f82a0106bf0 2024-07-26T07:48:49.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f8297400640 1 -- 172.21.15.97:0/124251150 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298018ad0 con 0x7f82a0106bf0 2024-07-26T07:48:49.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.057+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a019d570 con 0x7f82a0106bf0 2024-07-26T07:48:49.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.058+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a01a3b30 con 0x7f82a0106bf0 2024-07-26T07:48:49.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.059+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a010c380 con 0x7f82a0106bf0 2024-07-26T07:48:49.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.061+0000 7f8297400640 1 -- 172.21.15.97:0/124251150 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f829800f5f0 con 0x7f82a0106bf0 2024-07-26T07:48:49.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.063+0000 7f8297400640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f82800b3cd0 0x7f82800b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:49.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.063+0000 7f8297400640 1 -- 172.21.15.97:0/124251150 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f82980e46c0 con 0x7f82a0106bf0 2024-07-26T07:48:49.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.063+0000 7f829ea00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f82800b3cd0 0x7f82800b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:49.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.064+0000 7f829ea00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f82800b3cd0 0x7f82800b61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8290005e10 tx=0x7f8290005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:49.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.065+0000 7f8297400640 1 -- 172.21.15.97:0/124251150 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8298014030 con 0x7f82a0106bf0 2024-07-26T07:48:49.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.335+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f82a002e0c0 con 0x7f82a0106bf0 2024-07-26T07:48:49.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.344+0000 7f8297400640 1 -- 172.21.15.97:0/124251150 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v34) ==== 170+0+59 (secure 0 0 0) 0x7f82980abc40 con 0x7f82a0106bf0 2024-07-26T07:48:49.346 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-07-26T07:48:49.346 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDhVKNmt/glFBAAAGumBGhxLtbD2VhLrQIHgw== 2024-07-26T07:48:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.347+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f82800b3cd0 msgr2=0x7f82800b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.347+0000 7f829fe00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f82800b3cd0 0x7f82800b61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8290005e10 tx=0x7f8290005da0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.347+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106bf0 msgr2=0x7f82a019caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.347+0000 7f829fe00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106bf0 0x7f82a019caf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f829800cce0 tx=0x7f829800a620 comp rx=0 tx=0).stop 2024-07-26T07:48:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.348+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 shutdown_connections 2024-07-26T07:48:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.348+0000 7f829fe00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f82800b3cd0 0x7f82800b61e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.348+0000 7f829fe00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82a0106bf0 0x7f82a019caf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.348+0000 7f829fe00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82a0106250 0x7f82a019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.348+0000 7f829fe00640 1 --2- 172.21.15.97:0/124251150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a01051c0 0x7f82a019b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.348+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 >> 172.21.15.97:0/124251150 conn(0x7f82a0100b10 msgr2=0x7f82a0109550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.348+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 shutdown_connections 2024-07-26T07:48:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:49.348+0000 7f829fe00640 1 -- 172.21.15.97:0/124251150 wait complete. 2024-07-26T07:48:50.149 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:48:50.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-07-26T07:48:50.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-26T07:48:50.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph config log 1 --format=json 2024-07-26T07:48:50.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:50 smithi097 ceph-mon[32231]: pgmap v419: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T07:48:50.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:50 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/124251150' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T07:48:50.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:50 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/124251150' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T07:48:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:50 smithi115 ceph-mon[35364]: pgmap v419: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T07:48:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:50 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/124251150' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T07:48:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:50 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/124251150' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T07:48:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:50 smithi181 ceph-mon[35869]: pgmap v419: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T07:48:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:50 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/124251150' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T07:48:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:50 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/124251150' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T07:48:50.807 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:52.366 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:52 smithi097 ceph-mon[32231]: pgmap v420: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:48:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:52 smithi115 ceph-mon[35364]: pgmap v420: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:48:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:52 smithi181 ceph-mon[35869]: pgmap v420: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:48:52.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 -- 172.21.15.97:0/4172310383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7be81020b0 msgr2=0x7f7be81024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 --2- 172.21.15.97:0/4172310383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7be81020b0 0x7f7be81024c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009900 tx=0x7f7bd802a480 comp rx=0 tx=0).stop 2024-07-26T07:48:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 -- 172.21.15.97:0/4172310383 shutdown_connections 2024-07-26T07:48:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 --2- 172.21.15.97:0/4172310383 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7be810fc60 0x7f7be8112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 --2- 172.21.15.97:0/4172310383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be8102a00 0x7f7be810f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 --2- 172.21.15.97:0/4172310383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7be81020b0 0x7f7be81024c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 -- 172.21.15.97:0/4172310383 >> 172.21.15.97:0/4172310383 conn(0x7f7be8079180 msgr2=0x7f7be8079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 -- 172.21.15.97:0/4172310383 shutdown_connections 2024-07-26T07:48:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.921+0000 7f7beec00640 1 -- 172.21.15.97:0/4172310383 wait complete. 2024-07-26T07:48:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.922+0000 7f7beec00640 1 Processor -- start 2024-07-26T07:48:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.922+0000 7f7beec00640 1 -- start start 2024-07-26T07:48:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7beec00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be81020b0 0x7f7be819c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7beec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7be8102a00 0x7f7be819c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7beec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7be810fc60 0x7f7be81a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7beec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7be8115550 con 0x7f7be810fc60 2024-07-26T07:48:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7beec00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7be81153d0 con 0x7f7be81020b0 2024-07-26T07:48:52.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7beec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7be81156d0 con 0x7f7be8102a00 2024-07-26T07:48:52.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7becc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7be810fc60 0x7f7be81a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:52.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7becc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7be810fc60 0x7f7be81a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40522/0 (socket says 172.21.15.97:40522) 2024-07-26T07:48:52.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7becc00640 1 -- 172.21.15.97:0/4246823974 learned_addr learned my addr 172.21.15.97:0/4246823974 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:52.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7be7400640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7be8102a00 0x7f7be819c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:52.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.923+0000 7f7be7e00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be81020b0 0x7f7be819c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:52.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7becc00640 1 -- 172.21.15.97:0/4246823974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7be8102a00 msgr2=0x7f7be819c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:52.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7becc00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7be8102a00 0x7f7be819c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:52.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7becc00640 1 -- 172.21.15.97:0/4246823974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be81020b0 msgr2=0x7f7be819c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:52.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7becc00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be81020b0 0x7f7be819c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:52.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7becc00640 1 -- 172.21.15.97:0/4246823974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be81a1f90 con 0x7f7be810fc60 2024-07-26T07:48:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7becc00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7be810fc60 0x7f7be81a1860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7be000e970 tx=0x7f7be000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7be4c00640 1 -- 172.21.15.97:0/4246823974 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be000ccc0 con 0x7f7be810fc60 2024-07-26T07:48:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be81a2280 con 0x7f7be810fc60 2024-07-26T07:48:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7be4c00640 1 -- 172.21.15.97:0/4246823974 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7be000ce60 con 0x7f7be810fc60 2024-07-26T07:48:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7be4c00640 1 -- 172.21.15.97:0/4246823974 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be0010430 con 0x7f7be810fc60 2024-07-26T07:48:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.924+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be81a89e0 con 0x7f7be810fc60 2024-07-26T07:48:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.925+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be806bbd0 con 0x7f7be810fc60 2024-07-26T07:48:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.930+0000 7f7be4c00640 1 -- 172.21.15.97:0/4246823974 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f7be0004ef0 con 0x7f7be810fc60 2024-07-26T07:48:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.931+0000 7f7be4c00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7bc80b3ce0 0x7f7bc80b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.931+0000 7f7be4c00640 1 -- 172.21.15.97:0/4246823974 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f7be00e4450 con 0x7f7be810fc60 2024-07-26T07:48:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.932+0000 7f7be7e00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7bc80b3ce0 0x7f7bc80b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:52.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.932+0000 7f7be7e00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7bc80b3ce0 0x7f7bc80b61f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8000c00 tx=0x7f7bd8009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:52.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:52.933+0000 7f7be4c00640 1 -- 172.21.15.97:0/4246823974 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7be0055600 con 0x7f7be810fc60 2024-07-26T07:48:53.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.134+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f7be81024c0 con 0x7f7be810fc60 2024-07-26T07:48:53.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.135+0000 7f7be4c00640 1 -- 172.21.15.97:0/4246823974 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v52) ==== 86+0+133 (secure 0 0 0) 0x7f7be81024c0 con 0x7f7be810fc60 2024-07-26T07:48:53.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:48:53.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.137+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7bc80b3ce0 msgr2=0x7f7bc80b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:53.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.137+0000 7f7beec00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7bc80b3ce0 0x7f7bc80b61f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8000c00 tx=0x7f7bd8009680 comp rx=0 tx=0).stop 2024-07-26T07:48:53.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.137+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7be810fc60 msgr2=0x7f7be81a1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.137+0000 7f7beec00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7be810fc60 0x7f7be81a1860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7be000e970 tx=0x7f7be000ed70 comp rx=0 tx=0).stop 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.137+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 shutdown_connections 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.138+0000 7f7beec00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7bc80b3ce0 0x7f7bc80b61f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.138+0000 7f7beec00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7be810fc60 0x7f7be81a1860 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.138+0000 7f7beec00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7be8102a00 0x7f7be819c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.138+0000 7f7beec00640 1 --2- 172.21.15.97:0/4246823974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7be81020b0 0x7f7be819c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.138+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 >> 172.21.15.97:0/4246823974 conn(0x7f7be8079180 msgr2=0x7f7be8111230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.138+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 shutdown_connections 2024-07-26T07:48:53.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:53.138+0000 7f7beec00640 1 -- 172.21.15.97:0/4246823974 wait complete. 2024-07-26T07:48:53.900 INFO:teuthology.orchestra.run.smithi097.stdout:[{"version":52,"timestamp":"2024-07-26T07:47:28.330155+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-07-26T07:48:53.901 INFO:tasks.ceph_manager:config epoch is 52 2024-07-26T07:48:53.901 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-26T07:48:53.901 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-07-26T07:48:53.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph mgr dump --format=json 2024-07-26T07:48:54.368 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:54 smithi115 ceph-mon[35364]: pgmap v421: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T07:48:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:54 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4246823974' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T07:48:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:54 smithi181 ceph-mon[35869]: pgmap v421: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T07:48:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:54 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4246823974' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T07:48:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:54 smithi097 ceph-mon[32231]: pgmap v421: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T07:48:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:54 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4246823974' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T07:48:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:56 smithi097 ceph-mon[32231]: pgmap v422: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:48:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:56 smithi115 ceph-mon[35364]: pgmap v422: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:48:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:56 smithi181 ceph-mon[35869]: pgmap v422: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:48:56.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.490+0000 7f6458a00640 1 -- 172.21.15.97:0/318164974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 msgr2=0x7f64541120a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:56.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.490+0000 7f6458a00640 1 --2- 172.21.15.97:0/318164974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 0x7f64541120a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f644c009ca0 tx=0x7f644c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.490+0000 7f6458a00640 1 -- 172.21.15.97:0/318164974 shutdown_connections 2024-07-26T07:48:56.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.490+0000 7f6458a00640 1 --2- 172.21.15.97:0/318164974 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 0x7f64541120a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.490+0000 7f6458a00640 1 --2- 172.21.15.97:0/318164974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f645410af20 0x7f645410b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.490+0000 7f6458a00640 1 --2- 172.21.15.97:0/318164974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64541055e0 0x7f64541059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.490+0000 7f6458a00640 1 -- 172.21.15.97:0/318164974 >> 172.21.15.97:0/318164974 conn(0x7f64541010b0 msgr2=0x7f6454103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:56.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.491+0000 7f6458a00640 1 -- 172.21.15.97:0/318164974 shutdown_connections 2024-07-26T07:48:56.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.491+0000 7f6458a00640 1 -- 172.21.15.97:0/318164974 wait complete. 2024-07-26T07:48:56.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.492+0000 7f6458a00640 1 Processor -- start 2024-07-26T07:48:56.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.492+0000 7f6458a00640 1 -- start start 2024-07-26T07:48:56.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.492+0000 7f6458a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64541055e0 0x7f645419be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:56.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.492+0000 7f6458a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f645410af20 0x7f645419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.492+0000 7f6458a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 0x7f64541a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.492+0000 7f6458a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6454115420 con 0x7f645410b870 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.492+0000 7f6458a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f64541152a0 con 0x7f645410af20 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.492+0000 7f6458a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f64541155a0 con 0x7f64541055e0 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6453200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 0x7f64541a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6453200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 0x7f64541a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36768/0 (socket says 172.21.15.97:36768) 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6453200640 1 -- 172.21.15.97:0/1959402226 learned_addr learned my addr 172.21.15.97:0/1959402226 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f644be00640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f645410af20 0x7f645419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6452800640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64541055e0 0x7f645419be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6453200640 1 -- 172.21.15.97:0/1959402226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64541055e0 msgr2=0x7f645419be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:56.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6453200640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64541055e0 0x7f645419be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6453200640 1 -- 172.21.15.97:0/1959402226 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f645410af20 msgr2=0x7f645419c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:56.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6453200640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f645410af20 0x7f645419c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.493+0000 7f6453200640 1 -- 172.21.15.97:0/1959402226 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64541a1d20 con 0x7f645410b870 2024-07-26T07:48:56.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.494+0000 7f6453200640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 0x7f64541a16b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f644c005cc0 tx=0x7f644c032c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:56.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.494+0000 7f644b400640 1 -- 172.21.15.97:0/1959402226 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c004040 con 0x7f645410b870 2024-07-26T07:48:56.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.494+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64541a1fb0 con 0x7f645410b870 2024-07-26T07:48:56.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.494+0000 7f644b400640 1 -- 172.21.15.97:0/1959402226 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f644c004670 con 0x7f645410b870 2024-07-26T07:48:56.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.494+0000 7f644b400640 1 -- 172.21.15.97:0/1959402226 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c030cc0 con 0x7f645410b870 2024-07-26T07:48:56.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.494+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64541a24f0 con 0x7f645410b870 2024-07-26T07:48:56.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.495+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64541059f0 con 0x7f645410b870 2024-07-26T07:48:56.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.499+0000 7f644b400640 1 -- 172.21.15.97:0/1959402226 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f644c0041e0 con 0x7f645410b870 2024-07-26T07:48:56.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.501+0000 7f644b400640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f64380b3c90 0x7f64380b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.501+0000 7f644b400640 1 -- 172.21.15.97:0/1959402226 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f644c105590 con 0x7f645410b870 2024-07-26T07:48:56.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.501+0000 7f6452800640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f64380b3c90 0x7f64380b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:56.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.502+0000 7f6452800640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f64380b3c90 0x7f64380b61a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6444005e10 tx=0x7f6444005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.503+0000 7f644b400640 1 -- 172.21.15.97:0/1959402226 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f644c0ccb10 con 0x7f645410b870 2024-07-26T07:48:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.749+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f645419cfa0 con 0x7f645410b870 2024-07-26T07:48:56.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.765+0000 7f644b400640 1 -- 172.21.15.97:0/1959402226 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+291015 (secure 0 0 0) 0x7f645419cfa0 con 0x7f645410b870 2024-07-26T07:48:56.772 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:48:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f64380b3c90 msgr2=0x7f64380b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f64380b3c90 0x7f64380b61a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6444005e10 tx=0x7f6444005da0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 msgr2=0x7f64541a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 0x7f64541a16b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f644c005cc0 tx=0x7f644c032c40 comp rx=0 tx=0).stop 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 shutdown_connections 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f64380b3c90 0x7f64380b61a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f645410b870 0x7f64541a16b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f645410af20 0x7f645419c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 --2- 172.21.15.97:0/1959402226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64541055e0 0x7f645419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.771+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 >> 172.21.15.97:0/1959402226 conn(0x7f64541010b0 msgr2=0x7f64541094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.772+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 shutdown_connections 2024-07-26T07:48:56.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:56.772+0000 7f6458a00640 1 -- 172.21.15.97:0/1959402226 wait complete. 2024-07-26T07:48:57.125 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:57 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1959402226' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T07:48:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:57 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1959402226' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T07:48:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:57 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1959402226' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T07:48:57.529 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":2663280026},{"type":"v1","addr":"172.21.15.97:6801","nonce":2663280026}]},"active_addr":"172.21.15.97:6801/2663280026","active_change":"2024-07-26T07:37:17.917618+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":34106,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.97:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":455400142}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":84795799}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":391803286}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":1306606673}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":831939171}]}]} 2024-07-26T07:48:57.535 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-07-26T07:48:57.536 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-07-26T07:48:57.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:48:57.983 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:48:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:58 smithi115 ceph-mon[35364]: pgmap v423: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:48:58.475 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:58 smithi097 ceph-mon[32231]: pgmap v423: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:48:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:58 smithi181 ceph-mon[35869]: pgmap v423: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:48:59.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:48:59 smithi097 ceph-mon[32231]: pgmap v424: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:48:59.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:48:59 smithi115 ceph-mon[35364]: pgmap v424: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:48:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:48:59 smithi181 ceph-mon[35869]: pgmap v424: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:48:59.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.936+0000 7f9158a00640 1 -- 172.21.15.97:0/1470103374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9154108f00 msgr2=0x7f9154109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:59.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.936+0000 7f9158a00640 1 --2- 172.21.15.97:0/1470103374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9154108f00 0x7f9154109310 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9140009900 tx=0x7f914002a410 comp rx=0 tx=0).stop 2024-07-26T07:48:59.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.937+0000 7f9158a00640 1 -- 172.21.15.97:0/1470103374 shutdown_connections 2024-07-26T07:48:59.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.937+0000 7f9158a00640 1 --2- 172.21.15.97:0/1470103374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9154109850 0x7f9154110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:59.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.937+0000 7f9158a00640 1 --2- 172.21.15.97:0/1470103374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9154108f00 0x7f9154109310 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:59.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.937+0000 7f9158a00640 1 --2- 172.21.15.97:0/1470103374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91541035c0 0x7f91541039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:59.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.937+0000 7f9158a00640 1 -- 172.21.15.97:0/1470103374 >> 172.21.15.97:0/1470103374 conn(0x7f91540ff030 msgr2=0x7f9154101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:48:59.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.937+0000 7f9158a00640 1 -- 172.21.15.97:0/1470103374 shutdown_connections 2024-07-26T07:48:59.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.937+0000 7f9158a00640 1 -- 172.21.15.97:0/1470103374 wait complete. 2024-07-26T07:48:59.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.938+0000 7f9158a00640 1 Processor -- start 2024-07-26T07:48:59.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.938+0000 7f9158a00640 1 -- start start 2024-07-26T07:48:59.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9158a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91541035c0 0x7f9154199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:59.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9158a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9154108f00 0x7f915419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:59.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9158a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9154109850 0x7f915419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:59.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9158a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f91541133e0 con 0x7f91541035c0 2024-07-26T07:48:59.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9158a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9154113260 con 0x7f9154108f00 2024-07-26T07:48:59.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9158a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9154113560 con 0x7f9154109850 2024-07-26T07:48:59.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9151e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91541035c0 0x7f9154199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:59.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9151e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91541035c0 0x7f9154199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36780/0 (socket says 172.21.15.97:36780) 2024-07-26T07:48:59.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9151e00640 1 -- 172.21.15.97:0/653821737 learned_addr learned my addr 172.21.15.97:0/653821737 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:48:59.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9152800640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9154109850 0x7f915419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:59.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.939+0000 7f9151400640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9154108f00 0x7f915419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:59.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f9151e00640 1 -- 172.21.15.97:0/653821737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9154109850 msgr2=0x7f915419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:59.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f9151e00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9154109850 0x7f915419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:59.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f9151e00640 1 -- 172.21.15.97:0/653821737 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9154108f00 msgr2=0x7f915419a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:48:59.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f9151e00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9154108f00 0x7f915419a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:48:59.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f9151e00640 1 -- 172.21.15.97:0/653821737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915419fd20 con 0x7f91541035c0 2024-07-26T07:48:59.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f9151e00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91541035c0 0x7f9154199e70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f914400b4a0 tx=0x7f914400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:59.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f914aa00640 1 -- 172.21.15.97:0/653821737 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144014070 con 0x7f91541035c0 2024-07-26T07:48:59.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f914aa00640 1 -- 172.21.15.97:0/653821737 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9144010aa0 con 0x7f91541035c0 2024-07-26T07:48:59.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f914aa00640 1 -- 172.21.15.97:0/653821737 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144017b70 con 0x7f91541035c0 2024-07-26T07:48:59.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.940+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91541a0010 con 0x7f91541035c0 2024-07-26T07:48:59.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.941+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91541a6830 con 0x7f91541035c0 2024-07-26T07:48:59.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.942+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91541039d0 con 0x7f91541035c0 2024-07-26T07:48:59.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.947+0000 7f914aa00640 1 -- 172.21.15.97:0/653821737 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f9144017070 con 0x7f91541035c0 2024-07-26T07:48:59.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.947+0000 7f914aa00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f91380b3c90 0x7f91380b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:48:59.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.947+0000 7f914aa00640 1 -- 172.21.15.97:0/653821737 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f91440e3560 con 0x7f91541035c0 2024-07-26T07:48:59.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.948+0000 7f9151400640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f91380b3c90 0x7f91380b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:48:59.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.949+0000 7f9151400640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f91380b3c90 0x7f91380b61a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91400057c0 tx=0x7f9140000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:48:59.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:48:59.950+0000 7f914aa00640 1 -- 172.21.15.97:0/653821737 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f91440aaae0 con 0x7f91541035c0 2024-07-26T07:49:00.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.155+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f915419afa0 con 0x7f91541035c0 2024-07-26T07:49:00.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.159+0000 7f914aa00640 1 -- 172.21.15.97:0/653821737 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22701 (secure 0 0 0) 0x7f915419afa0 con 0x7f91541035c0 2024-07-26T07:49:00.162 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:49:00.162 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":84,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:48:19.900531+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T07:47:40.641206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T07:47:44.357412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.162+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f91380b3c90 msgr2=0x7f91380b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.162+0000 7f9158a00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f91380b3c90 0x7f91380b61a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91400057c0 tx=0x7f9140000bc0 comp rx=0 tx=0).stop 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.162+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91541035c0 msgr2=0x7f9154199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.162+0000 7f9158a00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91541035c0 0x7f9154199e70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f914400b4a0 tx=0x7f914400b960 comp rx=0 tx=0).stop 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.163+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 shutdown_connections 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.163+0000 7f9158a00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f91380b3c90 0x7f91380b61a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.163+0000 7f9158a00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9154109850 0x7f915419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.163+0000 7f9158a00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9154108f00 0x7f915419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.163+0000 7f9158a00640 1 --2- 172.21.15.97:0/653821737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91541035c0 0x7f9154199e70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:00.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.163+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 >> 172.21.15.97:0/653821737 conn(0x7f91540ff030 msgr2=0x7f9154101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:00.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.163+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 shutdown_connections 2024-07-26T07:49:00.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:00.163+0000 7f9158a00640 1 -- 172.21.15.97:0/653821737 wait complete. 2024-07-26T07:49:00.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:00 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/653821737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:49:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:00 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/653821737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:49:00.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:00 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/653821737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:49:00.880 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-07-26T07:49:00.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:49:01.321 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:01.469 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:01 smithi097 ceph-mon[32231]: pgmap v425: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:01.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:01 smithi115 ceph-mon[35364]: pgmap v425: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:01 smithi181 ceph-mon[35869]: pgmap v425: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:03.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 -- 172.21.15.97:0/2119793594 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 msgr2=0x7f724410f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:03.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 --2- 172.21.15.97:0/2119793594 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 0x7f724410f3e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f722c009900 tx=0x7f722c02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 -- 172.21.15.97:0/2119793594 shutdown_connections 2024-07-26T07:49:03.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 --2- 172.21.15.97:0/2119793594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f724410f920 0x7f7244111d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 --2- 172.21.15.97:0/2119793594 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 0x7f724410f3e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 --2- 172.21.15.97:0/2119793594 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72440751e0 0x7f72440755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 -- 172.21.15.97:0/2119793594 >> 172.21.15.97:0/2119793594 conn(0x7f72440fee10 msgr2=0x7f7244101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:03.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 -- 172.21.15.97:0/2119793594 shutdown_connections 2024-07-26T07:49:03.431 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:03 smithi097 ceph-mon[32231]: pgmap v426: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.429+0000 7f724a600640 1 -- 172.21.15.97:0/2119793594 wait complete. 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.430+0000 7f724a600640 1 Processor -- start 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.430+0000 7f724a600640 1 -- start start 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f724a600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72440751e0 0x7f724419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f724a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 0x7f724419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f724a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f724410f920 0x7f72441a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f724a600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7244115410 con 0x7f7244075b30 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f724a600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7244115290 con 0x7f72440751e0 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f724a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7244115590 con 0x7f724410f920 2024-07-26T07:49:03.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f7243e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 0x7f724419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f7243e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 0x7f724419c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36802/0 (socket says 172.21.15.97:36802) 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.431+0000 7f7243e00640 1 -- 172.21.15.97:0/150596181 learned_addr learned my addr 172.21.15.97:0/150596181 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7249200640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72440751e0 0x7f724419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7249c00640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f724410f920 0x7f72441a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7243e00640 1 -- 172.21.15.97:0/150596181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f724410f920 msgr2=0x7f72441a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7243e00640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f724410f920 0x7f72441a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7243e00640 1 -- 172.21.15.97:0/150596181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72440751e0 msgr2=0x7f724419bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7243e00640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72440751e0 0x7f724419bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7243e00640 1 -- 172.21.15.97:0/150596181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72441a1eb0 con 0x7f7244075b30 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7249200640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72440751e0 0x7f724419bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:49:03.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7243e00640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 0x7f724419c4e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f722c02a910 tx=0x7f722c003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:03.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.432+0000 7f7241600640 1 -- 172.21.15.97:0/150596181 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c03f070 con 0x7f7244075b30 2024-07-26T07:49:03.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.433+0000 7f7241600640 1 -- 172.21.15.97:0/150596181 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f722c03bdc0 con 0x7f7244075b30 2024-07-26T07:49:03.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.433+0000 7f7241600640 1 -- 172.21.15.97:0/150596181 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c035030 con 0x7f7244075b30 2024-07-26T07:49:03.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.433+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72441a2140 con 0x7f7244075b30 2024-07-26T07:49:03.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.433+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72441a8900 con 0x7f7244075b30 2024-07-26T07:49:03.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.434+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72440755f0 con 0x7f7244075b30 2024-07-26T07:49:03.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.436+0000 7f7241600640 1 -- 172.21.15.97:0/150596181 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f722c042070 con 0x7f7244075b30 2024-07-26T07:49:03.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.438+0000 7f7241600640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f72280b3ce0 0x7f72280b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:03.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.438+0000 7f7241600640 1 -- 172.21.15.97:0/150596181 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f722c104b40 con 0x7f7244075b30 2024-07-26T07:49:03.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.438+0000 7f7249200640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f72280b3ce0 0x7f72280b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:03.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.439+0000 7f7249200640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f72280b3ce0 0x7f72280b61f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72340058c0 tx=0x7f7234005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:03.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.440+0000 7f7241600640 1 -- 172.21.15.97:0/150596181 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f722c0cc0a0 con 0x7f7244075b30 2024-07-26T07:49:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:03 smithi115 ceph-mon[35364]: pgmap v426: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:49:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:03 smithi181 ceph-mon[35869]: pgmap v426: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:49:03.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.641+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72441a8b00 con 0x7f7244075b30 2024-07-26T07:49:03.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.644+0000 7f7241600640 1 -- 172.21.15.97:0/150596181 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22701 (secure 0 0 0) 0x7f722c0cb9a0 con 0x7f7244075b30 2024-07-26T07:49:03.646 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:49:03.646 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":84,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:48:19.900531+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T07:47:40.641206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T07:47:44.357412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:49:03.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.646+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f72280b3ce0 msgr2=0x7f72280b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:03.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.646+0000 7f724a600640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f72280b3ce0 0x7f72280b61f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72340058c0 tx=0x7f7234005ec0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.646+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 msgr2=0x7f724419c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:03.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.646+0000 7f724a600640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 0x7f724419c4e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f722c02a910 tx=0x7f722c003d60 comp rx=0 tx=0).stop 2024-07-26T07:49:03.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.647+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 shutdown_connections 2024-07-26T07:49:03.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.647+0000 7f724a600640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f72280b3ce0 0x7f72280b61f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.647+0000 7f724a600640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f724410f920 0x7f72441a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.647+0000 7f724a600640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7244075b30 0x7f724419c4e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.647+0000 7f724a600640 1 --2- 172.21.15.97:0/150596181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72440751e0 0x7f724419bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:03.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.647+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 >> 172.21.15.97:0/150596181 conn(0x7f72440fee10 msgr2=0x7f7244109500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:03.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.647+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 shutdown_connections 2024-07-26T07:49:03.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:03.647+0000 7f724a600640 1 -- 172.21.15.97:0/150596181 wait complete. 2024-07-26T07:49:04.434 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:04 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/150596181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:49:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:04 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/150596181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:49:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:04 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/150596181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:49:04.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-07-26T07:49:04.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-07-26T07:49:04.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-07-26T07:49:04.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-07-26T07:49:04.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-07-26T07:49:04.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-07-26T07:49:04.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-07-26T07:49:04.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-07-26T07:49:04.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-07-26T07:49:04.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-07-26T07:49:04.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-07-26T07:49:04.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-07-26T07:49:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:05 smithi115 ceph-mon[35364]: pgmap v427: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:49:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:05 smithi181 ceph-mon[35869]: pgmap v427: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:49:05.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:05 smithi097 ceph-mon[32231]: pgmap v427: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:49:05.528 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:05.552 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:05.568 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:05.585 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:05.585 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:06.808 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:07.058 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:07.231 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:07 smithi097 ceph-mon[32231]: pgmap v428: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:49:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:07 smithi181 ceph-mon[35869]: pgmap v428: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:49:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:07 smithi115 ceph-mon[35364]: pgmap v428: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:49:08.372 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:08.373 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:09.438 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:09 smithi097 ceph-mon[32231]: pgmap v429: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:09 smithi115 ceph-mon[35364]: pgmap v429: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:09 smithi181 ceph-mon[35869]: pgmap v429: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:10.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.470+0000 7f5d34a00640 1 -- 172.21.15.97:0/1922010967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 msgr2=0x7f5d3010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:10.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.470+0000 7f5d34a00640 1 --2- 172.21.15.97:0/1922010967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 0x7f5d3010f740 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009900 tx=0x7f5d1802a410 comp rx=0 tx=0).stop 2024-07-26T07:49:10.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.471+0000 7f5d34a00640 1 -- 172.21.15.97:0/1922010967 shutdown_connections 2024-07-26T07:49:10.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.471+0000 7f5d34a00640 1 --2- 172.21.15.97:0/1922010967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d3010fc80 0x7f5d301164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:10.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.471+0000 7f5d34a00640 1 --2- 172.21.15.97:0/1922010967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 0x7f5d3010f740 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:10.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.471+0000 7f5d34a00640 1 --2- 172.21.15.97:0/1922010967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d301099f0 0x7f5d30109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:10.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.471+0000 7f5d34a00640 1 -- 172.21.15.97:0/1922010967 >> 172.21.15.97:0/1922010967 conn(0x7f5d30078ff0 msgr2=0x7f5d30107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:10.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.471+0000 7f5d34a00640 1 -- 172.21.15.97:0/1922010967 shutdown_connections 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.471+0000 7f5d34a00640 1 -- 172.21.15.97:0/1922010967 wait complete. 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.472+0000 7f5d34a00640 1 Processor -- start 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.472+0000 7f5d34a00640 1 -- start start 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d34a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d301099f0 0x7f5d301a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d34a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 0x7f5d301a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d34a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d3010fc80 0x7f5d301a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d34a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5d301197f0 con 0x7f5d3010f330 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d34a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f5d30119670 con 0x7f5d3010fc80 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d34a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5d30119970 con 0x7f5d301099f0 2024-07-26T07:49:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d2ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 0x7f5d301a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d2ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 0x7f5d301a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58244/0 (socket says 172.21.15.97:58244) 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d2ea00640 1 -- 172.21.15.97:0/1488603725 learned_addr learned my addr 172.21.15.97:0/1488603725 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d2fe00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d3010fc80 0x7f5d301a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.473+0000 7f5d2f400640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d301099f0 0x7f5d301a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d2ea00640 1 -- 172.21.15.97:0/1488603725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d301099f0 msgr2=0x7f5d301a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d2ea00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d301099f0 0x7f5d301a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d2ea00640 1 -- 172.21.15.97:0/1488603725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d3010fc80 msgr2=0x7f5d301a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d2ea00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d3010fc80 0x7f5d301a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d2ea00640 1 -- 172.21.15.97:0/1488603725 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d301a61b0 con 0x7f5d3010f330 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d2ea00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 0x7f5d301a07e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5d18005940 tx=0x7f5d180040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:10.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1803f030 con 0x7f5d3010f330 2024-07-26T07:49:10.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d18003a80 con 0x7f5d3010f330 2024-07-26T07:49:10.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d301a6440 con 0x7f5d3010f330 2024-07-26T07:49:10.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.474+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18035070 con 0x7f5d3010f330 2024-07-26T07:49:10.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.475+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d301acc00 con 0x7f5d3010f330 2024-07-26T07:49:10.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.475+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d30063540 con 0x7f5d3010f330 2024-07-26T07:49:10.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.478+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f5d18042030 con 0x7f5d3010f330 2024-07-26T07:49:10.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.479+0000 7f5d27e00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5d140b3ce0 0x7f5d140b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.480+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7f5d181047d0 con 0x7f5d3010f330 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.480+0000 7f5d2f400640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5d140b3ce0 0x7f5d140b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.480+0000 7f5d27e00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] conn(0x7f5d140bd9e0 0x7f5d140bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.480+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 --> [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d18040e70 con 0x7f5d140bd9e0 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.480+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f5d18104b80 con 0x7f5d3010f330 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.481+0000 7f5d2f400640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5d140b3ce0 0x7f5d140b61f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005fd0 tx=0x7f5d20005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.481+0000 7f5d2fe00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] conn(0x7f5d140bd9e0 0x7f5d140bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.482+0000 7f5d2fe00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] conn(0x7f5d140bd9e0 0x7f5d140bfed0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.484+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 <== osd.4 v2:172.21.15.115:6808/1719675205 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5d18040e70 con 0x7f5d140bd9e0 2024-07-26T07:49:10.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.499+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 --> [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5d30063970 con 0x7f5d140bd9e0 2024-07-26T07:49:10.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.500+0000 7f5d27e00640 1 -- 172.21.15.97:0/1488603725 <== osd.4 v2:172.21.15.115:6808/1719675205 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5d30063970 con 0x7f5d140bd9e0 2024-07-26T07:49:10.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 >> [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] conn(0x7f5d140bd9e0 msgr2=0x7f5d140bfed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:10.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d34a00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] conn(0x7f5d140bd9e0 0x7f5d140bfed0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:10.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5d140b3ce0 msgr2=0x7f5d140b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:10.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d34a00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f5d140b3ce0 0x7f5d140b61f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005fd0 tx=0x7f5d20005850 comp rx=0 tx=0).stop 2024-07-26T07:49:10.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 msgr2=0x7f5d301a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:10.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d34a00640 1 --2- 172.21.15.97:0/1488603725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d3010f330 0x7f5d301a07e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5d18005940 tx=0x7f5d180040e0 comp rx=0 tx=0).stop 2024-07-26T07:49:10.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d2fe00640 1 -- 172.21.15.97:0/1488603725 reap_dead start 2024-07-26T07:49:10.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 shutdown_connections 2024-07-26T07:49:10.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.501+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 >> 172.21.15.97:0/1488603725 conn(0x7f5d30078ff0 msgr2=0x7f5d30079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:10.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.502+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 shutdown_connections 2024-07-26T07:49:10.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:10.502+0000 7f5d34a00640 1 -- 172.21.15.97:0/1488603725 wait complete. 2024-07-26T07:49:10.892 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:10.954 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:10.955 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:11 smithi097 ceph-mon[32231]: pgmap v430: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:11 smithi115 ceph-mon[35364]: pgmap v430: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:11 smithi181 ceph-mon[35869]: pgmap v430: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:13.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:13 smithi097 ceph-mon[32231]: pgmap v431: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:49:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:13 smithi115 ceph-mon[35364]: pgmap v431: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:49:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:13 smithi181 ceph-mon[35869]: pgmap v431: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:49:14.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 -- 172.21.15.97:0/182532038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7facf810fa40 msgr2=0x7facf8111e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:14.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 --2- 172.21.15.97:0/182532038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7facf810fa40 0x7facf8111e60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7facf0009be0 tx=0x7facf002a750 comp rx=0 tx=0).stop 2024-07-26T07:49:14.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 -- 172.21.15.97:0/182532038 shutdown_connections 2024-07-26T07:49:14.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 --2- 172.21.15.97:0/182532038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7facf810fa40 0x7facf8111e60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:14.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 --2- 172.21.15.97:0/182532038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf81027e0 0x7facf810f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:14.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 --2- 172.21.15.97:0/182532038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8101e90 0x7facf81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:14.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 -- 172.21.15.97:0/182532038 >> 172.21.15.97:0/182532038 conn(0x7facf80fdcd0 msgr2=0x7facf8100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:14.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 -- 172.21.15.97:0/182532038 shutdown_connections 2024-07-26T07:49:14.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.975+0000 7facf7200640 1 -- 172.21.15.97:0/182532038 wait complete. 2024-07-26T07:49:14.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.977+0000 7facf7200640 1 Processor -- start 2024-07-26T07:49:14.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.977+0000 7facf7200640 1 -- start start 2024-07-26T07:49:14.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf7200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8101e90 0x7facf819bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:14.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf7200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7facf81027e0 0x7facf819c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:14.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf810fa40 0x7facf81a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:14.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf7200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7facf8115350 con 0x7facf8101e90 2024-07-26T07:49:14.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf7200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7facf81151d0 con 0x7facf81027e0 2024-07-26T07:49:14.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7facf81154d0 con 0x7facf810fa40 2024-07-26T07:49:14.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8101e90 0x7facf819bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:14.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8101e90 0x7facf819bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58266/0 (socket says 172.21.15.97:58266) 2024-07-26T07:49:14.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf5e00640 1 -- 172.21.15.97:0/1874099111 learned_addr learned my addr 172.21.15.97:0/1874099111 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:14.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.978+0000 7facf6800640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf810fa40 0x7facf81a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:14.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.979+0000 7facf5400640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7facf81027e0 0x7facf819c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:14.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.979+0000 7facf5e00640 1 -- 172.21.15.97:0/1874099111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf810fa40 msgr2=0x7facf81a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:14.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.979+0000 7facf5e00640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf810fa40 0x7facf81a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:14.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.979+0000 7facf5e00640 1 -- 172.21.15.97:0/1874099111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7facf81027e0 msgr2=0x7facf819c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:14.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.979+0000 7facf5e00640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7facf81027e0 0x7facf819c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:14.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.979+0000 7facf5e00640 1 -- 172.21.15.97:0/1874099111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf81a1c90 con 0x7facf8101e90 2024-07-26T07:49:14.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.979+0000 7facf5400640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7facf81027e0 0x7facf819c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:49:14.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.979+0000 7facf5e00640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8101e90 0x7facf819bd20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7face800e9c0 tx=0x7face800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:14.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.980+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face800ccc0 con 0x7facf8101e90 2024-07-26T07:49:14.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.980+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7face80054b0 con 0x7facf8101e90 2024-07-26T07:49:14.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.980+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf81a1f80 con 0x7facf8101e90 2024-07-26T07:49:14.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.980+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8010430 con 0x7facf8101e90 2024-07-26T07:49:14.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.980+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf81a86e0 con 0x7facf8101e90 2024-07-26T07:49:14.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.981+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7facf8111ec0 con 0x7facf8101e90 2024-07-26T07:49:14.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.984+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7face80046c0 con 0x7facf8101e90 2024-07-26T07:49:14.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.985+0000 7faceea00640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7facdc0b3ce0 0x7facdc0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:14.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.986+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7face80e48b0 con 0x7facf8101e90 2024-07-26T07:49:14.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.986+0000 7facf5400640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7facdc0b3ce0 0x7facdc0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:14.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.986+0000 7faceea00640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] conn(0x7facdc0bd9e0 0x7facdc0bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:14.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.986+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 --> [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7face8015e80 con 0x7facdc0bd9e0 2024-07-26T07:49:14.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.986+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7face80e4c60 con 0x7facf8101e90 2024-07-26T07:49:14.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.987+0000 7facf5400640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7facdc0b3ce0 0x7facdc0b61f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7facf81038c0 tx=0x7face4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:14.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.987+0000 7facf6800640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] conn(0x7facdc0bd9e0 0x7facdc0bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:14.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.988+0000 7facf6800640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] conn(0x7facdc0bd9e0 0x7facdc0bfed0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:14.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:14.990+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 <== osd.2 v2:172.21.15.181:6800/1600494363 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7face8015e80 con 0x7facdc0bd9e0 2024-07-26T07:49:15.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.008+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 --> [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7facf806ba10 con 0x7facdc0bd9e0 2024-07-26T07:49:15.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.009+0000 7faceea00640 1 -- 172.21.15.97:0/1874099111 <== osd.2 v2:172.21.15.181:6800/1600494363 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7facf806ba10 con 0x7facdc0bd9e0 2024-07-26T07:49:15.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 >> [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] conn(0x7facdc0bd9e0 msgr2=0x7facdc0bfed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:15.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] conn(0x7facdc0bd9e0 0x7facdc0bfed0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:15.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7facdc0b3ce0 msgr2=0x7facdc0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:15.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7facdc0b3ce0 0x7facdc0b61f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7facf81038c0 tx=0x7face4005f90 comp rx=0 tx=0).stop 2024-07-26T07:49:15.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8101e90 msgr2=0x7facf819bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:15.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 --2- 172.21.15.97:0/1874099111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7facf8101e90 0x7facf819bd20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7face800e9c0 tx=0x7face800ee80 comp rx=0 tx=0).stop 2024-07-26T07:49:15.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf6800640 1 -- 172.21.15.97:0/1874099111 reap_dead start 2024-07-26T07:49:15.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 shutdown_connections 2024-07-26T07:49:15.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 >> 172.21.15.97:0/1874099111 conn(0x7facf80fdcd0 msgr2=0x7facf8100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:15.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 shutdown_connections 2024-07-26T07:49:15.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:15.010+0000 7facf7200640 1 -- 172.21.15.97:0/1874099111 wait complete. 2024-07-26T07:49:15.319 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:15 smithi097 ceph-mon[32231]: pgmap v432: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:49:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:15 smithi115 ceph-mon[35364]: pgmap v432: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:49:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:15 smithi181 ceph-mon[35869]: pgmap v432: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:49:17.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:17 smithi097 ceph-mon[32231]: pgmap v433: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:49:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:17 smithi115 ceph-mon[35364]: pgmap v433: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:49:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:17 smithi181 ceph-mon[35869]: pgmap v433: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:49:19.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 -- 172.21.15.97:0/2657621460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff168101e90 msgr2=0x7ff1681022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:19.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 --2- 172.21.15.97:0/2657621460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff168101e90 0x7ff1681022a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff158009900 tx=0x7ff15802a480 comp rx=0 tx=0).stop 2024-07-26T07:49:19.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 -- 172.21.15.97:0/2657621460 shutdown_connections 2024-07-26T07:49:19.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 --2- 172.21.15.97:0/2657621460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff16810fa40 0x7ff168111e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:19.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 --2- 172.21.15.97:0/2657621460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1681027e0 0x7ff16810f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 --2- 172.21.15.97:0/2657621460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff168101e90 0x7ff1681022a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 -- 172.21.15.97:0/2657621460 >> 172.21.15.97:0/2657621460 conn(0x7ff1680fdcf0 msgr2=0x7ff168100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 -- 172.21.15.97:0/2657621460 shutdown_connections 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.398+0000 7ff167200640 1 -- 172.21.15.97:0/2657621460 wait complete. 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.399+0000 7ff167200640 1 Processor -- start 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.400+0000 7ff167200640 1 -- start start 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.400+0000 7ff167200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff168101e90 0x7ff16819be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.400+0000 7ff167200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1681027e0 0x7ff16819c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.400+0000 7ff167200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff16810fa40 0x7ff1681a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:19.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.400+0000 7ff167200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff168115390 con 0x7ff16810fa40 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.400+0000 7ff167200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff168115210 con 0x7ff168101e90 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.400+0000 7ff167200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff168115510 con 0x7ff1681027e0 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff166800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff16810fa40 0x7ff1681a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff166800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff16810fa40 0x7ff1681a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33664/0 (socket says 172.21.15.97:33664) 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff166800640 1 -- 172.21.15.97:0/620448069 learned_addr learned my addr 172.21.15.97:0/620448069 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff165e00640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff168101e90 0x7ff16819be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff165400640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1681027e0 0x7ff16819c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff166800640 1 -- 172.21.15.97:0/620448069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1681027e0 msgr2=0x7ff16819c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff166800640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1681027e0 0x7ff16819c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff166800640 1 -- 172.21.15.97:0/620448069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff168101e90 msgr2=0x7ff16819be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:19.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff166800640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff168101e90 0x7ff16819be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.401+0000 7ff166800640 1 -- 172.21.15.97:0/620448069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1681a1d70 con 0x7ff16810fa40 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.402+0000 7ff166800640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff16810fa40 0x7ff1681a1640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff16000eaa0 tx=0x7ff16000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.402+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16000cdb0 con 0x7ff16810fa40 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.402+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1681a2060 con 0x7ff16810fa40 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.402+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff160005020 con 0x7ff16810fa40 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.402+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1600116d0 con 0x7ff16810fa40 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.402+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1681a87c0 con 0x7ff16810fa40 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.403+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff168063540 con 0x7ff16810fa40 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.406+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7ff1600051d0 con 0x7ff16810fa40 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.407+0000 7ff15ea00640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff14c0b3ce0 0x7ff14c0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:19.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.407+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8269+0+0 (secure 0 0 0) 0x7ff1600e43b0 con 0x7ff16810fa40 2024-07-26T07:49:19.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.408+0000 7ff165e00640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff14c0b3ce0 0x7ff14c0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:19.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.408+0000 7ff15ea00640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] conn(0x7ff14c0bd9e0 0x7ff14c0bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:19.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.408+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 --> [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff160014ec0 con 0x7ff14c0bd9e0 2024-07-26T07:49:19.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.408+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7ff1600e4760 con 0x7ff16810fa40 2024-07-26T07:49:19.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.408+0000 7ff165e00640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff14c0b3ce0 0x7ff14c0b61f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff158006010 tx=0x7ff158000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:19.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.408+0000 7ff165400640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] conn(0x7ff14c0bd9e0 0x7ff14c0bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:19.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.410+0000 7ff165400640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] conn(0x7ff14c0bd9e0 0x7ff14c0bfed0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:19.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.412+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 <== osd.11 v2:172.21.15.181:6824/2026606552 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff160014ec0 con 0x7ff14c0bd9e0 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.428+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 --> [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff1680639c0 con 0x7ff14c0bd9e0 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.429+0000 7ff15ea00640 1 -- 172.21.15.97:0/620448069 <== osd.11 v2:172.21.15.181:6824/2026606552 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff1680639c0 con 0x7ff14c0bd9e0 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.429+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 >> [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] conn(0x7ff14c0bd9e0 msgr2=0x7ff14c0bfed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.429+0000 7ff167200640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] conn(0x7ff14c0bd9e0 0x7ff14c0bfed0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.429+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff14c0b3ce0 msgr2=0x7ff14c0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.429+0000 7ff167200640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff14c0b3ce0 0x7ff14c0b61f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff158006010 tx=0x7ff158000bc0 comp rx=0 tx=0).stop 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.429+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff16810fa40 msgr2=0x7ff1681a1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.429+0000 7ff167200640 1 --2- 172.21.15.97:0/620448069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff16810fa40 0x7ff1681a1640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff16000eaa0 tx=0x7ff16000ef60 comp rx=0 tx=0).stop 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.429+0000 7ff166800640 1 -- 172.21.15.97:0/620448069 reap_dead start 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.430+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 shutdown_connections 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.430+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 >> 172.21.15.97:0/620448069 conn(0x7ff1680fdcf0 msgr2=0x7ff168100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.430+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 shutdown_connections 2024-07-26T07:49:19.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:17.430+0000 7ff167200640 1 -- 172.21.15.97:0/620448069 wait complete. 2024-07-26T07:49:19.583 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:19 smithi097 ceph-mon[32231]: pgmap v434: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:19.583 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T07:49:19.583 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]: dispatch 2024-07-26T07:49:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:19 smithi115 ceph-mon[35364]: pgmap v434: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T07:49:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]: dispatch 2024-07-26T07:49:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:19 smithi181 ceph-mon[35869]: pgmap v434: 97 pgs: 97 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T07:49:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]: dispatch 2024-07-26T07:49:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-07-26T07:49:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]': finished 2024-07-26T07:49:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:20 smithi097 ceph-mon[32231]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T07:49:20.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 -- 172.21.15.97:0/548525450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4109fc0 msgr2=0x7fcca410a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:20.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 --2- 172.21.15.97:0/548525450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4109fc0 0x7fcca410a3d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcc94009900 tx=0x7fcc9402a3f0 comp rx=0 tx=0).stop 2024-07-26T07:49:20.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 -- 172.21.15.97:0/548525450 shutdown_connections 2024-07-26T07:49:20.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 --2- 172.21.15.97:0/548525450 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcca4116070 0x7fcca4118580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:20.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 --2- 172.21.15.97:0/548525450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca410a910 0x7fcca4115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:20.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 --2- 172.21.15.97:0/548525450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4109fc0 0x7fcca410a3d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 -- 172.21.15.97:0/548525450 >> 172.21.15.97:0/548525450 conn(0x7fcca40708f0 msgr2=0x7fcca4070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 -- 172.21.15.97:0/548525450 shutdown_connections 2024-07-26T07:49:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.304+0000 7fccab200640 1 -- 172.21.15.97:0/548525450 wait complete. 2024-07-26T07:49:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.306+0000 7fccab200640 1 Processor -- start 2024-07-26T07:49:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.307+0000 7fccab200640 1 -- start start 2024-07-26T07:49:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.307+0000 7fccab200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4109fc0 0x7fcca41a98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.307+0000 7fccab200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcca410a910 0x7fcca41a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.307+0000 7fccab200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca4116070 0x7fcca41b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.307+0000 7fccab200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcca411bc10 con 0x7fcca4109fc0 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.307+0000 7fccab200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcca411ba90 con 0x7fcca410a910 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.307+0000 7fccab200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcca411bd90 con 0x7fcca4116070 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.308+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4109fc0 0x7fcca41a98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.308+0000 7fcca3400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcca410a910 0x7fcca41a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.308+0000 7fcca3400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcca410a910 0x7fcca41a9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:50576/0 (socket says 172.21.15.97:50576) 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.308+0000 7fcca3400640 1 -- 172.21.15.97:0/3487127484 learned_addr learned my addr 172.21.15.97:0/3487127484 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.308+0000 7fcca9200640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca4116070 0x7fcca41b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.309+0000 7fcca3400640 1 -- 172.21.15.97:0/3487127484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca4116070 msgr2=0x7fcca41b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.309+0000 7fcca3400640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca4116070 0x7fcca41b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.309+0000 7fcca3400640 1 -- 172.21.15.97:0/3487127484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4109fc0 msgr2=0x7fcca41a98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.309+0000 7fcca3400640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcca4109fc0 0x7fcca41a98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.309+0000 7fcca3400640 1 -- 172.21.15.97:0/3487127484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca41aa700 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.310+0000 7fcca3400640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcca410a910 0x7fcca41a9e30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc9000d750 tx=0x7fcc9000dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.310+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc90017070 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.311+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc900054b0 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.311+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc90004b90 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.311+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca41b4fd0 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.311+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca41b5430 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.312+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcca40732c0 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.322+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fcc90004d30 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.324+0000 7fcca0c00640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcc880b3ce0 0x7fcc880b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.324+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fcc900e3c70 con 0x7fcca410a910 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.324+0000 7fcca3e00640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcc880b3ce0 0x7fcc880b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.324+0000 7fcca0c00640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] conn(0x7fcc880bdae0 0x7fcc880bffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.324+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 --> [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc90001d30 con 0x7fcc880bdae0 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.324+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fcc900e4020 con 0x7fcca410a910 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.325+0000 7fcca3e00640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcc880b3ce0 0x7fcc880b61f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcca4111940 tx=0x7fcc94005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.335+0000 7fcca9200640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] conn(0x7fcc880bdae0 0x7fcc880bffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.336+0000 7fcca9200640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] conn(0x7fcc880bdae0 0x7fcc880bffd0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.339+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 <== osd.8 v2:172.21.15.181:6816/2477056065 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc90001d30 con 0x7fcc880bdae0 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.354+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 --> [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcca410b010 con 0x7fcc880bdae0 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.355+0000 7fcca0c00640 1 -- 172.21.15.97:0/3487127484 <== osd.8 v2:172.21.15.181:6816/2477056065 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcca410b010 con 0x7fcc880bdae0 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 >> [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] conn(0x7fcc880bdae0 msgr2=0x7fcc880bffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] conn(0x7fcc880bdae0 0x7fcc880bffd0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcc880b3ce0 msgr2=0x7fcc880b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcc880b3ce0 0x7fcc880b61f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcca4111940 tx=0x7fcc94005fd0 comp rx=0 tx=0).stop 2024-07-26T07:49:20.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcca410a910 msgr2=0x7fcca41a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:20.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 --2- 172.21.15.97:0/3487127484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcca410a910 0x7fcca41a9e30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc9000d750 tx=0x7fcc9000dc10 comp rx=0 tx=0).stop 2024-07-26T07:49:20.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fcca9200640 1 -- 172.21.15.97:0/3487127484 reap_dead start 2024-07-26T07:49:20.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 shutdown_connections 2024-07-26T07:49:20.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 >> 172.21.15.97:0/3487127484 conn(0x7fcca40708f0 msgr2=0x7fcca4071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:20.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 shutdown_connections 2024-07-26T07:49:20.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.356+0000 7fccab200640 1 -- 172.21.15.97:0/3487127484 wait complete. 2024-07-26T07:49:20.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-07-26T07:49:20.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]': finished 2024-07-26T07:49:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:20 smithi115 ceph-mon[35364]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T07:49:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-07-26T07:49:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]': finished 2024-07-26T07:49:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:20 smithi181 ceph-mon[35869]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T07:49:21.567 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:21 smithi097 ceph-mon[32231]: pgmap v436: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:21.567 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:21 smithi097 ceph-mon[32231]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T07:49:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:21 smithi115 ceph-mon[35364]: pgmap v436: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:21 smithi115 ceph-mon[35364]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T07:49:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:21 smithi181 ceph-mon[35869]: pgmap v436: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:21 smithi181 ceph-mon[35869]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T07:49:22.072 INFO:teuthology.orchestra.run.smithi097.stdout:317827579942 2024-07-26T07:49:22.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-07-26T07:49:22.154 INFO:teuthology.orchestra.run.smithi097.stdout:77309411426 2024-07-26T07:49:22.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-07-26T07:49:22.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 -- 172.21.15.97:0/701554594 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410b100 msgr2=0x7ff40410b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 --2- 172.21.15.97:0/701554594 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410b100 0x7ff40410b510 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009900 tx=0x7ff3f002a410 comp rx=0 tx=0).stop 2024-07-26T07:49:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 -- 172.21.15.97:0/701554594 shutdown_connections 2024-07-26T07:49:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 --2- 172.21.15.97:0/701554594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff40410ba50 0x7ff404112280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 --2- 172.21.15.97:0/701554594 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410b100 0x7ff40410b510 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 --2- 172.21.15.97:0/701554594 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4041057c0 0x7ff404105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 -- 172.21.15.97:0/701554594 >> 172.21.15.97:0/701554594 conn(0x7ff404101290 msgr2=0x7ff4041036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 -- 172.21.15.97:0/701554594 shutdown_connections 2024-07-26T07:49:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.694+0000 7ff40a400640 1 -- 172.21.15.97:0/701554594 wait complete. 2024-07-26T07:49:22.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.696+0000 7ff40a400640 1 Processor -- start 2024-07-26T07:49:22.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.696+0000 7ff40a400640 1 -- start start 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.696+0000 7ff40a400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041057c0 0x7ff40419c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.696+0000 7ff40a400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40410b100 0x7ff40419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.696+0000 7ff40a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410ba50 0x7ff4041a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.696+0000 7ff40a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff404115570 con 0x7ff40410ba50 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.696+0000 7ff40a400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff4041153f0 con 0x7ff40410b100 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.696+0000 7ff40a400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4041156f0 con 0x7ff4041057c0 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.697+0000 7ff403e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410ba50 0x7ff4041a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.697+0000 7ff403e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410ba50 0x7ff4041a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33736/0 (socket says 172.21.15.97:33736) 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.697+0000 7ff403e00640 1 -- 172.21.15.97:0/3179827371 learned_addr learned my addr 172.21.15.97:0/3179827371 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.697+0000 7ff403400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041057c0 0x7ff40419c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.697+0000 7ff402a00640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40410b100 0x7ff40419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff403e00640 1 -- 172.21.15.97:0/3179827371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041057c0 msgr2=0x7ff40419c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff403e00640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041057c0 0x7ff40419c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff403e00640 1 -- 172.21.15.97:0/3179827371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40410b100 msgr2=0x7ff40419c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff403e00640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40410b100 0x7ff40419c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff403e00640 1 -- 172.21.15.97:0/3179827371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4041a1ed0 con 0x7ff40410ba50 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff403e00640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410ba50 0x7ff4041a1860 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00cce0 tx=0x7ff3fc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc00f450 con 0x7ff40410ba50 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3fc00fa80 con 0x7ff40410ba50 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.698+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4041a21c0 con 0x7ff40410ba50 2024-07-26T07:49:22.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.699+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc018ad0 con 0x7ff40410ba50 2024-07-26T07:49:22.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.699+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4041a89e0 con 0x7ff40410ba50 2024-07-26T07:49:22.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.700+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff404063540 con 0x7ff40410ba50 2024-07-26T07:49:22.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.703+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7ff3fc00f5f0 con 0x7ff40410ba50 2024-07-26T07:49:22.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.704+0000 7ff3fbe00640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3d10 0x7ff3e80b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.704+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff3fc0e47e0 con 0x7ff40410ba50 2024-07-26T07:49:22.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.705+0000 7ff403400640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3d10 0x7ff3e80b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.705+0000 7ff3fbe00640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] conn(0x7ff3e80bdb10 0x7ff3e80c0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.705+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 --> [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3fc024ea0 con 0x7ff3e80bdb10 2024-07-26T07:49:22.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.705+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7ff3fc0e4c30 con 0x7ff40410ba50 2024-07-26T07:49:22.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.705+0000 7ff403400640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3d10 0x7ff3e80b6220 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400bbe0 tx=0x7ff3f400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:22.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.705+0000 7ff402a00640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] conn(0x7ff3e80bdb10 0x7ff3e80c0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.706+0000 7ff402a00640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] conn(0x7ff3e80bdb10 0x7ff3e80c0000 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:22.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.709+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 <== osd.7 v2:172.21.15.115:6816/2846701605 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3fc024ea0 con 0x7ff3e80bdb10 2024-07-26T07:49:22.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.724+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 --> [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff4040639f0 con 0x7ff3e80bdb10 2024-07-26T07:49:22.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.725+0000 7ff3fbe00640 1 -- 172.21.15.97:0/3179827371 <== osd.7 v2:172.21.15.115:6816/2846701605 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff4040639f0 con 0x7ff3e80bdb10 2024-07-26T07:49:22.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 >> [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] conn(0x7ff3e80bdb10 msgr2=0x7ff3e80c0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff40a400640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] conn(0x7ff3e80bdb10 0x7ff3e80c0000 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3d10 msgr2=0x7ff3e80b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff40a400640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3d10 0x7ff3e80b6220 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400bbe0 tx=0x7ff3f400b390 comp rx=0 tx=0).stop 2024-07-26T07:49:22.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410ba50 msgr2=0x7ff4041a1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff40a400640 1 --2- 172.21.15.97:0/3179827371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff40410ba50 0x7ff4041a1860 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00cce0 tx=0x7ff3fc00a620 comp rx=0 tx=0).stop 2024-07-26T07:49:22.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff403e00640 1 -- 172.21.15.97:0/3179827371 reap_dead start 2024-07-26T07:49:22.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 shutdown_connections 2024-07-26T07:49:22.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.726+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 >> 172.21.15.97:0/3179827371 conn(0x7ff404101290 msgr2=0x7ff4041036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:22.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.727+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 shutdown_connections 2024-07-26T07:49:22.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:22.727+0000 7ff40a400640 1 -- 172.21.15.97:0/3179827371 wait complete. 2024-07-26T07:49:22.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 -- 172.21.15.97:0/3620299870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 msgr2=0x7f468410d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 --2- 172.21.15.97:0/3620299870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 0x7f468410d0e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4670009900 tx=0x7f467002a410 comp rx=0 tx=0).stop 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 -- 172.21.15.97:0/3620299870 shutdown_connections 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 --2- 172.21.15.97:0/3620299870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f468410d6a0 0x7f468410fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 --2- 172.21.15.97:0/3620299870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 0x7f468410d0e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 --2- 172.21.15.97:0/3620299870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4684104240 0x7f4684104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 -- 172.21.15.97:0/3620299870 >> 172.21.15.97:0/3620299870 conn(0x7f46840fdd80 msgr2=0x7f46841001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 -- 172.21.15.97:0/3620299870 shutdown_connections 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.221+0000 7f468b000640 1 -- 172.21.15.97:0/3620299870 wait complete. 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.223+0000 7f468b000640 1 Processor -- start 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.223+0000 7f468b000640 1 -- start start 2024-07-26T07:49:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.223+0000 7f468b000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4684104240 0x7f468419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.223+0000 7f468b000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 0x7f468419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.223+0000 7f468b000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f468410d6a0 0x7f46841a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.223+0000 7f468b000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4684113190 con 0x7f4684104b90 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.223+0000 7f468b000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4684113010 con 0x7f468410d6a0 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.223+0000 7f468b000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4684113310 con 0x7f4684104240 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 0x7f468419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 0x7f468419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33678/0 (socket says 172.21.15.97:33678) 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689200640 1 -- 172.21.15.97:0/815844234 learned_addr learned my addr 172.21.15.97:0/815844234 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689c00640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4684104240 0x7f468419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f468a600640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f468410d6a0 0x7f46841a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689200640 1 -- 172.21.15.97:0/815844234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4684104240 msgr2=0x7f468419be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689200640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4684104240 0x7f468419be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689200640 1 -- 172.21.15.97:0/815844234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f468410d6a0 msgr2=0x7f46841a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689200640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f468410d6a0 0x7f46841a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.224+0000 7f4689200640 1 -- 172.21.15.97:0/815844234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46841a1df0 con 0x7f4684104b90 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.225+0000 7f4689200640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 0x7f468419c420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4670005940 tx=0x7f46700040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.225+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467003f030 con 0x7f4684104b90 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.225+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4670003a80 con 0x7f4684104b90 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.225+0000 7f468b000640 1 -- 172.21.15.97:0/815844234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46841a2080 con 0x7f4684104b90 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.225+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4670035070 con 0x7f4684104b90 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.225+0000 7f468b000640 1 -- 172.21.15.97:0/815844234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46841a8840 con 0x7f4684104b90 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.227+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4660001030 con 0x7f4684104b90 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.229+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f4670042030 con 0x7f4684104b90 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.231+0000 7f4682000640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f466c0b3ce0 0x7f466c0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.232+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8327+0+0 (secure 0 0 0) 0x7f46701047c0 con 0x7f4684104b90 2024-07-26T07:49:22.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.232+0000 7f4689c00640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f466c0b3ce0 0x7f466c0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.232+0000 7f4682000640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] conn(0x7f466c0bdae0 0x7f466c0bffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:22.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.232+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 --> [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4670040e70 con 0x7f466c0bdae0 2024-07-26T07:49:22.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.232+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4670104b70 con 0x7f4684104b90 2024-07-26T07:49:22.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.232+0000 7f468a600640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] conn(0x7f466c0bdae0 0x7f466c0bffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:22.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.233+0000 7f4689c00640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f466c0b3ce0 0x7f466c0b61f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4678005fd0 tx=0x7f4678005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:22.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.233+0000 7f468a600640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] conn(0x7f466c0bdae0 0x7f466c0bffd0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:22.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.234+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 <== osd.0 v2:172.21.15.97:6802/706181072 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4670040e70 con 0x7f466c0bdae0 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.249+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 --> [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4660002dd0 con 0x7f466c0bdae0 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.250+0000 7f4682000640 1 -- 172.21.15.97:0/815844234 <== osd.0 v2:172.21.15.97:6802/706181072 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4660002dd0 con 0x7f466c0bdae0 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] conn(0x7f466c0bdae0 msgr2=0x7f466c0bffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] conn(0x7f466c0bdae0 0x7f466c0bffd0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f466c0b3ce0 msgr2=0x7f466c0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f466c0b3ce0 0x7f466c0b61f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4678005fd0 tx=0x7f4678005850 comp rx=0 tx=0).stop 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 msgr2=0x7f468419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 --2- 172.21.15.97:0/815844234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684104b90 0x7f468419c420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4670005940 tx=0x7f46700040e0 comp rx=0 tx=0).stop 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f468a600640 1 -- 172.21.15.97:0/815844234 reap_dead start 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 shutdown_connections 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 >> 172.21.15.97:0/815844234 conn(0x7f46840fdd80 msgr2=0x7f4684101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:22.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 shutdown_connections 2024-07-26T07:49:22.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:20.251+0000 7f4677400640 1 -- 172.21.15.97:0/815844234 wait complete. 2024-07-26T07:49:23.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:23 smithi097 ceph-mon[32231]: pgmap v438: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T07:49:23.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:23 smithi115 ceph-mon[35364]: pgmap v438: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T07:49:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:23 smithi181 ceph-mon[35869]: pgmap v438: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T07:49:23.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.960+0000 7f322b200640 1 -- 172.21.15.97:0/208059673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c109670 msgr2=0x7f322c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.960+0000 7f322b200640 1 --2- 172.21.15.97:0/208059673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c109670 0x7f322c10fea0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3224009ca0 tx=0x7f322402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.960+0000 7f322b200640 1 -- 172.21.15.97:0/208059673 shutdown_connections 2024-07-26T07:49:23.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.960+0000 7f322b200640 1 --2- 172.21.15.97:0/208059673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c109670 0x7f322c10fea0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.960+0000 7f322b200640 1 --2- 172.21.15.97:0/208059673 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f322c108d20 0x7f322c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.960+0000 7f322b200640 1 --2- 172.21.15.97:0/208059673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f322c1033e0 0x7f322c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.960+0000 7f322b200640 1 -- 172.21.15.97:0/208059673 >> 172.21.15.97:0/208059673 conn(0x7f322c0fee30 msgr2=0x7f322c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:23.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.961+0000 7f322b200640 1 -- 172.21.15.97:0/208059673 shutdown_connections 2024-07-26T07:49:23.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.961+0000 7f322b200640 1 -- 172.21.15.97:0/208059673 wait complete. 2024-07-26T07:49:23.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.962+0000 7f322b200640 1 Processor -- start 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.962+0000 7f322b200640 1 -- start start 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f322b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f322c1033e0 0x7f322c19be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f322b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c108d20 0x7f322c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f322b200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f322c109670 0x7f322c1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f322b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f322c113240 con 0x7f322c108d20 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f322b200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1130c0 con 0x7f322c109670 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f322b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1133c0 con 0x7f322c1033e0 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f3229400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c108d20 0x7f322c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f3229e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f322c1033e0 0x7f322c19be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f3229400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c108d20 0x7f322c19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33762/0 (socket says 172.21.15.97:33762) 2024-07-26T07:49:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f3229400640 1 -- 172.21.15.97:0/3301225917 learned_addr learned my addr 172.21.15.97:0/3301225917 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.963+0000 7f322a800640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f322c109670 0x7f322c1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.964+0000 7f3229400640 1 -- 172.21.15.97:0/3301225917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f322c1033e0 msgr2=0x7f322c19be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.964+0000 7f3229400640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f322c1033e0 0x7f322c19be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.964+0000 7f3229400640 1 -- 172.21.15.97:0/3301225917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f322c109670 msgr2=0x7f322c1a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.964+0000 7f3229400640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f322c109670 0x7f322c1a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.964+0000 7f3229400640 1 -- 172.21.15.97:0/3301225917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322c1a1de0 con 0x7f322c108d20 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.964+0000 7f3229400640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c108d20 0x7f322c19c410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f321800d740 tx=0x7f321800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.965+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3218017070 con 0x7f322c108d20 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.965+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322c1a20d0 con 0x7f322c108d20 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.965+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f321800de50 con 0x7f322c108d20 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.965+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321800f620 con 0x7f322c108d20 2024-07-26T07:49:23.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.965+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f322c1a8830 con 0x7f322c108d20 2024-07-26T07:49:23.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.966+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f322c10fbd0 con 0x7f322c108d20 2024-07-26T07:49:23.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.970+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f321801a070 con 0x7f322c108d20 2024-07-26T07:49:23.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.971+0000 7f3222a00640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f32100b3ce0 0x7f32100b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.971+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f32180e3b40 con 0x7f322c108d20 2024-07-26T07:49:23.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.971+0000 7f3229e00640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f32100b3ce0 0x7f32100b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.972+0000 7f3222a00640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] conn(0x7f32100bdae0 0x7f32100bffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.972+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 --> [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3218011e50 con 0x7f32100bdae0 2024-07-26T07:49:23.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.972+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f32180e3ef0 con 0x7f322c108d20 2024-07-26T07:49:23.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.972+0000 7f3229e00640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f32100b3ce0 0x7f32100b61f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f321c009bd0 tx=0x7f321c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:23.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.973+0000 7f322a800640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] conn(0x7f32100bdae0 0x7f32100bffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.974+0000 7f322a800640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] conn(0x7f32100bdae0 0x7f32100bffd0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:23.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.976+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 <== osd.1 v2:172.21.15.115:6800/2733242052 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3218011e50 con 0x7f32100bdae0 2024-07-26T07:49:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.644+0000 7f6c0a800640 1 -- 172.21.15.97:0/3125223660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0410af30 msgr2=0x7f6c0410b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.644+0000 7f6c0a800640 1 --2- 172.21.15.97:0/3125223660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0410af30 0x7f6c0410b340 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009900 tx=0x7f6bf002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.644+0000 7f6c0a800640 1 -- 172.21.15.97:0/3125223660 shutdown_connections 2024-07-26T07:49:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.644+0000 7f6c0a800640 1 --2- 172.21.15.97:0/3125223660 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c0410b880 0x7f6c041120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.644+0000 7f6c0a800640 1 --2- 172.21.15.97:0/3125223660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0410af30 0x7f6c0410b340 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.644+0000 7f6c0a800640 1 --2- 172.21.15.97:0/3125223660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c041055f0 0x7f6c04105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.644+0000 7f6c0a800640 1 -- 172.21.15.97:0/3125223660 >> 172.21.15.97:0/3125223660 conn(0x7f6c04101060 msgr2=0x7f6c041034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.644+0000 7f6c0a800640 1 -- 172.21.15.97:0/3125223660 shutdown_connections 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.645+0000 7f6c0a800640 1 -- 172.21.15.97:0/3125223660 wait complete. 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c0a800640 1 Processor -- start 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c0a800640 1 -- start start 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c0a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c041055f0 0x7f6c0419beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c0a800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c0410af30 0x7f6c0419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c0a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c0410b880 0x7f6c041a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c0a800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6c041153c0 con 0x7f6c041055f0 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c0a800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6c04115240 con 0x7f6c0410af30 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c0a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6c04115540 con 0x7f6c0410b880 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c09400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c041055f0 0x7f6c0419beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c09400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c041055f0 0x7f6c0419beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33746/0 (socket says 172.21.15.97:33746) 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.646+0000 7f6c09400640 1 -- 172.21.15.97:0/551052300 learned_addr learned my addr 172.21.15.97:0/551052300 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.647+0000 7f6c08a00640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c0410af30 0x7f6c0419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.647+0000 7f6c09e00640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c0410b880 0x7f6c041a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.647+0000 7f6c09400640 1 -- 172.21.15.97:0/551052300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c0410b880 msgr2=0x7f6c041a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.647+0000 7f6c09400640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c0410b880 0x7f6c041a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.647+0000 7f6c09400640 1 -- 172.21.15.97:0/551052300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c0410af30 msgr2=0x7f6c0419c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.647+0000 7f6c09400640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c0410af30 0x7f6c0419c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.647+0000 7f6c09400640 1 -- 172.21.15.97:0/551052300 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c041a1d60 con 0x7f6c041055f0 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.648+0000 7f6c09400640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c041055f0 0x7f6c0419beb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800d770 tx=0x7f6bf800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.648+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8017070 con 0x7f6c041055f0 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.648+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c041a2050 con 0x7f6c041055f0 2024-07-26T07:49:23.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.648+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bf80054b0 con 0x7f6c041055f0 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.648+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8004c60 con 0x7f6c041055f0 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.648+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c041a2510 con 0x7f6c041055f0 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.649+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6c04063540 con 0x7f6c041055f0 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.652+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f6bf8004e00 con 0x7f6c041055f0 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.653+0000 7f6c02000640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bec0b3c90 0x7f6bec0b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.653+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f6bf80e3d00 con 0x7f6c041055f0 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.653+0000 7f6c08a00640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bec0b3c90 0x7f6bec0b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.653+0000 7f6c02000640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] conn(0x7f6bec0bda90 0x7f6bec0bff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.653+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 --> [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bf8001d30 con 0x7f6bec0bda90 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.653+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f6bf80e4150 con 0x7f6c041055f0 2024-07-26T07:49:23.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.654+0000 7f6c08a00640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bec0b3c90 0x7f6bec0b61a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00057c0 tx=0x7f6bf0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.654+0000 7f6c09e00640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] conn(0x7f6bec0bda90 0x7f6bec0bff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.655+0000 7f6c09e00640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] conn(0x7f6bec0bda90 0x7f6bec0bff80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.658+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 <== osd.5 v2:172.21.15.181:6808/1061806763 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bf8001d30 con 0x7f6bec0bda90 2024-07-26T07:49:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.674+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 --> [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6c040639c0 con 0x7f6bec0bda90 2024-07-26T07:49:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.675+0000 7f6c02000640 1 -- 172.21.15.97:0/551052300 <== osd.5 v2:172.21.15.181:6808/1061806763 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6c040639c0 con 0x7f6bec0bda90 2024-07-26T07:49:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 >> [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] conn(0x7f6bec0bda90 msgr2=0x7f6bec0bff80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] conn(0x7f6bec0bda90 0x7f6bec0bff80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bec0b3c90 msgr2=0x7f6bec0b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bec0b3c90 0x7f6bec0b61a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00057c0 tx=0x7f6bf0000bc0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c041055f0 msgr2=0x7f6c0419beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 --2- 172.21.15.97:0/551052300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c041055f0 0x7f6c0419beb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800d770 tx=0x7f6bf800dc30 comp rx=0 tx=0).stop 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c09e00640 1 -- 172.21.15.97:0/551052300 reap_dead start 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 shutdown_connections 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 >> 172.21.15.97:0/551052300 conn(0x7f6c04101060 msgr2=0x7f6c04103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 shutdown_connections 2024-07-26T07:49:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.676+0000 7f6c0a800640 1 -- 172.21.15.97:0/551052300 wait complete. 2024-07-26T07:49:23.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.992+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 --> [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f322c104660 con 0x7f32100bdae0 2024-07-26T07:49:23.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.993+0000 7f3222a00640 1 -- 172.21.15.97:0/3301225917 <== osd.1 v2:172.21.15.115:6800/2733242052 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f322c104660 con 0x7f32100bdae0 2024-07-26T07:49:23.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 >> [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] conn(0x7f32100bdae0 msgr2=0x7f32100bffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] conn(0x7f32100bdae0 0x7f32100bffd0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:23.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f32100b3ce0 msgr2=0x7f32100b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f32100b3ce0 0x7f32100b61f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f321c009bd0 tx=0x7f321c005f90 comp rx=0 tx=0).stop 2024-07-26T07:49:23.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c108d20 msgr2=0x7f322c19c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:23.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 --2- 172.21.15.97:0/3301225917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f322c108d20 0x7f322c19c410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f321800d740 tx=0x7f321800dc00 comp rx=0 tx=0).stop 2024-07-26T07:49:23.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322a800640 1 -- 172.21.15.97:0/3301225917 reap_dead start 2024-07-26T07:49:23.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 shutdown_connections 2024-07-26T07:49:23.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 >> 172.21.15.97:0/3301225917 conn(0x7f322c0fee30 msgr2=0x7f322c10e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:23.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 shutdown_connections 2024-07-26T07:49:23.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:23.994+0000 7f322b200640 1 -- 172.21.15.97:0/3301225917 wait complete. 2024-07-26T07:49:24.864 INFO:teuthology.orchestra.run.smithi097.stdout:163208757329 2024-07-26T07:49:24.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-07-26T07:49:25.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.434+0000 7f7666400640 1 -- 172.21.15.97:0/1838003494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7660109830 msgr2=0x7f7660110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:25.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.434+0000 7f7666400640 1 --2- 172.21.15.97:0/1838003494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7660109830 0x7f7660110060 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7658009ca0 tx=0x7f765802a8d0 comp rx=0 tx=0).stop 2024-07-26T07:49:25.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.434+0000 7f7666400640 1 -- 172.21.15.97:0/1838003494 shutdown_connections 2024-07-26T07:49:25.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.434+0000 7f7666400640 1 --2- 172.21.15.97:0/1838003494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7660109830 0x7f7660110060 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:25.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.434+0000 7f7666400640 1 --2- 172.21.15.97:0/1838003494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7660108ee0 0x7f76601092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:25.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.434+0000 7f7666400640 1 --2- 172.21.15.97:0/1838003494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76601035f0 0x7f7660103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:25.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.434+0000 7f7666400640 1 -- 172.21.15.97:0/1838003494 >> 172.21.15.97:0/1838003494 conn(0x7f766007b340 msgr2=0x7f766007b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:25.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.435+0000 7f7666400640 1 -- 172.21.15.97:0/1838003494 shutdown_connections 2024-07-26T07:49:25.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.435+0000 7f7666400640 1 -- 172.21.15.97:0/1838003494 wait complete. 2024-07-26T07:49:25.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.436+0000 7f7666400640 1 Processor -- start 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.436+0000 7f7666400640 1 -- start start 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.436+0000 7f7666400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76601035f0 0x7f7660075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.436+0000 7f7666400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7660108ee0 0x7f7660075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.436+0000 7f7666400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7660109830 0x7f7660075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.436+0000 7f7666400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f76601133e0 con 0x7f76601035f0 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.436+0000 7f7666400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7660113260 con 0x7f7660108ee0 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.436+0000 7f7666400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7660113560 con 0x7f7660109830 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76601035f0 0x7f7660075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76601035f0 0x7f7660075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33772/0 (socket says 172.21.15.97:33772) 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765f400640 1 -- 172.21.15.97:0/1601668235 learned_addr learned my addr 172.21.15.97:0/1601668235 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:25.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f7657e00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7660108ee0 0x7f7660075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765fe00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7660109830 0x7f7660075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765f400640 1 -- 172.21.15.97:0/1601668235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7660109830 msgr2=0x7f7660075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765f400640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7660109830 0x7f7660075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765f400640 1 -- 172.21.15.97:0/1601668235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7660108ee0 msgr2=0x7f7660075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765f400640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7660108ee0 0x7f7660075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765f400640 1 -- 172.21.15.97:0/1601668235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76601a7fa0 con 0x7f76601035f0 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f765fe00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7660109830 0x7f7660075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.437+0000 7f7657e00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7660108ee0 0x7f7660075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.438+0000 7f765f400640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76601035f0 0x7f7660075140 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f765000c920 tx=0x7f765000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.438+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650007c70 con 0x7f76601035f0 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.438+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7650007e10 con 0x7f76601035f0 2024-07-26T07:49:25.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.438+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650018710 con 0x7f76601035f0 2024-07-26T07:49:25.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.438+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76601a8290 con 0x7f76601035f0 2024-07-26T07:49:25.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.438+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76601a8850 con 0x7f76601035f0 2024-07-26T07:49:25.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.439+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7660063580 con 0x7f76601035f0 2024-07-26T07:49:25.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.442+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f765000f450 con 0x7f76601035f0 2024-07-26T07:49:25.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.444+0000 7f765cc00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76440b3c60 0x7f76440b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:25.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.444+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f76500e4450 con 0x7f76601035f0 2024-07-26T07:49:25.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.444+0000 7f7657e00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76440b3c60 0x7f76440b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:25.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.445+0000 7f765cc00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] conn(0x7f76440bda60 0x7f76440bff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:25.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.445+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 --> [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f765001ee50 con 0x7f76440bda60 2024-07-26T07:49:25.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.445+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f76500e48a0 con 0x7f76601035f0 2024-07-26T07:49:25.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.445+0000 7f765fe00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] conn(0x7f76440bda60 0x7f76440bff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:25.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.445+0000 7f7657e00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76440b3c60 0x7f76440b6170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7660108a70 tx=0x7f7648008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:25.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.446+0000 7f765fe00640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] conn(0x7f76440bda60 0x7f76440bff50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:25.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.447+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 <== osd.6 v2:172.21.15.97:6818/3873314113 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f765001ee50 con 0x7f76440bda60 2024-07-26T07:49:25.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.462+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 --> [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7660063980 con 0x7f76440bda60 2024-07-26T07:49:25.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.463+0000 7f765cc00640 1 -- 172.21.15.97:0/1601668235 <== osd.6 v2:172.21.15.97:6818/3873314113 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7660063980 con 0x7f76440bda60 2024-07-26T07:49:25.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.463+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] conn(0x7f76440bda60 msgr2=0x7f76440bff50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:25.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.463+0000 7f7666400640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] conn(0x7f76440bda60 0x7f76440bff50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:25.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.463+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76440b3c60 msgr2=0x7f76440b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:25.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.463+0000 7f7666400640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f76440b3c60 0x7f76440b6170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7660108a70 tx=0x7f7648008040 comp rx=0 tx=0).stop 2024-07-26T07:49:25.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.463+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76601035f0 msgr2=0x7f7660075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:25.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.463+0000 7f7666400640 1 --2- 172.21.15.97:0/1601668235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76601035f0 0x7f7660075140 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f765000c920 tx=0x7f765000cde0 comp rx=0 tx=0).stop 2024-07-26T07:49:25.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.463+0000 7f765fe00640 1 -- 172.21.15.97:0/1601668235 reap_dead start 2024-07-26T07:49:25.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.464+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 shutdown_connections 2024-07-26T07:49:25.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.464+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 >> 172.21.15.97:0/1601668235 conn(0x7f766007b340 msgr2=0x7f7660106f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:25.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.464+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 shutdown_connections 2024-07-26T07:49:25.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:25.464+0000 7f7666400640 1 -- 172.21.15.97:0/1601668235 wait complete. 2024-07-26T07:49:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:25 smithi097 ceph-mon[32231]: pgmap v439: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 13 B/s, 0 objects/s recovering 2024-07-26T07:49:25.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:25 smithi115 ceph-mon[35364]: pgmap v439: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 13 B/s, 0 objects/s recovering 2024-07-26T07:49:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:25 smithi181 ceph-mon[35869]: pgmap v439: 97 pgs: 1 peering, 96 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 13 B/s, 0 objects/s recovering 2024-07-26T07:49:25.975 INFO:teuthology.orchestra.run.smithi097.stdout:55834574957 2024-07-26T07:49:25.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-07-26T07:49:26.654 INFO:teuthology.orchestra.run.smithi097.stdout:137438953557 2024-07-26T07:49:26.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-07-26T07:49:27.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.510+0000 7f868b000640 1 -- 172.21.15.97:0/1009330837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410f330 msgr2=0x7f868410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:27.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.510+0000 7f868b000640 1 --2- 172.21.15.97:0/1009330837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410f330 0x7f868410f740 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8670009900 tx=0x7f867002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:49:27.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.511+0000 7f868b000640 1 -- 172.21.15.97:0/1009330837 shutdown_connections 2024-07-26T07:49:27.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.511+0000 7f868b000640 1 --2- 172.21.15.97:0/1009330837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f868410fc80 0x7f86841164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:27.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.511+0000 7f868b000640 1 --2- 172.21.15.97:0/1009330837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410f330 0x7f868410f740 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:27.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.511+0000 7f868b000640 1 --2- 172.21.15.97:0/1009330837 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86841099f0 0x7f8684109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:27.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.511+0000 7f868b000640 1 -- 172.21.15.97:0/1009330837 >> 172.21.15.97:0/1009330837 conn(0x7f8684078ff0 msgr2=0x7f8684107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.511+0000 7f868b000640 1 -- 172.21.15.97:0/1009330837 shutdown_connections 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.511+0000 7f868b000640 1 -- 172.21.15.97:0/1009330837 wait complete. 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.512+0000 7f868b000640 1 Processor -- start 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.512+0000 7f868b000640 1 -- start start 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868b000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86841099f0 0x7f86841a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868b000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f868410f330 0x7f86841a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868b000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410fc80 0x7f86841a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868b000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f86841197f0 con 0x7f868410fc80 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868b000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8684119670 con 0x7f868410f330 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868b000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8684119970 con 0x7f86841099f0 2024-07-26T07:49:27.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410fc80 0x7f86841a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410fc80 0x7f86841a5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49270/0 (socket says 172.21.15.97:49270) 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f868a600640 1 -- 172.21.15.97:0/1583615412 learned_addr learned my addr 172.21.15.97:0/1583615412 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f8689c00640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86841099f0 0x7f86841a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.513+0000 7f8689200640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f868410f330 0x7f86841a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f868a600640 1 -- 172.21.15.97:0/1583615412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86841099f0 msgr2=0x7f86841a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f868a600640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86841099f0 0x7f86841a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f868a600640 1 -- 172.21.15.97:0/1583615412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f868410f330 msgr2=0x7f86841a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f868a600640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f868410f330 0x7f86841a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f868a600640 1 -- 172.21.15.97:0/1583615412 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86841a6110 con 0x7f868410fc80 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f8689c00640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86841099f0 0x7f86841a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f868a600640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410fc80 0x7f86841a5aa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f867c00eaa0 tx=0x7f867c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:27.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c00cdb0 con 0x7f868410fc80 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f867c005020 con 0x7f868410fc80 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86841a6400 con 0x7f868410fc80 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.514+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c0116d0 con 0x7f868410fc80 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.515+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86841acc20 con 0x7f868410fc80 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.515+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8684063580 con 0x7f868410fc80 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.518+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f867c0051d0 con 0x7f868410fc80 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.520+0000 7f8682000640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f866c0b3d10 0x7f866c0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.520+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f867c0e43a0 con 0x7f868410fc80 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.520+0000 7f8689c00640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f866c0b3d10 0x7f866c0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.520+0000 7f8682000640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] conn(0x7f866c0bdb10 0x7f866c0c0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:27.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.520+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 --> [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f867c014ec0 con 0x7f866c0bdb10 2024-07-26T07:49:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.520+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f867c0e47f0 con 0x7f868410fc80 2024-07-26T07:49:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.521+0000 7f8689c00640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f866c0b3d10 0x7f866c0b6220 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f867800abe0 tx=0x7f867800a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.521+0000 7f8689200640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] conn(0x7f866c0bdb10 0x7f866c0c0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.522+0000 7f8689200640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] conn(0x7f866c0bdb10 0x7f866c0c0000 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.524+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 <== osd.10 v2:172.21.15.115:6824/2791495352 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f867c014ec0 con 0x7f866c0bdb10 2024-07-26T07:49:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.540+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 --> [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8684063a00 con 0x7f866c0bdb10 2024-07-26T07:49:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.541+0000 7f8682000640 1 -- 172.21.15.97:0/1583615412 <== osd.10 v2:172.21.15.115:6824/2791495352 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8684063a00 con 0x7f866c0bdb10 2024-07-26T07:49:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.541+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 >> [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] conn(0x7f866c0bdb10 msgr2=0x7f866c0c0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] conn(0x7f866c0bdb10 0x7f866c0c0000 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f866c0b3d10 msgr2=0x7f866c0b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f866c0b3d10 0x7f866c0b6220 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f867800abe0 tx=0x7f867800a5c0 comp rx=0 tx=0).stop 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410fc80 msgr2=0x7f86841a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 --2- 172.21.15.97:0/1583615412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f868410fc80 0x7f86841a5aa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f867c00eaa0 tx=0x7f867c00ef60 comp rx=0 tx=0).stop 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868a600640 1 -- 172.21.15.97:0/1583615412 reap_dead start 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 shutdown_connections 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 >> 172.21.15.97:0/1583615412 conn(0x7f8684078ff0 msgr2=0x7f86840798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 shutdown_connections 2024-07-26T07:49:27.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:26.542+0000 7f868b000640 1 -- 172.21.15.97:0/1583615412 wait complete. 2024-07-26T07:49:27.437 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:27 smithi097 ceph-mon[32231]: pgmap v440: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 87 B/s, 1 objects/s recovering 2024-07-26T07:49:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:27 smithi115 ceph-mon[35364]: pgmap v440: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 87 B/s, 1 objects/s recovering 2024-07-26T07:49:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:27 smithi181 ceph-mon[35869]: pgmap v440: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 87 B/s, 1 objects/s recovering 2024-07-26T07:49:27.920 INFO:teuthology.orchestra.run.smithi097.stdout:188978561097 2024-07-26T07:49:27.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-07-26T07:49:28.324 INFO:teuthology.orchestra.run.smithi097.stdout:292057776174 2024-07-26T07:49:28.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-07-26T07:49:28.416 INFO:teuthology.orchestra.run.smithi097.stdout:240518168636 2024-07-26T07:49:28.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-07-26T07:49:29.274 INFO:teuthology.orchestra.run.smithi097.stdout:210453397569 2024-07-26T07:49:29.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-07-26T07:49:29.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:29 smithi097 ceph-mon[32231]: pgmap v441: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 78 B/s, 1 objects/s recovering 2024-07-26T07:49:29.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:29 smithi115 ceph-mon[35364]: pgmap v441: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 78 B/s, 1 objects/s recovering 2024-07-26T07:49:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:29 smithi181 ceph-mon[35869]: pgmap v441: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 78 B/s, 1 objects/s recovering 2024-07-26T07:49:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 -- 172.21.15.97:0/144273918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff26410b860 msgr2=0x7ff264112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 --2- 172.21.15.97:0/144273918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff26410b860 0x7ff264112090 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009ca0 tx=0x7ff25c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:49:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 -- 172.21.15.97:0/144273918 shutdown_connections 2024-07-26T07:49:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 --2- 172.21.15.97:0/144273918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff26410b860 0x7ff264112090 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:30.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 --2- 172.21.15.97:0/144273918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff26410af10 0x7ff26410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 --2- 172.21.15.97:0/144273918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2641055d0 0x7ff2641059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 -- 172.21.15.97:0/144273918 >> 172.21.15.97:0/144273918 conn(0x7ff264101040 msgr2=0x7ff264103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 -- 172.21.15.97:0/144273918 shutdown_connections 2024-07-26T07:49:30.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.053+0000 7ff263e00640 1 -- 172.21.15.97:0/144273918 wait complete. 2024-07-26T07:49:30.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.054+0000 7ff263e00640 1 Processor -- start 2024-07-26T07:49:30.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.054+0000 7ff263e00640 1 -- start start 2024-07-26T07:49:30.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff263e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2641055d0 0x7ff26419be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:30.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff263e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff26410af10 0x7ff26419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:30.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff263e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff26410b860 0x7ff2641a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:30.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff263e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff264115430 con 0x7ff2641055d0 2024-07-26T07:49:30.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff263e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641152b0 con 0x7ff26410b860 2024-07-26T07:49:30.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff263e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641155b0 con 0x7ff26410af10 2024-07-26T07:49:30.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff262a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2641055d0 0x7ff26419be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:30.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff262a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2641055d0 0x7ff26419be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49292/0 (socket says 172.21.15.97:49292) 2024-07-26T07:49:30.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff262a00640 1 -- 172.21.15.97:0/1634445245 learned_addr learned my addr 172.21.15.97:0/1634445245 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:30.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff25be00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff26410af10 0x7ff26419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:30.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.055+0000 7ff263400640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff26410b860 0x7ff2641a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:30.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff262a00640 1 -- 172.21.15.97:0/1634445245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff26410af10 msgr2=0x7ff26419c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:30.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff262a00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff26410af10 0x7ff26419c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:30.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff262a00640 1 -- 172.21.15.97:0/1634445245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff26410b860 msgr2=0x7ff2641a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:30.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff262a00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff26410b860 0x7ff2641a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff262a00640 1 -- 172.21.15.97:0/1634445245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2641a1dd0 con 0x7ff2641055d0 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff263400640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff26410b860 0x7ff2641a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff262a00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2641055d0 0x7ff26419be60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff25400d770 tx=0x7ff25400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254017070 con 0x7ff2641055d0 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2641a20c0 con 0x7ff2641055d0 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2540054b0 con 0x7ff2641055d0 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254004c60 con 0x7ff2641055d0 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.056+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2641a8820 con 0x7ff2641055d0 2024-07-26T07:49:30.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.057+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff264063580 con 0x7ff2641055d0 2024-07-26T07:49:30.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.060+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7ff254004e00 con 0x7ff2641055d0 2024-07-26T07:49:30.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.061+0000 7ff25b400640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2440b3c60 0x7ff2440b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:30.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.062+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff2540e3d90 con 0x7ff2641055d0 2024-07-26T07:49:30.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.062+0000 7ff25be00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2440b3c60 0x7ff2440b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:30.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.062+0000 7ff25b400640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] conn(0x7ff2440bda60 0x7ff2440bff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:30.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.062+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 --> [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff254001d30 con 0x7ff2440bda60 2024-07-26T07:49:30.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.062+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7ff2540e4140 con 0x7ff2641055d0 2024-07-26T07:49:30.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.063+0000 7ff263400640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] conn(0x7ff2440bda60 0x7ff2440bff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:30.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.063+0000 7ff25be00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2440b3c60 0x7ff2440b6170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff24c005fd0 tx=0x7ff24c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:30.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.063+0000 7ff263400640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] conn(0x7ff2440bda60 0x7ff2440bff50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:30.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.064+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 <== osd.9 v2:172.21.15.97:6826/2688123967 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff254001d30 con 0x7ff2440bda60 2024-07-26T07:49:30.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.079+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 --> [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff264063a00 con 0x7ff2440bda60 2024-07-26T07:49:30.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.080+0000 7ff25b400640 1 -- 172.21.15.97:0/1634445245 <== osd.9 v2:172.21.15.97:6826/2688123967 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff264063a00 con 0x7ff2440bda60 2024-07-26T07:49:30.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] conn(0x7ff2440bda60 msgr2=0x7ff2440bff50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:30.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] conn(0x7ff2440bda60 0x7ff2440bff50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2440b3c60 msgr2=0x7ff2440b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2440b3c60 0x7ff2440b6170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff24c005fd0 tx=0x7ff24c005850 comp rx=0 tx=0).stop 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2641055d0 msgr2=0x7ff26419be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 --2- 172.21.15.97:0/1634445245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff2641055d0 0x7ff26419be60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff25400d770 tx=0x7ff25400dc30 comp rx=0 tx=0).stop 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263400640 1 -- 172.21.15.97:0/1634445245 reap_dead start 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 shutdown_connections 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 >> 172.21.15.97:0/1634445245 conn(0x7ff264101040 msgr2=0x7ff264103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 shutdown_connections 2024-07-26T07:49:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:30.081+0000 7ff263e00640 1 -- 172.21.15.97:0/1634445245 wait complete. 2024-07-26T07:49:30.382 INFO:teuthology.orchestra.run.smithi097.stdout:34359738483 2024-07-26T07:49:30.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-07-26T07:49:31.049 INFO:teuthology.orchestra.run.smithi097.stdout:266287972406 2024-07-26T07:49:31.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-07-26T07:49:31.340 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:31.343 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:31.343 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:31.358 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:31.367 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:31.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.398+0000 7f3943e00640 1 -- 172.21.15.97:0/421338593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c113e70 msgr2=0x7f393c116340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:31.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.398+0000 7f3943e00640 1 --2- 172.21.15.97:0/421338593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c113e70 0x7f393c116340 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3934009ca0 tx=0x7f393402a8f0 comp rx=0 tx=0).stop 2024-07-26T07:49:31.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.398+0000 7f3943e00640 1 -- 172.21.15.97:0/421338593 shutdown_connections 2024-07-26T07:49:31.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.398+0000 7f3943e00640 1 --2- 172.21.15.97:0/421338593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c113e70 0x7f393c116340 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:31.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.398+0000 7f3943e00640 1 --2- 172.21.15.97:0/421338593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c075800 0x7f393c075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:31.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.398+0000 7f3943e00640 1 --2- 172.21.15.97:0/421338593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f393c076db0 0x7f393c0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:31.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.398+0000 7f3943e00640 1 -- 172.21.15.97:0/421338593 >> 172.21.15.97:0/421338593 conn(0x7f393c0ff010 msgr2=0x7f393c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:31.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.400+0000 7f3943e00640 1 -- 172.21.15.97:0/421338593 shutdown_connections 2024-07-26T07:49:31.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.400+0000 7f3943e00640 1 -- 172.21.15.97:0/421338593 wait complete. 2024-07-26T07:49:31.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.402+0000 7f3943e00640 1 Processor -- start 2024-07-26T07:49:31.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.402+0000 7f3943e00640 1 -- start start 2024-07-26T07:49:31.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.403+0000 7f3943e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f393c075800 0x7f393c1a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:31.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.403+0000 7f3943e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c076db0 0x7f393c1a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.403+0000 7f3943e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c113e70 0x7f393c1a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.403+0000 7f3943e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f393c119a30 con 0x7f393c076db0 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.403+0000 7f3943e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f393c1198b0 con 0x7f393c113e70 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.403+0000 7f3943e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f393c119bb0 con 0x7f393c075800 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.403+0000 7f3940a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c076db0 0x7f393c1a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3940a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c076db0 0x7f393c1a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49308/0 (socket says 172.21.15.97:49308) 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3940a00640 1 -- 172.21.15.97:0/1682199196 learned_addr learned my addr 172.21.15.97:0/1682199196 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3941400640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f393c075800 0x7f393c1a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3941e00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c113e70 0x7f393c1a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3940a00640 1 -- 172.21.15.97:0/1682199196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f393c075800 msgr2=0x7f393c1a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3940a00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f393c075800 0x7f393c1a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3940a00640 1 -- 172.21.15.97:0/1682199196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c113e70 msgr2=0x7f393c1a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:31.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3940a00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f393c113e70 0x7f393c1a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:31.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.404+0000 7f3940a00640 1 -- 172.21.15.97:0/1682199196 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393c1a6290 con 0x7f393c076db0 2024-07-26T07:49:31.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.406+0000 7f3940a00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c076db0 0x7f393c1a0980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f393000c970 tx=0x7f393000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:31.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.407+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3930014070 con 0x7f393c076db0 2024-07-26T07:49:31.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.407+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3930004440 con 0x7f393c076db0 2024-07-26T07:49:31.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.407+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3930003d40 con 0x7f393c076db0 2024-07-26T07:49:31.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.407+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f393c1a6580 con 0x7f393c076db0 2024-07-26T07:49:31.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.408+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f393c1acda0 con 0x7f393c076db0 2024-07-26T07:49:31.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.409+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f391c001060 con 0x7f393c076db0 2024-07-26T07:49:31.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.415+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f3930017070 con 0x7f393c076db0 2024-07-26T07:49:31.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.419+0000 7f393a000640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f39240b3d90 0x7f39240b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:31.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.419+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f39300e3be0 con 0x7f393c076db0 2024-07-26T07:49:31.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.420+0000 7f393a000640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] conn(0x7f39240bdb90 0x7f39240c0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:31.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.420+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 --> [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39240c0750 con 0x7f39240bdb90 2024-07-26T07:49:31.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.420+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f39300e3fd0 con 0x7f393c076db0 2024-07-26T07:49:31.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.420+0000 7f3941400640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f39240b3d90 0x7f39240b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:31.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.421+0000 7f3941e00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] conn(0x7f39240bdb90 0x7f39240c0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:31.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.423+0000 7f3941400640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f39240b3d90 0x7f39240b62a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3928006fd0 tx=0x7f39280090e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:31.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.423+0000 7f3941e00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] conn(0x7f39240bdb90 0x7f39240c0080 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:31.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.425+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 <== osd.3 v2:172.21.15.97:6810/461651027 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39240c0750 con 0x7f39240bdb90 2024-07-26T07:49:31.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.462+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 --> [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f391c002e00 con 0x7f39240bdb90 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.463+0000 7f393a000640 1 -- 172.21.15.97:0/1682199196 <== osd.3 v2:172.21.15.97:6810/461651027 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f391c002e00 con 0x7f39240bdb90 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.464+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] conn(0x7f39240bdb90 msgr2=0x7f39240c0080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.464+0000 7f3943e00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] conn(0x7f39240bdb90 0x7f39240c0080 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.465+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f39240b3d90 msgr2=0x7f39240b62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.465+0000 7f3943e00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f39240b3d90 0x7f39240b62a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3928006fd0 tx=0x7f39280090e0 comp rx=0 tx=0).stop 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.465+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c076db0 msgr2=0x7f393c1a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.465+0000 7f3943e00640 1 --2- 172.21.15.97:0/1682199196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f393c076db0 0x7f393c1a0980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f393000c970 tx=0x7f393000ce30 comp rx=0 tx=0).stop 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.465+0000 7f3941e00640 1 -- 172.21.15.97:0/1682199196 reap_dead start 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.465+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 shutdown_connections 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.465+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 >> 172.21.15.97:0/1682199196 conn(0x7f393c0ff010 msgr2=0x7f393c10fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:31.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.466+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 shutdown_connections 2024-07-26T07:49:31.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:31.466+0000 7f3943e00640 1 -- 172.21.15.97:0/1682199196 wait complete. 2024-07-26T07:49:31.538 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:31 smithi097 ceph-mon[32231]: pgmap v442: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 69 B/s, 1 objects/s recovering 2024-07-26T07:49:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:31 smithi115 ceph-mon[35364]: pgmap v442: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 69 B/s, 1 objects/s recovering 2024-07-26T07:49:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:31 smithi181 ceph-mon[35869]: pgmap v442: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 69 B/s, 1 objects/s recovering 2024-07-26T07:49:31.948 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:31.951 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:31.955 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:33.576 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:33 smithi097 ceph-mon[32231]: pgmap v443: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 58 B/s, 0 objects/s recovering 2024-07-26T07:49:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:33 smithi115 ceph-mon[35364]: pgmap v443: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 58 B/s, 0 objects/s recovering 2024-07-26T07:49:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:33 smithi181 ceph-mon[35869]: pgmap v443: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 58 B/s, 0 objects/s recovering 2024-07-26T07:49:34.750 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:35 smithi097 ceph-mon[32231]: pgmap v444: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 58 B/s, 0 objects/s recovering 2024-07-26T07:49:35.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:35 smithi115 ceph-mon[35364]: pgmap v444: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 58 B/s, 0 objects/s recovering 2024-07-26T07:49:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:35 smithi181 ceph-mon[35869]: pgmap v444: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 58 B/s, 0 objects/s recovering 2024-07-26T07:49:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:37.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 -- 172.21.15.97:0/1216601464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc109840 msgr2=0x7fd7dc110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/1216601464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc109840 0x7fd7dc110070 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009900 tx=0x7fd7d402a410 comp rx=0 tx=0).stop 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 -- 172.21.15.97:0/1216601464 shutdown_connections 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/1216601464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc109840 0x7fd7dc110070 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/1216601464 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7dc108ef0 0x7fd7dc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/1216601464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7dc1035b0 0x7fd7dc1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 -- 172.21.15.97:0/1216601464 >> 172.21.15.97:0/1216601464 conn(0x7fd7dc0fefc0 msgr2=0x7fd7dc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 -- 172.21.15.97:0/1216601464 shutdown_connections 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.226+0000 7fd7e0e00640 1 -- 172.21.15.97:0/1216601464 wait complete. 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.227+0000 7fd7e0e00640 1 Processor -- start 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.227+0000 7fd7e0e00640 1 -- start start 2024-07-26T07:49:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7e0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7dc1035b0 0x7fd7dc199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7e0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc108ef0 0x7fd7dc19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7e0e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7dc109840 0x7fd7dc19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7e0e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc113410 con 0x7fd7dc108ef0 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7e0e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc113290 con 0x7fd7dc109840 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7e0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc113590 con 0x7fd7dc1035b0 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc108ef0 0x7fd7dc19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc108ef0 0x7fd7dc19a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33850/0 (socket says 172.21.15.97:33850) 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9400640 1 -- 172.21.15.97:0/2180494960 learned_addr learned my addr 172.21.15.97:0/2180494960 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7dc1035b0 0x7fd7dc199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9400640 1 -- 172.21.15.97:0/2180494960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7dc1035b0 msgr2=0x7fd7dc199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:37.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9400640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7dc1035b0 0x7fd7dc199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9400640 1 -- 172.21.15.97:0/2180494960 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7dc109840 msgr2=0x7fd7dc19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9400640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7dc109840 0x7fd7dc19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9400640 1 -- 172.21.15.97:0/2180494960 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc19fd50 con 0x7fd7dc108ef0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.228+0000 7fd7d9e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7dc1035b0 0x7fd7dc199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.229+0000 7fd7d9400640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc108ef0 0x7fd7dc19a380 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400cce0 tx=0x7fd7c400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.229+0000 7fd7d2a00640 1 -- 172.21.15.97:0/2180494960 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4004b20 con 0x7fd7dc108ef0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.229+0000 7fd7d2a00640 1 -- 172.21.15.97:0/2180494960 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd7c4004cc0 con 0x7fd7dc108ef0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.229+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7dc1a0040 con 0x7fd7dc108ef0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.229+0000 7fd7d2a00640 1 -- 172.21.15.97:0/2180494960 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c400f6b0 con 0x7fd7dc108ef0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.229+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7dc1a67a0 con 0x7fd7dc108ef0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.230+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7dc1039c0 con 0x7fd7dc108ef0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.233+0000 7fd7d2a00640 1 -- 172.21.15.97:0/2180494960 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fd7c400f850 con 0x7fd7dc108ef0 2024-07-26T07:49:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.235+0000 7fd7d2a00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd7c00b3ce0 0x7fd7c00b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:37.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.235+0000 7fd7d2a00640 1 -- 172.21.15.97:0/2180494960 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd7c40e3e60 con 0x7fd7dc108ef0 2024-07-26T07:49:37.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.235+0000 7fd7d9e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd7c00b3ce0 0x7fd7c00b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:37.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.236+0000 7fd7d9e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd7c00b3ce0 0x7fd7c00b61f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0058c0 tx=0x7fd7cc005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:37.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.239+0000 7fd7d2a00640 1 -- 172.21.15.97:0/2180494960 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd7c4014030 con 0x7fd7dc108ef0 2024-07-26T07:49:37.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.443+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fd7dc02e0c0 con 0x7fd7dc108ef0 2024-07-26T07:49:37.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.444+0000 7fd7d2a00640 1 -- 172.21.15.97:0/2180494960 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd7c40ab290 con 0x7fd7dc108ef0 2024-07-26T07:49:37.445 INFO:teuthology.orchestra.run.smithi097.stdout:317827579946 2024-07-26T07:49:37.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd7c00b3ce0 msgr2=0x7fd7c00b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:37.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd7c00b3ce0 0x7fd7c00b61f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0058c0 tx=0x7fd7cc005d30 comp rx=0 tx=0).stop 2024-07-26T07:49:37.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc108ef0 msgr2=0x7fd7dc19a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:37.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc108ef0 0x7fd7dc19a380 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400cce0 tx=0x7fd7c400a620 comp rx=0 tx=0).stop 2024-07-26T07:49:37.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 shutdown_connections 2024-07-26T07:49:37.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd7c00b3ce0 0x7fd7c00b61f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7dc109840 0x7fd7dc19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd7dc108ef0 0x7fd7dc19a380 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 --2- 172.21.15.97:0/2180494960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7dc1035b0 0x7fd7dc199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:37.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 >> 172.21.15.97:0/2180494960 conn(0x7fd7dc0fefc0 msgr2=0x7fd7dc1022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:37.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.447+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 shutdown_connections 2024-07-26T07:49:37.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:37.448+0000 7fd7e0e00640 1 -- 172.21.15.97:0/2180494960 wait complete. 2024-07-26T07:49:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:37 smithi115 ceph-mon[35364]: pgmap v445: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 48 B/s, 0 objects/s recovering 2024-07-26T07:49:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:37 smithi181 ceph-mon[35869]: pgmap v445: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 48 B/s, 0 objects/s recovering 2024-07-26T07:49:37.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:37 smithi097 ceph-mon[32231]: pgmap v445: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 48 B/s, 0 objects/s recovering 2024-07-26T07:49:38.606 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:38 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2180494960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T07:49:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:38 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2180494960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T07:49:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:38 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2180494960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T07:49:39.473 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:39 smithi097 ceph-mon[32231]: pgmap v446: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:49:39.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:39 smithi115 ceph-mon[35364]: pgmap v446: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:49:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:39 smithi181 ceph-mon[35869]: pgmap v446: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:49:40.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.369+0000 7fa364e00640 1 -- 172.21.15.97:0/1685421538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601029f0 msgr2=0x7fa36010f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.369+0000 7fa364e00640 1 --2- 172.21.15.97:0/1685421538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601029f0 0x7fa36010f710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009900 tx=0x7fa34c02a480 comp rx=0 tx=0).stop 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.370+0000 7fa364e00640 1 -- 172.21.15.97:0/1685421538 shutdown_connections 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.370+0000 7fa364e00640 1 --2- 172.21.15.97:0/1685421538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa36010fc50 0x7fa360112070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.370+0000 7fa364e00640 1 --2- 172.21.15.97:0/1685421538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601029f0 0x7fa36010f710 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.370+0000 7fa364e00640 1 --2- 172.21.15.97:0/1685421538 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3601020a0 0x7fa3601024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.370+0000 7fa364e00640 1 -- 172.21.15.97:0/1685421538 >> 172.21.15.97:0/1685421538 conn(0x7fa360079170 msgr2=0x7fa360079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.370+0000 7fa364e00640 1 -- 172.21.15.97:0/1685421538 shutdown_connections 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.370+0000 7fa364e00640 1 -- 172.21.15.97:0/1685421538 wait complete. 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.371+0000 7fa364e00640 1 Processor -- start 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.371+0000 7fa364e00640 1 -- start start 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.371+0000 7fa364e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601020a0 0x7fa36019c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa364e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3601029f0 0x7fa36019c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa364e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa36010fc50 0x7fa3601a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa364e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa360115590 con 0x7fa3601020a0 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa364e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa360115410 con 0x7fa36010fc50 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa364e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa360115710 con 0x7fa3601029f0 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601020a0 0x7fa36019c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601020a0 0x7fa36019c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33868/0 (socket says 172.21.15.97:33868) 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35de00640 1 -- 172.21.15.97:0/3219793362 learned_addr learned my addr 172.21.15.97:0/3219793362 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35d400640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3601029f0 0x7fa36019c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35e800640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa36010fc50 0x7fa3601a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35de00640 1 -- 172.21.15.97:0/3219793362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3601029f0 msgr2=0x7fa36019c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35de00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3601029f0 0x7fa36019c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35de00640 1 -- 172.21.15.97:0/3219793362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa36010fc50 msgr2=0x7fa3601a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35de00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa36010fc50 0x7fa3601a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35de00640 1 -- 172.21.15.97:0/3219793362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3601a1ed0 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.372+0000 7fa35e800640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa36010fc50 0x7fa3601a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.373+0000 7fa35de00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601020a0 0x7fa36019c020 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c920 tx=0x7fa35000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.373+0000 7fa356a00640 1 -- 172.21.15.97:0/3219793362 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350007c70 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.373+0000 7fa356a00640 1 -- 172.21.15.97:0/3219793362 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa350007e10 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.373+0000 7fa356a00640 1 -- 172.21.15.97:0/3219793362 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350018710 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.373+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3601a21c0 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.373+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3601a89e0 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.374+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3601024b0 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.377+0000 7fa356a00640 1 -- 172.21.15.97:0/3219793362 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fa35000f450 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.378+0000 7fa356a00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa3440b3d10 0x7fa3440b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.379+0000 7fa356a00640 1 -- 172.21.15.97:0/3219793362 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fa3500e4280 con 0x7fa3601020a0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.379+0000 7fa35d400640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa3440b3d10 0x7fa3440b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.380+0000 7fa35d400640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa3440b3d10 0x7fa3440b6220 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa34c000c00 tx=0x7fa34c009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.380+0000 7fa356a00640 1 -- 172.21.15.97:0/3219793362 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa350014030 con 0x7fa3601020a0 2024-07-26T07:49:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.587+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa36019d150 con 0x7fa3601020a0 2024-07-26T07:49:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.588+0000 7fa356a00640 1 -- 172.21.15.97:0/3219793362 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa3500ab7c0 con 0x7fa3601020a0 2024-07-26T07:49:40.390 INFO:teuthology.orchestra.run.smithi097.stdout:163208757332 2024-07-26T07:49:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa3440b3d10 msgr2=0x7fa3440b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa3440b3d10 0x7fa3440b6220 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa34c000c00 tx=0x7fa34c009680 comp rx=0 tx=0).stop 2024-07-26T07:49:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601020a0 msgr2=0x7fa36019c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601020a0 0x7fa36019c020 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c920 tx=0x7fa35000cde0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 shutdown_connections 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa3440b3d10 0x7fa3440b6220 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa36010fc50 0x7fa3601a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3601029f0 0x7fa36019c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 --2- 172.21.15.97:0/3219793362 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa3601020a0 0x7fa36019c020 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 >> 172.21.15.97:0/3219793362 conn(0x7fa360079170 msgr2=0x7fa360100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 shutdown_connections 2024-07-26T07:49:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.591+0000 7fa364e00640 1 -- 172.21.15.97:0/3219793362 wait complete. 2024-07-26T07:49:40.491 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:40 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3219793362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T07:49:40.491 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:40 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3930314972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T07:49:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:40 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3219793362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T07:49:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:40 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3930314972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T07:49:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:40 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3219793362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T07:49:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:40 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3930314972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T07:49:40.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 -- 172.21.15.97:0/3567570774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075ad0 msgr2=0x7f594c10f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 --2- 172.21.15.97:0/3567570774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075ad0 0x7f594c10f460 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5938009900 tx=0x7f593802a410 comp rx=0 tx=0).stop 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 -- 172.21.15.97:0/3567570774 shutdown_connections 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 --2- 172.21.15.97:0/3567570774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f594c10f9a0 0x7f594c111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 --2- 172.21.15.97:0/3567570774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075ad0 0x7f594c10f460 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 --2- 172.21.15.97:0/3567570774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f594c075180 0x7f594c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 -- 172.21.15.97:0/3567570774 >> 172.21.15.97:0/3567570774 conn(0x7f594c0fee70 msgr2=0x7f594c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 -- 172.21.15.97:0/3567570774 shutdown_connections 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.489+0000 7f5952c00640 1 -- 172.21.15.97:0/3567570774 wait complete. 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.490+0000 7f5952c00640 1 Processor -- start 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.490+0000 7f5952c00640 1 -- start start 2024-07-26T07:49:40.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5952c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075180 0x7f594c19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5952c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f594c075ad0 0x7f594c19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5952c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f594c10f9a0 0x7f594c1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5952c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f594c115390 con 0x7f594c075180 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5952c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f594c115210 con 0x7f594c10f9a0 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5952c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f594c115510 con 0x7f594c075ad0 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5951800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075180 0x7f594c19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5951800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075180 0x7f594c19be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33886/0 (socket says 172.21.15.97:33886) 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5951800640 1 -- 172.21.15.97:0/3930314972 learned_addr learned my addr 172.21.15.97:0/3930314972 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.491+0000 7f5950e00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f594c075ad0 0x7f594c19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f5952200640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f594c10f9a0 0x7f594c1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f5951800640 1 -- 172.21.15.97:0/3930314972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f594c075ad0 msgr2=0x7f594c19c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f5951800640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f594c075ad0 0x7f594c19c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f5951800640 1 -- 172.21.15.97:0/3930314972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f594c10f9a0 msgr2=0x7f594c1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f5951800640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f594c10f9a0 0x7f594c1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f5951800640 1 -- 172.21.15.97:0/3930314972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f594c1a1dd0 con 0x7f594c075180 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f5951800640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075180 0x7f594c19be60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f594000e9f0 tx=0x7f594000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f594a000640 1 -- 172.21.15.97:0/3930314972 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594000cd80 con 0x7f594c075180 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f594a000640 1 -- 172.21.15.97:0/3930314972 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59400108c0 con 0x7f594c075180 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.492+0000 7f594a000640 1 -- 172.21.15.97:0/3930314972 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59400187c0 con 0x7f594c075180 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.493+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f594c1a20c0 con 0x7f594c075180 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.493+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f594c1a8820 con 0x7f594c075180 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.494+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f594c075590 con 0x7f594c075180 2024-07-26T07:49:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.498+0000 7f594a000640 1 -- 172.21.15.97:0/3930314972 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f5940010430 con 0x7f594c075180 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.499+0000 7f594a000640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59340b3ce0 0x7f59340b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.499+0000 7f594a000640 1 -- 172.21.15.97:0/3930314972 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f59400e4710 con 0x7f594c075180 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.500+0000 7f5950e00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59340b3ce0 0x7f59340b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.500+0000 7f5950e00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59340b3ce0 0x7f59340b61f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f59380057c0 tx=0x7f5938000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.502+0000 7f594a000640 1 -- 172.21.15.97:0/3930314972 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f59400abba0 con 0x7f594c075180 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.709+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f594c02e0c0 con 0x7f594c075180 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.709+0000 7f594a000640 1 -- 172.21.15.97:0/3930314972 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f59400b2e20 con 0x7f594c075180 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59340b3ce0 msgr2=0x7f59340b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59340b3ce0 0x7f59340b61f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f59380057c0 tx=0x7f5938000bc0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075180 msgr2=0x7f594c19be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075180 0x7f594c19be60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f594000e9f0 tx=0x7f594000eeb0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 shutdown_connections 2024-07-26T07:49:40.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f59340b3ce0 0x7f59340b61f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f594c10f9a0 0x7f594c1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f594c075ad0 0x7f594c19c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 --2- 172.21.15.97:0/3930314972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f594c075180 0x7f594c19be60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 >> 172.21.15.97:0/3930314972 conn(0x7f594c0fee70 msgr2=0x7f594c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 shutdown_connections 2024-07-26T07:49:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:39.712+0000 7f5952c00640 1 -- 172.21.15.97:0/3930314972 wait complete. 2024-07-26T07:49:40.914 INFO:teuthology.orchestra.run.smithi097.stdout:292057776177 2024-07-26T07:49:41.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 -- 172.21.15.97:0/3228570674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 msgr2=0x7faa5c113c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:41.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 --2- 172.21.15.97:0/3228570674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 0x7faa5c113c60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faa4c009900 tx=0x7faa4c02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 -- 172.21.15.97:0/3228570674 shutdown_connections 2024-07-26T07:49:41.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 --2- 172.21.15.97:0/3228570674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa5c1141a0 0x7faa5c1165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 --2- 172.21.15.97:0/3228570674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 0x7faa5c113c60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 --2- 172.21.15.97:0/3228570674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa5c077370 0x7faa5c077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 -- 172.21.15.97:0/3228570674 >> 172.21.15.97:0/3228570674 conn(0x7faa5c1011f0 msgr2=0x7faa5c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:41.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 -- 172.21.15.97:0/3228570674 shutdown_connections 2024-07-26T07:49:41.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.206+0000 7faa64800640 1 -- 172.21.15.97:0/3228570674 wait complete. 2024-07-26T07:49:41.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.207+0000 7faa64800640 1 Processor -- start 2024-07-26T07:49:41.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa64800640 1 -- start start 2024-07-26T07:49:41.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa64800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa5c077370 0x7faa5c1a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:41.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa64800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 0x7faa5c1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa64800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa5c1141a0 0x7faa5c1a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa64800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c1199b0 con 0x7faa5c077cc0 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa64800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c119830 con 0x7faa5c1141a0 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa64800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c119b30 con 0x7faa5c077370 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa61400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 0x7faa5c1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa61400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 0x7faa5c1a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33904/0 (socket says 172.21.15.97:33904) 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.208+0000 7faa61400640 1 -- 172.21.15.97:0/49662182 learned_addr learned my addr 172.21.15.97:0/49662182 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.209+0000 7faa61e00640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa5c077370 0x7faa5c1a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.209+0000 7faa62800640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa5c1141a0 0x7faa5c1a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.209+0000 7faa61400640 1 -- 172.21.15.97:0/49662182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa5c077370 msgr2=0x7faa5c1a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:41.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.209+0000 7faa61400640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa5c077370 0x7faa5c1a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.209+0000 7faa61400640 1 -- 172.21.15.97:0/49662182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa5c1141a0 msgr2=0x7faa5c1a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:41.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.209+0000 7faa61400640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa5c1141a0 0x7faa5c1a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.209+0000 7faa61400640 1 -- 172.21.15.97:0/49662182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5c1a6270 con 0x7faa5c077cc0 2024-07-26T07:49:41.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.209+0000 7faa61400640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 0x7faa5c1a0960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faa4c005940 tx=0x7faa4c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:41.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.210+0000 7faa56a00640 1 -- 172.21.15.97:0/49662182 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c03f030 con 0x7faa5c077cc0 2024-07-26T07:49:41.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.210+0000 7faa56a00640 1 -- 172.21.15.97:0/49662182 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa4c003c90 con 0x7faa5c077cc0 2024-07-26T07:49:41.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.210+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa5c1a6500 con 0x7faa5c077cc0 2024-07-26T07:49:41.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.210+0000 7faa56a00640 1 -- 172.21.15.97:0/49662182 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c035070 con 0x7faa5c077cc0 2024-07-26T07:49:41.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.210+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa5c1a6a30 con 0x7faa5c077cc0 2024-07-26T07:49:41.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.211+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa5c077780 con 0x7faa5c077cc0 2024-07-26T07:49:41.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.214+0000 7faa56a00640 1 -- 172.21.15.97:0/49662182 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7faa4c042030 con 0x7faa5c077cc0 2024-07-26T07:49:41.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.215+0000 7faa56a00640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7faa440b3d10 0x7faa440b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:41.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.215+0000 7faa56a00640 1 -- 172.21.15.97:0/49662182 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7faa4c104850 con 0x7faa5c077cc0 2024-07-26T07:49:41.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.215+0000 7faa61e00640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7faa440b3d10 0x7faa440b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:41.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.216+0000 7faa61e00640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7faa440b3d10 0x7faa440b6220 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7faa50005e10 tx=0x7faa50005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:41.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.219+0000 7faa56a00640 1 -- 172.21.15.97:0/49662182 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7faa5c1a6a30 con 0x7faa5c077cc0 2024-07-26T07:49:41.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.429+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7faa5c1a14f0 con 0x7faa5c077cc0 2024-07-26T07:49:41.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.430+0000 7faa56a00640 1 -- 172.21.15.97:0/49662182 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7faa5c1a14f0 con 0x7faa5c077cc0 2024-07-26T07:49:41.431 INFO:teuthology.orchestra.run.smithi097.stdout:188978561100 2024-07-26T07:49:41.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7faa440b3d10 msgr2=0x7faa440b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7faa440b3d10 0x7faa440b6220 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7faa50005e10 tx=0x7faa50005da0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 msgr2=0x7faa5c1a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 0x7faa5c1a0960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faa4c005940 tx=0x7faa4c0042f0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 shutdown_connections 2024-07-26T07:49:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7faa440b3d10 0x7faa440b6220 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa5c1141a0 0x7faa5c1a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa5c077cc0 0x7faa5c1a0960 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 --2- 172.21.15.97:0/49662182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa5c077370 0x7faa5c1a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 >> 172.21.15.97:0/49662182 conn(0x7faa5c1011f0 msgr2=0x7faa5c103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 shutdown_connections 2024-07-26T07:49:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:41.433+0000 7faa64800640 1 -- 172.21.15.97:0/49662182 wait complete. 2024-07-26T07:49:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:41 smithi115 ceph-mon[35364]: pgmap v447: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T07:49:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:41 smithi181 ceph-mon[35869]: pgmap v447: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T07:49:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:41 smithi097 ceph-mon[32231]: pgmap v447: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T07:49:41.907 INFO:teuthology.orchestra.run.smithi097.stdout:111669149791 2024-07-26T07:49:41.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-07-26T07:49:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:42 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/49662182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T07:49:42.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:42 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/49662182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T07:49:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:42 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/49662182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T07:49:43.008 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757329 got 163208757332 for osd.5 2024-07-26T07:49:43.008 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:43.643 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:43 smithi097 ceph-mon[32231]: pgmap v448: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T07:49:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:43 smithi115 ceph-mon[35364]: pgmap v448: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T07:49:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:43 smithi181 ceph-mon[35869]: pgmap v448: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T07:49:43.787 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579942 got 317827579946 for osd.11 2024-07-26T07:49:43.787 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:44.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.291+0000 7f2638e00640 1 -- 172.21.15.97:0/1752051867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f263410faf0 msgr2=0x7f2634111f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.291+0000 7f2638e00640 1 --2- 172.21.15.97:0/1752051867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f263410faf0 0x7f2634111f30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f262c009ca0 tx=0x7f262c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.291+0000 7f2638e00640 1 -- 172.21.15.97:0/1752051867 shutdown_connections 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.291+0000 7f2638e00640 1 --2- 172.21.15.97:0/1752051867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f263410faf0 0x7f2634111f30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.291+0000 7f2638e00640 1 --2- 172.21.15.97:0/1752051867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2634075b30 0x7f263410f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.291+0000 7f2638e00640 1 --2- 172.21.15.97:0/1752051867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26340751e0 0x7f26340755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.291+0000 7f2638e00640 1 -- 172.21.15.97:0/1752051867 >> 172.21.15.97:0/1752051867 conn(0x7f263407b280 msgr2=0x7f263407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.292+0000 7f2638e00640 1 -- 172.21.15.97:0/1752051867 shutdown_connections 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.292+0000 7f2638e00640 1 -- 172.21.15.97:0/1752051867 wait complete. 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.293+0000 7f2638e00640 1 Processor -- start 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.293+0000 7f2638e00640 1 -- start start 2024-07-26T07:49:44.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.293+0000 7f2638e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26340751e0 0x7f263419bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.293+0000 7f2638e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2634075b30 0x7f263419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.293+0000 7f2638e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263410faf0 0x7f26341a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.293+0000 7f2638e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2634115600 con 0x7f2634075b30 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.293+0000 7f2638e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2634115480 con 0x7f26340751e0 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.293+0000 7f2638e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2634115780 con 0x7f263410faf0 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2634075b30 0x7f263419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2634075b30 0x7f263419c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33924/0 (socket says 172.21.15.97:33924) 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 -- 172.21.15.97:0/2976673517 learned_addr learned my addr 172.21.15.97:0/2976673517 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2631e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26340751e0 0x7f263419bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2632800640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263410faf0 0x7f26341a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:44.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 -- 172.21.15.97:0/2976673517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263410faf0 msgr2=0x7f26341a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:44.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263410faf0 0x7f26341a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 -- 172.21.15.97:0/2976673517 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26340751e0 msgr2=0x7f263419bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:44.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26340751e0 0x7f263419bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 -- 172.21.15.97:0/2976673517 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26341a1f40 con 0x7f2634075b30 2024-07-26T07:49:44.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.294+0000 7f2629400640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2634075b30 0x7f263419c570 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f261c00b450 tx=0x7f261c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:44.705 INFO:teuthology.orchestra.run.smithi097.stdout:77309411432 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.295+0000 7f262b400640 1 -- 172.21.15.97:0/2976673517 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c015020 con 0x7f2634075b30 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.295+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26341a2230 con 0x7f2634075b30 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.295+0000 7f262b400640 1 -- 172.21.15.97:0/2976673517 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f261c011d20 con 0x7f2634075b30 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.295+0000 7f262b400640 1 -- 172.21.15.97:0/2976673517 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c010e10 con 0x7f2634075b30 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.295+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26341a8990 con 0x7f2634075b30 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.296+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26340755f0 con 0x7f2634075b30 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.298+0000 7f262b400640 1 -- 172.21.15.97:0/2976673517 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f261c018020 con 0x7f2634075b30 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.300+0000 7f262b400640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f26180b3c60 0x7f26180b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.300+0000 7f262b400640 1 -- 172.21.15.97:0/2976673517 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f261c0e3970 con 0x7f2634075b30 2024-07-26T07:49:44.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.300+0000 7f2631e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f26180b3c60 0x7f26180b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.301+0000 7f2631e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f26180b3c60 0x7f26180b6170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2624005fd0 tx=0x7f2624005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.304+0000 7f262b400640 1 -- 172.21.15.97:0/2976673517 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f261c01f020 con 0x7f2634075b30 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.511+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f26341a8c50 con 0x7f2634075b30 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.512+0000 7f262b400640 1 -- 172.21.15.97:0/2976673517 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f261c00d040 con 0x7f2634075b30 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f26180b3c60 msgr2=0x7f26180b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f26180b3c60 0x7f26180b6170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2624005fd0 tx=0x7f2624005850 comp rx=0 tx=0).stop 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2634075b30 msgr2=0x7f263419c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2634075b30 0x7f263419c570 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f261c00b450 tx=0x7f261c00b910 comp rx=0 tx=0).stop 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 shutdown_connections 2024-07-26T07:49:44.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f26180b3c60 0x7f26180b6170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263410faf0 0x7f26341a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2634075b30 0x7f263419c570 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 --2- 172.21.15.97:0/2976673517 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26340751e0 0x7f263419bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:44.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.515+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 >> 172.21.15.97:0/2976673517 conn(0x7f263407b280 msgr2=0x7f2634101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:44.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.516+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 shutdown_connections 2024-07-26T07:49:44.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:44.516+0000 7f2638e00640 1 -- 172.21.15.97:0/2976673517 wait complete. 2024-07-26T07:49:45.520 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:45 smithi097 ceph-mon[32231]: pgmap v449: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T07:49:45.521 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:45 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2976673517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T07:49:45.521 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:49:45.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:45 smithi115 ceph-mon[35364]: pgmap v449: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T07:49:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:45 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2976673517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T07:49:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:49:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:45 smithi181 ceph-mon[35869]: pgmap v449: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T07:49:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:45 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2976673517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T07:49:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:49:47.056 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411426 got 77309411432 for osd.2 2024-07-26T07:49:47.056 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:47.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.569+0000 7f1406400640 1 -- 172.21.15.97:0/3839856290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1400108d20 msgr2=0x7f1400109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:47.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.569+0000 7f1406400640 1 --2- 172.21.15.97:0/3839856290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1400108d20 0x7f1400109130 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009900 tx=0x7f13ec02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.569+0000 7f1406400640 1 -- 172.21.15.97:0/3839856290 shutdown_connections 2024-07-26T07:49:47.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.569+0000 7f1406400640 1 --2- 172.21.15.97:0/3839856290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1400109670 0x7f140010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.569+0000 7f1406400640 1 --2- 172.21.15.97:0/3839856290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1400108d20 0x7f1400109130 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.569+0000 7f1406400640 1 --2- 172.21.15.97:0/3839856290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14001033e0 0x7f14001037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.569+0000 7f1406400640 1 -- 172.21.15.97:0/3839856290 >> 172.21.15.97:0/3839856290 conn(0x7f14000fee30 msgr2=0x7f1400101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:47.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.570+0000 7f1406400640 1 -- 172.21.15.97:0/3839856290 shutdown_connections 2024-07-26T07:49:47.572 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:47 smithi097 ceph-mon[32231]: pgmap v450: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:49:47.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.570+0000 7f1406400640 1 -- 172.21.15.97:0/3839856290 wait complete. 2024-07-26T07:49:47.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.571+0000 7f1406400640 1 Processor -- start 2024-07-26T07:49:47.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.572+0000 7f1406400640 1 -- start start 2024-07-26T07:49:47.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.572+0000 7f1406400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14001033e0 0x7f140019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:47.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.572+0000 7f1406400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1400108d20 0x7f140019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:47.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.572+0000 7f1406400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1400109670 0x7f14001a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:47.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.572+0000 7f1406400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1400113200 con 0x7f1400109670 2024-07-26T07:49:47.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.572+0000 7f1406400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1400113080 con 0x7f1400108d20 2024-07-26T07:49:47.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.572+0000 7f1406400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1400113380 con 0x7f14001033e0 2024-07-26T07:49:47.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f1405000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14001033e0 0x7f140019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:47.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.572+0000 7f1405a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1400109670 0x7f14001a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:47.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f1405000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14001033e0 0x7f140019beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:58840/0 (socket says 172.21.15.97:58840) 2024-07-26T07:49:47.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f1405000640 1 -- 172.21.15.97:0/4029573541 learned_addr learned my addr 172.21.15.97:0/4029573541 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:47.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f13ffe00640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1400108d20 0x7f140019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:47.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f1405000640 1 -- 172.21.15.97:0/4029573541 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1400108d20 msgr2=0x7f140019c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:47.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f1405000640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1400108d20 0x7f140019c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f1405000640 1 -- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1400109670 msgr2=0x7f14001a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:47.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f1405000640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1400109670 0x7f14001a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.573+0000 7f1405000640 1 -- 172.21.15.97:0/4029573541 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14001a1e20 con 0x7f14001033e0 2024-07-26T07:49:47.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.574+0000 7f1405000640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14001033e0 0x7f140019beb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f13f400d950 tx=0x7f13f400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:47.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.574+0000 7f13fd600640 1 -- 172.21.15.97:0/4029573541 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4017070 con 0x7f14001033e0 2024-07-26T07:49:47.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.574+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14001a2110 con 0x7f14001033e0 2024-07-26T07:49:47.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.574+0000 7f13fd600640 1 -- 172.21.15.97:0/4029573541 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13f4009c30 con 0x7f14001033e0 2024-07-26T07:49:47.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.574+0000 7f13fd600640 1 -- 172.21.15.97:0/4029573541 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f400f620 con 0x7f14001033e0 2024-07-26T07:49:47.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.574+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14001a8870 con 0x7f14001033e0 2024-07-26T07:49:47.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.575+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14001037f0 con 0x7f14001033e0 2024-07-26T07:49:47.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.578+0000 7f13fd600640 1 -- 172.21.15.97:0/4029573541 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f13f4005030 con 0x7f14001033e0 2024-07-26T07:49:47.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.579+0000 7f13fd600640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13e80b3c60 0x7f13e80b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:47.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.580+0000 7f13fd600640 1 -- 172.21.15.97:0/4029573541 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f13f40e3960 con 0x7f14001033e0 2024-07-26T07:49:47.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.580+0000 7f13ffe00640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13e80b3c60 0x7f13e80b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:47.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.581+0000 7f13ffe00640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13e80b3c60 0x7f13e80b6170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f13ec005940 tx=0x7f13ec0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:47.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.582+0000 7f13fd600640 1 -- 172.21.15.97:0/4029573541 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f13f4009de0 con 0x7f14001033e0 2024-07-26T07:49:47.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:47 smithi115 ceph-mon[35364]: pgmap v450: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:49:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:47 smithi181 ceph-mon[35869]: pgmap v450: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:49:47.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.788+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f14001a8b60 con 0x7f14001033e0 2024-07-26T07:49:47.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.788+0000 7f13fd600640 1 -- 172.21.15.97:0/4029573541 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f13f40aa740 con 0x7f14001033e0 2024-07-26T07:49:47.789 INFO:teuthology.orchestra.run.smithi097.stdout:55834574961 2024-07-26T07:49:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13e80b3c60 msgr2=0x7f13e80b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13e80b3c60 0x7f13e80b6170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f13ec005940 tx=0x7f13ec0058d0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14001033e0 msgr2=0x7f140019beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14001033e0 0x7f140019beb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f13f400d950 tx=0x7f13f400de10 comp rx=0 tx=0).stop 2024-07-26T07:49:47.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 shutdown_connections 2024-07-26T07:49:47.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f13e80b3c60 0x7f13e80b6170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1400109670 0x7f14001a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1400108d20 0x7f140019c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 --2- 172.21.15.97:0/4029573541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14001033e0 0x7f140019beb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:47.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.791+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 >> 172.21.15.97:0/4029573541 conn(0x7f14000fee30 msgr2=0x7f1400107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:47.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.792+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 shutdown_connections 2024-07-26T07:49:47.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:47.792+0000 7f1406400640 1 -- 172.21.15.97:0/4029573541 wait complete. 2024-07-26T07:49:47.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561097 got 188978561100 for osd.6 2024-07-26T07:49:47.891 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:48.846 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.846 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.846 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.846 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.846 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:48 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4029573541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T07:49:48.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776174 got 292057776177 for osd.10 2024-07-26T07:49:48.888 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:48 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4029573541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T07:49:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:48 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4029573541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T07:49:49.709 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574957 got 55834574961 for osd.1 2024-07-26T07:49:49.709 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:49.874 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:49 smithi097 ceph-mon[32231]: pgmap v451: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:49.875 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:49 smithi097 ceph-mon[32231]: mgrmap e18: x(active, since 12m), standbys: z, y 2024-07-26T07:49:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:49 smithi115 ceph-mon[35364]: pgmap v451: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:49 smithi115 ceph-mon[35364]: mgrmap e18: x(active, since 12m), standbys: z, y 2024-07-26T07:49:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:49 smithi181 ceph-mon[35869]: pgmap v451: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:49:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:49 smithi181 ceph-mon[35869]: mgrmap e18: x(active, since 12m), standbys: z, y 2024-07-26T07:49:50.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 -- 172.21.15.97:0/1025125255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec107de0 msgr2=0x7f6bec1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:50.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1025125255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec107de0 0x7f6bec1081f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009900 tx=0x7f6bd802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 -- 172.21.15.97:0/1025125255 shutdown_connections 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1025125255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bec108730 0x7f6bec10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1025125255 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec107de0 0x7f6bec1081f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1025125255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bec1024a0 0x7f6bec1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 -- 172.21.15.97:0/1025125255 >> 172.21.15.97:0/1025125255 conn(0x7f6bec079160 msgr2=0x7f6bec079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 -- 172.21.15.97:0/1025125255 shutdown_connections 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.598+0000 7f6bf3800640 1 -- 172.21.15.97:0/1025125255 wait complete. 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.599+0000 7f6bf3800640 1 Processor -- start 2024-07-26T07:49:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.599+0000 7f6bf3800640 1 -- start start 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf3800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec1024a0 0x7f6bec199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf3800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bec107de0 0x7f6bec19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf3800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bec108730 0x7f6bec19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf3800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec112290 con 0x7f6bec1024a0 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf3800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec112110 con 0x7f6bec108730 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf3800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec112410 con 0x7f6bec107de0 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec1024a0 0x7f6bec199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf0e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec1024a0 0x7f6bec199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59474/0 (socket says 172.21.15.97:59474) 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.600+0000 7f6bf0e00640 1 -- 172.21.15.97:0/1148104034 learned_addr learned my addr 172.21.15.97:0/1148104034 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bebe00640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bec107de0 0x7f6bec19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf1800640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bec108730 0x7f6bec19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf0e00640 1 -- 172.21.15.97:0/1148104034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bec107de0 msgr2=0x7f6bec19a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:50.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf0e00640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bec107de0 0x7f6bec19a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf0e00640 1 -- 172.21.15.97:0/1148104034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bec108730 msgr2=0x7f6bec19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf0e00640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bec108730 0x7f6bec19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf0e00640 1 -- 172.21.15.97:0/1148104034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec19fda0 con 0x7f6bec1024a0 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bebe00640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bec107de0 0x7f6bec19a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf1800640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bec108730 0x7f6bec19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf0e00640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec1024a0 0x7f6bec199e30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6be000c890 tx=0x7f6be000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6be9600640 1 -- 172.21.15.97:0/1148104034 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0004220 con 0x7f6bec1024a0 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6be9600640 1 -- 172.21.15.97:0/1148104034 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6be0007e40 con 0x7f6bec1024a0 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec1a0090 con 0x7f6bec1024a0 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6be9600640 1 -- 172.21.15.97:0/1148104034 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0004a00 con 0x7f6bec1024a0 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.601+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec1a67f0 con 0x7f6bec1024a0 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.603+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec103720 con 0x7f6bec1024a0 2024-07-26T07:49:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.605+0000 7f6be9600640 1 -- 172.21.15.97:0/1148104034 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f6be00079b0 con 0x7f6bec1024a0 2024-07-26T07:49:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.607+0000 7f6be9600640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bd40b3bd0 0x7f6bd40b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.607+0000 7f6be9600640 1 -- 172.21.15.97:0/1148104034 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f6be00e3e40 con 0x7f6bec1024a0 2024-07-26T07:49:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.607+0000 7f6bebe00640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bd40b3bd0 0x7f6bd40b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.608+0000 7f6bebe00640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bd40b3bd0 0x7f6bd40b60e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8006010 tx=0x7f6bd8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.609+0000 7f6be9600640 1 -- 172.21.15.97:0/1148104034 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6be0014030 con 0x7f6bec1024a0 2024-07-26T07:49:50.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.817+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f6bec1028b0 con 0x7f6bec1024a0 2024-07-26T07:49:50.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.818+0000 7f6be9600640 1 -- 172.21.15.97:0/1148104034 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6be00ab430 con 0x7f6bec1024a0 2024-07-26T07:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout:34359738489 2024-07-26T07:49:50.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bd40b3bd0 msgr2=0x7f6bd40b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:50.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bd40b3bd0 0x7f6bd40b60e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8006010 tx=0x7f6bd8000bc0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec1024a0 msgr2=0x7f6bec199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:50.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec1024a0 0x7f6bec199e30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6be000c890 tx=0x7f6be000cd50 comp rx=0 tx=0).stop 2024-07-26T07:49:50.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 shutdown_connections 2024-07-26T07:49:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6bd40b3bd0 0x7f6bd40b60e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bec108730 0x7f6bec19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bec107de0 0x7f6bec19a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 --2- 172.21.15.97:0/1148104034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6bec1024a0 0x7f6bec199e30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.821+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 >> 172.21.15.97:0/1148104034 conn(0x7f6bec079160 msgr2=0x7f6bec06b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.822+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 shutdown_connections 2024-07-26T07:49:50.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.822+0000 7f6bf3800640 1 -- 172.21.15.97:0/1148104034 wait complete. 2024-07-26T07:49:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 -- 172.21.15.97:0/481452822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 msgr2=0x7f29f4109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 --2- 172.21.15.97:0/481452822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 0x7f29f4109300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009900 tx=0x7f29e002a410 comp rx=0 tx=0).stop 2024-07-26T07:49:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 -- 172.21.15.97:0/481452822 shutdown_connections 2024-07-26T07:49:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 --2- 172.21.15.97:0/481452822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f4109840 0x7f29f4110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 --2- 172.21.15.97:0/481452822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 0x7f29f4109300 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 --2- 172.21.15.97:0/481452822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29f41035b0 0x7f29f41039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 -- 172.21.15.97:0/481452822 >> 172.21.15.97:0/481452822 conn(0x7f29f40fefc0 msgr2=0x7f29f4101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 -- 172.21.15.97:0/481452822 shutdown_connections 2024-07-26T07:49:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.937+0000 7f29fa600640 1 -- 172.21.15.97:0/481452822 wait complete. 2024-07-26T07:49:50.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.939+0000 7f29fa600640 1 Processor -- start 2024-07-26T07:49:50.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.939+0000 7f29fa600640 1 -- start start 2024-07-26T07:49:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.939+0000 7f29fa600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29f41035b0 0x7f29f4199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.939+0000 7f29fa600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 0x7f29f419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.939+0000 7f29fa600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f4109840 0x7f29f419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.939+0000 7f29fa600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f29f4113360 con 0x7f29f4108ef0 2024-07-26T07:49:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.939+0000 7f29fa600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f29f41131e0 con 0x7f29f41035b0 2024-07-26T07:49:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.939+0000 7f29fa600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f29f41134e0 con 0x7f29f4109840 2024-07-26T07:49:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.940+0000 7f29f2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 0x7f29f419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.940+0000 7f29f3400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29f41035b0 0x7f29f4199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.940+0000 7f29f3400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29f41035b0 0x7f29f4199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:34580/0 (socket says 172.21.15.97:34580) 2024-07-26T07:49:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.940+0000 7f29f3400640 1 -- 172.21.15.97:0/2384873946 learned_addr learned my addr 172.21.15.97:0/2384873946 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.940+0000 7f29f2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 0x7f29f419a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59508/0 (socket says 172.21.15.97:59508) 2024-07-26T07:49:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.940+0000 7f29f3e00640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f4109840 0x7f29f419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.940+0000 7f29f3400640 1 -- 172.21.15.97:0/2384873946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f4109840 msgr2=0x7f29f419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.940+0000 7f29f3400640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f4109840 0x7f29f419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.941+0000 7f29f3400640 1 -- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 msgr2=0x7f29f419a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.941+0000 7f29f3400640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 0x7f29f419a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.941+0000 7f29f3400640 1 -- 172.21.15.97:0/2384873946 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f419fde0 con 0x7f29f41035b0 2024-07-26T07:49:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.941+0000 7f29f3400640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29f41035b0 0x7f29f4199e70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f29e400c890 tx=0x7f29e400cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.941+0000 7f29ebe00640 1 -- 172.21.15.97:0/2384873946 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4004220 con 0x7f29f41035b0 2024-07-26T07:49:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.941+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f41a00d0 con 0x7f29f41035b0 2024-07-26T07:49:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.941+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f4075020 con 0x7f29f41035b0 2024-07-26T07:49:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.942+0000 7f29ebe00640 1 -- 172.21.15.97:0/2384873946 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29e4007e40 con 0x7f29f41035b0 2024-07-26T07:49:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.942+0000 7f29ebe00640 1 -- 172.21.15.97:0/2384873946 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e401d060 con 0x7f29f41035b0 2024-07-26T07:49:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.943+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d40052c0 con 0x7f29f41035b0 2024-07-26T07:49:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.946+0000 7f29ebe00640 1 -- 172.21.15.97:0/2384873946 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f29e4005020 con 0x7f29f41035b0 2024-07-26T07:49:50.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.947+0000 7f29ebe00640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f29d80b3bd0 0x7f29d80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:50.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.947+0000 7f29ebe00640 1 -- 172.21.15.97:0/2384873946 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f29e40e3c20 con 0x7f29f41035b0 2024-07-26T07:49:50.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.947+0000 7f29f2a00640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f29d80b3bd0 0x7f29d80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:50.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.948+0000 7f29f2a00640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f29d80b3bd0 0x7f29d80b60e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f29e0005940 tx=0x7f29e00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:50.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.959+0000 7f29ebe00640 1 -- 172.21.15.97:0/2384873946 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f29e4014030 con 0x7f29f41035b0 2024-07-26T07:49:51.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.740+0000 7f69a3200640 1 -- 172.21.15.97:0/2214020993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 msgr2=0x7f69a41163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:51.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.740+0000 7f69a3200640 1 --2- 172.21.15.97:0/2214020993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 0x7f69a41163d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f699c009ca0 tx=0x7f699c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.741+0000 7f69a3200640 1 -- 172.21.15.97:0/2214020993 shutdown_connections 2024-07-26T07:49:51.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.741+0000 7f69a3200640 1 --2- 172.21.15.97:0/2214020993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 0x7f69a41163d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.741+0000 7f69a3200640 1 --2- 172.21.15.97:0/2214020993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a4077cc0 0x7f69a4113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.741+0000 7f69a3200640 1 --2- 172.21.15.97:0/2214020993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69a4077370 0x7f69a4077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.741+0000 7f69a3200640 1 -- 172.21.15.97:0/2214020993 >> 172.21.15.97:0/2214020993 conn(0x7f69a41010a0 msgr2=0x7f69a41034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:51.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.741+0000 7f69a3200640 1 -- 172.21.15.97:0/2214020993 shutdown_connections 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.741+0000 7f69a3200640 1 -- 172.21.15.97:0/2214020993 wait complete. 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.742+0000 7f69a3200640 1 Processor -- start 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.742+0000 7f69a3200640 1 -- start start 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a3200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69a4077370 0x7f69a41a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a4077cc0 0x7f69a41a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a3200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 0x7f69a41a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a3200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f69a4119810 con 0x7f69a4113fb0 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a3200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f69a4119690 con 0x7f69a4077370 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f69a4119990 con 0x7f69a4077cc0 2024-07-26T07:49:51.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 0x7f69a41a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 0x7f69a41a5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59496/0 (socket says 172.21.15.97:59496) 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a2800640 1 -- 172.21.15.97:0/609427922 learned_addr learned my addr 172.21.15.97:0/609427922 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a1e00640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69a4077370 0x7f69a41a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.743+0000 7f69a1400640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a4077cc0 0x7f69a41a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.744+0000 7f69a2800640 1 -- 172.21.15.97:0/609427922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a4077cc0 msgr2=0x7f69a41a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.744+0000 7f69a2800640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a4077cc0 0x7f69a41a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.744+0000 7f69a2800640 1 -- 172.21.15.97:0/609427922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69a4077370 msgr2=0x7f69a41a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.744+0000 7f69a2800640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69a4077370 0x7f69a41a0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.744+0000 7f69a2800640 1 -- 172.21.15.97:0/609427922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a41a60b0 con 0x7f69a4113fb0 2024-07-26T07:49:51.162 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.744+0000 7f69a2800640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 0x7f69a41a5a40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f699c031040 tx=0x7f699c032bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.745+0000 7f699aa00640 1 -- 172.21.15.97:0/609427922 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c004040 con 0x7f69a4113fb0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.745+0000 7f699aa00640 1 -- 172.21.15.97:0/609427922 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f699c004670 con 0x7f69a4113fb0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.745+0000 7f699aa00640 1 -- 172.21.15.97:0/609427922 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c030bb0 con 0x7f69a4113fb0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.745+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a41a6340 con 0x7f69a4113fb0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.745+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a41acbc0 con 0x7f69a4113fb0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.746+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69a4078880 con 0x7f69a4113fb0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.748+0000 7f699aa00640 1 -- 172.21.15.97:0/609427922 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f699c0041e0 con 0x7f69a4113fb0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.750+0000 7f699aa00640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69880b3c80 0x7f69880b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.751+0000 7f699aa00640 1 -- 172.21.15.97:0/609427922 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f699c104fb0 con 0x7f69a4113fb0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.751+0000 7f69a1e00640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69880b3c80 0x7f69880b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:51.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.752+0000 7f69a1e00640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69880b3c80 0x7f69880b6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f699400abe0 tx=0x7f699400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:51.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.755+0000 7f699aa00640 1 -- 172.21.15.97:0/609427922 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f699c0cc540 con 0x7f69a4113fb0 2024-07-26T07:49:51.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.970+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f69a410f140 con 0x7f69a4113fb0 2024-07-26T07:49:51.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.970+0000 7f699aa00640 1 -- 172.21.15.97:0/609427922 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f699c0cbe40 con 0x7f69a4113fb0 2024-07-26T07:49:51.164 INFO:teuthology.orchestra.run.smithi097.stdout:137438953565 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69880b3c80 msgr2=0x7f69880b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69880b3c80 0x7f69880b6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f699400abe0 tx=0x7f699400a2d0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 msgr2=0x7f69a41a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 0x7f69a41a5a40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f699c031040 tx=0x7f699c032bc0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 shutdown_connections 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69880b3c80 0x7f69880b6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69a4113fb0 0x7f69a41a5a40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a4077cc0 0x7f69a41a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 --2- 172.21.15.97:0/609427922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69a4077370 0x7f69a41a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 >> 172.21.15.97:0/609427922 conn(0x7f69a41010a0 msgr2=0x7f69a4116060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:51.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 shutdown_connections 2024-07-26T07:49:51.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:50.973+0000 7f69a3200640 1 -- 172.21.15.97:0/609427922 wait complete. 2024-07-26T07:49:51.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.168+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f29d4005880 con 0x7f29f41035b0 2024-07-26T07:49:51.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.169+0000 7f29ebe00640 1 -- 172.21.15.97:0/2384873946 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f29e40aab60 con 0x7f29f41035b0 2024-07-26T07:49:51.170 INFO:teuthology.orchestra.run.smithi097.stdout:240518168642 2024-07-26T07:49:51.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f29d80b3bd0 msgr2=0x7f29d80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:51.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f29d80b3bd0 0x7f29d80b60e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f29e0005940 tx=0x7f29e00058d0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29f41035b0 msgr2=0x7f29f4199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:51.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29f41035b0 0x7f29f4199e70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f29e400c890 tx=0x7f29e400cd50 comp rx=0 tx=0).stop 2024-07-26T07:49:51.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 shutdown_connections 2024-07-26T07:49:51.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f29d80b3bd0 0x7f29d80b60e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f4109840 0x7f29f419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f29f4108ef0 0x7f29f419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 --2- 172.21.15.97:0/2384873946 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29f41035b0 0x7f29f4199e70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:51.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.172+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 >> 172.21.15.97:0/2384873946 conn(0x7f29f40fefc0 msgr2=0x7f29f41013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:51.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.173+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 shutdown_connections 2024-07-26T07:49:51.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:51.173+0000 7f29fa600640 1 -- 172.21.15.97:0/2384873946 wait complete. 2024-07-26T07:49:51.681 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:51 smithi097 ceph-mon[32231]: pgmap v452: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:51.681 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:51 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1148104034' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T07:49:51.681 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:51 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/609427922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T07:49:51.681 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:51 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2384873946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T07:49:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:51 smithi115 ceph-mon[35364]: pgmap v452: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:51 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1148104034' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T07:49:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:51 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/609427922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T07:49:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:51 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2384873946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T07:49:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:51 smithi181 ceph-mon[35869]: pgmap v452: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:49:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:51 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1148104034' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T07:49:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:51 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/609427922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T07:49:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:51 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2384873946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T07:49:52.122 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168636 got 240518168642 for osd.8 2024-07-26T07:49:52.122 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:52.770 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738483 got 34359738489 for osd.0 2024-07-26T07:49:52.770 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:53.068 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953557 got 137438953565 for osd.4 2024-07-26T07:49:53.068 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:53.185 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:53.339 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:53.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.369+0000 7f171f600640 1 -- 172.21.15.97:0/1012144856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718107610 msgr2=0x7f1718107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:53.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.369+0000 7f171f600640 1 --2- 172.21.15.97:0/1012144856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718107610 0x7f1718107af0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1710009cd0 tx=0x7f171002a8f0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.369+0000 7f171f600640 1 -- 172.21.15.97:0/1012144856 shutdown_connections 2024-07-26T07:49:53.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.369+0000 7f171f600640 1 --2- 172.21.15.97:0/1012144856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718107610 0x7f1718107af0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.369+0000 7f171f600640 1 --2- 172.21.15.97:0/1012144856 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1718102b20 0x7f1718107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.369+0000 7f171f600640 1 --2- 172.21.15.97:0/1012144856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17181021d0 0x7f17181025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.369+0000 7f171f600640 1 -- 172.21.15.97:0/1012144856 >> 172.21.15.97:0/1012144856 conn(0x7f1718079160 msgr2=0x7f1718079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:53.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.370+0000 7f171f600640 1 -- 172.21.15.97:0/1012144856 shutdown_connections 2024-07-26T07:49:53.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.370+0000 7f171f600640 1 -- 172.21.15.97:0/1012144856 wait complete. 2024-07-26T07:49:53.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.371+0000 7f171f600640 1 Processor -- start 2024-07-26T07:49:53.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.371+0000 7f171f600640 1 -- start start 2024-07-26T07:49:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f171f600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17181021d0 0x7f17181a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f171f600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718102b20 0x7f17181a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f171f600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1718107610 0x7f17181a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f171f600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f17181199b0 con 0x7f1718102b20 2024-07-26T07:49:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f171f600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1718119830 con 0x7f1718107610 2024-07-26T07:49:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f171f600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1718119b30 con 0x7f17181021d0 2024-07-26T07:49:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f1717e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718102b20 0x7f17181a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f1717e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718102b20 0x7f17181a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59522/0 (socket says 172.21.15.97:59522) 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f1717e00640 1 -- 172.21.15.97:0/580508187 learned_addr learned my addr 172.21.15.97:0/580508187 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.372+0000 7f171cc00640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17181021d0 0x7f17181a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.373+0000 7f171d600640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1718107610 0x7f17181a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.373+0000 7f1717e00640 1 -- 172.21.15.97:0/580508187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17181021d0 msgr2=0x7f17181a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.373+0000 7f1717e00640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17181021d0 0x7f17181a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.373+0000 7f1717e00640 1 -- 172.21.15.97:0/580508187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1718107610 msgr2=0x7f17181a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.373+0000 7f1717e00640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1718107610 0x7f17181a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.373+0000 7f1717e00640 1 -- 172.21.15.97:0/580508187 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17181a6330 con 0x7f1718102b20 2024-07-26T07:49:53.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.373+0000 7f1717e00640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718102b20 0x7f17181a0960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f170800e9f0 tx=0x7f170800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:53.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.374+0000 7f1715600640 1 -- 172.21.15.97:0/580508187 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170800ccc0 con 0x7f1718102b20 2024-07-26T07:49:53.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.374+0000 7f1715600640 1 -- 172.21.15.97:0/580508187 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17080108c0 con 0x7f1718102b20 2024-07-26T07:49:53.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.374+0000 7f1715600640 1 -- 172.21.15.97:0/580508187 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1708018710 con 0x7f1718102b20 2024-07-26T07:49:53.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.374+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17181a6620 con 0x7f1718102b20 2024-07-26T07:49:53.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.374+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17181acd80 con 0x7f1718102b20 2024-07-26T07:49:53.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.376+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f171806af90 con 0x7f1718102b20 2024-07-26T07:49:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.379+0000 7f1715600640 1 -- 172.21.15.97:0/580508187 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f1708010430 con 0x7f1718102b20 2024-07-26T07:49:53.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.381+0000 7f1715600640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f16fc0b3c50 0x7f16fc0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.381+0000 7f1715600640 1 -- 172.21.15.97:0/580508187 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f17080e4180 con 0x7f1718102b20 2024-07-26T07:49:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.381+0000 7f171cc00640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f16fc0b3c50 0x7f16fc0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:53.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.382+0000 7f171cc00640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f16fc0b3c50 0x7f16fc0b6160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1700005d00 tx=0x7f1700005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:53.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.388+0000 7f1715600640 1 -- 172.21.15.97:0/580508187 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1708055600 con 0x7f1718102b20 2024-07-26T07:49:53.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.622+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f17181acfb0 con 0x7f1718102b20 2024-07-26T07:49:53.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.622+0000 7f1715600640 1 -- 172.21.15.97:0/580508187 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f17080ab710 con 0x7f1718102b20 2024-07-26T07:49:53.623 INFO:teuthology.orchestra.run.smithi097.stdout:210453397575 2024-07-26T07:49:53.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f16fc0b3c50 msgr2=0x7f16fc0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f16fc0b3c50 0x7f16fc0b6160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1700005d00 tx=0x7f1700005c50 comp rx=0 tx=0).stop 2024-07-26T07:49:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718102b20 msgr2=0x7f17181a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718102b20 0x7f17181a0960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f170800e9f0 tx=0x7f170800eeb0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 shutdown_connections 2024-07-26T07:49:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f16fc0b3c50 0x7f16fc0b6160 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1718107610 0x7f17181a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1718102b20 0x7f17181a0960 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 --2- 172.21.15.97:0/580508187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17181021d0 0x7f17181a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 >> 172.21.15.97:0/580508187 conn(0x7f1718079160 msgr2=0x7f1718105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 shutdown_connections 2024-07-26T07:49:53.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:53.625+0000 7f171f600640 1 -- 172.21.15.97:0/580508187 wait complete. 2024-07-26T07:49:53.834 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:53 smithi097 ceph-mon[32231]: pgmap v453: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T07:49:53.834 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:53 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/580508187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T07:49:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:53 smithi115 ceph-mon[35364]: pgmap v453: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T07:49:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:53 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/580508187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T07:49:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:53 smithi181 ceph-mon[35869]: pgmap v453: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T07:49:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:53 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/580508187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T07:49:55.310 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397569 got 210453397575 for osd.7 2024-07-26T07:49:55.311 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:55.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:55 smithi097 ceph-mon[32231]: pgmap v454: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T07:49:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:55 smithi115 ceph-mon[35364]: pgmap v454: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T07:49:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:55 smithi181 ceph-mon[35869]: pgmap v454: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.069+0000 7ff554800640 1 -- 172.21.15.97:0/595454131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c10ba70 msgr2=0x7ff54c1122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.069+0000 7ff554800640 1 --2- 172.21.15.97:0/595454131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c10ba70 0x7ff54c1122a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff548009ca0 tx=0x7ff54802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.070+0000 7ff554800640 1 -- 172.21.15.97:0/595454131 shutdown_connections 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.070+0000 7ff554800640 1 --2- 172.21.15.97:0/595454131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c10ba70 0x7ff54c1122a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.070+0000 7ff554800640 1 --2- 172.21.15.97:0/595454131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff54c10b120 0x7ff54c10b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.070+0000 7ff554800640 1 --2- 172.21.15.97:0/595454131 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff54c1057e0 0x7ff54c105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.070+0000 7ff554800640 1 -- 172.21.15.97:0/595454131 >> 172.21.15.97:0/595454131 conn(0x7ff54c1011f0 msgr2=0x7ff54c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.070+0000 7ff554800640 1 -- 172.21.15.97:0/595454131 shutdown_connections 2024-07-26T07:49:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.070+0000 7ff554800640 1 -- 172.21.15.97:0/595454131 wait complete. 2024-07-26T07:49:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff554800640 1 Processor -- start 2024-07-26T07:49:56.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff554800640 1 -- start start 2024-07-26T07:49:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff554800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c1057e0 0x7ff54c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff554800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff54c10b120 0x7ff54c077350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff554800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff54c10ba70 0x7ff54c077a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:56.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff554800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c115620 con 0x7ff54c1057e0 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff554800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c1154a0 con 0x7ff54c10ba70 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff554800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c1157a0 con 0x7ff54c10b120 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff551e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c1057e0 0x7ff54c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff551e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c1057e0 0x7ff54c07b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41498/0 (socket says 172.21.15.97:41498) 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff551e00640 1 -- 172.21.15.97:0/4188807089 learned_addr learned my addr 172.21.15.97:0/4188807089 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff551400640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff54c10b120 0x7ff54c077350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.071+0000 7ff552800640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff54c10ba70 0x7ff54c077a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff551e00640 1 -- 172.21.15.97:0/4188807089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff54c10b120 msgr2=0x7ff54c077350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff551e00640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff54c10b120 0x7ff54c077350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff551e00640 1 -- 172.21.15.97:0/4188807089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff54c10ba70 msgr2=0x7ff54c077a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff551e00640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff54c10ba70 0x7ff54c077a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff551e00640 1 -- 172.21.15.97:0/4188807089 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54c1ac5e0 con 0x7ff54c1057e0 2024-07-26T07:49:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff551e00640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c1057e0 0x7ff54c07b070 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff54000d950 tx=0x7ff54000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:56.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff546a00640 1 -- 172.21.15.97:0/4188807089 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540017030 con 0x7ff54c1057e0 2024-07-26T07:49:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff546a00640 1 -- 172.21.15.97:0/4188807089 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff540004020 con 0x7ff54c1057e0 2024-07-26T07:49:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff54c1ac810 con 0x7ff54c1057e0 2024-07-26T07:49:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.073+0000 7ff546a00640 1 -- 172.21.15.97:0/4188807089 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5400055d0 con 0x7ff54c1057e0 2024-07-26T07:49:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.074+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54c1acd50 con 0x7ff54c1057e0 2024-07-26T07:49:56.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.075+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff54c105bf0 con 0x7ff54c1057e0 2024-07-26T07:49:56.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.076+0000 7ff546a00640 1 -- 172.21.15.97:0/4188807089 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7ff540009a30 con 0x7ff54c1057e0 2024-07-26T07:49:56.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.079+0000 7ff546a00640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5340b3bd0 0x7ff5340b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.079+0000 7ff546a00640 1 -- 172.21.15.97:0/4188807089 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff5400e3e90 con 0x7ff54c1057e0 2024-07-26T07:49:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.079+0000 7ff551400640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5340b3bd0 0x7ff5340b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:56.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.080+0000 7ff551400640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5340b3bd0 0x7ff5340b60e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff53c005e00 tx=0x7ff53c005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:56.083 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.081+0000 7ff546a00640 1 -- 172.21.15.97:0/4188807089 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff5400ab420 con 0x7ff54c1057e0 2024-07-26T07:49:56.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.293+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff54c02e0c0 con 0x7ff54c1057e0 2024-07-26T07:49:56.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.294+0000 7ff546a00640 1 -- 172.21.15.97:0/4188807089 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff5400aad20 con 0x7ff54c1057e0 2024-07-26T07:49:56.295 INFO:teuthology.orchestra.run.smithi097.stdout:111669149795 2024-07-26T07:49:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.296+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5340b3bd0 msgr2=0x7ff5340b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.296+0000 7ff554800640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5340b3bd0 0x7ff5340b60e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff53c005e00 tx=0x7ff53c005c50 comp rx=0 tx=0).stop 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.296+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c1057e0 msgr2=0x7ff54c07b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.296+0000 7ff554800640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c1057e0 0x7ff54c07b070 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff54000d950 tx=0x7ff54000de10 comp rx=0 tx=0).stop 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.297+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 shutdown_connections 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.297+0000 7ff554800640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5340b3bd0 0x7ff5340b60e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.297+0000 7ff554800640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff54c10ba70 0x7ff54c077a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.297+0000 7ff554800640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff54c10b120 0x7ff54c077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.297+0000 7ff554800640 1 --2- 172.21.15.97:0/4188807089 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff54c1057e0 0x7ff54c07b070 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.297+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 >> 172.21.15.97:0/4188807089 conn(0x7ff54c1011f0 msgr2=0x7ff54c1096c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.297+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 shutdown_connections 2024-07-26T07:49:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:56.297+0000 7ff554800640 1 -- 172.21.15.97:0/4188807089 wait complete. 2024-07-26T07:49:56.879 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:56 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4188807089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T07:49:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:56 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4188807089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T07:49:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:56 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4188807089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T07:49:57.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149791 got 111669149795 for osd.3 2024-07-26T07:49:57.175 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:57.785 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:57 smithi097 ceph-mon[32231]: pgmap v455: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:49:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:57 smithi115 ceph-mon[35364]: pgmap v455: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:49:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:57 smithi181 ceph-mon[35869]: pgmap v455: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:49:58.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.293+0000 7f8edb200640 1 -- 172.21.15.97:0/2423099889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed4108ef0 msgr2=0x7f8ed4109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.293+0000 7f8edb200640 1 --2- 172.21.15.97:0/2423099889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed4108ef0 0x7f8ed4109300 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009900 tx=0x7f8ebc02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.293+0000 7f8edb200640 1 -- 172.21.15.97:0/2423099889 shutdown_connections 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.293+0000 7f8edb200640 1 --2- 172.21.15.97:0/2423099889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed4109840 0x7f8ed4110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.293+0000 7f8edb200640 1 --2- 172.21.15.97:0/2423099889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed4108ef0 0x7f8ed4109300 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.293+0000 7f8edb200640 1 --2- 172.21.15.97:0/2423099889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ed41035b0 0x7f8ed41039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.293+0000 7f8edb200640 1 -- 172.21.15.97:0/2423099889 >> 172.21.15.97:0/2423099889 conn(0x7f8ed40fefc0 msgr2=0x7f8ed4101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.294+0000 7f8edb200640 1 -- 172.21.15.97:0/2423099889 shutdown_connections 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.294+0000 7f8edb200640 1 -- 172.21.15.97:0/2423099889 wait complete. 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8edb200640 1 Processor -- start 2024-07-26T07:49:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8edb200640 1 -- start start 2024-07-26T07:49:58.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8edb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed41035b0 0x7f8ed4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:58.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8edb200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed4108ef0 0x7f8ed4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:58.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8edb200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ed4109840 0x7f8ed4075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:58.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8edb200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41133b0 con 0x7f8ed41035b0 2024-07-26T07:49:58.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8edb200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed4113230 con 0x7f8ed4109840 2024-07-26T07:49:58.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8edb200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed4113530 con 0x7f8ed4108ef0 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.295+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed41035b0 0x7f8ed4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed41035b0 0x7f8ed4075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41514/0 (socket says 172.21.15.97:41514) 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed3e00640 1 -- 172.21.15.97:0/154142698 learned_addr learned my addr 172.21.15.97:0/154142698 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed9200640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ed4109840 0x7f8ed4075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ecb400640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed4108ef0 0x7f8ed4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed3e00640 1 -- 172.21.15.97:0/154142698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed4108ef0 msgr2=0x7f8ed4075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed3e00640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed4108ef0 0x7f8ed4075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed3e00640 1 -- 172.21.15.97:0/154142698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ed4109840 msgr2=0x7f8ed4075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed3e00640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ed4109840 0x7f8ed4075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed3e00640 1 -- 172.21.15.97:0/154142698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed41a8120 con 0x7f8ed41035b0 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.296+0000 7f8ed9200640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ed4109840 0x7f8ed4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.297+0000 7f8ed3e00640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed41035b0 0x7f8ed4075140 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400e9f0 tx=0x7f8ec400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:58.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.297+0000 7f8ed1600640 1 -- 172.21.15.97:0/154142698 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec400cd80 con 0x7f8ed41035b0 2024-07-26T07:49:58.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.297+0000 7f8ed1600640 1 -- 172.21.15.97:0/154142698 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ec40108c0 con 0x7f8ed41035b0 2024-07-26T07:49:58.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.297+0000 7f8ed1600640 1 -- 172.21.15.97:0/154142698 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec40187c0 con 0x7f8ed41035b0 2024-07-26T07:49:58.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.297+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed41a8410 con 0x7f8ed41035b0 2024-07-26T07:49:58.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.297+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed41a8920 con 0x7f8ed41035b0 2024-07-26T07:49:58.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.298+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed41039c0 con 0x7f8ed41035b0 2024-07-26T07:49:58.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.301+0000 7f8ed1600640 1 -- 172.21.15.97:0/154142698 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f8ec4010430 con 0x7f8ed41035b0 2024-07-26T07:49:58.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.302+0000 7f8ed1600640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8eb80b3bd0 0x7f8eb80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:49:58.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.303+0000 7f8ed1600640 1 -- 172.21.15.97:0/154142698 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f8ec40e4190 con 0x7f8ed41035b0 2024-07-26T07:49:58.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.303+0000 7f8ecb400640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8eb80b3bd0 0x7f8eb80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:49:58.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.304+0000 7f8ecb400640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8eb80b3bd0 0x7f8eb80b60e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc02a910 tx=0x7f8ebc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:49:58.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.307+0000 7f8ed1600640 1 -- 172.21.15.97:0/154142698 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8ec40ab720 con 0x7f8ed41035b0 2024-07-26T07:49:58.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.528+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f8ed402e0c0 con 0x7f8ed41035b0 2024-07-26T07:49:58.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.529+0000 7f8ed1600640 1 -- 172.21.15.97:0/154142698 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8ec40ab020 con 0x7f8ed41035b0 2024-07-26T07:49:58.530 INFO:teuthology.orchestra.run.smithi097.stdout:266287972411 2024-07-26T07:49:58.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.532+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8eb80b3bd0 msgr2=0x7f8eb80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.532+0000 7f8edb200640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8eb80b3bd0 0x7f8eb80b60e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc02a910 tx=0x7f8ebc000bc0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.532+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed41035b0 msgr2=0x7f8ed4075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.532+0000 7f8edb200640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed41035b0 0x7f8ed4075140 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400e9f0 tx=0x7f8ec400eeb0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.532+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 shutdown_connections 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.532+0000 7f8edb200640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8eb80b3bd0 0x7f8eb80b60e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.532+0000 7f8edb200640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ed4109840 0x7f8ed4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.532+0000 7f8edb200640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed4108ef0 0x7f8ed4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.533+0000 7f8edb200640 1 --2- 172.21.15.97:0/154142698 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed41035b0 0x7f8ed4075140 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.533+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 >> 172.21.15.97:0/154142698 conn(0x7f8ed40fefc0 msgr2=0x7f8ed41075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.533+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 shutdown_connections 2024-07-26T07:49:58.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:49:58.533+0000 7f8edb200640 1 -- 172.21.15.97:0/154142698 wait complete. 2024-07-26T07:49:59.194 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972406 got 266287972411 for osd.9 2024-07-26T07:49:59.194 DEBUG:teuthology.parallel:result is None 2024-07-26T07:49:59.194 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T07:49:59.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T07:49:59.630 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:49:59.777 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:59 smithi097 ceph-mon[32231]: pgmap v456: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:49:59.777 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:59.777 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:59.777 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:49:59 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/154142698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T07:49:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:59 smithi115 ceph-mon[35364]: pgmap v456: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:49:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:49:59 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/154142698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T07:49:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:59 smithi181 ceph-mon[35869]: pgmap v456: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:49:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:49:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:49:59 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/154142698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T07:50:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:50:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:50:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:50:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:00 smithi097 ceph-mon[32231]: overall HEALTH_OK 2024-07-26T07:50:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:50:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:50:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:50:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:00 smithi115 ceph-mon[35364]: overall HEALTH_OK 2024-07-26T07:50:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:50:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:50:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:50:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:00 smithi181 ceph-mon[35869]: overall HEALTH_OK 2024-07-26T07:50:01.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 -- 172.21.15.97:0/235264987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc1099f0 msgr2=0x7f63cc109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:01.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 --2- 172.21.15.97:0/235264987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc1099f0 0x7f63cc109e00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009900 tx=0x7f63bc02a410 comp rx=0 tx=0).stop 2024-07-26T07:50:01.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 -- 172.21.15.97:0/235264987 shutdown_connections 2024-07-26T07:50:01.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 --2- 172.21.15.97:0/235264987 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63cc10fc80 0x7f63cc1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:01.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 --2- 172.21.15.97:0/235264987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63cc10f330 0x7f63cc10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:01.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 --2- 172.21.15.97:0/235264987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc1099f0 0x7f63cc109e00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:01.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 -- 172.21.15.97:0/235264987 >> 172.21.15.97:0/235264987 conn(0x7f63cc078ff0 msgr2=0x7f63cc107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:01.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 -- 172.21.15.97:0/235264987 shutdown_connections 2024-07-26T07:50:01.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.774+0000 7f63cbe00640 1 -- 172.21.15.97:0/235264987 wait complete. 2024-07-26T07:50:01.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.776+0000 7f63cbe00640 1 Processor -- start 2024-07-26T07:50:01.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.776+0000 7f63cbe00640 1 -- start start 2024-07-26T07:50:01.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63cc1099f0 0x7f63cc1a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:01.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63cc10f330 0x7f63cc1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:01.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc10fc80 0x7f63cc1a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:01.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc119830 con 0x7f63cc10fc80 2024-07-26T07:50:01.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cbe00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1196b0 con 0x7f63cc1099f0 2024-07-26T07:50:01.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cbe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1199b0 con 0x7f63cc10f330 2024-07-26T07:50:01.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc10fc80 0x7f63cc1a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:01.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc10fc80 0x7f63cc1a5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41536/0 (socket says 172.21.15.97:41536) 2024-07-26T07:50:01.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63cb400640 1 -- 172.21.15.97:0/3323746315 learned_addr learned my addr 172.21.15.97:0/3323746315 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:01.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63caa00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63cc1099f0 0x7f63cc1a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:01.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.777+0000 7f63ca000640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63cc10f330 0x7f63cc1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:01.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.778+0000 7f63cb400640 1 -- 172.21.15.97:0/3323746315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63cc10f330 msgr2=0x7f63cc1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:01.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.778+0000 7f63cb400640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63cc10f330 0x7f63cc1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:01.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.778+0000 7f63cb400640 1 -- 172.21.15.97:0/3323746315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63cc1099f0 msgr2=0x7f63cc1a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.778+0000 7f63cb400640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63cc1099f0 0x7f63cc1a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.778+0000 7f63cb400640 1 -- 172.21.15.97:0/3323746315 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63cc1a6270 con 0x7f63cc10fc80 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.778+0000 7f63caa00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63cc1099f0 0x7f63cc1a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.778+0000 7f63cb400640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc10fc80 0x7f63cc1a5b40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f63c400cad0 tx=0x7f63c400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.778+0000 7f63c3400640 1 -- 172.21.15.97:0/3323746315 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c400ae20 con 0x7f63cc10fc80 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.779+0000 7f63c3400640 1 -- 172.21.15.97:0/3323746315 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63c4005020 con 0x7f63cc10fc80 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.779+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63cc1a6560 con 0x7f63cc10fc80 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.779+0000 7f63c3400640 1 -- 172.21.15.97:0/3323746315 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c400f6b0 con 0x7f63cc10fc80 2024-07-26T07:50:01.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.779+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63cc1accc0 con 0x7f63cc10fc80 2024-07-26T07:50:01.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.780+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63cc109e00 con 0x7f63cc10fc80 2024-07-26T07:50:01.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.782+0000 7f63c3400640 1 -- 172.21.15.97:0/3323746315 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f63c40051d0 con 0x7f63cc10fc80 2024-07-26T07:50:01.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.784+0000 7f63c3400640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63b00b3c50 0x7f63b00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.784+0000 7f63c3400640 1 -- 172.21.15.97:0/3323746315 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f63c40e40b0 con 0x7f63cc10fc80 2024-07-26T07:50:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.784+0000 7f63caa00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63b00b3c50 0x7f63b00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:01.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.785+0000 7f63caa00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63b00b3c50 0x7f63b00b6160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f63bc000c00 tx=0x7f63bc0058b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:01.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.788+0000 7f63c3400640 1 -- 172.21.15.97:0/3323746315 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f63c4014030 con 0x7f63cc10fc80 2024-07-26T07:50:01.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:01 smithi181 ceph-mon[35869]: pgmap v457: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:50:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:01 smithi115 ceph-mon[35364]: pgmap v457: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:50:01.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:01.988+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63cc063230 con 0x7f63b00b3c50 2024-07-26T07:50:01.990 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:01 smithi097 ceph-mon[32231]: pgmap v457: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:50:02.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.006+0000 7f63c3400640 1 -- 172.21.15.97:0/3323746315 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258885 (secure 0 0 0) 0x7f63cc063230 con 0x7f63b00b3c50 2024-07-26T07:50:02.015 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:50:02.016 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.012+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63b00b3c50 msgr2=0x7f63b00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.012+0000 7f63cbe00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63b00b3c50 0x7f63b00b6160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f63bc000c00 tx=0x7f63bc0058b0 comp rx=0 tx=0).stop 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.012+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc10fc80 msgr2=0x7f63cc1a5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc10fc80 0x7f63cc1a5b40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f63c400cad0 tx=0x7f63c400cf90 comp rx=0 tx=0).stop 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 shutdown_connections 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63b00b3c50 0x7f63b00b6160 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63cc10fc80 0x7f63cc1a5b40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63cc10f330 0x7f63cc1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:02.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 --2- 172.21.15.97:0/3323746315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63cc1099f0 0x7f63cc1a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:02.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 >> 172.21.15.97:0/3323746315 conn(0x7f63cc078ff0 msgr2=0x7f63cc079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:02.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 shutdown_connections 2024-07-26T07:50:02.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:02.013+0000 7f63cbe00640 1 -- 172.21.15.97:0/3323746315 wait complete. 2024-07-26T07:50:02.776 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-07-26T07:50:00.174154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657567,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1210,"num_read_kb":704,"num_write":376,"num_write_kb":2291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2953212,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":809,"ondisk_log_size":809,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":404156,"kb_used_data":13404,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537668,"statfs":{"total":1151940427776,"available":1151526572032,"internally_reserved":0,"allocated":13725696,"data_stored":7514781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":62,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005364"},"pg_stats":[{"pgid":"3.3f","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090598+0000","last_change":"2024-07-26T07:47:45.843670+0000","last_active":"2024-07-26T07:49:48.090598+0000","last_peered":"2024-07-26T07:49:48.090598+0000","last_clean":"2024-07-26T07:49:48.090598+0000","last_became_active":"2024-07-26T07:47:45.841137+0000","last_became_peered":"2024-07-26T07:47:45.841137+0000","last_unstale":"2024-07-26T07:49:48.090598+0000","last_undegraded":"2024-07-26T07:49:48.090598+0000","last_fullsized":"2024-07-26T07:49:48.090598+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:38.063503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085904+0000","last_change":"2024-07-26T07:47:45.849326+0000","last_active":"2024-07-26T07:49:48.085904+0000","last_peered":"2024-07-26T07:49:48.085904+0000","last_clean":"2024-07-26T07:49:48.085904+0000","last_became_active":"2024-07-26T07:47:45.847761+0000","last_became_peered":"2024-07-26T07:47:45.847761+0000","last_unstale":"2024-07-26T07:49:48.085904+0000","last_undegraded":"2024-07-26T07:49:48.085904+0000","last_fullsized":"2024-07-26T07:49:48.085904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:31.098794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092147+0000","last_change":"2024-07-26T07:47:45.852343+0000","last_active":"2024-07-26T07:49:48.092147+0000","last_peered":"2024-07-26T07:49:48.092147+0000","last_clean":"2024-07-26T07:49:48.092147+0000","last_became_active":"2024-07-26T07:47:45.851950+0000","last_became_peered":"2024-07-26T07:47:45.851950+0000","last_unstale":"2024-07-26T07:49:48.092147+0000","last_undegraded":"2024-07-26T07:49:48.092147+0000","last_fullsized":"2024-07-26T07:49:48.092147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:41.030903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091026+0000","last_change":"2024-07-26T07:47:45.857042+0000","last_active":"2024-07-26T07:49:48.091026+0000","last_peered":"2024-07-26T07:49:48.091026+0000","last_clean":"2024-07-26T07:49:48.091026+0000","last_became_active":"2024-07-26T07:47:45.855145+0000","last_became_peered":"2024-07-26T07:47:45.855145+0000","last_unstale":"2024-07-26T07:49:48.091026+0000","last_undegraded":"2024-07-26T07:49:48.091026+0000","last_fullsized":"2024-07-26T07:49:48.091026+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:32.233019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091838+0000","last_change":"2024-07-26T07:47:45.838560+0000","last_active":"2024-07-26T07:49:48.091838+0000","last_peered":"2024-07-26T07:49:48.091838+0000","last_clean":"2024-07-26T07:49:48.091838+0000","last_became_active":"2024-07-26T07:47:45.835578+0000","last_became_peered":"2024-07-26T07:47:45.835578+0000","last_unstale":"2024-07-26T07:49:48.091838+0000","last_undegraded":"2024-07-26T07:49:48.091838+0000","last_fullsized":"2024-07-26T07:49:48.091838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:09:40.445625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086316+0000","last_change":"2024-07-26T07:47:45.843980+0000","last_active":"2024-07-26T07:49:48.086316+0000","last_peered":"2024-07-26T07:49:48.086316+0000","last_clean":"2024-07-26T07:49:48.086316+0000","last_became_active":"2024-07-26T07:47:45.843546+0000","last_became_peered":"2024-07-26T07:47:45.843546+0000","last_unstale":"2024-07-26T07:49:48.086316+0000","last_undegraded":"2024-07-26T07:49:48.086316+0000","last_fullsized":"2024-07-26T07:49:48.086316+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:06.839136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091016+0000","last_change":"2024-07-26T07:47:45.854427+0000","last_active":"2024-07-26T07:49:48.091016+0000","last_peered":"2024-07-26T07:49:48.091016+0000","last_clean":"2024-07-26T07:49:48.091016+0000","last_became_active":"2024-07-26T07:47:45.853495+0000","last_became_peered":"2024-07-26T07:47:45.853495+0000","last_unstale":"2024-07-26T07:49:48.091016+0000","last_undegraded":"2024-07-26T07:49:48.091016+0000","last_fullsized":"2024-07-26T07:49:48.091016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:34:34.110581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096645+0000","last_change":"2024-07-26T07:47:45.847732+0000","last_active":"2024-07-26T07:49:48.096645+0000","last_peered":"2024-07-26T07:49:48.096645+0000","last_clean":"2024-07-26T07:49:48.096645+0000","last_became_active":"2024-07-26T07:47:45.845583+0000","last_became_peered":"2024-07-26T07:47:45.845583+0000","last_unstale":"2024-07-26T07:49:48.096645+0000","last_undegraded":"2024-07-26T07:49:48.096645+0000","last_fullsized":"2024-07-26T07:49:48.096645+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:07:09.370076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.106910+0000","last_change":"2024-07-26T07:47:45.849962+0000","last_active":"2024-07-26T07:49:48.106910+0000","last_peered":"2024-07-26T07:49:48.106910+0000","last_clean":"2024-07-26T07:49:48.106910+0000","last_became_active":"2024-07-26T07:47:45.845766+0000","last_became_peered":"2024-07-26T07:47:45.845766+0000","last_unstale":"2024-07-26T07:49:48.106910+0000","last_undegraded":"2024-07-26T07:49:48.106910+0000","last_fullsized":"2024-07-26T07:49:48.106910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:56.602825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086092+0000","last_change":"2024-07-26T07:47:45.857287+0000","last_active":"2024-07-26T07:49:48.086092+0000","last_peered":"2024-07-26T07:49:48.086092+0000","last_clean":"2024-07-26T07:49:48.086092+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.086092+0000","last_undegraded":"2024-07-26T07:49:48.086092+0000","last_fullsized":"2024-07-26T07:49:48.086092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:38:24.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.107067+0000","last_change":"2024-07-26T07:47:45.858369+0000","last_active":"2024-07-26T07:49:48.107067+0000","last_peered":"2024-07-26T07:49:48.107067+0000","last_clean":"2024-07-26T07:49:48.107067+0000","last_became_active":"2024-07-26T07:47:45.856685+0000","last_became_peered":"2024-07-26T07:47:45.856685+0000","last_unstale":"2024-07-26T07:49:48.107067+0000","last_undegraded":"2024-07-26T07:49:48.107067+0000","last_fullsized":"2024-07-26T07:49:48.107067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:44.877953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.083513+0000","last_change":"2024-07-26T07:48:20.001394+0000","last_active":"2024-07-26T07:49:48.083513+0000","last_peered":"2024-07-26T07:49:48.083513+0000","last_clean":"2024-07-26T07:49:48.083513+0000","last_became_active":"2024-07-26T07:48:20.000751+0000","last_became_peered":"2024-07-26T07:48:20.000751+0000","last_unstale":"2024-07-26T07:49:48.083513+0000","last_undegraded":"2024-07-26T07:49:48.083513+0000","last_fullsized":"2024-07-26T07:49:48.083513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:26:24.513652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.097061+0000","last_change":"2024-07-26T07:47:45.847931+0000","last_active":"2024-07-26T07:49:48.097061+0000","last_peered":"2024-07-26T07:49:48.097061+0000","last_clean":"2024-07-26T07:49:48.097061+0000","last_became_active":"2024-07-26T07:47:45.847484+0000","last_became_peered":"2024-07-26T07:47:45.847484+0000","last_unstale":"2024-07-26T07:49:48.097061+0000","last_undegraded":"2024-07-26T07:49:48.097061+0000","last_fullsized":"2024-07-26T07:49:48.097061+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:04.498360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.087054+0000","last_change":"2024-07-26T07:47:45.846484+0000","last_active":"2024-07-26T07:49:48.087054+0000","last_peered":"2024-07-26T07:49:48.087054+0000","last_clean":"2024-07-26T07:49:48.087054+0000","last_became_active":"2024-07-26T07:47:45.844772+0000","last_became_peered":"2024-07-26T07:47:45.844772+0000","last_unstale":"2024-07-26T07:49:48.087054+0000","last_undegraded":"2024-07-26T07:49:48.087054+0000","last_fullsized":"2024-07-26T07:49:48.087054+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:54.318026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088181+0000","last_change":"2024-07-26T07:47:45.849840+0000","last_active":"2024-07-26T07:49:48.088181+0000","last_peered":"2024-07-26T07:49:48.088181+0000","last_clean":"2024-07-26T07:49:48.088181+0000","last_became_active":"2024-07-26T07:47:45.848744+0000","last_became_peered":"2024-07-26T07:47:45.848744+0000","last_unstale":"2024-07-26T07:49:48.088181+0000","last_undegraded":"2024-07-26T07:49:48.088181+0000","last_fullsized":"2024-07-26T07:49:48.088181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:44.675382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.110885+0000","last_change":"2024-07-26T07:47:45.847176+0000","last_active":"2024-07-26T07:49:48.110885+0000","last_peered":"2024-07-26T07:49:48.110885+0000","last_clean":"2024-07-26T07:49:48.110885+0000","last_became_active":"2024-07-26T07:47:45.845936+0000","last_became_peered":"2024-07-26T07:47:45.845936+0000","last_unstale":"2024-07-26T07:49:48.110885+0000","last_undegraded":"2024-07-26T07:49:48.110885+0000","last_fullsized":"2024-07-26T07:49:48.110885+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:02.245098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090829+0000","last_change":"2024-07-26T07:47:45.853210+0000","last_active":"2024-07-26T07:49:48.090829+0000","last_peered":"2024-07-26T07:49:48.090829+0000","last_clean":"2024-07-26T07:49:48.090829+0000","last_became_active":"2024-07-26T07:47:45.852766+0000","last_became_peered":"2024-07-26T07:47:45.852766+0000","last_unstale":"2024-07-26T07:49:48.090829+0000","last_undegraded":"2024-07-26T07:49:48.090829+0000","last_fullsized":"2024-07-26T07:49:48.090829+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:57.406748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096098+0000","last_change":"2024-07-26T07:47:45.850699+0000","last_active":"2024-07-26T07:49:48.096098+0000","last_peered":"2024-07-26T07:49:48.096098+0000","last_clean":"2024-07-26T07:49:48.096098+0000","last_became_active":"2024-07-26T07:47:45.850314+0000","last_became_peered":"2024-07-26T07:47:45.850314+0000","last_unstale":"2024-07-26T07:49:48.096098+0000","last_undegraded":"2024-07-26T07:49:48.096098+0000","last_fullsized":"2024-07-26T07:49:48.096098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:09.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.082161+0000","last_change":"2024-07-26T07:47:45.851063+0000","last_active":"2024-07-26T07:49:48.082161+0000","last_peered":"2024-07-26T07:49:48.082161+0000","last_clean":"2024-07-26T07:49:48.082161+0000","last_became_active":"2024-07-26T07:47:45.850164+0000","last_became_peered":"2024-07-26T07:47:45.850164+0000","last_unstale":"2024-07-26T07:49:48.082161+0000","last_undegraded":"2024-07-26T07:49:48.082161+0000","last_fullsized":"2024-07-26T07:49:48.082161+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:31.597551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088610+0000","last_change":"2024-07-26T07:47:45.861207+0000","last_active":"2024-07-26T07:49:48.088610+0000","last_peered":"2024-07-26T07:49:48.088610+0000","last_clean":"2024-07-26T07:49:48.088610+0000","last_became_active":"2024-07-26T07:47:45.860179+0000","last_became_peered":"2024-07-26T07:47:45.860179+0000","last_unstale":"2024-07-26T07:49:48.088610+0000","last_undegraded":"2024-07-26T07:49:48.088610+0000","last_fullsized":"2024-07-26T07:49:48.088610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:12:51.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101574+0000","last_change":"2024-07-26T07:47:45.846197+0000","last_active":"2024-07-26T07:49:48.101574+0000","last_peered":"2024-07-26T07:49:48.101574+0000","last_clean":"2024-07-26T07:49:48.101574+0000","last_became_active":"2024-07-26T07:47:45.845238+0000","last_became_peered":"2024-07-26T07:47:45.845238+0000","last_unstale":"2024-07-26T07:49:48.101574+0000","last_undegraded":"2024-07-26T07:49:48.101574+0000","last_fullsized":"2024-07-26T07:49:48.101574+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:29.900684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112272+0000","last_change":"2024-07-26T07:47:45.847963+0000","last_active":"2024-07-26T07:49:48.112272+0000","last_peered":"2024-07-26T07:49:48.112272+0000","last_clean":"2024-07-26T07:49:48.112272+0000","last_became_active":"2024-07-26T07:47:45.847513+0000","last_became_peered":"2024-07-26T07:47:45.847513+0000","last_unstale":"2024-07-26T07:49:48.112272+0000","last_undegraded":"2024-07-26T07:49:48.112272+0000","last_fullsized":"2024-07-26T07:49:48.112272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:51.082816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091950+0000","last_change":"2024-07-26T07:47:45.853273+0000","last_active":"2024-07-26T07:49:48.091950+0000","last_peered":"2024-07-26T07:49:48.091950+0000","last_clean":"2024-07-26T07:49:48.091950+0000","last_became_active":"2024-07-26T07:47:45.852894+0000","last_became_peered":"2024-07-26T07:47:45.852894+0000","last_unstale":"2024-07-26T07:49:48.091950+0000","last_undegraded":"2024-07-26T07:49:48.091950+0000","last_fullsized":"2024-07-26T07:49:48.091950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:57:12.945228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092297+0000","last_change":"2024-07-26T07:48:19.931938+0000","last_active":"2024-07-26T07:49:48.092297+0000","last_peered":"2024-07-26T07:49:48.092297+0000","last_clean":"2024-07-26T07:49:48.092297+0000","last_became_active":"2024-07-26T07:48:19.930976+0000","last_became_peered":"2024-07-26T07:48:19.930976+0000","last_unstale":"2024-07-26T07:49:48.092297+0000","last_undegraded":"2024-07-26T07:49:48.092297+0000","last_fullsized":"2024-07-26T07:49:48.092297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:59.610464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090184+0000","last_change":"2024-07-26T07:47:45.849736+0000","last_active":"2024-07-26T07:49:48.090184+0000","last_peered":"2024-07-26T07:49:48.090184+0000","last_clean":"2024-07-26T07:49:48.090184+0000","last_became_active":"2024-07-26T07:47:45.849335+0000","last_became_peered":"2024-07-26T07:47:45.849335+0000","last_unstale":"2024-07-26T07:49:48.090184+0000","last_undegraded":"2024-07-26T07:49:48.090184+0000","last_fullsized":"2024-07-26T07:49:48.090184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:20:12.758331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104678+0000","last_change":"2024-07-26T07:47:45.847250+0000","last_active":"2024-07-26T07:49:48.104678+0000","last_peered":"2024-07-26T07:49:48.104678+0000","last_clean":"2024-07-26T07:49:48.104678+0000","last_became_active":"2024-07-26T07:47:45.845564+0000","last_became_peered":"2024-07-26T07:47:45.845564+0000","last_unstale":"2024-07-26T07:49:48.104678+0000","last_undegraded":"2024-07-26T07:49:48.104678+0000","last_fullsized":"2024-07-26T07:49:48.104678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:03.616782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086629+0000","last_change":"2024-07-26T07:47:45.850462+0000","last_active":"2024-07-26T07:49:48.086629+0000","last_peered":"2024-07-26T07:49:48.086629+0000","last_clean":"2024-07-26T07:49:48.086629+0000","last_became_active":"2024-07-26T07:47:45.850036+0000","last_became_peered":"2024-07-26T07:47:45.850036+0000","last_unstale":"2024-07-26T07:49:48.086629+0000","last_undegraded":"2024-07-26T07:49:48.086629+0000","last_fullsized":"2024-07-26T07:49:48.086629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:38.374782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.094143+0000","last_change":"2024-07-26T07:47:45.859294+0000","last_active":"2024-07-26T07:49:48.094143+0000","last_peered":"2024-07-26T07:49:48.094143+0000","last_clean":"2024-07-26T07:49:48.094143+0000","last_became_active":"2024-07-26T07:47:45.858812+0000","last_became_peered":"2024-07-26T07:47:45.858812+0000","last_unstale":"2024-07-26T07:49:48.094143+0000","last_undegraded":"2024-07-26T07:49:48.094143+0000","last_fullsized":"2024-07-26T07:49:48.094143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:11.689717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085416+0000","last_change":"2024-07-26T07:48:19.930943+0000","last_active":"2024-07-26T07:49:48.085416+0000","last_peered":"2024-07-26T07:49:48.085416+0000","last_clean":"2024-07-26T07:49:48.085416+0000","last_became_active":"2024-07-26T07:48:19.929758+0000","last_became_peered":"2024-07-26T07:48:19.929758+0000","last_unstale":"2024-07-26T07:49:48.085416+0000","last_undegraded":"2024-07-26T07:49:48.085416+0000","last_fullsized":"2024-07-26T07:49:48.085416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:29:27.657376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090071+0000","last_change":"2024-07-26T07:47:45.846867+0000","last_active":"2024-07-26T07:49:48.090071+0000","last_peered":"2024-07-26T07:49:48.090071+0000","last_clean":"2024-07-26T07:49:48.090071+0000","last_became_active":"2024-07-26T07:47:45.846455+0000","last_became_peered":"2024-07-26T07:47:45.846455+0000","last_unstale":"2024-07-26T07:49:48.090071+0000","last_undegraded":"2024-07-26T07:49:48.090071+0000","last_fullsized":"2024-07-26T07:49:48.090071+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:14.997452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092573+0000","last_change":"2024-07-26T07:47:45.846179+0000","last_active":"2024-07-26T07:49:48.092573+0000","last_peered":"2024-07-26T07:49:48.092573+0000","last_clean":"2024-07-26T07:49:48.092573+0000","last_became_active":"2024-07-26T07:47:45.844880+0000","last_became_peered":"2024-07-26T07:47:45.844880+0000","last_unstale":"2024-07-26T07:49:48.092573+0000","last_undegraded":"2024-07-26T07:49:48.092573+0000","last_fullsized":"2024-07-26T07:49:48.092573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:00:50.899573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091429+0000","last_change":"2024-07-26T07:48:19.935196+0000","last_active":"2024-07-26T07:49:48.091429+0000","last_peered":"2024-07-26T07:49:48.091429+0000","last_clean":"2024-07-26T07:49:48.091429+0000","last_became_active":"2024-07-26T07:48:19.933730+0000","last_became_peered":"2024-07-26T07:48:19.933730+0000","last_unstale":"2024-07-26T07:49:48.091429+0000","last_undegraded":"2024-07-26T07:49:48.091429+0000","last_fullsized":"2024-07-26T07:49:48.091429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:10.305122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090368+0000","last_change":"2024-07-26T07:47:45.843360+0000","last_active":"2024-07-26T07:49:48.090368+0000","last_peered":"2024-07-26T07:49:48.090368+0000","last_clean":"2024-07-26T07:49:48.090368+0000","last_became_active":"2024-07-26T07:47:45.842971+0000","last_became_peered":"2024-07-26T07:47:45.842971+0000","last_unstale":"2024-07-26T07:49:48.090368+0000","last_undegraded":"2024-07-26T07:49:48.090368+0000","last_fullsized":"2024-07-26T07:49:48.090368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:58.687638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"82'5","reported_seq":146,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:54.181059+0000","last_change":"2024-07-26T07:47:41.787760+0000","last_active":"2024-07-26T07:49:54.181059+0000","last_peered":"2024-07-26T07:49:54.181059+0000","last_clean":"2024-07-26T07:49:54.181059+0000","last_became_active":"2024-07-26T07:47:41.787197+0000","last_became_peered":"2024-07-26T07:47:41.787197+0000","last_unstale":"2024-07-26T07:49:54.181059+0000","last_undegraded":"2024-07-26T07:49:54.181059+0000","last_fullsized":"2024-07-26T07:49:54.181059+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:14.597759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092352+0000","last_change":"2024-07-26T07:47:45.846822+0000","last_active":"2024-07-26T07:49:48.092352+0000","last_peered":"2024-07-26T07:49:48.092352+0000","last_clean":"2024-07-26T07:49:48.092352+0000","last_became_active":"2024-07-26T07:47:45.846398+0000","last_became_peered":"2024-07-26T07:47:45.846398+0000","last_unstale":"2024-07-26T07:49:48.092352+0000","last_undegraded":"2024-07-26T07:49:48.092352+0000","last_fullsized":"2024-07-26T07:49:48.092352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:33:34.727843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.311772+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:49:20.311772+0000","last_peered":"2024-07-26T07:49:20.311772+0000","last_clean":"2024-07-26T07:49:20.311772+0000","last_became_active":"2024-07-26T07:47:41.782077+0000","last_became_peered":"2024-07-26T07:47:41.782077+0000","last_unstale":"2024-07-26T07:49:20.311772+0000","last_undegraded":"2024-07-26T07:49:20.311772+0000","last_fullsized":"2024-07-26T07:49:20.311772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:42.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429023+0000","last_change":"2024-07-26T07:47:41.784615+0000","last_active":"2024-07-26T07:49:20.429023+0000","last_peered":"2024-07-26T07:49:20.429023+0000","last_clean":"2024-07-26T07:49:20.429023+0000","last_became_active":"2024-07-26T07:47:41.783945+0000","last_became_peered":"2024-07-26T07:47:41.783945+0000","last_unstale":"2024-07-26T07:49:20.429023+0000","last_undegraded":"2024-07-26T07:49:20.429023+0000","last_fullsized":"2024-07-26T07:49:20.429023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:42.618524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095762+0000","last_change":"2024-07-26T07:47:45.838774+0000","last_active":"2024-07-26T07:49:48.095762+0000","last_peered":"2024-07-26T07:49:48.095762+0000","last_clean":"2024-07-26T07:49:48.095762+0000","last_became_active":"2024-07-26T07:47:45.837707+0000","last_became_peered":"2024-07-26T07:47:45.837707+0000","last_unstale":"2024-07-26T07:49:48.095762+0000","last_undegraded":"2024-07-26T07:49:48.095762+0000","last_fullsized":"2024-07-26T07:49:48.095762+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:59.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297370+0000","last_change":"2024-07-26T07:47:41.792031+0000","last_active":"2024-07-26T07:49:20.297370+0000","last_peered":"2024-07-26T07:49:20.297370+0000","last_clean":"2024-07-26T07:49:20.297370+0000","last_became_active":"2024-07-26T07:47:41.785955+0000","last_became_peered":"2024-07-26T07:47:41.785955+0000","last_unstale":"2024-07-26T07:49:20.297370+0000","last_undegraded":"2024-07-26T07:49:20.297370+0000","last_fullsized":"2024-07-26T07:49:20.297370+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:01:47.073970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093965+0000","last_change":"2024-07-26T07:47:45.844595+0000","last_active":"2024-07-26T07:49:48.093965+0000","last_peered":"2024-07-26T07:49:48.093965+0000","last_clean":"2024-07-26T07:49:48.093965+0000","last_became_active":"2024-07-26T07:47:45.843381+0000","last_became_peered":"2024-07-26T07:47:45.843381+0000","last_unstale":"2024-07-26T07:49:48.093965+0000","last_undegraded":"2024-07-26T07:49:48.093965+0000","last_fullsized":"2024-07-26T07:49:48.093965+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:39.692463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'5","reported_seq":256,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:57.189348+0000","last_change":"2024-07-26T07:47:41.788239+0000","last_active":"2024-07-26T07:49:57.189348+0000","last_peered":"2024-07-26T07:49:57.189348+0000","last_clean":"2024-07-26T07:49:57.189348+0000","last_became_active":"2024-07-26T07:47:41.787470+0000","last_became_peered":"2024-07-26T07:47:41.787470+0000","last_unstale":"2024-07-26T07:49:57.189348+0000","last_undegraded":"2024-07-26T07:49:57.189348+0000","last_fullsized":"2024-07-26T07:49:57.189348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:03.129563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":220,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111850+0000","last_change":"2024-07-26T07:47:45.852794+0000","last_active":"2024-07-26T07:49:48.111850+0000","last_peered":"2024-07-26T07:49:48.111850+0000","last_clean":"2024-07-26T07:49:48.111850+0000","last_became_active":"2024-07-26T07:47:45.849699+0000","last_became_peered":"2024-07-26T07:47:45.849699+0000","last_unstale":"2024-07-26T07:49:48.111850+0000","last_undegraded":"2024-07-26T07:49:48.111850+0000","last_fullsized":"2024-07-26T07:49:48.111850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:06:31.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.315365+0000","last_change":"2024-07-26T07:47:41.789414+0000","last_active":"2024-07-26T07:49:20.315365+0000","last_peered":"2024-07-26T07:49:20.315365+0000","last_clean":"2024-07-26T07:49:20.315365+0000","last_became_active":"2024-07-26T07:47:41.788679+0000","last_became_peered":"2024-07-26T07:47:41.788679+0000","last_unstale":"2024-07-26T07:49:20.315365+0000","last_undegraded":"2024-07-26T07:49:20.315365+0000","last_fullsized":"2024-07-26T07:49:20.315365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:51.809997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099189+0000","last_change":"2024-07-26T07:48:19.937603+0000","last_active":"2024-07-26T07:49:48.099189+0000","last_peered":"2024-07-26T07:49:48.099189+0000","last_clean":"2024-07-26T07:49:48.099189+0000","last_became_active":"2024-07-26T07:48:19.936889+0000","last_became_peered":"2024-07-26T07:48:19.936889+0000","last_unstale":"2024-07-26T07:49:48.099189+0000","last_undegraded":"2024-07-26T07:49:48.099189+0000","last_fullsized":"2024-07-26T07:49:48.099189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:46:35.138825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297818+0000","last_change":"2024-07-26T07:47:42.564288+0000","last_active":"2024-07-26T07:49:20.297818+0000","last_peered":"2024-07-26T07:49:20.297818+0000","last_clean":"2024-07-26T07:49:20.297818+0000","last_became_active":"2024-07-26T07:47:42.561766+0000","last_became_peered":"2024-07-26T07:47:42.561766+0000","last_unstale":"2024-07-26T07:49:20.297818+0000","last_undegraded":"2024-07-26T07:49:20.297818+0000","last_fullsized":"2024-07-26T07:49:20.297818+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:16.414030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112002+0000","last_change":"2024-07-26T07:47:45.861895+0000","last_active":"2024-07-26T07:49:48.112002+0000","last_peered":"2024-07-26T07:49:48.112002+0000","last_clean":"2024-07-26T07:49:48.112002+0000","last_became_active":"2024-07-26T07:47:45.861439+0000","last_became_peered":"2024-07-26T07:47:45.861439+0000","last_unstale":"2024-07-26T07:49:48.112002+0000","last_undegraded":"2024-07-26T07:49:48.112002+0000","last_fullsized":"2024-07-26T07:49:48.112002+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:26.276453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'34","reported_seq":295,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:57.206631+0000","last_change":"2024-07-26T07:47:41.789977+0000","last_active":"2024-07-26T07:49:57.206631+0000","last_peered":"2024-07-26T07:49:57.206631+0000","last_clean":"2024-07-26T07:49:57.206631+0000","last_became_active":"2024-07-26T07:47:41.789488+0000","last_became_peered":"2024-07-26T07:47:41.789488+0000","last_unstale":"2024-07-26T07:49:57.206631+0000","last_undegraded":"2024-07-26T07:49:57.206631+0000","last_fullsized":"2024-07-26T07:49:57.206631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:33.059279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60011,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":230,"num_read_kb":173,"num_write":36,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098160+0000","last_change":"2024-07-26T07:47:45.854772+0000","last_active":"2024-07-26T07:49:48.098160+0000","last_peered":"2024-07-26T07:49:48.098160+0000","last_clean":"2024-07-26T07:49:48.098160+0000","last_became_active":"2024-07-26T07:47:45.854354+0000","last_became_peered":"2024-07-26T07:47:45.854354+0000","last_unstale":"2024-07-26T07:49:48.098160+0000","last_undegraded":"2024-07-26T07:49:48.098160+0000","last_fullsized":"2024-07-26T07:49:48.098160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:31:54.856407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313226+0000","last_change":"2024-07-26T07:47:41.789752+0000","last_active":"2024-07-26T07:49:20.313226+0000","last_peered":"2024-07-26T07:49:20.313226+0000","last_clean":"2024-07-26T07:49:20.313226+0000","last_became_active":"2024-07-26T07:47:41.788952+0000","last_became_peered":"2024-07-26T07:47:41.788952+0000","last_unstale":"2024-07-26T07:49:20.313226+0000","last_undegraded":"2024-07-26T07:49:20.313226+0000","last_fullsized":"2024-07-26T07:49:20.313226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:45.141693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101147+0000","last_change":"2024-07-26T07:47:45.845120+0000","last_active":"2024-07-26T07:49:48.101147+0000","last_peered":"2024-07-26T07:49:48.101147+0000","last_clean":"2024-07-26T07:49:48.101147+0000","last_became_active":"2024-07-26T07:47:45.844699+0000","last_became_peered":"2024-07-26T07:47:45.844699+0000","last_unstale":"2024-07-26T07:49:48.101147+0000","last_undegraded":"2024-07-26T07:49:48.101147+0000","last_fullsized":"2024-07-26T07:49:48.101147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:23:42.130846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297374+0000","last_change":"2024-07-26T07:47:41.790765+0000","last_active":"2024-07-26T07:49:20.297374+0000","last_peered":"2024-07-26T07:49:20.297374+0000","last_clean":"2024-07-26T07:49:20.297374+0000","last_became_active":"2024-07-26T07:47:41.790288+0000","last_became_peered":"2024-07-26T07:47:41.790288+0000","last_unstale":"2024-07-26T07:49:20.297374+0000","last_undegraded":"2024-07-26T07:49:20.297374+0000","last_fullsized":"2024-07-26T07:49:20.297374+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:05.734468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.102481+0000","last_change":"2024-07-26T07:47:45.854648+0000","last_active":"2024-07-26T07:49:48.102481+0000","last_peered":"2024-07-26T07:49:48.102481+0000","last_clean":"2024-07-26T07:49:48.102481+0000","last_became_active":"2024-07-26T07:47:45.854240+0000","last_became_peered":"2024-07-26T07:47:45.854240+0000","last_unstale":"2024-07-26T07:49:48.102481+0000","last_undegraded":"2024-07-26T07:49:48.102481+0000","last_fullsized":"2024-07-26T07:49:48.102481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:41.775760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":165,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:58.445026+0000","last_change":"2024-07-26T07:49:20.331153+0000","last_active":"2024-07-26T07:49:58.445026+0000","last_peered":"2024-07-26T07:49:58.445026+0000","last_clean":"2024-07-26T07:49:58.445026+0000","last_became_active":"2024-07-26T07:49:20.315423+0000","last_became_peered":"2024-07-26T07:49:20.315423+0000","last_unstale":"2024-07-26T07:49:58.445026+0000","last_undegraded":"2024-07-26T07:49:58.445026+0000","last_fullsized":"2024-07-26T07:49:58.445026+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:24.476818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.100623+0000","last_change":"2024-07-26T07:47:45.848731+0000","last_active":"2024-07-26T07:49:48.100623+0000","last_peered":"2024-07-26T07:49:48.100623+0000","last_clean":"2024-07-26T07:49:48.100623+0000","last_became_active":"2024-07-26T07:47:45.846663+0000","last_became_peered":"2024-07-26T07:47:45.846663+0000","last_unstale":"2024-07-26T07:49:48.100623+0000","last_undegraded":"2024-07-26T07:49:48.100623+0000","last_fullsized":"2024-07-26T07:49:48.100623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:05.251423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.356788+0000","last_change":"2024-07-26T07:49:20.355238+0000","last_active":"2024-07-26T07:49:20.356788+0000","last_peered":"2024-07-26T07:49:20.356788+0000","last_clean":"2024-07-26T07:49:20.356788+0000","last_became_active":"2024-07-26T07:49:20.316591+0000","last_became_peered":"2024-07-26T07:49:20.316591+0000","last_unstale":"2024-07-26T07:49:20.356788+0000","last_undegraded":"2024-07-26T07:49:20.356788+0000","last_fullsized":"2024-07-26T07:49:20.356788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:12.739473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088493+0000","last_change":"2024-07-26T07:47:45.842572+0000","last_active":"2024-07-26T07:49:48.088493+0000","last_peered":"2024-07-26T07:49:48.088493+0000","last_clean":"2024-07-26T07:49:48.088493+0000","last_became_active":"2024-07-26T07:47:45.842116+0000","last_became_peered":"2024-07-26T07:47:45.842116+0000","last_unstale":"2024-07-26T07:49:48.088493+0000","last_undegraded":"2024-07-26T07:49:48.088493+0000","last_fullsized":"2024-07-26T07:49:48.088493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:06.913737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314657+0000","last_change":"2024-07-26T07:47:41.790215+0000","last_active":"2024-07-26T07:49:20.314657+0000","last_peered":"2024-07-26T07:49:20.314657+0000","last_clean":"2024-07-26T07:49:20.314657+0000","last_became_active":"2024-07-26T07:47:41.789691+0000","last_became_peered":"2024-07-26T07:47:41.789691+0000","last_unstale":"2024-07-26T07:49:20.314657+0000","last_undegraded":"2024-07-26T07:49:20.314657+0000","last_fullsized":"2024-07-26T07:49:20.314657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:01:47.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.082930+0000","last_change":"2024-07-26T07:47:45.860887+0000","last_active":"2024-07-26T07:49:48.082930+0000","last_peered":"2024-07-26T07:49:48.082930+0000","last_clean":"2024-07-26T07:49:48.082930+0000","last_became_active":"2024-07-26T07:47:45.860315+0000","last_became_peered":"2024-07-26T07:47:45.860315+0000","last_unstale":"2024-07-26T07:49:48.082930+0000","last_undegraded":"2024-07-26T07:49:48.082930+0000","last_fullsized":"2024-07-26T07:49:48.082930+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:17:02.231206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429700+0000","last_change":"2024-07-26T07:48:20.022885+0000","last_active":"2024-07-26T07:49:20.429700+0000","last_peered":"2024-07-26T07:49:20.429700+0000","last_clean":"2024-07-26T07:49:20.429700+0000","last_became_active":"2024-07-26T07:48:20.001946+0000","last_became_peered":"2024-07-26T07:48:20.001946+0000","last_unstale":"2024-07-26T07:49:20.429700+0000","last_undegraded":"2024-07-26T07:49:20.429700+0000","last_fullsized":"2024-07-26T07:49:20.429700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:45.892886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098492+0000","last_change":"2024-07-26T07:47:45.858267+0000","last_active":"2024-07-26T07:49:48.098492+0000","last_peered":"2024-07-26T07:49:48.098492+0000","last_clean":"2024-07-26T07:49:48.098492+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.098492+0000","last_undegraded":"2024-07-26T07:49:48.098492+0000","last_fullsized":"2024-07-26T07:49:48.098492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:32.055977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298215+0000","last_change":"2024-07-26T07:47:41.789846+0000","last_active":"2024-07-26T07:49:20.298215+0000","last_peered":"2024-07-26T07:49:20.298215+0000","last_clean":"2024-07-26T07:49:20.298215+0000","last_became_active":"2024-07-26T07:47:41.789237+0000","last_became_peered":"2024-07-26T07:47:41.789237+0000","last_unstale":"2024-07-26T07:49:20.298215+0000","last_undegraded":"2024-07-26T07:49:20.298215+0000","last_fullsized":"2024-07-26T07:49:20.298215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:39.031047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095656+0000","last_change":"2024-07-26T07:47:45.846294+0000","last_active":"2024-07-26T07:49:48.095656+0000","last_peered":"2024-07-26T07:49:48.095656+0000","last_clean":"2024-07-26T07:49:48.095656+0000","last_became_active":"2024-07-26T07:47:45.845861+0000","last_became_peered":"2024-07-26T07:47:45.845861+0000","last_unstale":"2024-07-26T07:49:48.095656+0000","last_undegraded":"2024-07-26T07:49:48.095656+0000","last_fullsized":"2024-07-26T07:49:48.095656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:56:10.594430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'7","reported_seq":150,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:54.183207+0000","last_change":"2024-07-26T07:47:42.563412+0000","last_active":"2024-07-26T07:49:54.183207+0000","last_peered":"2024-07-26T07:49:54.183207+0000","last_clean":"2024-07-26T07:49:54.183207+0000","last_became_active":"2024-07-26T07:47:42.562364+0000","last_became_peered":"2024-07-26T07:47:42.562364+0000","last_unstale":"2024-07-26T07:49:54.183207+0000","last_undegraded":"2024-07-26T07:49:54.183207+0000","last_fullsized":"2024-07-26T07:49:54.183207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:14.796738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.107332+0000","last_change":"2024-07-26T07:47:45.856170+0000","last_active":"2024-07-26T07:49:48.107332+0000","last_peered":"2024-07-26T07:49:48.107332+0000","last_clean":"2024-07-26T07:49:48.107332+0000","last_became_active":"2024-07-26T07:47:45.855782+0000","last_became_peered":"2024-07-26T07:47:45.855782+0000","last_unstale":"2024-07-26T07:49:48.107332+0000","last_undegraded":"2024-07-26T07:49:48.107332+0000","last_fullsized":"2024-07-26T07:49:48.107332+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:22.690910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":110,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313107+0000","last_change":"2024-07-26T07:45:54.310557+0000","last_active":"2024-07-26T07:49:20.313107+0000","last_peered":"2024-07-26T07:49:20.313107+0000","last_clean":"2024-07-26T07:49:20.313107+0000","last_became_active":"2024-07-26T07:45:54.290906+0000","last_became_peered":"2024-07-26T07:45:54.290906+0000","last_unstale":"2024-07-26T07:49:20.313107+0000","last_undegraded":"2024-07-26T07:49:20.313107+0000","last_fullsized":"2024-07-26T07:49:20.313107+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_clean_scrub_stamp":"2024-07-26T07:41:16.689463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:51.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.305857+0000","last_change":"2024-07-26T07:47:41.776420+0000","last_active":"2024-07-26T07:49:20.305857+0000","last_peered":"2024-07-26T07:49:20.305857+0000","last_clean":"2024-07-26T07:49:20.305857+0000","last_became_active":"2024-07-26T07:47:41.775858+0000","last_became_peered":"2024-07-26T07:47:41.775858+0000","last_unstale":"2024-07-26T07:49:20.305857+0000","last_undegraded":"2024-07-26T07:49:20.305857+0000","last_fullsized":"2024-07-26T07:49:20.305857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:18.880331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.109533+0000","last_change":"2024-07-26T07:48:19.933318+0000","last_active":"2024-07-26T07:49:48.109533+0000","last_peered":"2024-07-26T07:49:48.109533+0000","last_clean":"2024-07-26T07:49:48.109533+0000","last_became_active":"2024-07-26T07:48:19.932683+0000","last_became_peered":"2024-07-26T07:48:19.932683+0000","last_unstale":"2024-07-26T07:49:48.109533+0000","last_undegraded":"2024-07-26T07:49:48.109533+0000","last_fullsized":"2024-07-26T07:49:48.109533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:25:23.387428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313228+0000","last_change":"2024-07-26T07:47:41.786698+0000","last_active":"2024-07-26T07:49:20.313228+0000","last_peered":"2024-07-26T07:49:20.313228+0000","last_clean":"2024-07-26T07:49:20.313228+0000","last_became_active":"2024-07-26T07:47:41.784491+0000","last_became_peered":"2024-07-26T07:47:41.784491+0000","last_unstale":"2024-07-26T07:49:20.313228+0000","last_undegraded":"2024-07-26T07:49:20.313228+0000","last_fullsized":"2024-07-26T07:49:20.313228+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:09:08.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093849+0000","last_change":"2024-07-26T07:47:45.849698+0000","last_active":"2024-07-26T07:49:48.093849+0000","last_peered":"2024-07-26T07:49:48.093849+0000","last_clean":"2024-07-26T07:49:48.093849+0000","last_became_active":"2024-07-26T07:47:45.849266+0000","last_became_peered":"2024-07-26T07:47:45.849266+0000","last_unstale":"2024-07-26T07:49:48.093849+0000","last_undegraded":"2024-07-26T07:49:48.093849+0000","last_fullsized":"2024-07-26T07:49:48.093849+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:57.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":149,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:57.440461+0000","last_change":"2024-07-26T07:47:42.558680+0000","last_active":"2024-07-26T07:49:57.440461+0000","last_peered":"2024-07-26T07:49:57.440461+0000","last_clean":"2024-07-26T07:49:57.440461+0000","last_became_active":"2024-07-26T07:47:42.558197+0000","last_became_peered":"2024-07-26T07:47:42.558197+0000","last_unstale":"2024-07-26T07:49:57.440461+0000","last_undegraded":"2024-07-26T07:49:57.440461+0000","last_fullsized":"2024-07-26T07:49:57.440461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:52:52.505455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.078525+0000","last_change":"2024-07-26T07:48:19.929897+0000","last_active":"2024-07-26T07:49:48.078525+0000","last_peered":"2024-07-26T07:49:48.078525+0000","last_clean":"2024-07-26T07:49:48.078525+0000","last_became_active":"2024-07-26T07:48:19.927946+0000","last_became_peered":"2024-07-26T07:48:19.927946+0000","last_unstale":"2024-07-26T07:49:48.078525+0000","last_undegraded":"2024-07-26T07:49:48.078525+0000","last_fullsized":"2024-07-26T07:49:48.078525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:11.157824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"82'5","reported_seq":149,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:57.219657+0000","last_change":"2024-07-26T07:47:41.785513+0000","last_active":"2024-07-26T07:49:57.219657+0000","last_peered":"2024-07-26T07:49:57.219657+0000","last_clean":"2024-07-26T07:49:57.219657+0000","last_became_active":"2024-07-26T07:47:41.784783+0000","last_became_peered":"2024-07-26T07:47:41.784783+0000","last_unstale":"2024-07-26T07:49:57.219657+0000","last_undegraded":"2024-07-26T07:49:57.219657+0000","last_fullsized":"2024-07-26T07:49:57.219657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:54.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.082468+0000","last_change":"2024-07-26T07:47:45.856580+0000","last_active":"2024-07-26T07:49:48.082468+0000","last_peered":"2024-07-26T07:49:48.082468+0000","last_clean":"2024-07-26T07:49:48.082468+0000","last_became_active":"2024-07-26T07:47:45.852330+0000","last_became_peered":"2024-07-26T07:47:45.852330+0000","last_unstale":"2024-07-26T07:49:48.082468+0000","last_undegraded":"2024-07-26T07:49:48.082468+0000","last_fullsized":"2024-07-26T07:49:48.082468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:09.479563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.428822+0000","last_change":"2024-07-26T07:47:42.559255+0000","last_active":"2024-07-26T07:49:20.428822+0000","last_peered":"2024-07-26T07:49:20.428822+0000","last_clean":"2024-07-26T07:49:20.428822+0000","last_became_active":"2024-07-26T07:47:42.558123+0000","last_became_peered":"2024-07-26T07:47:42.558123+0000","last_unstale":"2024-07-26T07:49:20.428822+0000","last_undegraded":"2024-07-26T07:49:20.428822+0000","last_fullsized":"2024-07-26T07:49:20.428822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:49:46.576674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092093+0000","last_change":"2024-07-26T07:47:45.856256+0000","last_active":"2024-07-26T07:49:48.092093+0000","last_peered":"2024-07-26T07:49:48.092093+0000","last_clean":"2024-07-26T07:49:48.092093+0000","last_became_active":"2024-07-26T07:47:45.855696+0000","last_became_peered":"2024-07-26T07:47:45.855696+0000","last_unstale":"2024-07-26T07:49:48.092093+0000","last_undegraded":"2024-07-26T07:49:48.092093+0000","last_fullsized":"2024-07-26T07:49:48.092093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:24.451225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314969+0000","last_change":"2024-07-26T07:47:42.562631+0000","last_active":"2024-07-26T07:49:20.314969+0000","last_peered":"2024-07-26T07:49:20.314969+0000","last_clean":"2024-07-26T07:49:20.314969+0000","last_became_active":"2024-07-26T07:47:42.561379+0000","last_became_peered":"2024-07-26T07:47:42.561379+0000","last_unstale":"2024-07-26T07:49:20.314969+0000","last_undegraded":"2024-07-26T07:49:20.314969+0000","last_fullsized":"2024-07-26T07:49:20.314969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:03:30.471694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092440+0000","last_change":"2024-07-26T07:47:45.849580+0000","last_active":"2024-07-26T07:49:48.092440+0000","last_peered":"2024-07-26T07:49:48.092440+0000","last_clean":"2024-07-26T07:49:48.092440+0000","last_became_active":"2024-07-26T07:47:45.849107+0000","last_became_peered":"2024-07-26T07:47:45.849107+0000","last_unstale":"2024-07-26T07:49:48.092440+0000","last_undegraded":"2024-07-26T07:49:48.092440+0000","last_fullsized":"2024-07-26T07:49:48.092440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:57:13.694059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'16","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.879746+0000","last_change":"2024-07-26T07:47:41.774083+0000","last_active":"2024-07-26T07:49:48.879746+0000","last_peered":"2024-07-26T07:49:48.879746+0000","last_clean":"2024-07-26T07:49:48.879746+0000","last_became_active":"2024-07-26T07:47:41.773502+0000","last_became_peered":"2024-07-26T07:47:41.773502+0000","last_unstale":"2024-07-26T07:49:48.879746+0000","last_undegraded":"2024-07-26T07:49:48.879746+0000","last_fullsized":"2024-07-26T07:49:48.879746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:11:44.083944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099359+0000","last_change":"2024-07-26T07:47:45.845894+0000","last_active":"2024-07-26T07:49:48.099359+0000","last_peered":"2024-07-26T07:49:48.099359+0000","last_clean":"2024-07-26T07:49:48.099359+0000","last_became_active":"2024-07-26T07:47:45.845534+0000","last_became_peered":"2024-07-26T07:47:45.845534+0000","last_unstale":"2024-07-26T07:49:48.099359+0000","last_undegraded":"2024-07-26T07:49:48.099359+0000","last_fullsized":"2024-07-26T07:49:48.099359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:52:42.209561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.306134+0000","last_change":"2024-07-26T07:47:41.904220+0000","last_active":"2024-07-26T07:49:20.306134+0000","last_peered":"2024-07-26T07:49:20.306134+0000","last_clean":"2024-07-26T07:49:20.306134+0000","last_became_active":"2024-07-26T07:47:41.902349+0000","last_became_peered":"2024-07-26T07:47:41.902349+0000","last_unstale":"2024-07-26T07:49:20.306134+0000","last_undegraded":"2024-07-26T07:49:20.306134+0000","last_fullsized":"2024-07-26T07:49:20.306134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:08.520461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092199+0000","last_change":"2024-07-26T07:47:45.846920+0000","last_active":"2024-07-26T07:49:48.092199+0000","last_peered":"2024-07-26T07:49:48.092199+0000","last_clean":"2024-07-26T07:49:48.092199+0000","last_became_active":"2024-07-26T07:47:45.846458+0000","last_became_peered":"2024-07-26T07:47:45.846458+0000","last_unstale":"2024-07-26T07:49:48.092199+0000","last_undegraded":"2024-07-26T07:49:48.092199+0000","last_fullsized":"2024-07-26T07:49:48.092199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:06.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.295007+0000","last_change":"2024-07-26T07:47:42.560346+0000","last_active":"2024-07-26T07:49:20.295007+0000","last_peered":"2024-07-26T07:49:20.295007+0000","last_clean":"2024-07-26T07:49:20.295007+0000","last_became_active":"2024-07-26T07:47:42.558937+0000","last_became_peered":"2024-07-26T07:47:42.558937+0000","last_unstale":"2024-07-26T07:49:20.295007+0000","last_undegraded":"2024-07-26T07:49:20.295007+0000","last_fullsized":"2024-07-26T07:49:20.295007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.676625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.081730+0000","last_change":"2024-07-26T07:47:45.845553+0000","last_active":"2024-07-26T07:49:48.081730+0000","last_peered":"2024-07-26T07:49:48.081730+0000","last_clean":"2024-07-26T07:49:48.081730+0000","last_became_active":"2024-07-26T07:47:45.845125+0000","last_became_peered":"2024-07-26T07:47:45.845125+0000","last_unstale":"2024-07-26T07:49:48.081730+0000","last_undegraded":"2024-07-26T07:49:48.081730+0000","last_fullsized":"2024-07-26T07:49:48.081730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:10.896159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.292557+0000","last_change":"2024-07-26T07:47:41.779688+0000","last_active":"2024-07-26T07:49:20.292557+0000","last_peered":"2024-07-26T07:49:20.292557+0000","last_clean":"2024-07-26T07:49:20.292557+0000","last_became_active":"2024-07-26T07:47:41.779115+0000","last_became_peered":"2024-07-26T07:47:41.779115+0000","last_unstale":"2024-07-26T07:49:20.292557+0000","last_undegraded":"2024-07-26T07:49:20.292557+0000","last_fullsized":"2024-07-26T07:49:20.292557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:20.708336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091558+0000","last_change":"2024-07-26T07:47:45.853869+0000","last_active":"2024-07-26T07:49:48.091558+0000","last_peered":"2024-07-26T07:49:48.091558+0000","last_clean":"2024-07-26T07:49:48.091558+0000","last_became_active":"2024-07-26T07:47:45.852509+0000","last_became_peered":"2024-07-26T07:47:45.852509+0000","last_unstale":"2024-07-26T07:49:48.091558+0000","last_undegraded":"2024-07-26T07:49:48.091558+0000","last_fullsized":"2024-07-26T07:49:48.091558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:01:31.764025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.312497+0000","last_change":"2024-07-26T07:47:41.792331+0000","last_active":"2024-07-26T07:49:20.312497+0000","last_peered":"2024-07-26T07:49:20.312497+0000","last_clean":"2024-07-26T07:49:20.312497+0000","last_became_active":"2024-07-26T07:47:41.790668+0000","last_became_peered":"2024-07-26T07:47:41.790668+0000","last_unstale":"2024-07-26T07:49:20.312497+0000","last_undegraded":"2024-07-26T07:49:20.312497+0000","last_fullsized":"2024-07-26T07:49:20.312497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:59.430692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.089973+0000","last_change":"2024-07-26T07:47:45.843583+0000","last_active":"2024-07-26T07:49:48.089973+0000","last_peered":"2024-07-26T07:49:48.089973+0000","last_clean":"2024-07-26T07:49:48.089973+0000","last_became_active":"2024-07-26T07:47:45.843185+0000","last_became_peered":"2024-07-26T07:47:45.843185+0000","last_unstale":"2024-07-26T07:49:48.089973+0000","last_undegraded":"2024-07-26T07:49:48.089973+0000","last_fullsized":"2024-07-26T07:49:48.089973+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:01.119065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314240+0000","last_change":"2024-07-26T07:48:20.014411+0000","last_active":"2024-07-26T07:49:20.314240+0000","last_peered":"2024-07-26T07:49:20.314240+0000","last_clean":"2024-07-26T07:49:20.314240+0000","last_became_active":"2024-07-26T07:48:20.002085+0000","last_became_peered":"2024-07-26T07:48:20.002085+0000","last_unstale":"2024-07-26T07:49:20.314240+0000","last_undegraded":"2024-07-26T07:49:20.314240+0000","last_fullsized":"2024-07-26T07:49:20.314240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:46:59.335128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.110858+0000","last_change":"2024-07-26T07:47:45.851963+0000","last_active":"2024-07-26T07:49:48.110858+0000","last_peered":"2024-07-26T07:49:48.110858+0000","last_clean":"2024-07-26T07:49:48.110858+0000","last_became_active":"2024-07-26T07:47:45.850459+0000","last_became_peered":"2024-07-26T07:47:45.850459+0000","last_unstale":"2024-07-26T07:49:48.110858+0000","last_undegraded":"2024-07-26T07:49:48.110858+0000","last_fullsized":"2024-07-26T07:49:48.110858+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:18.092186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088035+0000","last_change":"2024-07-26T07:47:45.848744+0000","last_active":"2024-07-26T07:49:48.088035+0000","last_peered":"2024-07-26T07:49:48.088035+0000","last_clean":"2024-07-26T07:49:48.088035+0000","last_became_active":"2024-07-26T07:47:45.848292+0000","last_became_peered":"2024-07-26T07:47:45.848292+0000","last_unstale":"2024-07-26T07:49:48.088035+0000","last_undegraded":"2024-07-26T07:49:48.088035+0000","last_fullsized":"2024-07-26T07:49:48.088035+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:54.549508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314691+0000","last_change":"2024-07-26T07:47:41.788875+0000","last_active":"2024-07-26T07:49:20.314691+0000","last_peered":"2024-07-26T07:49:20.314691+0000","last_clean":"2024-07-26T07:49:20.314691+0000","last_became_active":"2024-07-26T07:47:41.786942+0000","last_became_peered":"2024-07-26T07:47:41.786942+0000","last_unstale":"2024-07-26T07:49:20.314691+0000","last_undegraded":"2024-07-26T07:49:20.314691+0000","last_fullsized":"2024-07-26T07:49:20.314691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:11:31.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111090+0000","last_change":"2024-07-26T07:47:45.854832+0000","last_active":"2024-07-26T07:49:48.111090+0000","last_peered":"2024-07-26T07:49:48.111090+0000","last_clean":"2024-07-26T07:49:48.111090+0000","last_became_active":"2024-07-26T07:47:45.853756+0000","last_became_peered":"2024-07-26T07:47:45.853756+0000","last_unstale":"2024-07-26T07:49:48.111090+0000","last_undegraded":"2024-07-26T07:49:48.111090+0000","last_fullsized":"2024-07-26T07:49:48.111090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:42:14.579177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314716+0000","last_change":"2024-07-26T07:47:41.787839+0000","last_active":"2024-07-26T07:49:20.314716+0000","last_peered":"2024-07-26T07:49:20.314716+0000","last_clean":"2024-07-26T07:49:20.314716+0000","last_became_active":"2024-07-26T07:47:41.785951+0000","last_became_peered":"2024-07-26T07:47:41.785951+0000","last_unstale":"2024-07-26T07:49:20.314716+0000","last_undegraded":"2024-07-26T07:49:20.314716+0000","last_fullsized":"2024-07-26T07:49:20.314716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:58.699056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104028+0000","last_change":"2024-07-26T07:47:45.841451+0000","last_active":"2024-07-26T07:49:48.104028+0000","last_peered":"2024-07-26T07:49:48.104028+0000","last_clean":"2024-07-26T07:49:48.104028+0000","last_became_active":"2024-07-26T07:47:45.840803+0000","last_became_peered":"2024-07-26T07:47:45.840803+0000","last_unstale":"2024-07-26T07:49:48.104028+0000","last_undegraded":"2024-07-26T07:49:48.104028+0000","last_fullsized":"2024-07-26T07:49:48.104028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:24:21.832317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.293557+0000","last_change":"2024-07-26T07:47:42.558775+0000","last_active":"2024-07-26T07:49:20.293557+0000","last_peered":"2024-07-26T07:49:20.293557+0000","last_clean":"2024-07-26T07:49:20.293557+0000","last_became_active":"2024-07-26T07:47:42.558300+0000","last_became_peered":"2024-07-26T07:47:42.558300+0000","last_unstale":"2024-07-26T07:49:20.293557+0000","last_undegraded":"2024-07-26T07:49:20.293557+0000","last_fullsized":"2024-07-26T07:49:20.293557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:59.643709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091602+0000","last_change":"2024-07-26T07:47:45.852401+0000","last_active":"2024-07-26T07:49:48.091602+0000","last_peered":"2024-07-26T07:49:48.091602+0000","last_clean":"2024-07-26T07:49:48.091602+0000","last_became_active":"2024-07-26T07:47:45.852054+0000","last_became_peered":"2024-07-26T07:47:45.852054+0000","last_unstale":"2024-07-26T07:49:48.091602+0000","last_undegraded":"2024-07-26T07:49:48.091602+0000","last_fullsized":"2024-07-26T07:49:48.091602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:25.347770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"82'13","reported_seq":178,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:57.218889+0000","last_change":"2024-07-26T07:48:19.961760+0000","last_active":"2024-07-26T07:49:57.218889+0000","last_peered":"2024-07-26T07:49:57.218889+0000","last_clean":"2024-07-26T07:49:57.218889+0000","last_became_active":"2024-07-26T07:48:19.937694+0000","last_became_peered":"2024-07-26T07:48:19.937694+0000","last_unstale":"2024-07-26T07:49:57.218889+0000","last_undegraded":"2024-07-26T07:49:57.218889+0000","last_fullsized":"2024-07-26T07:49:57.218889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:05:10.571086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66965,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1134,"num_read_kb":640,"num_write":251,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1372,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":200895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579951,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":464264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776181,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1114072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972412,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":462962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168644,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":462556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397577,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1054879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561104,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":462679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757336,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1589248,"data_stored":1055145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953567,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":462540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":462504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":524355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":524275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":464550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T07:50:02.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T07:50:02.855 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:02 smithi097 ceph-mon[32231]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:50:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:02 smithi115 ceph-mon[35364]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:50:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:02 smithi181 ceph-mon[35869]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:50:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:03.684 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:03 smithi097 ceph-mon[32231]: pgmap v458: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T07:50:03.875 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:03 smithi115 ceph-mon[35364]: pgmap v458: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T07:50:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:03 smithi181 ceph-mon[35869]: pgmap v458: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T07:50:05.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 -- 172.21.15.97:0/3755202462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd010b860 msgr2=0x7f4bd0112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:05.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/3755202462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd010b860 0x7f4bd0112090 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8009ca0 tx=0x7f4bc802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 -- 172.21.15.97:0/3755202462 shutdown_connections 2024-07-26T07:50:05.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/3755202462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd010b860 0x7f4bd0112090 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/3755202462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bd010af10 0x7f4bd010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/3755202462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bd01055d0 0x7f4bd01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 -- 172.21.15.97:0/3755202462 >> 172.21.15.97:0/3755202462 conn(0x7f4bd0101040 msgr2=0x7f4bd0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:05.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 -- 172.21.15.97:0/3755202462 shutdown_connections 2024-07-26T07:50:05.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.354+0000 7f4bcfe00640 1 -- 172.21.15.97:0/3755202462 wait complete. 2024-07-26T07:50:05.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.356+0000 7f4bcfe00640 1 Processor -- start 2024-07-26T07:50:05.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.357+0000 7f4bcfe00640 1 -- start start 2024-07-26T07:50:05.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.357+0000 7f4bcfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd01055d0 0x7f4bd019bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:05.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.357+0000 7f4bcfe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bd010af10 0x7f4bd019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.357+0000 7f4bcfe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bd010b860 0x7f4bd01a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.357+0000 7f4bcfe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0115410 con 0x7f4bd01055d0 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.357+0000 7f4bcfe00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0115290 con 0x7f4bd010af10 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.357+0000 7f4bcfe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0115590 con 0x7f4bd010b860 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd01055d0 0x7f4bd019bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd01055d0 0x7f4bd019bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41560/0 (socket says 172.21.15.97:41560) 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcea00640 1 -- 172.21.15.97:0/1446244528 learned_addr learned my addr 172.21.15.97:0/1446244528 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bc7e00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bd010af10 0x7f4bd019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcf400640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bd010b860 0x7f4bd01a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:05.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcea00640 1 -- 172.21.15.97:0/1446244528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bd010b860 msgr2=0x7f4bd01a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:05.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcea00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bd010b860 0x7f4bd01a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcea00640 1 -- 172.21.15.97:0/1446244528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bd010af10 msgr2=0x7f4bd019c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:05.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcea00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bd010af10 0x7f4bd019c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.358+0000 7f4bcea00640 1 -- 172.21.15.97:0/1446244528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd01a1d50 con 0x7f4bd01055d0 2024-07-26T07:50:05.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.359+0000 7f4bcea00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd01055d0 0x7f4bd019bde0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000d950 tx=0x7f4bc000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.359+0000 7f4bc7400640 1 -- 172.21.15.97:0/1446244528 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0017070 con 0x7f4bd01055d0 2024-07-26T07:50:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.359+0000 7f4bc7400640 1 -- 172.21.15.97:0/1446244528 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4bc0009c30 con 0x7f4bd01055d0 2024-07-26T07:50:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.359+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd01a2040 con 0x7f4bd01055d0 2024-07-26T07:50:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.359+0000 7f4bc7400640 1 -- 172.21.15.97:0/1446244528 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc000f620 con 0x7f4bd01055d0 2024-07-26T07:50:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.359+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd01a87a0 con 0x7f4bd01055d0 2024-07-26T07:50:05.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.360+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd01059e0 con 0x7f4bd01055d0 2024-07-26T07:50:05.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.363+0000 7f4bc7400640 1 -- 172.21.15.97:0/1446244528 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f4bc0005030 con 0x7f4bd01055d0 2024-07-26T07:50:05.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.365+0000 7f4bc7400640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4bb40b3bd0 0x7f4bb40b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:05.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.366+0000 7f4bc7400640 1 -- 172.21.15.97:0/1446244528 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f4bc00e38c0 con 0x7f4bd01055d0 2024-07-26T07:50:05.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.366+0000 7f4bc7e00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4bb40b3bd0 0x7f4bb40b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:05.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.366+0000 7f4bc7e00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4bb40b3bd0 0x7f4bb40b60e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4bd010a9e0 tx=0x7f4bb800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.368+0000 7f4bc7400640 1 -- 172.21.15.97:0/1446244528 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4bc00aae50 con 0x7f4bd01055d0 2024-07-26T07:50:05.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.563+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4bd0063230 con 0x7f4bb40b3bd0 2024-07-26T07:50:05.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.575+0000 7f4bc7400640 1 -- 172.21.15.97:0/1446244528 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258885 (secure 0 0 0) 0x7f4bd0063230 con 0x7f4bb40b3bd0 2024-07-26T07:50:05.577 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:50:05.581 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4bb40b3bd0 msgr2=0x7f4bb40b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4bb40b3bd0 0x7f4bb40b60e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4bd010a9e0 tx=0x7f4bb800b390 comp rx=0 tx=0).stop 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd01055d0 msgr2=0x7f4bd019bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd01055d0 0x7f4bd019bde0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000d950 tx=0x7f4bc000de10 comp rx=0 tx=0).stop 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 shutdown_connections 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4bb40b3bd0 0x7f4bb40b60e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bd010b860 0x7f4bd01a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4bd010af10 0x7f4bd019c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 --2- 172.21.15.97:0/1446244528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4bd01055d0 0x7f4bd019bde0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:05.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.582+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 >> 172.21.15.97:0/1446244528 conn(0x7f4bd0101040 msgr2=0x7f4bd0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:05.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.583+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 shutdown_connections 2024-07-26T07:50:05.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:05.583+0000 7f4bcfe00640 1 -- 172.21.15.97:0/1446244528 wait complete. 2024-07-26T07:50:05.960 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:05 smithi097 ceph-mon[32231]: pgmap v459: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T07:50:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:05 smithi115 ceph-mon[35364]: pgmap v459: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T07:50:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:05 smithi181 ceph-mon[35869]: pgmap v459: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T07:50:06.338 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-07-26T07:50:04.176291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657567,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1250,"num_read_kb":724,"num_write":376,"num_write_kb":2291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2953212,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":809,"ondisk_log_size":809,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":404156,"kb_used_data":13404,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537668,"statfs":{"total":1151940427776,"available":1151526572032,"internally_reserved":0,"allocated":13725696,"data_stored":7514781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":71,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005631"},"pg_stats":[{"pgid":"3.3f","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090598+0000","last_change":"2024-07-26T07:47:45.843670+0000","last_active":"2024-07-26T07:49:48.090598+0000","last_peered":"2024-07-26T07:49:48.090598+0000","last_clean":"2024-07-26T07:49:48.090598+0000","last_became_active":"2024-07-26T07:47:45.841137+0000","last_became_peered":"2024-07-26T07:47:45.841137+0000","last_unstale":"2024-07-26T07:49:48.090598+0000","last_undegraded":"2024-07-26T07:49:48.090598+0000","last_fullsized":"2024-07-26T07:49:48.090598+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:38.063503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085904+0000","last_change":"2024-07-26T07:47:45.849326+0000","last_active":"2024-07-26T07:49:48.085904+0000","last_peered":"2024-07-26T07:49:48.085904+0000","last_clean":"2024-07-26T07:49:48.085904+0000","last_became_active":"2024-07-26T07:47:45.847761+0000","last_became_peered":"2024-07-26T07:47:45.847761+0000","last_unstale":"2024-07-26T07:49:48.085904+0000","last_undegraded":"2024-07-26T07:49:48.085904+0000","last_fullsized":"2024-07-26T07:49:48.085904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:31.098794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092147+0000","last_change":"2024-07-26T07:47:45.852343+0000","last_active":"2024-07-26T07:49:48.092147+0000","last_peered":"2024-07-26T07:49:48.092147+0000","last_clean":"2024-07-26T07:49:48.092147+0000","last_became_active":"2024-07-26T07:47:45.851950+0000","last_became_peered":"2024-07-26T07:47:45.851950+0000","last_unstale":"2024-07-26T07:49:48.092147+0000","last_undegraded":"2024-07-26T07:49:48.092147+0000","last_fullsized":"2024-07-26T07:49:48.092147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:41.030903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091026+0000","last_change":"2024-07-26T07:47:45.857042+0000","last_active":"2024-07-26T07:49:48.091026+0000","last_peered":"2024-07-26T07:49:48.091026+0000","last_clean":"2024-07-26T07:49:48.091026+0000","last_became_active":"2024-07-26T07:47:45.855145+0000","last_became_peered":"2024-07-26T07:47:45.855145+0000","last_unstale":"2024-07-26T07:49:48.091026+0000","last_undegraded":"2024-07-26T07:49:48.091026+0000","last_fullsized":"2024-07-26T07:49:48.091026+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:32.233019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091838+0000","last_change":"2024-07-26T07:47:45.838560+0000","last_active":"2024-07-26T07:49:48.091838+0000","last_peered":"2024-07-26T07:49:48.091838+0000","last_clean":"2024-07-26T07:49:48.091838+0000","last_became_active":"2024-07-26T07:47:45.835578+0000","last_became_peered":"2024-07-26T07:47:45.835578+0000","last_unstale":"2024-07-26T07:49:48.091838+0000","last_undegraded":"2024-07-26T07:49:48.091838+0000","last_fullsized":"2024-07-26T07:49:48.091838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:09:40.445625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086316+0000","last_change":"2024-07-26T07:47:45.843980+0000","last_active":"2024-07-26T07:49:48.086316+0000","last_peered":"2024-07-26T07:49:48.086316+0000","last_clean":"2024-07-26T07:49:48.086316+0000","last_became_active":"2024-07-26T07:47:45.843546+0000","last_became_peered":"2024-07-26T07:47:45.843546+0000","last_unstale":"2024-07-26T07:49:48.086316+0000","last_undegraded":"2024-07-26T07:49:48.086316+0000","last_fullsized":"2024-07-26T07:49:48.086316+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:06.839136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091016+0000","last_change":"2024-07-26T07:47:45.854427+0000","last_active":"2024-07-26T07:49:48.091016+0000","last_peered":"2024-07-26T07:49:48.091016+0000","last_clean":"2024-07-26T07:49:48.091016+0000","last_became_active":"2024-07-26T07:47:45.853495+0000","last_became_peered":"2024-07-26T07:47:45.853495+0000","last_unstale":"2024-07-26T07:49:48.091016+0000","last_undegraded":"2024-07-26T07:49:48.091016+0000","last_fullsized":"2024-07-26T07:49:48.091016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:34:34.110581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096645+0000","last_change":"2024-07-26T07:47:45.847732+0000","last_active":"2024-07-26T07:49:48.096645+0000","last_peered":"2024-07-26T07:49:48.096645+0000","last_clean":"2024-07-26T07:49:48.096645+0000","last_became_active":"2024-07-26T07:47:45.845583+0000","last_became_peered":"2024-07-26T07:47:45.845583+0000","last_unstale":"2024-07-26T07:49:48.096645+0000","last_undegraded":"2024-07-26T07:49:48.096645+0000","last_fullsized":"2024-07-26T07:49:48.096645+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:07:09.370076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.106910+0000","last_change":"2024-07-26T07:47:45.849962+0000","last_active":"2024-07-26T07:49:48.106910+0000","last_peered":"2024-07-26T07:49:48.106910+0000","last_clean":"2024-07-26T07:49:48.106910+0000","last_became_active":"2024-07-26T07:47:45.845766+0000","last_became_peered":"2024-07-26T07:47:45.845766+0000","last_unstale":"2024-07-26T07:49:48.106910+0000","last_undegraded":"2024-07-26T07:49:48.106910+0000","last_fullsized":"2024-07-26T07:49:48.106910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:56.602825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086092+0000","last_change":"2024-07-26T07:47:45.857287+0000","last_active":"2024-07-26T07:49:48.086092+0000","last_peered":"2024-07-26T07:49:48.086092+0000","last_clean":"2024-07-26T07:49:48.086092+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.086092+0000","last_undegraded":"2024-07-26T07:49:48.086092+0000","last_fullsized":"2024-07-26T07:49:48.086092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:38:24.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.107067+0000","last_change":"2024-07-26T07:47:45.858369+0000","last_active":"2024-07-26T07:49:48.107067+0000","last_peered":"2024-07-26T07:49:48.107067+0000","last_clean":"2024-07-26T07:49:48.107067+0000","last_became_active":"2024-07-26T07:47:45.856685+0000","last_became_peered":"2024-07-26T07:47:45.856685+0000","last_unstale":"2024-07-26T07:49:48.107067+0000","last_undegraded":"2024-07-26T07:49:48.107067+0000","last_fullsized":"2024-07-26T07:49:48.107067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:44.877953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.083513+0000","last_change":"2024-07-26T07:48:20.001394+0000","last_active":"2024-07-26T07:49:48.083513+0000","last_peered":"2024-07-26T07:49:48.083513+0000","last_clean":"2024-07-26T07:49:48.083513+0000","last_became_active":"2024-07-26T07:48:20.000751+0000","last_became_peered":"2024-07-26T07:48:20.000751+0000","last_unstale":"2024-07-26T07:49:48.083513+0000","last_undegraded":"2024-07-26T07:49:48.083513+0000","last_fullsized":"2024-07-26T07:49:48.083513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:26:24.513652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.097061+0000","last_change":"2024-07-26T07:47:45.847931+0000","last_active":"2024-07-26T07:49:48.097061+0000","last_peered":"2024-07-26T07:49:48.097061+0000","last_clean":"2024-07-26T07:49:48.097061+0000","last_became_active":"2024-07-26T07:47:45.847484+0000","last_became_peered":"2024-07-26T07:47:45.847484+0000","last_unstale":"2024-07-26T07:49:48.097061+0000","last_undegraded":"2024-07-26T07:49:48.097061+0000","last_fullsized":"2024-07-26T07:49:48.097061+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:04.498360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.087054+0000","last_change":"2024-07-26T07:47:45.846484+0000","last_active":"2024-07-26T07:49:48.087054+0000","last_peered":"2024-07-26T07:49:48.087054+0000","last_clean":"2024-07-26T07:49:48.087054+0000","last_became_active":"2024-07-26T07:47:45.844772+0000","last_became_peered":"2024-07-26T07:47:45.844772+0000","last_unstale":"2024-07-26T07:49:48.087054+0000","last_undegraded":"2024-07-26T07:49:48.087054+0000","last_fullsized":"2024-07-26T07:49:48.087054+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:54.318026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088181+0000","last_change":"2024-07-26T07:47:45.849840+0000","last_active":"2024-07-26T07:49:48.088181+0000","last_peered":"2024-07-26T07:49:48.088181+0000","last_clean":"2024-07-26T07:49:48.088181+0000","last_became_active":"2024-07-26T07:47:45.848744+0000","last_became_peered":"2024-07-26T07:47:45.848744+0000","last_unstale":"2024-07-26T07:49:48.088181+0000","last_undegraded":"2024-07-26T07:49:48.088181+0000","last_fullsized":"2024-07-26T07:49:48.088181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:44.675382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.110885+0000","last_change":"2024-07-26T07:47:45.847176+0000","last_active":"2024-07-26T07:49:48.110885+0000","last_peered":"2024-07-26T07:49:48.110885+0000","last_clean":"2024-07-26T07:49:48.110885+0000","last_became_active":"2024-07-26T07:47:45.845936+0000","last_became_peered":"2024-07-26T07:47:45.845936+0000","last_unstale":"2024-07-26T07:49:48.110885+0000","last_undegraded":"2024-07-26T07:49:48.110885+0000","last_fullsized":"2024-07-26T07:49:48.110885+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:02.245098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090829+0000","last_change":"2024-07-26T07:47:45.853210+0000","last_active":"2024-07-26T07:49:48.090829+0000","last_peered":"2024-07-26T07:49:48.090829+0000","last_clean":"2024-07-26T07:49:48.090829+0000","last_became_active":"2024-07-26T07:47:45.852766+0000","last_became_peered":"2024-07-26T07:47:45.852766+0000","last_unstale":"2024-07-26T07:49:48.090829+0000","last_undegraded":"2024-07-26T07:49:48.090829+0000","last_fullsized":"2024-07-26T07:49:48.090829+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:57.406748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096098+0000","last_change":"2024-07-26T07:47:45.850699+0000","last_active":"2024-07-26T07:49:48.096098+0000","last_peered":"2024-07-26T07:49:48.096098+0000","last_clean":"2024-07-26T07:49:48.096098+0000","last_became_active":"2024-07-26T07:47:45.850314+0000","last_became_peered":"2024-07-26T07:47:45.850314+0000","last_unstale":"2024-07-26T07:49:48.096098+0000","last_undegraded":"2024-07-26T07:49:48.096098+0000","last_fullsized":"2024-07-26T07:49:48.096098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:09.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090575+0000","last_change":"2024-07-26T07:47:45.851063+0000","last_active":"2024-07-26T07:49:48.090575+0000","last_peered":"2024-07-26T07:49:48.090575+0000","last_clean":"2024-07-26T07:49:48.090575+0000","last_became_active":"2024-07-26T07:47:45.850164+0000","last_became_peered":"2024-07-26T07:47:45.850164+0000","last_unstale":"2024-07-26T07:49:48.090575+0000","last_undegraded":"2024-07-26T07:49:48.090575+0000","last_fullsized":"2024-07-26T07:49:48.090575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:31.597551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088724+0000","last_change":"2024-07-26T07:47:45.861207+0000","last_active":"2024-07-26T07:49:48.088724+0000","last_peered":"2024-07-26T07:49:48.088724+0000","last_clean":"2024-07-26T07:49:48.088724+0000","last_became_active":"2024-07-26T07:47:45.860179+0000","last_became_peered":"2024-07-26T07:47:45.860179+0000","last_unstale":"2024-07-26T07:49:48.088724+0000","last_undegraded":"2024-07-26T07:49:48.088724+0000","last_fullsized":"2024-07-26T07:49:48.088724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:12:51.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101574+0000","last_change":"2024-07-26T07:47:45.846197+0000","last_active":"2024-07-26T07:49:48.101574+0000","last_peered":"2024-07-26T07:49:48.101574+0000","last_clean":"2024-07-26T07:49:48.101574+0000","last_became_active":"2024-07-26T07:47:45.845238+0000","last_became_peered":"2024-07-26T07:47:45.845238+0000","last_unstale":"2024-07-26T07:49:48.101574+0000","last_undegraded":"2024-07-26T07:49:48.101574+0000","last_fullsized":"2024-07-26T07:49:48.101574+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:29.900684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112421+0000","last_change":"2024-07-26T07:47:45.847963+0000","last_active":"2024-07-26T07:49:48.112421+0000","last_peered":"2024-07-26T07:49:48.112421+0000","last_clean":"2024-07-26T07:49:48.112421+0000","last_became_active":"2024-07-26T07:47:45.847513+0000","last_became_peered":"2024-07-26T07:47:45.847513+0000","last_unstale":"2024-07-26T07:49:48.112421+0000","last_undegraded":"2024-07-26T07:49:48.112421+0000","last_fullsized":"2024-07-26T07:49:48.112421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:51.082816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091950+0000","last_change":"2024-07-26T07:47:45.853273+0000","last_active":"2024-07-26T07:49:48.091950+0000","last_peered":"2024-07-26T07:49:48.091950+0000","last_clean":"2024-07-26T07:49:48.091950+0000","last_became_active":"2024-07-26T07:47:45.852894+0000","last_became_peered":"2024-07-26T07:47:45.852894+0000","last_unstale":"2024-07-26T07:49:48.091950+0000","last_undegraded":"2024-07-26T07:49:48.091950+0000","last_fullsized":"2024-07-26T07:49:48.091950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:57:12.945228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092297+0000","last_change":"2024-07-26T07:48:19.931938+0000","last_active":"2024-07-26T07:49:48.092297+0000","last_peered":"2024-07-26T07:49:48.092297+0000","last_clean":"2024-07-26T07:49:48.092297+0000","last_became_active":"2024-07-26T07:48:19.930976+0000","last_became_peered":"2024-07-26T07:48:19.930976+0000","last_unstale":"2024-07-26T07:49:48.092297+0000","last_undegraded":"2024-07-26T07:49:48.092297+0000","last_fullsized":"2024-07-26T07:49:48.092297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:59.610464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090184+0000","last_change":"2024-07-26T07:47:45.849736+0000","last_active":"2024-07-26T07:49:48.090184+0000","last_peered":"2024-07-26T07:49:48.090184+0000","last_clean":"2024-07-26T07:49:48.090184+0000","last_became_active":"2024-07-26T07:47:45.849335+0000","last_became_peered":"2024-07-26T07:47:45.849335+0000","last_unstale":"2024-07-26T07:49:48.090184+0000","last_undegraded":"2024-07-26T07:49:48.090184+0000","last_fullsized":"2024-07-26T07:49:48.090184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:20:12.758331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104678+0000","last_change":"2024-07-26T07:47:45.847250+0000","last_active":"2024-07-26T07:49:48.104678+0000","last_peered":"2024-07-26T07:49:48.104678+0000","last_clean":"2024-07-26T07:49:48.104678+0000","last_became_active":"2024-07-26T07:47:45.845564+0000","last_became_peered":"2024-07-26T07:47:45.845564+0000","last_unstale":"2024-07-26T07:49:48.104678+0000","last_undegraded":"2024-07-26T07:49:48.104678+0000","last_fullsized":"2024-07-26T07:49:48.104678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:03.616782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086629+0000","last_change":"2024-07-26T07:47:45.850462+0000","last_active":"2024-07-26T07:49:48.086629+0000","last_peered":"2024-07-26T07:49:48.086629+0000","last_clean":"2024-07-26T07:49:48.086629+0000","last_became_active":"2024-07-26T07:47:45.850036+0000","last_became_peered":"2024-07-26T07:47:45.850036+0000","last_unstale":"2024-07-26T07:49:48.086629+0000","last_undegraded":"2024-07-26T07:49:48.086629+0000","last_fullsized":"2024-07-26T07:49:48.086629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:38.374782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.094143+0000","last_change":"2024-07-26T07:47:45.859294+0000","last_active":"2024-07-26T07:49:48.094143+0000","last_peered":"2024-07-26T07:49:48.094143+0000","last_clean":"2024-07-26T07:49:48.094143+0000","last_became_active":"2024-07-26T07:47:45.858812+0000","last_became_peered":"2024-07-26T07:47:45.858812+0000","last_unstale":"2024-07-26T07:49:48.094143+0000","last_undegraded":"2024-07-26T07:49:48.094143+0000","last_fullsized":"2024-07-26T07:49:48.094143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:11.689717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085416+0000","last_change":"2024-07-26T07:48:19.930943+0000","last_active":"2024-07-26T07:49:48.085416+0000","last_peered":"2024-07-26T07:49:48.085416+0000","last_clean":"2024-07-26T07:49:48.085416+0000","last_became_active":"2024-07-26T07:48:19.929758+0000","last_became_peered":"2024-07-26T07:48:19.929758+0000","last_unstale":"2024-07-26T07:49:48.085416+0000","last_undegraded":"2024-07-26T07:49:48.085416+0000","last_fullsized":"2024-07-26T07:49:48.085416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:29:27.657376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090071+0000","last_change":"2024-07-26T07:47:45.846867+0000","last_active":"2024-07-26T07:49:48.090071+0000","last_peered":"2024-07-26T07:49:48.090071+0000","last_clean":"2024-07-26T07:49:48.090071+0000","last_became_active":"2024-07-26T07:47:45.846455+0000","last_became_peered":"2024-07-26T07:47:45.846455+0000","last_unstale":"2024-07-26T07:49:48.090071+0000","last_undegraded":"2024-07-26T07:49:48.090071+0000","last_fullsized":"2024-07-26T07:49:48.090071+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:14.997452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092573+0000","last_change":"2024-07-26T07:47:45.846179+0000","last_active":"2024-07-26T07:49:48.092573+0000","last_peered":"2024-07-26T07:49:48.092573+0000","last_clean":"2024-07-26T07:49:48.092573+0000","last_became_active":"2024-07-26T07:47:45.844880+0000","last_became_peered":"2024-07-26T07:47:45.844880+0000","last_unstale":"2024-07-26T07:49:48.092573+0000","last_undegraded":"2024-07-26T07:49:48.092573+0000","last_fullsized":"2024-07-26T07:49:48.092573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:00:50.899573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091429+0000","last_change":"2024-07-26T07:48:19.935196+0000","last_active":"2024-07-26T07:49:48.091429+0000","last_peered":"2024-07-26T07:49:48.091429+0000","last_clean":"2024-07-26T07:49:48.091429+0000","last_became_active":"2024-07-26T07:48:19.933730+0000","last_became_peered":"2024-07-26T07:48:19.933730+0000","last_unstale":"2024-07-26T07:49:48.091429+0000","last_undegraded":"2024-07-26T07:49:48.091429+0000","last_fullsized":"2024-07-26T07:49:48.091429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:10.305122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090368+0000","last_change":"2024-07-26T07:47:45.843360+0000","last_active":"2024-07-26T07:49:48.090368+0000","last_peered":"2024-07-26T07:49:48.090368+0000","last_clean":"2024-07-26T07:49:48.090368+0000","last_became_active":"2024-07-26T07:47:45.842971+0000","last_became_peered":"2024-07-26T07:47:45.842971+0000","last_unstale":"2024-07-26T07:49:48.090368+0000","last_undegraded":"2024-07-26T07:49:48.090368+0000","last_fullsized":"2024-07-26T07:49:48.090368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:58.687638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"82'5","reported_seq":151,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:59.198189+0000","last_change":"2024-07-26T07:47:41.787760+0000","last_active":"2024-07-26T07:49:59.198189+0000","last_peered":"2024-07-26T07:49:59.198189+0000","last_clean":"2024-07-26T07:49:59.198189+0000","last_became_active":"2024-07-26T07:47:41.787197+0000","last_became_peered":"2024-07-26T07:47:41.787197+0000","last_unstale":"2024-07-26T07:49:59.198189+0000","last_undegraded":"2024-07-26T07:49:59.198189+0000","last_fullsized":"2024-07-26T07:49:59.198189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:14.597759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092352+0000","last_change":"2024-07-26T07:47:45.846822+0000","last_active":"2024-07-26T07:49:48.092352+0000","last_peered":"2024-07-26T07:49:48.092352+0000","last_clean":"2024-07-26T07:49:48.092352+0000","last_became_active":"2024-07-26T07:47:45.846398+0000","last_became_peered":"2024-07-26T07:47:45.846398+0000","last_unstale":"2024-07-26T07:49:48.092352+0000","last_undegraded":"2024-07-26T07:49:48.092352+0000","last_fullsized":"2024-07-26T07:49:48.092352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:33:34.727843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.311772+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:49:20.311772+0000","last_peered":"2024-07-26T07:49:20.311772+0000","last_clean":"2024-07-26T07:49:20.311772+0000","last_became_active":"2024-07-26T07:47:41.782077+0000","last_became_peered":"2024-07-26T07:47:41.782077+0000","last_unstale":"2024-07-26T07:49:20.311772+0000","last_undegraded":"2024-07-26T07:49:20.311772+0000","last_fullsized":"2024-07-26T07:49:20.311772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:42.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429023+0000","last_change":"2024-07-26T07:47:41.784615+0000","last_active":"2024-07-26T07:49:20.429023+0000","last_peered":"2024-07-26T07:49:20.429023+0000","last_clean":"2024-07-26T07:49:20.429023+0000","last_became_active":"2024-07-26T07:47:41.783945+0000","last_became_peered":"2024-07-26T07:47:41.783945+0000","last_unstale":"2024-07-26T07:49:20.429023+0000","last_undegraded":"2024-07-26T07:49:20.429023+0000","last_fullsized":"2024-07-26T07:49:20.429023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:42.618524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095762+0000","last_change":"2024-07-26T07:47:45.838774+0000","last_active":"2024-07-26T07:49:48.095762+0000","last_peered":"2024-07-26T07:49:48.095762+0000","last_clean":"2024-07-26T07:49:48.095762+0000","last_became_active":"2024-07-26T07:47:45.837707+0000","last_became_peered":"2024-07-26T07:47:45.837707+0000","last_unstale":"2024-07-26T07:49:48.095762+0000","last_undegraded":"2024-07-26T07:49:48.095762+0000","last_fullsized":"2024-07-26T07:49:48.095762+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:59.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297370+0000","last_change":"2024-07-26T07:47:41.792031+0000","last_active":"2024-07-26T07:49:20.297370+0000","last_peered":"2024-07-26T07:49:20.297370+0000","last_clean":"2024-07-26T07:49:20.297370+0000","last_became_active":"2024-07-26T07:47:41.785955+0000","last_became_peered":"2024-07-26T07:47:41.785955+0000","last_unstale":"2024-07-26T07:49:20.297370+0000","last_undegraded":"2024-07-26T07:49:20.297370+0000","last_fullsized":"2024-07-26T07:49:20.297370+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:01:47.073970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093965+0000","last_change":"2024-07-26T07:47:45.844595+0000","last_active":"2024-07-26T07:49:48.093965+0000","last_peered":"2024-07-26T07:49:48.093965+0000","last_clean":"2024-07-26T07:49:48.093965+0000","last_became_active":"2024-07-26T07:47:45.843381+0000","last_became_peered":"2024-07-26T07:47:45.843381+0000","last_unstale":"2024-07-26T07:49:48.093965+0000","last_undegraded":"2024-07-26T07:49:48.093965+0000","last_fullsized":"2024-07-26T07:49:48.093965+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:39.692463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'5","reported_seq":266,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:50:02.206846+0000","last_change":"2024-07-26T07:47:41.788239+0000","last_active":"2024-07-26T07:50:02.206846+0000","last_peered":"2024-07-26T07:50:02.206846+0000","last_clean":"2024-07-26T07:50:02.206846+0000","last_became_active":"2024-07-26T07:47:41.787470+0000","last_became_peered":"2024-07-26T07:47:41.787470+0000","last_unstale":"2024-07-26T07:50:02.206846+0000","last_undegraded":"2024-07-26T07:50:02.206846+0000","last_fullsized":"2024-07-26T07:50:02.206846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:03.129563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":230,"num_read_kb":230,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111850+0000","last_change":"2024-07-26T07:47:45.852794+0000","last_active":"2024-07-26T07:49:48.111850+0000","last_peered":"2024-07-26T07:49:48.111850+0000","last_clean":"2024-07-26T07:49:48.111850+0000","last_became_active":"2024-07-26T07:47:45.849699+0000","last_became_peered":"2024-07-26T07:47:45.849699+0000","last_unstale":"2024-07-26T07:49:48.111850+0000","last_undegraded":"2024-07-26T07:49:48.111850+0000","last_fullsized":"2024-07-26T07:49:48.111850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:06:31.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.315365+0000","last_change":"2024-07-26T07:47:41.789414+0000","last_active":"2024-07-26T07:49:20.315365+0000","last_peered":"2024-07-26T07:49:20.315365+0000","last_clean":"2024-07-26T07:49:20.315365+0000","last_became_active":"2024-07-26T07:47:41.788679+0000","last_became_peered":"2024-07-26T07:47:41.788679+0000","last_unstale":"2024-07-26T07:49:20.315365+0000","last_undegraded":"2024-07-26T07:49:20.315365+0000","last_fullsized":"2024-07-26T07:49:20.315365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:51.809997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099189+0000","last_change":"2024-07-26T07:48:19.937603+0000","last_active":"2024-07-26T07:49:48.099189+0000","last_peered":"2024-07-26T07:49:48.099189+0000","last_clean":"2024-07-26T07:49:48.099189+0000","last_became_active":"2024-07-26T07:48:19.936889+0000","last_became_peered":"2024-07-26T07:48:19.936889+0000","last_unstale":"2024-07-26T07:49:48.099189+0000","last_undegraded":"2024-07-26T07:49:48.099189+0000","last_fullsized":"2024-07-26T07:49:48.099189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:46:35.138825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297818+0000","last_change":"2024-07-26T07:47:42.564288+0000","last_active":"2024-07-26T07:49:20.297818+0000","last_peered":"2024-07-26T07:49:20.297818+0000","last_clean":"2024-07-26T07:49:20.297818+0000","last_became_active":"2024-07-26T07:47:42.561766+0000","last_became_peered":"2024-07-26T07:47:42.561766+0000","last_unstale":"2024-07-26T07:49:20.297818+0000","last_undegraded":"2024-07-26T07:49:20.297818+0000","last_fullsized":"2024-07-26T07:49:20.297818+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:16.414030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112002+0000","last_change":"2024-07-26T07:47:45.861895+0000","last_active":"2024-07-26T07:49:48.112002+0000","last_peered":"2024-07-26T07:49:48.112002+0000","last_clean":"2024-07-26T07:49:48.112002+0000","last_became_active":"2024-07-26T07:47:45.861439+0000","last_became_peered":"2024-07-26T07:47:45.861439+0000","last_unstale":"2024-07-26T07:49:48.112002+0000","last_undegraded":"2024-07-26T07:49:48.112002+0000","last_fullsized":"2024-07-26T07:49:48.112002+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:26.276453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'34","reported_seq":306,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:50:02.223683+0000","last_change":"2024-07-26T07:47:41.789977+0000","last_active":"2024-07-26T07:50:02.223683+0000","last_peered":"2024-07-26T07:50:02.223683+0000","last_clean":"2024-07-26T07:50:02.223683+0000","last_became_active":"2024-07-26T07:47:41.789488+0000","last_became_peered":"2024-07-26T07:47:41.789488+0000","last_unstale":"2024-07-26T07:50:02.223683+0000","last_undegraded":"2024-07-26T07:50:02.223683+0000","last_fullsized":"2024-07-26T07:50:02.223683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:33.059279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60011,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":240,"num_read_kb":178,"num_write":36,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098160+0000","last_change":"2024-07-26T07:47:45.854772+0000","last_active":"2024-07-26T07:49:48.098160+0000","last_peered":"2024-07-26T07:49:48.098160+0000","last_clean":"2024-07-26T07:49:48.098160+0000","last_became_active":"2024-07-26T07:47:45.854354+0000","last_became_peered":"2024-07-26T07:47:45.854354+0000","last_unstale":"2024-07-26T07:49:48.098160+0000","last_undegraded":"2024-07-26T07:49:48.098160+0000","last_fullsized":"2024-07-26T07:49:48.098160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:31:54.856407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313226+0000","last_change":"2024-07-26T07:47:41.789752+0000","last_active":"2024-07-26T07:49:20.313226+0000","last_peered":"2024-07-26T07:49:20.313226+0000","last_clean":"2024-07-26T07:49:20.313226+0000","last_became_active":"2024-07-26T07:47:41.788952+0000","last_became_peered":"2024-07-26T07:47:41.788952+0000","last_unstale":"2024-07-26T07:49:20.313226+0000","last_undegraded":"2024-07-26T07:49:20.313226+0000","last_fullsized":"2024-07-26T07:49:20.313226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:45.141693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101147+0000","last_change":"2024-07-26T07:47:45.845120+0000","last_active":"2024-07-26T07:49:48.101147+0000","last_peered":"2024-07-26T07:49:48.101147+0000","last_clean":"2024-07-26T07:49:48.101147+0000","last_became_active":"2024-07-26T07:47:45.844699+0000","last_became_peered":"2024-07-26T07:47:45.844699+0000","last_unstale":"2024-07-26T07:49:48.101147+0000","last_undegraded":"2024-07-26T07:49:48.101147+0000","last_fullsized":"2024-07-26T07:49:48.101147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:23:42.130846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297374+0000","last_change":"2024-07-26T07:47:41.790765+0000","last_active":"2024-07-26T07:49:20.297374+0000","last_peered":"2024-07-26T07:49:20.297374+0000","last_clean":"2024-07-26T07:49:20.297374+0000","last_became_active":"2024-07-26T07:47:41.790288+0000","last_became_peered":"2024-07-26T07:47:41.790288+0000","last_unstale":"2024-07-26T07:49:20.297374+0000","last_undegraded":"2024-07-26T07:49:20.297374+0000","last_fullsized":"2024-07-26T07:49:20.297374+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:05.734468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.102481+0000","last_change":"2024-07-26T07:47:45.854648+0000","last_active":"2024-07-26T07:49:48.102481+0000","last_peered":"2024-07-26T07:49:48.102481+0000","last_clean":"2024-07-26T07:49:48.102481+0000","last_became_active":"2024-07-26T07:47:45.854240+0000","last_became_peered":"2024-07-26T07:47:45.854240+0000","last_unstale":"2024-07-26T07:49:48.102481+0000","last_undegraded":"2024-07-26T07:49:48.102481+0000","last_fullsized":"2024-07-26T07:49:48.102481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:41.775760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'5","reported_seq":165,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:58.445026+0000","last_change":"2024-07-26T07:49:20.331153+0000","last_active":"2024-07-26T07:49:58.445026+0000","last_peered":"2024-07-26T07:49:58.445026+0000","last_clean":"2024-07-26T07:49:58.445026+0000","last_became_active":"2024-07-26T07:49:20.315423+0000","last_became_peered":"2024-07-26T07:49:20.315423+0000","last_unstale":"2024-07-26T07:49:58.445026+0000","last_undegraded":"2024-07-26T07:49:58.445026+0000","last_fullsized":"2024-07-26T07:49:58.445026+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:24.476818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.100623+0000","last_change":"2024-07-26T07:47:45.848731+0000","last_active":"2024-07-26T07:49:48.100623+0000","last_peered":"2024-07-26T07:49:48.100623+0000","last_clean":"2024-07-26T07:49:48.100623+0000","last_became_active":"2024-07-26T07:47:45.846663+0000","last_became_peered":"2024-07-26T07:47:45.846663+0000","last_unstale":"2024-07-26T07:49:48.100623+0000","last_undegraded":"2024-07-26T07:49:48.100623+0000","last_fullsized":"2024-07-26T07:49:48.100623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:05.251423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.356788+0000","last_change":"2024-07-26T07:49:20.355238+0000","last_active":"2024-07-26T07:49:20.356788+0000","last_peered":"2024-07-26T07:49:20.356788+0000","last_clean":"2024-07-26T07:49:20.356788+0000","last_became_active":"2024-07-26T07:49:20.316591+0000","last_became_peered":"2024-07-26T07:49:20.316591+0000","last_unstale":"2024-07-26T07:49:20.356788+0000","last_undegraded":"2024-07-26T07:49:20.356788+0000","last_fullsized":"2024-07-26T07:49:20.356788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:12.739473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088493+0000","last_change":"2024-07-26T07:47:45.842572+0000","last_active":"2024-07-26T07:49:48.088493+0000","last_peered":"2024-07-26T07:49:48.088493+0000","last_clean":"2024-07-26T07:49:48.088493+0000","last_became_active":"2024-07-26T07:47:45.842116+0000","last_became_peered":"2024-07-26T07:47:45.842116+0000","last_unstale":"2024-07-26T07:49:48.088493+0000","last_undegraded":"2024-07-26T07:49:48.088493+0000","last_fullsized":"2024-07-26T07:49:48.088493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:06.913737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314657+0000","last_change":"2024-07-26T07:47:41.790215+0000","last_active":"2024-07-26T07:49:20.314657+0000","last_peered":"2024-07-26T07:49:20.314657+0000","last_clean":"2024-07-26T07:49:20.314657+0000","last_became_active":"2024-07-26T07:47:41.789691+0000","last_became_peered":"2024-07-26T07:47:41.789691+0000","last_unstale":"2024-07-26T07:49:20.314657+0000","last_undegraded":"2024-07-26T07:49:20.314657+0000","last_fullsized":"2024-07-26T07:49:20.314657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:01:47.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086806+0000","last_change":"2024-07-26T07:47:45.860887+0000","last_active":"2024-07-26T07:49:48.086806+0000","last_peered":"2024-07-26T07:49:48.086806+0000","last_clean":"2024-07-26T07:49:48.086806+0000","last_became_active":"2024-07-26T07:47:45.860315+0000","last_became_peered":"2024-07-26T07:47:45.860315+0000","last_unstale":"2024-07-26T07:49:48.086806+0000","last_undegraded":"2024-07-26T07:49:48.086806+0000","last_fullsized":"2024-07-26T07:49:48.086806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:17:02.231206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429700+0000","last_change":"2024-07-26T07:48:20.022885+0000","last_active":"2024-07-26T07:49:20.429700+0000","last_peered":"2024-07-26T07:49:20.429700+0000","last_clean":"2024-07-26T07:49:20.429700+0000","last_became_active":"2024-07-26T07:48:20.001946+0000","last_became_peered":"2024-07-26T07:48:20.001946+0000","last_unstale":"2024-07-26T07:49:20.429700+0000","last_undegraded":"2024-07-26T07:49:20.429700+0000","last_fullsized":"2024-07-26T07:49:20.429700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:45.892886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098632+0000","last_change":"2024-07-26T07:47:45.858267+0000","last_active":"2024-07-26T07:49:48.098632+0000","last_peered":"2024-07-26T07:49:48.098632+0000","last_clean":"2024-07-26T07:49:48.098632+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.098632+0000","last_undegraded":"2024-07-26T07:49:48.098632+0000","last_fullsized":"2024-07-26T07:49:48.098632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:32.055977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298413+0000","last_change":"2024-07-26T07:47:41.789846+0000","last_active":"2024-07-26T07:49:20.298413+0000","last_peered":"2024-07-26T07:49:20.298413+0000","last_clean":"2024-07-26T07:49:20.298413+0000","last_became_active":"2024-07-26T07:47:41.789237+0000","last_became_peered":"2024-07-26T07:47:41.789237+0000","last_unstale":"2024-07-26T07:49:20.298413+0000","last_undegraded":"2024-07-26T07:49:20.298413+0000","last_fullsized":"2024-07-26T07:49:20.298413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:39.031047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095798+0000","last_change":"2024-07-26T07:47:45.846294+0000","last_active":"2024-07-26T07:49:48.095798+0000","last_peered":"2024-07-26T07:49:48.095798+0000","last_clean":"2024-07-26T07:49:48.095798+0000","last_became_active":"2024-07-26T07:47:45.845861+0000","last_became_peered":"2024-07-26T07:47:45.845861+0000","last_unstale":"2024-07-26T07:49:48.095798+0000","last_undegraded":"2024-07-26T07:49:48.095798+0000","last_fullsized":"2024-07-26T07:49:48.095798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:56:10.594430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'7","reported_seq":156,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:50:00.202039+0000","last_change":"2024-07-26T07:47:42.563412+0000","last_active":"2024-07-26T07:50:00.202039+0000","last_peered":"2024-07-26T07:50:00.202039+0000","last_clean":"2024-07-26T07:50:00.202039+0000","last_became_active":"2024-07-26T07:47:42.562364+0000","last_became_peered":"2024-07-26T07:47:42.562364+0000","last_unstale":"2024-07-26T07:50:00.202039+0000","last_undegraded":"2024-07-26T07:50:00.202039+0000","last_fullsized":"2024-07-26T07:50:00.202039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:14.796738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.108677+0000","last_change":"2024-07-26T07:47:45.856170+0000","last_active":"2024-07-26T07:49:48.108677+0000","last_peered":"2024-07-26T07:49:48.108677+0000","last_clean":"2024-07-26T07:49:48.108677+0000","last_became_active":"2024-07-26T07:47:45.855782+0000","last_became_peered":"2024-07-26T07:47:45.855782+0000","last_unstale":"2024-07-26T07:49:48.108677+0000","last_undegraded":"2024-07-26T07:49:48.108677+0000","last_fullsized":"2024-07-26T07:49:48.108677+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:22.690910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":110,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313107+0000","last_change":"2024-07-26T07:45:54.310557+0000","last_active":"2024-07-26T07:49:20.313107+0000","last_peered":"2024-07-26T07:49:20.313107+0000","last_clean":"2024-07-26T07:49:20.313107+0000","last_became_active":"2024-07-26T07:45:54.290906+0000","last_became_peered":"2024-07-26T07:45:54.290906+0000","last_unstale":"2024-07-26T07:49:20.313107+0000","last_undegraded":"2024-07-26T07:49:20.313107+0000","last_fullsized":"2024-07-26T07:49:20.313107+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_clean_scrub_stamp":"2024-07-26T07:41:16.689463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:51.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.305857+0000","last_change":"2024-07-26T07:47:41.776420+0000","last_active":"2024-07-26T07:49:20.305857+0000","last_peered":"2024-07-26T07:49:20.305857+0000","last_clean":"2024-07-26T07:49:20.305857+0000","last_became_active":"2024-07-26T07:47:41.775858+0000","last_became_peered":"2024-07-26T07:47:41.775858+0000","last_unstale":"2024-07-26T07:49:20.305857+0000","last_undegraded":"2024-07-26T07:49:20.305857+0000","last_fullsized":"2024-07-26T07:49:20.305857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:18.880331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.109533+0000","last_change":"2024-07-26T07:48:19.933318+0000","last_active":"2024-07-26T07:49:48.109533+0000","last_peered":"2024-07-26T07:49:48.109533+0000","last_clean":"2024-07-26T07:49:48.109533+0000","last_became_active":"2024-07-26T07:48:19.932683+0000","last_became_peered":"2024-07-26T07:48:19.932683+0000","last_unstale":"2024-07-26T07:49:48.109533+0000","last_undegraded":"2024-07-26T07:49:48.109533+0000","last_fullsized":"2024-07-26T07:49:48.109533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:25:23.387428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313228+0000","last_change":"2024-07-26T07:47:41.786698+0000","last_active":"2024-07-26T07:49:20.313228+0000","last_peered":"2024-07-26T07:49:20.313228+0000","last_clean":"2024-07-26T07:49:20.313228+0000","last_became_active":"2024-07-26T07:47:41.784491+0000","last_became_peered":"2024-07-26T07:47:41.784491+0000","last_unstale":"2024-07-26T07:49:20.313228+0000","last_undegraded":"2024-07-26T07:49:20.313228+0000","last_fullsized":"2024-07-26T07:49:20.313228+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:09:08.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093849+0000","last_change":"2024-07-26T07:47:45.849698+0000","last_active":"2024-07-26T07:49:48.093849+0000","last_peered":"2024-07-26T07:49:48.093849+0000","last_clean":"2024-07-26T07:49:48.093849+0000","last_became_active":"2024-07-26T07:47:45.849266+0000","last_became_peered":"2024-07-26T07:47:45.849266+0000","last_unstale":"2024-07-26T07:49:48.093849+0000","last_undegraded":"2024-07-26T07:49:48.093849+0000","last_fullsized":"2024-07-26T07:49:48.093849+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:57.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"82'5","reported_seq":149,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:57.440461+0000","last_change":"2024-07-26T07:47:42.558680+0000","last_active":"2024-07-26T07:49:57.440461+0000","last_peered":"2024-07-26T07:49:57.440461+0000","last_clean":"2024-07-26T07:49:57.440461+0000","last_became_active":"2024-07-26T07:47:42.558197+0000","last_became_peered":"2024-07-26T07:47:42.558197+0000","last_unstale":"2024-07-26T07:49:57.440461+0000","last_undegraded":"2024-07-26T07:49:57.440461+0000","last_fullsized":"2024-07-26T07:49:57.440461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:52:52.505455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.078525+0000","last_change":"2024-07-26T07:48:19.929897+0000","last_active":"2024-07-26T07:49:48.078525+0000","last_peered":"2024-07-26T07:49:48.078525+0000","last_clean":"2024-07-26T07:49:48.078525+0000","last_became_active":"2024-07-26T07:48:19.927946+0000","last_became_peered":"2024-07-26T07:48:19.927946+0000","last_unstale":"2024-07-26T07:49:48.078525+0000","last_undegraded":"2024-07-26T07:49:48.078525+0000","last_fullsized":"2024-07-26T07:49:48.078525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:11.157824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"82'5","reported_seq":154,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:50:02.233979+0000","last_change":"2024-07-26T07:47:41.785513+0000","last_active":"2024-07-26T07:50:02.233979+0000","last_peered":"2024-07-26T07:50:02.233979+0000","last_clean":"2024-07-26T07:50:02.233979+0000","last_became_active":"2024-07-26T07:47:41.784783+0000","last_became_peered":"2024-07-26T07:47:41.784783+0000","last_unstale":"2024-07-26T07:50:02.233979+0000","last_undegraded":"2024-07-26T07:50:02.233979+0000","last_fullsized":"2024-07-26T07:50:02.233979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:54.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.082468+0000","last_change":"2024-07-26T07:47:45.856580+0000","last_active":"2024-07-26T07:49:48.082468+0000","last_peered":"2024-07-26T07:49:48.082468+0000","last_clean":"2024-07-26T07:49:48.082468+0000","last_became_active":"2024-07-26T07:47:45.852330+0000","last_became_peered":"2024-07-26T07:47:45.852330+0000","last_unstale":"2024-07-26T07:49:48.082468+0000","last_undegraded":"2024-07-26T07:49:48.082468+0000","last_fullsized":"2024-07-26T07:49:48.082468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:09.479563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.428822+0000","last_change":"2024-07-26T07:47:42.559255+0000","last_active":"2024-07-26T07:49:20.428822+0000","last_peered":"2024-07-26T07:49:20.428822+0000","last_clean":"2024-07-26T07:49:20.428822+0000","last_became_active":"2024-07-26T07:47:42.558123+0000","last_became_peered":"2024-07-26T07:47:42.558123+0000","last_unstale":"2024-07-26T07:49:20.428822+0000","last_undegraded":"2024-07-26T07:49:20.428822+0000","last_fullsized":"2024-07-26T07:49:20.428822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:49:46.576674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092093+0000","last_change":"2024-07-26T07:47:45.856256+0000","last_active":"2024-07-26T07:49:48.092093+0000","last_peered":"2024-07-26T07:49:48.092093+0000","last_clean":"2024-07-26T07:49:48.092093+0000","last_became_active":"2024-07-26T07:47:45.855696+0000","last_became_peered":"2024-07-26T07:47:45.855696+0000","last_unstale":"2024-07-26T07:49:48.092093+0000","last_undegraded":"2024-07-26T07:49:48.092093+0000","last_fullsized":"2024-07-26T07:49:48.092093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:24.451225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314969+0000","last_change":"2024-07-26T07:47:42.562631+0000","last_active":"2024-07-26T07:49:20.314969+0000","last_peered":"2024-07-26T07:49:20.314969+0000","last_clean":"2024-07-26T07:49:20.314969+0000","last_became_active":"2024-07-26T07:47:42.561379+0000","last_became_peered":"2024-07-26T07:47:42.561379+0000","last_unstale":"2024-07-26T07:49:20.314969+0000","last_undegraded":"2024-07-26T07:49:20.314969+0000","last_fullsized":"2024-07-26T07:49:20.314969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:03:30.471694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092440+0000","last_change":"2024-07-26T07:47:45.849580+0000","last_active":"2024-07-26T07:49:48.092440+0000","last_peered":"2024-07-26T07:49:48.092440+0000","last_clean":"2024-07-26T07:49:48.092440+0000","last_became_active":"2024-07-26T07:47:45.849107+0000","last_became_peered":"2024-07-26T07:47:45.849107+0000","last_unstale":"2024-07-26T07:49:48.092440+0000","last_undegraded":"2024-07-26T07:49:48.092440+0000","last_fullsized":"2024-07-26T07:49:48.092440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:57:13.694059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'16","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.879746+0000","last_change":"2024-07-26T07:47:41.774083+0000","last_active":"2024-07-26T07:49:48.879746+0000","last_peered":"2024-07-26T07:49:48.879746+0000","last_clean":"2024-07-26T07:49:48.879746+0000","last_became_active":"2024-07-26T07:47:41.773502+0000","last_became_peered":"2024-07-26T07:47:41.773502+0000","last_unstale":"2024-07-26T07:49:48.879746+0000","last_undegraded":"2024-07-26T07:49:48.879746+0000","last_fullsized":"2024-07-26T07:49:48.879746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:11:44.083944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099359+0000","last_change":"2024-07-26T07:47:45.845894+0000","last_active":"2024-07-26T07:49:48.099359+0000","last_peered":"2024-07-26T07:49:48.099359+0000","last_clean":"2024-07-26T07:49:48.099359+0000","last_became_active":"2024-07-26T07:47:45.845534+0000","last_became_peered":"2024-07-26T07:47:45.845534+0000","last_unstale":"2024-07-26T07:49:48.099359+0000","last_undegraded":"2024-07-26T07:49:48.099359+0000","last_fullsized":"2024-07-26T07:49:48.099359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:52:42.209561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.306134+0000","last_change":"2024-07-26T07:47:41.904220+0000","last_active":"2024-07-26T07:49:20.306134+0000","last_peered":"2024-07-26T07:49:20.306134+0000","last_clean":"2024-07-26T07:49:20.306134+0000","last_became_active":"2024-07-26T07:47:41.902349+0000","last_became_peered":"2024-07-26T07:47:41.902349+0000","last_unstale":"2024-07-26T07:49:20.306134+0000","last_undegraded":"2024-07-26T07:49:20.306134+0000","last_fullsized":"2024-07-26T07:49:20.306134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:08.520461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092199+0000","last_change":"2024-07-26T07:47:45.846920+0000","last_active":"2024-07-26T07:49:48.092199+0000","last_peered":"2024-07-26T07:49:48.092199+0000","last_clean":"2024-07-26T07:49:48.092199+0000","last_became_active":"2024-07-26T07:47:45.846458+0000","last_became_peered":"2024-07-26T07:47:45.846458+0000","last_unstale":"2024-07-26T07:49:48.092199+0000","last_undegraded":"2024-07-26T07:49:48.092199+0000","last_fullsized":"2024-07-26T07:49:48.092199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:06.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.295007+0000","last_change":"2024-07-26T07:47:42.560346+0000","last_active":"2024-07-26T07:49:20.295007+0000","last_peered":"2024-07-26T07:49:20.295007+0000","last_clean":"2024-07-26T07:49:20.295007+0000","last_became_active":"2024-07-26T07:47:42.558937+0000","last_became_peered":"2024-07-26T07:47:42.558937+0000","last_unstale":"2024-07-26T07:49:20.295007+0000","last_undegraded":"2024-07-26T07:49:20.295007+0000","last_fullsized":"2024-07-26T07:49:20.295007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.676625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.081730+0000","last_change":"2024-07-26T07:47:45.845553+0000","last_active":"2024-07-26T07:49:48.081730+0000","last_peered":"2024-07-26T07:49:48.081730+0000","last_clean":"2024-07-26T07:49:48.081730+0000","last_became_active":"2024-07-26T07:47:45.845125+0000","last_became_peered":"2024-07-26T07:47:45.845125+0000","last_unstale":"2024-07-26T07:49:48.081730+0000","last_undegraded":"2024-07-26T07:49:48.081730+0000","last_fullsized":"2024-07-26T07:49:48.081730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:10.896159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.292557+0000","last_change":"2024-07-26T07:47:41.779688+0000","last_active":"2024-07-26T07:49:20.292557+0000","last_peered":"2024-07-26T07:49:20.292557+0000","last_clean":"2024-07-26T07:49:20.292557+0000","last_became_active":"2024-07-26T07:47:41.779115+0000","last_became_peered":"2024-07-26T07:47:41.779115+0000","last_unstale":"2024-07-26T07:49:20.292557+0000","last_undegraded":"2024-07-26T07:49:20.292557+0000","last_fullsized":"2024-07-26T07:49:20.292557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:20.708336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091558+0000","last_change":"2024-07-26T07:47:45.853869+0000","last_active":"2024-07-26T07:49:48.091558+0000","last_peered":"2024-07-26T07:49:48.091558+0000","last_clean":"2024-07-26T07:49:48.091558+0000","last_became_active":"2024-07-26T07:47:45.852509+0000","last_became_peered":"2024-07-26T07:47:45.852509+0000","last_unstale":"2024-07-26T07:49:48.091558+0000","last_undegraded":"2024-07-26T07:49:48.091558+0000","last_fullsized":"2024-07-26T07:49:48.091558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:01:31.764025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'3","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.312497+0000","last_change":"2024-07-26T07:47:41.792331+0000","last_active":"2024-07-26T07:49:20.312497+0000","last_peered":"2024-07-26T07:49:20.312497+0000","last_clean":"2024-07-26T07:49:20.312497+0000","last_became_active":"2024-07-26T07:47:41.790668+0000","last_became_peered":"2024-07-26T07:47:41.790668+0000","last_unstale":"2024-07-26T07:49:20.312497+0000","last_undegraded":"2024-07-26T07:49:20.312497+0000","last_fullsized":"2024-07-26T07:49:20.312497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:59.430692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.089973+0000","last_change":"2024-07-26T07:47:45.843583+0000","last_active":"2024-07-26T07:49:48.089973+0000","last_peered":"2024-07-26T07:49:48.089973+0000","last_clean":"2024-07-26T07:49:48.089973+0000","last_became_active":"2024-07-26T07:47:45.843185+0000","last_became_peered":"2024-07-26T07:47:45.843185+0000","last_unstale":"2024-07-26T07:49:48.089973+0000","last_undegraded":"2024-07-26T07:49:48.089973+0000","last_fullsized":"2024-07-26T07:49:48.089973+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:01.119065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314240+0000","last_change":"2024-07-26T07:48:20.014411+0000","last_active":"2024-07-26T07:49:20.314240+0000","last_peered":"2024-07-26T07:49:20.314240+0000","last_clean":"2024-07-26T07:49:20.314240+0000","last_became_active":"2024-07-26T07:48:20.002085+0000","last_became_peered":"2024-07-26T07:48:20.002085+0000","last_unstale":"2024-07-26T07:49:20.314240+0000","last_undegraded":"2024-07-26T07:49:20.314240+0000","last_fullsized":"2024-07-26T07:49:20.314240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:46:59.335128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.110858+0000","last_change":"2024-07-26T07:47:45.851963+0000","last_active":"2024-07-26T07:49:48.110858+0000","last_peered":"2024-07-26T07:49:48.110858+0000","last_clean":"2024-07-26T07:49:48.110858+0000","last_became_active":"2024-07-26T07:47:45.850459+0000","last_became_peered":"2024-07-26T07:47:45.850459+0000","last_unstale":"2024-07-26T07:49:48.110858+0000","last_undegraded":"2024-07-26T07:49:48.110858+0000","last_fullsized":"2024-07-26T07:49:48.110858+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:18.092186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088035+0000","last_change":"2024-07-26T07:47:45.848744+0000","last_active":"2024-07-26T07:49:48.088035+0000","last_peered":"2024-07-26T07:49:48.088035+0000","last_clean":"2024-07-26T07:49:48.088035+0000","last_became_active":"2024-07-26T07:47:45.848292+0000","last_became_peered":"2024-07-26T07:47:45.848292+0000","last_unstale":"2024-07-26T07:49:48.088035+0000","last_undegraded":"2024-07-26T07:49:48.088035+0000","last_fullsized":"2024-07-26T07:49:48.088035+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:54.549508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314691+0000","last_change":"2024-07-26T07:47:41.788875+0000","last_active":"2024-07-26T07:49:20.314691+0000","last_peered":"2024-07-26T07:49:20.314691+0000","last_clean":"2024-07-26T07:49:20.314691+0000","last_became_active":"2024-07-26T07:47:41.786942+0000","last_became_peered":"2024-07-26T07:47:41.786942+0000","last_unstale":"2024-07-26T07:49:20.314691+0000","last_undegraded":"2024-07-26T07:49:20.314691+0000","last_fullsized":"2024-07-26T07:49:20.314691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:11:31.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111090+0000","last_change":"2024-07-26T07:47:45.854832+0000","last_active":"2024-07-26T07:49:48.111090+0000","last_peered":"2024-07-26T07:49:48.111090+0000","last_clean":"2024-07-26T07:49:48.111090+0000","last_became_active":"2024-07-26T07:47:45.853756+0000","last_became_peered":"2024-07-26T07:47:45.853756+0000","last_unstale":"2024-07-26T07:49:48.111090+0000","last_undegraded":"2024-07-26T07:49:48.111090+0000","last_fullsized":"2024-07-26T07:49:48.111090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:42:14.579177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314716+0000","last_change":"2024-07-26T07:47:41.787839+0000","last_active":"2024-07-26T07:49:20.314716+0000","last_peered":"2024-07-26T07:49:20.314716+0000","last_clean":"2024-07-26T07:49:20.314716+0000","last_became_active":"2024-07-26T07:47:41.785951+0000","last_became_peered":"2024-07-26T07:47:41.785951+0000","last_unstale":"2024-07-26T07:49:20.314716+0000","last_undegraded":"2024-07-26T07:49:20.314716+0000","last_fullsized":"2024-07-26T07:49:20.314716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:58.699056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104028+0000","last_change":"2024-07-26T07:47:45.841451+0000","last_active":"2024-07-26T07:49:48.104028+0000","last_peered":"2024-07-26T07:49:48.104028+0000","last_clean":"2024-07-26T07:49:48.104028+0000","last_became_active":"2024-07-26T07:47:45.840803+0000","last_became_peered":"2024-07-26T07:47:45.840803+0000","last_unstale":"2024-07-26T07:49:48.104028+0000","last_undegraded":"2024-07-26T07:49:48.104028+0000","last_fullsized":"2024-07-26T07:49:48.104028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:24:21.832317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.293557+0000","last_change":"2024-07-26T07:47:42.558775+0000","last_active":"2024-07-26T07:49:20.293557+0000","last_peered":"2024-07-26T07:49:20.293557+0000","last_clean":"2024-07-26T07:49:20.293557+0000","last_became_active":"2024-07-26T07:47:42.558300+0000","last_became_peered":"2024-07-26T07:47:42.558300+0000","last_unstale":"2024-07-26T07:49:20.293557+0000","last_undegraded":"2024-07-26T07:49:20.293557+0000","last_fullsized":"2024-07-26T07:49:20.293557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:59.643709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091602+0000","last_change":"2024-07-26T07:47:45.852401+0000","last_active":"2024-07-26T07:49:48.091602+0000","last_peered":"2024-07-26T07:49:48.091602+0000","last_clean":"2024-07-26T07:49:48.091602+0000","last_became_active":"2024-07-26T07:47:45.852054+0000","last_became_peered":"2024-07-26T07:47:45.852054+0000","last_unstale":"2024-07-26T07:49:48.091602+0000","last_undegraded":"2024-07-26T07:49:48.091602+0000","last_fullsized":"2024-07-26T07:49:48.091602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:25.347770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"82'13","reported_seq":183,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:50:02.232523+0000","last_change":"2024-07-26T07:48:19.961760+0000","last_active":"2024-07-26T07:50:02.232523+0000","last_peered":"2024-07-26T07:50:02.232523+0000","last_clean":"2024-07-26T07:50:02.232523+0000","last_became_active":"2024-07-26T07:48:19.937694+0000","last_became_peered":"2024-07-26T07:48:19.937694+0000","last_unstale":"2024-07-26T07:50:02.232523+0000","last_undegraded":"2024-07-26T07:50:02.232523+0000","last_fullsized":"2024-07-26T07:50:02.232523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:05:10.571086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66965,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1174,"num_read_kb":660,"num_write":251,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1372,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":200895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":464264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776182,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1114072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972413,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":462962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168644,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":462556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397578,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1054879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561104,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":462679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757337,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1589248,"data_stored":1055145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953567,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":462540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":462504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411436,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":524355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":524275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":464550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T07:50:06.342 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T07:50:06.342 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-26T07:50:06.343 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-07-26T07:50:06.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph health --format=json 2024-07-26T07:50:06.805 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:06.955 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:06 smithi097 ceph-mon[32231]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:50:06.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:06 smithi115 ceph-mon[35364]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:50:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:06 smithi181 ceph-mon[35869]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:50:07.965 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:07 smithi097 ceph-mon[32231]: pgmap v460: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:07.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:07 smithi115 ceph-mon[35364]: pgmap v460: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:07 smithi181 ceph-mon[35869]: pgmap v460: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:09.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.998+0000 7f02b9000640 1 -- 172.21.15.97:0/3340550840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b406cf90 msgr2=0x7f02b406d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:09.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.998+0000 7f02b9000640 1 --2- 172.21.15.97:0/3340550840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b406cf90 0x7f02b406d3a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f029c009900 tx=0x7f029c02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.998+0000 7f02b9000640 1 -- 172.21.15.97:0/3340550840 shutdown_connections 2024-07-26T07:50:09.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.998+0000 7f02b9000640 1 --2- 172.21.15.97:0/3340550840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b411c670 0x7f02b411eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.999+0000 7f02b9000640 1 --2- 172.21.15.97:0/3340550840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b406cf90 0x7f02b406d3a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.999+0000 7f02b9000640 1 --2- 172.21.15.97:0/3340550840 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02b406e540 0x7f02b406c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.999+0000 7f02b9000640 1 -- 172.21.15.97:0/3340550840 >> 172.21.15.97:0/3340550840 conn(0x7f02b40714b0 msgr2=0x7f02b4109df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:09.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.999+0000 7f02b9000640 1 -- 172.21.15.97:0/3340550840 shutdown_connections 2024-07-26T07:50:09.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:08.999+0000 7f02b9000640 1 -- 172.21.15.97:0/3340550840 wait complete. 2024-07-26T07:50:09.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.000+0000 7f02b9000640 1 Processor -- start 2024-07-26T07:50:09.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.000+0000 7f02b9000640 1 -- start start 2024-07-26T07:50:09.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b9000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02b406cf90 0x7f02b41a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:09.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b9000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b406e540 0x7f02b41a9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:09.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b9000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b411c670 0x7f02b41ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:09.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b9000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41221f0 con 0x7f02b411c670 2024-07-26T07:50:09.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b9000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f02b4122070 con 0x7f02b406cf90 2024-07-26T07:50:09.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b9000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f02b4122370 con 0x7f02b406e540 2024-07-26T07:50:09.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b411c670 0x7f02b41ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:09.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b411c670 0x7f02b41ae4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44652/0 (socket says 172.21.15.97:44652) 2024-07-26T07:50:09.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b2800640 1 -- 172.21.15.97:0/1026178584 learned_addr learned my addr 172.21.15.97:0/1026178584 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:09.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02abe00640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b406e540 0x7f02b41a9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.001+0000 7f02b1e00640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02b406cf90 0x7f02b41a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02b2800640 1 -- 172.21.15.97:0/1026178584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b406e540 msgr2=0x7f02b41a9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02b2800640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b406e540 0x7f02b41a9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02b2800640 1 -- 172.21.15.97:0/1026178584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02b406cf90 msgr2=0x7f02b41a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02b2800640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02b406cf90 0x7f02b41a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02b2800640 1 -- 172.21.15.97:0/1026178584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b41aeb10 con 0x7f02b411c670 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02b2800640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b411c670 0x7f02b41ae4a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00b970 tx=0x7f02ac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02aaa00640 1 -- 172.21.15.97:0/1026178584 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac019070 con 0x7f02b411c670 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b41aee00 con 0x7f02b411c670 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02aaa00640 1 -- 172.21.15.97:0/1026178584 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02ac014440 con 0x7f02b411c670 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02aaa00640 1 -- 172.21.15.97:0/1026178584 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0075d0 con 0x7f02b411c670 2024-07-26T07:50:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.002+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b41b5620 con 0x7f02b411c670 2024-07-26T07:50:09.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.004+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b406d3a0 con 0x7f02b411c670 2024-07-26T07:50:09.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.006+0000 7f02aaa00640 1 -- 172.21.15.97:0/1026178584 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f02ac01c070 con 0x7f02b411c670 2024-07-26T07:50:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.007+0000 7f02aaa00640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f02940b3c80 0x7f02940b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:09.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.008+0000 7f02aaa00640 1 -- 172.21.15.97:0/1026178584 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f02ac0e36f0 con 0x7f02b411c670 2024-07-26T07:50:09.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.008+0000 7f02b1e00640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f02940b3c80 0x7f02940b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:09.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.009+0000 7f02b1e00640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f02940b3c80 0x7f02940b6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02a4005e10 tx=0x7f02a4005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:09.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.010+0000 7f02aaa00640 1 -- 172.21.15.97:0/1026178584 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f02ac0aac80 con 0x7f02b411c670 2024-07-26T07:50:09.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.254+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f02b41a9d90 con 0x7f02b411c670 2024-07-26T07:50:09.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.255+0000 7f02aaa00640 1 -- 172.21.15.97:0/1026178584 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f02b41a9d90 con 0x7f02b411c670 2024-07-26T07:50:09.256 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:50:09.256 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-26T07:50:09.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f02940b3c80 msgr2=0x7f02940b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f02940b3c80 0x7f02940b6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02a4005e10 tx=0x7f02a4005da0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b411c670 msgr2=0x7f02b41ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b411c670 0x7f02b41ae4a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00b970 tx=0x7f02ac00be30 comp rx=0 tx=0).stop 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 shutdown_connections 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f02940b3c80 0x7f02940b6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02b411c670 0x7f02b41ae4a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b406e540 0x7f02b41a9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 --2- 172.21.15.97:0/1026178584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02b406cf90 0x7f02b41a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 >> 172.21.15.97:0/1026178584 conn(0x7f02b40714b0 msgr2=0x7f02b4118570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.258+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 shutdown_connections 2024-07-26T07:50:09.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:09.259+0000 7f02b9000640 1 -- 172.21.15.97:0/1026178584 wait complete. 2024-07-26T07:50:09.936 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:09 smithi097 ceph-mon[32231]: pgmap v461: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:09.936 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:09 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1026178584' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T07:50:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:09 smithi115 ceph-mon[35364]: pgmap v461: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:09 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1026178584' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T07:50:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:09 smithi181 ceph-mon[35869]: pgmap v461: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:09 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1026178584' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T07:50:09.991 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-07-26T07:50:09.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:50:10.431 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:11 smithi115 ceph-mon[35364]: pgmap v462: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:11 smithi181 ceph-mon[35869]: pgmap v462: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:11.987 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:11 smithi097 ceph-mon[32231]: pgmap v462: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:12.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 -- 172.21.15.97:0/4200317265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c6810af50 msgr2=0x7f0c6810b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:12.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 --2- 172.21.15.97:0/4200317265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c6810af50 0x7f0c6810b360 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009900 tx=0x7f0c5402a3f0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 -- 172.21.15.97:0/4200317265 shutdown_connections 2024-07-26T07:50:12.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 --2- 172.21.15.97:0/4200317265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c6810b8a0 0x7f0c681120d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 --2- 172.21.15.97:0/4200317265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c6810af50 0x7f0c6810b360 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 --2- 172.21.15.97:0/4200317265 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c68105610 0x7f0c68105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 -- 172.21.15.97:0/4200317265 >> 172.21.15.97:0/4200317265 conn(0x7f0c68101060 msgr2=0x7f0c681034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:12.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 -- 172.21.15.97:0/4200317265 shutdown_connections 2024-07-26T07:50:12.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.500+0000 7f0c70200640 1 -- 172.21.15.97:0/4200317265 wait complete. 2024-07-26T07:50:12.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c70200640 1 Processor -- start 2024-07-26T07:50:12.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c70200640 1 -- start start 2024-07-26T07:50:12.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c70200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c68105610 0x7f0c6819be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:12.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c70200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c6810af50 0x7f0c6819c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:12.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c70200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c6810b8a0 0x7f0c681a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:12.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c70200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0c681153e0 con 0x7f0c68105610 2024-07-26T07:50:12.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c70200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0c68115260 con 0x7f0c6810af50 2024-07-26T07:50:12.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c70200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0c68115560 con 0x7f0c6810b8a0 2024-07-26T07:50:12.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.502+0000 7f0c6d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c68105610 0x7f0c6819be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:12.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c68105610 0x7f0c6819be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44680/0 (socket says 172.21.15.97:44680) 2024-07-26T07:50:12.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6d800640 1 -- 172.21.15.97:0/637465885 learned_addr learned my addr 172.21.15.97:0/637465885 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:12.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6ce00640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c6810af50 0x7f0c6819c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:12.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6e200640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c6810b8a0 0x7f0c681a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:12.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6d800640 1 -- 172.21.15.97:0/637465885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c6810b8a0 msgr2=0x7f0c681a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:12.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6d800640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c6810b8a0 0x7f0c681a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6d800640 1 -- 172.21.15.97:0/637465885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c6810af50 msgr2=0x7f0c6819c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:12.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6d800640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c6810af50 0x7f0c6819c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6d800640 1 -- 172.21.15.97:0/637465885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c681a1d40 con 0x7f0c68105610 2024-07-26T07:50:12.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.503+0000 7f0c6ce00640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c6810af50 0x7f0c6819c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:12.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.504+0000 7f0c6d800640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c68105610 0x7f0c6819be90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00c920 tx=0x7f0c5c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:12.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.504+0000 7f0c66000640 1 -- 172.21.15.97:0/637465885 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c007c70 con 0x7f0c68105610 2024-07-26T07:50:12.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.504+0000 7f0c66000640 1 -- 172.21.15.97:0/637465885 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c5c007e10 con 0x7f0c68105610 2024-07-26T07:50:12.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.504+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c681a2030 con 0x7f0c68105610 2024-07-26T07:50:12.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.504+0000 7f0c66000640 1 -- 172.21.15.97:0/637465885 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00f450 con 0x7f0c68105610 2024-07-26T07:50:12.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.504+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c681a8850 con 0x7f0c68105610 2024-07-26T07:50:12.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.505+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c68105a20 con 0x7f0c68105610 2024-07-26T07:50:12.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.508+0000 7f0c66000640 1 -- 172.21.15.97:0/637465885 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f0c5c004e90 con 0x7f0c68105610 2024-07-26T07:50:12.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.509+0000 7f0c66000640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c500b3c80 0x7f0c500b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:12.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.510+0000 7f0c66000640 1 -- 172.21.15.97:0/637465885 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f0c5c0e4100 con 0x7f0c68105610 2024-07-26T07:50:12.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.510+0000 7f0c6ce00640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c500b3c80 0x7f0c500b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:12.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.511+0000 7f0c6ce00640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c500b3c80 0x7f0c500b6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0c540057c0 tx=0x7f0c54009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:12.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.511+0000 7f0c66000640 1 -- 172.21.15.97:0/637465885 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0c5c025020 con 0x7f0c68105610 2024-07-26T07:50:12.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.753+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c6819cfc0 con 0x7f0c68105610 2024-07-26T07:50:12.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.755+0000 7f0c66000640 1 -- 172.21.15.97:0/637465885 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11990 (secure 0 0 0) 0x7f0c5c0ab780 con 0x7f0c68105610 2024-07-26T07:50:12.756 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:50:12.757 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 27 2024-07-26T07:50:12.757 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":27,"btime":"2024-07-26T07:48:11:830543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:48:11.504745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:50:12.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c500b3c80 msgr2=0x7f0c500b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:12.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c500b3c80 0x7f0c500b6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0c540057c0 tx=0x7f0c54009680 comp rx=0 tx=0).stop 2024-07-26T07:50:12.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c68105610 msgr2=0x7f0c6819be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:12.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c68105610 0x7f0c6819be90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00c920 tx=0x7f0c5c00cde0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 shutdown_connections 2024-07-26T07:50:12.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c500b3c80 0x7f0c500b6190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c6810b8a0 0x7f0c681a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c6810af50 0x7f0c6819c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 --2- 172.21.15.97:0/637465885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c68105610 0x7f0c6819be90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:12.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 >> 172.21.15.97:0/637465885 conn(0x7f0c68101060 msgr2=0x7f0c68103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:12.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.758+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 shutdown_connections 2024-07-26T07:50:12.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:12.759+0000 7f0c70200640 1 -- 172.21.15.97:0/637465885 wait complete. 2024-07-26T07:50:13.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:50:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:13 smithi097 ceph-mon[32231]: pgmap v463: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:13 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/637465885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:13 smithi115 ceph-mon[35364]: pgmap v463: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:13 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/637465885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:13 smithi181 ceph-mon[35869]: pgmap v463: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:13 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/637465885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:13.997 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:15.732 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:15 smithi097 ceph-mon[32231]: pgmap v464: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:15.921 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:15 smithi115 ceph-mon[35364]: pgmap v464: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:15 smithi181 ceph-mon[35869]: pgmap v464: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:16.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.030+0000 7fb06a200640 1 -- 172.21.15.97:0/2906787040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410af10 msgr2=0x7fb06410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:16.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.030+0000 7fb06a200640 1 --2- 172.21.15.97:0/2906787040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410af10 0x7fb06410b320 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb050009900 tx=0x7fb05002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.031+0000 7fb06a200640 1 -- 172.21.15.97:0/2906787040 shutdown_connections 2024-07-26T07:50:16.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.031+0000 7fb06a200640 1 --2- 172.21.15.97:0/2906787040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb06410b860 0x7fb064112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.031+0000 7fb06a200640 1 --2- 172.21.15.97:0/2906787040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410af10 0x7fb06410b320 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.031+0000 7fb06a200640 1 --2- 172.21.15.97:0/2906787040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0641055d0 0x7fb0641059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.031+0000 7fb06a200640 1 -- 172.21.15.97:0/2906787040 >> 172.21.15.97:0/2906787040 conn(0x7fb064101040 msgr2=0x7fb064103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:16.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.031+0000 7fb06a200640 1 -- 172.21.15.97:0/2906787040 shutdown_connections 2024-07-26T07:50:16.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.031+0000 7fb06a200640 1 -- 172.21.15.97:0/2906787040 wait complete. 2024-07-26T07:50:16.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.032+0000 7fb06a200640 1 Processor -- start 2024-07-26T07:50:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.032+0000 7fb06a200640 1 -- start start 2024-07-26T07:50:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb06a200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0641055d0 0x7fb06419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb06a200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb06410af10 0x7fb06419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb06a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410b860 0x7fb0641a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb06a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641153f0 con 0x7fb06410b860 2024-07-26T07:50:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb06a200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb064115270 con 0x7fb0641055d0 2024-07-26T07:50:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb06a200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb064115570 con 0x7fb06410af10 2024-07-26T07:50:16.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb069800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410b860 0x7fb0641a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:16.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb069800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410b860 0x7fb0641a16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39680/0 (socket says 172.21.15.97:39680) 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.033+0000 7fb069800640 1 -- 172.21.15.97:0/2910900598 learned_addr learned my addr 172.21.15.97:0/2910900598 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb063e00640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb06410af10 0x7fb06419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb068e00640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0641055d0 0x7fb06419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb069800640 1 -- 172.21.15.97:0/2910900598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb06410af10 msgr2=0x7fb06419c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb069800640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb06410af10 0x7fb06419c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb069800640 1 -- 172.21.15.97:0/2910900598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0641055d0 msgr2=0x7fb06419be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb069800640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0641055d0 0x7fb06419be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb069800640 1 -- 172.21.15.97:0/2910900598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0641a1df0 con 0x7fb06410b860 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb069800640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410b860 0x7fb0641a16c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00eaa0 tx=0x7fb05c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb061600640 1 -- 172.21.15.97:0/2910900598 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c00cdb0 con 0x7fb06410b860 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb061600640 1 -- 172.21.15.97:0/2910900598 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb05c005020 con 0x7fb06410b860 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.034+0000 7fb061600640 1 -- 172.21.15.97:0/2910900598 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0116d0 con 0x7fb06410b860 2024-07-26T07:50:16.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.035+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0641a20e0 con 0x7fb06410b860 2024-07-26T07:50:16.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.035+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0641a8840 con 0x7fb06410b860 2024-07-26T07:50:16.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.036+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0641059e0 con 0x7fb06410b860 2024-07-26T07:50:16.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.038+0000 7fb061600640 1 -- 172.21.15.97:0/2910900598 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fb05c0051d0 con 0x7fb06410b860 2024-07-26T07:50:16.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.040+0000 7fb061600640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb04c0b3bd0 0x7fb04c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:16.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.040+0000 7fb061600640 1 -- 172.21.15.97:0/2910900598 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fb05c0e4360 con 0x7fb06410b860 2024-07-26T07:50:16.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.040+0000 7fb068e00640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb04c0b3bd0 0x7fb04c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:16.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.041+0000 7fb068e00640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb04c0b3bd0 0x7fb04c0b60e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb058005fd0 tx=0x7fb058005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:16.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.042+0000 7fb061600640 1 -- 172.21.15.97:0/2910900598 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb05c016030 con 0x7fb06410b860 2024-07-26T07:50:16.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.277+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb06419cfb0 con 0x7fb06410b860 2024-07-26T07:50:16.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.279+0000 7fb061600640 1 -- 172.21.15.97:0/2910900598 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11990 (secure 0 0 0) 0x7fb05c0ab8f0 con 0x7fb06410b860 2024-07-26T07:50:16.280 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:50:16.281 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":27,"btime":"2024-07-26T07:48:11:830543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:48:11.504745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:50:16.281 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 27 2024-07-26T07:50:16.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb04c0b3bd0 msgr2=0x7fb04c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:16.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb04c0b3bd0 0x7fb04c0b60e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb058005fd0 tx=0x7fb058005850 comp rx=0 tx=0).stop 2024-07-26T07:50:16.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410b860 msgr2=0x7fb0641a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:16.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410b860 0x7fb0641a16c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00eaa0 tx=0x7fb05c00ef60 comp rx=0 tx=0).stop 2024-07-26T07:50:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 shutdown_connections 2024-07-26T07:50:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb04c0b3bd0 0x7fb04c0b60e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb06410b860 0x7fb0641a16c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb06410af10 0x7fb06419c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 --2- 172.21.15.97:0/2910900598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0641055d0 0x7fb06419be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.282+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 >> 172.21.15.97:0/2910900598 conn(0x7fb064101040 msgr2=0x7fb064103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.283+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 shutdown_connections 2024-07-26T07:50:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:16.283+0000 7fb06a200640 1 -- 172.21.15.97:0/2910900598 wait complete. 2024-07-26T07:50:16.957 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:16 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2910900598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:16 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2910900598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:16 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2910900598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:17.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:50:17.495 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:17.946 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:17 smithi097 ceph-mon[32231]: pgmap v465: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:17 smithi115 ceph-mon[35364]: pgmap v465: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:17 smithi181 ceph-mon[35869]: pgmap v465: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:19.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 -- 172.21.15.97:0/3235029815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd8410af10 msgr2=0x7efd8410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:19.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 --2- 172.21.15.97:0/3235029815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd8410af10 0x7efd8410b320 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7efd70009900 tx=0x7efd7002a410 comp rx=0 tx=0).stop 2024-07-26T07:50:19.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 -- 172.21.15.97:0/3235029815 shutdown_connections 2024-07-26T07:50:19.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 --2- 172.21.15.97:0/3235029815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd8410b860 0x7efd84112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 --2- 172.21.15.97:0/3235029815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd8410af10 0x7efd8410b320 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 --2- 172.21.15.97:0/3235029815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd841055d0 0x7efd841059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 -- 172.21.15.97:0/3235029815 >> 172.21.15.97:0/3235029815 conn(0x7efd84101040 msgr2=0x7efd84103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:19.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 -- 172.21.15.97:0/3235029815 shutdown_connections 2024-07-26T07:50:19.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.498+0000 7efd89800640 1 -- 172.21.15.97:0/3235029815 wait complete. 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.499+0000 7efd89800640 1 Processor -- start 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.500+0000 7efd89800640 1 -- start start 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.500+0000 7efd89800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd841055d0 0x7efd8419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.500+0000 7efd89800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd8410af10 0x7efd8419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.500+0000 7efd89800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd8410b860 0x7efd841a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.500+0000 7efd89800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7efd841153a0 con 0x7efd841055d0 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.500+0000 7efd89800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7efd84115220 con 0x7efd8410af10 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.500+0000 7efd89800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efd84115520 con 0x7efd8410b860 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.500+0000 7efd83e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd841055d0 0x7efd8419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd83e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd841055d0 0x7efd8419be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39698/0 (socket says 172.21.15.97:39698) 2024-07-26T07:50:19.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd83e00640 1 -- 172.21.15.97:0/2496241635 learned_addr learned my addr 172.21.15.97:0/2496241635 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd83400640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd8410af10 0x7efd8419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd88e00640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd8410b860 0x7efd841a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd83e00640 1 -- 172.21.15.97:0/2496241635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd8410b860 msgr2=0x7efd841a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd83e00640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd8410b860 0x7efd841a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd83e00640 1 -- 172.21.15.97:0/2496241635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd8410af10 msgr2=0x7efd8419c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd83e00640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd8410af10 0x7efd8419c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.501+0000 7efd83e00640 1 -- 172.21.15.97:0/2496241635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd841a1e00 con 0x7efd841055d0 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.502+0000 7efd83e00640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd841055d0 0x7efd8419be90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7efd7400c890 tx=0x7efd7400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.502+0000 7efd80c00640 1 -- 172.21.15.97:0/2496241635 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd74004220 con 0x7efd841055d0 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.502+0000 7efd80c00640 1 -- 172.21.15.97:0/2496241635 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd74007e40 con 0x7efd841055d0 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.502+0000 7efd80c00640 1 -- 172.21.15.97:0/2496241635 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd74004a00 con 0x7efd841055d0 2024-07-26T07:50:19.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.502+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd841a20f0 con 0x7efd841055d0 2024-07-26T07:50:19.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.502+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd841a8850 con 0x7efd841055d0 2024-07-26T07:50:19.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.503+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd841059e0 con 0x7efd841055d0 2024-07-26T07:50:19.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.506+0000 7efd80c00640 1 -- 172.21.15.97:0/2496241635 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7efd740079b0 con 0x7efd841055d0 2024-07-26T07:50:19.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.507+0000 7efd80c00640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7efd680b3c50 0x7efd680b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:19.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.507+0000 7efd80c00640 1 -- 172.21.15.97:0/2496241635 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7efd740e3a30 con 0x7efd841055d0 2024-07-26T07:50:19.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.507+0000 7efd83400640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7efd680b3c50 0x7efd680b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:19.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.508+0000 7efd83400640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7efd680b3c50 0x7efd680b6160 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efd700057c0 tx=0x7efd70000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:19.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.511+0000 7efd80c00640 1 -- 172.21.15.97:0/2496241635 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efd74014030 con 0x7efd841055d0 2024-07-26T07:50:19.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.715+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd8419cfc0 con 0x7efd841055d0 2024-07-26T07:50:19.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.718+0000 7efd80c00640 1 -- 172.21.15.97:0/2496241635 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22853 (secure 0 0 0) 0x7efd740aafc0 con 0x7efd841055d0 2024-07-26T07:50:19.720 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:50:19.720 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":86,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:49:20.276546+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T07:47:40.641206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T07:47:44.357412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:50:19.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.721+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7efd680b3c50 msgr2=0x7efd680b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:19.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.721+0000 7efd89800640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7efd680b3c50 0x7efd680b6160 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efd700057c0 tx=0x7efd70000bc0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.721+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd841055d0 msgr2=0x7efd8419be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.721+0000 7efd89800640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd841055d0 0x7efd8419be90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7efd7400c890 tx=0x7efd7400cd50 comp rx=0 tx=0).stop 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.722+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 shutdown_connections 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.722+0000 7efd89800640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7efd680b3c50 0x7efd680b6160 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.722+0000 7efd89800640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd8410b860 0x7efd841a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.722+0000 7efd89800640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd8410af10 0x7efd8419c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.722+0000 7efd89800640 1 --2- 172.21.15.97:0/2496241635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd841055d0 0x7efd8419be90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.722+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 >> 172.21.15.97:0/2496241635 conn(0x7efd84101040 msgr2=0x7efd84103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.722+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 shutdown_connections 2024-07-26T07:50:19.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:19.722+0000 7efd89800640 1 -- 172.21.15.97:0/2496241635 wait complete. 2024-07-26T07:50:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:19 smithi115 ceph-mon[35364]: pgmap v466: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:19 smithi181 ceph-mon[35869]: pgmap v466: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:20.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:19 smithi097 ceph-mon[32231]: pgmap v466: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:20.487 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:50:20.956 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:20 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2496241635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:50:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:20 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2496241635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:50:21.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:20 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2496241635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:50:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:21 smithi115 ceph-mon[35364]: pgmap v467: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:21 smithi181 ceph-mon[35869]: pgmap v467: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:21 smithi097 ceph-mon[32231]: pgmap v467: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:22.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.919+0000 7fa594c00640 1 -- 172.21.15.97:0/129560667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 msgr2=0x7fa590112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:22.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.919+0000 7fa594c00640 1 --2- 172.21.15.97:0/129560667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 0x7fa590112090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa588009ca0 tx=0x7fa58802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:50:22.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.920+0000 7fa594c00640 1 -- 172.21.15.97:0/129560667 shutdown_connections 2024-07-26T07:50:22.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.920+0000 7fa594c00640 1 --2- 172.21.15.97:0/129560667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 0x7fa590112090 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:22.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.920+0000 7fa594c00640 1 --2- 172.21.15.97:0/129560667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa59010af10 0x7fa59010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:22.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.920+0000 7fa594c00640 1 --2- 172.21.15.97:0/129560667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5901055d0 0x7fa5901059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:22.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.920+0000 7fa594c00640 1 -- 172.21.15.97:0/129560667 >> 172.21.15.97:0/129560667 conn(0x7fa590101040 msgr2=0x7fa590103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:22.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.920+0000 7fa594c00640 1 -- 172.21.15.97:0/129560667 shutdown_connections 2024-07-26T07:50:22.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.920+0000 7fa594c00640 1 -- 172.21.15.97:0/129560667 wait complete. 2024-07-26T07:50:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.921+0000 7fa594c00640 1 Processor -- start 2024-07-26T07:50:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.921+0000 7fa594c00640 1 -- start start 2024-07-26T07:50:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa594c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5901055d0 0x7fa59019bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa594c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa59010af10 0x7fa59019c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa594c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 0x7fa5901a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa594c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa590115410 con 0x7fa59010b860 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa594c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa590115290 con 0x7fa5901055d0 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa594c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa590115590 con 0x7fa59010af10 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa58fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 0x7fa5901a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa58fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 0x7fa5901a1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39706/0 (socket says 172.21.15.97:39706) 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa58fe00640 1 -- 172.21.15.97:0/1668021609 learned_addr learned my addr 172.21.15.97:0/1668021609 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa58f400640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5901055d0 0x7fa59019bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.922+0000 7fa58ea00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa59010af10 0x7fa59019c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa58fe00640 1 -- 172.21.15.97:0/1668021609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa59010af10 msgr2=0x7fa59019c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa58fe00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa59010af10 0x7fa59019c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa58fe00640 1 -- 172.21.15.97:0/1668021609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5901055d0 msgr2=0x7fa59019bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa58fe00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5901055d0 0x7fa59019bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa58fe00640 1 -- 172.21.15.97:0/1668021609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5901a1d30 con 0x7fa59010b860 2024-07-26T07:50:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa58fe00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 0x7fa5901a1600 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa588031040 tx=0x7fa588032bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa587e00640 1 -- 172.21.15.97:0/1668021609 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa588004040 con 0x7fa59010b860 2024-07-26T07:50:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa587e00640 1 -- 172.21.15.97:0/1668021609 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa588004670 con 0x7fa59010b860 2024-07-26T07:50:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa587e00640 1 -- 172.21.15.97:0/1668021609 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa588030bb0 con 0x7fa59010b860 2024-07-26T07:50:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.923+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5901a1fc0 con 0x7fa59010b860 2024-07-26T07:50:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.924+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5901a8780 con 0x7fa59010b860 2024-07-26T07:50:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.925+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa590106850 con 0x7fa59010b860 2024-07-26T07:50:22.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.929+0000 7fa587e00640 1 -- 172.21.15.97:0/1668021609 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fa5880041e0 con 0x7fa59010b860 2024-07-26T07:50:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.930+0000 7fa587e00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5740b3c50 0x7fa5740b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.931+0000 7fa587e00640 1 -- 172.21.15.97:0/1668021609 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fa5881051f0 con 0x7fa59010b860 2024-07-26T07:50:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.931+0000 7fa58f400640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5740b3c50 0x7fa5740b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.931+0000 7fa58f400640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5740b3c50 0x7fa5740b6160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa58000bbe0 tx=0x7fa58000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:22.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:22.932+0000 7fa587e00640 1 -- 172.21.15.97:0/1668021609 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa5880cc780 con 0x7fa59010b860 2024-07-26T07:50:23.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.168+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5901059e0 con 0x7fa59010b860 2024-07-26T07:50:23.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.169+0000 7fa587e00640 1 -- 172.21.15.97:0/1668021609 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11990 (secure 0 0 0) 0x7fa5901059e0 con 0x7fa59010b860 2024-07-26T07:50:23.171 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:50:23.171 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":27,"btime":"2024-07-26T07:48:11:830543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:48:11.504745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:50:23.171 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 27 2024-07-26T07:50:23.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.172+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5740b3c50 msgr2=0x7fa5740b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:23.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.172+0000 7fa594c00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5740b3c50 0x7fa5740b6160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa58000bbe0 tx=0x7fa58000b390 comp rx=0 tx=0).stop 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.172+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 msgr2=0x7fa5901a1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 0x7fa5901a1600 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa588031040 tx=0x7fa588032bc0 comp rx=0 tx=0).stop 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 shutdown_connections 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5740b3c50 0x7fa5740b6160 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa59010b860 0x7fa5901a1600 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa59010af10 0x7fa59019c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 --2- 172.21.15.97:0/1668021609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5901055d0 0x7fa59019bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 >> 172.21.15.97:0/1668021609 conn(0x7fa590101040 msgr2=0x7fa590103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 shutdown_connections 2024-07-26T07:50:23.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:23.173+0000 7fa594c00640 1 -- 172.21.15.97:0/1668021609 wait complete. 2024-07-26T07:50:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:23 smithi097 ceph-mon[32231]: pgmap v468: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:23 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1668021609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:23.882 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T07:47:48.482950+0000', 'modified': '2024-07-26T07:48:11.504745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 34583, 'mds_1': 34556, 'mds_2': 34535, 'mds_3': 34517, 'mds_4': 24562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14748': {'gid': 14748, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6839/4174464953', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6838', 'nonce': 4174464953}, {'type': 'v1', 'addr': '172.21.15.97:6839', 'nonce': 4174464953}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14760': {'gid': 14760, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6841/4022202837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6840', 'nonce': 4022202837}, {'type': 'v1', 'addr': '172.21.15.97:6841', 'nonce': 4022202837}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6833/1511551515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6832', 'nonce': 1511551515}, {'type': 'v1', 'addr': '172.21.15.181:6833', 'nonce': 1511551515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6835/4001900111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 4001900111}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 4001900111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6837/3053625422', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 3053625422}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 3053625422}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'i', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.181:6839/4258683328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 4258683328}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 4258683328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34517': {'gid': 34517, 'name': 'k', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.115:6833/1028321962', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6832', 'nonce': 1028321962}, {'type': 'v1', 'addr': '172.21.15.115:6833', 'nonce': 1028321962}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34535': {'gid': 34535, 'name': 'e', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.115:6835/4018170367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6834', 'nonce': 4018170367}, {'type': 'v1', 'addr': '172.21.15.115:6835', 'nonce': 4018170367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34556': {'gid': 34556, 'name': 'h', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.115:6837/3478625369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6836', 'nonce': 3478625369}, {'type': 'v1', 'addr': '172.21.15.115:6837', 'nonce': 3478625369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34583': {'gid': 34583, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.115:6839/1903638498', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6838', 'nonce': 1903638498}, {'type': 'v1', 'addr': '172.21.15.115:6839', 'nonce': 1903638498}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34583, 'qdb_cluster': [24562, 34517, 34556, 34535, 34583]} 2024-07-26T07:50:23.882 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T07:50:23.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.i status 2024-07-26T07:50:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:23 smithi115 ceph-mon[35364]: pgmap v468: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:23 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1668021609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:23 smithi181 ceph-mon[35869]: pgmap v468: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:23 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1668021609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:50:24.351 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:25.846 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:25 smithi097 ceph-mon[32231]: pgmap v469: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:25 smithi115 ceph-mon[35364]: pgmap v469: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:25 smithi181 ceph-mon[35869]: pgmap v469: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:26.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 -- 172.21.15.97:0/802689536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d90104b40 msgr2=0x7f9d9010d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 --2- 172.21.15.97:0/802689536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d90104b40 0x7f9d9010d090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9d78009900 tx=0x7f9d7802a410 comp rx=0 tx=0).stop 2024-07-26T07:50:26.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 -- 172.21.15.97:0/802689536 shutdown_connections 2024-07-26T07:50:26.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 --2- 172.21.15.97:0/802689536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d9010d650 0x7f9d9010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 --2- 172.21.15.97:0/802689536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d90104b40 0x7f9d9010d090 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 --2- 172.21.15.97:0/802689536 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d901041f0 0x7f9d90104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 -- 172.21.15.97:0/802689536 >> 172.21.15.97:0/802689536 conn(0x7f9d900fdcd0 msgr2=0x7f9d90100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:26.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 -- 172.21.15.97:0/802689536 shutdown_connections 2024-07-26T07:50:26.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.347+0000 7f9d95000640 1 -- 172.21.15.97:0/802689536 wait complete. 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.348+0000 7f9d95000640 1 Processor -- start 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d95000640 1 -- start start 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d95000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d901041f0 0x7f9d90199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d95000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d90104b40 0x7f9d9019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d95000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d9010d650 0x7f9d9019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d95000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9d901131f0 con 0x7f9d901041f0 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d95000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9d90113070 con 0x7f9d9010d650 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d95000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9d90113370 con 0x7f9d90104b40 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d8f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d901041f0 0x7f9d90199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d8f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d901041f0 0x7f9d90199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58022/0 (socket says 172.21.15.97:58022) 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.349+0000 7f9d8f400640 1 -- 172.21.15.97:0/2014833133 learned_addr learned my addr 172.21.15.97:0/2014833133 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.350+0000 7f9d87e00640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d90104b40 0x7f9d9019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.350+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d9010d650 0x7f9d9019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.350+0000 7f9d8f400640 1 -- 172.21.15.97:0/2014833133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d90104b40 msgr2=0x7f9d9019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.350+0000 7f9d8f400640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d90104b40 0x7f9d9019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.350+0000 7f9d8f400640 1 -- 172.21.15.97:0/2014833133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d9010d650 msgr2=0x7f9d9019f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.350+0000 7f9d8f400640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d9010d650 0x7f9d9019f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.350+0000 7f9d8f400640 1 -- 172.21.15.97:0/2014833133 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9019fbb0 con 0x7f9d901041f0 2024-07-26T07:50:26.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.350+0000 7f9d8f400640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d901041f0 0x7f9d90199c40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000d950 tx=0x7f9d8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:26.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.351+0000 7f9d8cc00640 1 -- 172.21.15.97:0/2014833133 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80017070 con 0x7f9d901041f0 2024-07-26T07:50:26.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.351+0000 7f9d8cc00640 1 -- 172.21.15.97:0/2014833133 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d80009c30 con 0x7f9d901041f0 2024-07-26T07:50:26.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.351+0000 7f9d8cc00640 1 -- 172.21.15.97:0/2014833133 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8000f620 con 0x7f9d901041f0 2024-07-26T07:50:26.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.351+0000 7f9d95000640 1 -- 172.21.15.97:0/2014833133 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d9019fea0 con 0x7f9d901041f0 2024-07-26T07:50:26.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.351+0000 7f9d95000640 1 -- 172.21.15.97:0/2014833133 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d901a6600 con 0x7f9d901041f0 2024-07-26T07:50:26.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.354+0000 7f9d9d246740 1 Processor -- start 2024-07-26T07:50:26.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.354+0000 7f9d9d246740 1 -- start start 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.354+0000 7f9d8cc00640 1 -- 172.21.15.97:0/2014833133 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f9d80010460 con 0x7f9d901041f0 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762fcf60 0x561d766a2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762d40a0 0x561d766a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 0x561d766a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d9d246740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561d76618970 con 0x561d762d40a0 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d9d246740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561d76607e70 con 0x561d762dc280 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d9d246740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561d7660beb0 con 0x561d762fcf60 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d8f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762d40a0 0x561d766a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d8f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762d40a0 0x561d766a2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58038/0 (socket says 172.21.15.97:58038) 2024-07-26T07:50:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d8f400640 1 -- 172.21.15.97:0/308017567 learned_addr learned my addr 172.21.15.97:0/308017567 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:26.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762fcf60 0x561d766a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d87e00640 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 0x561d766a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d8f400640 1 -- 172.21.15.97:0/308017567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762fcf60 msgr2=0x561d766a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:26.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d8f400640 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762fcf60 0x561d766a2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d8f400640 1 -- 172.21.15.97:0/308017567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 msgr2=0x561d766a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:26.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.355+0000 7f9d8f400640 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 0x561d766a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.356+0000 7f9d8f400640 1 -- 172.21.15.97:0/308017567 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d765af110 con 0x561d762d40a0 2024-07-26T07:50:26.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.356+0000 7f9d8f400640 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762d40a0 0x561d766a2cc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9d800e9c10 tx=0x7f9d800eb690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522a11d9474cd040 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:26.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.356+0000 7f9d84c00640 1 -- 172.21.15.97:0/308017567 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d800f2070 con 0x561d762d40a0 2024-07-26T07:50:26.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.356+0000 7f9d84c00640 1 -- 172.21.15.97:0/308017567 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d800eead0 con 0x561d762d40a0 2024-07-26T07:50:26.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.356+0000 7f9d84c00640 1 -- 172.21.15.97:0/308017567 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d800ed990 con 0x561d762d40a0 2024-07-26T07:50:26.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.357+0000 7f9d8cc00640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9d740b3c50 0x7f9d740b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.357+0000 7f9d8cc00640 1 -- 172.21.15.97:0/2014833133 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f9d800e38a0 con 0x7f9d901041f0 2024-07-26T07:50:26.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.357+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9d740b3c50 0x7f9d740b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.357+0000 7f9d9d246740 1 -- 172.21.15.97:0/308017567 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762d40a0 msgr2=0x561d766a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.357+0000 7f9d9d246740 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762d40a0 0x561d766a2cc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9d800e9c10 tx=0x7f9d800eb690 comp rx=0 tx=0).stop 2024-07-26T07:50:26.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d9d246740 1 -- 172.21.15.97:0/308017567 shutdown_connections 2024-07-26T07:50:26.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d9d246740 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 0x561d766a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d9d246740 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762d40a0 0x561d766a2cc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d9d246740 1 --2- 172.21.15.97:0/308017567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762fcf60 0x561d766a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d9d246740 1 -- 172.21.15.97:0/308017567 >> 172.21.15.97:0/308017567 conn(0x561d762f1350 msgr2=0x561d766a0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d9d246740 1 -- 172.21.15.97:0/308017567 shutdown_connections 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9d740b3c50 0x7f9d740b6160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009fd0 tx=0x7f9d8800e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d9d246740 1 -- 172.21.15.97:0/308017567 wait complete. 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.358+0000 7f9d9d246740 1 -- start start 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.359+0000 7f9d9d246740 1 Processor -- start 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.359+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 0x561d766adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.359+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762dc280 0x561d766ae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.359+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762fcf60 0x561d766ba280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.359+0000 7f9d9d246740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561d7646b5f0 con 0x561d762dc280 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.359+0000 7f9d9d246740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561d765a4540 con 0x561d762fcf60 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.359+0000 7f9d9d246740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561d766b0f60 con 0x561d762d40a0 2024-07-26T07:50:26.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.359+0000 7f9d9d246740 10 client.0 _ll_register_callbacks cb 0x561d765b3f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:26.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762dc280 0x561d766ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762dc280 0x561d766ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58044/0 (socket says 172.21.15.97:58044) 2024-07-26T07:50:26.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d87e00640 1 -- 172.21.15.97:0/4117446346 learned_addr learned my addr 172.21.15.97:0/4117446346 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:26.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d8f400640 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 0x561d766adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762fcf60 0x561d766ba280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d87e00640 1 -- 172.21.15.97:0/4117446346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 msgr2=0x561d766adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:26.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d87e00640 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 0x561d766adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d87e00640 1 -- 172.21.15.97:0/4117446346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762fcf60 msgr2=0x561d766ba280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:26.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d87e00640 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762fcf60 0x561d766ba280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.360+0000 7f9d87e00640 1 -- 172.21.15.97:0/4117446346 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d76368540 con 0x561d762dc280 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.361+0000 7f9d87e00640 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762dc280 0x561d766ae570 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9d9006af60 tx=0x7f9d78005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc16a24693a0d42 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.361+0000 7f9d7d600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.361+0000 7f9d7d600640 1 -- 172.21.15.97:0/4117446346 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78035030 con 0x561d762dc280 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.361+0000 7f9d7d600640 1 -- 172.21.15.97:0/4117446346 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f9d780048c0 con 0x561d762dc280 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.361+0000 7f9d9d246740 1 -- 172.21.15.97:0/4117446346 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d800eead0 con 0x561d762dc280 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.361+0000 7f9d7d600640 1 -- 172.21.15.97:0/4117446346 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9d78004ac0 con 0x561d762dc280 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.361+0000 7f9d9d246740 10 client.15120 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.361+0000 7f9d9d246740 1 -- 172.21.15.97:0/4117446346 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561d766bce20 con 0x561d762dc280 2024-07-26T07:50:26.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.362+0000 7f9d7d600640 1 -- 172.21.15.97:0/4117446346 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d800eead0 con 0x561d762dc280 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.362+0000 7f9d7d600640 1 -- 172.21.15.97:0/4117446346 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9d78044470 con 0x561d762dc280 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.362+0000 7f9d9d246740 10 client.15120 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.362+0000 7f9d9d246740 10 client.15120 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.362+0000 7f9d9d246740 20 client.15120 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.362+0000 7f9d9d246740 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561d762bb8e0 0x561d766c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.362+0000 7f9d9d246740 4 client.15120 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.362+0000 7f9d9d246740 1 -- 172.21.15.97:0/4117446346 --> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d7661a8c0 con 0x561d762bb8e0 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.363+0000 7f9d8f400640 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561d762bb8e0 0x561d766c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.364+0000 7f9d8f400640 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561d762bb8e0 0x561d766c08a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7ba6918a7aa1b89d server_cookie=7ffad529d997e7a9 in_seq=0 out_seq=0 2024-07-26T07:50:26.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.364+0000 7f9d7d600640 10 client.15120 ms_handle_connect on v2:172.21.15.181:6838/4258683328 2024-07-26T07:50:26.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.366+0000 7f9d7d600640 1 -- 172.21.15.97:0/4117446346 <== mds.4 v2:172.21.15.181:6838/4258683328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561d7661a8c0 con 0x561d762bb8e0 2024-07-26T07:50:26.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.366+0000 7f9d7d600640 10 client.15120 handle_command_reply: tid=0 2024-07-26T07:50:26.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.366+0000 7f9d9d246740 1 client.15120 shutdown 2024-07-26T07:50:26.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.366+0000 7f9d9d246740 1 -- 172.21.15.97:0/4117446346 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762dc280 msgr2=0x561d766ae570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.366+0000 7f9d9d246740 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762dc280 0x561d766ae570 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9d9006af60 tx=0x7f9d78005250 comp rx=0 tx=0).stop 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 1 -- 172.21.15.97:0/4117446346 shutdown_connections 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561d762bb8e0 0x561d766c08a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762fcf60 0x561d766ba280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762dc280 0x561d766ae570 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 1 --2- 172.21.15.97:0/4117446346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 0x561d766adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d7d600640 0 client.15120 ms_handle_reset on v2:172.21.15.181:6838/4258683328 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 1 -- 172.21.15.97:0/4117446346 >> 172.21.15.97:0/4117446346 conn(0x561d762f1350 msgr2=0x561d766a5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 1 -- 172.21.15.97:0/4117446346 shutdown_connections 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 1 -- 172.21.15.97:0/4117446346 wait complete. 2024-07-26T07:50:26.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.367+0000 7f9d9d246740 20 client.15120 trim_cache size 0 max 16384 2024-07-26T07:50:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.386+0000 7f9d9d246740 1 Processor -- start 2024-07-26T07:50:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.386+0000 7f9d9d246740 1 -- start start 2024-07-26T07:50:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762d40a0 0x561d766a88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762dc280 0x561d766c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766ba1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d9d246740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561d766bebe0 con 0x561d762fcf60 2024-07-26T07:50:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d9d246740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561d766bf2b0 con 0x561d762d40a0 2024-07-26T07:50:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d9d246740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561d766bf980 con 0x561d762dc280 2024-07-26T07:50:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d8fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766ba1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d8fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766ba1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58048/0 (socket says 172.21.15.97:58048) 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d8fe00640 1 -- 172.21.15.97:0/1338475655 learned_addr learned my addr 172.21.15.97:0/1338475655 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d87e00640 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762dc280 0x561d766c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.387+0000 7f9d8f400640 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762d40a0 0x561d766a88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d8fe00640 1 -- 172.21.15.97:0/1338475655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762dc280 msgr2=0x561d766c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762dc280 0x561d766c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d8fe00640 1 -- 172.21.15.97:0/1338475655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762d40a0 msgr2=0x561d766a88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762d40a0 0x561d766a88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d8fe00640 1 -- 172.21.15.97:0/1338475655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d7669b7f0 con 0x561d762fcf60 2024-07-26T07:50:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d8f400640 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762d40a0 0x561d766a88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:50:26.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766ba1f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9d880070b0 tx=0x7f9d8800c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b867214217c572 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:26.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d7fe00640 1 -- 172.21.15.97:0/1338475655 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d88027070 con 0x561d762fcf60 2024-07-26T07:50:26.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d7fe00640 1 -- 172.21.15.97:0/1338475655 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d88016ca0 con 0x561d762fcf60 2024-07-26T07:50:26.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.388+0000 7f9d7fe00640 1 -- 172.21.15.97:0/1338475655 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d880075d0 con 0x561d762fcf60 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 -- 172.21.15.97:0/1338475655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 msgr2=0x561d766ba1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766ba1f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9d880070b0 tx=0x7f9d8800c050 comp rx=0 tx=0).stop 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 -- 172.21.15.97:0/1338475655 shutdown_connections 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766ba1f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762dc280 0x561d766c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 --2- 172.21.15.97:0/1338475655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762d40a0 0x561d766a88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 -- 172.21.15.97:0/1338475655 >> 172.21.15.97:0/1338475655 conn(0x561d762f1350 msgr2=0x561d766ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 -- 172.21.15.97:0/1338475655 shutdown_connections 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.389+0000 7f9d9d246740 1 -- 172.21.15.97:0/1338475655 wait complete. 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.390+0000 7f9d9d246740 1 -- start start 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.390+0000 7f9d9d246740 1 Processor -- start 2024-07-26T07:50:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 0x561d766c6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 0x561d766c6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d9d246740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766c1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d9d246740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561d766c0720 con 0x561d762fcf60 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d9d246740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561d766c0050 con 0x561d762dc280 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d9d246740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561d766c0df0 con 0x561d762d40a0 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d9d246740 10 client.0 _ll_register_callbacks cb 0x7f9d78004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d8fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766c1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d8fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766c1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58056/0 (socket says 172.21.15.97:58056) 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d8fe00640 1 -- 172.21.15.97:0/2175081399 learned_addr learned my addr 172.21.15.97:0/2175081399 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:26.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d8f400640 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 0x561d766c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.391+0000 7f9d87e00640 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 0x561d766c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d8fe00640 1 -- 172.21.15.97:0/2175081399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 msgr2=0x561d766c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 0x561d766c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d8fe00640 1 -- 172.21.15.97:0/2175081399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 msgr2=0x561d766c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 0x561d766c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d8fe00640 1 -- 172.21.15.97:0/2175081399 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d766a55e0 con 0x561d762fcf60 2024-07-26T07:50:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d8fe00640 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766c1aa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009e00 tx=0x7f9d880175b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b52aa50130e8594 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d7d600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:26.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d7d600640 1 -- 172.21.15.97:0/2175081399 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d88032060 con 0x561d762fcf60 2024-07-26T07:50:26.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d9d246740 1 -- 172.21.15.97:0/2175081399 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d88016ca0 con 0x561d762fcf60 2024-07-26T07:50:26.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.392+0000 7f9d7d600640 1 -- 172.21.15.97:0/2175081399 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f9d88016aa0 con 0x561d762fcf60 2024-07-26T07:50:26.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.393+0000 7f9d7d600640 1 -- 172.21.15.97:0/2175081399 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9d88027030 con 0x561d762fcf60 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.393+0000 7f9d9d246740 10 client.15126 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.393+0000 7f9d9d246740 1 -- 172.21.15.97:0/2175081399 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561d766c8770 con 0x561d762fcf60 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.393+0000 7f9d7d600640 1 -- 172.21.15.97:0/2175081399 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d88016ca0 con 0x561d762fcf60 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.393+0000 7f9d7d600640 1 -- 172.21.15.97:0/2175081399 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9d88036030 con 0x561d762fcf60 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.393+0000 7f9d9d246740 10 client.15126 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.393+0000 7f9d9d246740 10 client.15126 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.393+0000 7f9d9d246740 20 client.15126 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.394+0000 7f9d9d246740 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561d76290c90 0x561d76734640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.394+0000 7f9d9d246740 4 client.15126 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.394+0000 7f9d9d246740 1 -- 172.21.15.97:0/2175081399 --> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] -- command(tid 0: {"prefix": "status"}) -- 0x561d76616360 con 0x561d76290c90 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.394+0000 7f9d8f400640 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561d76290c90 0x561d76734640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.395+0000 7f9d8f400640 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561d76290c90 0x561d76734640 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c33578c6ca45d59a server_cookie=b65bde03b92d0167 in_seq=0 out_seq=0 2024-07-26T07:50:26.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.395+0000 7f9d7d600640 10 client.15126 ms_handle_connect on v2:172.21.15.181:6838/4258683328 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d7d600640 1 -- 172.21.15.97:0/2175081399 <== mds.4 v2:172.21.15.181:6838/4258683328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561d76616360 con 0x561d76290c90 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d7d600640 10 client.15126 handle_command_reply: tid=0 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 client.15126 shutdown 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 -- 172.21.15.97:0/2175081399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 msgr2=0x561d766c1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766c1aa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009e00 tx=0x7f9d880175b0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 -- 172.21.15.97:0/2175081399 shutdown_connections 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561d76290c90 0x561d76734640 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d762fcf60 0x561d766c1aa0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561d762dc280 0x561d766c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 --2- 172.21.15.97:0/2175081399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d762d40a0 0x561d766c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 -- 172.21.15.97:0/2175081399 >> 172.21.15.97:0/2175081399 conn(0x561d762f1350 msgr2=0x561d766add80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d7d600640 0 client.15126 ms_handle_reset on v2:172.21.15.181:6838/4258683328 2024-07-26T07:50:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 -- 172.21.15.97:0/2175081399 shutdown_connections 2024-07-26T07:50:26.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.396+0000 7f9d9d246740 1 -- 172.21.15.97:0/2175081399 wait complete. 2024-07-26T07:50:26.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.397+0000 7f9d9d246740 20 client.15126 trim_cache size 0 max 16384 2024-07-26T07:50:26.400 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:50:26.400 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:50:26.400 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 4, 2024-07-26T07:50:26.400 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24562, 2024-07-26T07:50:26.400 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:50:26.400 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:50:26.401 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:50:26.401 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 139.598236125, 2024-07-26T07:50:26.401 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:50:26.401 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:50:26.401 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:50:26.401 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 190.98774839399999 2024-07-26T07:50:26.401 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.397+0000 7f9d95000640 1 -- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9d740b3c50 msgr2=0x7f9d740b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.397+0000 7f9d95000640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9d740b3c50 0x7f9d740b6160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009fd0 tx=0x7f9d8800e040 comp rx=0 tx=0).stop 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.397+0000 7f9d95000640 1 -- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d901041f0 msgr2=0x7f9d90199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.397+0000 7f9d95000640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d901041f0 0x7f9d90199c40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000d950 tx=0x7f9d8000de10 comp rx=0 tx=0).stop 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.397+0000 7f9d95000640 1 -- 172.21.15.97:0/2014833133 shutdown_connections 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.397+0000 7f9d95000640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9d740b3c50 0x7f9d740b6160 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.397+0000 7f9d95000640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d9010d650 0x7f9d9019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.398+0000 7f9d95000640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d90104b40 0x7f9d9019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.398+0000 7f9d95000640 1 --2- 172.21.15.97:0/2014833133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9d901041f0 0x7f9d90199c40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.398+0000 7f9d95000640 1 -- 172.21.15.97:0/2014833133 >> 172.21.15.97:0/2014833133 conn(0x7f9d900fdcd0 msgr2=0x7f9d901071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.398+0000 7f9d95000640 1 -- 172.21.15.97:0/2014833133 shutdown_connections 2024-07-26T07:50:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:26.398+0000 7f9d95000640 1 -- 172.21.15.97:0/2014833133 wait complete. 2024-07-26T07:50:27.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.k status 2024-07-26T07:50:27.542 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:27.958 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:27 smithi097 ceph-mon[32231]: pgmap v470: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:27 smithi115 ceph-mon[35364]: pgmap v470: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:27 smithi181 ceph-mon[35869]: pgmap v470: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.483+0000 7fabeec00640 1 -- 172.21.15.97:0/3666389280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 msgr2=0x7fabe81059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.483+0000 7fabeec00640 1 --2- 172.21.15.97:0/3666389280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 0x7fabe81059e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009900 tx=0x7fabdc02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.484+0000 7fabeec00640 1 -- 172.21.15.97:0/3666389280 shutdown_connections 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.484+0000 7fabeec00640 1 --2- 172.21.15.97:0/3666389280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe810b860 0x7fabe8112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.484+0000 7fabeec00640 1 --2- 172.21.15.97:0/3666389280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe810af10 0x7fabe810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.484+0000 7fabeec00640 1 --2- 172.21.15.97:0/3666389280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 0x7fabe81059e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.484+0000 7fabeec00640 1 -- 172.21.15.97:0/3666389280 >> 172.21.15.97:0/3666389280 conn(0x7fabe8101040 msgr2=0x7fabe8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.484+0000 7fabeec00640 1 -- 172.21.15.97:0/3666389280 shutdown_connections 2024-07-26T07:50:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.484+0000 7fabeec00640 1 -- 172.21.15.97:0/3666389280 wait complete. 2024-07-26T07:50:29.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.485+0000 7fabeec00640 1 Processor -- start 2024-07-26T07:50:29.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.485+0000 7fabeec00640 1 -- start start 2024-07-26T07:50:29.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.485+0000 7fabeec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 0x7fabe819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.485+0000 7fabeec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe810af10 0x7fabe819c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.485+0000 7fabeec00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe810b860 0x7fabe81a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.485+0000 7fabeec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fabe81153e0 con 0x7fabe81055d0 2024-07-26T07:50:29.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.485+0000 7fabeec00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fabe8115260 con 0x7fabe810b860 2024-07-26T07:50:29.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.485+0000 7fabeec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fabe8115560 con 0x7fabe810af10 2024-07-26T07:50:29.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabed800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 0x7fabe819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabed800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 0x7fabe819bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58084/0 (socket says 172.21.15.97:58084) 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabed800640 1 -- 172.21.15.97:0/3903679153 learned_addr learned my addr 172.21.15.97:0/3903679153 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabee200640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe810b860 0x7fabe81a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabece00640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe810af10 0x7fabe819c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabed800640 1 -- 172.21.15.97:0/3903679153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe810af10 msgr2=0x7fabe819c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabed800640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe810af10 0x7fabe819c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabed800640 1 -- 172.21.15.97:0/3903679153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe810b860 msgr2=0x7fabe81a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabed800640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe810b860 0x7fabe81a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.486+0000 7fabed800640 1 -- 172.21.15.97:0/3903679153 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe81a1de0 con 0x7fabe81055d0 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.487+0000 7fabed800640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 0x7fabe819bf30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009a20 tx=0x7fabdc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.487+0000 7fabe6000640 1 -- 172.21.15.97:0/3903679153 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc03f030 con 0x7fabe81055d0 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.487+0000 7fabe6000640 1 -- 172.21.15.97:0/3903679153 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fabdc0035f0 con 0x7fabe81055d0 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.487+0000 7fabeec00640 1 -- 172.21.15.97:0/3903679153 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe81a2070 con 0x7fabe81055d0 2024-07-26T07:50:29.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.487+0000 7fabe6000640 1 -- 172.21.15.97:0/3903679153 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc035070 con 0x7fabe81055d0 2024-07-26T07:50:29.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.487+0000 7fabeec00640 1 -- 172.21.15.97:0/3903679153 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe81a2550 con 0x7fabe81055d0 2024-07-26T07:50:29.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabf7025740 1 Processor -- start 2024-07-26T07:50:29.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabf7025740 1 -- start start 2024-07-26T07:50:29.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabe6000640 1 -- 172.21.15.97:0/3903679153 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fabdc003aa0 con 0x7fabe81055d0 2024-07-26T07:50:29.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabf7025740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd44df60 0x55f2cd7f3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabf7025740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd4250a0 0x55f2cd7f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabf7025740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 0x55f2cd7f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabf7025740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd769950 con 0x55f2cd4250a0 2024-07-26T07:50:29.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabf7025740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd758e50 con 0x55f2cd44df60 2024-07-26T07:50:29.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabf7025740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd75ce90 con 0x55f2cd42d280 2024-07-26T07:50:29.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.491+0000 7fabed800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd4250a0 0x55f2cd7f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabed800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd4250a0 0x55f2cd7f3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58098/0 (socket says 172.21.15.97:58098) 2024-07-26T07:50:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabed800640 1 -- 172.21.15.97:0/4171037796 learned_addr learned my addr 172.21.15.97:0/4171037796 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabee200640 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd44df60 0x55f2cd7f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabece00640 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 0x55f2cd7f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabed800640 1 -- 172.21.15.97:0/4171037796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 msgr2=0x55f2cd7f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabed800640 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 0x55f2cd7f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabed800640 1 -- 172.21.15.97:0/4171037796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd44df60 msgr2=0x55f2cd7f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabed800640 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd44df60 0x55f2cd7f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.492+0000 7fabed800640 1 -- 172.21.15.97:0/4171037796 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2cd7000f0 con 0x55f2cd4250a0 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.493+0000 7fabed800640 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd4250a0 0x55f2cd7f3ca0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fabdc10ac10 tx=0x7fabdc10ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6449bb4d382a3a5f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.493+0000 7fabdaa00640 1 -- 172.21.15.97:0/4171037796 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc113070 con 0x55f2cd4250a0 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.493+0000 7fabe6000640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fabd00b3c00 0x7fabd00b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.493+0000 7fabdaa00640 1 -- 172.21.15.97:0/4171037796 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fabdc10ccb0 con 0x55f2cd4250a0 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.493+0000 7fabdaa00640 1 -- 172.21.15.97:0/4171037796 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc10e600 con 0x55f2cd4250a0 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.493+0000 7fabe6000640 1 -- 172.21.15.97:0/3903679153 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fabdc104890 con 0x7fabe81055d0 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.493+0000 7fabee200640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fabd00b3c00 0x7fabd00b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 -- 172.21.15.97:0/4171037796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd4250a0 msgr2=0x55f2cd7f3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabee200640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fabd00b3c00 0x7fabd00b6110 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fabe0007e50 tx=0x7fabe0007600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd4250a0 0x55f2cd7f3ca0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fabdc10ac10 tx=0x7fabdc10ad00 comp rx=0 tx=0).stop 2024-07-26T07:50:29.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 -- 172.21.15.97:0/4171037796 shutdown_connections 2024-07-26T07:50:29.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 0x55f2cd7f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd4250a0 0x55f2cd7f3ca0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 --2- 172.21.15.97:0/4171037796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd44df60 0x55f2cd7f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 -- 172.21.15.97:0/4171037796 >> 172.21.15.97:0/4171037796 conn(0x55f2cd442350 msgr2=0x55f2cd7f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:29.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 -- 172.21.15.97:0/4171037796 shutdown_connections 2024-07-26T07:50:29.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.494+0000 7fabf7025740 1 -- 172.21.15.97:0/4171037796 wait complete. 2024-07-26T07:50:29.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.495+0000 7fabf7025740 1 -- start start 2024-07-26T07:50:29.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.495+0000 7fabf7025740 1 Processor -- start 2024-07-26T07:50:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabf7025740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd7feff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabf7025740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd7ff5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabf7025740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabf7025740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd5bc5f0 con 0x55f2cd44df60 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabf7025740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd6f5520 con 0x55f2cd42d280 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabf7025740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd769950 con 0x55f2cd4250a0 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabf7025740 10 client.0 _ll_register_callbacks cb 0x55f2cd704ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabee200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabee200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58102/0 (socket says 172.21.15.97:58102) 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabee200640 1 -- 172.21.15.97:0/907489509 learned_addr learned my addr 172.21.15.97:0/907489509 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.496+0000 7fabed800640 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd7feff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabece00640 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd7ff5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabee200640 1 -- 172.21.15.97:0/907489509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 msgr2=0x55f2cd7feff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabee200640 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd7feff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabee200640 1 -- 172.21.15.97:0/907489509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 msgr2=0x55f2cd7ff5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabee200640 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd7ff5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabee200640 1 -- 172.21.15.97:0/907489509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2cd4b9540 con 0x55f2cd44df60 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabee200640 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80ade0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fabe000ed80 tx=0x7fabe0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d4c22d7ae0c304 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabcf400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.497+0000 7fabcf400640 1 -- 172.21.15.97:0/907489509 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe001ccb0 con 0x55f2cd44df60 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabcf400640 1 -- 172.21.15.97:0/907489509 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fabe0027430 con 0x55f2cd44df60 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabf7025740 1 -- 172.21.15.97:0/907489509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabdc10ccb0 con 0x55f2cd44df60 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabcf400640 1 -- 172.21.15.97:0/907489509 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fabe001ce50 con 0x55f2cd44df60 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabf7025740 10 client.15138 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabf7025740 1 -- 172.21.15.97:0/907489509 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f2cd80da60 con 0x55f2cd44df60 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabcf400640 1 -- 172.21.15.97:0/907489509 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fabdc10ccb0 con 0x55f2cd44df60 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabcf400640 1 -- 172.21.15.97:0/907489509 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fabe00316a0 con 0x55f2cd44df60 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabf7025740 10 client.15138 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.498+0000 7fabf7025740 10 client.15138 resolve_mds: resolved name 'k' to daemon mds.k 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.499+0000 7fabf7025740 20 client.15138 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:29.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.499+0000 7fabf7025740 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55f2cd40c8e0 0x55f2cd8114f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.499+0000 7fabf7025740 4 client.15138 mds_command: new command op to 34517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:50:29.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.499+0000 7fabf7025740 1 -- 172.21.15.97:0/907489509 --> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2cd76b8a0 con 0x55f2cd40c8e0 2024-07-26T07:50:29.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.499+0000 7fabed800640 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55f2cd40c8e0 0x55f2cd8114f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.500+0000 7fabed800640 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55f2cd40c8e0 0x55f2cd8114f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5ff021f51ccbda79 server_cookie=16efa18bc0efafa6 in_seq=0 out_seq=0 2024-07-26T07:50:29.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.500+0000 7fabcf400640 10 client.15138 ms_handle_connect on v2:172.21.15.115:6832/1028321962 2024-07-26T07:50:29.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabcf400640 1 -- 172.21.15.97:0/907489509 <== mds.3 v2:172.21.15.115:6832/1028321962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f2cd76b8a0 con 0x55f2cd40c8e0 2024-07-26T07:50:29.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabcf400640 10 client.15138 handle_command_reply: tid=0 2024-07-26T07:50:29.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 client.15138 shutdown 2024-07-26T07:50:29.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 -- 172.21.15.97:0/907489509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 msgr2=0x55f2cd80ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80ade0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fabe000ed80 tx=0x7fabe0004a40 comp rx=0 tx=0).stop 2024-07-26T07:50:29.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 -- 172.21.15.97:0/907489509 shutdown_connections 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80ade0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55f2cd40c8e0 0x55f2cd8114f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd7ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 --2- 172.21.15.97:0/907489509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd7feff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 -- 172.21.15.97:0/907489509 >> 172.21.15.97:0/907489509 conn(0x55f2cd442350 msgr2=0x55f2cd7f68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabcf400640 0 client.15138 ms_handle_reset on v2:172.21.15.115:6832/1028321962 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 -- 172.21.15.97:0/907489509 shutdown_connections 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.503+0000 7fabf7025740 1 -- 172.21.15.97:0/907489509 wait complete. 2024-07-26T07:50:29.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.504+0000 7fabf7025740 20 client.15138 trim_cache size 0 max 16384 2024-07-26T07:50:29.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.523+0000 7fabf7025740 1 Processor -- start 2024-07-26T07:50:29.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.523+0000 7fabf7025740 1 -- start start 2024-07-26T07:50:29.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.523+0000 7fabf7025740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd8114f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.523+0000 7fabf7025740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd80ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.523+0000 7fabf7025740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.523+0000 7fabf7025740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd80f830 con 0x55f2cd44df60 2024-07-26T07:50:29.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.523+0000 7fabf7025740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd80ff00 con 0x55f2cd42d280 2024-07-26T07:50:29.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.523+0000 7fabf7025740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd8105d0 con 0x55f2cd4250a0 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabee200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabee200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58116/0 (socket says 172.21.15.97:58116) 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabee200640 1 -- 172.21.15.97:0/4211398444 learned_addr learned my addr 172.21.15.97:0/4211398444 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabed800640 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd8114f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabece00640 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd80ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabee200640 1 -- 172.21.15.97:0/4211398444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 msgr2=0x55f2cd8114f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabee200640 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd8114f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabee200640 1 -- 172.21.15.97:0/4211398444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 msgr2=0x55f2cd80ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabee200640 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd80ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabee200640 1 -- 172.21.15.97:0/4211398444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2cd7fa880 con 0x55f2cd44df60 2024-07-26T07:50:29.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabece00640 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd80ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:29.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.524+0000 7fabed800640 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd8114f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:29.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.525+0000 7fabee200640 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80b290 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fabe0016040 tx=0x7fabe0007aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a6bf9f5e42b0a1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:29.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.525+0000 7fabda000640 1 -- 172.21.15.97:0/4211398444 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0031c30 con 0x55f2cd44df60 2024-07-26T07:50:29.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.525+0000 7fabda000640 1 -- 172.21.15.97:0/4211398444 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fabe0031dd0 con 0x55f2cd44df60 2024-07-26T07:50:29.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.525+0000 7fabda000640 1 -- 172.21.15.97:0/4211398444 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe00276d0 con 0x55f2cd44df60 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 -- 172.21.15.97:0/4211398444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 msgr2=0x55f2cd80b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80b290 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fabe0016040 tx=0x7fabe0007aa0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 -- 172.21.15.97:0/4211398444 shutdown_connections 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd80b290 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd42d280 0x55f2cd80ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 --2- 172.21.15.97:0/4211398444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd4250a0 0x55f2cd8114f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 -- 172.21.15.97:0/4211398444 >> 172.21.15.97:0/4211398444 conn(0x55f2cd442350 msgr2=0x55f2cd7fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 -- 172.21.15.97:0/4211398444 shutdown_connections 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.526+0000 7fabf7025740 1 -- 172.21.15.97:0/4211398444 wait complete. 2024-07-26T07:50:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.527+0000 7fabf7025740 1 -- start start 2024-07-26T07:50:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.527+0000 7fabf7025740 1 Processor -- start 2024-07-26T07:50:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.527+0000 7fabf7025740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd4250a0 0x55f2cd812a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.527+0000 7fabf7025740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 0x55f2cd812f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.527+0000 7fabf7025740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd878780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.527+0000 7fabf7025740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd811370 con 0x55f2cd44df60 2024-07-26T07:50:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.527+0000 7fabf7025740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd810ca0 con 0x55f2cd4250a0 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.527+0000 7fabf7025740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f2cd811a40 con 0x55f2cd42d280 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabf7025740 10 client.0 _ll_register_callbacks cb 0x7fabe000e750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabee200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd878780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabee200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd878780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58120/0 (socket says 172.21.15.97:58120) 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabee200640 1 -- 172.21.15.97:0/1795729618 learned_addr learned my addr 172.21.15.97:0/1795729618 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabed800640 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd4250a0 0x55f2cd812a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabece00640 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 0x55f2cd812f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabee200640 1 -- 172.21.15.97:0/1795729618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 msgr2=0x55f2cd812f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabee200640 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 0x55f2cd812f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabee200640 1 -- 172.21.15.97:0/1795729618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd4250a0 msgr2=0x55f2cd812a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabee200640 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd4250a0 0x55f2cd812a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.528+0000 7fabee200640 1 -- 172.21.15.97:0/1795729618 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2cd8137c0 con 0x55f2cd44df60 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.529+0000 7fabee200640 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd878780 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fabe0031a40 tx=0x7fabe0007380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a207d60d58cf3a26 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.529+0000 7fabcf400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:29.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.529+0000 7fabcf400640 1 -- 172.21.15.97:0/1795729618 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0041070 con 0x55f2cd44df60 2024-07-26T07:50:29.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.529+0000 7fabcf400640 1 -- 172.21.15.97:0/1795729618 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fabe0004dd0 con 0x55f2cd44df60 2024-07-26T07:50:29.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.529+0000 7fabf7025740 1 -- 172.21.15.97:0/1795729618 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabe0031dd0 con 0x55f2cd44df60 2024-07-26T07:50:29.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.529+0000 7fabcf400640 1 -- 172.21.15.97:0/1795729618 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fabe00276d0 con 0x55f2cd44df60 2024-07-26T07:50:29.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.529+0000 7fabf7025740 10 client.15144 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:29.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.529+0000 7fabf7025740 1 -- 172.21.15.97:0/1795729618 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f2cd879cd0 con 0x55f2cd44df60 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.530+0000 7fabcf400640 1 -- 172.21.15.97:0/1795729618 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fabe0031dd0 con 0x55f2cd44df60 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.530+0000 7fabcf400640 1 -- 172.21.15.97:0/1795729618 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fabe0005380 con 0x55f2cd44df60 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.530+0000 7fabf7025740 10 client.15144 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.530+0000 7fabf7025740 10 client.15144 resolve_mds: resolved name 'k' to daemon mds.k 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.530+0000 7fabf7025740 20 client.15144 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.530+0000 7fabf7025740 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55f2cd3e1c90 0x55f2cd87d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.530+0000 7fabf7025740 4 client.15144 mds_command: new command op to 34517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.530+0000 7fabf7025740 1 -- 172.21.15.97:0/1795729618 --> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] -- command(tid 0: {"prefix": "status"}) -- 0x55f2cd767340 con 0x55f2cd3e1c90 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.531+0000 7fabed800640 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55f2cd3e1c90 0x55f2cd87d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.532+0000 7fabed800640 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55f2cd3e1c90 0x55f2cd87d1d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=89c9591edcd0f8d7 server_cookie=7c482f45dd1043f6 in_seq=0 out_seq=0 2024-07-26T07:50:29.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.532+0000 7fabcf400640 10 client.15144 ms_handle_connect on v2:172.21.15.115:6832/1028321962 2024-07-26T07:50:29.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.532+0000 7fabcf400640 1 -- 172.21.15.97:0/1795729618 <== mds.3 v2:172.21.15.115:6832/1028321962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55f2cd767340 con 0x55f2cd3e1c90 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.532+0000 7fabcf400640 10 client.15144 handle_command_reply: tid=0 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.532+0000 7fabf7025740 1 client.15144 shutdown 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 -- 172.21.15.97:0/1795729618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 msgr2=0x55f2cd878780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd878780 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fabe0031a40 tx=0x7fabe0007380 comp rx=0 tx=0).stop 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 -- 172.21.15.97:0/1795729618 shutdown_connections 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f2cd44df60 0x55f2cd878780 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2cd42d280 0x55f2cd812f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55f2cd3e1c90 0x55f2cd87d1d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 --2- 172.21.15.97:0/1795729618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f2cd4250a0 0x55f2cd812a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabcf400640 0 client.15144 ms_handle_reset on v2:172.21.15.115:6832/1028321962 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 -- 172.21.15.97:0/1795729618 >> 172.21.15.97:0/1795729618 conn(0x55f2cd442350 msgr2=0x55f2cd7f6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 -- 172.21.15.97:0/1795729618 shutdown_connections 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 1 -- 172.21.15.97:0/1795729618 wait complete. 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.533+0000 7fabf7025740 20 client.15144 trim_cache size 0 max 16384 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 3, 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34517, 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:50:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 143.75837561, 2024-07-26T07:50:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:50:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:50:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:50:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 221.35129692300001 2024-07-26T07:50:29.537 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:50:29.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 -- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fabd00b3c00 msgr2=0x7fabd00b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fabd00b3c00 0x7fabd00b6110 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fabe0007e50 tx=0x7fabe0007600 comp rx=0 tx=0).stop 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 -- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 msgr2=0x7fabe819bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 0x7fabe819bf30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009a20 tx=0x7fabdc004270 comp rx=0 tx=0).stop 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 -- 172.21.15.97:0/3903679153 shutdown_connections 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fabd00b3c00 0x7fabd00b6110 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fabe810b860 0x7fabe81a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe810af10 0x7fabe819c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 --2- 172.21.15.97:0/3903679153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabe81055d0 0x7fabe819bf30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 -- 172.21.15.97:0/3903679153 >> 172.21.15.97:0/3903679153 conn(0x7fabe8101040 msgr2=0x7fabe8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 -- 172.21.15.97:0/3903679153 shutdown_connections 2024-07-26T07:50:29.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:29.534+0000 7fabeec00640 1 -- 172.21.15.97:0/3903679153 wait complete. 2024-07-26T07:50:29.845 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:29 smithi097 ceph-mon[32231]: pgmap v471: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:29 smithi115 ceph-mon[35364]: pgmap v471: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:29 smithi181 ceph-mon[35869]: pgmap v471: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:50:30.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.e status 2024-07-26T07:50:30.679 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:31 smithi181 ceph-mon[35869]: pgmap v472: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:32.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:31 smithi097 ceph-mon[32231]: pgmap v472: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:31 smithi115 ceph-mon[35364]: pgmap v472: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:32.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 -- 172.21.15.97:0/1617362714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812410f920 msgr2=0x7f8124111d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 --2- 172.21.15.97:0/1617362714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812410f920 0x7f8124111d60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f811c009ca0 tx=0x7f811c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 -- 172.21.15.97:0/1617362714 shutdown_connections 2024-07-26T07:50:32.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 --2- 172.21.15.97:0/1617362714 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f812410f920 0x7f8124111d60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 --2- 172.21.15.97:0/1617362714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8124075b30 0x7f812410f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 --2- 172.21.15.97:0/1617362714 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f81240751e0 0x7f81240755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 -- 172.21.15.97:0/1617362714 >> 172.21.15.97:0/1617362714 conn(0x7f81240fee10 msgr2=0x7f8124101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:32.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 -- 172.21.15.97:0/1617362714 shutdown_connections 2024-07-26T07:50:32.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.725+0000 7f8123200640 1 -- 172.21.15.97:0/1617362714 wait complete. 2024-07-26T07:50:32.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.726+0000 7f8123200640 1 Processor -- start 2024-07-26T07:50:32.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.727+0000 7f8123200640 1 -- start start 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.727+0000 7f8123200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f81240751e0 0x7f812419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.727+0000 7f8123200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8124075b30 0x7f812419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.727+0000 7f8123200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f812410f920 0x7f81241a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.727+0000 7f8123200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f81241153d0 con 0x7f81240751e0 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.727+0000 7f8123200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8124115250 con 0x7f812410f920 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.727+0000 7f8123200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8124115550 con 0x7f8124075b30 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8121e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f81240751e0 0x7f812419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8121e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f81240751e0 0x7f812419be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58136/0 (socket says 172.21.15.97:58136) 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8121e00640 1 -- 172.21.15.97:0/706449370 learned_addr learned my addr 172.21.15.97:0/706449370 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:32.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f811be00640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8124075b30 0x7f812419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8122800640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f812410f920 0x7f81241a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8121e00640 1 -- 172.21.15.97:0/706449370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8124075b30 msgr2=0x7f812419c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8121e00640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8124075b30 0x7f812419c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8121e00640 1 -- 172.21.15.97:0/706449370 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f812410f920 msgr2=0x7f81241a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8121e00640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f812410f920 0x7f81241a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.728+0000 7f8121e00640 1 -- 172.21.15.97:0/706449370 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81241a1cb0 con 0x7f81240751e0 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.729+0000 7f8121e00640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f81240751e0 0x7f812419be00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f811400c920 tx=0x7f811400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.729+0000 7f811aa00640 1 -- 172.21.15.97:0/706449370 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8114007c70 con 0x7f81240751e0 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.729+0000 7f811aa00640 1 -- 172.21.15.97:0/706449370 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8114007e10 con 0x7f81240751e0 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.729+0000 7f811aa00640 1 -- 172.21.15.97:0/706449370 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8114018710 con 0x7f81240751e0 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.730+0000 7f8123200640 1 -- 172.21.15.97:0/706449370 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81241a1fa0 con 0x7f81240751e0 2024-07-26T07:50:32.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.730+0000 7f8123200640 1 -- 172.21.15.97:0/706449370 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81241a87c0 con 0x7f81240751e0 2024-07-26T07:50:32.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.733+0000 7f812f522740 1 Processor -- start 2024-07-26T07:50:32.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.733+0000 7f812f522740 1 -- start start 2024-07-26T07:50:32.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f811aa00640 1 -- 172.21.15.97:0/706449370 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f811400f450 con 0x7f81240751e0 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f812f522740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56465738ff60 0x564657735780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f812f522740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646573670a0 0x564657735cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f812f522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x564657736200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f812f522740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5646576ab970 con 0x5646573670a0 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f812f522740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56465769ae70 con 0x56465738ff60 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f812f522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56465769eeb0 con 0x56465736f280 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f8121e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646573670a0 0x564657735cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f8121e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646573670a0 0x564657735cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58146/0 (socket says 172.21.15.97:58146) 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f8121e00640 1 -- 172.21.15.97:0/308073905 learned_addr learned my addr 172.21.15.97:0/308073905 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f811be00640 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x564657736200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.734+0000 7f8122800640 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56465738ff60 0x564657735780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f8121e00640 1 -- 172.21.15.97:0/308073905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 msgr2=0x564657736200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f8121e00640 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x564657736200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f8121e00640 1 -- 172.21.15.97:0/308073905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56465738ff60 msgr2=0x564657735780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f8121e00640 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56465738ff60 0x564657735780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f8121e00640 1 -- 172.21.15.97:0/308073905 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564657642110 con 0x5646573670a0 2024-07-26T07:50:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f811be00640 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x564657736200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:32.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f8122800640 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56465738ff60 0x564657735780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:32.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f8121e00640 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646573670a0 0x564657735cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f81140e7f10 tx=0x7f81140ea460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36c1b448f36a41f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.735+0000 7f811aa00640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81040b3c80 0x7f81040b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.736+0000 7f811aa00640 1 -- 172.21.15.97:0/706449370 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f81140e3e60 con 0x7f81240751e0 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.736+0000 7f8122800640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81040b3c80 0x7f81040b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.736+0000 7f8113000640 1 -- 172.21.15.97:0/308073905 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81140aa460 con 0x5646573670a0 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.736+0000 7f8113000640 1 -- 172.21.15.97:0/308073905 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81140aa800 con 0x5646573670a0 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.736+0000 7f8113000640 1 -- 172.21.15.97:0/308073905 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81140aa640 con 0x5646573670a0 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 -- 172.21.15.97:0/308073905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646573670a0 msgr2=0x564657735cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f8122800640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81040b3c80 0x7f81040b6190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f811c005cc0 tx=0x7f811c005c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646573670a0 0x564657735cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f81140e7f10 tx=0x7f81140ea460 comp rx=0 tx=0).stop 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 -- 172.21.15.97:0/308073905 shutdown_connections 2024-07-26T07:50:32.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x564657736200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646573670a0 0x564657735cc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 --2- 172.21.15.97:0/308073905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56465738ff60 0x564657735780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 -- 172.21.15.97:0/308073905 >> 172.21.15.97:0/308073905 conn(0x564657384350 msgr2=0x564657733670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 -- 172.21.15.97:0/308073905 shutdown_connections 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.737+0000 7f812f522740 1 -- 172.21.15.97:0/308073905 wait complete. 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 1 -- start start 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 1 Processor -- start 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x5646577412b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x5646577417f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465738ff60 0x56465774d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5646574fe5f0 con 0x56465738ff60 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564657637540 con 0x5646573670a0 2024-07-26T07:50:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564657743f60 con 0x56465736f280 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.738+0000 7f812f522740 10 client.0 _ll_register_callbacks cb 0x564657646f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8122800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465738ff60 0x56465774d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8122800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465738ff60 0x56465774d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58154/0 (socket says 172.21.15.97:58154) 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8122800640 1 -- 172.21.15.97:0/3728453420 learned_addr learned my addr 172.21.15.97:0/3728453420 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8121e00640 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x5646577412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f811be00640 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x5646577417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8122800640 1 -- 172.21.15.97:0/3728453420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 msgr2=0x5646577417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8122800640 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x5646577417f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8122800640 1 -- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 msgr2=0x5646577412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8122800640 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x5646577412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8122800640 1 -- 172.21.15.97:0/3728453420 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646573fb540 con 0x56465738ff60 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.739+0000 7f8121e00640 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x5646577412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.740+0000 7f8122800640 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465738ff60 0x56465774d250 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f811c035000 tx=0x7f811c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88347445b6ab83d8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.740+0000 7f810b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:32.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.740+0000 7f810b400640 1 -- 172.21.15.97:0/3728453420 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c04f070 con 0x56465738ff60 2024-07-26T07:50:32.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.740+0000 7f812f522740 1 -- 172.21.15.97:0/3728453420 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81140aa800 con 0x56465738ff60 2024-07-26T07:50:32.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.740+0000 7f810b400640 1 -- 172.21.15.97:0/3728453420 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f811c004510 con 0x56465738ff60 2024-07-26T07:50:32.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.740+0000 7f810b400640 1 -- 172.21.15.97:0/3728453420 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f811c004710 con 0x56465738ff60 2024-07-26T07:50:32.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.740+0000 7f812f522740 10 client.15156 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:32.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.740+0000 7f812f522740 1 -- 172.21.15.97:0/3728453420 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56465774fec0 con 0x56465738ff60 2024-07-26T07:50:32.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.741+0000 7f810b400640 1 -- 172.21.15.97:0/3728453420 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f81140aa800 con 0x56465738ff60 2024-07-26T07:50:32.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.741+0000 7f810b400640 1 -- 172.21.15.97:0/3728453420 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f811c044040 con 0x56465738ff60 2024-07-26T07:50:32.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.741+0000 7f812f522740 10 client.15156 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.741+0000 7f812f522740 10 client.15156 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T07:50:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.741+0000 7f812f522740 20 client.15156 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.741+0000 7f812f522740 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x56465734e8e0 0x564657753910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.741+0000 7f812f522740 4 client.15156 mds_command: new command op to 34535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:50:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.741+0000 7f812f522740 1 -- 172.21.15.97:0/3728453420 --> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646576ad8c0 con 0x56465734e8e0 2024-07-26T07:50:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.742+0000 7f8121e00640 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x56465734e8e0 0x564657753910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.743+0000 7f8121e00640 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x56465734e8e0 0x564657753910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1546ec28715954af server_cookie=1415b53eea05f777 in_seq=0 out_seq=0 2024-07-26T07:50:32.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.743+0000 7f810b400640 10 client.15156 ms_handle_connect on v2:172.21.15.115:6834/4018170367 2024-07-26T07:50:32.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.745+0000 7f810b400640 1 -- 172.21.15.97:0/3728453420 <== mds.2 v2:172.21.15.115:6834/4018170367 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5646576ad8c0 con 0x56465734e8e0 2024-07-26T07:50:32.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.745+0000 7f810b400640 10 client.15156 handle_command_reply: tid=0 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.745+0000 7f812f522740 1 client.15156 shutdown 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 -- 172.21.15.97:0/3728453420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465738ff60 msgr2=0x56465774d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465738ff60 0x56465774d250 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f811c035000 tx=0x7f811c0095c0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 -- 172.21.15.97:0/3728453420 shutdown_connections 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x56465734e8e0 0x564657753910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465738ff60 0x56465774d250 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465736f280 0x5646577417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 --2- 172.21.15.97:0/3728453420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x5646577412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 -- 172.21.15.97:0/3728453420 >> 172.21.15.97:0/3728453420 conn(0x564657384350 msgr2=0x564657738960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f810b400640 0 client.15156 ms_handle_reset on v2:172.21.15.115:6834/4018170367 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 -- 172.21.15.97:0/3728453420 shutdown_connections 2024-07-26T07:50:32.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 1 -- 172.21.15.97:0/3728453420 wait complete. 2024-07-26T07:50:32.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.746+0000 7f812f522740 20 client.15156 trim_cache size 0 max 16384 2024-07-26T07:50:32.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.766+0000 7f812f522740 1 Processor -- start 2024-07-26T07:50:32.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.766+0000 7f812f522740 1 -- start start 2024-07-26T07:50:32.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.766+0000 7f812f522740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x564657753240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.766+0000 7f812f522740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465773b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.766+0000 7f812f522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 0x56465774d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.766+0000 7f812f522740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x564657751c50 con 0x56465736f280 2024-07-26T07:50:32.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.766+0000 7f812f522740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564657752320 con 0x5646573670a0 2024-07-26T07:50:32.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.766+0000 7f812f522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5646577529f0 con 0x56465738ff60 2024-07-26T07:50:32.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465773b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465773b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58170/0 (socket says 172.21.15.97:58170) 2024-07-26T07:50:32.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 -- 172.21.15.97:0/2426224961 learned_addr learned my addr 172.21.15.97:0/2426224961 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:32.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f8121e00640 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x564657753240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 -- 172.21.15.97:0/2426224961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 msgr2=0x56465774d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:32.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 0x56465774d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 -- 172.21.15.97:0/2426224961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 msgr2=0x564657753240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:32.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x564657753240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 -- 172.21.15.97:0/2426224961 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564657742370 con 0x56465736f280 2024-07-26T07:50:32.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f8121e00640 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x564657753240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:32.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.767+0000 7f811be00640 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465773b8a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f810c010c20 tx=0x7f810c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72344643c349fb8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:32.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.768+0000 7f8112600640 1 -- 172.21.15.97:0/2426224961 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c004230 con 0x56465736f280 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.768+0000 7f8112600640 1 -- 172.21.15.97:0/2426224961 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f810c0043d0 con 0x56465736f280 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.768+0000 7f8112600640 1 -- 172.21.15.97:0/2426224961 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c00f930 con 0x56465736f280 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 -- 172.21.15.97:0/2426224961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 msgr2=0x56465773b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465773b8a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f810c010c20 tx=0x7f810c012820 comp rx=0 tx=0).stop 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 -- 172.21.15.97:0/2426224961 shutdown_connections 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 0x56465774d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465773b8a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 --2- 172.21.15.97:0/2426224961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x564657753240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 -- 172.21.15.97:0/2426224961 >> 172.21.15.97:0/2426224961 conn(0x564657384350 msgr2=0x56465773c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 -- 172.21.15.97:0/2426224961 shutdown_connections 2024-07-26T07:50:32.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.769+0000 7f812f522740 1 -- 172.21.15.97:0/2426224961 wait complete. 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 1 -- start start 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 1 Processor -- start 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x56465775bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465775c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 0x56465775ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x564657753790 con 0x56465736f280 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5646577530c0 con 0x5646573670a0 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564657753e60 con 0x56465738ff60 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.770+0000 7f812f522740 10 client.0 _ll_register_callbacks cb 0x7f811c003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:32.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465775c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465775c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58172/0 (socket says 172.21.15.97:58172) 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 -- 172.21.15.97:0/2233290865 learned_addr learned my addr 172.21.15.97:0/2233290865 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f8121e00640 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x56465775bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f8122800640 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 0x56465775ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 -- 172.21.15.97:0/2233290865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 msgr2=0x56465775ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 0x56465775ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 -- 172.21.15.97:0/2233290865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 msgr2=0x56465775bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x56465775bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 -- 172.21.15.97:0/2233290865 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56465775b510 con 0x56465736f280 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f811be00640 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465775c480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f810c003820 tx=0x7f810c003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6385c1a0376257 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.771+0000 7f810b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:32.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.772+0000 7f810b400640 1 -- 172.21.15.97:0/2233290865 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c02c060 con 0x56465736f280 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.772+0000 7f810b400640 1 -- 172.21.15.97:0/2233290865 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f810c003c80 con 0x56465736f280 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.772+0000 7f812f522740 1 -- 172.21.15.97:0/2233290865 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f810c0043d0 con 0x56465736f280 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.772+0000 7f810b400640 1 -- 172.21.15.97:0/2233290865 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f810c00f590 con 0x56465736f280 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.772+0000 7f812f522740 10 client.15162 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.772+0000 7f812f522740 1 -- 172.21.15.97:0/2233290865 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5646577bb700 con 0x56465736f280 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.772+0000 7f810b400640 1 -- 172.21.15.97:0/2233290865 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f810c0043d0 con 0x56465736f280 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.773+0000 7f810b400640 1 -- 172.21.15.97:0/2233290865 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f810c030030 con 0x56465736f280 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.773+0000 7f812f522740 10 client.15162 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.773+0000 7f812f522740 10 client.15162 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.773+0000 7f812f522740 20 client.15162 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:32.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.773+0000 7f812f522740 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x564657323c90 0x5646577bed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.773+0000 7f812f522740 4 client.15162 mds_command: new command op to 34535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.773+0000 7f812f522740 1 -- 172.21.15.97:0/2233290865 --> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] -- command(tid 0: {"prefix": "status"}) -- 0x5646576a9360 con 0x564657323c90 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.773+0000 7f8121e00640 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x564657323c90 0x5646577bed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.775+0000 7f8121e00640 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x564657323c90 0x5646577bed20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=712c73fd7635fc1c server_cookie=4bbee6e915594ea5 in_seq=0 out_seq=0 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.775+0000 7f810b400640 10 client.15162 ms_handle_connect on v2:172.21.15.115:6834/4018170367 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.775+0000 7f810b400640 1 -- 172.21.15.97:0/2233290865 <== mds.2 v2:172.21.15.115:6834/4018170367 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5646576a9360 con 0x564657323c90 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.775+0000 7f810b400640 10 client.15162 handle_command_reply: tid=0 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.775+0000 7f812f522740 1 client.15162 shutdown 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 -- 172.21.15.97:0/2233290865 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 msgr2=0x56465775c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465775c480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f810c003820 tx=0x7f810c003850 comp rx=0 tx=0).stop 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 -- 172.21.15.97:0/2233290865 shutdown_connections 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x564657323c90 0x5646577bed20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56465738ff60 0x56465775ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56465736f280 0x56465775c480 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 --2- 172.21.15.97:0/2233290865 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646573670a0 0x56465775bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f810b400640 0 client.15162 ms_handle_reset on v2:172.21.15.115:6834/4018170367 2024-07-26T07:50:32.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 -- 172.21.15.97:0/2233290865 >> 172.21.15.97:0/2233290865 conn(0x564657384350 msgr2=0x56465773f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:32.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 -- 172.21.15.97:0/2233290865 shutdown_connections 2024-07-26T07:50:32.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 1 -- 172.21.15.97:0/2233290865 wait complete. 2024-07-26T07:50:32.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.776+0000 7f812f522740 20 client.15162 trim_cache size 0 max 16384 2024-07-26T07:50:32.782 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:50:32.782 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:50:32.782 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 2, 2024-07-26T07:50:32.782 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34535, 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 148.01816231800001, 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 212.02054831300001 2024-07-26T07:50:32.783 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:50:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 -- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81040b3c80 msgr2=0x7f81040b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81040b3c80 0x7f81040b6190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f811c005cc0 tx=0x7f811c005c30 comp rx=0 tx=0).stop 2024-07-26T07:50:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 -- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f81240751e0 msgr2=0x7f812419be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f81240751e0 0x7f812419be00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f811400c920 tx=0x7f811400cde0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 -- 172.21.15.97:0/706449370 shutdown_connections 2024-07-26T07:50:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f81040b3c80 0x7f81040b6190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f812410f920 0x7f81241a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8124075b30 0x7f812419c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 --2- 172.21.15.97:0/706449370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f81240751e0 0x7f812419be00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.777+0000 7f8123200640 1 -- 172.21.15.97:0/706449370 >> 172.21.15.97:0/706449370 conn(0x7f81240fee10 msgr2=0x7f8124109430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.778+0000 7f8123200640 1 -- 172.21.15.97:0/706449370 shutdown_connections 2024-07-26T07:50:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:32.778+0000 7f8123200640 1 -- 172.21.15.97:0/706449370 wait complete. 2024-07-26T07:50:33.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.h status 2024-07-26T07:50:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:33 smithi181 ceph-mon[35869]: pgmap v473: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:34.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:33 smithi097 ceph-mon[32231]: pgmap v473: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:34.026 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:33 smithi115 ceph-mon[35364]: pgmap v473: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:35.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.950+0000 7fb030800640 1 -- 172.21.15.97:0/1679021740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02810fe70 msgr2=0x7fb0281166a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:35.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.950+0000 7fb030800640 1 --2- 172.21.15.97:0/1679021740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02810fe70 0x7fb0281166a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb024009ca0 tx=0x7fb02402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.950+0000 7fb030800640 1 -- 172.21.15.97:0/1679021740 shutdown_connections 2024-07-26T07:50:35.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.950+0000 7fb030800640 1 --2- 172.21.15.97:0/1679021740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02810fe70 0x7fb0281166a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.950+0000 7fb030800640 1 --2- 172.21.15.97:0/1679021740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb02810f520 0x7fb02810f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.950+0000 7fb030800640 1 --2- 172.21.15.97:0/1679021740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb028109be0 0x7fb028109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.950+0000 7fb030800640 1 -- 172.21.15.97:0/1679021740 >> 172.21.15.97:0/1679021740 conn(0x7fb028077b10 msgr2=0x7fb028107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.950+0000 7fb030800640 1 -- 172.21.15.97:0/1679021740 shutdown_connections 2024-07-26T07:50:35.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.951+0000 7fb030800640 1 -- 172.21.15.97:0/1679021740 wait complete. 2024-07-26T07:50:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb030800640 1 Processor -- start 2024-07-26T07:50:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb030800640 1 -- start start 2024-07-26T07:50:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb030800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb028109be0 0x7fb0281a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb030800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb02810f520 0x7fb0281a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb030800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb02810fe70 0x7fb0281a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb030800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb028119a30 con 0x7fb028109be0 2024-07-26T07:50:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb030800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb0281198b0 con 0x7fb02810f520 2024-07-26T07:50:35.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb030800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb028119bb0 con 0x7fb02810fe70 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.952+0000 7fb02de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb028109be0 0x7fb0281a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb028109be0 0x7fb0281a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53478/0 (socket says 172.21.15.97:53478) 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02de00640 1 -- 172.21.15.97:0/2955950580 learned_addr learned my addr 172.21.15.97:0/2955950580 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02d400640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb02810f520 0x7fb0281a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02e800640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb02810fe70 0x7fb0281a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02de00640 1 -- 172.21.15.97:0/2955950580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb02810fe70 msgr2=0x7fb0281a5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02de00640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb02810fe70 0x7fb0281a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02de00640 1 -- 172.21.15.97:0/2955950580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb02810f520 msgr2=0x7fb0281a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02de00640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb02810f520 0x7fb0281a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02de00640 1 -- 172.21.15.97:0/2955950580 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0281a62b0 con 0x7fb028109be0 2024-07-26T07:50:35.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02d400640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb02810f520 0x7fb0281a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.953+0000 7fb02de00640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb028109be0 0x7fb0281a0400 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00c970 tx=0x7fb01c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.954+0000 7fb022a00640 1 -- 172.21.15.97:0/2955950580 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c007bb0 con 0x7fb028109be0 2024-07-26T07:50:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.954+0000 7fb022a00640 1 -- 172.21.15.97:0/2955950580 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb01c007d50 con 0x7fb028109be0 2024-07-26T07:50:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.954+0000 7fb030800640 1 -- 172.21.15.97:0/2955950580 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0281a65a0 con 0x7fb028109be0 2024-07-26T07:50:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.954+0000 7fb022a00640 1 -- 172.21.15.97:0/2955950580 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c004e40 con 0x7fb028109be0 2024-07-26T07:50:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.954+0000 7fb030800640 1 -- 172.21.15.97:0/2955950580 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0281acdc0 con 0x7fb028109be0 2024-07-26T07:50:35.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.957+0000 7fb0373c3740 1 Processor -- start 2024-07-26T07:50:35.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb0373c3740 1 -- start start 2024-07-26T07:50:35.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb022a00640 1 -- 172.21.15.97:0/2955950580 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fb01c005020 con 0x7fb028109be0 2024-07-26T07:50:35.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9c0ef60 0x561ea9fb4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fb4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9bee280 0x561ea9fb4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb0373c3740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561ea9f2a970 con 0x561ea9c0ef60 2024-07-26T07:50:35.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb0373c3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561ea9f19e70 con 0x561ea9bee280 2024-07-26T07:50:35.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb0373c3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561ea9f1deb0 con 0x561ea9be60a0 2024-07-26T07:50:35.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb02e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9c0ef60 0x561ea9fb4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb02e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9c0ef60 0x561ea9fb4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53482/0 (socket says 172.21.15.97:53482) 2024-07-26T07:50:35.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.958+0000 7fb02e800640 1 -- 172.21.15.97:0/4019123605 learned_addr learned my addr 172.21.15.97:0/4019123605 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:35.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.959+0000 7fb02de00640 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fb4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.959+0000 7fb02d400640 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9bee280 0x561ea9fb4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.959+0000 7fb02e800640 1 -- 172.21.15.97:0/4019123605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 msgr2=0x561ea9fb4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:35.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.959+0000 7fb02e800640 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fb4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.959+0000 7fb02e800640 1 -- 172.21.15.97:0/4019123605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9bee280 msgr2=0x561ea9fb4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.959+0000 7fb02e800640 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9bee280 0x561ea9fb4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.959+0000 7fb02e800640 1 -- 172.21.15.97:0/4019123605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea9ec1110 con 0x561ea9c0ef60 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.959+0000 7fb02e800640 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9c0ef60 0x561ea9fb4570 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb024006010 tx=0x7fb024031570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c615024e026753c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.960+0000 7fb022a00640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb0100b3c00 0x7fb0100b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.960+0000 7fb017000640 1 -- 172.21.15.97:0/4019123605 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb024031b80 con 0x561ea9c0ef60 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.960+0000 7fb017000640 1 -- 172.21.15.97:0/4019123605 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb024036070 con 0x561ea9c0ef60 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.960+0000 7fb017000640 1 -- 172.21.15.97:0/4019123605 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0240044a0 con 0x561ea9c0ef60 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.960+0000 7fb022a00640 1 -- 172.21.15.97:0/2955950580 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fb01c0e3a70 con 0x7fb028109be0 2024-07-26T07:50:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.960+0000 7fb02de00640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb0100b3c00 0x7fb0100b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 -- 172.21.15.97:0/4019123605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9c0ef60 msgr2=0x561ea9fb4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb02de00640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb0100b3c00 0x7fb0100b6110 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0a6980 tx=0x7fb01c0e9240 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9c0ef60 0x561ea9fb4570 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb024006010 tx=0x7fb024031570 comp rx=0 tx=0).stop 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 -- 172.21.15.97:0/4019123605 shutdown_connections 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9bee280 0x561ea9fb4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fb4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 --2- 172.21.15.97:0/4019123605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9c0ef60 0x561ea9fb4570 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 -- 172.21.15.97:0/4019123605 >> 172.21.15.97:0/4019123605 conn(0x561ea9c03350 msgr2=0x561ea9fb2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 -- 172.21.15.97:0/4019123605 shutdown_connections 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.961+0000 7fb0373c3740 1 -- 172.21.15.97:0/4019123605 wait complete. 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.962+0000 7fb0373c3740 1 -- start start 2024-07-26T07:50:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.962+0000 7fb0373c3740 1 Processor -- start 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9be60a0 0x561ea9fc0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fc0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9c0ef60 0x561ea9fcc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb0373c3740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561ea9d7d5f0 con 0x561ea9bee280 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb0373c3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561ea9eb6540 con 0x561ea9be60a0 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb0373c3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561ea9fc2cf0 con 0x561ea9c0ef60 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb0373c3740 10 client.0 _ll_register_callbacks cb 0x561ea9ec5f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb02d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fc0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb02d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fc0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53494/0 (socket says 172.21.15.97:53494) 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb02d400640 1 -- 172.21.15.97:0/267399873 learned_addr learned my addr 172.21.15.97:0/267399873 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:35.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb02e800640 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9be60a0 0x561ea9fc0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.963+0000 7fb02e800640 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9c0ef60 0x561ea9fcc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb02d400640 1 -- 172.21.15.97:0/267399873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9c0ef60 msgr2=0x561ea9fcc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb02d400640 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9c0ef60 0x561ea9fcc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb02d400640 1 -- 172.21.15.97:0/267399873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9be60a0 msgr2=0x561ea9fc0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb02d400640 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9be60a0 0x561ea9fc0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb02d400640 1 -- 172.21.15.97:0/267399873 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea9c7a540 con 0x561ea9bee280 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb02e800640 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9c0ef60 0x561ea9fcc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb02d400640 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fc0590 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb018010810 tx=0x7fb018010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805cb4ef0b64e94c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:35.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb00b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb00b400640 1 -- 172.21.15.97:0/267399873 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01800ed40 con 0x561ea9bee280 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb00b400640 1 -- 172.21.15.97:0/267399873 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fb01801c480 con 0x561ea9bee280 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.964+0000 7fb0373c3740 1 -- 172.21.15.97:0/267399873 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb024036070 con 0x561ea9bee280 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb00b400640 1 -- 172.21.15.97:0/267399873 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb01801c680 con 0x561ea9bee280 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb0373c3740 10 client.15174 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb0373c3740 1 -- 172.21.15.97:0/267399873 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561ea9fcec80 con 0x561ea9bee280 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb00b400640 1 -- 172.21.15.97:0/267399873 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb024036070 con 0x561ea9bee280 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb00b400640 1 -- 172.21.15.97:0/267399873 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb018025050 con 0x561ea9bee280 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb0373c3740 10 client.15174 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb0373c3740 10 client.15174 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb0373c3740 20 client.15174 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb0373c3740 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x561ea9bcd8e0 0x561ea9fd2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.965+0000 7fb0373c3740 4 client.15174 mds_command: new command op to 34556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:50:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.966+0000 7fb0373c3740 1 -- 172.21.15.97:0/267399873 --> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ea9f2c8c0 con 0x561ea9bcd8e0 2024-07-26T07:50:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.966+0000 7fb02e800640 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x561ea9bcd8e0 0x561ea9fd2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.967+0000 7fb02e800640 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x561ea9bcd8e0 0x561ea9fd2710 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=475e1522c6041346 server_cookie=ed97466796c51643 in_seq=0 out_seq=0 2024-07-26T07:50:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.967+0000 7fb00b400640 10 client.15174 ms_handle_connect on v2:172.21.15.115:6836/3478625369 2024-07-26T07:50:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.969+0000 7fb00b400640 1 -- 172.21.15.97:0/267399873 <== mds.1 v2:172.21.15.115:6836/3478625369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ea9f2c8c0 con 0x561ea9bcd8e0 2024-07-26T07:50:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.969+0000 7fb00b400640 10 client.15174 handle_command_reply: tid=0 2024-07-26T07:50:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.969+0000 7fb0373c3740 1 client.15174 shutdown 2024-07-26T07:50:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 -- 172.21.15.97:0/267399873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 msgr2=0x561ea9fc0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fc0590 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb018010810 tx=0x7fb018010cd0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 -- 172.21.15.97:0/267399873 shutdown_connections 2024-07-26T07:50:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x561ea9bcd8e0 0x561ea9fd2710 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9c0ef60 0x561ea9fcc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fc0590 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 --2- 172.21.15.97:0/267399873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9be60a0 0x561ea9fc0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 -- 172.21.15.97:0/267399873 >> 172.21.15.97:0/267399873 conn(0x561ea9c03350 msgr2=0x561ea9fb7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb00b400640 0 client.15174 ms_handle_reset on v2:172.21.15.115:6836/3478625369 2024-07-26T07:50:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 -- 172.21.15.97:0/267399873 shutdown_connections 2024-07-26T07:50:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 1 -- 172.21.15.97:0/267399873 wait complete. 2024-07-26T07:50:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.970+0000 7fb0373c3740 20 client.15174 trim_cache size 0 max 16384 2024-07-26T07:50:35.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.990+0000 7fb0373c3740 1 Processor -- start 2024-07-26T07:50:35.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.990+0000 7fb0373c3740 1 -- start start 2024-07-26T07:50:35.991 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:35 smithi097 ceph-mon[32231]: pgmap v474: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T07:50:35.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.990+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fba690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.990+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9facd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.990+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fcbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.990+0000 7fb0373c3740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561ea9fd0a50 con 0x561ea9bee280 2024-07-26T07:50:35.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.990+0000 7fb0373c3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561ea9fd1120 con 0x561ea9c0ef60 2024-07-26T07:50:35.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.990+0000 7fb0373c3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561ea9fd17f0 con 0x561ea9be60a0 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9facd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9facd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53506/0 (socket says 172.21.15.97:53506) 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02d400640 1 -- 172.21.15.97:0/2058875719 learned_addr learned my addr 172.21.15.97:0/2058875719 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02e800640 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fba690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02e800640 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fcbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02d400640 1 -- 172.21.15.97:0/2058875719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 msgr2=0x561ea9fba690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02d400640 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fba690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02d400640 1 -- 172.21.15.97:0/2058875719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 msgr2=0x561ea9fcbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02d400640 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fcbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02d400640 1 -- 172.21.15.97:0/2058875719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea9fb73d0 con 0x561ea9bee280 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.991+0000 7fb02e800640 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fcbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:50:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.992+0000 7fb02d400640 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9facd80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb01800c5b0 tx=0x7fb01800c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4e05c787831ecd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:35.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.992+0000 7fb016600640 1 -- 172.21.15.97:0/2058875719 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb018011ac0 con 0x561ea9bee280 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.992+0000 7fb016600640 1 -- 172.21.15.97:0/2058875719 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb01802b070 con 0x561ea9bee280 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.992+0000 7fb016600640 1 -- 172.21.15.97:0/2058875719 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01801c820 con 0x561ea9bee280 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 -- 172.21.15.97:0/2058875719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 msgr2=0x561ea9facd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9facd80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb01800c5b0 tx=0x7fb01800c5e0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 -- 172.21.15.97:0/2058875719 shutdown_connections 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fcbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9facd80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 --2- 172.21.15.97:0/2058875719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fba690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 -- 172.21.15.97:0/2058875719 >> 172.21.15.97:0/2058875719 conn(0x561ea9c03350 msgr2=0x561ea9fbe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 -- 172.21.15.97:0/2058875719 shutdown_connections 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.993+0000 7fb0373c3740 1 -- 172.21.15.97:0/2058875719 wait complete. 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.994+0000 7fb0373c3740 1 -- start start 2024-07-26T07:50:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.994+0000 7fb0373c3740 1 Processor -- start 2024-07-26T07:50:35.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.994+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fd3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fd4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb0373c3740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fd9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb0373c3740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561ea9fd2590 con 0x561ea9bee280 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb0373c3740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561ea9fd1ec0 con 0x561ea9c0ef60 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb0373c3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561ea9fd2c60 con 0x561ea9be60a0 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb0373c3740 10 client.0 _ll_register_callbacks cb 0x7fb0180103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fd4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fd4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53508/0 (socket says 172.21.15.97:53508) 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02d400640 1 -- 172.21.15.97:0/3608730014 learned_addr learned my addr 172.21.15.97:0/3608730014 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02e800640 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fd3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02e800640 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fd9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:35.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02d400640 1 -- 172.21.15.97:0/3608730014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 msgr2=0x561ea9fd3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:35.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02d400640 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fd3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02d400640 1 -- 172.21.15.97:0/3608730014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 msgr2=0x561ea9fd9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:35.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02d400640 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fd9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:35.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.995+0000 7fb02d400640 1 -- 172.21.15.97:0/3608730014 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea9fd99e0 con 0x561ea9bee280 2024-07-26T07:50:35.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.996+0000 7fb02e800640 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fd3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:35.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.996+0000 7fb02d400640 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fd4500 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb0180118c0 tx=0x7fb01800c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6534e1b7510025f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:35.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.996+0000 7fb00b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:35.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.996+0000 7fb00b400640 1 -- 172.21.15.97:0/3608730014 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01800eb40 con 0x561ea9bee280 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.996+0000 7fb00b400640 1 -- 172.21.15.97:0/3608730014 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fb01800ece0 con 0x561ea9bee280 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.996+0000 7fb0373c3740 1 -- 172.21.15.97:0/3608730014 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb01802b070 con 0x561ea9bee280 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb00b400640 1 -- 172.21.15.97:0/3608730014 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0180050c0 con 0x561ea9bee280 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb0373c3740 10 client.15180 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb0373c3740 1 -- 172.21.15.97:0/3608730014 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561ea9fdad10 con 0x561ea9bee280 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb00b400640 1 -- 172.21.15.97:0/3608730014 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb01802b070 con 0x561ea9bee280 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb00b400640 1 -- 172.21.15.97:0/3608730014 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb018042020 con 0x561ea9bee280 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb0373c3740 10 client.15180 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb0373c3740 10 client.15180 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb0373c3740 20 client.15180 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb0373c3740 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x561ea9ba2c90 0x561eaa03e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb0373c3740 4 client.15180 mds_command: new command op to 34556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:50:36.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.997+0000 7fb0373c3740 1 -- 172.21.15.97:0/3608730014 --> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] -- command(tid 0: {"prefix": "status"}) -- 0x561ea9fad3e0 con 0x561ea9ba2c90 2024-07-26T07:50:36.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.998+0000 7fb02e800640 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x561ea9ba2c90 0x561eaa03e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:36.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.999+0000 7fb02e800640 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x561ea9ba2c90 0x561eaa03e570 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef2e181439a8f941 server_cookie=d6179c2bf7c0919 in_seq=0 out_seq=0 2024-07-26T07:50:36.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:35.999+0000 7fb00b400640 10 client.15180 ms_handle_connect on v2:172.21.15.115:6836/3478625369 2024-07-26T07:50:36.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb00b400640 1 -- 172.21.15.97:0/3608730014 <== mds.1 v2:172.21.15.115:6836/3478625369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561ea9fad3e0 con 0x561ea9ba2c90 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb00b400640 10 client.15180 handle_command_reply: tid=0 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 client.15180 shutdown 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 -- 172.21.15.97:0/3608730014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 msgr2=0x561ea9fd4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fd4500 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb0180118c0 tx=0x7fb01800c540 comp rx=0 tx=0).stop 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 -- 172.21.15.97:0/3608730014 shutdown_connections 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x561ea9ba2c90 0x561eaa03e570 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561ea9c0ef60 0x561ea9fd9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561ea9bee280 0x561ea9fd4500 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 --2- 172.21.15.97:0/3608730014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea9be60a0 0x561ea9fd3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 -- 172.21.15.97:0/3608730014 >> 172.21.15.97:0/3608730014 conn(0x561ea9c03350 msgr2=0x561ea9fbe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb00b400640 0 client.15180 ms_handle_reset on v2:172.21.15.115:6836/3478625369 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 -- 172.21.15.97:0/3608730014 shutdown_connections 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.000+0000 7fb0373c3740 1 -- 172.21.15.97:0/3608730014 wait complete. 2024-07-26T07:50:36.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb0373c3740 20 client.15180 trim_cache size 0 max 16384 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 1, 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34556, 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 152.256800949, 2024-07-26T07:50:36.003 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:50:36.004 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:50:36.004 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:50:36.004 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 204.18406999600001 2024-07-26T07:50:36.004 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:50:36.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 -- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb0100b3c00 msgr2=0x7fb0100b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:36.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb0100b3c00 0x7fb0100b6110 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0a6980 tx=0x7fb01c0e9240 comp rx=0 tx=0).stop 2024-07-26T07:50:36.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 -- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb028109be0 msgr2=0x7fb0281a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:36.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb028109be0 0x7fb0281a0400 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00c970 tx=0x7fb01c00ce30 comp rx=0 tx=0).stop 2024-07-26T07:50:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 -- 172.21.15.97:0/2955950580 shutdown_connections 2024-07-26T07:50:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb0100b3c00 0x7fb0100b6110 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb02810fe70 0x7fb0281a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb02810f520 0x7fb0281a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 --2- 172.21.15.97:0/2955950580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb028109be0 0x7fb0281a0400 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.001+0000 7fb030800640 1 -- 172.21.15.97:0/2955950580 >> 172.21.15.97:0/2955950580 conn(0x7fb028077b10 msgr2=0x7fb028107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.002+0000 7fb030800640 1 -- 172.21.15.97:0/2955950580 shutdown_connections 2024-07-26T07:50:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:36.002+0000 7fb030800640 1 -- 172.21.15.97:0/2955950580 wait complete. 2024-07-26T07:50:36.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:35 smithi115 ceph-mon[35364]: pgmap v474: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T07:50:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:35 smithi181 ceph-mon[35869]: pgmap v474: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T07:50:36.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.b status 2024-07-26T07:50:37.244 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:37 smithi097 ceph-mon[32231]: pgmap v475: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:50:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:37 smithi115 ceph-mon[35364]: pgmap v475: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:50:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:37 smithi181 ceph-mon[35869]: pgmap v475: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:50:39.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 -- 172.21.15.97:0/4020618012 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410f330 msgr2=0x7fbfa410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 --2- 172.21.15.97:0/4020618012 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410f330 0x7fbfa410f740 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009900 tx=0x7fbf9002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 -- 172.21.15.97:0/4020618012 shutdown_connections 2024-07-26T07:50:39.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 --2- 172.21.15.97:0/4020618012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfa410fc80 0x7fbfa41164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 --2- 172.21.15.97:0/4020618012 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410f330 0x7fbfa410f740 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 --2- 172.21.15.97:0/4020618012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa41099f0 0x7fbfa4109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 -- 172.21.15.97:0/4020618012 >> 172.21.15.97:0/4020618012 conn(0x7fbfa4078ff0 msgr2=0x7fbfa4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:39.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 -- 172.21.15.97:0/4020618012 shutdown_connections 2024-07-26T07:50:39.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.407+0000 7fbfaac00640 1 -- 172.21.15.97:0/4020618012 wait complete. 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.408+0000 7fbfaac00640 1 Processor -- start 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.408+0000 7fbfaac00640 1 -- start start 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaac00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfa41099f0 0x7fbfa41a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa410f330 0x7fbfa41a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaac00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410fc80 0x7fbfa41a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaac00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41197f0 con 0x7fbfa410fc80 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaac00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa4119670 con 0x7fbfa41099f0 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa4119970 con 0x7fbfa410f330 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaa200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410fc80 0x7fbfa41a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaa200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410fc80 0x7fbfa41a5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53526/0 (socket says 172.21.15.97:53526) 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfaa200640 1 -- 172.21.15.97:0/624335385 learned_addr learned my addr 172.21.15.97:0/624335385 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.409+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa410f330 0x7fbfa41a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfa9800640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfa41099f0 0x7fbfa41a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfaa200640 1 -- 172.21.15.97:0/624335385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa410f330 msgr2=0x7fbfa41a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfaa200640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa410f330 0x7fbfa41a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfaa200640 1 -- 172.21.15.97:0/624335385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfa41099f0 msgr2=0x7fbfa41a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfaa200640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfa41099f0 0x7fbfa41a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfaa200640 1 -- 172.21.15.97:0/624335385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa41a6130 con 0x7fbfa410fc80 2024-07-26T07:50:39.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfaa200640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410fc80 0x7fbfa41a5ac0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00eaa0 tx=0x7fbf9c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:39.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfa2000640 1 -- 172.21.15.97:0/624335385 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c00cdb0 con 0x7fbfa410fc80 2024-07-26T07:50:39.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfa2000640 1 -- 172.21.15.97:0/624335385 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf9c005020 con 0x7fbfa410fc80 2024-07-26T07:50:39.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.410+0000 7fbfaac00640 1 -- 172.21.15.97:0/624335385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa41a6420 con 0x7fbfa410fc80 2024-07-26T07:50:39.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.411+0000 7fbfa2000640 1 -- 172.21.15.97:0/624335385 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0116d0 con 0x7fbfa410fc80 2024-07-26T07:50:39.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.411+0000 7fbfaac00640 1 -- 172.21.15.97:0/624335385 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa41a68e0 con 0x7fbfa410fc80 2024-07-26T07:50:39.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.414+0000 7fbfb2e01740 1 Processor -- start 2024-07-26T07:50:39.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.415+0000 7fbfa2000640 1 -- 172.21.15.97:0/624335385 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fbf9c0051d0 con 0x7fbfa410fc80 2024-07-26T07:50:39.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.415+0000 7fbfb2e01740 1 -- start start 2024-07-26T07:50:39.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.415+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 0x5587dc7d4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.415+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7d4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.415+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7d5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.415+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5587dc74a970 con 0x5587dc40e280 2024-07-26T07:50:39.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.415+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5587dc739e70 con 0x5587dc4060a0 2024-07-26T07:50:39.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.415+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5587dc73deb0 con 0x5587dc42ef60 2024-07-26T07:50:39.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7d5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7d5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53530/0 (socket says 172.21.15.97:53530) 2024-07-26T07:50:39.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 -- 172.21.15.97:0/3320840927 learned_addr learned my addr 172.21.15.97:0/3320840927 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:39.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 0x5587dc7d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa9800640 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7d4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 -- 172.21.15.97:0/3320840927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 msgr2=0x5587dc7d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 0x5587dc7d4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 -- 172.21.15.97:0/3320840927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 msgr2=0x5587dc7d4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7d4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.416+0000 7fbfa8e00640 1 -- 172.21.15.97:0/3320840927 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587dc6e1110 con 0x5587dc40e280 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.417+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7d5200 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbf90005f70 tx=0x7fbf90005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb302fc4c4626b25 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.417+0000 7fbfa2000640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbf8c0b3c00 0x7fbf8c0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.417+0000 7fbfa2000640 1 -- 172.21.15.97:0/624335385 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fbf9c0e3ea0 con 0x7fbfa410fc80 2024-07-26T07:50:39.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.417+0000 7fbfa9800640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbf8c0b3c00 0x7fbf8c0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.417+0000 7fbf96a00640 1 -- 172.21.15.97:0/3320840927 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf90043020 con 0x5587dc40e280 2024-07-26T07:50:39.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.417+0000 7fbf96a00640 1 -- 172.21.15.97:0/3320840927 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf900044c0 con 0x5587dc40e280 2024-07-26T07:50:39.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.417+0000 7fbf96a00640 1 -- 172.21.15.97:0/3320840927 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf90035040 con 0x5587dc40e280 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 -- 172.21.15.97:0/3320840927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 msgr2=0x5587dc7d5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7d5200 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbf90005f70 tx=0x7fbf90005a40 comp rx=0 tx=0).stop 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfa9800640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbf8c0b3c00 0x7fbf8c0b6110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbf980058c0 tx=0x7fbf98005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 -- 172.21.15.97:0/3320840927 shutdown_connections 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7d5200 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/3320840927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 0x5587dc7d4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 -- 172.21.15.97:0/3320840927 >> 172.21.15.97:0/3320840927 conn(0x5587dc423350 msgr2=0x5587dc7d2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 -- 172.21.15.97:0/3320840927 shutdown_connections 2024-07-26T07:50:39.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.418+0000 7fbfb2e01740 1 -- 172.21.15.97:0/3320840927 wait complete. 2024-07-26T07:50:39.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.419+0000 7fbfb2e01740 1 -- start start 2024-07-26T07:50:39.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.419+0000 7fbfb2e01740 1 Processor -- start 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc4060a0 0x5587dc7e0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc40e280 0x5587dc7e0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7ec1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5587dc59d5f0 con 0x5587dc42ef60 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5587dc6d6540 con 0x5587dc40e280 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5587dc7e2f00 con 0x5587dc4060a0 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfb2e01740 10 client.0 _ll_register_callbacks cb 0x5587dc6e5f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfaa200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7ec1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfaa200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7ec1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53540/0 (socket says 172.21.15.97:53540) 2024-07-26T07:50:39.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfaa200640 1 -- 172.21.15.97:0/2251066653 learned_addr learned my addr 172.21.15.97:0/2251066653 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfa9800640 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc4060a0 0x5587dc7e0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc40e280 0x5587dc7e0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfaa200640 1 -- 172.21.15.97:0/2251066653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc4060a0 msgr2=0x5587dc7e0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfaa200640 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc4060a0 0x5587dc7e0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfaa200640 1 -- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc40e280 msgr2=0x5587dc7e0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.420+0000 7fbfaa200640 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc40e280 0x5587dc7e0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbfaa200640 1 -- 172.21.15.97:0/2251066653 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587dc49a540 con 0x5587dc42ef60 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc40e280 0x5587dc7e0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbfaa200640 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7ec1f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c0a8000 tx=0x7fbf9c0ab5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d744176c11c9a54 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbf8b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbf8b400640 1 -- 172.21.15.97:0/2251066653 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c01f040 con 0x5587dc42ef60 2024-07-26T07:50:39.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbf8b400640 1 -- 172.21.15.97:0/2251066653 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fbf9c0a7380 con 0x5587dc42ef60 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2251066653 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5587dc74c8c0 con 0x5587dc42ef60 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbf8b400640 1 -- 172.21.15.97:0/2251066653 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf9c0b3e40 con 0x5587dc42ef60 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbfb2e01740 10 client.15192 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.421+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2251066653 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5587dc7eee80 con 0x5587dc42ef60 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.422+0000 7fbf8b400640 1 -- 172.21.15.97:0/2251066653 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf9c06c080 con 0x5587dc42ef60 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.422+0000 7fbf8b400640 1 -- 172.21.15.97:0/2251066653 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbf9c00f040 con 0x5587dc42ef60 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.422+0000 7fbfb2e01740 10 client.15192 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.422+0000 7fbfb2e01740 10 client.15192 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.422+0000 7fbfb2e01740 20 client.15192 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.422+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5587dc3ed8e0 0x5587dc7f2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.422+0000 7fbfb2e01740 4 client.15192 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:50:39.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.422+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2251066653 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587dc748360 con 0x5587dc3ed8e0 2024-07-26T07:50:39.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.423+0000 7fbfa9800640 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5587dc3ed8e0 0x5587dc7f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.424+0000 7fbfa9800640 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5587dc3ed8e0 0x5587dc7f2920 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62e7d93dbb30f1f9 server_cookie=d97be64944ab9d52 in_seq=0 out_seq=0 2024-07-26T07:50:39.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.424+0000 7fbf8b400640 10 client.15192 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:50:39.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbf8b400640 1 -- 172.21.15.97:0/2251066653 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5587dc748360 con 0x5587dc3ed8e0 2024-07-26T07:50:39.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbf8b400640 10 client.15192 handle_command_reply: tid=0 2024-07-26T07:50:39.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 client.15192 shutdown 2024-07-26T07:50:39.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2251066653 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 msgr2=0x5587dc7ec1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7ec1f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c0a8000 tx=0x7fbf9c0ab5d0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2251066653 shutdown_connections 2024-07-26T07:50:39.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5587dc3ed8e0 0x5587dc7f2920 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7ec1f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc40e280 0x5587dc7e0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2251066653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc4060a0 0x5587dc7e0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbf8b400640 0 client.15192 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:50:39.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.426+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2251066653 >> 172.21.15.97:0/2251066653 conn(0x5587dc423350 msgr2=0x5587dc7d7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:39.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.427+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2251066653 shutdown_connections 2024-07-26T07:50:39.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.427+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2251066653 wait complete. 2024-07-26T07:50:39.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.427+0000 7fbfb2e01740 20 client.15192 trim_cache size 0 max 16384 2024-07-26T07:50:39.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.455+0000 7fbfb2e01740 1 Processor -- start 2024-07-26T07:50:39.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.455+0000 7fbfb2e01740 1 -- start start 2024-07-26T07:50:39.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.455+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7df540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.455+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7ec160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.455+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 0x5587dc7ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.455+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5587dc7f0c60 con 0x5587dc40e280 2024-07-26T07:50:39.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.455+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5587dc7f1330 con 0x5587dc4060a0 2024-07-26T07:50:39.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.455+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5587dc7f1a00 con 0x5587dc42ef60 2024-07-26T07:50:39.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa9800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7df540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7ec160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53552/0 (socket says 172.21.15.97:53552) 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa8e00640 1 -- 172.21.15.97:0/2386604703 learned_addr learned my addr 172.21.15.97:0/2386604703 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfaa200640 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 0x5587dc7ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa9800640 1 -- 172.21.15.97:0/2386604703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 msgr2=0x5587dc7ec6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa9800640 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 0x5587dc7ec6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa9800640 1 -- 172.21.15.97:0/2386604703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 msgr2=0x5587dc7ec160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa9800640 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7ec160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa9800640 1 -- 172.21.15.97:0/2386604703 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587dc7db8e0 con 0x5587dc4060a0 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.456+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7ec160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.457+0000 7fbfa9800640 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7df540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x5587dc7f32b0 tx=0x7fbf98013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62118df7d4ba5ce1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.457+0000 7fbf96000640 1 -- 172.21.15.97:0/2386604703 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf980043d0 con 0x5587dc4060a0 2024-07-26T07:50:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.457+0000 7fbf96000640 1 -- 172.21.15.97:0/2386604703 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf98011950 con 0x5587dc4060a0 2024-07-26T07:50:39.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.457+0000 7fbf96000640 1 -- 172.21.15.97:0/2386604703 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf98028790 con 0x5587dc4060a0 2024-07-26T07:50:39.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.458+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2386604703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 msgr2=0x5587dc7df540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.458+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7df540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x5587dc7f32b0 tx=0x7fbf98013000 comp rx=0 tx=0).stop 2024-07-26T07:50:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.458+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2386604703 shutdown_connections 2024-07-26T07:50:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.459+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc42ef60 0x5587dc7ec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.459+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc40e280 0x5587dc7ec160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.459+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/2386604703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7df540 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.459+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2386604703 >> 172.21.15.97:0/2386604703 conn(0x5587dc423350 msgr2=0x5587dc7de8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.459+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2386604703 shutdown_connections 2024-07-26T07:50:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.459+0000 7fbfb2e01740 1 -- 172.21.15.97:0/2386604703 wait complete. 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.459+0000 7fbfb2e01740 1 -- start start 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.459+0000 7fbfb2e01740 1 Processor -- start 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc40e280 0x5587dc7f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfb2e01740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7f94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5587dc7f27a0 con 0x5587dc42ef60 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5587dc7f20d0 con 0x5587dc4060a0 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfb2e01740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5587dc7f2e70 con 0x5587dc40e280 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfb2e01740 10 client.0 _ll_register_callbacks cb 0x7fbf9c0aa8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:50:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfaa200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7f94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfaa200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7f94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53564/0 (socket says 172.21.15.97:53564) 2024-07-26T07:50:39.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.460+0000 7fbfaa200640 1 -- 172.21.15.97:0/586819939 learned_addr learned my addr 172.21.15.97:0/586819939 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfa9800640 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfa8e00640 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc40e280 0x5587dc7f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfaa200640 1 -- 172.21.15.97:0/586819939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc40e280 msgr2=0x5587dc7f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfaa200640 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc40e280 0x5587dc7f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfaa200640 1 -- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 msgr2=0x5587dc7f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfaa200640 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7f3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfaa200640 1 -- 172.21.15.97:0/586819939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587dc7f9b30 con 0x5587dc42ef60 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfa9800640 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbfaa200640 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7f94c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c0e41c0 tx=0x7fbf9c0b0890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ff664ff52f41ea server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbf8b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.461+0000 7fbf8b400640 1 -- 172.21.15.97:0/586819939 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c01f040 con 0x5587dc42ef60 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbf8b400640 1 -- 172.21.15.97:0/586819939 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fbf9c06d020 con 0x5587dc42ef60 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbfb2e01740 1 -- 172.21.15.97:0/586819939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf98011950 con 0x5587dc42ef60 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbf8b400640 1 -- 172.21.15.97:0/586819939 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf9c0aab60 con 0x5587dc42ef60 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbfb2e01740 10 client.15198 fetch_fsmap learned FSMap version 27 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbfb2e01740 1 -- 172.21.15.97:0/586819939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5587dc7faef0 con 0x5587dc42ef60 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbf8b400640 1 -- 172.21.15.97:0/586819939 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf98011950 con 0x5587dc42ef60 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbf8b400640 1 -- 172.21.15.97:0/586819939 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbf9c08d050 con 0x5587dc42ef60 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbfb2e01740 10 client.15198 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:50:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbfb2e01740 10 client.15198 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T07:50:39.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.462+0000 7fbfb2e01740 20 client.15198 populate_metadata read hostname 'smithi097' 2024-07-26T07:50:39.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.463+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5587dc3c2c90 0x5587dc85e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:39.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.463+0000 7fbfb2e01740 4 client.15198 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:50:39.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.463+0000 7fbfb2e01740 1 -- 172.21.15.97:0/586819939 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "status"}) -- 0x5587dc745aa0 con 0x5587dc3c2c90 2024-07-26T07:50:39.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.463+0000 7fbfa9800640 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5587dc3c2c90 0x5587dc85e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:39.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.464+0000 7fbfa9800640 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5587dc3c2c90 0x5587dc85e6f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f41534e568368758 server_cookie=8f357814c1e991dd in_seq=0 out_seq=0 2024-07-26T07:50:39.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.464+0000 7fbf8b400640 10 client.15198 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:50:39.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.465+0000 7fbf8b400640 1 -- 172.21.15.97:0/586819939 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5587dc745aa0 con 0x5587dc3c2c90 2024-07-26T07:50:39.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.465+0000 7fbf8b400640 10 client.15198 handle_command_reply: tid=0 2024-07-26T07:50:39.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.465+0000 7fbfb2e01740 1 client.15198 shutdown 2024-07-26T07:50:39.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.465+0000 7fbfb2e01740 1 -- 172.21.15.97:0/586819939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 msgr2=0x5587dc7f94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.465+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7f94c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c0e41c0 tx=0x7fbf9c0b0890 comp rx=0 tx=0).stop 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.465+0000 7fbfb2e01740 1 -- 172.21.15.97:0/586819939 shutdown_connections 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.465+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5587dc3c2c90 0x5587dc85e6f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.466+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5587dc42ef60 0x5587dc7f94c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.466+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587dc40e280 0x5587dc7f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.466+0000 7fbf8b400640 0 client.15198 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.466+0000 7fbfb2e01740 1 --2- 172.21.15.97:0/586819939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5587dc4060a0 0x5587dc7f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.466+0000 7fbfb2e01740 1 -- 172.21.15.97:0/586819939 >> 172.21.15.97:0/586819939 conn(0x5587dc423350 msgr2=0x5587dc7d7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.466+0000 7fbfb2e01740 1 -- 172.21.15.97:0/586819939 shutdown_connections 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.466+0000 7fbfb2e01740 1 -- 172.21.15.97:0/586819939 wait complete. 2024-07-26T07:50:39.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.466+0000 7fbfb2e01740 20 client.15198 trim_cache size 0 max 16384 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34583, 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 170.621969224, 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:50:39.471 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 191.255832142 2024-07-26T07:50:39.472 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:50:39.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 -- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbf8c0b3c00 msgr2=0x7fbf8c0b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbf8c0b3c00 0x7fbf8c0b6110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbf980058c0 tx=0x7fbf98005850 comp rx=0 tx=0).stop 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 -- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410fc80 msgr2=0x7fbfa41a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410fc80 0x7fbfa41a5ac0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00eaa0 tx=0x7fbf9c00ef60 comp rx=0 tx=0).stop 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 -- 172.21.15.97:0/624335385 shutdown_connections 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbf8c0b3c00 0x7fbf8c0b6110 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfa410fc80 0x7fbfa41a5ac0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa410f330 0x7fbfa41a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 --2- 172.21.15.97:0/624335385 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfa41099f0 0x7fbfa41a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 -- 172.21.15.97:0/624335385 >> 172.21.15.97:0/624335385 conn(0x7fbfa4078ff0 msgr2=0x7fbfa4079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 -- 172.21.15.97:0/624335385 shutdown_connections 2024-07-26T07:50:39.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:39.467+0000 7fbfaac00640 1 -- 172.21.15.97:0/624335385 wait complete. 2024-07-26T07:50:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:39 smithi097 ceph-mon[32231]: pgmap v476: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:50:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:39 smithi115 ceph-mon[35364]: pgmap v476: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:50:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:39 smithi181 ceph-mon[35869]: pgmap v476: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:50:40.241 INFO:tasks.cephadm:Setup complete, yielding 2024-07-26T07:50:40.242 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T07:50:40.260 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-07-26T07:50:40.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch status' 2024-07-26T07:50:40.716 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:41.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:41 smithi097 ceph-mon[32231]: pgmap v477: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T07:50:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:41 smithi115 ceph-mon[35364]: pgmap v477: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T07:50:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:41 smithi181 ceph-mon[35869]: pgmap v477: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 -- 172.21.15.97:0/4177840109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc51010af40 msgr2=0x7fc51010d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 --2- 172.21.15.97:0/4177840109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc51010af40 0x7fc51010d3e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc508009900 tx=0x7fc50802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 -- 172.21.15.97:0/4177840109 shutdown_connections 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 --2- 172.21.15.97:0/4177840109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc51010af40 0x7fc51010d3e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 --2- 172.21.15.97:0/4177840109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc510075720 0x7fc510075b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 --2- 172.21.15.97:0/4177840109 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc510076b10 0x7fc5100751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 -- 172.21.15.97:0/4177840109 >> 172.21.15.97:0/4177840109 conn(0x7fc5100fe8f0 msgr2=0x7fc510100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 -- 172.21.15.97:0/4177840109 shutdown_connections 2024-07-26T07:50:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.741+0000 7fc50f200640 1 -- 172.21.15.97:0/4177840109 wait complete. 2024-07-26T07:50:42.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.743+0000 7fc50f200640 1 Processor -- start 2024-07-26T07:50:42.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.743+0000 7fc50f200640 1 -- start start 2024-07-26T07:50:42.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.743+0000 7fc50f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc510075720 0x7fc51019b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:42.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.743+0000 7fc50f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc510076b10 0x7fc51019bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:42.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.743+0000 7fc50f200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51010af40 0x7fc51019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:42.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.743+0000 7fc50f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc510110ce0 con 0x7fc510075720 2024-07-26T07:50:42.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.743+0000 7fc50f200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc510110b60 con 0x7fc51010af40 2024-07-26T07:50:42.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.743+0000 7fc50f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc510110e60 con 0x7fc510076b10 2024-07-26T07:50:42.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc510075720 0x7fc51019b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:42.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc510075720 0x7fc51019b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53588/0 (socket says 172.21.15.97:53588) 2024-07-26T07:50:42.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50de00640 1 -- 172.21.15.97:0/3329872215 learned_addr learned my addr 172.21.15.97:0/3329872215 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:42.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50e800640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51010af40 0x7fc51019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:42.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc505400640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc510076b10 0x7fc51019bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:42.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50de00640 1 -- 172.21.15.97:0/3329872215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc510076b10 msgr2=0x7fc51019bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:42.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50de00640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc510076b10 0x7fc51019bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50de00640 1 -- 172.21.15.97:0/3329872215 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51010af40 msgr2=0x7fc51019cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:42.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50de00640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51010af40 0x7fc51019cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.744+0000 7fc50de00640 1 -- 172.21.15.97:0/3329872215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51019d360 con 0x7fc510075720 2024-07-26T07:50:42.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.745+0000 7fc50de00640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc510075720 0x7fc51019b930 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc50000ba50 tx=0x7fc50000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:42.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.745+0000 7fc507400640 1 -- 172.21.15.97:0/3329872215 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500019070 con 0x7fc510075720 2024-07-26T07:50:42.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.745+0000 7fc507400640 1 -- 172.21.15.97:0/3329872215 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc50000cbe0 con 0x7fc510075720 2024-07-26T07:50:42.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.745+0000 7fc507400640 1 -- 172.21.15.97:0/3329872215 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000075d0 con 0x7fc510075720 2024-07-26T07:50:42.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.745+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc51019d650 con 0x7fc510075720 2024-07-26T07:50:42.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.745+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5101a3d50 con 0x7fc510075720 2024-07-26T07:50:42.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.746+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc510109d50 con 0x7fc510075720 2024-07-26T07:50:42.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.749+0000 7fc507400640 1 -- 172.21.15.97:0/3329872215 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fc50001c070 con 0x7fc510075720 2024-07-26T07:50:42.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.750+0000 7fc507400640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc4f40b3c50 0x7fc4f40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:42.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.751+0000 7fc507400640 1 -- 172.21.15.97:0/3329872215 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fc5000e3420 con 0x7fc510075720 2024-07-26T07:50:42.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.751+0000 7fc505400640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc4f40b3c50 0x7fc4f40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:42.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.752+0000 7fc505400640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc4f40b3c50 0x7fc4f40b6160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc4f80058c0 tx=0x7fc4f800a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:42.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.755+0000 7fc507400640 1 -- 172.21.15.97:0/3329872215 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc500052600 con 0x7fc510075720 2024-07-26T07:50:42.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.970+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fc510063230 con 0x7fc4f40b3c50 2024-07-26T07:50:42.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.972+0000 7fc507400640 1 -- 172.21.15.97:0/3329872215 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fc510063230 con 0x7fc4f40b3c50 2024-07-26T07:50:42.973 INFO:teuthology.orchestra.run.smithi097.stdout:Backend: cephadm 2024-07-26T07:50:42.973 INFO:teuthology.orchestra.run.smithi097.stdout:Available: Yes 2024-07-26T07:50:42.973 INFO:teuthology.orchestra.run.smithi097.stdout:Paused: No 2024-07-26T07:50:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.974+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc4f40b3c50 msgr2=0x7fc4f40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.974+0000 7fc50f200640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc4f40b3c50 0x7fc4f40b6160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc4f80058c0 tx=0x7fc4f800a4c0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.974+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc510075720 msgr2=0x7fc51019b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.974+0000 7fc50f200640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc510075720 0x7fc51019b930 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc50000ba50 tx=0x7fc50000bf10 comp rx=0 tx=0).stop 2024-07-26T07:50:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.975+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 shutdown_connections 2024-07-26T07:50:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.975+0000 7fc50f200640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc4f40b3c50 0x7fc4f40b6160 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.975+0000 7fc50f200640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51010af40 0x7fc51019cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.975+0000 7fc50f200640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc510076b10 0x7fc51019bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.975+0000 7fc50f200640 1 --2- 172.21.15.97:0/3329872215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc510075720 0x7fc51019b930 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:42.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.975+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 >> 172.21.15.97:0/3329872215 conn(0x7fc5100fe8f0 msgr2=0x7fc510106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:42.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.975+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 shutdown_connections 2024-07-26T07:50:42.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:42.975+0000 7fc50f200640 1 -- 172.21.15.97:0/3329872215 wait complete. 2024-07-26T07:50:43.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch ps' 2024-07-26T07:50:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:43 smithi097 ceph-mon[32231]: pgmap v478: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T07:50:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:43 smithi097 ceph-mon[32231]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:44.221 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:43 smithi115 ceph-mon[35364]: pgmap v478: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T07:50:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:43 smithi115 ceph-mon[35364]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:43 smithi181 ceph-mon[35869]: pgmap v478: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T07:50:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:43 smithi181 ceph-mon[35869]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:45 smithi097 ceph-mon[32231]: pgmap v479: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T07:50:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:45 smithi115 ceph-mon[35364]: pgmap v479: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T07:50:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:45 smithi181 ceph-mon[35869]: pgmap v479: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T07:50:46.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 -- 172.21.15.97:0/2949460084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 msgr2=0x7f204010d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:46.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 --2- 172.21.15.97:0/2949460084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 0x7f204010d590 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2038009ca0 tx=0x7f203802a950 comp rx=0 tx=0).stop 2024-07-26T07:50:46.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 -- 172.21.15.97:0/2949460084 shutdown_connections 2024-07-26T07:50:46.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 --2- 172.21.15.97:0/2949460084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 0x7f204010d590 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 --2- 172.21.15.97:0/2949460084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2040106250 0x7f2040106660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 --2- 172.21.15.97:0/2949460084 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20401051c0 0x7f2040105630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 -- 172.21.15.97:0/2949460084 >> 172.21.15.97:0/2949460084 conn(0x7f2040100b30 msgr2=0x7f2040102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:46.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 -- 172.21.15.97:0/2949460084 shutdown_connections 2024-07-26T07:50:46.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.251+0000 7f203f200640 1 -- 172.21.15.97:0/2949460084 wait complete. 2024-07-26T07:50:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.252+0000 7f203f200640 1 Processor -- start 2024-07-26T07:50:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.252+0000 7f203f200640 1 -- start start 2024-07-26T07:50:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203f200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20401051c0 0x7f204019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2040106250 0x7f204019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 0x7f204019cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2040110d20 con 0x7f2040106bf0 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203f200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2040110ba0 con 0x7f20401051c0 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2040110ea0 con 0x7f2040106250 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 0x7f204019cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 0x7f204019cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34246/0 (socket says 172.21.15.97:34246) 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203e800640 1 -- 172.21.15.97:0/1729570030 learned_addr learned my addr 172.21.15.97:0/1729570030 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203de00640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20401051c0 0x7f204019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203d400640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2040106250 0x7f204019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203e800640 1 -- 172.21.15.97:0/1729570030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2040106250 msgr2=0x7f204019bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:46.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.253+0000 7f203e800640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2040106250 0x7f204019bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f203e800640 1 -- 172.21.15.97:0/1729570030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20401051c0 msgr2=0x7f204019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:46.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f203e800640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20401051c0 0x7f204019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f203e800640 1 -- 172.21.15.97:0/1729570030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204019d300 con 0x7f2040106bf0 2024-07-26T07:50:46.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f203e800640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 0x7f204019cae0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f203802af40 tx=0x7f2038003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:46.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f2036a00640 1 -- 172.21.15.97:0/1729570030 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038034040 con 0x7f2040106bf0 2024-07-26T07:50:46.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f2036a00640 1 -- 172.21.15.97:0/1729570030 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f203802fdf0 con 0x7f2040106bf0 2024-07-26T07:50:46.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204019d590 con 0x7f2040106bf0 2024-07-26T07:50:46.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f2036a00640 1 -- 172.21.15.97:0/1729570030 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038045480 con 0x7f2040106bf0 2024-07-26T07:50:46.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.254+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20401a3cf0 con 0x7f2040106bf0 2024-07-26T07:50:46.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.256+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f204010c380 con 0x7f2040106bf0 2024-07-26T07:50:46.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.258+0000 7f2036a00640 1 -- 172.21.15.97:0/1729570030 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f2038037020 con 0x7f2040106bf0 2024-07-26T07:50:46.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.260+0000 7f2036a00640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20240b3c50 0x7f20240b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:46.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.260+0000 7f2036a00640 1 -- 172.21.15.97:0/1729570030 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f2038104e20 con 0x7f2040106bf0 2024-07-26T07:50:46.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.260+0000 7f203de00640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20240b3c50 0x7f20240b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:46.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.261+0000 7f203de00640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20240b3c50 0x7f20240b6160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f20300058c0 tx=0x7f203000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:46.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.264+0000 7f2036a00640 1 -- 172.21.15.97:0/1729570030 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f20380cc410 con 0x7f2040106bf0 2024-07-26T07:50:46.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.472+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2040063230 con 0x7f20240b3c50 2024-07-26T07:50:46.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.507+0000 7f2036a00640 1 -- 172.21.15.97:0/1729570030 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f2040063230 con 0x7f20240b3c50 2024-07-26T07:50:46.509 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-07-26T07:50:46.509 INFO:teuthology.orchestra.run.smithi097.stdout:mds.a smithi097 running (3m) 47s ago 3m 86.3M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 53a37cf587b4 2024-07-26T07:50:46.509 INFO:teuthology.orchestra.run.smithi097.stdout:mds.b smithi115 running (3m) 58s ago 3m 92.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa ec7f3da47c46 2024-07-26T07:50:46.509 INFO:teuthology.orchestra.run.smithi097.stdout:mds.c smithi181 running (3m) 58s ago 3m 87.0M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 8970b3366a51 2024-07-26T07:50:46.509 INFO:teuthology.orchestra.run.smithi097.stdout:mds.d smithi097 running (3m) 47s ago 3m 81.1M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa c053db36be3d 2024-07-26T07:50:46.509 INFO:teuthology.orchestra.run.smithi097.stdout:mds.e smithi115 running (3m) 58s ago 3m 85.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 0940ab2ad1c4 2024-07-26T07:50:46.509 INFO:teuthology.orchestra.run.smithi097.stdout:mds.f smithi181 running (3m) 58s ago 3m 85.6M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 15af4714afed 2024-07-26T07:50:46.509 INFO:teuthology.orchestra.run.smithi097.stdout:mds.g smithi097 running (3m) 47s ago 3m 86.3M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 023a8794e0a1 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mds.h smithi115 running (3m) 58s ago 3m 86.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa fa4447fff829 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mds.i smithi181 running (3m) 58s ago 3m 85.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 51179b6e8ec8 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mds.j smithi097 running (3m) 47s ago 3m 81.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa e1efe33a3a8a 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mds.k smithi115 running (3m) 58s ago 3m 86.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa a7747372ca68 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mds.l smithi181 running (4m) 58s ago 4m 85.8M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 4af81ce494c9 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi097 *:9283,8765 running (14m) 47s ago 14m 592M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa bd9819eb29f0 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi115 *:8443,8765 running (11m) 58s ago 11m 522M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d2f28bd7fa4f 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.z smithi181 *:8443,8765 running (12m) 58s ago 12m 522M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 5b65aa30ade1 2024-07-26T07:50:46.510 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (14m) 47s ago 14m 121M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa aa7ca637e85b 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi115 running (12m) 58s ago 12m 109M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 15eec0fbd5e2 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi181 running (12m) 58s ago 12m 110M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 2c20e9d8ef7f 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (10m) 47s ago 10m 123M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa e55aa5ae8be4 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi115 running (10m) 58s ago 10m 122M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 81b6c15599e4 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi181 running (9m) 58s ago 9m 116M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 5fc7b65d6b76 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (9m) 47s ago 9m 115M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 76317a420734 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi115 running (8m) 58s ago 8m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa c432fde95d51 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi181 running (7m) 58s ago 7m 120M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 29bd4a299611 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi097 running (7m) 47s ago 7m 121M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa b40773062769 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi115 running (6m) 58s ago 6m 113M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 8cbcee71a5af 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8 smithi181 running (6m) 58s ago 6m 121M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 23efb18fff5f 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9 smithi097 running (5m) 47s ago 5m 119M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa bbc6d64f58d4 2024-07-26T07:50:46.511 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10 smithi115 running (5m) 58s ago 5m 120M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 35fa6fd2a715 2024-07-26T07:50:46.512 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11 smithi181 running (4m) 58s ago 4m 116M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa e1264dd63316 2024-07-26T07:50:46.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20240b3c50 msgr2=0x7f20240b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:46.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20240b3c50 0x7f20240b6160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f20300058c0 tx=0x7f203000a4c0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 msgr2=0x7f204019cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 0x7f204019cae0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f203802af40 tx=0x7f2038003f10 comp rx=0 tx=0).stop 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 shutdown_connections 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20240b3c50 0x7f20240b6160 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2040106bf0 0x7f204019cae0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2040106250 0x7f204019bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 --2- 172.21.15.97:0/1729570030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20401051c0 0x7f204019b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.510+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 >> 172.21.15.97:0/1729570030 conn(0x7f2040100b30 msgr2=0x7f2040109550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.511+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 shutdown_connections 2024-07-26T07:50:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:46.511+0000 7f203f200640 1 -- 172.21.15.97:0/1729570030 wait complete. 2024-07-26T07:50:47.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch ls' 2024-07-26T07:50:47.721 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:48.059 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:47 smithi097 ceph-mon[32231]: pgmap v480: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T07:50:48.059 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:47 smithi097 ceph-mon[32231]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:47 smithi115 ceph-mon[35364]: pgmap v480: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T07:50:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:47 smithi115 ceph-mon[35364]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:47 smithi181 ceph-mon[35869]: pgmap v480: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T07:50:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:47 smithi181 ceph-mon[35869]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:49.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.737+0000 7fe19d200640 1 -- 172.21.15.97:0/3213094076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1981095d0 msgr2=0x7fe198109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:49.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.737+0000 7fe19d200640 1 --2- 172.21.15.97:0/3213094076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1981095d0 0x7fe198109a40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe188009900 tx=0x7fe18802a480 comp rx=0 tx=0).stop 2024-07-26T07:50:49.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.738+0000 7fe19d200640 1 -- 172.21.15.97:0/3213094076 shutdown_connections 2024-07-26T07:50:49.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.738+0000 7fe19d200640 1 --2- 172.21.15.97:0/3213094076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe19810b000 0x7fe1981119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.738+0000 7fe19d200640 1 --2- 172.21.15.97:0/3213094076 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe19810a660 0x7fe19810aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.738+0000 7fe19d200640 1 --2- 172.21.15.97:0/3213094076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1981095d0 0x7fe198109a40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.738+0000 7fe19d200640 1 -- 172.21.15.97:0/3213094076 >> 172.21.15.97:0/3213094076 conn(0x7fe198078970 msgr2=0x7fe19807ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:49.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.738+0000 7fe19d200640 1 -- 172.21.15.97:0/3213094076 shutdown_connections 2024-07-26T07:50:49.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.738+0000 7fe19d200640 1 -- 172.21.15.97:0/3213094076 wait complete. 2024-07-26T07:50:49.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.739+0000 7fe19d200640 1 Processor -- start 2024-07-26T07:50:49.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.740+0000 7fe19d200640 1 -- start start 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.740+0000 7fe19d200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe1981095d0 0x7fe19819fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.740+0000 7fe19d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe19810a660 0x7fe1981a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.740+0000 7fe19d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe19810b000 0x7fe1981a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.740+0000 7fe19d200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe1981150d0 con 0x7fe19810a660 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.740+0000 7fe19d200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe198114f50 con 0x7fe1981095d0 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.740+0000 7fe19d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe198115250 con 0x7fe19810b000 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe18ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe19810a660 0x7fe1981a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe18ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe19810a660 0x7fe1981a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34264/0 (socket says 172.21.15.97:34264) 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe18ea00640 1 -- 172.21.15.97:0/3621851078 learned_addr learned my addr 172.21.15.97:0/3621851078 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe197400640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe1981095d0 0x7fe19819fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:49.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe197e00640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe19810b000 0x7fe1981a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:49.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe18ea00640 1 -- 172.21.15.97:0/3621851078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe19810b000 msgr2=0x7fe1981a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe18ea00640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe19810b000 0x7fe1981a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe18ea00640 1 -- 172.21.15.97:0/3621851078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe1981095d0 msgr2=0x7fe19819fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe18ea00640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe1981095d0 0x7fe19819fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe18ea00640 1 -- 172.21.15.97:0/3621851078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1981a17b0 con 0x7fe19810a660 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.741+0000 7fe197e00640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe19810b000 0x7fe1981a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.742+0000 7fe18ea00640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe19810a660 0x7fe1981a03f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9f0 tx=0x7fe18000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.742+0000 7fe194c00640 1 -- 172.21.15.97:0/3621851078 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000cd80 con 0x7fe19810a660 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.742+0000 7fe194c00640 1 -- 172.21.15.97:0/3621851078 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1800108c0 con 0x7fe19810a660 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.742+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1981a1aa0 con 0x7fe19810a660 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.742+0000 7fe194c00640 1 -- 172.21.15.97:0/3621851078 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1800187c0 con 0x7fe19810a660 2024-07-26T07:50:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.742+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1981a81a0 con 0x7fe19810a660 2024-07-26T07:50:49.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.743+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe198109a40 con 0x7fe19810a660 2024-07-26T07:50:49.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.746+0000 7fe194c00640 1 -- 172.21.15.97:0/3621851078 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe180010430 con 0x7fe19810a660 2024-07-26T07:50:49.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.748+0000 7fe194c00640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe17c0b3bd0 0x7fe17c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:49.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.748+0000 7fe194c00640 1 -- 172.21.15.97:0/3621851078 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fe1800e4580 con 0x7fe19810a660 2024-07-26T07:50:49.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.749+0000 7fe197400640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe17c0b3bd0 0x7fe17c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:49.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.749+0000 7fe197400640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe17c0b3bd0 0x7fe17c0b60e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe1880057c0 tx=0x7fe188000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:49.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.750+0000 7fe194c00640 1 -- 172.21.15.97:0/3621851078 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe1800abb10 con 0x7fe19810a660 2024-07-26T07:50:49.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.963+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fe198063230 con 0x7fe17c0b3bd0 2024-07-26T07:50:49.964 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:49 smithi097 ceph-mon[32231]: pgmap v481: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:50:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.979+0000 7fe194c00640 1 -- 172.21.15.97:0/3621851078 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fe198063230 con 0x7fe17c0b3bd0 2024-07-26T07:50:49.980 INFO:teuthology.orchestra.run.smithi097.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-07-26T07:50:49.980 INFO:teuthology.orchestra.run.smithi097.stdout:mds.all 12/12 62s ago 4m smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi115=b;smithi115=e;smithi115=h;smithi115=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-07-26T07:50:49.980 INFO:teuthology.orchestra.run.smithi097.stdout:mgr 3/3 62s ago 12m smithi097=x;smithi115=y;smithi181=z;count:3 2024-07-26T07:50:49.980 INFO:teuthology.orchestra.run.smithi097.stdout:mon 3/3 62s ago 12m smithi097:172.21.15.97=a;smithi115:172.21.15.115=b;smithi181:172.21.15.181=c;count:3 2024-07-26T07:50:49.981 INFO:teuthology.orchestra.run.smithi097.stdout:osd 12 62s ago - 2024-07-26T07:50:49.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe17c0b3bd0 msgr2=0x7fe17c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:49.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe17c0b3bd0 0x7fe17c0b60e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe1880057c0 tx=0x7fe188000bc0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe19810a660 msgr2=0x7fe1981a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:49.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe19810a660 0x7fe1981a03f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9f0 tx=0x7fe18000eeb0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 shutdown_connections 2024-07-26T07:50:49.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe17c0b3bd0 0x7fe17c0b60e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe19810b000 0x7fe1981a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe19810a660 0x7fe1981a03f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 --2- 172.21.15.97:0/3621851078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe1981095d0 0x7fe19819fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:49.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.981+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 >> 172.21.15.97:0/3621851078 conn(0x7fe198078970 msgr2=0x7fe19810d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:49.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.982+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 shutdown_connections 2024-07-26T07:50:49.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:49.982+0000 7fe19d200640 1 -- 172.21.15.97:0/3621851078 wait complete. 2024-07-26T07:50:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:49 smithi115 ceph-mon[35364]: pgmap v481: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:50:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:49 smithi181 ceph-mon[35869]: pgmap v481: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T07:50:50.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch host ls' 2024-07-26T07:50:51.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:50 smithi097 ceph-mon[32231]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:51.092 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:50 smithi115 ceph-mon[35364]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:50 smithi181 ceph-mon[35869]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:52.052 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:51 smithi097 ceph-mon[32231]: pgmap v482: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:52.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:51 smithi115 ceph-mon[35364]: pgmap v482: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:51 smithi181 ceph-mon[35869]: pgmap v482: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.106+0000 7eff20c00640 1 -- 172.21.15.97:0/4029897052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c104050 msgr2=0x7eff1c104460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.106+0000 7eff20c00640 1 --2- 172.21.15.97:0/4029897052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c104050 0x7eff1c104460 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7eff08009900 tx=0x7eff0802a480 comp rx=0 tx=0).stop 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.106+0000 7eff20c00640 1 -- 172.21.15.97:0/4029897052 shutdown_connections 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.106+0000 7eff20c00640 1 --2- 172.21.15.97:0/4029897052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff1c1049f0 0x7eff1c10b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.106+0000 7eff20c00640 1 --2- 172.21.15.97:0/4029897052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c104050 0x7eff1c104460 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.106+0000 7eff20c00640 1 --2- 172.21.15.97:0/4029897052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff1c102fc0 0x7eff1c103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.106+0000 7eff20c00640 1 -- 172.21.15.97:0/4029897052 >> 172.21.15.97:0/4029897052 conn(0x7eff1c0fe930 msgr2=0x7eff1c100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.106+0000 7eff20c00640 1 -- 172.21.15.97:0/4029897052 shutdown_connections 2024-07-26T07:50:53.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.107+0000 7eff20c00640 1 -- 172.21.15.97:0/4029897052 wait complete. 2024-07-26T07:50:53.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.108+0000 7eff20c00640 1 Processor -- start 2024-07-26T07:50:53.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.108+0000 7eff20c00640 1 -- start start 2024-07-26T07:50:53.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff20c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff1c102fc0 0x7eff1c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:53.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff20c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff1c104050 0x7eff1c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:53.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff20c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c1049f0 0x7eff1c19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:53.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff20c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c10ead0 con 0x7eff1c1049f0 2024-07-26T07:50:53.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff20c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c10e950 con 0x7eff1c102fc0 2024-07-26T07:50:53.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff20c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c10ec50 con 0x7eff1c104050 2024-07-26T07:50:53.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff1be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c1049f0 0x7eff1c19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:53.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff1be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c1049f0 0x7eff1c19a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34280/0 (socket says 172.21.15.97:34280) 2024-07-26T07:50:53.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff1be00640 1 -- 172.21.15.97:0/595432321 learned_addr learned my addr 172.21.15.97:0/595432321 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:53.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff1aa00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff1c104050 0x7eff1c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:53.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.109+0000 7eff1b400640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff1c102fc0 0x7eff1c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:53.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff1be00640 1 -- 172.21.15.97:0/595432321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff1c104050 msgr2=0x7eff1c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff1be00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff1c104050 0x7eff1c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff1be00640 1 -- 172.21.15.97:0/595432321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff1c102fc0 msgr2=0x7eff1c199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff1be00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff1c102fc0 0x7eff1c199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff1be00640 1 -- 172.21.15.97:0/595432321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c19b170 con 0x7eff1c1049f0 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff1aa00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff1c104050 0x7eff1c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff1be00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c1049f0 0x7eff1c19a950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7eff1400eaa0 tx=0x7eff1400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff13e00640 1 -- 172.21.15.97:0/595432321 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1400cdb0 con 0x7eff1c1049f0 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff13e00640 1 -- 172.21.15.97:0/595432321 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7eff14005020 con 0x7eff1c1049f0 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1c19b460 con 0x7eff1c1049f0 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff13e00640 1 -- 172.21.15.97:0/595432321 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff140116d0 con 0x7eff1c1049f0 2024-07-26T07:50:53.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.110+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1c1a1b60 con 0x7eff1c1049f0 2024-07-26T07:50:53.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.112+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff1c10a1a0 con 0x7eff1c1049f0 2024-07-26T07:50:53.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.114+0000 7eff13e00640 1 -- 172.21.15.97:0/595432321 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7eff140051d0 con 0x7eff1c1049f0 2024-07-26T07:50:53.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.115+0000 7eff13e00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff000b3c50 0x7eff000b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:53.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.116+0000 7eff13e00640 1 -- 172.21.15.97:0/595432321 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7eff140e3e90 con 0x7eff1c1049f0 2024-07-26T07:50:53.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.116+0000 7eff1b400640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff000b3c50 0x7eff000b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:53.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.116+0000 7eff1b400640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff000b3c50 0x7eff000b6160 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7eff0c005fd0 tx=0x7eff0c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:53.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.119+0000 7eff13e00640 1 -- 172.21.15.97:0/595432321 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7eff14016030 con 0x7eff1c1049f0 2024-07-26T07:50:53.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.323+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7eff1c063230 con 0x7eff000b3c50 2024-07-26T07:50:53.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.325+0000 7eff13e00640 1 -- 172.21.15.97:0/595432321 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7eff1c063230 con 0x7eff000b3c50 2024-07-26T07:50:53.326 INFO:teuthology.orchestra.run.smithi097.stdout:HOST ADDR LABELS STATUS 2024-07-26T07:50:53.327 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 172.21.15.97 2024-07-26T07:50:53.327 INFO:teuthology.orchestra.run.smithi097.stdout:smithi115 172.21.15.115 2024-07-26T07:50:53.327 INFO:teuthology.orchestra.run.smithi097.stdout:smithi181 172.21.15.181 2024-07-26T07:50:53.327 INFO:teuthology.orchestra.run.smithi097.stdout:3 hosts in cluster 2024-07-26T07:50:53.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff000b3c50 msgr2=0x7eff000b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:53.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff000b3c50 0x7eff000b6160 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7eff0c005fd0 tx=0x7eff0c005850 comp rx=0 tx=0).stop 2024-07-26T07:50:53.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c1049f0 msgr2=0x7eff1c19a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:53.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c1049f0 0x7eff1c19a950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7eff1400eaa0 tx=0x7eff1400ef60 comp rx=0 tx=0).stop 2024-07-26T07:50:53.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 shutdown_connections 2024-07-26T07:50:53.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff000b3c50 0x7eff000b6160 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff1c1049f0 0x7eff1c19a950 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff1c104050 0x7eff1c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 --2- 172.21.15.97:0/595432321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff1c102fc0 0x7eff1c199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:53.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.328+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 >> 172.21.15.97:0/595432321 conn(0x7eff1c0fe930 msgr2=0x7eff1c107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:53.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.329+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 shutdown_connections 2024-07-26T07:50:53.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:53.329+0000 7eff20c00640 1 -- 172.21.15.97:0/595432321 wait complete. 2024-07-26T07:50:53.994 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:53 smithi097 ceph-mon[32231]: pgmap v483: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:54.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch device ls' 2024-07-26T07:50:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:53 smithi115 ceph-mon[35364]: pgmap v483: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:53 smithi181 ceph-mon[35869]: pgmap v483: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:50:54.521 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:54.931 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:54 smithi097 ceph-mon[32231]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:54 smithi115 ceph-mon[35364]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:54 smithi181 ceph-mon[35869]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:55.969 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:55 smithi097 ceph-mon[32231]: pgmap v484: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:55 smithi115 ceph-mon[35364]: pgmap v484: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:55 smithi181 ceph-mon[35869]: pgmap v484: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:50:56.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 -- 172.21.15.97:0/3423106349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe518108cc0 msgr2=0x7fe51810b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:56.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 --2- 172.21.15.97:0/3423106349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe518108cc0 0x7fe51810b1d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe510009ca0 tx=0x7fe51002a8c0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 -- 172.21.15.97:0/3423106349 shutdown_connections 2024-07-26T07:50:56.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 --2- 172.21.15.97:0/3423106349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe518108cc0 0x7fe51810b1d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 --2- 172.21.15.97:0/3423106349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe518106180 0x7fe5181085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 --2- 172.21.15.97:0/3423106349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51806b350 0x7fe518105c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 -- 172.21.15.97:0/3423106349 >> 172.21.15.97:0/3423106349 conn(0x7fe5180fd800 msgr2=0x7fe5180ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:56.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 -- 172.21.15.97:0/3423106349 shutdown_connections 2024-07-26T07:50:56.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.469+0000 7fe51d000640 1 -- 172.21.15.97:0/3423106349 wait complete. 2024-07-26T07:50:56.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.470+0000 7fe51d000640 1 Processor -- start 2024-07-26T07:50:56.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.470+0000 7fe51d000640 1 -- start start 2024-07-26T07:50:56.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.470+0000 7fe51d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe51806b350 0x7fe5181996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.470+0000 7fe51d000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe518106180 0x7fe518199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe51d000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe518108cc0 0x7fe51819a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe51d000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810eae0 con 0x7fe51806b350 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe51d000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810e960 con 0x7fe518108cc0 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe51d000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810ec60 con 0x7fe518106180 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe51806b350 0x7fe5181996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe51806b350 0x7fe5181996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44414/0 (socket says 172.21.15.97:44414) 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517400640 1 -- 172.21.15.97:0/1000825017 learned_addr learned my addr 172.21.15.97:0/1000825017 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe50ea00640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe518106180 0x7fe518199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517e00640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe518108cc0 0x7fe51819a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517400640 1 -- 172.21.15.97:0/1000825017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe518106180 msgr2=0x7fe518199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:56.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517400640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe518106180 0x7fe518199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517400640 1 -- 172.21.15.97:0/1000825017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe518108cc0 msgr2=0x7fe51819a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:56.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517400640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe518108cc0 0x7fe51819a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.471+0000 7fe517400640 1 -- 172.21.15.97:0/1000825017 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51819b110 con 0x7fe51806b350 2024-07-26T07:50:56.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.472+0000 7fe517400640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe51806b350 0x7fe5181996e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe50800d770 tx=0x7fe50800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:56.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.472+0000 7fe514c00640 1 -- 172.21.15.97:0/1000825017 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508017070 con 0x7fe51806b350 2024-07-26T07:50:56.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.472+0000 7fe514c00640 1 -- 172.21.15.97:0/1000825017 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5080054b0 con 0x7fe51806b350 2024-07-26T07:50:56.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.472+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51819b400 con 0x7fe51806b350 2024-07-26T07:50:56.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.472+0000 7fe514c00640 1 -- 172.21.15.97:0/1000825017 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508004c60 con 0x7fe51806b350 2024-07-26T07:50:56.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.472+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5181a1b00 con 0x7fe51806b350 2024-07-26T07:50:56.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.474+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5180ff320 con 0x7fe51806b350 2024-07-26T07:50:56.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.476+0000 7fe514c00640 1 -- 172.21.15.97:0/1000825017 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe508004e00 con 0x7fe51806b350 2024-07-26T07:50:56.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.477+0000 7fe514c00640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4fc0b3c50 0x7fe4fc0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:56.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.478+0000 7fe514c00640 1 -- 172.21.15.97:0/1000825017 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fe5080e36f0 con 0x7fe51806b350 2024-07-26T07:50:56.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.478+0000 7fe50ea00640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4fc0b3c50 0x7fe4fc0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:56.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.478+0000 7fe50ea00640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4fc0b3c50 0x7fe4fc0b6160 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe500005fd0 tx=0x7fe500005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:56.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.482+0000 7fe514c00640 1 -- 172.21.15.97:0/1000825017 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe5080aace0 con 0x7fe51806b350 2024-07-26T07:50:56.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.685+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fe518063230 con 0x7fe4fc0b3c50 2024-07-26T07:50:56.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.690+0000 7fe514c00640 1 -- 172.21.15.97:0/1000825017 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fe518063230 con 0x7fe4fc0b3c50 2024-07-26T07:50:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-07-26T07:50:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013T400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-07-26T07:50:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:smithi115 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UU400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-07-26T07:50:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:smithi181 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6246000E400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-07-26T07:50:56.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.693+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4fc0b3c50 msgr2=0x7fe4fc0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:56.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.693+0000 7fe51d000640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4fc0b3c50 0x7fe4fc0b6160 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe500005fd0 tx=0x7fe500005850 comp rx=0 tx=0).stop 2024-07-26T07:50:56.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.693+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe51806b350 msgr2=0x7fe5181996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:56.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.693+0000 7fe51d000640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe51806b350 0x7fe5181996e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe50800d770 tx=0x7fe50800dc30 comp rx=0 tx=0).stop 2024-07-26T07:50:56.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.694+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 shutdown_connections 2024-07-26T07:50:56.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.694+0000 7fe51d000640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4fc0b3c50 0x7fe4fc0b6160 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.694+0000 7fe51d000640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe518108cc0 0x7fe51819a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.694+0000 7fe51d000640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe518106180 0x7fe518199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.694+0000 7fe51d000640 1 --2- 172.21.15.97:0/1000825017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe51806b350 0x7fe5181996e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:56.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.694+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 >> 172.21.15.97:0/1000825017 conn(0x7fe5180fd800 msgr2=0x7fe5181046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:56.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.694+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 shutdown_connections 2024-07-26T07:50:56.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:56.694+0000 7fe51d000640 1 -- 172.21.15.97:0/1000825017 wait complete. 2024-07-26T07:50:57.386 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T07:50:57.396 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-07-26T07:50:57.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- bash -c 'ceph fs dump' 2024-07-26T07:50:57.860 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:50:58.188 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:57 smithi097 ceph-mon[32231]: pgmap v485: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:58.189 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:57 smithi097 ceph-mon[32231]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:57 smithi115 ceph-mon[35364]: pgmap v485: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:57 smithi115 ceph-mon[35364]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:57 smithi181 ceph-mon[35869]: pgmap v485: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:50:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:57 smithi181 ceph-mon[35869]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.788+0000 7fc8db200640 1 -- 172.21.15.97:0/2313900822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 msgr2=0x7fc8dc109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.788+0000 7fc8db200640 1 --2- 172.21.15.97:0/2313900822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 0x7fc8dc109a40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009900 tx=0x7fc8cc02a480 comp rx=0 tx=0).stop 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.788+0000 7fc8db200640 1 -- 172.21.15.97:0/2313900822 shutdown_connections 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.788+0000 7fc8db200640 1 --2- 172.21.15.97:0/2313900822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8dc10b000 0x7fc8dc1119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.788+0000 7fc8db200640 1 --2- 172.21.15.97:0/2313900822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8dc10a660 0x7fc8dc10aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.788+0000 7fc8db200640 1 --2- 172.21.15.97:0/2313900822 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 0x7fc8dc109a40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.788+0000 7fc8db200640 1 -- 172.21.15.97:0/2313900822 >> 172.21.15.97:0/2313900822 conn(0x7fc8dc078970 msgr2=0x7fc8dc07ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.789+0000 7fc8db200640 1 -- 172.21.15.97:0/2313900822 shutdown_connections 2024-07-26T07:50:59.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.789+0000 7fc8db200640 1 -- 172.21.15.97:0/2313900822 wait complete. 2024-07-26T07:50:59.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.790+0000 7fc8db200640 1 Processor -- start 2024-07-26T07:50:59.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.790+0000 7fc8db200640 1 -- start start 2024-07-26T07:50:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8db200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 0x7fc8dc19fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8db200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8dc10a660 0x7fc8dc1a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8db200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8dc10b000 0x7fc8dc1a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8db200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc115110 con 0x7fc8dc1095d0 2024-07-26T07:50:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8db200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc114f90 con 0x7fc8dc10b000 2024-07-26T07:50:59.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8db200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc115290 con 0x7fc8dc10a660 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 0x7fc8dc19fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 0x7fc8dc19fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44428/0 (socket says 172.21.15.97:44428) 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d9e00640 1 -- 172.21.15.97:0/96587539 learned_addr learned my addr 172.21.15.97:0/96587539 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d1400640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8dc10a660 0x7fc8dc1a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8da800640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8dc10b000 0x7fc8dc1a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d9e00640 1 -- 172.21.15.97:0/96587539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8dc10a660 msgr2=0x7fc8dc1a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d9e00640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8dc10a660 0x7fc8dc1a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d9e00640 1 -- 172.21.15.97:0/96587539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8dc10b000 msgr2=0x7fc8dc1a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d9e00640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8dc10b000 0x7fc8dc1a0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d9e00640 1 -- 172.21.15.97:0/96587539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc1a17d0 con 0x7fc8dc1095d0 2024-07-26T07:50:59.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.791+0000 7fc8d1400640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8dc10a660 0x7fc8dc1a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:50:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.792+0000 7fc8d9e00640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 0x7fc8dc19fda0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc005760 tx=0x7fc8cc003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.792+0000 7fc8d3400640 1 -- 172.21.15.97:0/96587539 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc03f070 con 0x7fc8dc1095d0 2024-07-26T07:50:59.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.792+0000 7fc8d3400640 1 -- 172.21.15.97:0/96587539 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8cc031da0 con 0x7fc8dc1095d0 2024-07-26T07:50:59.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.792+0000 7fc8d3400640 1 -- 172.21.15.97:0/96587539 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc035030 con 0x7fc8dc1095d0 2024-07-26T07:50:59.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.792+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8dc1a1a60 con 0x7fc8dc1095d0 2024-07-26T07:50:59.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.792+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8dc1a81c0 con 0x7fc8dc1095d0 2024-07-26T07:50:59.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.793+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8dc109a40 con 0x7fc8dc1095d0 2024-07-26T07:50:59.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.796+0000 7fc8d3400640 1 -- 172.21.15.97:0/96587539 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fc8cc042070 con 0x7fc8dc1095d0 2024-07-26T07:50:59.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.797+0000 7fc8d3400640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc8c00b3cd0 0x7fc8c00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:50:59.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.797+0000 7fc8d3400640 1 -- 172.21.15.97:0/96587539 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fc8cc104da0 con 0x7fc8dc1095d0 2024-07-26T07:50:59.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.798+0000 7fc8d1400640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc8c00b3cd0 0x7fc8c00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:50:59.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.798+0000 7fc8d1400640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc8c00b3cd0 0x7fc8c00b61e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009b40 tx=0x7fc8c4005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:50:59.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:50:59.800+0000 7fc8d3400640 1 -- 172.21.15.97:0/96587539 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc8cc0cc330 con 0x7fc8dc1095d0 2024-07-26T07:51:00.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:50:59 smithi097 ceph-mon[32231]: pgmap v486: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:00.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.055+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fc8dc1107b0 con 0x7fc8dc1095d0 2024-07-26T07:51:00.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.055+0000 7fc8d3400640 1 -- 172.21.15.97:0/96587539 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3140 (secure 0 0 0) 0x7fc8dc1107b0 con 0x7fc8dc1095d0 2024-07-26T07:51:00.057 INFO:teuthology.orchestra.run.smithi097.stdout:e27 2024-07-26T07:51:00.058 INFO:teuthology.orchestra.run.smithi097.stdout:btime 2024-07-26T07:48:11:830543+0000 2024-07-26T07:51:00.058 INFO:teuthology.orchestra.run.smithi097.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T07:51:00.058 INFO:teuthology.orchestra.run.smithi097.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T07:51:00.058 INFO:teuthology.orchestra.run.smithi097.stdout:legacy client fscid: 1 2024-07-26T07:51:00.058 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:00.058 INFO:teuthology.orchestra.run.smithi097.stdout:Filesystem 'cephfs' (1) 2024-07-26T07:51:00.058 INFO:teuthology.orchestra.run.smithi097.stdout:fs_name cephfs 2024-07-26T07:51:00.058 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 27 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-07-26T07:47:48.482950+0000 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:modified 2024-07-26T07:48:11.504745+0000 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:tableserver 0 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:root 0 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:session_timeout 300 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:session_autoclose 300 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:max_file_size 1099511627776 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:max_xattr_size 65536 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:required_client_features {} 2024-07-26T07:51:00.059 INFO:teuthology.orchestra.run.smithi097.stdout:last_failure 0 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:last_failure_osd_epoch 0 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:max_mds 5 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:in 0,1,2,3,4 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:up {0=34583,1=34556,2=34535,3=34517,4=24562} 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:failed 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:damaged 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:stopped 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:data_pools [3] 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:metadata_pool 2 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:inline_data disabled 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:balancer 2024-07-26T07:51:00.060 INFO:teuthology.orchestra.run.smithi097.stdout:bal_rank_mask -1 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:standby_count_wanted 0 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:qdb_cluster leader: 34583 members: 24562,34517,34556,34535,34583 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b{0:34583} state up:active seq 7 addr [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c{0:24544} state up:standby-replay seq 1 addr [v2:172.21.15.181:6836/3053625422,v1:172.21.15.181:6837/3053625422] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.h{1:34556} state up:active seq 14 addr [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.f{1:24538} state up:standby-replay seq 1 addr [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.e{2:34535} state up:active seq 18 addr [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.l{2:24505} state up:standby-replay seq 1 addr [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.k{3:34517} state up:active seq 21 addr [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a{3:14760} state up:standby-replay seq 1 addr [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.i{4:24562} state up:active seq 14 addr [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.g{4:14748} state up:standby-replay seq 1 addr [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:00.061 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:00.062 INFO:teuthology.orchestra.run.smithi097.stdout:Standby daemons: 2024-07-26T07:51:00.062 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:00.062 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.j{-1:14685} state up:standby seq 1 addr [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.062 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d{-1:14691} state up:standby seq 1 addr [v2:172.21.15.97:6836/2971688113,v1:172.21.15.97:6837/2971688113] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T07:51:00.062 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 27 2024-07-26T07:51:00.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.059+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc8c00b3cd0 msgr2=0x7fc8c00b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:00.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.059+0000 7fc8db200640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc8c00b3cd0 0x7fc8c00b61e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009b40 tx=0x7fc8c4005e50 comp rx=0 tx=0).stop 2024-07-26T07:51:00.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.059+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 msgr2=0x7fc8dc19fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.059+0000 7fc8db200640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 0x7fc8dc19fda0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc005760 tx=0x7fc8cc003d50 comp rx=0 tx=0).stop 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.060+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 shutdown_connections 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.060+0000 7fc8db200640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc8c00b3cd0 0x7fc8c00b61e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.060+0000 7fc8db200640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc8dc10b000 0x7fc8dc1a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.060+0000 7fc8db200640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8dc10a660 0x7fc8dc1a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.060+0000 7fc8db200640 1 --2- 172.21.15.97:0/96587539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8dc1095d0 0x7fc8dc19fda0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.060+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 >> 172.21.15.97:0/96587539 conn(0x7fc8dc078970 msgr2=0x7fc8dc10d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.060+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 shutdown_connections 2024-07-26T07:51:00.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:00.060+0000 7fc8db200640 1 -- 172.21.15.97:0/96587539 wait complete. 2024-07-26T07:51:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:50:59 smithi115 ceph-mon[35364]: pgmap v486: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:50:59 smithi181 ceph-mon[35869]: pgmap v486: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:00.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- bash -c 'ceph osd dump' 2024-07-26T07:51:01.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:51:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:00 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/96587539' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T07:51:01.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:51:01.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:00 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/96587539' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T07:51:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:51:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:00 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/96587539' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T07:51:01.253 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:01 smithi097 ceph-mon[32231]: pgmap v487: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:51:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:51:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:51:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:01 smithi115 ceph-mon[35364]: pgmap v487: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:51:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:51:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:51:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:01 smithi181 ceph-mon[35869]: pgmap v487: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:51:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:51:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:51:03.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.299+0000 7f3227200640 1 -- 172.21.15.97:0/1304903883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3228106260 msgr2=0x7f3228106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:03.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.299+0000 7f3227200640 1 --2- 172.21.15.97:0/1304903883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3228106260 0x7f3228106670 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3214009900 tx=0x7f321402a480 comp rx=0 tx=0).stop 2024-07-26T07:51:03.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.299+0000 7f3227200640 1 -- 172.21.15.97:0/1304903883 shutdown_connections 2024-07-26T07:51:03.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.299+0000 7f3227200640 1 --2- 172.21.15.97:0/1304903883 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3228106c00 0x7f322810d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.299+0000 7f3227200640 1 --2- 172.21.15.97:0/1304903883 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3228106260 0x7f3228106670 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.299+0000 7f3227200640 1 --2- 172.21.15.97:0/1304903883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32281051d0 0x7f3228105640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.300+0000 7f3227200640 1 -- 172.21.15.97:0/1304903883 >> 172.21.15.97:0/1304903883 conn(0x7f3228100b60 msgr2=0x7f3228102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:03.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.300+0000 7f3227200640 1 -- 172.21.15.97:0/1304903883 shutdown_connections 2024-07-26T07:51:03.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.300+0000 7f3227200640 1 -- 172.21.15.97:0/1304903883 wait complete. 2024-07-26T07:51:03.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.301+0000 7f3227200640 1 Processor -- start 2024-07-26T07:51:03.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.301+0000 7f3227200640 1 -- start start 2024-07-26T07:51:03.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3227200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32281051d0 0x7f322819b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:03.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3227200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3228106260 0x7f322819c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:03.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3227200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3228106c00 0x7f322819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:03.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3227200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3228110ce0 con 0x7f32281051d0 2024-07-26T07:51:03.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3227200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3228110b60 con 0x7f3228106c00 2024-07-26T07:51:03.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3227200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3228110e60 con 0x7f3228106260 2024-07-26T07:51:03.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3225e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32281051d0 0x7f322819b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:03.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3225e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32281051d0 0x7f322819b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44446/0 (socket says 172.21.15.97:44446) 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3225e00640 1 -- 172.21.15.97:0/1317878740 learned_addr learned my addr 172.21.15.97:0/1317878740 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3225400640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3228106260 0x7f322819c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.302+0000 7f3226800640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3228106c00 0x7f322819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f3225e00640 1 -- 172.21.15.97:0/1317878740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3228106260 msgr2=0x7f322819c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f3225e00640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3228106260 0x7f322819c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f3225e00640 1 -- 172.21.15.97:0/1317878740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3228106c00 msgr2=0x7f322819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f3225e00640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3228106c00 0x7f322819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f3225e00640 1 -- 172.21.15.97:0/1317878740 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322819d370 con 0x7f32281051d0 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f3225e00640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32281051d0 0x7f322819b9d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f321800d770 tx=0x7f321800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f321ea00640 1 -- 172.21.15.97:0/1317878740 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3218017070 con 0x7f32281051d0 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f321ea00640 1 -- 172.21.15.97:0/1317878740 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32180054b0 con 0x7f32281051d0 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.303+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322819d660 con 0x7f32281051d0 2024-07-26T07:51:03.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.304+0000 7f321ea00640 1 -- 172.21.15.97:0/1317878740 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3218004c60 con 0x7f32281051d0 2024-07-26T07:51:03.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.304+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32281a3c20 con 0x7f32281051d0 2024-07-26T07:51:03.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.305+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f322810c3b0 con 0x7f32281051d0 2024-07-26T07:51:03.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.309+0000 7f321ea00640 1 -- 172.21.15.97:0/1317878740 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f3218004e00 con 0x7f32281051d0 2024-07-26T07:51:03.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.311+0000 7f321ea00640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f320c0b3bd0 0x7f320c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:03.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.311+0000 7f321ea00640 1 -- 172.21.15.97:0/1317878740 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f32180e3af0 con 0x7f32281051d0 2024-07-26T07:51:03.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.311+0000 7f3225400640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f320c0b3bd0 0x7f320c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:03.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.312+0000 7f3225400640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f320c0b3bd0 0x7f320c0b60e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f32140057c0 tx=0x7f3214000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:03.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.313+0000 7f321ea00640 1 -- 172.21.15.97:0/1317878740 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f32180ab0e0 con 0x7f32281051d0 2024-07-26T07:51:03.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.511+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f322802e0c0 con 0x7f32281051d0 2024-07-26T07:51:03.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.514+0000 7f321ea00640 1 -- 172.21.15.97:0/1317878740 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v86) ==== 56+0+5392 (secure 0 0 0) 0x7f32180aa9e0 con 0x7f32281051d0 2024-07-26T07:51:03.516 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 86 2024-07-26T07:51:03.516 INFO:teuthology.orchestra.run.smithi097.stdout:fsid 9621a072-4b21-11ef-bcb6-c7b262605968 2024-07-26T07:51:03.516 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-07-26T07:36:12.782822+0000 2024-07-26T07:51:03.516 INFO:teuthology.orchestra.run.smithi097.stdout:modified 2024-07-26T07:49:20.276546+0000 2024-07-26T07:51:03.516 INFO:teuthology.orchestra.run.smithi097.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:crush_version 26 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:full_ratio 0.95 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:backfillfull_ratio 0.9 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:nearfull_ratio 0.85 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:require_min_compat_client luminous 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:min_compat_client luminous 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:require_osd_release squid 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:stretch_mode_enabled false 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-07-26T07:51:03.517 INFO:teuthology.orchestra.run.smithi097.stdout:max_osd 12 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6802/706181072,v1:172.21.15.97:6803/706181072] [v2:172.21.15.97:6804/706181072,v1:172.21.15.97:6805/706181072] exists,up d860fc33-42a6-4123-af2c-5adff0e9b852 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 up in weight 1 up_from 13 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6800/2733242052,v1:172.21.15.115:6801/2733242052] [v2:172.21.15.115:6802/2733242052,v1:172.21.15.115:6803/2733242052] exists,up b2130819-8356-471c-b82d-89ce0ac47dc6 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 up in weight 1 up_from 18 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/1600494363,v1:172.21.15.181:6801/1600494363] [v2:172.21.15.181:6802/1600494363,v1:172.21.15.181:6803/1600494363] exists,up c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 up in weight 1 up_from 26 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6810/461651027,v1:172.21.15.97:6811/461651027] [v2:172.21.15.97:6812/461651027,v1:172.21.15.97:6813/461651027] exists,up 144e0dff-895b-471b-bd0c-8fcf3f51ac36 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 up in weight 1 up_from 32 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6808/1719675205,v1:172.21.15.115:6809/1719675205] [v2:172.21.15.115:6810/1719675205,v1:172.21.15.115:6811/1719675205] exists,up 7fddb4b8-44aa-42d7-8df4-c304b85c69b1 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6808/1061806763,v1:172.21.15.181:6809/1061806763] [v2:172.21.15.181:6810/1061806763,v1:172.21.15.181:6811/1061806763] exists,up 1272d9a0-8173-47b6-bfe2-017da6dc526d 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 up in weight 1 up_from 44 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6818/3873314113,v1:172.21.15.97:6819/3873314113] [v2:172.21.15.97:6820/3873314113,v1:172.21.15.97:6821/3873314113] exists,up 9567a0ec-9e3d-4b09-afc6-050d6ef59a83 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 up in weight 1 up_from 49 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6816/2846701605,v1:172.21.15.115:6817/2846701605] [v2:172.21.15.115:6818/2846701605,v1:172.21.15.115:6819/2846701605] exists,up 256172cc-f069-4599-bd65-14deca123b42 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8 up in weight 1 up_from 56 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6816/2477056065,v1:172.21.15.181:6817/2477056065] [v2:172.21.15.181:6818/2477056065,v1:172.21.15.181:6819/2477056065] exists,up 63b664e5-d348-4a9a-afb9-fe1fd1892a27 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9 up in weight 1 up_from 62 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6826/2688123967,v1:172.21.15.97:6827/2688123967] [v2:172.21.15.97:6828/2688123967,v1:172.21.15.97:6829/2688123967] exists,up 774b752e-42fc-4201-9051-9a9cdb1f495c 2024-07-26T07:51:03.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10 up in weight 1 up_from 68 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6824/2791495352,v1:172.21.15.115:6825/2791495352] [v2:172.21.15.115:6826/2791495352,v1:172.21.15.115:6827/2791495352] exists,up 3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11 up in weight 1 up_from 74 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6824/2026606552,v1:172.21.15.181:6825/2026606552] [v2:172.21.15.181:6826/2026606552,v1:172.21.15.181:6827/2026606552] exists,up 0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 2.b [2,4] 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 2.e [2,4] 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 2.10 [8,11] 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 2.11 [7,11] 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 2.1f [8,1] 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.0 [4,6] 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.2 [3,9] 2024-07-26T07:51:03.519 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.17 [4,7] 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.20 [5,9] 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.23 [4,8] 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.28 [5,8] 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.34 [4,9] 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/4135605223 expires 2024-07-27T07:37:17.917351+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:6801/4230832806 expires 2024-07-27T07:37:17.917351+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:6800/4230832806 expires 2024-07-27T07:37:17.917351+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/2767087495 expires 2024-07-27T07:37:17.917351+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/2403991941 expires 2024-07-27T07:37:17.917351+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:6801/3845319015 expires 2024-07-27T07:36:48.864300+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:6800/3845319015 expires 2024-07-27T07:36:48.864300+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/1962341976 expires 2024-07-27T07:36:48.864300+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/2498853950 expires 2024-07-27T07:36:48.864300+0000 2024-07-26T07:51:03.520 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/1642500127 expires 2024-07-27T07:36:48.864300+0000 2024-07-26T07:51:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.516+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f320c0b3bd0 msgr2=0x7f320c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f320c0b3bd0 0x7f320c0b60e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f32140057c0 tx=0x7f3214000bc0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32281051d0 msgr2=0x7f322819b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32281051d0 0x7f322819b9d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f321800d770 tx=0x7f321800dc30 comp rx=0 tx=0).stop 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 shutdown_connections 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f320c0b3bd0 0x7f320c0b60e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3228106c00 0x7f322819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3228106260 0x7f322819c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 --2- 172.21.15.97:0/1317878740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32281051d0 0x7f322819b9d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 >> 172.21.15.97:0/1317878740 conn(0x7f3228100b60 msgr2=0x7f3228109580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 shutdown_connections 2024-07-26T07:51:03.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:03.517+0000 7f3227200640 1 -- 172.21.15.97:0/1317878740 wait complete. 2024-07-26T07:51:04.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:03 smithi097 ceph-mon[32231]: pgmap v488: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:04.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:03 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1317878740' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T07:51:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:03 smithi115 ceph-mon[35364]: pgmap v488: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:03 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1317878740' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T07:51:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:03 smithi181 ceph-mon[35869]: pgmap v488: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:03 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1317878740' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T07:51:04.260 INFO:teuthology.run_tasks:Running task fs.ready... 2024-07-26T07:51:04.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:51:04.818 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:06.062 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:05 smithi097 ceph-mon[32231]: pgmap v489: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:06.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:05 smithi115 ceph-mon[35364]: pgmap v489: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:05 smithi181 ceph-mon[35869]: pgmap v489: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 -- 172.21.15.97:0/523836198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 msgr2=0x7ff4381164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 --2- 172.21.15.97:0/523836198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 0x7ff4381164b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff430009ca0 tx=0x7ff43002a8c0 comp rx=0 tx=0).stop 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 -- 172.21.15.97:0/523836198 shutdown_connections 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 --2- 172.21.15.97:0/523836198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 0x7ff4381164b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 --2- 172.21.15.97:0/523836198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff43810f330 0x7ff43810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 --2- 172.21.15.97:0/523836198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4381099f0 0x7ff438109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 -- 172.21.15.97:0/523836198 >> 172.21.15.97:0/523836198 conn(0x7ff438078ff0 msgr2=0x7ff438107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 -- 172.21.15.97:0/523836198 shutdown_connections 2024-07-26T07:51:06.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.817+0000 7ff43d400640 1 -- 172.21.15.97:0/523836198 wait complete. 2024-07-26T07:51:06.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.818+0000 7ff43d400640 1 Processor -- start 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.819+0000 7ff43d400640 1 -- start start 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.819+0000 7ff43d400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4381099f0 0x7ff4381a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.819+0000 7ff43d400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff43810f330 0x7ff4381a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.819+0000 7ff43d400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 0x7ff4381a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.819+0000 7ff43d400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff438119840 con 0x7ff43810fc80 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.819+0000 7ff43d400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff4381196c0 con 0x7ff4381099f0 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.819+0000 7ff43d400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4381199c0 con 0x7ff43810f330 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.819+0000 7ff43ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 0x7ff4381a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff43ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 0x7ff4381a5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52920/0 (socket says 172.21.15.97:52920) 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff43ca00640 1 -- 172.21.15.97:0/2935101891 learned_addr learned my addr 172.21.15.97:0/2935101891 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:06.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff42f400640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff43810f330 0x7ff4381a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:06.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff437e00640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4381099f0 0x7ff4381a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:06.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff43ca00640 1 -- 172.21.15.97:0/2935101891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff43810f330 msgr2=0x7ff4381a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff43ca00640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff43810f330 0x7ff4381a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff43ca00640 1 -- 172.21.15.97:0/2935101891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4381099f0 msgr2=0x7ff4381a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff43ca00640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4381099f0 0x7ff4381a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.820+0000 7ff43ca00640 1 -- 172.21.15.97:0/2935101891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4381a6140 con 0x7ff43810fc80 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.821+0000 7ff43ca00640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 0x7ff4381a5a10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff430031040 tx=0x7ff430032bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.821+0000 7ff435600640 1 -- 172.21.15.97:0/2935101891 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430004040 con 0x7ff43810fc80 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.821+0000 7ff435600640 1 -- 172.21.15.97:0/2935101891 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff430004670 con 0x7ff43810fc80 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.821+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4381a63d0 con 0x7ff43810fc80 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.821+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4381acb90 con 0x7ff43810fc80 2024-07-26T07:51:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.821+0000 7ff435600640 1 -- 172.21.15.97:0/2935101891 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430030bb0 con 0x7ff43810fc80 2024-07-26T07:51:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.822+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff438109e00 con 0x7ff43810fc80 2024-07-26T07:51:06.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.825+0000 7ff435600640 1 -- 172.21.15.97:0/2935101891 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7ff4300041e0 con 0x7ff43810fc80 2024-07-26T07:51:06.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.826+0000 7ff435600640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff41c0b3c50 0x7ff41c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:06.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.826+0000 7ff435600640 1 -- 172.21.15.97:0/2935101891 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff4301051b0 con 0x7ff43810fc80 2024-07-26T07:51:06.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.826+0000 7ff437e00640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff41c0b3c50 0x7ff41c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:06.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.827+0000 7ff437e00640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff41c0b3c50 0x7ff41c0b6160 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff428005fd0 tx=0x7ff42800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:06.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:06.828+0000 7ff435600640 1 -- 172.21.15.97:0/2935101891 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff4300cc7d0 con 0x7ff43810fc80 2024-07-26T07:51:07.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.069+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4381a1300 con 0x7ff43810fc80 2024-07-26T07:51:07.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.070+0000 7ff435600640 1 -- 172.21.15.97:0/2935101891 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11990 (secure 0 0 0) 0x7ff4381a1300 con 0x7ff43810fc80 2024-07-26T07:51:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:07.072 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":27,"btime":"2024-07-26T07:48:11:830543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:48:11.504745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:51:07.072 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 27 2024-07-26T07:51:07.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.073+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff41c0b3c50 msgr2=0x7ff41c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:07.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff41c0b3c50 0x7ff41c0b6160 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff428005fd0 tx=0x7ff42800b390 comp rx=0 tx=0).stop 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 msgr2=0x7ff4381a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 0x7ff4381a5a10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff430031040 tx=0x7ff430032bc0 comp rx=0 tx=0).stop 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 shutdown_connections 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff41c0b3c50 0x7ff41c0b6160 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff43810fc80 0x7ff4381a5a10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff43810f330 0x7ff4381a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 --2- 172.21.15.97:0/2935101891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4381099f0 0x7ff4381a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 >> 172.21.15.97:0/2935101891 conn(0x7ff438078ff0 msgr2=0x7ff4380796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 shutdown_connections 2024-07-26T07:51:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:07.074+0000 7ff43d400640 1 -- 172.21.15.97:0/2935101891 wait complete. 2024-07-26T07:51:07.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:51:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:07 smithi097 ceph-mon[32231]: pgmap v490: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:07 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2935101891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:07 smithi115 ceph-mon[35364]: pgmap v490: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:07 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2935101891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:07 smithi181 ceph-mon[35869]: pgmap v490: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:07 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2935101891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:08.287 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:09 smithi097 ceph-mon[32231]: pgmap v491: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:09 smithi115 ceph-mon[35364]: pgmap v491: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:09 smithi181 ceph-mon[35869]: pgmap v491: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:10.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.563+0000 7f3693200640 1 -- 172.21.15.97:0/1755504703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 msgr2=0x7f369410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:10.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.563+0000 7f3693200640 1 --2- 172.21.15.97:0/1755504703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 0x7f369410f740 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f367c009900 tx=0x7f367c02a410 comp rx=0 tx=0).stop 2024-07-26T07:51:10.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.564+0000 7f3693200640 1 -- 172.21.15.97:0/1755504703 shutdown_connections 2024-07-26T07:51:10.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.564+0000 7f3693200640 1 --2- 172.21.15.97:0/1755504703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f369410fc80 0x7f36941164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.564+0000 7f3693200640 1 --2- 172.21.15.97:0/1755504703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 0x7f369410f740 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.564+0000 7f3693200640 1 --2- 172.21.15.97:0/1755504703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36941099f0 0x7f3694109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.564+0000 7f3693200640 1 -- 172.21.15.97:0/1755504703 >> 172.21.15.97:0/1755504703 conn(0x7f3694078ff0 msgr2=0x7f3694107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:10.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.564+0000 7f3693200640 1 -- 172.21.15.97:0/1755504703 shutdown_connections 2024-07-26T07:51:10.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.564+0000 7f3693200640 1 -- 172.21.15.97:0/1755504703 wait complete. 2024-07-26T07:51:10.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.565+0000 7f3693200640 1 Processor -- start 2024-07-26T07:51:10.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.565+0000 7f3693200640 1 -- start start 2024-07-26T07:51:10.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3693200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36941099f0 0x7f36941a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:10.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3693200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 0x7f36941a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:10.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3693200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f369410fc80 0x7f36941a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:10.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3693200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3694119840 con 0x7f369410f330 2024-07-26T07:51:10.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3693200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f36941196c0 con 0x7f369410fc80 2024-07-26T07:51:10.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3693200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f36941199c0 con 0x7f36941099f0 2024-07-26T07:51:10.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3689400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 0x7f36941a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:10.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3689400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 0x7f36941a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52944/0 (socket says 172.21.15.97:52944) 2024-07-26T07:51:10.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.566+0000 7f3689400640 1 -- 172.21.15.97:0/1189079409 learned_addr learned my addr 172.21.15.97:0/1189079409 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:10.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.567+0000 7f3691e00640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36941099f0 0x7f36941a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:10.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.567+0000 7f3692800640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f369410fc80 0x7f36941a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:10.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.567+0000 7f3689400640 1 -- 172.21.15.97:0/1189079409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36941099f0 msgr2=0x7f36941a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:10.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.567+0000 7f3689400640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36941099f0 0x7f36941a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.567+0000 7f3689400640 1 -- 172.21.15.97:0/1189079409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f369410fc80 msgr2=0x7f36941a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:10.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.567+0000 7f3689400640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f369410fc80 0x7f36941a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.567+0000 7f3689400640 1 -- 172.21.15.97:0/1189079409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36941a6280 con 0x7f369410f330 2024-07-26T07:51:10.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.567+0000 7f3692800640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f369410fc80 0x7f36941a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:10.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.568+0000 7f3689400640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 0x7f36941a08b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f367c02a930 tx=0x7f367c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:10.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.568+0000 7f368b400640 1 -- 172.21.15.97:0/1189079409 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c03f070 con 0x7f369410f330 2024-07-26T07:51:10.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.568+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36941a6510 con 0x7f369410f330 2024-07-26T07:51:10.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.568+0000 7f368b400640 1 -- 172.21.15.97:0/1189079409 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f367c003e10 con 0x7f369410f330 2024-07-26T07:51:10.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.568+0000 7f368b400640 1 -- 172.21.15.97:0/1189079409 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c035030 con 0x7f369410f330 2024-07-26T07:51:10.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.568+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36941accd0 con 0x7f369410f330 2024-07-26T07:51:10.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.570+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f369410ac70 con 0x7f369410f330 2024-07-26T07:51:10.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.572+0000 7f368b400640 1 -- 172.21.15.97:0/1189079409 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f367c042070 con 0x7f369410f330 2024-07-26T07:51:10.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.574+0000 7f368b400640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f36780b3bd0 0x7f36780b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:10.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.574+0000 7f368b400640 1 -- 172.21.15.97:0/1189079409 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f367c104460 con 0x7f369410f330 2024-07-26T07:51:10.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.574+0000 7f3691e00640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f36780b3bd0 0x7f36780b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:10.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.575+0000 7f3691e00640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f36780b3bd0 0x7f36780b60e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3684005fd0 tx=0x7f3684009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:10.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.578+0000 7f368b400640 1 -- 172.21.15.97:0/1189079409 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f367c0cb9f0 con 0x7f369410f330 2024-07-26T07:51:10.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.820+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3694111a70 con 0x7f369410f330 2024-07-26T07:51:10.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.822+0000 7f368b400640 1 -- 172.21.15.97:0/1189079409 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11990 (secure 0 0 0) 0x7f367c0cb2f0 con 0x7f369410f330 2024-07-26T07:51:10.823 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:10.823 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":27,"btime":"2024-07-26T07:48:11:830543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:48:11.504745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:51:10.824 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 27 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.824+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f36780b3bd0 msgr2=0x7f36780b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.824+0000 7f3693200640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f36780b3bd0 0x7f36780b60e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3684005fd0 tx=0x7f3684009680 comp rx=0 tx=0).stop 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.824+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 msgr2=0x7f36941a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.824+0000 7f3693200640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 0x7f36941a08b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f367c02a930 tx=0x7f367c003c20 comp rx=0 tx=0).stop 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.825+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 shutdown_connections 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.825+0000 7f3693200640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f36780b3bd0 0x7f36780b60e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.825+0000 7f3693200640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f369410fc80 0x7f36941a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.825+0000 7f3693200640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f369410f330 0x7f36941a08b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.825+0000 7f3693200640 1 --2- 172.21.15.97:0/1189079409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36941099f0 0x7f36941a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:10.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.825+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 >> 172.21.15.97:0/1189079409 conn(0x7f3694078ff0 msgr2=0x7f3694108040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:10.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.825+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 shutdown_connections 2024-07-26T07:51:10.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:10.825+0000 7f3693200640 1 -- 172.21.15.97:0/1189079409 wait complete. 2024-07-26T07:51:11.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:10 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1189079409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:10 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1189079409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:10 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1189079409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:11.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:51:12.000 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:12.150 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:11 smithi097 ceph-mon[32231]: pgmap v492: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:11 smithi115 ceph-mon[35364]: pgmap v492: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:11 smithi181 ceph-mon[35869]: pgmap v492: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:14.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 -- 172.21.15.97:0/249933109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 msgr2=0x7fcbb01059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:14.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 --2- 172.21.15.97:0/249933109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 0x7fcbb01059e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009900 tx=0x7fcba002a410 comp rx=0 tx=0).stop 2024-07-26T07:51:14.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 -- 172.21.15.97:0/249933109 shutdown_connections 2024-07-26T07:51:14.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 --2- 172.21.15.97:0/249933109 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbb010b860 0x7fcbb0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 --2- 172.21.15.97:0/249933109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbb010af10 0x7fcbb010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 --2- 172.21.15.97:0/249933109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 0x7fcbb01059e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 -- 172.21.15.97:0/249933109 >> 172.21.15.97:0/249933109 conn(0x7fcbb0101040 msgr2=0x7fcbb0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:14.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 -- 172.21.15.97:0/249933109 shutdown_connections 2024-07-26T07:51:14.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.013+0000 7fcbaf200640 1 -- 172.21.15.97:0/249933109 wait complete. 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.014+0000 7fcbaf200640 1 Processor -- start 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.014+0000 7fcbaf200640 1 -- start start 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbaf200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 0x7fcbb019be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbaf200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbb010af10 0x7fcbb019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbaf200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbb010b860 0x7fcbb01a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbaf200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb01153c0 con 0x7fcbb01055d0 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbaf200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb0115240 con 0x7fcbb010b860 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbaf200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb0115540 con 0x7fcbb010af10 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbade00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 0x7fcbb019be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:14.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbade00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 0x7fcbb019be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52954/0 (socket says 172.21.15.97:52954) 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcbade00640 1 -- 172.21.15.97:0/3334405290 learned_addr learned my addr 172.21.15.97:0/3334405290 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.015+0000 7fcba5400640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbb010af10 0x7fcbb019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcbae800640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbb010b860 0x7fcbb01a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcbade00640 1 -- 172.21.15.97:0/3334405290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbb010af10 msgr2=0x7fcbb019c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcbade00640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbb010af10 0x7fcbb019c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcbade00640 1 -- 172.21.15.97:0/3334405290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbb010b860 msgr2=0x7fcbb01a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcbade00640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbb010b860 0x7fcbb01a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcbade00640 1 -- 172.21.15.97:0/3334405290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb01a1d20 con 0x7fcbb01055d0 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcba5400640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbb010af10 0x7fcbb019c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcbade00640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 0x7fcbb019be70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009230 tx=0x7fcba0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.016+0000 7fcba7400640 1 -- 172.21.15.97:0/3334405290 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba003f030 con 0x7fcbb01055d0 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.017+0000 7fcba7400640 1 -- 172.21.15.97:0/3334405290 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcba0003a80 con 0x7fcbb01055d0 2024-07-26T07:51:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.017+0000 7fcba7400640 1 -- 172.21.15.97:0/3334405290 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0035070 con 0x7fcbb01055d0 2024-07-26T07:51:14.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.017+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb01a1fb0 con 0x7fcbb01055d0 2024-07-26T07:51:14.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.017+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb01a2490 con 0x7fcbb01055d0 2024-07-26T07:51:14.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.018+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb01059e0 con 0x7fcbb01055d0 2024-07-26T07:51:14.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.020+0000 7fcba7400640 1 -- 172.21.15.97:0/3334405290 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fcba0042030 con 0x7fcbb01055d0 2024-07-26T07:51:14.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.022+0000 7fcba7400640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb940b3c00 0x7fcb940b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:14.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.022+0000 7fcba7400640 1 -- 172.21.15.97:0/3334405290 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fcba0104900 con 0x7fcbb01055d0 2024-07-26T07:51:14.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.022+0000 7fcba5400640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb940b3c00 0x7fcb940b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:14.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.023+0000 7fcba5400640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb940b3c00 0x7fcb940b6110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcbb010ab30 tx=0x7fcb9800a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:14.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.024+0000 7fcba7400640 1 -- 172.21.15.97:0/3334405290 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcba00cbe90 con 0x7fcbb01055d0 2024-07-26T07:51:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:13 smithi115 ceph-mon[35364]: pgmap v493: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:14.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.225+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbb010d050 con 0x7fcbb01055d0 2024-07-26T07:51:14.226 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:13 smithi097 ceph-mon[32231]: pgmap v493: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:14.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.229+0000 7fcba7400640 1 -- 172.21.15.97:0/3334405290 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22853 (secure 0 0 0) 0x7fcba00cb790 con 0x7fcbb01055d0 2024-07-26T07:51:14.231 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:14.231 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":86,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:49:20.276546+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T07:47:40.641206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T07:47:44.357412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:51:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:13 smithi181 ceph-mon[35869]: pgmap v493: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:14.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb940b3c00 msgr2=0x7fcb940b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:14.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb940b3c00 0x7fcb940b6110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcbb010ab30 tx=0x7fcb9800a5c0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 msgr2=0x7fcbb019be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 0x7fcbb019be70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009230 tx=0x7fcba0004270 comp rx=0 tx=0).stop 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 shutdown_connections 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb940b3c00 0x7fcb940b6110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbb010b860 0x7fcbb01a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbb010af10 0x7fcbb019c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 --2- 172.21.15.97:0/3334405290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbb01055d0 0x7fcbb019be70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.232+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 >> 172.21.15.97:0/3334405290 conn(0x7fcbb0101040 msgr2=0x7fcbb0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.233+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 shutdown_connections 2024-07-26T07:51:14.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:14.233+0000 7fcbaf200640 1 -- 172.21.15.97:0/3334405290 wait complete. 2024-07-26T07:51:14.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T07:47:48.482950+0000', 'modified': '2024-07-26T07:48:11.504745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 34583, 'mds_1': 34556, 'mds_2': 34535, 'mds_3': 34517, 'mds_4': 24562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14748': {'gid': 14748, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6839/4174464953', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6838', 'nonce': 4174464953}, {'type': 'v1', 'addr': '172.21.15.97:6839', 'nonce': 4174464953}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14760': {'gid': 14760, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6841/4022202837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6840', 'nonce': 4022202837}, {'type': 'v1', 'addr': '172.21.15.97:6841', 'nonce': 4022202837}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6833/1511551515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6832', 'nonce': 1511551515}, {'type': 'v1', 'addr': '172.21.15.181:6833', 'nonce': 1511551515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6835/4001900111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 4001900111}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 4001900111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6837/3053625422', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 3053625422}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 3053625422}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'i', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.181:6839/4258683328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 4258683328}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 4258683328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34517': {'gid': 34517, 'name': 'k', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.115:6833/1028321962', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6832', 'nonce': 1028321962}, {'type': 'v1', 'addr': '172.21.15.115:6833', 'nonce': 1028321962}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34535': {'gid': 34535, 'name': 'e', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.115:6835/4018170367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6834', 'nonce': 4018170367}, {'type': 'v1', 'addr': '172.21.15.115:6835', 'nonce': 4018170367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34556': {'gid': 34556, 'name': 'h', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.115:6837/3478625369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6836', 'nonce': 3478625369}, {'type': 'v1', 'addr': '172.21.15.115:6837', 'nonce': 3478625369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34583': {'gid': 34583, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.115:6839/1903638498', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6838', 'nonce': 1903638498}, {'type': 'v1', 'addr': '172.21.15.115:6839', 'nonce': 1903638498}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34583, 'qdb_cluster': [24562, 34517, 34556, 34535, 34583]} 2024-07-26T07:51:14.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T07:51:14.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.i status 2024-07-26T07:51:15.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:14 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3334405290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:51:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:14 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3334405290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:51:15.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:14 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3334405290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:51:15.326 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:16.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:15 smithi115 ceph-mon[35364]: pgmap v494: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:15 smithi181 ceph-mon[35869]: pgmap v494: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:15 smithi097 ceph-mon[32231]: pgmap v494: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 -- 172.21.15.97:0/449982753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410b860 msgr2=0x7f18a4112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 --2- 172.21.15.97:0/449982753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410b860 0x7f18a4112090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f189c009ca0 tx=0x7f189c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 -- 172.21.15.97:0/449982753 shutdown_connections 2024-07-26T07:51:17.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 --2- 172.21.15.97:0/449982753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410b860 0x7f18a4112090 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 --2- 172.21.15.97:0/449982753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18a410af10 0x7f18a410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 --2- 172.21.15.97:0/449982753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18a41055d0 0x7f18a41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 -- 172.21.15.97:0/449982753 >> 172.21.15.97:0/449982753 conn(0x7f18a4101040 msgr2=0x7f18a4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:17.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 -- 172.21.15.97:0/449982753 shutdown_connections 2024-07-26T07:51:17.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.527+0000 7f18a3e00640 1 -- 172.21.15.97:0/449982753 wait complete. 2024-07-26T07:51:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.528+0000 7f18a3e00640 1 Processor -- start 2024-07-26T07:51:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.528+0000 7f18a3e00640 1 -- start start 2024-07-26T07:51:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f18a3e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18a41055d0 0x7f18a419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f18a3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410af10 0x7f18a419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f18a3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18a410b860 0x7f18a41a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f18a3e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f18a4115430 con 0x7f18a410af10 2024-07-26T07:51:17.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f18a3e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41152b0 con 0x7f18a41055d0 2024-07-26T07:51:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f18a3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41155b0 con 0x7f18a410b860 2024-07-26T07:51:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f189a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410af10 0x7f18a419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f189a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410af10 0x7f18a419c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41376/0 (socket says 172.21.15.97:41376) 2024-07-26T07:51:17.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.529+0000 7f189a000640 1 -- 172.21.15.97:0/3737032101 learned_addr learned my addr 172.21.15.97:0/3737032101 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f18a2a00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18a41055d0 0x7f18a419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f18a3400640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18a410b860 0x7f18a41a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f189a000640 1 -- 172.21.15.97:0/3737032101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18a410b860 msgr2=0x7f18a41a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f189a000640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18a410b860 0x7f18a41a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f189a000640 1 -- 172.21.15.97:0/3737032101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18a41055d0 msgr2=0x7f18a419be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f189a000640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18a41055d0 0x7f18a419be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f189a000640 1 -- 172.21.15.97:0/3737032101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a41a1d70 con 0x7f18a410af10 2024-07-26T07:51:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f18a2a00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18a41055d0 0x7f18a419be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:17.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f189a000640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410af10 0x7f18a419c3a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f188c00d750 tx=0x7f188c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:17.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.530+0000 7f189be00640 1 -- 172.21.15.97:0/3737032101 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c017070 con 0x7f18a410af10 2024-07-26T07:51:17.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.531+0000 7f189be00640 1 -- 172.21.15.97:0/3737032101 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f188c0054b0 con 0x7f18a410af10 2024-07-26T07:51:17.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.531+0000 7f18a3e00640 1 -- 172.21.15.97:0/3737032101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a41a2060 con 0x7f18a410af10 2024-07-26T07:51:17.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.531+0000 7f189be00640 1 -- 172.21.15.97:0/3737032101 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c004bc0 con 0x7f18a410af10 2024-07-26T07:51:17.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.531+0000 7f18a3e00640 1 -- 172.21.15.97:0/3737032101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a41a87c0 con 0x7f18a410af10 2024-07-26T07:51:17.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.534+0000 7f18b01f9740 1 Processor -- start 2024-07-26T07:51:17.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18b01f9740 1 -- start start 2024-07-26T07:51:17.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f189be00640 1 -- 172.21.15.97:0/3737032101 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f188c009c30 con 0x7f18a410af10 2024-07-26T07:51:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893e0f60 0x561789786760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893b80a0 0x561789786ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893c0280 0x5617897871e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18b01f9740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5617896fc950 con 0x5617893e0f60 2024-07-26T07:51:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18b01f9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5617896ebe50 con 0x5617893b80a0 2024-07-26T07:51:17.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18b01f9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5617896efe90 con 0x5617893c0280 2024-07-26T07:51:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18a3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893e0f60 0x561789786760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18a3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893e0f60 0x561789786760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41390/0 (socket says 172.21.15.97:41390) 2024-07-26T07:51:17.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.535+0000 7f18a3400640 1 -- 172.21.15.97:0/3576834502 learned_addr learned my addr 172.21.15.97:0/3576834502 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f189a000640 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893c0280 0x5617897871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f18a2a00640 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893b80a0 0x561789786ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f18a3400640 1 -- 172.21.15.97:0/3576834502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893c0280 msgr2=0x5617897871e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f18a3400640 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893c0280 0x5617897871e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f18a3400640 1 -- 172.21.15.97:0/3576834502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893b80a0 msgr2=0x561789786ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f18a3400640 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893b80a0 0x561789786ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f18a3400640 1 -- 172.21.15.97:0/3576834502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617896930f0 con 0x5617893e0f60 2024-07-26T07:51:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f18a2a00640 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893b80a0 0x561789786ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.536+0000 7f18a3400640 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893e0f60 0x561789786760 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f189c005e30 tx=0x7f189c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79dca9a12498726e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.537+0000 7f189be00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18880b3bd0 0x7f18880b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.537+0000 7f1893400640 1 -- 172.21.15.97:0/3576834502 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c031980 con 0x5617893e0f60 2024-07-26T07:51:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.537+0000 7f1893400640 1 -- 172.21.15.97:0/3576834502 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f189c036040 con 0x5617893e0f60 2024-07-26T07:51:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.537+0000 7f1893400640 1 -- 172.21.15.97:0/3576834502 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c0043c0 con 0x5617893e0f60 2024-07-26T07:51:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.537+0000 7f189be00640 1 -- 172.21.15.97:0/3737032101 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f188c0e3780 con 0x7f18a410af10 2024-07-26T07:51:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.537+0000 7f18a2a00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18880b3bd0 0x7f18880b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 -- 172.21.15.97:0/3576834502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893e0f60 msgr2=0x561789786760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18a2a00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18880b3bd0 0x7f18880b60e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1894005fd0 tx=0x7f189400b2c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:17.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893e0f60 0x561789786760 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f189c005e30 tx=0x7f189c009a20 comp rx=0 tx=0).stop 2024-07-26T07:51:17.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 -- 172.21.15.97:0/3576834502 shutdown_connections 2024-07-26T07:51:17.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893c0280 0x5617897871e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893b80a0 0x561789786ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 --2- 172.21.15.97:0/3576834502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893e0f60 0x561789786760 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 -- 172.21.15.97:0/3576834502 >> 172.21.15.97:0/3576834502 conn(0x5617893d5350 msgr2=0x561789784650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:17.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 -- 172.21.15.97:0/3576834502 shutdown_connections 2024-07-26T07:51:17.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.538+0000 7f18b01f9740 1 -- 172.21.15.97:0/3576834502 wait complete. 2024-07-26T07:51:17.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.539+0000 7f18b01f9740 1 -- start start 2024-07-26T07:51:17.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.539+0000 7f18b01f9740 1 Processor -- start 2024-07-26T07:51:17.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897922a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897927e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x56178979e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18b01f9740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56178954f5f0 con 0x5617893b80a0 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18b01f9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561789688520 con 0x5617893c0280 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18b01f9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561789794ee0 con 0x5617893e0f60 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18b01f9740 10 client.0 _ll_register_callbacks cb 0x561789697ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18a2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897922a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18a2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897922a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41394/0 (socket says 172.21.15.97:41394) 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18a2a00640 1 -- 172.21.15.97:0/2217886685 learned_addr learned my addr 172.21.15.97:0/2217886685 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f18a3400640 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x56178979e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.540+0000 7f189a000640 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f18a2a00640 1 -- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 msgr2=0x56178979e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:17.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f18a2a00640 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x56178979e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f18a2a00640 1 -- 172.21.15.97:0/2217886685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 msgr2=0x5617897927e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f18a2a00640 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897927e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f18a2a00640 1 -- 172.21.15.97:0/2217886685 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56178944c540 con 0x5617893b80a0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f18a3400640 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x56178979e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f18a2a00640 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897922a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1894012700 tx=0x7f1894004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5856c714f0f6ac48 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f1887e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f1887e00640 1 -- 172.21.15.97:0/2217886685 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894023070 con 0x5617893b80a0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f1887e00640 1 -- 172.21.15.97:0/2217886685 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f1894011ae0 con 0x5617893b80a0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.541+0000 7f18b01f9740 1 -- 172.21.15.97:0/2217886685 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f189c036040 con 0x5617893b80a0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.542+0000 7f1887e00640 1 -- 172.21.15.97:0/2217886685 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f189402f050 con 0x5617893b80a0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.542+0000 7f18b01f9740 10 client.15270 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.542+0000 7f18b01f9740 1 -- 172.21.15.97:0/2217886685 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5617897a0e40 con 0x5617893b80a0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.542+0000 7f1887e00640 1 -- 172.21.15.97:0/2217886685 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f189c036040 con 0x5617893b80a0 2024-07-26T07:51:17.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.542+0000 7f1887e00640 1 -- 172.21.15.97:0/2217886685 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f189400f040 con 0x5617893b80a0 2024-07-26T07:51:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.542+0000 7f18b01f9740 10 client.15270 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.543+0000 7f18b01f9740 10 client.15270 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T07:51:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.543+0000 7f18b01f9740 20 client.15270 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.543+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x56178939f8e0 0x5617897a4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.543+0000 7f18b01f9740 4 client.15270 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:51:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.543+0000 7f18b01f9740 1 -- 172.21.15.97:0/2217886685 --> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617896fe8a0 con 0x56178939f8e0 2024-07-26T07:51:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.543+0000 7f189a000640 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x56178939f8e0 0x5617897a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.544+0000 7f189a000640 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x56178939f8e0 0x5617897a4900 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=43769b6831d06fdc server_cookie=ce8f28e0775016c3 in_seq=0 out_seq=0 2024-07-26T07:51:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.544+0000 7f1887e00640 10 client.15270 ms_handle_connect on v2:172.21.15.181:6838/4258683328 2024-07-26T07:51:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.546+0000 7f1887e00640 1 -- 172.21.15.97:0/2217886685 <== mds.4 v2:172.21.15.181:6838/4258683328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5617896fe8a0 con 0x56178939f8e0 2024-07-26T07:51:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.546+0000 7f1887e00640 10 client.15270 handle_command_reply: tid=0 2024-07-26T07:51:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 client.15270 shutdown 2024-07-26T07:51:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 -- 172.21.15.97:0/2217886685 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 msgr2=0x5617897922a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897922a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1894012700 tx=0x7f1894004730 comp rx=0 tx=0).stop 2024-07-26T07:51:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 -- 172.21.15.97:0/2217886685 shutdown_connections 2024-07-26T07:51:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x56178939f8e0 0x5617897a4900 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x56178979e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2217886685 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897922a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f1887e00640 0 client.15270 ms_handle_reset on v2:172.21.15.181:6838/4258683328 2024-07-26T07:51:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 -- 172.21.15.97:0/2217886685 >> 172.21.15.97:0/2217886685 conn(0x5617893d5350 msgr2=0x561789789940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 -- 172.21.15.97:0/2217886685 shutdown_connections 2024-07-26T07:51:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 1 -- 172.21.15.97:0/2217886685 wait complete. 2024-07-26T07:51:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.547+0000 7f18b01f9740 20 client.15270 trim_cache size 0 max 16384 2024-07-26T07:51:17.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.567+0000 7f18b01f9740 1 Processor -- start 2024-07-26T07:51:17.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.567+0000 7f18b01f9740 1 -- start start 2024-07-26T07:51:17.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.567+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893b80a0 0x56178979e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.567+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893c0280 0x56178979e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.567+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893e0f60 0x561789790920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.567+0000 7f18b01f9740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5617897a2c40 con 0x5617893c0280 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.567+0000 7f18b01f9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5617897a3310 con 0x5617893e0f60 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.567+0000 7f18b01f9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5617897a39e0 con 0x5617893b80a0 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893c0280 0x56178979e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893c0280 0x56178979e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41410/0 (socket says 172.21.15.97:41410) 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 -- 172.21.15.97:0/3555043108 learned_addr learned my addr 172.21.15.97:0/3555043108 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f18a2a00640 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893b80a0 0x56178979e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f18a3400640 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893e0f60 0x561789790920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 -- 172.21.15.97:0/3555043108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893b80a0 msgr2=0x56178979e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:17.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893b80a0 0x56178979e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 -- 172.21.15.97:0/3555043108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893e0f60 msgr2=0x561789790920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893e0f60 0x561789790920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 -- 172.21.15.97:0/3555043108 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56178977f7d0 con 0x5617893c0280 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.568+0000 7f189a000640 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893c0280 0x56178979e6b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f188c0aa5a0 tx=0x7f188c0b3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b5f66451afade1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.569+0000 7f1892a00640 1 -- 172.21.15.97:0/3555043108 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c020040 con 0x5617893c0280 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.569+0000 7f1892a00640 1 -- 172.21.15.97:0/3555043108 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f188c0aa610 con 0x5617893c0280 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.569+0000 7f1892a00640 1 -- 172.21.15.97:0/3555043108 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c0a68a0 con 0x5617893c0280 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.569+0000 7f18b01f9740 1 -- 172.21.15.97:0/3555043108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893c0280 msgr2=0x56178979e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.569+0000 7f18b01f9740 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893c0280 0x56178979e6b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f188c0aa5a0 tx=0x7f188c0b3040 comp rx=0 tx=0).stop 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.570+0000 7f18b01f9740 1 -- 172.21.15.97:0/3555043108 shutdown_connections 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.570+0000 7f18b01f9740 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893e0f60 0x561789790920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.570+0000 7f18b01f9740 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893c0280 0x56178979e6b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.570+0000 7f18b01f9740 1 --2- 172.21.15.97:0/3555043108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893b80a0 0x56178979e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.570+0000 7f18b01f9740 1 -- 172.21.15.97:0/3555043108 >> 172.21.15.97:0/3555043108 conn(0x5617893d5350 msgr2=0x56178978d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:17.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.570+0000 7f18b01f9740 1 -- 172.21.15.97:0/3555043108 shutdown_connections 2024-07-26T07:51:17.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.570+0000 7f18b01f9740 1 -- 172.21.15.97:0/3555043108 wait complete. 2024-07-26T07:51:17.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.570+0000 7f18b01f9740 1 -- start start 2024-07-26T07:51:17.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18b01f9740 1 Processor -- start 2024-07-26T07:51:17.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897a7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18b01f9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x5617897a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18b01f9740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5617897a4780 con 0x5617893b80a0 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18b01f9740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5617897a40b0 con 0x5617893c0280 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18b01f9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5617897a4e50 con 0x5617893e0f60 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18b01f9740 10 client.0 _ll_register_callbacks cb 0x7f189400be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18a2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18a2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41418/0 (socket says 172.21.15.97:41418) 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.571+0000 7f18a2a00640 1 -- 172.21.15.97:0/2414465631 learned_addr learned my addr 172.21.15.97:0/2414465631 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f189a000640 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897a7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f18a3400640 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x5617897a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f18a2a00640 1 -- 172.21.15.97:0/2414465631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 msgr2=0x5617897a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:17.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f18a2a00640 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x5617897a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f18a2a00640 1 -- 172.21.15.97:0/2414465631 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 msgr2=0x5617897a7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f18a2a00640 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897a7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f18a2a00640 1 -- 172.21.15.97:0/2414465631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56178980bb90 con 0x5617893b80a0 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f189a000640 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897a7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f18a2a00640 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897a7a30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f189400b980 tx=0x7f18940275c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a4b7b779a20f65 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.572+0000 7f1887e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f1887e00640 1 -- 172.21.15.97:0/2414465631 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894037020 con 0x5617893b80a0 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f1887e00640 1 -- 172.21.15.97:0/2414465631 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f1894004240 con 0x5617893b80a0 2024-07-26T07:51:17.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f18b01f9740 1 -- 172.21.15.97:0/2414465631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f188c0aa610 con 0x5617893b80a0 2024-07-26T07:51:17.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f1887e00640 1 -- 172.21.15.97:0/2414465631 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1894023070 con 0x5617893b80a0 2024-07-26T07:51:17.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f18b01f9740 10 client.15276 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:17.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f18b01f9740 1 -- 172.21.15.97:0/2414465631 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56178980cdf0 con 0x5617893b80a0 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f1887e00640 1 -- 172.21.15.97:0/2414465631 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f188c0aa610 con 0x5617893b80a0 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f1887e00640 1 -- 172.21.15.97:0/2414465631 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f189401f650 con 0x5617893b80a0 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.573+0000 7f18b01f9740 10 client.15276 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.574+0000 7f18b01f9740 10 client.15276 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.574+0000 7f18b01f9740 20 client.15276 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.574+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561789374c90 0x5617898107b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.574+0000 7f18b01f9740 4 client.15276 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.574+0000 7f18b01f9740 1 -- 172.21.15.97:0/2414465631 --> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] -- command(tid 0: {"prefix": "status"}) -- 0x5617896fa340 con 0x561789374c90 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.574+0000 7f189a000640 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561789374c90 0x5617898107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.575+0000 7f189a000640 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561789374c90 0x5617898107b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2f99b8c9ebff7c84 server_cookie=2ef622d086f7753d in_seq=0 out_seq=0 2024-07-26T07:51:17.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.575+0000 7f1887e00640 10 client.15276 ms_handle_connect on v2:172.21.15.181:6838/4258683328 2024-07-26T07:51:17.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f1887e00640 1 -- 172.21.15.97:0/2414465631 <== mds.4 v2:172.21.15.181:6838/4258683328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5617896fa340 con 0x561789374c90 2024-07-26T07:51:17.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f1887e00640 10 client.15276 handle_command_reply: tid=0 2024-07-26T07:51:17.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 client.15276 shutdown 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 -- 172.21.15.97:0/2414465631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 msgr2=0x5617897a7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897a7a30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f189400b980 tx=0x7f18940275c0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 -- 172.21.15.97:0/2414465631 shutdown_connections 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x561789374c90 0x5617898107b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617893e0f60 0x5617897a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5617893c0280 0x5617897a7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 --2- 172.21.15.97:0/2414465631 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5617893b80a0 0x5617897a7a30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.576+0000 7f18b01f9740 1 -- 172.21.15.97:0/2414465631 >> 172.21.15.97:0/2414465631 conn(0x5617893d5350 msgr2=0x561789789860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.577+0000 7f1887e00640 0 client.15276 ms_handle_reset on v2:172.21.15.181:6838/4258683328 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.577+0000 7f18b01f9740 1 -- 172.21.15.97:0/2414465631 shutdown_connections 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.577+0000 7f18b01f9740 1 -- 172.21.15.97:0/2414465631 wait complete. 2024-07-26T07:51:17.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.577+0000 7f18b01f9740 20 client.15276 trim_cache size 0 max 16384 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 4, 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24562, 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 190.778424546, 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:51:17.580 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 242.167937696 2024-07-26T07:51:17.581 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:51:17.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.577+0000 7f18a3e00640 1 -- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18880b3bd0 msgr2=0x7f18880b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.577+0000 7f18a3e00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18880b3bd0 0x7f18880b60e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1894005fd0 tx=0x7f189400b2c0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.577+0000 7f18a3e00640 1 -- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410af10 msgr2=0x7f18a419c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:17.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.577+0000 7f18a3e00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410af10 0x7f18a419c3a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f188c00d750 tx=0x7f188c00dc10 comp rx=0 tx=0).stop 2024-07-26T07:51:17.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.578+0000 7f18a3e00640 1 -- 172.21.15.97:0/3737032101 shutdown_connections 2024-07-26T07:51:17.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.578+0000 7f18a3e00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f18880b3bd0 0x7f18880b60e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.578+0000 7f18a3e00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18a410b860 0x7f18a41a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.578+0000 7f18a3e00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f18a410af10 0x7f18a419c3a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.578+0000 7f18a3e00640 1 --2- 172.21.15.97:0/3737032101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18a41055d0 0x7f18a419be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:17.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.578+0000 7f18a3e00640 1 -- 172.21.15.97:0/3737032101 >> 172.21.15.97:0/3737032101 conn(0x7f18a4101040 msgr2=0x7f18a4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:17.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.578+0000 7f18a3e00640 1 -- 172.21.15.97:0/3737032101 shutdown_connections 2024-07-26T07:51:17.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:17.578+0000 7f18a3e00640 1 -- 172.21.15.97:0/3737032101 wait complete. 2024-07-26T07:51:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:17 smithi115 ceph-mon[35364]: pgmap v495: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:17 smithi181 ceph-mon[35869]: pgmap v495: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:17 smithi097 ceph-mon[32231]: pgmap v495: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:18.323 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.k status 2024-07-26T07:51:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:19.987 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:19 smithi097 ceph-mon[32231]: pgmap v496: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:19 smithi115 ceph-mon[35364]: pgmap v496: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:19 smithi181 ceph-mon[35869]: pgmap v496: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.821+0000 7f8961a00640 1 -- 172.21.15.97:0/662880041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 msgr2=0x7f895c1059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.821+0000 7f8961a00640 1 --2- 172.21.15.97:0/662880041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 0x7f895c1059e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f894c009900 tx=0x7f894c02a410 comp rx=0 tx=0).stop 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.821+0000 7f8961a00640 1 -- 172.21.15.97:0/662880041 shutdown_connections 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.821+0000 7f8961a00640 1 --2- 172.21.15.97:0/662880041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f895c10b860 0x7f895c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.821+0000 7f8961a00640 1 --2- 172.21.15.97:0/662880041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f895c10af10 0x7f895c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.821+0000 7f8961a00640 1 --2- 172.21.15.97:0/662880041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 0x7f895c1059e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.821+0000 7f8961a00640 1 -- 172.21.15.97:0/662880041 >> 172.21.15.97:0/662880041 conn(0x7f895c101040 msgr2=0x7f895c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.821+0000 7f8961a00640 1 -- 172.21.15.97:0/662880041 shutdown_connections 2024-07-26T07:51:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.822+0000 7f8961a00640 1 -- 172.21.15.97:0/662880041 wait complete. 2024-07-26T07:51:20.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.823+0000 7f8961a00640 1 Processor -- start 2024-07-26T07:51:20.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.823+0000 7f8961a00640 1 -- start start 2024-07-26T07:51:20.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.823+0000 7f8961a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 0x7f895c19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.823+0000 7f8961a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f895c10af10 0x7f895c19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.823+0000 7f8961a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f895c10b860 0x7f895c1a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.823+0000 7f8961a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1153e0 con 0x7f895c1055d0 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.823+0000 7f8961a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f895c115260 con 0x7f895c10af10 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.823+0000 7f8961a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f895c115560 con 0x7f895c10b860 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f895be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 0x7f895c19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f895be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 0x7f895c19bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41432/0 (socket says 172.21.15.97:41432) 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f895be00640 1 -- 172.21.15.97:0/2434371683 learned_addr learned my addr 172.21.15.97:0/2434371683 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f8961000640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f895c10b860 0x7f895c1a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f8953e00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f895c10af10 0x7f895c19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f895be00640 1 -- 172.21.15.97:0/2434371683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f895c10b860 msgr2=0x7f895c1a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f895be00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f895c10b860 0x7f895c1a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f895be00640 1 -- 172.21.15.97:0/2434371683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f895c10af10 msgr2=0x7f895c19c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f895be00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f895c10af10 0x7f895c19c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f895be00640 1 -- 172.21.15.97:0/2434371683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c1a1ea0 con 0x7f895c1055d0 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f8961000640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f895c10b860 0x7f895c1a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.824+0000 7f8953e00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f895c10af10 0x7f895c19c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.825+0000 7f895be00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 0x7f895c19bf30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f894c009230 tx=0x7f894c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.825+0000 7f8959600640 1 -- 172.21.15.97:0/2434371683 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c03f030 con 0x7f895c1055d0 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.825+0000 7f8961a00640 1 -- 172.21.15.97:0/2434371683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c1a2130 con 0x7f895c1055d0 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.825+0000 7f8959600640 1 -- 172.21.15.97:0/2434371683 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f894c003a80 con 0x7f895c1055d0 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.825+0000 7f8959600640 1 -- 172.21.15.97:0/2434371683 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c035070 con 0x7f895c1055d0 2024-07-26T07:51:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.825+0000 7f8961a00640 1 -- 172.21.15.97:0/2434371683 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c1a88f0 con 0x7f895c1055d0 2024-07-26T07:51:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.828+0000 7f8969cd2740 1 Processor -- start 2024-07-26T07:51:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f8969cd2740 1 -- start start 2024-07-26T07:51:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f8959600640 1 -- 172.21.15.97:0/2434371683 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f894c042030 con 0x7f895c1055d0 2024-07-26T07:51:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f442bf60 0x5580f47d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f440b280 0x5580f47d2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f8969cd2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5580f4747970 con 0x5580f44030a0 2024-07-26T07:51:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f8969cd2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5580f4736e70 con 0x5580f440b280 2024-07-26T07:51:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f8969cd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5580f473aeb0 con 0x5580f442bf60 2024-07-26T07:51:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.829+0000 7f895be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f895be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41434/0 (socket says 172.21.15.97:41434) 2024-07-26T07:51:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f895be00640 1 -- 172.21.15.97:0/3767794549 learned_addr learned my addr 172.21.15.97:0/3767794549 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f8953e00640 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f440b280 0x5580f47d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f8961000640 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f442bf60 0x5580f47d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f895be00640 1 -- 172.21.15.97:0/3767794549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f442bf60 msgr2=0x5580f47d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:20.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f895be00640 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f442bf60 0x5580f47d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f895be00640 1 -- 172.21.15.97:0/3767794549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f440b280 msgr2=0x5580f47d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:20.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f895be00640 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f440b280 0x5580f47d2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.830+0000 7f895be00640 1 -- 172.21.15.97:0/3767794549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580f46de110 con 0x5580f44030a0 2024-07-26T07:51:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.831+0000 7f895be00640 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d1cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f894c10c5c0 tx=0x7f894c10c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29d97cb463dc7bb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.831+0000 7f8959600640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f89400b3bd0 0x7f89400b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.831+0000 7f8951600640 1 -- 172.21.15.97:0/3767794549 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c0c79a0 con 0x5580f44030a0 2024-07-26T07:51:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.831+0000 7f8959600640 1 -- 172.21.15.97:0/2434371683 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f894c1044c0 con 0x7f895c1055d0 2024-07-26T07:51:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.831+0000 7f8961000640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f89400b3bd0 0x7f89400b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.831+0000 7f8951600640 1 -- 172.21.15.97:0/3767794549 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f894c111030 con 0x5580f44030a0 2024-07-26T07:51:20.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.831+0000 7f8951600640 1 -- 172.21.15.97:0/3767794549 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c0caac0 con 0x5580f44030a0 2024-07-26T07:51:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.832+0000 7f8969cd2740 1 -- 172.21.15.97:0/3767794549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 msgr2=0x5580f47d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.832+0000 7f8961000640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f89400b3bd0 0x7f89400b60e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8954005fd0 tx=0x7f89540095a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.832+0000 7f8969cd2740 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d1cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f894c10c5c0 tx=0x7f894c10c6b0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 -- 172.21.15.97:0/3767794549 shutdown_connections 2024-07-26T07:51:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f440b280 0x5580f47d2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d1cc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 --2- 172.21.15.97:0/3767794549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f442bf60 0x5580f47d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 -- 172.21.15.97:0/3767794549 >> 172.21.15.97:0/3767794549 conn(0x5580f4420350 msgr2=0x5580f47cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 -- 172.21.15.97:0/3767794549 shutdown_connections 2024-07-26T07:51:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 -- 172.21.15.97:0/3767794549 wait complete. 2024-07-26T07:51:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 -- start start 2024-07-26T07:51:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.833+0000 7f8969cd2740 1 Processor -- start 2024-07-26T07:51:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47dcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 0x5580f47dd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47e8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f8969cd2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5580f459a5f0 con 0x5580f44030a0 2024-07-26T07:51:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f8969cd2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5580f46d3540 con 0x5580f442bf60 2024-07-26T07:51:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f8969cd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5580f47dfd20 con 0x5580f440b280 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f8969cd2740 10 client.0 _ll_register_callbacks cb 0x5580f46e2f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f895be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47dcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f895be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47dcfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41436/0 (socket says 172.21.15.97:41436) 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.834+0000 7f895be00640 1 -- 172.21.15.97:0/2446872121 learned_addr learned my addr 172.21.15.97:0/2446872121 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f8953e00640 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 0x5580f47dd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f8961000640 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47e8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f895be00640 1 -- 172.21.15.97:0/2446872121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 msgr2=0x5580f47dd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f895be00640 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 0x5580f47dd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f895be00640 1 -- 172.21.15.97:0/2446872121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 msgr2=0x5580f47e8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f895be00640 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47e8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f895be00640 1 -- 172.21.15.97:0/2446872121 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580f4497540 con 0x5580f44030a0 2024-07-26T07:51:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f895be00640 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47dcfe0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f894c000c00 tx=0x7f894c104740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65e56f95dca4d69 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f894a000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f894a000640 1 -- 172.21.15.97:0/2446872121 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c049070 con 0x5580f44030a0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f894a000640 1 -- 172.21.15.97:0/2446872121 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f894c102010 con 0x5580f44030a0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.835+0000 7f8969cd2740 1 -- 172.21.15.97:0/2446872121 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f894c111030 con 0x5580f44030a0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f894a000640 1 -- 172.21.15.97:0/2446872121 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f894c0caca0 con 0x5580f44030a0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f8969cd2740 10 client.15288 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f8969cd2740 1 -- 172.21.15.97:0/2446872121 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5580f47ebb20 con 0x5580f44030a0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f894a000640 1 -- 172.21.15.97:0/2446872121 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f894c111030 con 0x5580f44030a0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f894a000640 1 -- 172.21.15.97:0/2446872121 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f894c10cd20 con 0x5580f44030a0 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f8969cd2740 10 client.15288 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f8969cd2740 10 client.15288 resolve_mds: resolved name 'k' to daemon mds.k 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f8969cd2740 20 client.15288 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:20.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x5580f43ea8e0 0x5580f47ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f8969cd2740 4 client.15288 mds_command: new command op to 34517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:51:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.836+0000 7f8969cd2740 1 -- 172.21.15.97:0/2446872121 --> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580f47498c0 con 0x5580f43ea8e0 2024-07-26T07:51:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.837+0000 7f8953e00640 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x5580f43ea8e0 0x5580f47ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.838+0000 7f8953e00640 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x5580f43ea8e0 0x5580f47ef2c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6026e41ab1f92a08 server_cookie=878bf53b4cf46aa2 in_seq=0 out_seq=0 2024-07-26T07:51:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.838+0000 7f894a000640 10 client.15288 ms_handle_connect on v2:172.21.15.115:6832/1028321962 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.840+0000 7f894a000640 1 -- 172.21.15.97:0/2446872121 <== mds.3 v2:172.21.15.115:6832/1028321962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5580f47498c0 con 0x5580f43ea8e0 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.840+0000 7f894a000640 10 client.15288 handle_command_reply: tid=0 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.840+0000 7f8969cd2740 1 client.15288 shutdown 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.840+0000 7f8969cd2740 1 -- 172.21.15.97:0/2446872121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 msgr2=0x5580f47dcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.840+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47dcfe0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f894c000c00 tx=0x7f894c104740 comp rx=0 tx=0).stop 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.840+0000 7f8969cd2740 1 -- 172.21.15.97:0/2446872121 shutdown_connections 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x5580f43ea8e0 0x5580f47ef2c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47e8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 0x5580f47dd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2446872121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47dcfe0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f8969cd2740 1 -- 172.21.15.97:0/2446872121 >> 172.21.15.97:0/2446872121 conn(0x5580f4420350 msgr2=0x5580f47d4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:20.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f894a000640 0 client.15288 ms_handle_reset on v2:172.21.15.115:6832/1028321962 2024-07-26T07:51:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f8969cd2740 1 -- 172.21.15.97:0/2446872121 shutdown_connections 2024-07-26T07:51:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f8969cd2740 1 -- 172.21.15.97:0/2446872121 wait complete. 2024-07-26T07:51:20.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.841+0000 7f8969cd2740 20 client.15288 trim_cache size 0 max 16384 2024-07-26T07:51:20.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.860+0000 7f8969cd2740 1 Processor -- start 2024-07-26T07:51:20.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f8969cd2740 1 -- start start 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 0x5580f47ece70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47db860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f8969cd2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5580f47ed600 con 0x5580f44030a0 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f8969cd2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5580f47edcd0 con 0x5580f442bf60 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f8969cd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5580f47ee3a0 con 0x5580f440b280 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f895be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f895be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41448/0 (socket says 172.21.15.97:41448) 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.861+0000 7f895be00640 1 -- 172.21.15.97:0/3918336025 learned_addr learned my addr 172.21.15.97:0/3918336025 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f8953e00640 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 0x5580f47ece70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f8961000640 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47db860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f895be00640 1 -- 172.21.15.97:0/3918336025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 msgr2=0x5580f47ece70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:20.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f895be00640 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 0x5580f47ece70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f895be00640 1 -- 172.21.15.97:0/3918336025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 msgr2=0x5580f47db860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:20.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f895be00640 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47db860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f895be00640 1 -- 172.21.15.97:0/3918336025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580f47ca7f0 con 0x5580f44030a0 2024-07-26T07:51:20.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f8961000640 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47db860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:20.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f895be00640 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d78a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f894c0cb2e0 tx=0x7f894c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb55c33d89ea894 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:20.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f8950c00640 1 -- 172.21.15.97:0/3918336025 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c049070 con 0x5580f44030a0 2024-07-26T07:51:20.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f8950c00640 1 -- 172.21.15.97:0/3918336025 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f894c111070 con 0x5580f44030a0 2024-07-26T07:51:20.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.862+0000 7f8950c00640 1 -- 172.21.15.97:0/3918336025 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c0cae90 con 0x5580f44030a0 2024-07-26T07:51:20.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.863+0000 7f8969cd2740 1 -- 172.21.15.97:0/3918336025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 msgr2=0x5580f47d78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.863+0000 7f8969cd2740 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d78a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f894c0cb2e0 tx=0x7f894c02a410 comp rx=0 tx=0).stop 2024-07-26T07:51:20.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.863+0000 7f8969cd2740 1 -- 172.21.15.97:0/3918336025 shutdown_connections 2024-07-26T07:51:20.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.863+0000 7f8969cd2740 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47db860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.863+0000 7f8969cd2740 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f440b280 0x5580f47ece70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.863+0000 7f8969cd2740 1 --2- 172.21.15.97:0/3918336025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f44030a0 0x5580f47d78a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.863+0000 7f8969cd2740 1 -- 172.21.15.97:0/3918336025 >> 172.21.15.97:0/3918336025 conn(0x5580f4420350 msgr2=0x5580f47dcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:20.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.864+0000 7f8969cd2740 1 -- 172.21.15.97:0/3918336025 shutdown_connections 2024-07-26T07:51:20.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.864+0000 7f8969cd2740 1 -- 172.21.15.97:0/3918336025 wait complete. 2024-07-26T07:51:20.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.864+0000 7f8969cd2740 1 -- start start 2024-07-26T07:51:20.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.864+0000 7f8969cd2740 1 Processor -- start 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f44030a0 0x5580f47f0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f440b280 0x5580f47f09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8969cd2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8969cd2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5580f47ef140 con 0x5580f440b280 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8969cd2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5580f47eea70 con 0x5580f442bf60 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8969cd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5580f47ef810 con 0x5580f44030a0 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8969cd2740 10 client.0 _ll_register_callbacks cb 0x7f894c03b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8953e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f440b280 0x5580f47f09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8953e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f440b280 0x5580f47f09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41458/0 (socket says 172.21.15.97:41458) 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8953e00640 1 -- 172.21.15.97:0/2897585765 learned_addr learned my addr 172.21.15.97:0/2897585765 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f895be00640 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f44030a0 0x5580f47f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.865+0000 7f8961000640 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.866+0000 7f8953e00640 1 -- 172.21.15.97:0/2897585765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f44030a0 msgr2=0x5580f47f0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.866+0000 7f8953e00640 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f44030a0 0x5580f47f0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.866+0000 7f8953e00640 1 -- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 msgr2=0x5580f47f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.866+0000 7f8953e00640 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.866+0000 7f8953e00640 1 -- 172.21.15.97:0/2897585765 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580f47f5f10 con 0x5580f440b280 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.866+0000 7f8961000640 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.866+0000 7f8953e00640 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f440b280 0x5580f47f09e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8944004700 tx=0x7f8944013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65e90b2db64d91f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.866+0000 7f894a000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.867+0000 7f894a000640 1 -- 172.21.15.97:0/2897585765 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894401d620 con 0x5580f440b280 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.867+0000 7f8969cd2740 1 -- 172.21.15.97:0/2897585765 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f894c111070 con 0x5580f440b280 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.867+0000 7f894a000640 1 -- 172.21.15.97:0/2897585765 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f894401d7c0 con 0x5580f440b280 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.867+0000 7f894a000640 1 -- 172.21.15.97:0/2897585765 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f894401d9c0 con 0x5580f440b280 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.867+0000 7f8969cd2740 10 client.15294 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:20.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.867+0000 7f8969cd2740 1 -- 172.21.15.97:0/2897585765 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5580f47f72a0 con 0x5580f440b280 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.867+0000 7f894a000640 1 -- 172.21.15.97:0/2897585765 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f894c111070 con 0x5580f440b280 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.868+0000 7f894a000640 1 -- 172.21.15.97:0/2897585765 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f8944011ce0 con 0x5580f440b280 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.868+0000 7f8969cd2740 10 client.15294 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.868+0000 7f8969cd2740 10 client.15294 resolve_mds: resolved name 'k' to daemon mds.k 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.868+0000 7f8969cd2740 20 client.15294 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.868+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x5580f43bfc90 0x5580f485ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.868+0000 7f8969cd2740 4 client.15294 mds_command: new command op to 34517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.868+0000 7f8969cd2740 1 -- 172.21.15.97:0/2897585765 --> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] -- command(tid 0: {"prefix": "status"}) -- 0x5580f4745360 con 0x5580f43bfc90 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.868+0000 7f895be00640 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x5580f43bfc90 0x5580f485ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.869+0000 7f895be00640 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x5580f43bfc90 0x5580f485ad20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=49096237e9fa5db4 server_cookie=1481d90bdf9a3828 in_seq=0 out_seq=0 2024-07-26T07:51:20.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.869+0000 7f894a000640 10 client.15294 ms_handle_connect on v2:172.21.15.115:6832/1028321962 2024-07-26T07:51:20.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f894a000640 1 -- 172.21.15.97:0/2897585765 <== mds.3 v2:172.21.15.115:6832/1028321962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5580f4745360 con 0x5580f43bfc90 2024-07-26T07:51:20.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f894a000640 10 client.15294 handle_command_reply: tid=0 2024-07-26T07:51:20.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 client.15294 shutdown 2024-07-26T07:51:20.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 -- 172.21.15.97:0/2897585765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f440b280 msgr2=0x5580f47f09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f440b280 0x5580f47f09e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8944004700 tx=0x7f8944013830 comp rx=0 tx=0).stop 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 -- 172.21.15.97:0/2897585765 shutdown_connections 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x5580f43bfc90 0x5580f485ad20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5580f442bf60 0x5580f47f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5580f440b280 0x5580f47f09e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 --2- 172.21.15.97:0/2897585765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5580f44030a0 0x5580f47f0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 -- 172.21.15.97:0/2897585765 >> 172.21.15.97:0/2897585765 conn(0x5580f4420350 msgr2=0x5580f47dcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f894a000640 0 client.15294 ms_handle_reset on v2:172.21.15.115:6832/1028321962 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.870+0000 7f8969cd2740 1 -- 172.21.15.97:0/2897585765 shutdown_connections 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.871+0000 7f8969cd2740 1 -- 172.21.15.97:0/2897585765 wait complete. 2024-07-26T07:51:20.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.871+0000 7f8969cd2740 20 client.15294 trim_cache size 0 max 16384 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 3, 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34517, 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 195.09581379900001, 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 272.68873514500001 2024-07-26T07:51:20.874 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:51:20.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.871+0000 7f8961a00640 1 -- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f89400b3bd0 msgr2=0x7f89400b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.871+0000 7f8961a00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f89400b3bd0 0x7f89400b60e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8954005fd0 tx=0x7f89540095a0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.871+0000 7f8961a00640 1 -- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 msgr2=0x7f895c19bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.871+0000 7f8961a00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 0x7f895c19bf30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f894c009230 tx=0x7f894c004270 comp rx=0 tx=0).stop 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.872+0000 7f8961a00640 1 -- 172.21.15.97:0/2434371683 shutdown_connections 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.872+0000 7f8961a00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f89400b3bd0 0x7f89400b60e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.872+0000 7f8961a00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f895c10b860 0x7f895c1a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.872+0000 7f8961a00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f895c10af10 0x7f895c19c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.872+0000 7f8961a00640 1 --2- 172.21.15.97:0/2434371683 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f895c1055d0 0x7f895c19bf30 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.872+0000 7f8961a00640 1 -- 172.21.15.97:0/2434371683 >> 172.21.15.97:0/2434371683 conn(0x7f895c101040 msgr2=0x7f895c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.872+0000 7f8961a00640 1 -- 172.21.15.97:0/2434371683 shutdown_connections 2024-07-26T07:51:20.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:20.872+0000 7f8961a00640 1 -- 172.21.15.97:0/2434371683 wait complete. 2024-07-26T07:51:21.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.e status 2024-07-26T07:51:22.098 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:21 smithi115 ceph-mon[35364]: pgmap v497: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:21 smithi181 ceph-mon[35869]: pgmap v497: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:22.244 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:21 smithi097 ceph-mon[32231]: pgmap v497: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 -- 172.21.15.97:0/3263971955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b010fc80 msgr2=0x7f45b01164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 --2- 172.21.15.97:0/3263971955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b010fc80 0x7f45b01164b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009ca0 tx=0x7f45a802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 -- 172.21.15.97:0/3263971955 shutdown_connections 2024-07-26T07:51:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 --2- 172.21.15.97:0/3263971955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b010fc80 0x7f45b01164b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 --2- 172.21.15.97:0/3263971955 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b010f330 0x7f45b010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 --2- 172.21.15.97:0/3263971955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b01099f0 0x7f45b0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 -- 172.21.15.97:0/3263971955 >> 172.21.15.97:0/3263971955 conn(0x7f45b0078ff0 msgr2=0x7f45b0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:24.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 -- 172.21.15.97:0/3263971955 shutdown_connections 2024-07-26T07:51:24.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.100+0000 7f45b6000640 1 -- 172.21.15.97:0/3263971955 wait complete. 2024-07-26T07:51:24.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.101+0000 7f45b6000640 1 Processor -- start 2024-07-26T07:51:24.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b6000640 1 -- start start 2024-07-26T07:51:24.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b01099f0 0x7f45b01a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b6000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b010f330 0x7f45b01a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b6000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b010fc80 0x7f45b01a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b6000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f45b0119840 con 0x7f45b01099f0 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b6000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f45b01196c0 con 0x7f45b010f330 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b6000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f45b01199c0 con 0x7f45b010fc80 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b01099f0 0x7f45b01a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b01099f0 0x7f45b01a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41470/0 (socket says 172.21.15.97:41470) 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.102+0000 7f45b4c00640 1 -- 172.21.15.97:0/1279272120 learned_addr learned my addr 172.21.15.97:0/1279272120 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45b5600640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b010fc80 0x7f45b01a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45afe00640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b010f330 0x7f45b01a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45b4c00640 1 -- 172.21.15.97:0/1279272120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b010fc80 msgr2=0x7f45b01a5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45b4c00640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b010fc80 0x7f45b01a5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45b4c00640 1 -- 172.21.15.97:0/1279272120 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b010f330 msgr2=0x7f45b01a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45b4c00640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b010f330 0x7f45b01a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45b4c00640 1 -- 172.21.15.97:0/1279272120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b01a6140 con 0x7f45b01099f0 2024-07-26T07:51:24.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45b4c00640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b01099f0 0x7f45b01a01d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f45a400d770 tx=0x7f45a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:24.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.103+0000 7f45ad600640 1 -- 172.21.15.97:0/1279272120 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4017070 con 0x7f45b01099f0 2024-07-26T07:51:24.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.104+0000 7f45ad600640 1 -- 172.21.15.97:0/1279272120 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45a40054b0 con 0x7f45b01099f0 2024-07-26T07:51:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.104+0000 7f45b6000640 1 -- 172.21.15.97:0/1279272120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b01a6430 con 0x7f45b01099f0 2024-07-26T07:51:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.104+0000 7f45ad600640 1 -- 172.21.15.97:0/1279272120 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4004c60 con 0x7f45b01099f0 2024-07-26T07:51:24.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.104+0000 7f45b6000640 1 -- 172.21.15.97:0/1279272120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b01acb90 con 0x7f45b01099f0 2024-07-26T07:51:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.107+0000 7f45be241740 1 Processor -- start 2024-07-26T07:51:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.107+0000 7f45be241740 1 -- start start 2024-07-26T07:51:24.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45be241740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a033371760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45be241740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a033371ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45be241740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 0x55a0333721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45be241740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a0332e7950 con 0x55a032fa30a0 2024-07-26T07:51:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45be241740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a0332d6e50 con 0x55a032fcbf60 2024-07-26T07:51:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45be241740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a0332dae90 con 0x55a032fab280 2024-07-26T07:51:24.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45ad600640 1 -- 172.21.15.97:0/1279272120 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f45a4004e00 con 0x7f45b01099f0 2024-07-26T07:51:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45b4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a033371ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45b4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a033371ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41474/0 (socket says 172.21.15.97:41474) 2024-07-26T07:51:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45b4c00640 1 -- 172.21.15.97:0/4247536246 learned_addr learned my addr 172.21.15.97:0/4247536246 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45b5600640 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a033371760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.108+0000 7f45afe00640 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 0x55a0333721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.109+0000 7f45b4c00640 1 -- 172.21.15.97:0/4247536246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 msgr2=0x55a0333721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.109+0000 7f45b4c00640 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 0x55a0333721e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.109+0000 7f45b4c00640 1 -- 172.21.15.97:0/4247536246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 msgr2=0x55a033371760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:24.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.109+0000 7f45b4c00640 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a033371760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.109+0000 7f45b4c00640 1 -- 172.21.15.97:0/4247536246 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a03327e0f0 con 0x55a032fa30a0 2024-07-26T07:51:24.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.109+0000 7f45b4c00640 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a033371ca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f45a40e9c20 tx=0x7f45a40eb670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e667bcddb008ac server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:24.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.110+0000 7f45a1c00640 1 -- 172.21.15.97:0/4247536246 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a40f2070 con 0x55a032fa30a0 2024-07-26T07:51:24.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.110+0000 7f45a1c00640 1 -- 172.21.15.97:0/4247536246 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45a40e3ce0 con 0x55a032fa30a0 2024-07-26T07:51:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.110+0000 7f45a1c00640 1 -- 172.21.15.97:0/4247536246 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a40ee620 con 0x55a032fa30a0 2024-07-26T07:51:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45ad600640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f45980b3cd0 0x7f45980b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 -- 172.21.15.97:0/4247536246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 msgr2=0x55a033371ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a033371ca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f45a40e9c20 tx=0x7f45a40eb670 comp rx=0 tx=0).stop 2024-07-26T07:51:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45ad600640 1 -- 172.21.15.97:0/1279272120 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f45a40e3730 con 0x7f45b01099f0 2024-07-26T07:51:24.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45b5600640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f45980b3cd0 0x7f45980b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 -- 172.21.15.97:0/4247536246 shutdown_connections 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 0x55a0333721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a033371ca0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 --2- 172.21.15.97:0/4247536246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a033371760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 -- 172.21.15.97:0/4247536246 >> 172.21.15.97:0/4247536246 conn(0x55a032fc0350 msgr2=0x55a03336f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 -- 172.21.15.97:0/4247536246 shutdown_connections 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.111+0000 7f45be241740 1 -- 172.21.15.97:0/4247536246 wait complete. 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.112+0000 7f45b5600640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f45980b3cd0 0x7f45980b61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45a8005e30 tx=0x7f45a8005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:24.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.112+0000 7f45be241740 1 -- start start 2024-07-26T07:51:24.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.112+0000 7f45be241740 1 Processor -- start 2024-07-26T07:51:24.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45be241740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 0x55a03337cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45be241740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a03337d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45be241740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a033389260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45be241740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a03313a5f0 con 0x55a032fab280 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45be241740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a033273520 con 0x55a032fcbf60 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45be241740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a03337ff40 con 0x55a032fa30a0 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45be241740 10 client.0 _ll_register_callbacks cb 0x55a033282ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a03337d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a03337d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41486/0 (socket says 172.21.15.97:41486) 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45afe00640 1 -- 172.21.15.97:0/772733522 learned_addr learned my addr 172.21.15.97:0/772733522 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45b4c00640 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 0x55a03337cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.113+0000 7f45b5600640 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a033389260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.114+0000 7f45afe00640 1 -- 172.21.15.97:0/772733522 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 msgr2=0x55a03337cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.114+0000 7f45afe00640 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 0x55a03337cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.114+0000 7f45afe00640 1 -- 172.21.15.97:0/772733522 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 msgr2=0x55a033389260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:24.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.114+0000 7f45afe00640 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a033389260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.114+0000 7f45afe00640 1 -- 172.21.15.97:0/772733522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a033037540 con 0x55a032fab280 2024-07-26T07:51:24.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.114+0000 7f45afe00640 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a03337d5d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f459c00be80 tx=0x7f459c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7acb3053787878d6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:24.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.114+0000 7f4596000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:24.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.114+0000 7f4596000640 1 -- 172.21.15.97:0/772733522 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c01f060 con 0x55a032fab280 2024-07-26T07:51:24.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.115+0000 7f4596000640 1 -- 172.21.15.97:0/772733522 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f459c0108a0 con 0x55a032fab280 2024-07-26T07:51:24.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.115+0000 7f45be241740 1 -- 172.21.15.97:0/772733522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a40e3ce0 con 0x55a032fab280 2024-07-26T07:51:24.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.115+0000 7f4596000640 1 -- 172.21.15.97:0/772733522 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f459c010aa0 con 0x55a032fab280 2024-07-26T07:51:24.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.115+0000 7f45be241740 10 client.15306 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:24.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.115+0000 7f45be241740 1 -- 172.21.15.97:0/772733522 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a03338bee0 con 0x55a032fab280 2024-07-26T07:51:24.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.115+0000 7f4596000640 1 -- 172.21.15.97:0/772733522 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45a40e3ce0 con 0x55a032fab280 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.115+0000 7f4596000640 1 -- 172.21.15.97:0/772733522 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f459c023030 con 0x55a032fab280 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.116+0000 7f45be241740 10 client.15306 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.116+0000 7f45be241740 10 client.15306 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.116+0000 7f45be241740 20 client.15306 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.116+0000 7f45be241740 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x55a032f8a8e0 0x55a03338f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.116+0000 7f45be241740 4 client.15306 mds_command: new command op to 34535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.116+0000 7f45be241740 1 -- 172.21.15.97:0/772733522 --> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0332e98a0 con 0x55a032f8a8e0 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.116+0000 7f45b4c00640 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x55a032f8a8e0 0x55a03338f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.117+0000 7f45b4c00640 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x55a032f8a8e0 0x55a03338f990 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=24ecd4d75d603452 server_cookie=63307f977d511648 in_seq=0 out_seq=0 2024-07-26T07:51:24.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.117+0000 7f4596000640 10 client.15306 ms_handle_connect on v2:172.21.15.115:6834/4018170367 2024-07-26T07:51:24.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.119+0000 7f4596000640 1 -- 172.21.15.97:0/772733522 <== mds.2 v2:172.21.15.115:6834/4018170367 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a0332e98a0 con 0x55a032f8a8e0 2024-07-26T07:51:24.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.119+0000 7f4596000640 10 client.15306 handle_command_reply: tid=0 2024-07-26T07:51:24.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 client.15306 shutdown 2024-07-26T07:51:24.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 -- 172.21.15.97:0/772733522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 msgr2=0x55a03337d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a03337d5d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f459c00be80 tx=0x7f459c012820 comp rx=0 tx=0).stop 2024-07-26T07:51:24.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 -- 172.21.15.97:0/772733522 shutdown_connections 2024-07-26T07:51:24.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x55a032f8a8e0 0x55a03338f990 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a033389260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a03337d5d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 --2- 172.21.15.97:0/772733522 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 0x55a03337cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f4596000640 0 client.15306 ms_handle_reset on v2:172.21.15.115:6834/4018170367 2024-07-26T07:51:24.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 -- 172.21.15.97:0/772733522 >> 172.21.15.97:0/772733522 conn(0x55a032fc0350 msgr2=0x55a0333748f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:24.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 -- 172.21.15.97:0/772733522 shutdown_connections 2024-07-26T07:51:24.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 1 -- 172.21.15.97:0/772733522 wait complete. 2024-07-26T07:51:24.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.120+0000 7f45be241740 20 client.15306 trim_cache size 0 max 16384 2024-07-26T07:51:24.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.139+0000 7f45be241740 1 Processor -- start 2024-07-26T07:51:24.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45be241740 1 -- start start 2024-07-26T07:51:24.141 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:23 smithi097 ceph-mon[32231]: pgmap v498: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45be241740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a03338f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45be241740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 0x55a033374f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45be241740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a0333891d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45be241740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a03338dcd0 con 0x55a032fa30a0 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45be241740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a03338e3a0 con 0x55a032fcbf60 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45be241740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a03338ea70 con 0x55a032fab280 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45b4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a03338f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45b4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a03338f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41496/0 (socket says 172.21.15.97:41496) 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.140+0000 7f45b4c00640 1 -- 172.21.15.97:0/2230899666 learned_addr learned my addr 172.21.15.97:0/2230899666 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:24.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.141+0000 7f45afe00640 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 0x55a033374f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.141+0000 7f45b5600640 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a0333891d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.141+0000 7f45b4c00640 1 -- 172.21.15.97:0/2230899666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 msgr2=0x55a033374f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:24.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.141+0000 7f45b4c00640 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 0x55a033374f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.141+0000 7f45b4c00640 1 -- 172.21.15.97:0/2230899666 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 msgr2=0x55a0333891d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:24.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.141+0000 7f45b4c00640 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a0333891d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.141+0000 7f45b4c00640 1 -- 172.21.15.97:0/2230899666 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a03336a7d0 con 0x55a032fa30a0 2024-07-26T07:51:24.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.141+0000 7f45b4c00640 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a03338f990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f45a40ebb70 tx=0x7f45a40b3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7372e38429a40ed2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:24.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.142+0000 7f45a1200640 1 -- 172.21.15.97:0/2230899666 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4020070 con 0x55a032fa30a0 2024-07-26T07:51:24.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.142+0000 7f45a1200640 1 -- 172.21.15.97:0/2230899666 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45a40e9300 con 0x55a032fa30a0 2024-07-26T07:51:24.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.142+0000 7f45a1200640 1 -- 172.21.15.97:0/2230899666 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a40f2040 con 0x55a032fa30a0 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 -- 172.21.15.97:0/2230899666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 msgr2=0x55a03338f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a03338f990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f45a40ebb70 tx=0x7f45a40b3040 comp rx=0 tx=0).stop 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 -- 172.21.15.97:0/2230899666 shutdown_connections 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a0333891d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fab280 0x55a033374f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 --2- 172.21.15.97:0/2230899666 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fa30a0 0x55a03338f990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 -- 172.21.15.97:0/2230899666 >> 172.21.15.97:0/2230899666 conn(0x55a032fc0350 msgr2=0x55a033374860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 -- 172.21.15.97:0/2230899666 shutdown_connections 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 -- 172.21.15.97:0/2230899666 wait complete. 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.143+0000 7f45be241740 1 -- start start 2024-07-26T07:51:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.144+0000 7f45be241740 1 Processor -- start 2024-07-26T07:51:24.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.144+0000 7f45be241740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 0x55a033390d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.144+0000 7f45be241740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a033391350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.144+0000 7f45be241740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a0333960a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.144+0000 7f45be241740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a03338f810 con 0x55a032fab280 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.144+0000 7f45be241740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a03338f140 con 0x55a032fcbf60 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.144+0000 7f45be241740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a03338fee0 con 0x55a032fa30a0 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.144+0000 7f45be241740 10 client.0 _ll_register_callbacks cb 0x7f459c00ba90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a033391350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a033391350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41502/0 (socket says 172.21.15.97:41502) 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45afe00640 1 -- 172.21.15.97:0/814089762 learned_addr learned my addr 172.21.15.97:0/814089762 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45b5600640 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a0333960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45b4c00640 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 0x55a033390d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45afe00640 1 -- 172.21.15.97:0/814089762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 msgr2=0x55a033390d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45afe00640 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 0x55a033390d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45afe00640 1 -- 172.21.15.97:0/814089762 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 msgr2=0x55a0333960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:24.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45afe00640 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a0333960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.145+0000 7f45afe00640 1 -- 172.21.15.97:0/814089762 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a033396710 con 0x55a032fab280 2024-07-26T07:51:24.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.146+0000 7f45afe00640 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a033391350 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f459c00be50 tx=0x7f459c010950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9738e759d59c82 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:24.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.146+0000 7f4596000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:24.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.146+0000 7f4596000640 1 -- 172.21.15.97:0/814089762 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c031060 con 0x55a032fab280 2024-07-26T07:51:24.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.146+0000 7f4596000640 1 -- 172.21.15.97:0/814089762 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f459c005450 con 0x55a032fab280 2024-07-26T07:51:24.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.146+0000 7f45be241740 1 -- 172.21.15.97:0/814089762 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a40e9300 con 0x55a032fab280 2024-07-26T07:51:24.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.146+0000 7f4596000640 1 -- 172.21.15.97:0/814089762 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f459c01f030 con 0x55a032fab280 2024-07-26T07:51:24.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f45be241740 10 client.15312 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:24.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f45be241740 1 -- 172.21.15.97:0/814089762 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a033397ad0 con 0x55a032fab280 2024-07-26T07:51:24.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f4596000640 1 -- 172.21.15.97:0/814089762 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45a40e9300 con 0x55a032fab280 2024-07-26T07:51:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f4596000640 1 -- 172.21.15.97:0/814089762 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f459c023070 con 0x55a032fab280 2024-07-26T07:51:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f45be241740 10 client.15312 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f45be241740 10 client.15312 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T07:51:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f45be241740 20 client.15312 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f45be241740 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x55a032f5fc90 0x55a0333fb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f45be241740 4 client.15312 mds_command: new command op to 34535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:51:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.147+0000 7f45be241740 1 -- 172.21.15.97:0/814089762 --> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] -- command(tid 0: {"prefix": "status"}) -- 0x55a0332e5340 con 0x55a032f5fc90 2024-07-26T07:51:24.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.148+0000 7f45b4c00640 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x55a032f5fc90 0x55a0333fb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:24.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.149+0000 7f45b4c00640 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x55a032f5fc90 0x55a0333fb540 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9a28fc9d894b9ff server_cookie=532a2cc09f18cd0c in_seq=0 out_seq=0 2024-07-26T07:51:24.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.149+0000 7f4596000640 10 client.15312 ms_handle_connect on v2:172.21.15.115:6834/4018170367 2024-07-26T07:51:24.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f4596000640 1 -- 172.21.15.97:0/814089762 <== mds.2 v2:172.21.15.115:6834/4018170367 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a0332e5340 con 0x55a032f5fc90 2024-07-26T07:51:24.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f4596000640 10 client.15312 handle_command_reply: tid=0 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 client.15312 shutdown 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 -- 172.21.15.97:0/814089762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 msgr2=0x55a033391350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a033391350 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f459c00be50 tx=0x7f459c010950 comp rx=0 tx=0).stop 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 -- 172.21.15.97:0/814089762 shutdown_connections 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x55a032f5fc90 0x55a0333fb540 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a032fcbf60 0x55a0333960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a032fab280 0x55a033391350 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 --2- 172.21.15.97:0/814089762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a032fa30a0 0x55a033390d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 -- 172.21.15.97:0/814089762 >> 172.21.15.97:0/814089762 conn(0x55a032fc0350 msgr2=0x55a03337cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f4596000640 0 client.15312 ms_handle_reset on v2:172.21.15.115:6834/4018170367 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 -- 172.21.15.97:0/814089762 shutdown_connections 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.150+0000 7f45be241740 1 -- 172.21.15.97:0/814089762 wait complete. 2024-07-26T07:51:24.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.151+0000 7f45be241740 20 client.15312 trim_cache size 0 max 16384 2024-07-26T07:51:24.153 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:51:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:51:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 2, 2024-07-26T07:51:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34535, 2024-07-26T07:51:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:51:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:51:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:51:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 199.392687193, 2024-07-26T07:51:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:51:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:51:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:51:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 263.39507227799999 2024-07-26T07:51:24.154 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:51:24.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.151+0000 7f45b6000640 1 -- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f45980b3cd0 msgr2=0x7f45980b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.151+0000 7f45b6000640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f45980b3cd0 0x7f45980b61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45a8005e30 tx=0x7f45a8005c50 comp rx=0 tx=0).stop 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.151+0000 7f45b6000640 1 -- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b01099f0 msgr2=0x7f45b01a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.151+0000 7f45b6000640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b01099f0 0x7f45b01a01d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f45a400d770 tx=0x7f45a400dc30 comp rx=0 tx=0).stop 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.152+0000 7f45b6000640 1 -- 172.21.15.97:0/1279272120 shutdown_connections 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.152+0000 7f45b6000640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f45980b3cd0 0x7f45980b61e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.152+0000 7f45b6000640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b010fc80 0x7f45b01a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.152+0000 7f45b6000640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f45b010f330 0x7f45b01a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.152+0000 7f45b6000640 1 --2- 172.21.15.97:0/1279272120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45b01099f0 0x7f45b01a01d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.152+0000 7f45b6000640 1 -- 172.21.15.97:0/1279272120 >> 172.21.15.97:0/1279272120 conn(0x7f45b0078ff0 msgr2=0x7f45b0079670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.152+0000 7f45b6000640 1 -- 172.21.15.97:0/1279272120 shutdown_connections 2024-07-26T07:51:24.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:24.152+0000 7f45b6000640 1 -- 172.21.15.97:0/1279272120 wait complete. 2024-07-26T07:51:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:23 smithi115 ceph-mon[35364]: pgmap v498: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:23 smithi181 ceph-mon[35869]: pgmap v498: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:24.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.h status 2024-07-26T07:51:25.390 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:26.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:25 smithi115 ceph-mon[35364]: pgmap v499: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:25 smithi181 ceph-mon[35869]: pgmap v499: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:25 smithi097 ceph-mon[32231]: pgmap v499: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:27.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.431+0000 7f100f000640 1 -- 172.21.15.97:0/2955819181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10081099f0 msgr2=0x7f1008109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.431+0000 7f100f000640 1 --2- 172.21.15.97:0/2955819181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10081099f0 0x7f1008109e00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009900 tx=0x7f0ffc02a410 comp rx=0 tx=0).stop 2024-07-26T07:51:27.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.431+0000 7f100f000640 1 -- 172.21.15.97:0/2955819181 shutdown_connections 2024-07-26T07:51:27.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.431+0000 7f100f000640 1 --2- 172.21.15.97:0/2955819181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f100810fc80 0x7f10081164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.431+0000 7f100f000640 1 --2- 172.21.15.97:0/2955819181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100810f330 0x7f100810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.431+0000 7f100f000640 1 --2- 172.21.15.97:0/2955819181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f10081099f0 0x7f1008109e00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.432+0000 7f100f000640 1 -- 172.21.15.97:0/2955819181 >> 172.21.15.97:0/2955819181 conn(0x7f1008078ff0 msgr2=0x7f1008107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:27.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.432+0000 7f100f000640 1 -- 172.21.15.97:0/2955819181 shutdown_connections 2024-07-26T07:51:27.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.432+0000 7f100f000640 1 -- 172.21.15.97:0/2955819181 wait complete. 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.433+0000 7f100f000640 1 Processor -- start 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.433+0000 7f100f000640 1 -- start start 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.433+0000 7f100f000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10081099f0 0x7f10081a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.433+0000 7f100f000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f100810f330 0x7f10081a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.433+0000 7f100f000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f100810fc80 0x7f10081a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.433+0000 7f100f000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1008119880 con 0x7f100810fc80 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.433+0000 7f100f000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1008119700 con 0x7f100810f330 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.433+0000 7f100f000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1008119a00 con 0x7f10081099f0 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f100810fc80 0x7f10081a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f100810fc80 0x7f10081a5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58030/0 (socket says 172.21.15.97:58030) 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100e600640 1 -- 172.21.15.97:0/163492889 learned_addr learned my addr 172.21.15.97:0/163492889 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:27.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100d200640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f100810f330 0x7f10081a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100dc00640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10081099f0 0x7f10081a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100e600640 1 -- 172.21.15.97:0/163492889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10081099f0 msgr2=0x7f10081a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100e600640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10081099f0 0x7f10081a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100e600640 1 -- 172.21.15.97:0/163492889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f100810f330 msgr2=0x7f10081a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100e600640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f100810f330 0x7f10081a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.434+0000 7f100e600640 1 -- 172.21.15.97:0/163492889 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10081a6200 con 0x7f100810fc80 2024-07-26T07:51:27.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.435+0000 7f100e600640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f100810fc80 0x7f10081a5b90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f100000cce0 tx=0x7f100000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:27.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.435+0000 7f1006000640 1 -- 172.21.15.97:0/163492889 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100000f450 con 0x7f100810fc80 2024-07-26T07:51:27.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.435+0000 7f1006000640 1 -- 172.21.15.97:0/163492889 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f100000fa80 con 0x7f100810fc80 2024-07-26T07:51:27.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.435+0000 7f100f000640 1 -- 172.21.15.97:0/163492889 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10081a64f0 con 0x7f100810fc80 2024-07-26T07:51:27.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.435+0000 7f1006000640 1 -- 172.21.15.97:0/163492889 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000018ad0 con 0x7f100810fc80 2024-07-26T07:51:27.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.435+0000 7f100f000640 1 -- 172.21.15.97:0/163492889 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10081acd10 con 0x7f100810fc80 2024-07-26T07:51:27.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.438+0000 7f10171e1740 1 Processor -- start 2024-07-26T07:51:27.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.439+0000 7f1006000640 1 -- 172.21.15.97:0/163492889 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f100000f5f0 con 0x7f100810fc80 2024-07-26T07:51:27.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.439+0000 7f10171e1740 1 -- start start 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f10171e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915a8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f10171e1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911da0a0 0x558e915a8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f10171e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915a91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f10171e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558e9151e950 con 0x558e91202f60 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f10171e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558e9150de50 con 0x558e911e2280 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f10171e1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558e91511e90 con 0x558e911da0a0 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f100d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915a8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f100d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915a8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58034/0 (socket says 172.21.15.97:58034) 2024-07-26T07:51:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f100d200640 1 -- 172.21.15.97:0/121910494 learned_addr learned my addr 172.21.15.97:0/121910494 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.440+0000 7f100dc00640 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911da0a0 0x558e915a8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f100d200640 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915a91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f100d200640 1 -- 172.21.15.97:0/121910494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911da0a0 msgr2=0x558e915a8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f100d200640 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911da0a0 0x558e915a8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f100d200640 1 -- 172.21.15.97:0/121910494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 msgr2=0x558e915a91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f100d200640 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915a91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f100d200640 1 -- 172.21.15.97:0/121910494 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e914b50f0 con 0x558e91202f60 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f100d200640 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915a8760 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4006aa0 tx=0x7f0ff400b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0c5c88aceffdd3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:27.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f1006000640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ff00b3c80 0x7f0ff00b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.441+0000 7f0ffaa00640 1 -- 172.21.15.97:0/121910494 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff401a070 con 0x558e91202f60 2024-07-26T07:51:27.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.442+0000 7f1006000640 1 -- 172.21.15.97:0/163492889 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f10000e4470 con 0x7f100810fc80 2024-07-26T07:51:27.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.442+0000 7f0ffaa00640 1 -- 172.21.15.97:0/121910494 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ff4010040 con 0x558e91202f60 2024-07-26T07:51:27.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.442+0000 7f100dc00640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ff00b3c80 0x7f0ff00b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.442+0000 7f0ffaa00640 1 -- 172.21.15.97:0/121910494 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4013570 con 0x558e91202f60 2024-07-26T07:51:27.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 -- 172.21.15.97:0/121910494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 msgr2=0x558e915a8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f100dc00640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ff00b3c80 0x7f0ff00b6190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc006010 tx=0x7f0ffc009560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915a8760 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4006aa0 tx=0x7f0ff400b400 comp rx=0 tx=0).stop 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 -- 172.21.15.97:0/121910494 shutdown_connections 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911da0a0 0x558e915a8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 --2- 172.21.15.97:0/121910494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915a8760 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 -- 172.21.15.97:0/121910494 >> 172.21.15.97:0/121910494 conn(0x558e911f7350 msgr2=0x558e915a6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 -- 172.21.15.97:0/121910494 shutdown_connections 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.443+0000 7f10171e1740 1 -- 172.21.15.97:0/121910494 wait complete. 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 1 -- start start 2024-07-26T07:51:27.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 1 Processor -- start 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915b4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 0x558e915c0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558e913715f0 con 0x558e911da0a0 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558e914aa520 con 0x558e911e2280 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558e915b6f40 con 0x558e91202f60 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.444+0000 7f10171e1740 10 client.0 _ll_register_callbacks cb 0x558e914b9ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58040/0 (socket says 172.21.15.97:58040) 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100dc00640 1 -- 172.21.15.97:0/2507241575 learned_addr learned my addr 172.21.15.97:0/2507241575 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100d200640 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100e600640 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 0x558e915c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100dc00640 1 -- 172.21.15.97:0/2507241575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 msgr2=0x558e915c0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:27.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100dc00640 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 0x558e915c0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100dc00640 1 -- 172.21.15.97:0/2507241575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 msgr2=0x558e915b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:27.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100dc00640 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.445+0000 7f100dc00640 1 -- 172.21.15.97:0/2507241575 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e9126e540 con 0x558e911da0a0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.446+0000 7f100dc00640 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b42e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc033000 tx=0x7f0ffc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb5f19a253c0378 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.446+0000 7f0fef400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.446+0000 7f0fef400640 1 -- 172.21.15.97:0/2507241575 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc04d070 con 0x558e911da0a0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.446+0000 7f0fef400640 1 -- 172.21.15.97:0/2507241575 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f0ffc049650 con 0x558e911da0a0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.446+0000 7f10171e1740 1 -- 172.21.15.97:0/2507241575 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ff4010040 con 0x558e911da0a0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.446+0000 7f0fef400640 1 -- 172.21.15.97:0/2507241575 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ffc059050 con 0x558e911da0a0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.446+0000 7f10171e1740 10 client.15324 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.446+0000 7f10171e1740 1 -- 172.21.15.97:0/2507241575 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558e915c2f20 con 0x558e911da0a0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.447+0000 7f0fef400640 1 -- 172.21.15.97:0/2507241575 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ff4010040 con 0x558e911da0a0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.447+0000 7f0fef400640 1 -- 172.21.15.97:0/2507241575 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0ffc03c790 con 0x558e911da0a0 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.447+0000 7f10171e1740 10 client.15324 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.447+0000 7f10171e1740 10 client.15324 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T07:51:27.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.447+0000 7f10171e1740 20 client.15324 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:27.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.447+0000 7f10171e1740 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x558e911c18e0 0x558e915c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.447+0000 7f10171e1740 4 client.15324 mds_command: new command op to 34556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:51:27.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.447+0000 7f10171e1740 1 -- 172.21.15.97:0/2507241575 --> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e915208a0 con 0x558e911c18e0 2024-07-26T07:51:27.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.448+0000 7f100d200640 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x558e911c18e0 0x558e915c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.449+0000 7f100d200640 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x558e911c18e0 0x558e915c6980 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2542669fbfb5f21f server_cookie=c9f0890475a8486e in_seq=0 out_seq=0 2024-07-26T07:51:27.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.449+0000 7f0fef400640 10 client.15324 ms_handle_connect on v2:172.21.15.115:6836/3478625369 2024-07-26T07:51:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.451+0000 7f0fef400640 1 -- 172.21.15.97:0/2507241575 <== mds.1 v2:172.21.15.115:6836/3478625369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558e915208a0 con 0x558e911c18e0 2024-07-26T07:51:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.451+0000 7f0fef400640 10 client.15324 handle_command_reply: tid=0 2024-07-26T07:51:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.451+0000 7f10171e1740 1 client.15324 shutdown 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.451+0000 7f10171e1740 1 -- 172.21.15.97:0/2507241575 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 msgr2=0x558e915b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.451+0000 7f10171e1740 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b42e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc033000 tx=0x7f0ffc009260 comp rx=0 tx=0).stop 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.451+0000 7f10171e1740 1 -- 172.21.15.97:0/2507241575 shutdown_connections 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f10171e1740 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x558e911c18e0 0x558e915c6980 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f10171e1740 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 0x558e915c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f10171e1740 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f10171e1740 1 --2- 172.21.15.97:0/2507241575 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b42e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f0fef400640 0 client.15324 ms_handle_reset on v2:172.21.15.115:6836/3478625369 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f10171e1740 1 -- 172.21.15.97:0/2507241575 >> 172.21.15.97:0/2507241575 conn(0x558e911f7350 msgr2=0x558e915ab940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f10171e1740 1 -- 172.21.15.97:0/2507241575 shutdown_connections 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f10171e1740 1 -- 172.21.15.97:0/2507241575 wait complete. 2024-07-26T07:51:27.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.452+0000 7f10171e1740 20 client.15324 trim_cache size 0 max 16384 2024-07-26T07:51:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.471+0000 7f10171e1740 1 Processor -- start 2024-07-26T07:51:27.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.471+0000 7f10171e1740 1 -- start start 2024-07-26T07:51:27.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f10171e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f10171e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f10171e1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 0x558e915c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f10171e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558e915c4cc0 con 0x558e911da0a0 2024-07-26T07:51:27.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f10171e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558e915c5390 con 0x558e911e2280 2024-07-26T07:51:27.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f10171e1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558e915c5a60 con 0x558e91202f60 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f100dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f100dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58054/0 (socket says 172.21.15.97:58054) 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f100dc00640 1 -- 172.21.15.97:0/19268118 learned_addr learned my addr 172.21.15.97:0/19268118 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f100e600640 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 0x558e915c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f100d200640 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f100dc00640 1 -- 172.21.15.97:0/19268118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 msgr2=0x558e915c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.472+0000 7f100dc00640 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 0x558e915c06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.473+0000 7f100dc00640 1 -- 172.21.15.97:0/19268118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 msgr2=0x558e915c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.473+0000 7f100dc00640 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.473+0000 7f100dc00640 1 -- 172.21.15.97:0/19268118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e915af970 con 0x558e911da0a0 2024-07-26T07:51:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.473+0000 7f100d200640 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:27.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.473+0000 7f100dc00640 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b35d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc03cb00 tx=0x7f0ffc003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90d6108d6897934 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:27.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.473+0000 7f0ffa000640 1 -- 172.21.15.97:0/19268118 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc04d070 con 0x558e911da0a0 2024-07-26T07:51:27.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.473+0000 7f0ffa000640 1 -- 172.21.15.97:0/19268118 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ffc03c8d0 con 0x558e911da0a0 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.473+0000 7f0ffa000640 1 -- 172.21.15.97:0/19268118 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc035050 con 0x558e911da0a0 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 -- 172.21.15.97:0/19268118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 msgr2=0x558e915b35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b35d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc03cb00 tx=0x7f0ffc003a90 comp rx=0 tx=0).stop 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 -- 172.21.15.97:0/19268118 shutdown_connections 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e91202f60 0x558e915c06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911e2280 0x558e915c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 --2- 172.21.15.97:0/19268118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e911da0a0 0x558e915b35d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 -- 172.21.15.97:0/19268118 >> 172.21.15.97:0/19268118 conn(0x558e911f7350 msgr2=0x558e915b2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 -- 172.21.15.97:0/19268118 shutdown_connections 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.474+0000 7f10171e1740 1 -- 172.21.15.97:0/19268118 wait complete. 2024-07-26T07:51:27.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.475+0000 7f10171e1740 1 -- start start 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.475+0000 7f10171e1740 1 Processor -- start 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f10171e1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911da0a0 0x558e915c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f10171e1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911e2280 0x558e915c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f10171e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915c8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f10171e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558e915c6800 con 0x558e91202f60 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f10171e1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558e915c6130 con 0x558e911da0a0 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f10171e1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558e915c6ed0 con 0x558e911e2280 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f10171e1740 10 client.0 _ll_register_callbacks cb 0x7f0ffc0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58068/0 (socket says 172.21.15.97:58068) 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100e600640 1 -- 172.21.15.97:0/1474489625 learned_addr learned my addr 172.21.15.97:0/1474489625 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100dc00640 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911da0a0 0x558e915c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100e600640 1 -- 172.21.15.97:0/1474489625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911e2280 msgr2=0x558e915c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100d200640 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911e2280 0x558e915c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100e600640 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911e2280 0x558e915c80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100e600640 1 -- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911da0a0 msgr2=0x558e915c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100e600640 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911da0a0 0x558e915c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100e600640 1 -- 172.21.15.97:0/1474489625 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e9162dc10 con 0x558e91202f60 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.476+0000 7f100dc00640 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911da0a0 0x558e915c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f100e600640 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915c8600 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f10000a73e0 tx=0x7f100000f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408dbc633061f06 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f0fef400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:27.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f0fef400640 1 -- 172.21.15.97:0/1474489625 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10000c4070 con 0x558e91202f60 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f0fef400640 1 -- 172.21.15.97:0/1474489625 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f10000ab5c0 con 0x558e91202f60 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f10171e1740 1 -- 172.21.15.97:0/1474489625 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ffc03c8d0 con 0x558e91202f60 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f0fef400640 1 -- 172.21.15.97:0/1474489625 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f10000af960 con 0x558e91202f60 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f10171e1740 10 client.15330 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f10171e1740 1 -- 172.21.15.97:0/1474489625 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558e9162eda0 con 0x558e91202f60 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.477+0000 7f0fef400640 1 -- 172.21.15.97:0/1474489625 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ffc03c8d0 con 0x558e91202f60 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.478+0000 7f0fef400640 1 -- 172.21.15.97:0/1474489625 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f10000cc2f0 con 0x558e91202f60 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.478+0000 7f10171e1740 10 client.15330 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.478+0000 7f10171e1740 10 client.15330 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.478+0000 7f10171e1740 20 client.15330 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.478+0000 7f10171e1740 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x558e91196c90 0x558e91632660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.478+0000 7f10171e1740 4 client.15330 mds_command: new command op to 34556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:51:27.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.478+0000 7f10171e1740 1 -- 172.21.15.97:0/1474489625 --> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] -- command(tid 0: {"prefix": "status"}) -- 0x558e9151c340 con 0x558e91196c90 2024-07-26T07:51:27.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.479+0000 7f100dc00640 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x558e91196c90 0x558e91632660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.479+0000 7f100dc00640 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x558e91196c90 0x558e91632660 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e719ed80920b27d5 server_cookie=a757386601153853 in_seq=0 out_seq=0 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.479+0000 7f0fef400640 10 client.15330 ms_handle_connect on v2:172.21.15.115:6836/3478625369 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f0fef400640 1 -- 172.21.15.97:0/1474489625 <== mds.1 v2:172.21.15.115:6836/3478625369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558e9151c340 con 0x558e91196c90 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f0fef400640 10 client.15330 handle_command_reply: tid=0 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 client.15330 shutdown 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 -- 172.21.15.97:0/1474489625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 msgr2=0x558e915c8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915c8600 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f10000a73e0 tx=0x7f100000f9a0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 -- 172.21.15.97:0/1474489625 shutdown_connections 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x558e91196c90 0x558e91632660 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558e91202f60 0x558e915c8600 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e911e2280 0x558e915c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f0fef400640 0 client.15330 ms_handle_reset on v2:172.21.15.115:6836/3478625369 2024-07-26T07:51:27.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 --2- 172.21.15.97:0/1474489625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558e911da0a0 0x558e915c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.480+0000 7f10171e1740 1 -- 172.21.15.97:0/1474489625 >> 172.21.15.97:0/1474489625 conn(0x558e911f7350 msgr2=0x558e915b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.481+0000 7f10171e1740 1 -- 172.21.15.97:0/1474489625 shutdown_connections 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.481+0000 7f10171e1740 1 -- 172.21.15.97:0/1474489625 wait complete. 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.481+0000 7f10171e1740 20 client.15330 trim_cache size 0 max 16384 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 1, 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34556, 2024-07-26T07:51:27.484 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:51:27.485 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:51:27.485 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:51:27.485 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 203.73713366499999, 2024-07-26T07:51:27.485 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:51:27.485 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:51:27.485 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:51:27.485 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 255.66440846099999 2024-07-26T07:51:27.485 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 -- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ff00b3c80 msgr2=0x7f0ff00b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ff00b3c80 0x7f0ff00b6190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc006010 tx=0x7f0ffc009560 comp rx=0 tx=0).stop 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 -- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f100810fc80 msgr2=0x7f10081a5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f100810fc80 0x7f10081a5b90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f100000cce0 tx=0x7f100000a620 comp rx=0 tx=0).stop 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 -- 172.21.15.97:0/163492889 shutdown_connections 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ff00b3c80 0x7f0ff00b6190 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f100810fc80 0x7f10081a5b90 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f100810f330 0x7f10081a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 --2- 172.21.15.97:0/163492889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10081099f0 0x7f10081a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 -- 172.21.15.97:0/163492889 >> 172.21.15.97:0/163492889 conn(0x7f1008078ff0 msgr2=0x7f1008079950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:27.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 -- 172.21.15.97:0/163492889 shutdown_connections 2024-07-26T07:51:27.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:27.482+0000 7f100f000640 1 -- 172.21.15.97:0/163492889 wait complete. 2024-07-26T07:51:28.131 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:27 smithi097 ceph-mon[32231]: pgmap v500: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:28.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.b status 2024-07-26T07:51:28.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:27 smithi115 ceph-mon[35364]: pgmap v500: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:27 smithi181 ceph-mon[35869]: pgmap v500: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:28.660 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:29 smithi115 ceph-mon[35364]: pgmap v501: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:29 smithi181 ceph-mon[35869]: pgmap v501: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:29 smithi097 ceph-mon[32231]: pgmap v501: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 -- 172.21.15.97:0/1158471166 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 msgr2=0x7f2c0410d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 --2- 172.21.15.97:0/1158471166 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 0x7f2c0410d090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009900 tx=0x7f2bf002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 -- 172.21.15.97:0/1158471166 shutdown_connections 2024-07-26T07:51:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 --2- 172.21.15.97:0/1158471166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c0410d650 0x7f2c0410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 --2- 172.21.15.97:0/1158471166 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 0x7f2c0410d090 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 --2- 172.21.15.97:0/1158471166 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c041041f0 0x7f2c04104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 -- 172.21.15.97:0/1158471166 >> 172.21.15.97:0/1158471166 conn(0x7f2c040fdcf0 msgr2=0x7f2c04100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 -- 172.21.15.97:0/1158471166 shutdown_connections 2024-07-26T07:51:30.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.796+0000 7f2c03200640 1 -- 172.21.15.97:0/1158471166 wait complete. 2024-07-26T07:51:30.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.798+0000 7f2c03200640 1 Processor -- start 2024-07-26T07:51:30.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.798+0000 7f2c03200640 1 -- start start 2024-07-26T07:51:30.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.798+0000 7f2c03200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c041041f0 0x7f2c0419be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.798+0000 7f2c03200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 0x7f2c0419c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.798+0000 7f2c03200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c0410d650 0x7f2c041a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.798+0000 7f2c03200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2c04113150 con 0x7f2c04104b40 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.798+0000 7f2c03200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2c04112fd0 con 0x7f2c0410d650 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.798+0000 7f2c03200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2c041132d0 con 0x7f2c041041f0 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 0x7f2c0419c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 0x7f2c0419c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58082/0 (socket says 172.21.15.97:58082) 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 -- 172.21.15.97:0/751294077 learned_addr learned my addr 172.21.15.97:0/751294077 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01e00640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c041041f0 0x7f2c0419be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c02800640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c0410d650 0x7f2c041a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 -- 172.21.15.97:0/751294077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c041041f0 msgr2=0x7f2c0419be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c041041f0 0x7f2c0419be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 -- 172.21.15.97:0/751294077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c0410d650 msgr2=0x7f2c041a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c0410d650 0x7f2c041a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 -- 172.21.15.97:0/751294077 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c041a1cf0 con 0x7f2c04104b40 2024-07-26T07:51:30.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.799+0000 7f2c01400640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 0x7f2c0419c3e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0005940 tx=0x7f2bf00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:30.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.800+0000 7f2bfaa00640 1 -- 172.21.15.97:0/751294077 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf003f030 con 0x7f2c04104b40 2024-07-26T07:51:30.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.800+0000 7f2bfaa00640 1 -- 172.21.15.97:0/751294077 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bf0003c90 con 0x7f2c04104b40 2024-07-26T07:51:30.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.800+0000 7f2c03200640 1 -- 172.21.15.97:0/751294077 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c041a1f80 con 0x7f2c04104b40 2024-07-26T07:51:30.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.800+0000 7f2bfaa00640 1 -- 172.21.15.97:0/751294077 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0035070 con 0x7f2c04104b40 2024-07-26T07:51:30.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.800+0000 7f2c03200640 1 -- 172.21.15.97:0/751294077 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c041a2460 con 0x7f2c04104b40 2024-07-26T07:51:30.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.804+0000 7f2c0fac1740 1 Processor -- start 2024-07-26T07:51:30.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.804+0000 7f2c0fac1740 1 -- start start 2024-07-26T07:51:30.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83baf60 0x5607c8760760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83920a0 0x5607c8760ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c839a280 0x5607c87611e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5607c86d6950 con 0x5607c83920a0 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5607c86c5e50 con 0x5607c83baf60 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5607c86c9e90 con 0x5607c839a280 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c01e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83920a0 0x5607c8760ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c01e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83920a0 0x5607c8760ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58092/0 (socket says 172.21.15.97:58092) 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c01e00640 1 -- 172.21.15.97:0/3640400342 learned_addr learned my addr 172.21.15.97:0/3640400342 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2bfaa00640 1 -- 172.21.15.97:0/751294077 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f2bf0042030 con 0x7f2c04104b40 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.805+0000 7f2c02800640 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83baf60 0x5607c8760760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.806+0000 7f2c01e00640 1 -- 172.21.15.97:0/3640400342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c839a280 msgr2=0x5607c87611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.806+0000 7f2c01e00640 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c839a280 0x5607c87611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.806+0000 7f2c01e00640 1 -- 172.21.15.97:0/3640400342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83baf60 msgr2=0x5607c8760760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.806+0000 7f2c01e00640 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83baf60 0x5607c8760760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.806+0000 7f2c01e00640 1 -- 172.21.15.97:0/3640400342 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607c866d0f0 con 0x5607c83920a0 2024-07-26T07:51:30.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.806+0000 7f2c02800640 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83baf60 0x5607c8760760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.806+0000 7f2c01e00640 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83920a0 0x5607c8760ca0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400cc60 tx=0x7f2bf400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ee13ca0af571e8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.806+0000 7f2bef400640 1 -- 172.21.15.97:0/3640400342 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4014070 con 0x5607c83920a0 2024-07-26T07:51:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.807+0000 7f2bef400640 1 -- 172.21.15.97:0/3640400342 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bf40047a0 con 0x5607c83920a0 2024-07-26T07:51:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.807+0000 7f2bef400640 1 -- 172.21.15.97:0/3640400342 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4010690 con 0x5607c83920a0 2024-07-26T07:51:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.807+0000 7f2bfaa00640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2be80b3c00 0x7f2be80b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.807+0000 7f2c0fac1740 1 -- 172.21.15.97:0/3640400342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83920a0 msgr2=0x5607c8760ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.807+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83920a0 0x5607c8760ca0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400cc60 tx=0x7f2bf400e720 comp rx=0 tx=0).stop 2024-07-26T07:51:30.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2c0fac1740 1 -- 172.21.15.97:0/3640400342 shutdown_connections 2024-07-26T07:51:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c839a280 0x5607c87611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2bfaa00640 1 -- 172.21.15.97:0/751294077 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f2bf01042e0 con 0x7f2c04104b40 2024-07-26T07:51:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83920a0 0x5607c8760ca0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/3640400342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83baf60 0x5607c8760760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2c0fac1740 1 -- 172.21.15.97:0/3640400342 >> 172.21.15.97:0/3640400342 conn(0x5607c83af350 msgr2=0x5607c875e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2c02800640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2be80b3c00 0x7f2be80b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2c0fac1740 1 -- 172.21.15.97:0/3640400342 shutdown_connections 2024-07-26T07:51:30.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.808+0000 7f2c0fac1740 1 -- 172.21.15.97:0/3640400342 wait complete. 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.809+0000 7f2c02800640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2be80b3c00 0x7f2be80b6110 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00ae20 tx=0x7f2bfc00a3d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.809+0000 7f2c0fac1740 1 -- start start 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.809+0000 7f2c0fac1740 1 Processor -- start 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c876c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c876c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 0x5607c8778290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5607c85295f0 con 0x5607c839a280 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5607c8662520 con 0x5607c83920a0 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5607c876ef40 con 0x5607c83baf60 2024-07-26T07:51:30.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c0fac1740 10 client.0 _ll_register_callbacks cb 0x5607c8671ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c876c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c876c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58104/0 (socket says 172.21.15.97:58104) 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01400640 1 -- 172.21.15.97:0/2295666863 learned_addr learned my addr 172.21.15.97:0/2295666863 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01e00640 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c876c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c02800640 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 0x5607c8778290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01400640 1 -- 172.21.15.97:0/2295666863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 msgr2=0x5607c8778290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01400640 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 0x5607c8778290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01400640 1 -- 172.21.15.97:0/2295666863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 msgr2=0x5607c876c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01400640 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c876c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.810+0000 7f2c01400640 1 -- 172.21.15.97:0/2295666863 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607c8426540 con 0x5607c839a280 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.811+0000 7f2c01400640 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c876c850 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00cb100 tx=0x7f2bf00caac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1238f83a6c00ce server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.811+0000 7f2be3e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.811+0000 7f2be3e00640 1 -- 172.21.15.97:0/2295666863 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0049070 con 0x5607c839a280 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.811+0000 7f2be3e00640 1 -- 172.21.15.97:0/2295666863 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f2bf0101e30 con 0x5607c839a280 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.811+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2295666863 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bf40047a0 con 0x5607c839a280 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.811+0000 7f2be3e00640 1 -- 172.21.15.97:0/2295666863 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2bf00cac80 con 0x5607c839a280 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.811+0000 7f2c0fac1740 10 client.15342 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.811+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2295666863 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607c877aef0 con 0x5607c839a280 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.812+0000 7f2be3e00640 1 -- 172.21.15.97:0/2295666863 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bf40047a0 con 0x5607c839a280 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.812+0000 7f2be3e00640 1 -- 172.21.15.97:0/2295666863 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2bf00d3d90 con 0x5607c839a280 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.812+0000 7f2c0fac1740 10 client.15342 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.812+0000 7f2c0fac1740 10 client.15342 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.812+0000 7f2c0fac1740 20 client.15342 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.812+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5607c83798e0 0x5607c877e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.812+0000 7f2c0fac1740 4 client.15342 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:51:30.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.812+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2295666863 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607c86d88a0 con 0x5607c83798e0 2024-07-26T07:51:30.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.813+0000 7f2c01e00640 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5607c83798e0 0x5607c877e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.814+0000 7f2c01e00640 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5607c83798e0 0x5607c877e9b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aa572f65da9c8ed server_cookie=9660863a5c992234 in_seq=0 out_seq=0 2024-07-26T07:51:30.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.814+0000 7f2be3e00640 10 client.15342 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:51:30.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2be3e00640 1 -- 172.21.15.97:0/2295666863 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5607c86d88a0 con 0x5607c83798e0 2024-07-26T07:51:30.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2be3e00640 10 client.15342 handle_command_reply: tid=0 2024-07-26T07:51:30.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 client.15342 shutdown 2024-07-26T07:51:30.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2295666863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 msgr2=0x5607c876c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c876c850 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00cb100 tx=0x7f2bf00caac0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2295666863 shutdown_connections 2024-07-26T07:51:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5607c83798e0 0x5607c877e9b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 0x5607c8778290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c876c850 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2295666863 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c876c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2be3e00640 0 client.15342 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:51:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2295666863 >> 172.21.15.97:0/2295666863 conn(0x5607c83af350 msgr2=0x5607c87638f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.816+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2295666863 shutdown_connections 2024-07-26T07:51:30.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.817+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2295666863 wait complete. 2024-07-26T07:51:30.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.817+0000 7f2c0fac1740 20 client.15342 trim_cache size 0 max 16384 2024-07-26T07:51:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.836+0000 7f2c0fac1740 1 Processor -- start 2024-07-26T07:51:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.836+0000 7f2c0fac1740 1 -- start start 2024-07-26T07:51:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.836+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c877dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.836+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c8778200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.836+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 0x5607c877c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.836+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5607c877ccf0 con 0x5607c839a280 2024-07-26T07:51:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.836+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5607c877d3c0 con 0x5607c83920a0 2024-07-26T07:51:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.836+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5607c877da90 con 0x5607c83baf60 2024-07-26T07:51:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c8778200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c8778200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58106/0 (socket says 172.21.15.97:58106) 2024-07-26T07:51:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01400640 1 -- 172.21.15.97:0/2439421746 learned_addr learned my addr 172.21.15.97:0/2439421746 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01e00640 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c877dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01400640 1 -- 172.21.15.97:0/2439421746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 msgr2=0x5607c877c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01400640 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 0x5607c877c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01400640 1 -- 172.21.15.97:0/2439421746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 msgr2=0x5607c877dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01400640 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c877dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01400640 1 -- 172.21.15.97:0/2439421746 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607c87597d0 con 0x5607c839a280 2024-07-26T07:51:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.837+0000 7f2c01e00640 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c877dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.838+0000 7f2c01400640 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c8778200 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2bf003d040 tx=0x7f2bf00cabf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f41784e29a3dba server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:30.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.838+0000 7f2beea00640 1 -- 172.21.15.97:0/2439421746 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf00ea070 con 0x5607c839a280 2024-07-26T07:51:30.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.838+0000 7f2beea00640 1 -- 172.21.15.97:0/2439421746 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bf00cb870 con 0x5607c839a280 2024-07-26T07:51:30.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.838+0000 7f2beea00640 1 -- 172.21.15.97:0/2439421746 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0049050 con 0x5607c839a280 2024-07-26T07:51:30.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2439421746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 msgr2=0x5607c8778200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c8778200 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2bf003d040 tx=0x7f2bf00cabf0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2439421746 shutdown_connections 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83baf60 0x5607c877c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c839a280 0x5607c8778200 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2439421746 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c83920a0 0x5607c877dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2439421746 >> 172.21.15.97:0/2439421746 conn(0x5607c83af350 msgr2=0x5607c876c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2439421746 shutdown_connections 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2439421746 wait complete. 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.839+0000 7f2c0fac1740 1 -- start start 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c0fac1740 1 Processor -- start 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83920a0 0x5607c875e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c839a280 0x5607c875cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c0fac1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83baf60 0x5607c875d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5607c877e830 con 0x5607c83baf60 2024-07-26T07:51:30.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5607c877e160 con 0x5607c839a280 2024-07-26T07:51:30.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c0fac1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5607c877ef00 con 0x5607c83920a0 2024-07-26T07:51:30.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c0fac1740 10 client.0 _ll_register_callbacks cb 0x7f2bf00c7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:30.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.840+0000 7f2c02800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83baf60 0x5607c875d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c02800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83baf60 0x5607c875d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58118/0 (socket says 172.21.15.97:58118) 2024-07-26T07:51:30.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c02800640 1 -- 172.21.15.97:0/2338791245 learned_addr learned my addr 172.21.15.97:0/2338791245 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:30.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c01400640 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c839a280 0x5607c875cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c01e00640 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83920a0 0x5607c875e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c02800640 1 -- 172.21.15.97:0/2338791245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83920a0 msgr2=0x5607c875e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c02800640 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83920a0 0x5607c875e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c02800640 1 -- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c839a280 msgr2=0x5607c875cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c02800640 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c839a280 0x5607c875cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c02800640 1 -- 172.21.15.97:0/2338791245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607c875da20 con 0x5607c83baf60 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c01400640 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c839a280 0x5607c875cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2c02800640 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83baf60 0x5607c875d290 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc0119d0 tx=0x7f2bfc011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5124dd6d51f8a5b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.841+0000 7f2be3e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.842+0000 7f2be3e00640 1 -- 172.21.15.97:0/2338791245 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc010ce0 con 0x5607c83baf60 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.842+0000 7f2be3e00640 1 -- 172.21.15.97:0/2338791245 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f2bfc003970 con 0x5607c83baf60 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.842+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2338791245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bf00cb870 con 0x5607c83baf60 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.842+0000 7f2be3e00640 1 -- 172.21.15.97:0/2338791245 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2bfc003b70 con 0x5607c83baf60 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.842+0000 7f2c0fac1740 10 client.15348 fetch_fsmap learned FSMap version 27 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.842+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2338791245 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607c8786c70 con 0x5607c83baf60 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.843+0000 7f2be3e00640 1 -- 172.21.15.97:0/2338791245 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bf00cb870 con 0x5607c83baf60 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.843+0000 7f2be3e00640 1 -- 172.21.15.97:0/2338791245 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2bfc030410 con 0x5607c83baf60 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.843+0000 7f2c0fac1740 10 client.15348 fetch_fsmap finished waiting for FSMap version 27 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.843+0000 7f2c0fac1740 10 client.15348 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.843+0000 7f2c0fac1740 20 client.15348 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.843+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5607c834ec90 0x5607c87ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:30.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.843+0000 7f2c0fac1740 4 client.15348 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:51:30.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.843+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2338791245 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "status"}) -- 0x5607c86d4340 con 0x5607c834ec90 2024-07-26T07:51:30.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.844+0000 7f2c01e00640 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5607c834ec90 0x5607c87ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.845+0000 7f2c01e00640 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5607c834ec90 0x5607c87ea760 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a50e6d801836dd7f server_cookie=b9f5a04872083a65 in_seq=0 out_seq=0 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.845+0000 7f2be3e00640 10 client.15348 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.845+0000 7f2be3e00640 1 -- 172.21.15.97:0/2338791245 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5607c86d4340 con 0x5607c834ec90 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.845+0000 7f2be3e00640 10 client.15348 handle_command_reply: tid=0 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.845+0000 7f2c0fac1740 1 client.15348 shutdown 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.845+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2338791245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83baf60 msgr2=0x5607c875d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.845+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83baf60 0x5607c875d290 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc0119d0 tx=0x7f2bfc011dd0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2338791245 shutdown_connections 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5607c834ec90 0x5607c87ea760 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5607c83baf60 0x5607c875d290 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5607c839a280 0x5607c875cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 1 --2- 172.21.15.97:0/2338791245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5607c83920a0 0x5607c875e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2338791245 >> 172.21.15.97:0/2338791245 conn(0x5607c83af350 msgr2=0x5607c876a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2be3e00640 0 client.15348 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2338791245 shutdown_connections 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 1 -- 172.21.15.97:0/2338791245 wait complete. 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.846+0000 7f2c0fac1740 20 client.15348 trim_cache size 0 max 16384 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:51:30.849 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34583, 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 222.002059347, 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 27, 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 82, 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 242.63592672799999 2024-07-26T07:51:30.850 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:51:30.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 -- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2be80b3c00 msgr2=0x7f2be80b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2be80b3c00 0x7f2be80b6110 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00ae20 tx=0x7f2bfc00a3d0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 -- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 msgr2=0x7f2c0419c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:30.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 0x7f2c0419c3e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0005940 tx=0x7f2bf00042f0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 -- 172.21.15.97:0/751294077 shutdown_connections 2024-07-26T07:51:30.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f2be80b3c00 0x7f2be80b6110 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c0410d650 0x7f2c041a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c04104b40 0x7f2c0419c3e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 --2- 172.21.15.97:0/751294077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c041041f0 0x7f2c0419be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:30.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 -- 172.21.15.97:0/751294077 >> 172.21.15.97:0/751294077 conn(0x7f2c040fdcf0 msgr2=0x7f2c040ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:30.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.847+0000 7f2c03200640 1 -- 172.21.15.97:0/751294077 shutdown_connections 2024-07-26T07:51:30.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:30.848+0000 7f2c03200640 1 -- 172.21.15.97:0/751294077 wait complete. 2024-07-26T07:51:31.637 INFO:teuthology.run_tasks:Running task sequential... 2024-07-26T07:51:31.745 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-07-26T07:51:31.745 INFO:teuthology.task.sequential:In sequential, running task print... 2024-07-26T07:51:31.753 INFO:teuthology.task.print:Enabling mgr modules 2024-07-26T07:51:31.753 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-07-26T07:51:31.813 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-07-26T07:51:31.813 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-07-26T07:51:31.814 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-07-26T07:51:31.814 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T07:51:31.814 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T07:51:31.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:31.814 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-07-26T07:51:31.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:31.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link delete ceph-brx 2024-07-26T07:51:31.947 INFO:teuthology.orchestra.run.smithi097.stderr:Cannot find device "ceph-brx" 2024-07-26T07:51:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T07:51:31.950 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-07-26T07:51:31.950 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-07-26T07:51:31.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs ls 2024-07-26T07:51:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:31 smithi115 ceph-mon[35364]: pgmap v502: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:31 smithi181 ceph-mon[35869]: pgmap v502: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:32.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:31 smithi097 ceph-mon[32231]: pgmap v502: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T07:51:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:34.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:33 smithi097 ceph-mon[32231]: pgmap v503: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:34.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:33 smithi115 ceph-mon[35364]: pgmap v503: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:33 smithi181 ceph-mon[35869]: pgmap v503: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T07:51:34.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.415+0000 7fed97200640 1 -- 172.21.15.97:0/530180826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed90108d20 msgr2=0x7fed90109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:34.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.415+0000 7fed97200640 1 --2- 172.21.15.97:0/530180826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed90108d20 0x7fed90109130 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fed80009900 tx=0x7fed8002a410 comp rx=0 tx=0).stop 2024-07-26T07:51:34.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.416+0000 7fed97200640 1 -- 172.21.15.97:0/530180826 shutdown_connections 2024-07-26T07:51:34.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.416+0000 7fed97200640 1 --2- 172.21.15.97:0/530180826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed90109670 0x7fed9010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.416+0000 7fed97200640 1 --2- 172.21.15.97:0/530180826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed90108d20 0x7fed90109130 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.416+0000 7fed97200640 1 --2- 172.21.15.97:0/530180826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed901033e0 0x7fed901037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.416+0000 7fed97200640 1 -- 172.21.15.97:0/530180826 >> 172.21.15.97:0/530180826 conn(0x7fed900fee70 msgr2=0x7fed901012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:34.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.416+0000 7fed97200640 1 -- 172.21.15.97:0/530180826 shutdown_connections 2024-07-26T07:51:34.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.416+0000 7fed97200640 1 -- 172.21.15.97:0/530180826 wait complete. 2024-07-26T07:51:34.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.417+0000 7fed97200640 1 Processor -- start 2024-07-26T07:51:34.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.417+0000 7fed97200640 1 -- start start 2024-07-26T07:51:34.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.417+0000 7fed97200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901033e0 0x7fed9019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:34.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.417+0000 7fed97200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed90108d20 0x7fed9019c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.417+0000 7fed97200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed90109670 0x7fed901a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.417+0000 7fed97200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fed901132d0 con 0x7fed901033e0 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.417+0000 7fed97200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fed90113150 con 0x7fed90108d20 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.417+0000 7fed97200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fed90113450 con 0x7fed90109670 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901033e0 0x7fed9019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901033e0 0x7fed9019c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58140/0 (socket says 172.21.15.97:58140) 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95e00640 1 -- 172.21.15.97:0/2733498312 learned_addr learned my addr 172.21.15.97:0/2733498312 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95400640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed90108d20 0x7fed9019c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed96800640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed90109670 0x7fed901a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95e00640 1 -- 172.21.15.97:0/2733498312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed90109670 msgr2=0x7fed901a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95e00640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed90109670 0x7fed901a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95e00640 1 -- 172.21.15.97:0/2733498312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed90108d20 msgr2=0x7fed9019c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:34.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95e00640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed90108d20 0x7fed9019c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.418+0000 7fed95e00640 1 -- 172.21.15.97:0/2733498312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed901a1f70 con 0x7fed901033e0 2024-07-26T07:51:34.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.419+0000 7fed95e00640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901033e0 0x7fed9019c000 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fed8400b4a0 tx=0x7fed8400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:34.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.419+0000 7fed8aa00640 1 -- 172.21.15.97:0/2733498312 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84014070 con 0x7fed901033e0 2024-07-26T07:51:34.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.419+0000 7fed8aa00640 1 -- 172.21.15.97:0/2733498312 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed84010aa0 con 0x7fed901033e0 2024-07-26T07:51:34.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.419+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed901a2260 con 0x7fed901033e0 2024-07-26T07:51:34.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.419+0000 7fed8aa00640 1 -- 172.21.15.97:0/2733498312 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84017b70 con 0x7fed901033e0 2024-07-26T07:51:34.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.419+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed901a89c0 con 0x7fed901033e0 2024-07-26T07:51:34.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.420+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed90103870 con 0x7fed901033e0 2024-07-26T07:51:34.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.423+0000 7fed8aa00640 1 -- 172.21.15.97:0/2733498312 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fed84017070 con 0x7fed901033e0 2024-07-26T07:51:34.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.426+0000 7fed8aa00640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed780b3c50 0x7fed780b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:34.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.426+0000 7fed8aa00640 1 -- 172.21.15.97:0/2733498312 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fed840e33c0 con 0x7fed901033e0 2024-07-26T07:51:34.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.426+0000 7fed95400640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed780b3c50 0x7fed780b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:34.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.427+0000 7fed95400640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed780b3c50 0x7fed780b6160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fed8002a930 tx=0x7fed80000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:34.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.428+0000 7fed8aa00640 1 -- 172.21.15.97:0/2733498312 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fed840aa950 con 0x7fed901033e0 2024-07-26T07:51:34.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.671+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fed9019d130 con 0x7fed901033e0 2024-07-26T07:51:34.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.672+0000 7fed8aa00640 1 -- 172.21.15.97:0/2733498312 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7fed9019d130 con 0x7fed901033e0 2024-07-26T07:51:34.673 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T07:51:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed780b3c50 msgr2=0x7fed780b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed780b3c50 0x7fed780b6160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fed8002a930 tx=0x7fed80000bc0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901033e0 msgr2=0x7fed9019c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901033e0 0x7fed9019c000 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fed8400b4a0 tx=0x7fed8400b960 comp rx=0 tx=0).stop 2024-07-26T07:51:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 shutdown_connections 2024-07-26T07:51:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed780b3c50 0x7fed780b6160 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed90109670 0x7fed901a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed90108d20 0x7fed9019c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 --2- 172.21.15.97:0/2733498312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901033e0 0x7fed9019c000 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:34.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 >> 172.21.15.97:0/2733498312 conn(0x7fed900fee70 msgr2=0x7fed9010e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:34.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.675+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 shutdown_connections 2024-07-26T07:51:34.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:34.676+0000 7fed97200640 1 -- 172.21.15.97:0/2733498312 wait complete. 2024-07-26T07:51:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:34 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2733498312' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T07:51:35.411 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T07:51:35.412 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T07:51:35.412 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-07-26T07:51:35.412 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T07:51:35.412 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T07:51:35.412 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T07:51:35.412 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T07:51:35.412 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T07:51:35.412 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T07:51:35.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:35.413 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-07-26T07:51:35.466 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T07:51:35.466 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T07:51:35.466 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T07:51:35.466 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-07-26T07:51:35.466 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 41648sec preferred_lft 41648sec 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link 2024-07-26T07:51:35.467 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-07-26T07:51:35.468 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T07:51:35.469 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T07:51:35.469 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-07-26T07:51:35.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add name ceph-brx type bridge 2024-07-26T07:51:35.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr flush dev ceph-brx 2024-07-26T07:51:35.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set ceph-brx up 2024-07-26T07:51:35.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T07:51:35.469 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-07-26T07:51:35.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:34 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2733498312' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T07:51:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:34 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2733498312' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T07:51:35.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T07:51:35.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T07:51:35.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:35.940 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T07:51:36.009 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-07-26T07:51:36.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:36.013 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-07-26T07:51:36.066 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-07-26T07:51:36.066 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T07:51:36.067 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T07:51:36.067 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T07:51:36.067 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T07:51:36.067 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T07:51:36.067 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-07-26T07:51:36.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T07:51:36.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T07:51:36.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T07:51:36.068 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-07-26T07:51:36.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T07:51:36.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:35 smithi097 ceph-mon[32231]: pgmap v504: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:35 smithi115 ceph-mon[35364]: pgmap v504: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:35 smithi181 ceph-mon[35869]: pgmap v504: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T07:51:36.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T07:51:36.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:36.794 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-07-26T07:51:36.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:36.847 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list-id 2024-07-26T07:51:36.896 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T07:51:36.896 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-07-26T07:51:36.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T07:51:36.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-07-26T07:51:36.897 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-07-26T07:51:36.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T07:51:37.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T07:51:37.073 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-07-26T07:51:37.074 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T07:51:37.074 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-07-26T07:51:37.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-07-26T07:51:37.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T07:51:37.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-07-26T07:51:37.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-07-26T07:51:37.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-07-26T07:51:37.074 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-07-26T07:51:37.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T07:51:37.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T07:51:37.368 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T07:51:37.368 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-07-26T07:51:37.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-07-26T07:51:37.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T07:51:37.369 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-07-26T07:51:37.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T07:51:37.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T07:51:37.569 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-07-26T07:51:37.570 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T07:51:37.570 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T07:51:37.618 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T07:51:37.618 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T07:51:37.618 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T07:51:37.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo modprobe fuse 2024-07-26T07:51:37.790 DEBUG:teuthology.orchestra.run.smithi097:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T07:51:37.835 INFO:teuthology.orchestra.run.smithi097.stdout:/proc 2024-07-26T07:51:37.835 INFO:teuthology.orchestra.run.smithi097.stdout:/sys 2024-07-26T07:51:37.835 INFO:teuthology.orchestra.run.smithi097.stdout:/dev 2024-07-26T07:51:37.835 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/security 2024-07-26T07:51:37.835 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/shm 2024-07-26T07:51:37.835 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/pts 2024-07-26T07:51:37.835 INFO:teuthology.orchestra.run.smithi097.stdout:/run 2024-07-26T07:51:37.835 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/cgroup 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/pstore 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/bpf 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/ 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/selinux 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/mqueue 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/hugepages 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/debug 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/tracing 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/fuse/connections 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/config 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T07:51:37.836 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/1000 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/ceph 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/0657a44fdc77c0f4cfbbced5685e503eabf2af7eab0572fd380472e61d36ee78/merged 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/984196a82e3d483c5871e86f26b59ddc65c11a9ed6a837daf925adca538069e3/merged 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/0 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/33612c61b06b81bc8312481d704c9333c5d37059f50d6aa72af8342e7ceced8f/merged 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/083f708bf63f45286d32e89bbacd714bcb6bb6fd2cb4625d91cc8ba0086210e2/merged 2024-07-26T07:51:37.837 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/8dd23742a86cd2102822a59c45264d8ce58b2413589f125dadb49f46fe49c435/merged 2024-07-26T07:51:37.838 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/0198330f209462ddffd1a473593a18aa87de752098d098d658c2a7c3f15a2d12/merged 2024-07-26T07:51:37.838 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/1a939d3d17096c3420ec4eb78848ccb7bbfc5492dd2129d621291bac755550b3/merged 2024-07-26T07:51:37.838 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/24660ab7b905e66714b01f35e760a9399ffedcab24c56bc305acd9cdababc9d7/merged 2024-07-26T07:51:37.838 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/6e7119be3039a527e98a3dd43f8a593b2d6d536aec3504de9b221e4d746b363d/merged 2024-07-26T07:51:37.838 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/2c6f5e1797710abb855bb26a559e8da94be2a8d415552b6f45ed3cdc4db233bc/merged 2024-07-26T07:51:37.838 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns 2024-07-26T07:51:37.838 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T07:51:37.838 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T07:51:37.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:37.839 DEBUG:teuthology.orchestra.run.smithi097:> ls /sys/fs/fuse/connections 2024-07-26T07:51:37.933 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-07-26T07:51:37.933 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/4e92e127-3c83-480e-8968-682139f34bdb --client_fs=cephfs) 2024-07-26T07:51:37.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo modprobe fuse 2024-07-26T07:51:38.006 DEBUG:teuthology.orchestra.run.smithi097:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T07:51:38.049 INFO:teuthology.orchestra.run.smithi097.stdout:/proc 2024-07-26T07:51:38.049 INFO:teuthology.orchestra.run.smithi097.stdout:/sys 2024-07-26T07:51:38.049 INFO:teuthology.orchestra.run.smithi097.stdout:/dev 2024-07-26T07:51:38.049 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/security 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/shm 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/pts 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/run 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/cgroup 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/pstore 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/bpf 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/ 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/selinux 2024-07-26T07:51:38.050 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/mqueue 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/hugepages 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/debug 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/tracing 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/fuse/connections 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/config 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/1000 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/ceph 2024-07-26T07:51:38.051 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/0657a44fdc77c0f4cfbbced5685e503eabf2af7eab0572fd380472e61d36ee78/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/984196a82e3d483c5871e86f26b59ddc65c11a9ed6a837daf925adca538069e3/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/0 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/33612c61b06b81bc8312481d704c9333c5d37059f50d6aa72af8342e7ceced8f/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/083f708bf63f45286d32e89bbacd714bcb6bb6fd2cb4625d91cc8ba0086210e2/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/8dd23742a86cd2102822a59c45264d8ce58b2413589f125dadb49f46fe49c435/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/0198330f209462ddffd1a473593a18aa87de752098d098d658c2a7c3f15a2d12/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/1a939d3d17096c3420ec4eb78848ccb7bbfc5492dd2129d621291bac755550b3/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/24660ab7b905e66714b01f35e760a9399ffedcab24c56bc305acd9cdababc9d7/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/6e7119be3039a527e98a3dd43f8a593b2d6d536aec3504de9b221e4d746b363d/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/2c6f5e1797710abb855bb26a559e8da94be2a8d415552b6f45ed3cdc4db233bc/merged 2024-07-26T07:51:38.052 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns 2024-07-26T07:51:38.053 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T07:51:38.053 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T07:51:38.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:38.054 DEBUG:teuthology.orchestra.run.smithi097:> ls /sys/fs/fuse/connections 2024-07-26T07:51:38.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:37 smithi097 ceph-mon[32231]: pgmap v505: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:38.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi097.stderr:2024-07-26T07:51:38.250+0000 7f43eebe3480 -1 init, newargv = 0x7f43d4004d30 newargc=15 2024-07-26T07:51:38.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi097.stderr:ceph-fuse[91686]: starting ceph client 2024-07-26T07:51:38.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi097.stderr:ceph-fuse[91686]: starting fuse 2024-07-26T07:51:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:37 smithi115 ceph-mon[35364]: pgmap v505: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:37 smithi181 ceph-mon[35869]: pgmap v505: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T07:51:39.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo modprobe fuse 2024-07-26T07:51:39.332 DEBUG:teuthology.orchestra.run.smithi097:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T07:51:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:/proc 2024-07-26T07:51:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:/sys 2024-07-26T07:51:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:/dev 2024-07-26T07:51:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/security 2024-07-26T07:51:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/shm 2024-07-26T07:51:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/pts 2024-07-26T07:51:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:/run 2024-07-26T07:51:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/cgroup 2024-07-26T07:51:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/pstore 2024-07-26T07:51:39.390 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/bpf 2024-07-26T07:51:39.390 INFO:teuthology.orchestra.run.smithi097.stdout:/ 2024-07-26T07:51:39.390 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/selinux 2024-07-26T07:51:39.390 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T07:51:39.390 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/mqueue 2024-07-26T07:51:39.391 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/hugepages 2024-07-26T07:51:39.391 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/debug 2024-07-26T07:51:39.391 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/tracing 2024-07-26T07:51:39.391 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/fuse/connections 2024-07-26T07:51:39.391 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/config 2024-07-26T07:51:39.392 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T07:51:39.392 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T07:51:39.392 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T07:51:39.392 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T07:51:39.392 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/1000 2024-07-26T07:51:39.392 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/ceph 2024-07-26T07:51:39.393 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay 2024-07-26T07:51:39.393 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/0657a44fdc77c0f4cfbbced5685e503eabf2af7eab0572fd380472e61d36ee78/merged 2024-07-26T07:51:39.393 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/984196a82e3d483c5871e86f26b59ddc65c11a9ed6a837daf925adca538069e3/merged 2024-07-26T07:51:39.393 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/0 2024-07-26T07:51:39.393 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T07:51:39.394 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/33612c61b06b81bc8312481d704c9333c5d37059f50d6aa72af8342e7ceced8f/merged 2024-07-26T07:51:39.394 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/083f708bf63f45286d32e89bbacd714bcb6bb6fd2cb4625d91cc8ba0086210e2/merged 2024-07-26T07:51:39.394 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/8dd23742a86cd2102822a59c45264d8ce58b2413589f125dadb49f46fe49c435/merged 2024-07-26T07:51:39.394 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/0198330f209462ddffd1a473593a18aa87de752098d098d658c2a7c3f15a2d12/merged 2024-07-26T07:51:39.394 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/1a939d3d17096c3420ec4eb78848ccb7bbfc5492dd2129d621291bac755550b3/merged 2024-07-26T07:51:39.395 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/24660ab7b905e66714b01f35e760a9399ffedcab24c56bc305acd9cdababc9d7/merged 2024-07-26T07:51:39.395 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/6e7119be3039a527e98a3dd43f8a593b2d6d536aec3504de9b221e4d746b363d/merged 2024-07-26T07:51:39.395 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/2c6f5e1797710abb855bb26a559e8da94be2a8d415552b6f45ed3cdc4db233bc/merged 2024-07-26T07:51:39.395 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns 2024-07-26T07:51:39.395 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T07:51:39.396 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T07:51:39.396 INFO:teuthology.orchestra.run.smithi097.stdout:/home/ubuntu/cephtest/mnt.0 2024-07-26T07:51:39.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:39.398 DEBUG:teuthology.orchestra.run.smithi097:> ls /sys/fs/fuse/connections 2024-07-26T07:51:39.454 INFO:teuthology.orchestra.run.smithi097.stdout:119 2024-07-26T07:51:39.454 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-07-26T07:51:39.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo stdin-killer -- python3 -c ' 2024-07-26T07:51:39.454 DEBUG:teuthology.orchestra.run.smithi097:> import glob 2024-07-26T07:51:39.454 DEBUG:teuthology.orchestra.run.smithi097:> import re 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> import subprocess 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> def _find_admin_socket(client_name): 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> files = glob.glob(asok_path) 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> # Given a non-glob path, it better be there 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> if "*" not in asok_path: 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> assert(len(files) == 1) 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> return files[0] 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> 2024-07-26T07:51:39.455 DEBUG:teuthology.orchestra.run.smithi097:> for f in files: 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> if os.path.exists("/proc/{0}".format(pid)): 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> contents = proc_f.read() 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> if mountpoint in contents: 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> return f 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> print(_find_admin_socket("client.0")) 2024-07-26T07:51:39.456 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-07-26T07:51:39.622 INFO:teuthology.orchestra.run.smithi097.stdout:/var/run/ceph/ceph-client.0.91686.asok 2024-07-26T07:51:39.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T07:51:39.640 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.91686.asok 2024-07-26T07:51:39.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:39.640 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.91686.asok status 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: "metadata": { 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_sha1": "e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528", 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: "entity_id": "0", 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: "hostname": "smithi097", 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: "pid": "91686", 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: "root": "/volumes/qa/sv_0/4e92e127-3c83-480e-8968-682139f34bdb" 2024-07-26T07:51:39.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "dentry_count": 0, 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "dentry_pinned_count": 0, 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 15360, 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "inst": { 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "name": { 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "client", 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "num": 15360 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": { 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "192.168.0.1:0", 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1832020397 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:51:39.929 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": { 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "192.168.0.1:0", 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1832020397 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "inst_str": "client.15360 192.168.0.1:0/1832020397", 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "addr_str": "192.168.0.1:0/1832020397", 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "inode_count": 5, 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_epoch": 27, 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_epoch": 86, 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_epoch_barrier": 0, 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "blocklisted": false, 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs" 2024-07-26T07:51:39.930 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:51:39.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:39.950 DEBUG:teuthology.orchestra.run.smithi097:> stat --file-system '--printf=%T 2024-07-26T07:51:39.950 DEBUG:teuthology.orchestra.run.smithi097:> ' -- /home/ubuntu/cephtest/mnt.0 2024-07-26T07:51:40.007 INFO:teuthology.orchestra.run.smithi097.stdout:fuseblk 2024-07-26T07:51:40.008 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-07-26T07:51:40.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T07:51:40.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T07:51:40.093 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T07:51:40.112 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T07:51:40.112 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-07-26T07:51:40.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-07-26T07:51:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:39 smithi097 ceph-mon[32231]: pgmap v506: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:40.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.380+0000 7fa52be00640 1 -- 172.21.15.97:0/3775676823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c106b60 msgr2=0x7fa52c10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:40.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.380+0000 7fa52be00640 1 --2- 172.21.15.97:0/3775676823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c106b60 0x7fa52c10d500 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa524009ca0 tx=0x7fa52402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:51:40.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.380+0000 7fa52be00640 1 -- 172.21.15.97:0/3775676823 shutdown_connections 2024-07-26T07:51:40.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.380+0000 7fa52be00640 1 --2- 172.21.15.97:0/3775676823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c106b60 0x7fa52c10d500 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.380+0000 7fa52be00640 1 --2- 172.21.15.97:0/3775676823 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa52c1061c0 0x7fa52c1065d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.380+0000 7fa52be00640 1 --2- 172.21.15.97:0/3775676823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa52c105140 0x7fa52c1055b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.380+0000 7fa52be00640 1 -- 172.21.15.97:0/3775676823 >> 172.21.15.97:0/3775676823 conn(0x7fa52c100a80 msgr2=0x7fa52c102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.381+0000 7fa52be00640 1 -- 172.21.15.97:0/3775676823 shutdown_connections 2024-07-26T07:51:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.381+0000 7fa52be00640 1 -- 172.21.15.97:0/3775676823 wait complete. 2024-07-26T07:51:40.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52be00640 1 Processor -- start 2024-07-26T07:51:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52be00640 1 -- start start 2024-07-26T07:51:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c105140 0x7fa52c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa52c1061c0 0x7fa52c19bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52be00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa52c106b60 0x7fa52c19caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c110cc0 con 0x7fa52c105140 2024-07-26T07:51:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52be00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c110b40 con 0x7fa52c106b60 2024-07-26T07:51:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c110e40 con 0x7fa52c1061c0 2024-07-26T07:51:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c105140 0x7fa52c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.382+0000 7fa52aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c105140 0x7fa52c19b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44546/0 (socket says 172.21.15.97:44546) 2024-07-26T07:51:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.383+0000 7fa52aa00640 1 -- 172.21.15.97:0/2320181010 learned_addr learned my addr 172.21.15.97:0/2320181010 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.383+0000 7fa52b400640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa52c106b60 0x7fa52c19caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.383+0000 7fa52aa00640 1 -- 172.21.15.97:0/2320181010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa52c1061c0 msgr2=0x7fa52c19bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.383+0000 7fa52aa00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa52c1061c0 0x7fa52c19bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.383+0000 7fa52aa00640 1 -- 172.21.15.97:0/2320181010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa52c106b60 msgr2=0x7fa52c19caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.383+0000 7fa52aa00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa52c106b60 0x7fa52c19caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.383+0000 7fa52aa00640 1 -- 172.21.15.97:0/2320181010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52c19d310 con 0x7fa52c105140 2024-07-26T07:51:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.383+0000 7fa52aa00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c105140 0x7fa52c19b8d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00d770 tx=0x7fa51c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.384+0000 7fa523400640 1 -- 172.21.15.97:0/2320181010 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c017070 con 0x7fa52c105140 2024-07-26T07:51:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.384+0000 7fa523400640 1 -- 172.21.15.97:0/2320181010 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa51c0054b0 con 0x7fa52c105140 2024-07-26T07:51:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.384+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52c19d600 con 0x7fa52c105140 2024-07-26T07:51:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.384+0000 7fa523400640 1 -- 172.21.15.97:0/2320181010 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c004c60 con 0x7fa52c105140 2024-07-26T07:51:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.384+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa52c1a3cf0 con 0x7fa52c105140 2024-07-26T07:51:40.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.385+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa52c10c310 con 0x7fa52c105140 2024-07-26T07:51:40.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.387+0000 7fa523400640 1 -- 172.21.15.97:0/2320181010 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fa51c004e00 con 0x7fa52c105140 2024-07-26T07:51:40.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.389+0000 7fa523400640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5100b3bd0 0x7fa5100b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:40.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.389+0000 7fa523400640 1 -- 172.21.15.97:0/2320181010 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fa51c0e3c00 con 0x7fa52c105140 2024-07-26T07:51:40.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.389+0000 7fa52a000640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5100b3bd0 0x7fa5100b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:40.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.390+0000 7fa52a000640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5100b3bd0 0x7fa5100b60e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa518005fd0 tx=0x7fa518005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:40.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.391+0000 7fa523400640 1 -- 172.21.15.97:0/2320181010 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa51c0ab190 con 0x7fa52c105140 2024-07-26T07:51:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:40 smithi115 ceph-mon[35364]: pgmap v506: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:40 smithi181 ceph-mon[35869]: pgmap v506: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T07:51:40.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:40.645+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fa52c02e0c0 con 0x7fa52c105140 2024-07-26T07:51:41.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.015+0000 7fa523400640 1 -- 172.21.15.97:0/2320181010 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7fa51c0a6020 con 0x7fa52c105140 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.020+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5100b3bd0 msgr2=0x7fa5100b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.020+0000 7fa52be00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5100b3bd0 0x7fa5100b60e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa518005fd0 tx=0x7fa518005850 comp rx=0 tx=0).stop 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.020+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c105140 msgr2=0x7fa52c19b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c105140 0x7fa52c19b8d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00d770 tx=0x7fa51c00dc30 comp rx=0 tx=0).stop 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 shutdown_connections 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa5100b3bd0 0x7fa5100b60e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa52c106b60 0x7fa52c19caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa52c1061c0 0x7fa52c19bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 --2- 172.21.15.97:0/2320181010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa52c105140 0x7fa52c19b8d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 >> 172.21.15.97:0/2320181010 conn(0x7fa52c100a80 msgr2=0x7fa52c1094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:41.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 shutdown_connections 2024-07-26T07:51:41.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.021+0000 7fa52be00640 1 -- 172.21.15.97:0/2320181010 wait complete. 2024-07-26T07:51:41.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-07-26T07:51:41.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 -- 172.21.15.97:0/3933008910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 msgr2=0x7fbfc81065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:41.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 --2- 172.21.15.97:0/3933008910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 0x7fbfc81065b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009900 tx=0x7fbfb002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 -- 172.21.15.97:0/3933008910 shutdown_connections 2024-07-26T07:51:41.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 --2- 172.21.15.97:0/3933008910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfc8106b40 0x7fbfc810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 --2- 172.21.15.97:0/3933008910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 0x7fbfc81065b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 --2- 172.21.15.97:0/3933008910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfc8105120 0x7fbfc8105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 -- 172.21.15.97:0/3933008910 >> 172.21.15.97:0/3933008910 conn(0x7fbfc8100a80 msgr2=0x7fbfc8102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:41.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 -- 172.21.15.97:0/3933008910 shutdown_connections 2024-07-26T07:51:41.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.312+0000 7fbfcce00640 1 -- 172.21.15.97:0/3933008910 wait complete. 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.313+0000 7fbfcce00640 1 Processor -- start 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.313+0000 7fbfcce00640 1 -- start start 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfcce00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfc8105120 0x7fbfc819b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfcce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 0x7fbfc819bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfcce00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfc8106b40 0x7fbfc819cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfcce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc8110ca0 con 0x7fbfc81061a0 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfcce00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc8110b20 con 0x7fbfc8105120 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfcce00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc8110e20 con 0x7fbfc8106b40 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfbfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 0x7fbfc819bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:41.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfbfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 0x7fbfc819bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44564/0 (socket says 172.21.15.97:44564) 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfbfe00640 1 -- 172.21.15.97:0/2802337407 learned_addr learned my addr 172.21.15.97:0/2802337407 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfc7400640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfc8105120 0x7fbfc819b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfbfe00640 1 -- 172.21.15.97:0/2802337407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfc8106b40 msgr2=0x7fbfc819cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfbfe00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfc8106b40 0x7fbfc819cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfbfe00640 1 -- 172.21.15.97:0/2802337407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfc8105120 msgr2=0x7fbfc819b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfbfe00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfc8105120 0x7fbfc819b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.314+0000 7fbfbfe00640 1 -- 172.21.15.97:0/2802337407 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc819d2e0 con 0x7fbfc81061a0 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.315+0000 7fbfbfe00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 0x7fbfc819bfa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0005760 tx=0x7fbfb0003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.315+0000 7fbfc4c00640 1 -- 172.21.15.97:0/2802337407 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb003f070 con 0x7fbfc81061a0 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.315+0000 7fbfc4c00640 1 -- 172.21.15.97:0/2802337407 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbfb0031da0 con 0x7fbfc81061a0 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.315+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc819d570 con 0x7fbfc81061a0 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.315+0000 7fbfc4c00640 1 -- 172.21.15.97:0/2802337407 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0035030 con 0x7fbfc81061a0 2024-07-26T07:51:41.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.315+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc81a3b70 con 0x7fbfc81061a0 2024-07-26T07:51:41.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.316+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfc810c2d0 con 0x7fbfc81061a0 2024-07-26T07:51:41.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.319+0000 7fbfc4c00640 1 -- 172.21.15.97:0/2802337407 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fbfb0042070 con 0x7fbfc81061a0 2024-07-26T07:51:41.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.321+0000 7fbfc4c00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbfa80b3c50 0x7fbfa80b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:41.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.322+0000 7fbfc4c00640 1 -- 172.21.15.97:0/2802337407 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fbfb0104da0 con 0x7fbfc81061a0 2024-07-26T07:51:41.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.322+0000 7fbfc7400640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbfa80b3c50 0x7fbfa80b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:41.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.322+0000 7fbfc7400640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbfa80b3c50 0x7fbfa80b6160 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8005fd0 tx=0x7fbfb8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:41.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.324+0000 7fbfc4c00640 1 -- 172.21.15.97:0/2802337407 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbfb00cc330 con 0x7fbfc81061a0 2024-07-26T07:51:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:41 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2320181010' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-07-26T07:51:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:41 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2320181010' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-07-26T07:51:41.505 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:41 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2320181010' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-07-26T07:51:41.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.563+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fbfc8063390 con 0x7fbfa80b3c50 2024-07-26T07:51:41.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.607+0000 7fbfc4c00640 1 -- 172.21.15.97:0/2802337407 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fbfc8063390 con 0x7fbfa80b3c50 2024-07-26T07:51:41.608 INFO:teuthology.orchestra.run.smithi097.stdout:{} 2024-07-26T07:51:41.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbfa80b3c50 msgr2=0x7fbfa80b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:41.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbfa80b3c50 0x7fbfa80b6160 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8005fd0 tx=0x7fbfb8005850 comp rx=0 tx=0).stop 2024-07-26T07:51:41.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 msgr2=0x7fbfc819bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:41.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 0x7fbfc819bfa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0005760 tx=0x7fbfb0003d50 comp rx=0 tx=0).stop 2024-07-26T07:51:41.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 shutdown_connections 2024-07-26T07:51:41.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbfa80b3c50 0x7fbfa80b6160 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfc8106b40 0x7fbfc819cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbfc81061a0 0x7fbfc819bfa0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 --2- 172.21.15.97:0/2802337407 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbfc8105120 0x7fbfc819b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:41.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 >> 172.21.15.97:0/2802337407 conn(0x7fbfc8100a80 msgr2=0x7fbfc81094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:41.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 shutdown_connections 2024-07-26T07:51:41.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:41.610+0000 7fbfcce00640 1 -- 172.21.15.97:0/2802337407 wait complete. 2024-07-26T07:51:41.637 INFO:teuthology.run_tasks:Running task check-counter... 2024-07-26T07:51:41.658 INFO:teuthology.run_tasks:Running task background_exec... 2024-07-26T07:51:41.685 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-07-26T07:51:41.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-07-26T07:51:41.692 INFO:teuthology.run_tasks:Running task quiescer... 2024-07-26T07:51:41.731 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T07:51:41.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T07:51:42.001 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:51:41 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:51:41.571+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:51:42.001 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:51:41 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:51:41.571+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:51:42.001 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:51:41 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:51:41.571+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:51:42.001 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:51:41 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:51:41.571+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:51:42.001 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jul 26 07:51:41 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mgr-x[32459]: 2024-07-26T07:51:41.571+0000 7fcfe4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T07:51:42.166 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:42.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:42 smithi097 ceph-mon[32231]: pgmap v507: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:51:42.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:42 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2320181010' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-07-26T07:51:42.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:42 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:51:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:42 smithi115 ceph-mon[35364]: pgmap v507: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:51:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:42 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2320181010' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-07-26T07:51:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:42 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:51:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:42 smithi181 ceph-mon[35869]: pgmap v507: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T07:51:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:42 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2320181010' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-07-26T07:51:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:42 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:51:43.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:43 smithi097 ceph-mon[32231]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:51:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:43 smithi115 ceph-mon[35364]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:51:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:43 smithi181 ceph-mon[35869]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T07:51:44.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 -- 172.21.15.97:0/872497927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10f330 msgr2=0x7fdfdc10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:44.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/872497927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10f330 0x7fdfdc10f740 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009900 tx=0x7fdfc802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 -- 172.21.15.97:0/872497927 shutdown_connections 2024-07-26T07:51:44.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/872497927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc10fc80 0x7fdfdc1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/872497927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10f330 0x7fdfdc10f740 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/872497927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfdc1099f0 0x7fdfdc109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 -- 172.21.15.97:0/872497927 >> 172.21.15.97:0/872497927 conn(0x7fdfdc078ff0 msgr2=0x7fdfdc107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:44.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 -- 172.21.15.97:0/872497927 shutdown_connections 2024-07-26T07:51:44.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.187+0000 7fdfe0a00640 1 -- 172.21.15.97:0/872497927 wait complete. 2024-07-26T07:51:44.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.188+0000 7fdfe0a00640 1 Processor -- start 2024-07-26T07:51:44.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfe0a00640 1 -- start start 2024-07-26T07:51:44.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfe0a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfdc1099f0 0x7fdfdc1a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:44.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfe0a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc10f330 0x7fdfdc1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:44.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfe0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10fc80 0x7fdfdc1a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:44.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfe0a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc119840 con 0x7fdfdc10fc80 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfe0a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1196c0 con 0x7fdfdc1099f0 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfe0a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1199c0 con 0x7fdfdc10f330 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfdb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10fc80 0x7fdfdc1a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfdb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10fc80 0x7fdfdc1a5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44600/0 (socket says 172.21.15.97:44600) 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.189+0000 7fdfdb200640 1 -- 172.21.15.97:0/3151017733 learned_addr learned my addr 172.21.15.97:0/3151017733 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfd9e00640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc10f330 0x7fdfdc1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfda800640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfdc1099f0 0x7fdfdc1a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfdb200640 1 -- 172.21.15.97:0/3151017733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc10f330 msgr2=0x7fdfdc1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfdb200640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc10f330 0x7fdfdc1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfdb200640 1 -- 172.21.15.97:0/3151017733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfdc1099f0 msgr2=0x7fdfdc1a02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfdb200640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfdc1099f0 0x7fdfdc1a02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfdb200640 1 -- 172.21.15.97:0/3151017733 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc1a6220 con 0x7fdfdc10fc80 2024-07-26T07:51:44.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfdb200640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10fc80 0x7fdfdc1a5af0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400cad0 tx=0x7fdfd400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:44.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.190+0000 7fdfd3400640 1 -- 172.21.15.97:0/3151017733 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd400ae20 con 0x7fdfdc10fc80 2024-07-26T07:51:44.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.191+0000 7fdfd3400640 1 -- 172.21.15.97:0/3151017733 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfd4005020 con 0x7fdfdc10fc80 2024-07-26T07:51:44.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.191+0000 7fdfd3400640 1 -- 172.21.15.97:0/3151017733 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd400f6b0 con 0x7fdfdc10fc80 2024-07-26T07:51:44.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.191+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfdc1a6510 con 0x7fdfdc10fc80 2024-07-26T07:51:44.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.191+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfdc1acc70 con 0x7fdfdc10fc80 2024-07-26T07:51:44.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.192+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfdc109e00 con 0x7fdfdc10fc80 2024-07-26T07:51:44.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.194+0000 7fdfd3400640 1 -- 172.21.15.97:0/3151017733 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7fdfd40051d0 con 0x7fdfdc10fc80 2024-07-26T07:51:44.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.196+0000 7fdfd3400640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfc00b3ce0 0x7fdfc00b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:44.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.196+0000 7fdfd3400640 1 -- 172.21.15.97:0/3151017733 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fdfd40e4210 con 0x7fdfdc10fc80 2024-07-26T07:51:44.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.196+0000 7fdfda800640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfc00b3ce0 0x7fdfc00b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:44.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.197+0000 7fdfda800640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfc00b3ce0 0x7fdfc00b61f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0058c0 tx=0x7fdfcc00a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:44.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.198+0000 7fdfd3400640 1 -- 172.21.15.97:0/3151017733 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdfd4014030 con 0x7fdfdc10fc80 2024-07-26T07:51:44.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:44 smithi097 ceph-mon[32231]: pgmap v508: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-26T07:51:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:44 smithi097 ceph-mon[32231]: mgrmap e19: x(active, since 14m), standbys: z, y 2024-07-26T07:51:44.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.397+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdfdc063230 con 0x7fdfc00b3ce0 2024-07-26T07:51:44.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.404+0000 7fdfd3400640 1 -- 172.21.15.97:0/3151017733 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259003 (secure 0 0 0) 0x7fdfdc063230 con 0x7fdfc00b3ce0 2024-07-26T07:51:44.409 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:44.410 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-07-26T07:51:44.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfc00b3ce0 msgr2=0x7fdfc00b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:44.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfc00b3ce0 0x7fdfc00b61f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0058c0 tx=0x7fdfcc00a4c0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10fc80 msgr2=0x7fdfdc1a5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:44.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10fc80 0x7fdfdc1a5af0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400cad0 tx=0x7fdfd400cf90 comp rx=0 tx=0).stop 2024-07-26T07:51:44.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 shutdown_connections 2024-07-26T07:51:44.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfc00b3ce0 0x7fdfc00b61f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfdc10fc80 0x7fdfdc1a5af0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc10f330 0x7fdfdc1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 --2- 172.21.15.97:0/3151017733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfdc1099f0 0x7fdfdc1a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:44.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 >> 172.21.15.97:0/3151017733 conn(0x7fdfdc078ff0 msgr2=0x7fdfdc079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:44.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 shutdown_connections 2024-07-26T07:51:44.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:44.411+0000 7fdfe0a00640 1 -- 172.21.15.97:0/3151017733 wait complete. 2024-07-26T07:51:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:44 smithi115 ceph-mon[35364]: pgmap v508: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-26T07:51:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:44 smithi115 ceph-mon[35364]: mgrmap e19: x(active, since 14m), standbys: z, y 2024-07-26T07:51:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:44 smithi181 ceph-mon[35869]: pgmap v508: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-26T07:51:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:44 smithi181 ceph-mon[35869]: mgrmap e19: x(active, since 14m), standbys: z, y 2024-07-26T07:51:45.352 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-07-26T07:51:44.224066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664737,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2249,"num_read_kb":1226,"num_write":395,"num_write_kb":2310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2953212,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":824,"ondisk_log_size":824,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":404164,"kb_used_data":13412,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537660,"statfs":{"total":1151940427776,"available":1151526563840,"internally_reserved":0,"allocated":13733888,"data_stored":7533150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7170,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":75,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004966"},"pg_stats":[{"pgid":"3.3f","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090598+0000","last_change":"2024-07-26T07:47:45.843670+0000","last_active":"2024-07-26T07:49:48.090598+0000","last_peered":"2024-07-26T07:49:48.090598+0000","last_clean":"2024-07-26T07:49:48.090598+0000","last_became_active":"2024-07-26T07:47:45.841137+0000","last_became_peered":"2024-07-26T07:47:45.841137+0000","last_unstale":"2024-07-26T07:49:48.090598+0000","last_undegraded":"2024-07-26T07:49:48.090598+0000","last_fullsized":"2024-07-26T07:49:48.090598+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:38.063503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085904+0000","last_change":"2024-07-26T07:47:45.849326+0000","last_active":"2024-07-26T07:49:48.085904+0000","last_peered":"2024-07-26T07:49:48.085904+0000","last_clean":"2024-07-26T07:49:48.085904+0000","last_became_active":"2024-07-26T07:47:45.847761+0000","last_became_peered":"2024-07-26T07:47:45.847761+0000","last_unstale":"2024-07-26T07:49:48.085904+0000","last_undegraded":"2024-07-26T07:49:48.085904+0000","last_fullsized":"2024-07-26T07:49:48.085904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:31.098794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092147+0000","last_change":"2024-07-26T07:47:45.852343+0000","last_active":"2024-07-26T07:49:48.092147+0000","last_peered":"2024-07-26T07:49:48.092147+0000","last_clean":"2024-07-26T07:49:48.092147+0000","last_became_active":"2024-07-26T07:47:45.851950+0000","last_became_peered":"2024-07-26T07:47:45.851950+0000","last_unstale":"2024-07-26T07:49:48.092147+0000","last_undegraded":"2024-07-26T07:49:48.092147+0000","last_fullsized":"2024-07-26T07:49:48.092147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:41.030903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091026+0000","last_change":"2024-07-26T07:47:45.857042+0000","last_active":"2024-07-26T07:49:48.091026+0000","last_peered":"2024-07-26T07:49:48.091026+0000","last_clean":"2024-07-26T07:49:48.091026+0000","last_became_active":"2024-07-26T07:47:45.855145+0000","last_became_peered":"2024-07-26T07:47:45.855145+0000","last_unstale":"2024-07-26T07:49:48.091026+0000","last_undegraded":"2024-07-26T07:49:48.091026+0000","last_fullsized":"2024-07-26T07:49:48.091026+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:32.233019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092139+0000","last_change":"2024-07-26T07:47:45.838560+0000","last_active":"2024-07-26T07:49:48.092139+0000","last_peered":"2024-07-26T07:49:48.092139+0000","last_clean":"2024-07-26T07:49:48.092139+0000","last_became_active":"2024-07-26T07:47:45.835578+0000","last_became_peered":"2024-07-26T07:47:45.835578+0000","last_unstale":"2024-07-26T07:49:48.092139+0000","last_undegraded":"2024-07-26T07:49:48.092139+0000","last_fullsized":"2024-07-26T07:49:48.092139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:09:40.445625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.089953+0000","last_change":"2024-07-26T07:47:45.843980+0000","last_active":"2024-07-26T07:49:48.089953+0000","last_peered":"2024-07-26T07:49:48.089953+0000","last_clean":"2024-07-26T07:49:48.089953+0000","last_became_active":"2024-07-26T07:47:45.843546+0000","last_became_peered":"2024-07-26T07:47:45.843546+0000","last_unstale":"2024-07-26T07:49:48.089953+0000","last_undegraded":"2024-07-26T07:49:48.089953+0000","last_fullsized":"2024-07-26T07:49:48.089953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:06.839136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093383+0000","last_change":"2024-07-26T07:47:45.854427+0000","last_active":"2024-07-26T07:49:48.093383+0000","last_peered":"2024-07-26T07:49:48.093383+0000","last_clean":"2024-07-26T07:49:48.093383+0000","last_became_active":"2024-07-26T07:47:45.853495+0000","last_became_peered":"2024-07-26T07:47:45.853495+0000","last_unstale":"2024-07-26T07:49:48.093383+0000","last_undegraded":"2024-07-26T07:49:48.093383+0000","last_fullsized":"2024-07-26T07:49:48.093383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:34:34.110581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096799+0000","last_change":"2024-07-26T07:47:45.847732+0000","last_active":"2024-07-26T07:49:48.096799+0000","last_peered":"2024-07-26T07:49:48.096799+0000","last_clean":"2024-07-26T07:49:48.096799+0000","last_became_active":"2024-07-26T07:47:45.845583+0000","last_became_peered":"2024-07-26T07:47:45.845583+0000","last_unstale":"2024-07-26T07:49:48.096799+0000","last_undegraded":"2024-07-26T07:49:48.096799+0000","last_fullsized":"2024-07-26T07:49:48.096799+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:07:09.370076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.108141+0000","last_change":"2024-07-26T07:47:45.849962+0000","last_active":"2024-07-26T07:49:48.108141+0000","last_peered":"2024-07-26T07:49:48.108141+0000","last_clean":"2024-07-26T07:49:48.108141+0000","last_became_active":"2024-07-26T07:47:45.845766+0000","last_became_peered":"2024-07-26T07:47:45.845766+0000","last_unstale":"2024-07-26T07:49:48.108141+0000","last_undegraded":"2024-07-26T07:49:48.108141+0000","last_fullsized":"2024-07-26T07:49:48.108141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:56.602825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086656+0000","last_change":"2024-07-26T07:47:45.857287+0000","last_active":"2024-07-26T07:49:48.086656+0000","last_peered":"2024-07-26T07:49:48.086656+0000","last_clean":"2024-07-26T07:49:48.086656+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.086656+0000","last_undegraded":"2024-07-26T07:49:48.086656+0000","last_fullsized":"2024-07-26T07:49:48.086656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:38:24.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.107214+0000","last_change":"2024-07-26T07:47:45.858369+0000","last_active":"2024-07-26T07:49:48.107214+0000","last_peered":"2024-07-26T07:49:48.107214+0000","last_clean":"2024-07-26T07:49:48.107214+0000","last_became_active":"2024-07-26T07:47:45.856685+0000","last_became_peered":"2024-07-26T07:47:45.856685+0000","last_unstale":"2024-07-26T07:49:48.107214+0000","last_undegraded":"2024-07-26T07:49:48.107214+0000","last_fullsized":"2024-07-26T07:49:48.107214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:44.877953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.083667+0000","last_change":"2024-07-26T07:48:20.001394+0000","last_active":"2024-07-26T07:49:48.083667+0000","last_peered":"2024-07-26T07:49:48.083667+0000","last_clean":"2024-07-26T07:49:48.083667+0000","last_became_active":"2024-07-26T07:48:20.000751+0000","last_became_peered":"2024-07-26T07:48:20.000751+0000","last_unstale":"2024-07-26T07:49:48.083667+0000","last_undegraded":"2024-07-26T07:49:48.083667+0000","last_fullsized":"2024-07-26T07:49:48.083667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:26:24.513652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.097228+0000","last_change":"2024-07-26T07:47:45.847931+0000","last_active":"2024-07-26T07:49:48.097228+0000","last_peered":"2024-07-26T07:49:48.097228+0000","last_clean":"2024-07-26T07:49:48.097228+0000","last_became_active":"2024-07-26T07:47:45.847484+0000","last_became_peered":"2024-07-26T07:47:45.847484+0000","last_unstale":"2024-07-26T07:49:48.097228+0000","last_undegraded":"2024-07-26T07:49:48.097228+0000","last_fullsized":"2024-07-26T07:49:48.097228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:04.498360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.087242+0000","last_change":"2024-07-26T07:47:45.846484+0000","last_active":"2024-07-26T07:49:48.087242+0000","last_peered":"2024-07-26T07:49:48.087242+0000","last_clean":"2024-07-26T07:49:48.087242+0000","last_became_active":"2024-07-26T07:47:45.844772+0000","last_became_peered":"2024-07-26T07:47:45.844772+0000","last_unstale":"2024-07-26T07:49:48.087242+0000","last_undegraded":"2024-07-26T07:49:48.087242+0000","last_fullsized":"2024-07-26T07:49:48.087242+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:54.318026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088688+0000","last_change":"2024-07-26T07:47:45.849840+0000","last_active":"2024-07-26T07:49:48.088688+0000","last_peered":"2024-07-26T07:49:48.088688+0000","last_clean":"2024-07-26T07:49:48.088688+0000","last_became_active":"2024-07-26T07:47:45.848744+0000","last_became_peered":"2024-07-26T07:47:45.848744+0000","last_unstale":"2024-07-26T07:49:48.088688+0000","last_undegraded":"2024-07-26T07:49:48.088688+0000","last_fullsized":"2024-07-26T07:49:48.088688+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:44.675382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111006+0000","last_change":"2024-07-26T07:47:45.847176+0000","last_active":"2024-07-26T07:49:48.111006+0000","last_peered":"2024-07-26T07:49:48.111006+0000","last_clean":"2024-07-26T07:49:48.111006+0000","last_became_active":"2024-07-26T07:47:45.845936+0000","last_became_peered":"2024-07-26T07:47:45.845936+0000","last_unstale":"2024-07-26T07:49:48.111006+0000","last_undegraded":"2024-07-26T07:49:48.111006+0000","last_fullsized":"2024-07-26T07:49:48.111006+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:02.245098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090942+0000","last_change":"2024-07-26T07:47:45.853210+0000","last_active":"2024-07-26T07:49:48.090942+0000","last_peered":"2024-07-26T07:49:48.090942+0000","last_clean":"2024-07-26T07:49:48.090942+0000","last_became_active":"2024-07-26T07:47:45.852766+0000","last_became_peered":"2024-07-26T07:47:45.852766+0000","last_unstale":"2024-07-26T07:49:48.090942+0000","last_undegraded":"2024-07-26T07:49:48.090942+0000","last_fullsized":"2024-07-26T07:49:48.090942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:57.406748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.103824+0000","last_change":"2024-07-26T07:47:45.850699+0000","last_active":"2024-07-26T07:49:48.103824+0000","last_peered":"2024-07-26T07:49:48.103824+0000","last_clean":"2024-07-26T07:49:48.103824+0000","last_became_active":"2024-07-26T07:47:45.850314+0000","last_became_peered":"2024-07-26T07:47:45.850314+0000","last_unstale":"2024-07-26T07:49:48.103824+0000","last_undegraded":"2024-07-26T07:49:48.103824+0000","last_fullsized":"2024-07-26T07:49:48.103824+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:09.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090575+0000","last_change":"2024-07-26T07:47:45.851063+0000","last_active":"2024-07-26T07:49:48.090575+0000","last_peered":"2024-07-26T07:49:48.090575+0000","last_clean":"2024-07-26T07:49:48.090575+0000","last_became_active":"2024-07-26T07:47:45.850164+0000","last_became_peered":"2024-07-26T07:47:45.850164+0000","last_unstale":"2024-07-26T07:49:48.090575+0000","last_undegraded":"2024-07-26T07:49:48.090575+0000","last_fullsized":"2024-07-26T07:49:48.090575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:31.597551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088724+0000","last_change":"2024-07-26T07:47:45.861207+0000","last_active":"2024-07-26T07:49:48.088724+0000","last_peered":"2024-07-26T07:49:48.088724+0000","last_clean":"2024-07-26T07:49:48.088724+0000","last_became_active":"2024-07-26T07:47:45.860179+0000","last_became_peered":"2024-07-26T07:47:45.860179+0000","last_unstale":"2024-07-26T07:49:48.088724+0000","last_undegraded":"2024-07-26T07:49:48.088724+0000","last_fullsized":"2024-07-26T07:49:48.088724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:12:51.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101706+0000","last_change":"2024-07-26T07:47:45.846197+0000","last_active":"2024-07-26T07:49:48.101706+0000","last_peered":"2024-07-26T07:49:48.101706+0000","last_clean":"2024-07-26T07:49:48.101706+0000","last_became_active":"2024-07-26T07:47:45.845238+0000","last_became_peered":"2024-07-26T07:47:45.845238+0000","last_unstale":"2024-07-26T07:49:48.101706+0000","last_undegraded":"2024-07-26T07:49:48.101706+0000","last_fullsized":"2024-07-26T07:49:48.101706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:29.900684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112421+0000","last_change":"2024-07-26T07:47:45.847963+0000","last_active":"2024-07-26T07:49:48.112421+0000","last_peered":"2024-07-26T07:49:48.112421+0000","last_clean":"2024-07-26T07:49:48.112421+0000","last_became_active":"2024-07-26T07:47:45.847513+0000","last_became_peered":"2024-07-26T07:47:45.847513+0000","last_unstale":"2024-07-26T07:49:48.112421+0000","last_undegraded":"2024-07-26T07:49:48.112421+0000","last_fullsized":"2024-07-26T07:49:48.112421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:51.082816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092151+0000","last_change":"2024-07-26T07:47:45.853273+0000","last_active":"2024-07-26T07:49:48.092151+0000","last_peered":"2024-07-26T07:49:48.092151+0000","last_clean":"2024-07-26T07:49:48.092151+0000","last_became_active":"2024-07-26T07:47:45.852894+0000","last_became_peered":"2024-07-26T07:47:45.852894+0000","last_unstale":"2024-07-26T07:49:48.092151+0000","last_undegraded":"2024-07-26T07:49:48.092151+0000","last_fullsized":"2024-07-26T07:49:48.092151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:57:12.945228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093387+0000","last_change":"2024-07-26T07:48:19.931938+0000","last_active":"2024-07-26T07:49:48.093387+0000","last_peered":"2024-07-26T07:49:48.093387+0000","last_clean":"2024-07-26T07:49:48.093387+0000","last_became_active":"2024-07-26T07:48:19.930976+0000","last_became_peered":"2024-07-26T07:48:19.930976+0000","last_unstale":"2024-07-26T07:49:48.093387+0000","last_undegraded":"2024-07-26T07:49:48.093387+0000","last_fullsized":"2024-07-26T07:49:48.093387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:59.610464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090184+0000","last_change":"2024-07-26T07:47:45.849736+0000","last_active":"2024-07-26T07:49:48.090184+0000","last_peered":"2024-07-26T07:49:48.090184+0000","last_clean":"2024-07-26T07:49:48.090184+0000","last_became_active":"2024-07-26T07:47:45.849335+0000","last_became_peered":"2024-07-26T07:47:45.849335+0000","last_unstale":"2024-07-26T07:49:48.090184+0000","last_undegraded":"2024-07-26T07:49:48.090184+0000","last_fullsized":"2024-07-26T07:49:48.090184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:20:12.758331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104678+0000","last_change":"2024-07-26T07:47:45.847250+0000","last_active":"2024-07-26T07:49:48.104678+0000","last_peered":"2024-07-26T07:49:48.104678+0000","last_clean":"2024-07-26T07:49:48.104678+0000","last_became_active":"2024-07-26T07:47:45.845564+0000","last_became_peered":"2024-07-26T07:47:45.845564+0000","last_unstale":"2024-07-26T07:49:48.104678+0000","last_undegraded":"2024-07-26T07:49:48.104678+0000","last_fullsized":"2024-07-26T07:49:48.104678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:03.616782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086629+0000","last_change":"2024-07-26T07:47:45.850462+0000","last_active":"2024-07-26T07:49:48.086629+0000","last_peered":"2024-07-26T07:49:48.086629+0000","last_clean":"2024-07-26T07:49:48.086629+0000","last_became_active":"2024-07-26T07:47:45.850036+0000","last_became_peered":"2024-07-26T07:47:45.850036+0000","last_unstale":"2024-07-26T07:49:48.086629+0000","last_undegraded":"2024-07-26T07:49:48.086629+0000","last_fullsized":"2024-07-26T07:49:48.086629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:38.374782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.094143+0000","last_change":"2024-07-26T07:47:45.859294+0000","last_active":"2024-07-26T07:49:48.094143+0000","last_peered":"2024-07-26T07:49:48.094143+0000","last_clean":"2024-07-26T07:49:48.094143+0000","last_became_active":"2024-07-26T07:47:45.858812+0000","last_became_peered":"2024-07-26T07:47:45.858812+0000","last_unstale":"2024-07-26T07:49:48.094143+0000","last_undegraded":"2024-07-26T07:49:48.094143+0000","last_fullsized":"2024-07-26T07:49:48.094143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:11.689717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085416+0000","last_change":"2024-07-26T07:48:19.930943+0000","last_active":"2024-07-26T07:49:48.085416+0000","last_peered":"2024-07-26T07:49:48.085416+0000","last_clean":"2024-07-26T07:49:48.085416+0000","last_became_active":"2024-07-26T07:48:19.929758+0000","last_became_peered":"2024-07-26T07:48:19.929758+0000","last_unstale":"2024-07-26T07:49:48.085416+0000","last_undegraded":"2024-07-26T07:49:48.085416+0000","last_fullsized":"2024-07-26T07:49:48.085416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:29:27.657376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090071+0000","last_change":"2024-07-26T07:47:45.846867+0000","last_active":"2024-07-26T07:49:48.090071+0000","last_peered":"2024-07-26T07:49:48.090071+0000","last_clean":"2024-07-26T07:49:48.090071+0000","last_became_active":"2024-07-26T07:47:45.846455+0000","last_became_peered":"2024-07-26T07:47:45.846455+0000","last_unstale":"2024-07-26T07:49:48.090071+0000","last_undegraded":"2024-07-26T07:49:48.090071+0000","last_fullsized":"2024-07-26T07:49:48.090071+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:14.997452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092573+0000","last_change":"2024-07-26T07:47:45.846179+0000","last_active":"2024-07-26T07:49:48.092573+0000","last_peered":"2024-07-26T07:49:48.092573+0000","last_clean":"2024-07-26T07:49:48.092573+0000","last_became_active":"2024-07-26T07:47:45.844880+0000","last_became_peered":"2024-07-26T07:47:45.844880+0000","last_unstale":"2024-07-26T07:49:48.092573+0000","last_undegraded":"2024-07-26T07:49:48.092573+0000","last_fullsized":"2024-07-26T07:49:48.092573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:00:50.899573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091429+0000","last_change":"2024-07-26T07:48:19.935196+0000","last_active":"2024-07-26T07:49:48.091429+0000","last_peered":"2024-07-26T07:49:48.091429+0000","last_clean":"2024-07-26T07:49:48.091429+0000","last_became_active":"2024-07-26T07:48:19.933730+0000","last_became_peered":"2024-07-26T07:48:19.933730+0000","last_unstale":"2024-07-26T07:49:48.091429+0000","last_undegraded":"2024-07-26T07:49:48.091429+0000","last_fullsized":"2024-07-26T07:49:48.091429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:10.305122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090542+0000","last_change":"2024-07-26T07:47:45.843360+0000","last_active":"2024-07-26T07:49:48.090542+0000","last_peered":"2024-07-26T07:49:48.090542+0000","last_clean":"2024-07-26T07:49:48.090542+0000","last_became_active":"2024-07-26T07:47:45.842971+0000","last_became_peered":"2024-07-26T07:47:45.842971+0000","last_unstale":"2024-07-26T07:49:48.090542+0000","last_undegraded":"2024-07-26T07:49:48.090542+0000","last_fullsized":"2024-07-26T07:49:48.090542+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:58.687638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":253,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:38.537839+0000","last_change":"2024-07-26T07:47:41.787760+0000","last_active":"2024-07-26T07:51:38.537839+0000","last_peered":"2024-07-26T07:51:38.537839+0000","last_clean":"2024-07-26T07:51:38.537839+0000","last_became_active":"2024-07-26T07:47:41.787197+0000","last_became_peered":"2024-07-26T07:47:41.787197+0000","last_unstale":"2024-07-26T07:51:38.537839+0000","last_undegraded":"2024-07-26T07:51:38.537839+0000","last_fullsized":"2024-07-26T07:51:38.537839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:14.597759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":215,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092722+0000","last_change":"2024-07-26T07:47:45.846822+0000","last_active":"2024-07-26T07:49:48.092722+0000","last_peered":"2024-07-26T07:49:48.092722+0000","last_clean":"2024-07-26T07:49:48.092722+0000","last_became_active":"2024-07-26T07:47:45.846398+0000","last_became_peered":"2024-07-26T07:47:45.846398+0000","last_unstale":"2024-07-26T07:49:48.092722+0000","last_undegraded":"2024-07-26T07:49:48.092722+0000","last_fullsized":"2024-07-26T07:49:48.092722+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:33:34.727843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.311933+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:49:20.311933+0000","last_peered":"2024-07-26T07:49:20.311933+0000","last_clean":"2024-07-26T07:49:20.311933+0000","last_became_active":"2024-07-26T07:47:41.782077+0000","last_became_peered":"2024-07-26T07:47:41.782077+0000","last_unstale":"2024-07-26T07:49:20.311933+0000","last_undegraded":"2024-07-26T07:49:20.311933+0000","last_fullsized":"2024-07-26T07:49:20.311933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:42.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429076+0000","last_change":"2024-07-26T07:47:41.784615+0000","last_active":"2024-07-26T07:49:20.429076+0000","last_peered":"2024-07-26T07:49:20.429076+0000","last_clean":"2024-07-26T07:49:20.429076+0000","last_became_active":"2024-07-26T07:47:41.783945+0000","last_became_peered":"2024-07-26T07:47:41.783945+0000","last_unstale":"2024-07-26T07:49:20.429076+0000","last_undegraded":"2024-07-26T07:49:20.429076+0000","last_fullsized":"2024-07-26T07:49:20.429076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:42.618524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095869+0000","last_change":"2024-07-26T07:47:45.838774+0000","last_active":"2024-07-26T07:49:48.095869+0000","last_peered":"2024-07-26T07:49:48.095869+0000","last_clean":"2024-07-26T07:49:48.095869+0000","last_became_active":"2024-07-26T07:47:45.837707+0000","last_became_peered":"2024-07-26T07:47:45.837707+0000","last_unstale":"2024-07-26T07:49:48.095869+0000","last_undegraded":"2024-07-26T07:49:48.095869+0000","last_fullsized":"2024-07-26T07:49:48.095869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:59.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297619+0000","last_change":"2024-07-26T07:47:41.792031+0000","last_active":"2024-07-26T07:49:20.297619+0000","last_peered":"2024-07-26T07:49:20.297619+0000","last_clean":"2024-07-26T07:49:20.297619+0000","last_became_active":"2024-07-26T07:47:41.785955+0000","last_became_peered":"2024-07-26T07:47:41.785955+0000","last_unstale":"2024-07-26T07:49:20.297619+0000","last_undegraded":"2024-07-26T07:49:20.297619+0000","last_fullsized":"2024-07-26T07:49:20.297619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:01:47.073970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.094145+0000","last_change":"2024-07-26T07:47:45.844595+0000","last_active":"2024-07-26T07:49:48.094145+0000","last_peered":"2024-07-26T07:49:48.094145+0000","last_clean":"2024-07-26T07:49:48.094145+0000","last_became_active":"2024-07-26T07:47:45.843381+0000","last_became_peered":"2024-07-26T07:47:45.843381+0000","last_unstale":"2024-07-26T07:49:48.094145+0000","last_undegraded":"2024-07-26T07:49:48.094145+0000","last_fullsized":"2024-07-26T07:49:48.094145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:39.692463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'7","reported_seq":467,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:41.545148+0000","last_change":"2024-07-26T07:47:41.788239+0000","last_active":"2024-07-26T07:51:41.545148+0000","last_peered":"2024-07-26T07:51:41.545148+0000","last_clean":"2024-07-26T07:51:41.545148+0000","last_became_active":"2024-07-26T07:47:41.787470+0000","last_became_peered":"2024-07-26T07:47:41.787470+0000","last_unstale":"2024-07-26T07:51:41.545148+0000","last_undegraded":"2024-07-26T07:51:41.545148+0000","last_fullsized":"2024-07-26T07:51:41.545148+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:03.129563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":428,"num_read_kb":428,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112494+0000","last_change":"2024-07-26T07:47:45.852794+0000","last_active":"2024-07-26T07:49:48.112494+0000","last_peered":"2024-07-26T07:49:48.112494+0000","last_clean":"2024-07-26T07:49:48.112494+0000","last_became_active":"2024-07-26T07:47:45.849699+0000","last_became_peered":"2024-07-26T07:47:45.849699+0000","last_unstale":"2024-07-26T07:49:48.112494+0000","last_undegraded":"2024-07-26T07:49:48.112494+0000","last_fullsized":"2024-07-26T07:49:48.112494+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:06:31.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.315536+0000","last_change":"2024-07-26T07:47:41.789414+0000","last_active":"2024-07-26T07:49:20.315536+0000","last_peered":"2024-07-26T07:49:20.315536+0000","last_clean":"2024-07-26T07:49:20.315536+0000","last_became_active":"2024-07-26T07:47:41.788679+0000","last_became_peered":"2024-07-26T07:47:41.788679+0000","last_unstale":"2024-07-26T07:49:20.315536+0000","last_undegraded":"2024-07-26T07:49:20.315536+0000","last_fullsized":"2024-07-26T07:49:20.315536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:51.809997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099375+0000","last_change":"2024-07-26T07:48:19.937603+0000","last_active":"2024-07-26T07:49:48.099375+0000","last_peered":"2024-07-26T07:49:48.099375+0000","last_clean":"2024-07-26T07:49:48.099375+0000","last_became_active":"2024-07-26T07:48:19.936889+0000","last_became_peered":"2024-07-26T07:48:19.936889+0000","last_unstale":"2024-07-26T07:49:48.099375+0000","last_undegraded":"2024-07-26T07:49:48.099375+0000","last_fullsized":"2024-07-26T07:49:48.099375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:46:35.138825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298028+0000","last_change":"2024-07-26T07:47:42.564288+0000","last_active":"2024-07-26T07:49:20.298028+0000","last_peered":"2024-07-26T07:49:20.298028+0000","last_clean":"2024-07-26T07:49:20.298028+0000","last_became_active":"2024-07-26T07:47:42.561766+0000","last_became_peered":"2024-07-26T07:47:42.561766+0000","last_unstale":"2024-07-26T07:49:20.298028+0000","last_undegraded":"2024-07-26T07:49:20.298028+0000","last_fullsized":"2024-07-26T07:49:20.298028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:16.414030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.113511+0000","last_change":"2024-07-26T07:47:45.861895+0000","last_active":"2024-07-26T07:49:48.113511+0000","last_peered":"2024-07-26T07:49:48.113511+0000","last_clean":"2024-07-26T07:49:48.113511+0000","last_became_active":"2024-07-26T07:47:45.861439+0000","last_became_peered":"2024-07-26T07:47:45.861439+0000","last_unstale":"2024-07-26T07:49:48.113511+0000","last_undegraded":"2024-07-26T07:49:48.113511+0000","last_fullsized":"2024-07-26T07:49:48.113511+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:26.276453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'37","reported_seq":508,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:41.548437+0000","last_change":"2024-07-26T07:47:41.789977+0000","last_active":"2024-07-26T07:51:41.548437+0000","last_peered":"2024-07-26T07:51:41.548437+0000","last_clean":"2024-07-26T07:51:41.548437+0000","last_became_active":"2024-07-26T07:47:41.789488+0000","last_became_peered":"2024-07-26T07:47:41.789488+0000","last_unstale":"2024-07-26T07:51:41.548437+0000","last_undegraded":"2024-07-26T07:51:41.548437+0000","last_fullsized":"2024-07-26T07:51:41.548437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:33.059279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65209,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":439,"num_read_kb":278,"num_write":39,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098321+0000","last_change":"2024-07-26T07:47:45.854772+0000","last_active":"2024-07-26T07:49:48.098321+0000","last_peered":"2024-07-26T07:49:48.098321+0000","last_clean":"2024-07-26T07:49:48.098321+0000","last_became_active":"2024-07-26T07:47:45.854354+0000","last_became_peered":"2024-07-26T07:47:45.854354+0000","last_unstale":"2024-07-26T07:49:48.098321+0000","last_undegraded":"2024-07-26T07:49:48.098321+0000","last_fullsized":"2024-07-26T07:49:48.098321+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:31:54.856407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313468+0000","last_change":"2024-07-26T07:47:41.789752+0000","last_active":"2024-07-26T07:49:20.313468+0000","last_peered":"2024-07-26T07:49:20.313468+0000","last_clean":"2024-07-26T07:49:20.313468+0000","last_became_active":"2024-07-26T07:47:41.788952+0000","last_became_peered":"2024-07-26T07:47:41.788952+0000","last_unstale":"2024-07-26T07:49:20.313468+0000","last_undegraded":"2024-07-26T07:49:20.313468+0000","last_fullsized":"2024-07-26T07:49:20.313468+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:45.141693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101283+0000","last_change":"2024-07-26T07:47:45.845120+0000","last_active":"2024-07-26T07:49:48.101283+0000","last_peered":"2024-07-26T07:49:48.101283+0000","last_clean":"2024-07-26T07:49:48.101283+0000","last_became_active":"2024-07-26T07:47:45.844699+0000","last_became_peered":"2024-07-26T07:47:45.844699+0000","last_unstale":"2024-07-26T07:49:48.101283+0000","last_undegraded":"2024-07-26T07:49:48.101283+0000","last_fullsized":"2024-07-26T07:49:48.101283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:23:42.130846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298579+0000","last_change":"2024-07-26T07:47:41.790765+0000","last_active":"2024-07-26T07:49:20.298579+0000","last_peered":"2024-07-26T07:49:20.298579+0000","last_clean":"2024-07-26T07:49:20.298579+0000","last_became_active":"2024-07-26T07:47:41.790288+0000","last_became_peered":"2024-07-26T07:47:41.790288+0000","last_unstale":"2024-07-26T07:49:20.298579+0000","last_undegraded":"2024-07-26T07:49:20.298579+0000","last_fullsized":"2024-07-26T07:49:20.298579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:05.734468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.102627+0000","last_change":"2024-07-26T07:47:45.854648+0000","last_active":"2024-07-26T07:49:48.102627+0000","last_peered":"2024-07-26T07:49:48.102627+0000","last_clean":"2024-07-26T07:49:48.102627+0000","last_became_active":"2024-07-26T07:47:45.854240+0000","last_became_peered":"2024-07-26T07:47:45.854240+0000","last_unstale":"2024-07-26T07:49:48.102627+0000","last_undegraded":"2024-07-26T07:49:48.102627+0000","last_fullsized":"2024-07-26T07:49:48.102627+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:41.775760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":268,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:38.774261+0000","last_change":"2024-07-26T07:49:20.331153+0000","last_active":"2024-07-26T07:51:38.774261+0000","last_peered":"2024-07-26T07:51:38.774261+0000","last_clean":"2024-07-26T07:51:38.774261+0000","last_became_active":"2024-07-26T07:49:20.315423+0000","last_became_peered":"2024-07-26T07:49:20.315423+0000","last_unstale":"2024-07-26T07:51:38.774261+0000","last_undegraded":"2024-07-26T07:51:38.774261+0000","last_fullsized":"2024-07-26T07:51:38.774261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:24.476818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":214,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.100845+0000","last_change":"2024-07-26T07:47:45.848731+0000","last_active":"2024-07-26T07:49:48.100845+0000","last_peered":"2024-07-26T07:49:48.100845+0000","last_clean":"2024-07-26T07:49:48.100845+0000","last_became_active":"2024-07-26T07:47:45.846663+0000","last_became_peered":"2024-07-26T07:47:45.846663+0000","last_unstale":"2024-07-26T07:49:48.100845+0000","last_undegraded":"2024-07-26T07:49:48.100845+0000","last_fullsized":"2024-07-26T07:49:48.100845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:05.251423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:23.742120+0000","last_change":"2024-07-26T07:49:20.355238+0000","last_active":"2024-07-26T07:49:23.742120+0000","last_peered":"2024-07-26T07:49:23.742120+0000","last_clean":"2024-07-26T07:49:23.742120+0000","last_became_active":"2024-07-26T07:49:20.316591+0000","last_became_peered":"2024-07-26T07:49:20.316591+0000","last_unstale":"2024-07-26T07:49:23.742120+0000","last_undegraded":"2024-07-26T07:49:23.742120+0000","last_fullsized":"2024-07-26T07:49:23.742120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:12.739473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088622+0000","last_change":"2024-07-26T07:47:45.842572+0000","last_active":"2024-07-26T07:49:48.088622+0000","last_peered":"2024-07-26T07:49:48.088622+0000","last_clean":"2024-07-26T07:49:48.088622+0000","last_became_active":"2024-07-26T07:47:45.842116+0000","last_became_peered":"2024-07-26T07:47:45.842116+0000","last_unstale":"2024-07-26T07:49:48.088622+0000","last_undegraded":"2024-07-26T07:49:48.088622+0000","last_fullsized":"2024-07-26T07:49:48.088622+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:06.913737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:59.385245+0000","last_change":"2024-07-26T07:47:41.790215+0000","last_active":"2024-07-26T07:49:59.385245+0000","last_peered":"2024-07-26T07:49:59.385245+0000","last_clean":"2024-07-26T07:49:59.385245+0000","last_became_active":"2024-07-26T07:47:41.789691+0000","last_became_peered":"2024-07-26T07:47:41.789691+0000","last_unstale":"2024-07-26T07:49:59.385245+0000","last_undegraded":"2024-07-26T07:49:59.385245+0000","last_fullsized":"2024-07-26T07:49:59.385245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:01:47.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086806+0000","last_change":"2024-07-26T07:47:45.860887+0000","last_active":"2024-07-26T07:49:48.086806+0000","last_peered":"2024-07-26T07:49:48.086806+0000","last_clean":"2024-07-26T07:49:48.086806+0000","last_became_active":"2024-07-26T07:47:45.860315+0000","last_became_peered":"2024-07-26T07:47:45.860315+0000","last_unstale":"2024-07-26T07:49:48.086806+0000","last_undegraded":"2024-07-26T07:49:48.086806+0000","last_fullsized":"2024-07-26T07:49:48.086806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:17:02.231206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429700+0000","last_change":"2024-07-26T07:48:20.022885+0000","last_active":"2024-07-26T07:49:20.429700+0000","last_peered":"2024-07-26T07:49:20.429700+0000","last_clean":"2024-07-26T07:49:20.429700+0000","last_became_active":"2024-07-26T07:48:20.001946+0000","last_became_peered":"2024-07-26T07:48:20.001946+0000","last_unstale":"2024-07-26T07:49:20.429700+0000","last_undegraded":"2024-07-26T07:49:20.429700+0000","last_fullsized":"2024-07-26T07:49:20.429700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:45.892886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098632+0000","last_change":"2024-07-26T07:47:45.858267+0000","last_active":"2024-07-26T07:49:48.098632+0000","last_peered":"2024-07-26T07:49:48.098632+0000","last_clean":"2024-07-26T07:49:48.098632+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.098632+0000","last_undegraded":"2024-07-26T07:49:48.098632+0000","last_fullsized":"2024-07-26T07:49:48.098632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:32.055977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298413+0000","last_change":"2024-07-26T07:47:41.789846+0000","last_active":"2024-07-26T07:49:20.298413+0000","last_peered":"2024-07-26T07:49:20.298413+0000","last_clean":"2024-07-26T07:49:20.298413+0000","last_became_active":"2024-07-26T07:47:41.789237+0000","last_became_peered":"2024-07-26T07:47:41.789237+0000","last_unstale":"2024-07-26T07:49:20.298413+0000","last_undegraded":"2024-07-26T07:49:20.298413+0000","last_fullsized":"2024-07-26T07:49:20.298413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:39.031047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095798+0000","last_change":"2024-07-26T07:47:45.846294+0000","last_active":"2024-07-26T07:49:48.095798+0000","last_peered":"2024-07-26T07:49:48.095798+0000","last_clean":"2024-07-26T07:49:48.095798+0000","last_became_active":"2024-07-26T07:47:45.845861+0000","last_became_peered":"2024-07-26T07:47:45.845861+0000","last_unstale":"2024-07-26T07:49:48.095798+0000","last_undegraded":"2024-07-26T07:49:48.095798+0000","last_fullsized":"2024-07-26T07:49:48.095798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:56:10.594430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'8","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:39.539913+0000","last_change":"2024-07-26T07:47:42.563412+0000","last_active":"2024-07-26T07:51:39.539913+0000","last_peered":"2024-07-26T07:51:39.539913+0000","last_clean":"2024-07-26T07:51:39.539913+0000","last_became_active":"2024-07-26T07:47:42.562364+0000","last_became_peered":"2024-07-26T07:47:42.562364+0000","last_unstale":"2024-07-26T07:51:39.539913+0000","last_undegraded":"2024-07-26T07:51:39.539913+0000","last_fullsized":"2024-07-26T07:51:39.539913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:14.796738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2195,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.108677+0000","last_change":"2024-07-26T07:47:45.856170+0000","last_active":"2024-07-26T07:49:48.108677+0000","last_peered":"2024-07-26T07:49:48.108677+0000","last_clean":"2024-07-26T07:49:48.108677+0000","last_became_active":"2024-07-26T07:47:45.855782+0000","last_became_peered":"2024-07-26T07:47:45.855782+0000","last_unstale":"2024-07-26T07:49:48.108677+0000","last_undegraded":"2024-07-26T07:49:48.108677+0000","last_fullsized":"2024-07-26T07:49:48.108677+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:22.690910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313258+0000","last_change":"2024-07-26T07:45:54.310557+0000","last_active":"2024-07-26T07:49:20.313258+0000","last_peered":"2024-07-26T07:49:20.313258+0000","last_clean":"2024-07-26T07:49:20.313258+0000","last_became_active":"2024-07-26T07:45:54.290906+0000","last_became_peered":"2024-07-26T07:45:54.290906+0000","last_unstale":"2024-07-26T07:49:20.313258+0000","last_undegraded":"2024-07-26T07:49:20.313258+0000","last_fullsized":"2024-07-26T07:49:20.313258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_clean_scrub_stamp":"2024-07-26T07:41:16.689463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:51.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.305857+0000","last_change":"2024-07-26T07:47:41.776420+0000","last_active":"2024-07-26T07:49:20.305857+0000","last_peered":"2024-07-26T07:49:20.305857+0000","last_clean":"2024-07-26T07:49:20.305857+0000","last_became_active":"2024-07-26T07:47:41.775858+0000","last_became_peered":"2024-07-26T07:47:41.775858+0000","last_unstale":"2024-07-26T07:49:20.305857+0000","last_undegraded":"2024-07-26T07:49:20.305857+0000","last_fullsized":"2024-07-26T07:49:20.305857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:18.880331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.109733+0000","last_change":"2024-07-26T07:48:19.933318+0000","last_active":"2024-07-26T07:49:48.109733+0000","last_peered":"2024-07-26T07:49:48.109733+0000","last_clean":"2024-07-26T07:49:48.109733+0000","last_became_active":"2024-07-26T07:48:19.932683+0000","last_became_peered":"2024-07-26T07:48:19.932683+0000","last_unstale":"2024-07-26T07:49:48.109733+0000","last_undegraded":"2024-07-26T07:49:48.109733+0000","last_fullsized":"2024-07-26T07:49:48.109733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:25:23.387428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:43.375132+0000","last_change":"2024-07-26T07:47:41.786698+0000","last_active":"2024-07-26T07:49:43.375132+0000","last_peered":"2024-07-26T07:49:43.375132+0000","last_clean":"2024-07-26T07:49:43.375132+0000","last_became_active":"2024-07-26T07:47:41.784491+0000","last_became_peered":"2024-07-26T07:47:41.784491+0000","last_unstale":"2024-07-26T07:49:43.375132+0000","last_undegraded":"2024-07-26T07:49:43.375132+0000","last_fullsized":"2024-07-26T07:49:43.375132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:09:08.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093849+0000","last_change":"2024-07-26T07:47:45.849698+0000","last_active":"2024-07-26T07:49:48.093849+0000","last_peered":"2024-07-26T07:49:48.093849+0000","last_clean":"2024-07-26T07:49:48.093849+0000","last_became_active":"2024-07-26T07:47:45.849266+0000","last_became_peered":"2024-07-26T07:47:45.849266+0000","last_unstale":"2024-07-26T07:49:48.093849+0000","last_undegraded":"2024-07-26T07:49:48.093849+0000","last_fullsized":"2024-07-26T07:49:48.093849+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:57.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":250,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:38.277250+0000","last_change":"2024-07-26T07:47:42.558680+0000","last_active":"2024-07-26T07:51:38.277250+0000","last_peered":"2024-07-26T07:51:38.277250+0000","last_clean":"2024-07-26T07:51:38.277250+0000","last_became_active":"2024-07-26T07:47:42.558197+0000","last_became_peered":"2024-07-26T07:47:42.558197+0000","last_unstale":"2024-07-26T07:51:38.277250+0000","last_undegraded":"2024-07-26T07:51:38.277250+0000","last_fullsized":"2024-07-26T07:51:38.277250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:52:52.505455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":212,"num_read_kb":212,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.078525+0000","last_change":"2024-07-26T07:48:19.929897+0000","last_active":"2024-07-26T07:49:48.078525+0000","last_peered":"2024-07-26T07:49:48.078525+0000","last_clean":"2024-07-26T07:49:48.078525+0000","last_became_active":"2024-07-26T07:48:19.927946+0000","last_became_peered":"2024-07-26T07:48:19.927946+0000","last_unstale":"2024-07-26T07:49:48.078525+0000","last_undegraded":"2024-07-26T07:49:48.078525+0000","last_fullsized":"2024-07-26T07:49:48.078525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:11.157824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'6","reported_seq":255,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:41.529095+0000","last_change":"2024-07-26T07:47:41.785513+0000","last_active":"2024-07-26T07:51:41.529095+0000","last_peered":"2024-07-26T07:51:41.529095+0000","last_clean":"2024-07-26T07:51:41.529095+0000","last_became_active":"2024-07-26T07:47:41.784783+0000","last_became_peered":"2024-07-26T07:47:41.784783+0000","last_unstale":"2024-07-26T07:51:41.529095+0000","last_undegraded":"2024-07-26T07:51:41.529095+0000","last_fullsized":"2024-07-26T07:51:41.529095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:54.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.082468+0000","last_change":"2024-07-26T07:47:45.856580+0000","last_active":"2024-07-26T07:49:48.082468+0000","last_peered":"2024-07-26T07:49:48.082468+0000","last_clean":"2024-07-26T07:49:48.082468+0000","last_became_active":"2024-07-26T07:47:45.852330+0000","last_became_peered":"2024-07-26T07:47:45.852330+0000","last_unstale":"2024-07-26T07:49:48.082468+0000","last_undegraded":"2024-07-26T07:49:48.082468+0000","last_fullsized":"2024-07-26T07:49:48.082468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:09.479563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.428822+0000","last_change":"2024-07-26T07:47:42.559255+0000","last_active":"2024-07-26T07:49:20.428822+0000","last_peered":"2024-07-26T07:49:20.428822+0000","last_clean":"2024-07-26T07:49:20.428822+0000","last_became_active":"2024-07-26T07:47:42.558123+0000","last_became_peered":"2024-07-26T07:47:42.558123+0000","last_unstale":"2024-07-26T07:49:20.428822+0000","last_undegraded":"2024-07-26T07:49:20.428822+0000","last_fullsized":"2024-07-26T07:49:20.428822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:49:46.576674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092093+0000","last_change":"2024-07-26T07:47:45.856256+0000","last_active":"2024-07-26T07:49:48.092093+0000","last_peered":"2024-07-26T07:49:48.092093+0000","last_clean":"2024-07-26T07:49:48.092093+0000","last_became_active":"2024-07-26T07:47:45.855696+0000","last_became_peered":"2024-07-26T07:47:45.855696+0000","last_unstale":"2024-07-26T07:49:48.092093+0000","last_undegraded":"2024-07-26T07:49:48.092093+0000","last_fullsized":"2024-07-26T07:49:48.092093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:24.451225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314969+0000","last_change":"2024-07-26T07:47:42.562631+0000","last_active":"2024-07-26T07:49:20.314969+0000","last_peered":"2024-07-26T07:49:20.314969+0000","last_clean":"2024-07-26T07:49:20.314969+0000","last_became_active":"2024-07-26T07:47:42.561379+0000","last_became_peered":"2024-07-26T07:47:42.561379+0000","last_unstale":"2024-07-26T07:49:20.314969+0000","last_undegraded":"2024-07-26T07:49:20.314969+0000","last_fullsized":"2024-07-26T07:49:20.314969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:03:30.471694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092440+0000","last_change":"2024-07-26T07:47:45.849580+0000","last_active":"2024-07-26T07:49:48.092440+0000","last_peered":"2024-07-26T07:49:48.092440+0000","last_clean":"2024-07-26T07:49:48.092440+0000","last_became_active":"2024-07-26T07:47:45.849107+0000","last_became_peered":"2024-07-26T07:47:45.849107+0000","last_unstale":"2024-07-26T07:49:48.092440+0000","last_undegraded":"2024-07-26T07:49:48.092440+0000","last_fullsized":"2024-07-26T07:49:48.092440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:57:13.694059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'18","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:38.899550+0000","last_change":"2024-07-26T07:47:41.774083+0000","last_active":"2024-07-26T07:51:38.899550+0000","last_peered":"2024-07-26T07:51:38.899550+0000","last_clean":"2024-07-26T07:51:38.899550+0000","last_became_active":"2024-07-26T07:47:41.773502+0000","last_became_peered":"2024-07-26T07:47:41.773502+0000","last_unstale":"2024-07-26T07:51:38.899550+0000","last_undegraded":"2024-07-26T07:51:38.899550+0000","last_fullsized":"2024-07-26T07:51:38.899550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:11:44.083944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099359+0000","last_change":"2024-07-26T07:47:45.845894+0000","last_active":"2024-07-26T07:49:48.099359+0000","last_peered":"2024-07-26T07:49:48.099359+0000","last_clean":"2024-07-26T07:49:48.099359+0000","last_became_active":"2024-07-26T07:47:45.845534+0000","last_became_peered":"2024-07-26T07:47:45.845534+0000","last_unstale":"2024-07-26T07:49:48.099359+0000","last_undegraded":"2024-07-26T07:49:48.099359+0000","last_fullsized":"2024-07-26T07:49:48.099359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:52:42.209561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.306134+0000","last_change":"2024-07-26T07:47:41.904220+0000","last_active":"2024-07-26T07:49:20.306134+0000","last_peered":"2024-07-26T07:49:20.306134+0000","last_clean":"2024-07-26T07:49:20.306134+0000","last_became_active":"2024-07-26T07:47:41.902349+0000","last_became_peered":"2024-07-26T07:47:41.902349+0000","last_unstale":"2024-07-26T07:49:20.306134+0000","last_undegraded":"2024-07-26T07:49:20.306134+0000","last_fullsized":"2024-07-26T07:49:20.306134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:08.520461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092199+0000","last_change":"2024-07-26T07:47:45.846920+0000","last_active":"2024-07-26T07:49:48.092199+0000","last_peered":"2024-07-26T07:49:48.092199+0000","last_clean":"2024-07-26T07:49:48.092199+0000","last_became_active":"2024-07-26T07:47:45.846458+0000","last_became_peered":"2024-07-26T07:47:45.846458+0000","last_unstale":"2024-07-26T07:49:48.092199+0000","last_undegraded":"2024-07-26T07:49:48.092199+0000","last_fullsized":"2024-07-26T07:49:48.092199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:06.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.295007+0000","last_change":"2024-07-26T07:47:42.560346+0000","last_active":"2024-07-26T07:49:20.295007+0000","last_peered":"2024-07-26T07:49:20.295007+0000","last_clean":"2024-07-26T07:49:20.295007+0000","last_became_active":"2024-07-26T07:47:42.558937+0000","last_became_peered":"2024-07-26T07:47:42.558937+0000","last_unstale":"2024-07-26T07:49:20.295007+0000","last_undegraded":"2024-07-26T07:49:20.295007+0000","last_fullsized":"2024-07-26T07:49:20.295007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.676625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.081730+0000","last_change":"2024-07-26T07:47:45.845553+0000","last_active":"2024-07-26T07:49:48.081730+0000","last_peered":"2024-07-26T07:49:48.081730+0000","last_clean":"2024-07-26T07:49:48.081730+0000","last_became_active":"2024-07-26T07:47:45.845125+0000","last_became_peered":"2024-07-26T07:47:45.845125+0000","last_unstale":"2024-07-26T07:49:48.081730+0000","last_undegraded":"2024-07-26T07:49:48.081730+0000","last_fullsized":"2024-07-26T07:49:48.081730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:10.896159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.292557+0000","last_change":"2024-07-26T07:47:41.779688+0000","last_active":"2024-07-26T07:49:20.292557+0000","last_peered":"2024-07-26T07:49:20.292557+0000","last_clean":"2024-07-26T07:49:20.292557+0000","last_became_active":"2024-07-26T07:47:41.779115+0000","last_became_peered":"2024-07-26T07:47:41.779115+0000","last_unstale":"2024-07-26T07:49:20.292557+0000","last_undegraded":"2024-07-26T07:49:20.292557+0000","last_fullsized":"2024-07-26T07:49:20.292557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:20.708336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091558+0000","last_change":"2024-07-26T07:47:45.853869+0000","last_active":"2024-07-26T07:49:48.091558+0000","last_peered":"2024-07-26T07:49:48.091558+0000","last_clean":"2024-07-26T07:49:48.091558+0000","last_became_active":"2024-07-26T07:47:45.852509+0000","last_became_peered":"2024-07-26T07:47:45.852509+0000","last_unstale":"2024-07-26T07:49:48.091558+0000","last_undegraded":"2024-07-26T07:49:48.091558+0000","last_fullsized":"2024-07-26T07:49:48.091558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:01:31.764025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'4","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:55.750934+0000","last_change":"2024-07-26T07:47:41.792331+0000","last_active":"2024-07-26T07:49:55.750934+0000","last_peered":"2024-07-26T07:49:55.750934+0000","last_clean":"2024-07-26T07:49:55.750934+0000","last_became_active":"2024-07-26T07:47:41.790668+0000","last_became_peered":"2024-07-26T07:47:41.790668+0000","last_unstale":"2024-07-26T07:49:55.750934+0000","last_undegraded":"2024-07-26T07:49:55.750934+0000","last_fullsized":"2024-07-26T07:49:55.750934+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:59.430692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096588+0000","last_change":"2024-07-26T07:47:45.843583+0000","last_active":"2024-07-26T07:49:48.096588+0000","last_peered":"2024-07-26T07:49:48.096588+0000","last_clean":"2024-07-26T07:49:48.096588+0000","last_became_active":"2024-07-26T07:47:45.843185+0000","last_became_peered":"2024-07-26T07:47:45.843185+0000","last_unstale":"2024-07-26T07:49:48.096588+0000","last_undegraded":"2024-07-26T07:49:48.096588+0000","last_fullsized":"2024-07-26T07:49:48.096588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:01.119065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314240+0000","last_change":"2024-07-26T07:48:20.014411+0000","last_active":"2024-07-26T07:49:20.314240+0000","last_peered":"2024-07-26T07:49:20.314240+0000","last_clean":"2024-07-26T07:49:20.314240+0000","last_became_active":"2024-07-26T07:48:20.002085+0000","last_became_peered":"2024-07-26T07:48:20.002085+0000","last_unstale":"2024-07-26T07:49:20.314240+0000","last_undegraded":"2024-07-26T07:49:20.314240+0000","last_fullsized":"2024-07-26T07:49:20.314240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:46:59.335128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.110858+0000","last_change":"2024-07-26T07:47:45.851963+0000","last_active":"2024-07-26T07:49:48.110858+0000","last_peered":"2024-07-26T07:49:48.110858+0000","last_clean":"2024-07-26T07:49:48.110858+0000","last_became_active":"2024-07-26T07:47:45.850459+0000","last_became_peered":"2024-07-26T07:47:45.850459+0000","last_unstale":"2024-07-26T07:49:48.110858+0000","last_undegraded":"2024-07-26T07:49:48.110858+0000","last_fullsized":"2024-07-26T07:49:48.110858+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:18.092186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096201+0000","last_change":"2024-07-26T07:47:45.848744+0000","last_active":"2024-07-26T07:49:48.096201+0000","last_peered":"2024-07-26T07:49:48.096201+0000","last_clean":"2024-07-26T07:49:48.096201+0000","last_became_active":"2024-07-26T07:47:45.848292+0000","last_became_peered":"2024-07-26T07:47:45.848292+0000","last_unstale":"2024-07-26T07:49:48.096201+0000","last_undegraded":"2024-07-26T07:49:48.096201+0000","last_fullsized":"2024-07-26T07:49:48.096201+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:54.549508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314854+0000","last_change":"2024-07-26T07:47:41.788875+0000","last_active":"2024-07-26T07:49:20.314854+0000","last_peered":"2024-07-26T07:49:20.314854+0000","last_clean":"2024-07-26T07:49:20.314854+0000","last_became_active":"2024-07-26T07:47:41.786942+0000","last_became_peered":"2024-07-26T07:47:41.786942+0000","last_unstale":"2024-07-26T07:49:20.314854+0000","last_undegraded":"2024-07-26T07:49:20.314854+0000","last_fullsized":"2024-07-26T07:49:20.314854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:11:31.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111571+0000","last_change":"2024-07-26T07:47:45.854832+0000","last_active":"2024-07-26T07:49:48.111571+0000","last_peered":"2024-07-26T07:49:48.111571+0000","last_clean":"2024-07-26T07:49:48.111571+0000","last_became_active":"2024-07-26T07:47:45.853756+0000","last_became_peered":"2024-07-26T07:47:45.853756+0000","last_unstale":"2024-07-26T07:49:48.111571+0000","last_undegraded":"2024-07-26T07:49:48.111571+0000","last_fullsized":"2024-07-26T07:49:48.111571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:42:14.579177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314889+0000","last_change":"2024-07-26T07:47:41.787839+0000","last_active":"2024-07-26T07:49:20.314889+0000","last_peered":"2024-07-26T07:49:20.314889+0000","last_clean":"2024-07-26T07:49:20.314889+0000","last_became_active":"2024-07-26T07:47:41.785951+0000","last_became_peered":"2024-07-26T07:47:41.785951+0000","last_unstale":"2024-07-26T07:49:20.314889+0000","last_undegraded":"2024-07-26T07:49:20.314889+0000","last_fullsized":"2024-07-26T07:49:20.314889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:58.699056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104028+0000","last_change":"2024-07-26T07:47:45.841451+0000","last_active":"2024-07-26T07:49:48.104028+0000","last_peered":"2024-07-26T07:49:48.104028+0000","last_clean":"2024-07-26T07:49:48.104028+0000","last_became_active":"2024-07-26T07:47:45.840803+0000","last_became_peered":"2024-07-26T07:47:45.840803+0000","last_unstale":"2024-07-26T07:49:48.104028+0000","last_undegraded":"2024-07-26T07:49:48.104028+0000","last_fullsized":"2024-07-26T07:49:48.104028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:24:21.832317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:44.189574+0000","last_change":"2024-07-26T07:47:42.558775+0000","last_active":"2024-07-26T07:49:44.189574+0000","last_peered":"2024-07-26T07:49:44.189574+0000","last_clean":"2024-07-26T07:49:44.189574+0000","last_became_active":"2024-07-26T07:47:42.558300+0000","last_became_peered":"2024-07-26T07:47:42.558300+0000","last_unstale":"2024-07-26T07:49:44.189574+0000","last_undegraded":"2024-07-26T07:49:44.189574+0000","last_fullsized":"2024-07-26T07:49:44.189574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:59.643709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091602+0000","last_change":"2024-07-26T07:47:45.852401+0000","last_active":"2024-07-26T07:49:48.091602+0000","last_peered":"2024-07-26T07:49:48.091602+0000","last_clean":"2024-07-26T07:49:48.091602+0000","last_became_active":"2024-07-26T07:47:45.852054+0000","last_became_peered":"2024-07-26T07:47:45.852054+0000","last_unstale":"2024-07-26T07:49:48.091602+0000","last_undegraded":"2024-07-26T07:49:48.091602+0000","last_fullsized":"2024-07-26T07:49:48.091602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:25.347770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"86'14","reported_seq":284,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:42.531152+0000","last_change":"2024-07-26T07:48:19.961760+0000","last_active":"2024-07-26T07:51:42.531152+0000","last_peered":"2024-07-26T07:51:42.531152+0000","last_clean":"2024-07-26T07:51:42.531152+0000","last_became_active":"2024-07-26T07:48:19.937694+0000","last_became_peered":"2024-07-26T07:48:19.937694+0000","last_unstale":"2024-07-26T07:51:42.531152+0000","last_undegraded":"2024-07-26T07:51:42.531152+0000","last_fullsized":"2024-07-26T07:51:42.531152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:05:10.571086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":211,"num_read_kb":211,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74135,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2173,"num_read_kb":1162,"num_write":270,"num_write_kb":158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1372,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":221784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579972,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":466032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776202,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1662976,"data_stored":1120154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972433,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":462962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168664,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":462556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397598,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1055865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561124,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":463172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757357,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1589248,"data_stored":1055638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953587,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":463033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":462504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411456,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":525834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":529966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":465434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":70646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T07:51:45.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T07:51:45.815 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:46 smithi097 ceph-mon[32231]: pgmap v509: 97 pgs: 97 active+clean; 649 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-07-26T07:51:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:46 smithi097 ceph-mon[32231]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:46 smithi115 ceph-mon[35364]: pgmap v509: 97 pgs: 97 active+clean; 649 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-07-26T07:51:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:46 smithi115 ceph-mon[35364]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:46 smithi181 ceph-mon[35869]: pgmap v509: 97 pgs: 97 active+clean; 649 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-07-26T07:51:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:46 smithi181 ceph-mon[35869]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:47.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 -- 172.21.15.97:0/2666248395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8108720 msgr2=0x7feaf810ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:47.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 --2- 172.21.15.97:0/2666248395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8108720 0x7feaf810ef50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009900 tx=0x7feaf002a410 comp rx=0 tx=0).stop 2024-07-26T07:51:47.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 -- 172.21.15.97:0/2666248395 shutdown_connections 2024-07-26T07:51:47.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 --2- 172.21.15.97:0/2666248395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8108720 0x7feaf810ef50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:47.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 --2- 172.21.15.97:0/2666248395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaf8107dd0 0x7feaf81081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:47.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 --2- 172.21.15.97:0/2666248395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaf8102490 0x7feaf81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:47.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 -- 172.21.15.97:0/2666248395 >> 172.21.15.97:0/2666248395 conn(0x7feaf8079160 msgr2=0x7feaf8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:47.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 -- 172.21.15.97:0/2666248395 shutdown_connections 2024-07-26T07:51:47.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.826+0000 7feafec00640 1 -- 172.21.15.97:0/2666248395 wait complete. 2024-07-26T07:51:47.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.828+0000 7feafec00640 1 Processor -- start 2024-07-26T07:51:47.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.828+0000 7feafec00640 1 -- start start 2024-07-26T07:51:47.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feafec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8102490 0x7feaf8199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:47.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feafec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaf8107dd0 0x7feaf819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:47.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feafec00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaf8108720 0x7feaf819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:47.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feafec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7feaf8112280 con 0x7feaf8102490 2024-07-26T07:51:47.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feafec00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7feaf8112100 con 0x7feaf8108720 2024-07-26T07:51:47.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feafec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feaf8112400 con 0x7feaf8107dd0 2024-07-26T07:51:47.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feaf7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8102490 0x7feaf8199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feaf7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8102490 0x7feaf8199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42308/0 (socket says 172.21.15.97:42308) 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feaf7e00640 1 -- 172.21.15.97:0/1273985367 learned_addr learned my addr 172.21.15.97:0/1273985367 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.829+0000 7feaf7400640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaf8107dd0 0x7feaf819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feafcc00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaf8108720 0x7feaf819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf7e00640 1 -- 172.21.15.97:0/1273985367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaf8107dd0 msgr2=0x7feaf819a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf7e00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaf8107dd0 0x7feaf819a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf7e00640 1 -- 172.21.15.97:0/1273985367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaf8108720 msgr2=0x7feaf819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf7e00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaf8108720 0x7feaf819f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf7e00640 1 -- 172.21.15.97:0/1273985367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf819fcb0 con 0x7feaf8102490 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf7400640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaf8107dd0 0x7feaf819a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf7e00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8102490 0x7feaf8199d40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7feae800eab0 tx=0x7feae800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf4c00640 1 -- 172.21.15.97:0/1273985367 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae800ce40 con 0x7feaf8102490 2024-07-26T07:51:47.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.830+0000 7feaf4c00640 1 -- 172.21.15.97:0/1273985367 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feae8011900 con 0x7feaf8102490 2024-07-26T07:51:47.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.831+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaf819ffa0 con 0x7feaf8102490 2024-07-26T07:51:47.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.831+0000 7feaf4c00640 1 -- 172.21.15.97:0/1273985367 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae8010990 con 0x7feaf8102490 2024-07-26T07:51:47.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.831+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaf81a6700 con 0x7feaf8102490 2024-07-26T07:51:47.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.832+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaf81028a0 con 0x7feaf8102490 2024-07-26T07:51:47.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.834+0000 7feaf4c00640 1 -- 172.21.15.97:0/1273985367 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7feae8011470 con 0x7feaf8102490 2024-07-26T07:51:47.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.836+0000 7feaf4c00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feadc0b3c60 0x7feadc0b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:47.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.836+0000 7feaf4c00640 1 -- 172.21.15.97:0/1273985367 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7feae80e4520 con 0x7feaf8102490 2024-07-26T07:51:47.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.836+0000 7feaf7400640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feadc0b3c60 0x7feadc0b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:47.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.837+0000 7feaf7400640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feadc0b3c60 0x7feadc0b6170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feae00058c0 tx=0x7feae0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:47.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:47.841+0000 7feaf4c00640 1 -- 172.21.15.97:0/1273985367 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feae8016030 con 0x7feaf8102490 2024-07-26T07:51:48.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.039+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feaf8063230 con 0x7feadc0b3c60 2024-07-26T07:51:48.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.044+0000 7feaf4c00640 1 -- 172.21.15.97:0/1273985367 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258991 (secure 0 0 0) 0x7feaf8063230 con 0x7feadc0b3c60 2024-07-26T07:51:48.049 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:48.050 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feadc0b3c60 msgr2=0x7feadc0b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feadc0b3c60 0x7feadc0b6170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feae00058c0 tx=0x7feae0005850 comp rx=0 tx=0).stop 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8102490 msgr2=0x7feaf8199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8102490 0x7feaf8199d40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7feae800eab0 tx=0x7feae800ef70 comp rx=0 tx=0).stop 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 shutdown_connections 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7feadc0b3c60 0x7feadc0b6170 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feaf8108720 0x7feaf819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feaf8107dd0 0x7feaf819a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:48.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 --2- 172.21.15.97:0/1273985367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feaf8102490 0x7feaf8199d40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:48.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.050+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 >> 172.21.15.97:0/1273985367 conn(0x7feaf8079160 msgr2=0x7feaf806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:48.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.051+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 shutdown_connections 2024-07-26T07:51:48.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:48.051+0000 7feafec00640 1 -- 172.21.15.97:0/1273985367 wait complete. 2024-07-26T07:51:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:48 smithi097 ceph-mon[32231]: pgmap v510: 97 pgs: 97 active+clean; 650 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-26T07:51:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:48 smithi115 ceph-mon[35364]: pgmap v510: 97 pgs: 97 active+clean; 650 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-26T07:51:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:48 smithi181 ceph-mon[35869]: pgmap v510: 97 pgs: 97 active+clean; 650 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-26T07:51:48.844 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-07-26T07:51:46.225131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665519,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2271,"num_read_kb":1233,"num_write":399,"num_write_kb":2313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2953212,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":827,"ondisk_log_size":827,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":404172,"kb_used_data":13420,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537652,"statfs":{"total":1151940427776,"available":1151526555648,"internally_reserved":0,"allocated":13742080,"data_stored":7541157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":62,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005350"},"pg_stats":[{"pgid":"3.3f","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090598+0000","last_change":"2024-07-26T07:47:45.843670+0000","last_active":"2024-07-26T07:49:48.090598+0000","last_peered":"2024-07-26T07:49:48.090598+0000","last_clean":"2024-07-26T07:49:48.090598+0000","last_became_active":"2024-07-26T07:47:45.841137+0000","last_became_peered":"2024-07-26T07:47:45.841137+0000","last_unstale":"2024-07-26T07:49:48.090598+0000","last_undegraded":"2024-07-26T07:49:48.090598+0000","last_fullsized":"2024-07-26T07:49:48.090598+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:38.063503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085904+0000","last_change":"2024-07-26T07:47:45.849326+0000","last_active":"2024-07-26T07:49:48.085904+0000","last_peered":"2024-07-26T07:49:48.085904+0000","last_clean":"2024-07-26T07:49:48.085904+0000","last_became_active":"2024-07-26T07:47:45.847761+0000","last_became_peered":"2024-07-26T07:47:45.847761+0000","last_unstale":"2024-07-26T07:49:48.085904+0000","last_undegraded":"2024-07-26T07:49:48.085904+0000","last_fullsized":"2024-07-26T07:49:48.085904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:31.098794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092147+0000","last_change":"2024-07-26T07:47:45.852343+0000","last_active":"2024-07-26T07:49:48.092147+0000","last_peered":"2024-07-26T07:49:48.092147+0000","last_clean":"2024-07-26T07:49:48.092147+0000","last_became_active":"2024-07-26T07:47:45.851950+0000","last_became_peered":"2024-07-26T07:47:45.851950+0000","last_unstale":"2024-07-26T07:49:48.092147+0000","last_undegraded":"2024-07-26T07:49:48.092147+0000","last_fullsized":"2024-07-26T07:49:48.092147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:41.030903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091199+0000","last_change":"2024-07-26T07:47:45.857042+0000","last_active":"2024-07-26T07:49:48.091199+0000","last_peered":"2024-07-26T07:49:48.091199+0000","last_clean":"2024-07-26T07:49:48.091199+0000","last_became_active":"2024-07-26T07:47:45.855145+0000","last_became_peered":"2024-07-26T07:47:45.855145+0000","last_unstale":"2024-07-26T07:49:48.091199+0000","last_undegraded":"2024-07-26T07:49:48.091199+0000","last_fullsized":"2024-07-26T07:49:48.091199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:32.233019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092139+0000","last_change":"2024-07-26T07:47:45.838560+0000","last_active":"2024-07-26T07:49:48.092139+0000","last_peered":"2024-07-26T07:49:48.092139+0000","last_clean":"2024-07-26T07:49:48.092139+0000","last_became_active":"2024-07-26T07:47:45.835578+0000","last_became_peered":"2024-07-26T07:47:45.835578+0000","last_unstale":"2024-07-26T07:49:48.092139+0000","last_undegraded":"2024-07-26T07:49:48.092139+0000","last_fullsized":"2024-07-26T07:49:48.092139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:09:40.445625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.089953+0000","last_change":"2024-07-26T07:47:45.843980+0000","last_active":"2024-07-26T07:49:48.089953+0000","last_peered":"2024-07-26T07:49:48.089953+0000","last_clean":"2024-07-26T07:49:48.089953+0000","last_became_active":"2024-07-26T07:47:45.843546+0000","last_became_peered":"2024-07-26T07:47:45.843546+0000","last_unstale":"2024-07-26T07:49:48.089953+0000","last_undegraded":"2024-07-26T07:49:48.089953+0000","last_fullsized":"2024-07-26T07:49:48.089953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:06.839136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093383+0000","last_change":"2024-07-26T07:47:45.854427+0000","last_active":"2024-07-26T07:49:48.093383+0000","last_peered":"2024-07-26T07:49:48.093383+0000","last_clean":"2024-07-26T07:49:48.093383+0000","last_became_active":"2024-07-26T07:47:45.853495+0000","last_became_peered":"2024-07-26T07:47:45.853495+0000","last_unstale":"2024-07-26T07:49:48.093383+0000","last_undegraded":"2024-07-26T07:49:48.093383+0000","last_fullsized":"2024-07-26T07:49:48.093383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:34:34.110581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096799+0000","last_change":"2024-07-26T07:47:45.847732+0000","last_active":"2024-07-26T07:49:48.096799+0000","last_peered":"2024-07-26T07:49:48.096799+0000","last_clean":"2024-07-26T07:49:48.096799+0000","last_became_active":"2024-07-26T07:47:45.845583+0000","last_became_peered":"2024-07-26T07:47:45.845583+0000","last_unstale":"2024-07-26T07:49:48.096799+0000","last_undegraded":"2024-07-26T07:49:48.096799+0000","last_fullsized":"2024-07-26T07:49:48.096799+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:07:09.370076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.108141+0000","last_change":"2024-07-26T07:47:45.849962+0000","last_active":"2024-07-26T07:49:48.108141+0000","last_peered":"2024-07-26T07:49:48.108141+0000","last_clean":"2024-07-26T07:49:48.108141+0000","last_became_active":"2024-07-26T07:47:45.845766+0000","last_became_peered":"2024-07-26T07:47:45.845766+0000","last_unstale":"2024-07-26T07:49:48.108141+0000","last_undegraded":"2024-07-26T07:49:48.108141+0000","last_fullsized":"2024-07-26T07:49:48.108141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:56.602825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086656+0000","last_change":"2024-07-26T07:47:45.857287+0000","last_active":"2024-07-26T07:49:48.086656+0000","last_peered":"2024-07-26T07:49:48.086656+0000","last_clean":"2024-07-26T07:49:48.086656+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.086656+0000","last_undegraded":"2024-07-26T07:49:48.086656+0000","last_fullsized":"2024-07-26T07:49:48.086656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:38:24.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.107214+0000","last_change":"2024-07-26T07:47:45.858369+0000","last_active":"2024-07-26T07:49:48.107214+0000","last_peered":"2024-07-26T07:49:48.107214+0000","last_clean":"2024-07-26T07:49:48.107214+0000","last_became_active":"2024-07-26T07:47:45.856685+0000","last_became_peered":"2024-07-26T07:47:45.856685+0000","last_unstale":"2024-07-26T07:49:48.107214+0000","last_undegraded":"2024-07-26T07:49:48.107214+0000","last_fullsized":"2024-07-26T07:49:48.107214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:44.877953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.083667+0000","last_change":"2024-07-26T07:48:20.001394+0000","last_active":"2024-07-26T07:49:48.083667+0000","last_peered":"2024-07-26T07:49:48.083667+0000","last_clean":"2024-07-26T07:49:48.083667+0000","last_became_active":"2024-07-26T07:48:20.000751+0000","last_became_peered":"2024-07-26T07:48:20.000751+0000","last_unstale":"2024-07-26T07:49:48.083667+0000","last_undegraded":"2024-07-26T07:49:48.083667+0000","last_fullsized":"2024-07-26T07:49:48.083667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:26:24.513652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.097228+0000","last_change":"2024-07-26T07:47:45.847931+0000","last_active":"2024-07-26T07:49:48.097228+0000","last_peered":"2024-07-26T07:49:48.097228+0000","last_clean":"2024-07-26T07:49:48.097228+0000","last_became_active":"2024-07-26T07:47:45.847484+0000","last_became_peered":"2024-07-26T07:47:45.847484+0000","last_unstale":"2024-07-26T07:49:48.097228+0000","last_undegraded":"2024-07-26T07:49:48.097228+0000","last_fullsized":"2024-07-26T07:49:48.097228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:04.498360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.087242+0000","last_change":"2024-07-26T07:47:45.846484+0000","last_active":"2024-07-26T07:49:48.087242+0000","last_peered":"2024-07-26T07:49:48.087242+0000","last_clean":"2024-07-26T07:49:48.087242+0000","last_became_active":"2024-07-26T07:47:45.844772+0000","last_became_peered":"2024-07-26T07:47:45.844772+0000","last_unstale":"2024-07-26T07:49:48.087242+0000","last_undegraded":"2024-07-26T07:49:48.087242+0000","last_fullsized":"2024-07-26T07:49:48.087242+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:54.318026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088688+0000","last_change":"2024-07-26T07:47:45.849840+0000","last_active":"2024-07-26T07:49:48.088688+0000","last_peered":"2024-07-26T07:49:48.088688+0000","last_clean":"2024-07-26T07:49:48.088688+0000","last_became_active":"2024-07-26T07:47:45.848744+0000","last_became_peered":"2024-07-26T07:47:45.848744+0000","last_unstale":"2024-07-26T07:49:48.088688+0000","last_undegraded":"2024-07-26T07:49:48.088688+0000","last_fullsized":"2024-07-26T07:49:48.088688+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:44.675382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111006+0000","last_change":"2024-07-26T07:47:45.847176+0000","last_active":"2024-07-26T07:49:48.111006+0000","last_peered":"2024-07-26T07:49:48.111006+0000","last_clean":"2024-07-26T07:49:48.111006+0000","last_became_active":"2024-07-26T07:47:45.845936+0000","last_became_peered":"2024-07-26T07:47:45.845936+0000","last_unstale":"2024-07-26T07:49:48.111006+0000","last_undegraded":"2024-07-26T07:49:48.111006+0000","last_fullsized":"2024-07-26T07:49:48.111006+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:02.245098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090942+0000","last_change":"2024-07-26T07:47:45.853210+0000","last_active":"2024-07-26T07:49:48.090942+0000","last_peered":"2024-07-26T07:49:48.090942+0000","last_clean":"2024-07-26T07:49:48.090942+0000","last_became_active":"2024-07-26T07:47:45.852766+0000","last_became_peered":"2024-07-26T07:47:45.852766+0000","last_unstale":"2024-07-26T07:49:48.090942+0000","last_undegraded":"2024-07-26T07:49:48.090942+0000","last_fullsized":"2024-07-26T07:49:48.090942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:57.406748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.103824+0000","last_change":"2024-07-26T07:47:45.850699+0000","last_active":"2024-07-26T07:49:48.103824+0000","last_peered":"2024-07-26T07:49:48.103824+0000","last_clean":"2024-07-26T07:49:48.103824+0000","last_became_active":"2024-07-26T07:47:45.850314+0000","last_became_peered":"2024-07-26T07:47:45.850314+0000","last_unstale":"2024-07-26T07:49:48.103824+0000","last_undegraded":"2024-07-26T07:49:48.103824+0000","last_fullsized":"2024-07-26T07:49:48.103824+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:09.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090575+0000","last_change":"2024-07-26T07:47:45.851063+0000","last_active":"2024-07-26T07:49:48.090575+0000","last_peered":"2024-07-26T07:49:48.090575+0000","last_clean":"2024-07-26T07:49:48.090575+0000","last_became_active":"2024-07-26T07:47:45.850164+0000","last_became_peered":"2024-07-26T07:47:45.850164+0000","last_unstale":"2024-07-26T07:49:48.090575+0000","last_undegraded":"2024-07-26T07:49:48.090575+0000","last_fullsized":"2024-07-26T07:49:48.090575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:31.597551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088724+0000","last_change":"2024-07-26T07:47:45.861207+0000","last_active":"2024-07-26T07:49:48.088724+0000","last_peered":"2024-07-26T07:49:48.088724+0000","last_clean":"2024-07-26T07:49:48.088724+0000","last_became_active":"2024-07-26T07:47:45.860179+0000","last_became_peered":"2024-07-26T07:47:45.860179+0000","last_unstale":"2024-07-26T07:49:48.088724+0000","last_undegraded":"2024-07-26T07:49:48.088724+0000","last_fullsized":"2024-07-26T07:49:48.088724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:12:51.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101706+0000","last_change":"2024-07-26T07:47:45.846197+0000","last_active":"2024-07-26T07:49:48.101706+0000","last_peered":"2024-07-26T07:49:48.101706+0000","last_clean":"2024-07-26T07:49:48.101706+0000","last_became_active":"2024-07-26T07:47:45.845238+0000","last_became_peered":"2024-07-26T07:47:45.845238+0000","last_unstale":"2024-07-26T07:49:48.101706+0000","last_undegraded":"2024-07-26T07:49:48.101706+0000","last_fullsized":"2024-07-26T07:49:48.101706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:29.900684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112421+0000","last_change":"2024-07-26T07:47:45.847963+0000","last_active":"2024-07-26T07:49:48.112421+0000","last_peered":"2024-07-26T07:49:48.112421+0000","last_clean":"2024-07-26T07:49:48.112421+0000","last_became_active":"2024-07-26T07:47:45.847513+0000","last_became_peered":"2024-07-26T07:47:45.847513+0000","last_unstale":"2024-07-26T07:49:48.112421+0000","last_undegraded":"2024-07-26T07:49:48.112421+0000","last_fullsized":"2024-07-26T07:49:48.112421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:51.082816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092151+0000","last_change":"2024-07-26T07:47:45.853273+0000","last_active":"2024-07-26T07:49:48.092151+0000","last_peered":"2024-07-26T07:49:48.092151+0000","last_clean":"2024-07-26T07:49:48.092151+0000","last_became_active":"2024-07-26T07:47:45.852894+0000","last_became_peered":"2024-07-26T07:47:45.852894+0000","last_unstale":"2024-07-26T07:49:48.092151+0000","last_undegraded":"2024-07-26T07:49:48.092151+0000","last_fullsized":"2024-07-26T07:49:48.092151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:57:12.945228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093387+0000","last_change":"2024-07-26T07:48:19.931938+0000","last_active":"2024-07-26T07:49:48.093387+0000","last_peered":"2024-07-26T07:49:48.093387+0000","last_clean":"2024-07-26T07:49:48.093387+0000","last_became_active":"2024-07-26T07:48:19.930976+0000","last_became_peered":"2024-07-26T07:48:19.930976+0000","last_unstale":"2024-07-26T07:49:48.093387+0000","last_undegraded":"2024-07-26T07:49:48.093387+0000","last_fullsized":"2024-07-26T07:49:48.093387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:59.610464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090184+0000","last_change":"2024-07-26T07:47:45.849736+0000","last_active":"2024-07-26T07:49:48.090184+0000","last_peered":"2024-07-26T07:49:48.090184+0000","last_clean":"2024-07-26T07:49:48.090184+0000","last_became_active":"2024-07-26T07:47:45.849335+0000","last_became_peered":"2024-07-26T07:47:45.849335+0000","last_unstale":"2024-07-26T07:49:48.090184+0000","last_undegraded":"2024-07-26T07:49:48.090184+0000","last_fullsized":"2024-07-26T07:49:48.090184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:20:12.758331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104678+0000","last_change":"2024-07-26T07:47:45.847250+0000","last_active":"2024-07-26T07:49:48.104678+0000","last_peered":"2024-07-26T07:49:48.104678+0000","last_clean":"2024-07-26T07:49:48.104678+0000","last_became_active":"2024-07-26T07:47:45.845564+0000","last_became_peered":"2024-07-26T07:47:45.845564+0000","last_unstale":"2024-07-26T07:49:48.104678+0000","last_undegraded":"2024-07-26T07:49:48.104678+0000","last_fullsized":"2024-07-26T07:49:48.104678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:03.616782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086629+0000","last_change":"2024-07-26T07:47:45.850462+0000","last_active":"2024-07-26T07:49:48.086629+0000","last_peered":"2024-07-26T07:49:48.086629+0000","last_clean":"2024-07-26T07:49:48.086629+0000","last_became_active":"2024-07-26T07:47:45.850036+0000","last_became_peered":"2024-07-26T07:47:45.850036+0000","last_unstale":"2024-07-26T07:49:48.086629+0000","last_undegraded":"2024-07-26T07:49:48.086629+0000","last_fullsized":"2024-07-26T07:49:48.086629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:38.374782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.094143+0000","last_change":"2024-07-26T07:47:45.859294+0000","last_active":"2024-07-26T07:49:48.094143+0000","last_peered":"2024-07-26T07:49:48.094143+0000","last_clean":"2024-07-26T07:49:48.094143+0000","last_became_active":"2024-07-26T07:47:45.858812+0000","last_became_peered":"2024-07-26T07:47:45.858812+0000","last_unstale":"2024-07-26T07:49:48.094143+0000","last_undegraded":"2024-07-26T07:49:48.094143+0000","last_fullsized":"2024-07-26T07:49:48.094143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:11.689717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085416+0000","last_change":"2024-07-26T07:48:19.930943+0000","last_active":"2024-07-26T07:49:48.085416+0000","last_peered":"2024-07-26T07:49:48.085416+0000","last_clean":"2024-07-26T07:49:48.085416+0000","last_became_active":"2024-07-26T07:48:19.929758+0000","last_became_peered":"2024-07-26T07:48:19.929758+0000","last_unstale":"2024-07-26T07:49:48.085416+0000","last_undegraded":"2024-07-26T07:49:48.085416+0000","last_fullsized":"2024-07-26T07:49:48.085416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:29:27.657376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090071+0000","last_change":"2024-07-26T07:47:45.846867+0000","last_active":"2024-07-26T07:49:48.090071+0000","last_peered":"2024-07-26T07:49:48.090071+0000","last_clean":"2024-07-26T07:49:48.090071+0000","last_became_active":"2024-07-26T07:47:45.846455+0000","last_became_peered":"2024-07-26T07:47:45.846455+0000","last_unstale":"2024-07-26T07:49:48.090071+0000","last_undegraded":"2024-07-26T07:49:48.090071+0000","last_fullsized":"2024-07-26T07:49:48.090071+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:14.997452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092573+0000","last_change":"2024-07-26T07:47:45.846179+0000","last_active":"2024-07-26T07:49:48.092573+0000","last_peered":"2024-07-26T07:49:48.092573+0000","last_clean":"2024-07-26T07:49:48.092573+0000","last_became_active":"2024-07-26T07:47:45.844880+0000","last_became_peered":"2024-07-26T07:47:45.844880+0000","last_unstale":"2024-07-26T07:49:48.092573+0000","last_undegraded":"2024-07-26T07:49:48.092573+0000","last_fullsized":"2024-07-26T07:49:48.092573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:00:50.899573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091429+0000","last_change":"2024-07-26T07:48:19.935196+0000","last_active":"2024-07-26T07:49:48.091429+0000","last_peered":"2024-07-26T07:49:48.091429+0000","last_clean":"2024-07-26T07:49:48.091429+0000","last_became_active":"2024-07-26T07:48:19.933730+0000","last_became_peered":"2024-07-26T07:48:19.933730+0000","last_unstale":"2024-07-26T07:49:48.091429+0000","last_undegraded":"2024-07-26T07:49:48.091429+0000","last_fullsized":"2024-07-26T07:49:48.091429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:10.305122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090542+0000","last_change":"2024-07-26T07:47:45.843360+0000","last_active":"2024-07-26T07:49:48.090542+0000","last_peered":"2024-07-26T07:49:48.090542+0000","last_clean":"2024-07-26T07:49:48.090542+0000","last_became_active":"2024-07-26T07:47:45.842971+0000","last_became_peered":"2024-07-26T07:47:45.842971+0000","last_unstale":"2024-07-26T07:49:48.090542+0000","last_undegraded":"2024-07-26T07:49:48.090542+0000","last_fullsized":"2024-07-26T07:49:48.090542+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:58.687638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:43.556551+0000","last_change":"2024-07-26T07:47:41.787760+0000","last_active":"2024-07-26T07:51:43.556551+0000","last_peered":"2024-07-26T07:51:43.556551+0000","last_clean":"2024-07-26T07:51:43.556551+0000","last_became_active":"2024-07-26T07:47:41.787197+0000","last_became_peered":"2024-07-26T07:47:41.787197+0000","last_unstale":"2024-07-26T07:51:43.556551+0000","last_undegraded":"2024-07-26T07:51:43.556551+0000","last_fullsized":"2024-07-26T07:51:43.556551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:14.597759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092722+0000","last_change":"2024-07-26T07:47:45.846822+0000","last_active":"2024-07-26T07:49:48.092722+0000","last_peered":"2024-07-26T07:49:48.092722+0000","last_clean":"2024-07-26T07:49:48.092722+0000","last_became_active":"2024-07-26T07:47:45.846398+0000","last_became_peered":"2024-07-26T07:47:45.846398+0000","last_unstale":"2024-07-26T07:49:48.092722+0000","last_undegraded":"2024-07-26T07:49:48.092722+0000","last_fullsized":"2024-07-26T07:49:48.092722+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:33:34.727843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.311933+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:49:20.311933+0000","last_peered":"2024-07-26T07:49:20.311933+0000","last_clean":"2024-07-26T07:49:20.311933+0000","last_became_active":"2024-07-26T07:47:41.782077+0000","last_became_peered":"2024-07-26T07:47:41.782077+0000","last_unstale":"2024-07-26T07:49:20.311933+0000","last_undegraded":"2024-07-26T07:49:20.311933+0000","last_fullsized":"2024-07-26T07:49:20.311933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:42.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429076+0000","last_change":"2024-07-26T07:47:41.784615+0000","last_active":"2024-07-26T07:49:20.429076+0000","last_peered":"2024-07-26T07:49:20.429076+0000","last_clean":"2024-07-26T07:49:20.429076+0000","last_became_active":"2024-07-26T07:47:41.783945+0000","last_became_peered":"2024-07-26T07:47:41.783945+0000","last_unstale":"2024-07-26T07:49:20.429076+0000","last_undegraded":"2024-07-26T07:49:20.429076+0000","last_fullsized":"2024-07-26T07:49:20.429076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:42.618524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095869+0000","last_change":"2024-07-26T07:47:45.838774+0000","last_active":"2024-07-26T07:49:48.095869+0000","last_peered":"2024-07-26T07:49:48.095869+0000","last_clean":"2024-07-26T07:49:48.095869+0000","last_became_active":"2024-07-26T07:47:45.837707+0000","last_became_peered":"2024-07-26T07:47:45.837707+0000","last_unstale":"2024-07-26T07:49:48.095869+0000","last_undegraded":"2024-07-26T07:49:48.095869+0000","last_fullsized":"2024-07-26T07:49:48.095869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:59.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297619+0000","last_change":"2024-07-26T07:47:41.792031+0000","last_active":"2024-07-26T07:49:20.297619+0000","last_peered":"2024-07-26T07:49:20.297619+0000","last_clean":"2024-07-26T07:49:20.297619+0000","last_became_active":"2024-07-26T07:47:41.785955+0000","last_became_peered":"2024-07-26T07:47:41.785955+0000","last_unstale":"2024-07-26T07:49:20.297619+0000","last_undegraded":"2024-07-26T07:49:20.297619+0000","last_fullsized":"2024-07-26T07:49:20.297619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:01:47.073970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.094145+0000","last_change":"2024-07-26T07:47:45.844595+0000","last_active":"2024-07-26T07:49:48.094145+0000","last_peered":"2024-07-26T07:49:48.094145+0000","last_clean":"2024-07-26T07:49:48.094145+0000","last_became_active":"2024-07-26T07:47:45.843381+0000","last_became_peered":"2024-07-26T07:47:45.843381+0000","last_unstale":"2024-07-26T07:49:48.094145+0000","last_undegraded":"2024-07-26T07:49:48.094145+0000","last_fullsized":"2024-07-26T07:49:48.094145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:39.692463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'7","reported_seq":467,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:41.545148+0000","last_change":"2024-07-26T07:47:41.788239+0000","last_active":"2024-07-26T07:51:41.545148+0000","last_peered":"2024-07-26T07:51:41.545148+0000","last_clean":"2024-07-26T07:51:41.545148+0000","last_became_active":"2024-07-26T07:47:41.787470+0000","last_became_peered":"2024-07-26T07:47:41.787470+0000","last_unstale":"2024-07-26T07:51:41.545148+0000","last_undegraded":"2024-07-26T07:51:41.545148+0000","last_fullsized":"2024-07-26T07:51:41.545148+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:03.129563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":428,"num_read_kb":428,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112494+0000","last_change":"2024-07-26T07:47:45.852794+0000","last_active":"2024-07-26T07:49:48.112494+0000","last_peered":"2024-07-26T07:49:48.112494+0000","last_clean":"2024-07-26T07:49:48.112494+0000","last_became_active":"2024-07-26T07:47:45.849699+0000","last_became_peered":"2024-07-26T07:47:45.849699+0000","last_unstale":"2024-07-26T07:49:48.112494+0000","last_undegraded":"2024-07-26T07:49:48.112494+0000","last_fullsized":"2024-07-26T07:49:48.112494+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:06:31.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.315536+0000","last_change":"2024-07-26T07:47:41.789414+0000","last_active":"2024-07-26T07:49:20.315536+0000","last_peered":"2024-07-26T07:49:20.315536+0000","last_clean":"2024-07-26T07:49:20.315536+0000","last_became_active":"2024-07-26T07:47:41.788679+0000","last_became_peered":"2024-07-26T07:47:41.788679+0000","last_unstale":"2024-07-26T07:49:20.315536+0000","last_undegraded":"2024-07-26T07:49:20.315536+0000","last_fullsized":"2024-07-26T07:49:20.315536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:51.809997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099375+0000","last_change":"2024-07-26T07:48:19.937603+0000","last_active":"2024-07-26T07:49:48.099375+0000","last_peered":"2024-07-26T07:49:48.099375+0000","last_clean":"2024-07-26T07:49:48.099375+0000","last_became_active":"2024-07-26T07:48:19.936889+0000","last_became_peered":"2024-07-26T07:48:19.936889+0000","last_unstale":"2024-07-26T07:49:48.099375+0000","last_undegraded":"2024-07-26T07:49:48.099375+0000","last_fullsized":"2024-07-26T07:49:48.099375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:46:35.138825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298028+0000","last_change":"2024-07-26T07:47:42.564288+0000","last_active":"2024-07-26T07:49:20.298028+0000","last_peered":"2024-07-26T07:49:20.298028+0000","last_clean":"2024-07-26T07:49:20.298028+0000","last_became_active":"2024-07-26T07:47:42.561766+0000","last_became_peered":"2024-07-26T07:47:42.561766+0000","last_unstale":"2024-07-26T07:49:20.298028+0000","last_undegraded":"2024-07-26T07:49:20.298028+0000","last_fullsized":"2024-07-26T07:49:20.298028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:16.414030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.113511+0000","last_change":"2024-07-26T07:47:45.861895+0000","last_active":"2024-07-26T07:49:48.113511+0000","last_peered":"2024-07-26T07:49:48.113511+0000","last_clean":"2024-07-26T07:49:48.113511+0000","last_became_active":"2024-07-26T07:47:45.861439+0000","last_became_peered":"2024-07-26T07:47:45.861439+0000","last_unstale":"2024-07-26T07:49:48.113511+0000","last_undegraded":"2024-07-26T07:49:48.113511+0000","last_fullsized":"2024-07-26T07:49:48.113511+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:26.276453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'37","reported_seq":508,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:41.548437+0000","last_change":"2024-07-26T07:47:41.789977+0000","last_active":"2024-07-26T07:51:41.548437+0000","last_peered":"2024-07-26T07:51:41.548437+0000","last_clean":"2024-07-26T07:51:41.548437+0000","last_became_active":"2024-07-26T07:47:41.789488+0000","last_became_peered":"2024-07-26T07:47:41.789488+0000","last_unstale":"2024-07-26T07:51:41.548437+0000","last_undegraded":"2024-07-26T07:51:41.548437+0000","last_fullsized":"2024-07-26T07:51:41.548437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:33.059279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65209,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":439,"num_read_kb":278,"num_write":39,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098321+0000","last_change":"2024-07-26T07:47:45.854772+0000","last_active":"2024-07-26T07:49:48.098321+0000","last_peered":"2024-07-26T07:49:48.098321+0000","last_clean":"2024-07-26T07:49:48.098321+0000","last_became_active":"2024-07-26T07:47:45.854354+0000","last_became_peered":"2024-07-26T07:47:45.854354+0000","last_unstale":"2024-07-26T07:49:48.098321+0000","last_undegraded":"2024-07-26T07:49:48.098321+0000","last_fullsized":"2024-07-26T07:49:48.098321+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:31:54.856407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313468+0000","last_change":"2024-07-26T07:47:41.789752+0000","last_active":"2024-07-26T07:49:20.313468+0000","last_peered":"2024-07-26T07:49:20.313468+0000","last_clean":"2024-07-26T07:49:20.313468+0000","last_became_active":"2024-07-26T07:47:41.788952+0000","last_became_peered":"2024-07-26T07:47:41.788952+0000","last_unstale":"2024-07-26T07:49:20.313468+0000","last_undegraded":"2024-07-26T07:49:20.313468+0000","last_fullsized":"2024-07-26T07:49:20.313468+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:45.141693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101283+0000","last_change":"2024-07-26T07:47:45.845120+0000","last_active":"2024-07-26T07:49:48.101283+0000","last_peered":"2024-07-26T07:49:48.101283+0000","last_clean":"2024-07-26T07:49:48.101283+0000","last_became_active":"2024-07-26T07:47:45.844699+0000","last_became_peered":"2024-07-26T07:47:45.844699+0000","last_unstale":"2024-07-26T07:49:48.101283+0000","last_undegraded":"2024-07-26T07:49:48.101283+0000","last_fullsized":"2024-07-26T07:49:48.101283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:23:42.130846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298579+0000","last_change":"2024-07-26T07:47:41.790765+0000","last_active":"2024-07-26T07:49:20.298579+0000","last_peered":"2024-07-26T07:49:20.298579+0000","last_clean":"2024-07-26T07:49:20.298579+0000","last_became_active":"2024-07-26T07:47:41.790288+0000","last_became_peered":"2024-07-26T07:47:41.790288+0000","last_unstale":"2024-07-26T07:49:20.298579+0000","last_undegraded":"2024-07-26T07:49:20.298579+0000","last_fullsized":"2024-07-26T07:49:20.298579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:05.734468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.102627+0000","last_change":"2024-07-26T07:47:45.854648+0000","last_active":"2024-07-26T07:49:48.102627+0000","last_peered":"2024-07-26T07:49:48.102627+0000","last_clean":"2024-07-26T07:49:48.102627+0000","last_became_active":"2024-07-26T07:47:45.854240+0000","last_became_peered":"2024-07-26T07:47:45.854240+0000","last_unstale":"2024-07-26T07:49:48.102627+0000","last_undegraded":"2024-07-26T07:49:48.102627+0000","last_fullsized":"2024-07-26T07:49:48.102627+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:41.775760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'7","reported_seq":275,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:43.790014+0000","last_change":"2024-07-26T07:49:20.331153+0000","last_active":"2024-07-26T07:51:43.790014+0000","last_peered":"2024-07-26T07:51:43.790014+0000","last_clean":"2024-07-26T07:51:43.790014+0000","last_became_active":"2024-07-26T07:49:20.315423+0000","last_became_peered":"2024-07-26T07:49:20.315423+0000","last_unstale":"2024-07-26T07:51:43.790014+0000","last_undegraded":"2024-07-26T07:51:43.790014+0000","last_fullsized":"2024-07-26T07:51:43.790014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:24.476818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1472,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.100845+0000","last_change":"2024-07-26T07:47:45.848731+0000","last_active":"2024-07-26T07:49:48.100845+0000","last_peered":"2024-07-26T07:49:48.100845+0000","last_clean":"2024-07-26T07:49:48.100845+0000","last_became_active":"2024-07-26T07:47:45.846663+0000","last_became_peered":"2024-07-26T07:47:45.846663+0000","last_unstale":"2024-07-26T07:49:48.100845+0000","last_undegraded":"2024-07-26T07:49:48.100845+0000","last_fullsized":"2024-07-26T07:49:48.100845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:05.251423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:23.742120+0000","last_change":"2024-07-26T07:49:20.355238+0000","last_active":"2024-07-26T07:49:23.742120+0000","last_peered":"2024-07-26T07:49:23.742120+0000","last_clean":"2024-07-26T07:49:23.742120+0000","last_became_active":"2024-07-26T07:49:20.316591+0000","last_became_peered":"2024-07-26T07:49:20.316591+0000","last_unstale":"2024-07-26T07:49:23.742120+0000","last_undegraded":"2024-07-26T07:49:23.742120+0000","last_fullsized":"2024-07-26T07:49:23.742120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:12.739473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088622+0000","last_change":"2024-07-26T07:47:45.842572+0000","last_active":"2024-07-26T07:49:48.088622+0000","last_peered":"2024-07-26T07:49:48.088622+0000","last_clean":"2024-07-26T07:49:48.088622+0000","last_became_active":"2024-07-26T07:47:45.842116+0000","last_became_peered":"2024-07-26T07:47:45.842116+0000","last_unstale":"2024-07-26T07:49:48.088622+0000","last_undegraded":"2024-07-26T07:49:48.088622+0000","last_fullsized":"2024-07-26T07:49:48.088622+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:06.913737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:59.385245+0000","last_change":"2024-07-26T07:47:41.790215+0000","last_active":"2024-07-26T07:49:59.385245+0000","last_peered":"2024-07-26T07:49:59.385245+0000","last_clean":"2024-07-26T07:49:59.385245+0000","last_became_active":"2024-07-26T07:47:41.789691+0000","last_became_peered":"2024-07-26T07:47:41.789691+0000","last_unstale":"2024-07-26T07:49:59.385245+0000","last_undegraded":"2024-07-26T07:49:59.385245+0000","last_fullsized":"2024-07-26T07:49:59.385245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:01:47.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086806+0000","last_change":"2024-07-26T07:47:45.860887+0000","last_active":"2024-07-26T07:49:48.086806+0000","last_peered":"2024-07-26T07:49:48.086806+0000","last_clean":"2024-07-26T07:49:48.086806+0000","last_became_active":"2024-07-26T07:47:45.860315+0000","last_became_peered":"2024-07-26T07:47:45.860315+0000","last_unstale":"2024-07-26T07:49:48.086806+0000","last_undegraded":"2024-07-26T07:49:48.086806+0000","last_fullsized":"2024-07-26T07:49:48.086806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:17:02.231206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429700+0000","last_change":"2024-07-26T07:48:20.022885+0000","last_active":"2024-07-26T07:49:20.429700+0000","last_peered":"2024-07-26T07:49:20.429700+0000","last_clean":"2024-07-26T07:49:20.429700+0000","last_became_active":"2024-07-26T07:48:20.001946+0000","last_became_peered":"2024-07-26T07:48:20.001946+0000","last_unstale":"2024-07-26T07:49:20.429700+0000","last_undegraded":"2024-07-26T07:49:20.429700+0000","last_fullsized":"2024-07-26T07:49:20.429700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:45.892886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098632+0000","last_change":"2024-07-26T07:47:45.858267+0000","last_active":"2024-07-26T07:49:48.098632+0000","last_peered":"2024-07-26T07:49:48.098632+0000","last_clean":"2024-07-26T07:49:48.098632+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.098632+0000","last_undegraded":"2024-07-26T07:49:48.098632+0000","last_fullsized":"2024-07-26T07:49:48.098632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:32.055977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298413+0000","last_change":"2024-07-26T07:47:41.789846+0000","last_active":"2024-07-26T07:49:20.298413+0000","last_peered":"2024-07-26T07:49:20.298413+0000","last_clean":"2024-07-26T07:49:20.298413+0000","last_became_active":"2024-07-26T07:47:41.789237+0000","last_became_peered":"2024-07-26T07:47:41.789237+0000","last_unstale":"2024-07-26T07:49:20.298413+0000","last_undegraded":"2024-07-26T07:49:20.298413+0000","last_fullsized":"2024-07-26T07:49:20.298413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:39.031047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095798+0000","last_change":"2024-07-26T07:47:45.846294+0000","last_active":"2024-07-26T07:49:48.095798+0000","last_peered":"2024-07-26T07:49:48.095798+0000","last_clean":"2024-07-26T07:49:48.095798+0000","last_became_active":"2024-07-26T07:47:45.845861+0000","last_became_peered":"2024-07-26T07:47:45.845861+0000","last_unstale":"2024-07-26T07:49:48.095798+0000","last_undegraded":"2024-07-26T07:49:48.095798+0000","last_fullsized":"2024-07-26T07:49:48.095798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:56:10.594430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'9","reported_seq":265,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:44.558871+0000","last_change":"2024-07-26T07:47:42.563412+0000","last_active":"2024-07-26T07:51:44.558871+0000","last_peered":"2024-07-26T07:51:44.558871+0000","last_clean":"2024-07-26T07:51:44.558871+0000","last_became_active":"2024-07-26T07:47:42.562364+0000","last_became_peered":"2024-07-26T07:47:42.562364+0000","last_unstale":"2024-07-26T07:51:44.558871+0000","last_undegraded":"2024-07-26T07:51:44.558871+0000","last_fullsized":"2024-07-26T07:51:44.558871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:14.796738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2586,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":224,"num_read_kb":6,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.108677+0000","last_change":"2024-07-26T07:47:45.856170+0000","last_active":"2024-07-26T07:49:48.108677+0000","last_peered":"2024-07-26T07:49:48.108677+0000","last_clean":"2024-07-26T07:49:48.108677+0000","last_became_active":"2024-07-26T07:47:45.855782+0000","last_became_peered":"2024-07-26T07:47:45.855782+0000","last_unstale":"2024-07-26T07:49:48.108677+0000","last_undegraded":"2024-07-26T07:49:48.108677+0000","last_fullsized":"2024-07-26T07:49:48.108677+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:22.690910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313258+0000","last_change":"2024-07-26T07:45:54.310557+0000","last_active":"2024-07-26T07:49:20.313258+0000","last_peered":"2024-07-26T07:49:20.313258+0000","last_clean":"2024-07-26T07:49:20.313258+0000","last_became_active":"2024-07-26T07:45:54.290906+0000","last_became_peered":"2024-07-26T07:45:54.290906+0000","last_unstale":"2024-07-26T07:49:20.313258+0000","last_undegraded":"2024-07-26T07:49:20.313258+0000","last_fullsized":"2024-07-26T07:49:20.313258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_clean_scrub_stamp":"2024-07-26T07:41:16.689463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:51.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.305857+0000","last_change":"2024-07-26T07:47:41.776420+0000","last_active":"2024-07-26T07:49:20.305857+0000","last_peered":"2024-07-26T07:49:20.305857+0000","last_clean":"2024-07-26T07:49:20.305857+0000","last_became_active":"2024-07-26T07:47:41.775858+0000","last_became_peered":"2024-07-26T07:47:41.775858+0000","last_unstale":"2024-07-26T07:49:20.305857+0000","last_undegraded":"2024-07-26T07:49:20.305857+0000","last_fullsized":"2024-07-26T07:49:20.305857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:18.880331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.109733+0000","last_change":"2024-07-26T07:48:19.933318+0000","last_active":"2024-07-26T07:49:48.109733+0000","last_peered":"2024-07-26T07:49:48.109733+0000","last_clean":"2024-07-26T07:49:48.109733+0000","last_became_active":"2024-07-26T07:48:19.932683+0000","last_became_peered":"2024-07-26T07:48:19.932683+0000","last_unstale":"2024-07-26T07:49:48.109733+0000","last_undegraded":"2024-07-26T07:49:48.109733+0000","last_fullsized":"2024-07-26T07:49:48.109733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:25:23.387428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:43.375132+0000","last_change":"2024-07-26T07:47:41.786698+0000","last_active":"2024-07-26T07:49:43.375132+0000","last_peered":"2024-07-26T07:49:43.375132+0000","last_clean":"2024-07-26T07:49:43.375132+0000","last_became_active":"2024-07-26T07:47:41.784491+0000","last_became_peered":"2024-07-26T07:47:41.784491+0000","last_unstale":"2024-07-26T07:49:43.375132+0000","last_undegraded":"2024-07-26T07:49:43.375132+0000","last_fullsized":"2024-07-26T07:49:43.375132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:09:08.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093849+0000","last_change":"2024-07-26T07:47:45.849698+0000","last_active":"2024-07-26T07:49:48.093849+0000","last_peered":"2024-07-26T07:49:48.093849+0000","last_clean":"2024-07-26T07:49:48.093849+0000","last_became_active":"2024-07-26T07:47:45.849266+0000","last_became_peered":"2024-07-26T07:47:45.849266+0000","last_unstale":"2024-07-26T07:49:48.093849+0000","last_undegraded":"2024-07-26T07:49:48.093849+0000","last_fullsized":"2024-07-26T07:49:48.093849+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:57.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":255,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:42.785841+0000","last_change":"2024-07-26T07:47:42.558680+0000","last_active":"2024-07-26T07:51:42.785841+0000","last_peered":"2024-07-26T07:51:42.785841+0000","last_clean":"2024-07-26T07:51:42.785841+0000","last_became_active":"2024-07-26T07:47:42.558197+0000","last_became_peered":"2024-07-26T07:47:42.558197+0000","last_unstale":"2024-07-26T07:51:42.785841+0000","last_undegraded":"2024-07-26T07:51:42.785841+0000","last_fullsized":"2024-07-26T07:51:42.785841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:52:52.505455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":217,"num_read_kb":217,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.078802+0000","last_change":"2024-07-26T07:48:19.929897+0000","last_active":"2024-07-26T07:49:48.078802+0000","last_peered":"2024-07-26T07:49:48.078802+0000","last_clean":"2024-07-26T07:49:48.078802+0000","last_became_active":"2024-07-26T07:48:19.927946+0000","last_became_peered":"2024-07-26T07:48:19.927946+0000","last_unstale":"2024-07-26T07:49:48.078802+0000","last_undegraded":"2024-07-26T07:49:48.078802+0000","last_fullsized":"2024-07-26T07:49:48.078802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:11.157824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'6","reported_seq":255,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:41.529095+0000","last_change":"2024-07-26T07:47:41.785513+0000","last_active":"2024-07-26T07:51:41.529095+0000","last_peered":"2024-07-26T07:51:41.529095+0000","last_clean":"2024-07-26T07:51:41.529095+0000","last_became_active":"2024-07-26T07:47:41.784783+0000","last_became_peered":"2024-07-26T07:47:41.784783+0000","last_unstale":"2024-07-26T07:51:41.529095+0000","last_undegraded":"2024-07-26T07:51:41.529095+0000","last_fullsized":"2024-07-26T07:51:41.529095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:54.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.082468+0000","last_change":"2024-07-26T07:47:45.856580+0000","last_active":"2024-07-26T07:49:48.082468+0000","last_peered":"2024-07-26T07:49:48.082468+0000","last_clean":"2024-07-26T07:49:48.082468+0000","last_became_active":"2024-07-26T07:47:45.852330+0000","last_became_peered":"2024-07-26T07:47:45.852330+0000","last_unstale":"2024-07-26T07:49:48.082468+0000","last_undegraded":"2024-07-26T07:49:48.082468+0000","last_fullsized":"2024-07-26T07:49:48.082468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:09.479563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.428822+0000","last_change":"2024-07-26T07:47:42.559255+0000","last_active":"2024-07-26T07:49:20.428822+0000","last_peered":"2024-07-26T07:49:20.428822+0000","last_clean":"2024-07-26T07:49:20.428822+0000","last_became_active":"2024-07-26T07:47:42.558123+0000","last_became_peered":"2024-07-26T07:47:42.558123+0000","last_unstale":"2024-07-26T07:49:20.428822+0000","last_undegraded":"2024-07-26T07:49:20.428822+0000","last_fullsized":"2024-07-26T07:49:20.428822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:49:46.576674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092093+0000","last_change":"2024-07-26T07:47:45.856256+0000","last_active":"2024-07-26T07:49:48.092093+0000","last_peered":"2024-07-26T07:49:48.092093+0000","last_clean":"2024-07-26T07:49:48.092093+0000","last_became_active":"2024-07-26T07:47:45.855696+0000","last_became_peered":"2024-07-26T07:47:45.855696+0000","last_unstale":"2024-07-26T07:49:48.092093+0000","last_undegraded":"2024-07-26T07:49:48.092093+0000","last_fullsized":"2024-07-26T07:49:48.092093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:24.451225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314969+0000","last_change":"2024-07-26T07:47:42.562631+0000","last_active":"2024-07-26T07:49:20.314969+0000","last_peered":"2024-07-26T07:49:20.314969+0000","last_clean":"2024-07-26T07:49:20.314969+0000","last_became_active":"2024-07-26T07:47:42.561379+0000","last_became_peered":"2024-07-26T07:47:42.561379+0000","last_unstale":"2024-07-26T07:49:20.314969+0000","last_undegraded":"2024-07-26T07:49:20.314969+0000","last_fullsized":"2024-07-26T07:49:20.314969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:03:30.471694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092440+0000","last_change":"2024-07-26T07:47:45.849580+0000","last_active":"2024-07-26T07:49:48.092440+0000","last_peered":"2024-07-26T07:49:48.092440+0000","last_clean":"2024-07-26T07:49:48.092440+0000","last_became_active":"2024-07-26T07:47:45.849107+0000","last_became_peered":"2024-07-26T07:47:45.849107+0000","last_unstale":"2024-07-26T07:49:48.092440+0000","last_undegraded":"2024-07-26T07:49:48.092440+0000","last_fullsized":"2024-07-26T07:49:48.092440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:57:13.694059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'18","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:38.899550+0000","last_change":"2024-07-26T07:47:41.774083+0000","last_active":"2024-07-26T07:51:38.899550+0000","last_peered":"2024-07-26T07:51:38.899550+0000","last_clean":"2024-07-26T07:51:38.899550+0000","last_became_active":"2024-07-26T07:47:41.773502+0000","last_became_peered":"2024-07-26T07:47:41.773502+0000","last_unstale":"2024-07-26T07:51:38.899550+0000","last_undegraded":"2024-07-26T07:51:38.899550+0000","last_fullsized":"2024-07-26T07:51:38.899550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:11:44.083944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099359+0000","last_change":"2024-07-26T07:47:45.845894+0000","last_active":"2024-07-26T07:49:48.099359+0000","last_peered":"2024-07-26T07:49:48.099359+0000","last_clean":"2024-07-26T07:49:48.099359+0000","last_became_active":"2024-07-26T07:47:45.845534+0000","last_became_peered":"2024-07-26T07:47:45.845534+0000","last_unstale":"2024-07-26T07:49:48.099359+0000","last_undegraded":"2024-07-26T07:49:48.099359+0000","last_fullsized":"2024-07-26T07:49:48.099359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:52:42.209561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.306134+0000","last_change":"2024-07-26T07:47:41.904220+0000","last_active":"2024-07-26T07:49:20.306134+0000","last_peered":"2024-07-26T07:49:20.306134+0000","last_clean":"2024-07-26T07:49:20.306134+0000","last_became_active":"2024-07-26T07:47:41.902349+0000","last_became_peered":"2024-07-26T07:47:41.902349+0000","last_unstale":"2024-07-26T07:49:20.306134+0000","last_undegraded":"2024-07-26T07:49:20.306134+0000","last_fullsized":"2024-07-26T07:49:20.306134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:08.520461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092199+0000","last_change":"2024-07-26T07:47:45.846920+0000","last_active":"2024-07-26T07:49:48.092199+0000","last_peered":"2024-07-26T07:49:48.092199+0000","last_clean":"2024-07-26T07:49:48.092199+0000","last_became_active":"2024-07-26T07:47:45.846458+0000","last_became_peered":"2024-07-26T07:47:45.846458+0000","last_unstale":"2024-07-26T07:49:48.092199+0000","last_undegraded":"2024-07-26T07:49:48.092199+0000","last_fullsized":"2024-07-26T07:49:48.092199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:06.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.295007+0000","last_change":"2024-07-26T07:47:42.560346+0000","last_active":"2024-07-26T07:49:20.295007+0000","last_peered":"2024-07-26T07:49:20.295007+0000","last_clean":"2024-07-26T07:49:20.295007+0000","last_became_active":"2024-07-26T07:47:42.558937+0000","last_became_peered":"2024-07-26T07:47:42.558937+0000","last_unstale":"2024-07-26T07:49:20.295007+0000","last_undegraded":"2024-07-26T07:49:20.295007+0000","last_fullsized":"2024-07-26T07:49:20.295007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.676625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.081730+0000","last_change":"2024-07-26T07:47:45.845553+0000","last_active":"2024-07-26T07:49:48.081730+0000","last_peered":"2024-07-26T07:49:48.081730+0000","last_clean":"2024-07-26T07:49:48.081730+0000","last_became_active":"2024-07-26T07:47:45.845125+0000","last_became_peered":"2024-07-26T07:47:45.845125+0000","last_unstale":"2024-07-26T07:49:48.081730+0000","last_undegraded":"2024-07-26T07:49:48.081730+0000","last_fullsized":"2024-07-26T07:49:48.081730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:10.896159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.292557+0000","last_change":"2024-07-26T07:47:41.779688+0000","last_active":"2024-07-26T07:49:20.292557+0000","last_peered":"2024-07-26T07:49:20.292557+0000","last_clean":"2024-07-26T07:49:20.292557+0000","last_became_active":"2024-07-26T07:47:41.779115+0000","last_became_peered":"2024-07-26T07:47:41.779115+0000","last_unstale":"2024-07-26T07:49:20.292557+0000","last_undegraded":"2024-07-26T07:49:20.292557+0000","last_fullsized":"2024-07-26T07:49:20.292557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:20.708336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091558+0000","last_change":"2024-07-26T07:47:45.853869+0000","last_active":"2024-07-26T07:49:48.091558+0000","last_peered":"2024-07-26T07:49:48.091558+0000","last_clean":"2024-07-26T07:49:48.091558+0000","last_became_active":"2024-07-26T07:47:45.852509+0000","last_became_peered":"2024-07-26T07:47:45.852509+0000","last_unstale":"2024-07-26T07:49:48.091558+0000","last_undegraded":"2024-07-26T07:49:48.091558+0000","last_fullsized":"2024-07-26T07:49:48.091558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:01:31.764025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'5","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:38.797885+0000","last_change":"2024-07-26T07:47:41.792331+0000","last_active":"2024-07-26T07:51:38.797885+0000","last_peered":"2024-07-26T07:51:38.797885+0000","last_clean":"2024-07-26T07:51:38.797885+0000","last_became_active":"2024-07-26T07:47:41.790668+0000","last_became_peered":"2024-07-26T07:47:41.790668+0000","last_unstale":"2024-07-26T07:51:38.797885+0000","last_undegraded":"2024-07-26T07:51:38.797885+0000","last_fullsized":"2024-07-26T07:51:38.797885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:59.430692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096588+0000","last_change":"2024-07-26T07:47:45.843583+0000","last_active":"2024-07-26T07:49:48.096588+0000","last_peered":"2024-07-26T07:49:48.096588+0000","last_clean":"2024-07-26T07:49:48.096588+0000","last_became_active":"2024-07-26T07:47:45.843185+0000","last_became_peered":"2024-07-26T07:47:45.843185+0000","last_unstale":"2024-07-26T07:49:48.096588+0000","last_undegraded":"2024-07-26T07:49:48.096588+0000","last_fullsized":"2024-07-26T07:49:48.096588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:01.119065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314240+0000","last_change":"2024-07-26T07:48:20.014411+0000","last_active":"2024-07-26T07:49:20.314240+0000","last_peered":"2024-07-26T07:49:20.314240+0000","last_clean":"2024-07-26T07:49:20.314240+0000","last_became_active":"2024-07-26T07:48:20.002085+0000","last_became_peered":"2024-07-26T07:48:20.002085+0000","last_unstale":"2024-07-26T07:49:20.314240+0000","last_undegraded":"2024-07-26T07:49:20.314240+0000","last_fullsized":"2024-07-26T07:49:20.314240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:46:59.335128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.110858+0000","last_change":"2024-07-26T07:47:45.851963+0000","last_active":"2024-07-26T07:49:48.110858+0000","last_peered":"2024-07-26T07:49:48.110858+0000","last_clean":"2024-07-26T07:49:48.110858+0000","last_became_active":"2024-07-26T07:47:45.850459+0000","last_became_peered":"2024-07-26T07:47:45.850459+0000","last_unstale":"2024-07-26T07:49:48.110858+0000","last_undegraded":"2024-07-26T07:49:48.110858+0000","last_fullsized":"2024-07-26T07:49:48.110858+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:18.092186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096201+0000","last_change":"2024-07-26T07:47:45.848744+0000","last_active":"2024-07-26T07:49:48.096201+0000","last_peered":"2024-07-26T07:49:48.096201+0000","last_clean":"2024-07-26T07:49:48.096201+0000","last_became_active":"2024-07-26T07:47:45.848292+0000","last_became_peered":"2024-07-26T07:47:45.848292+0000","last_unstale":"2024-07-26T07:49:48.096201+0000","last_undegraded":"2024-07-26T07:49:48.096201+0000","last_fullsized":"2024-07-26T07:49:48.096201+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:54.549508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314854+0000","last_change":"2024-07-26T07:47:41.788875+0000","last_active":"2024-07-26T07:49:20.314854+0000","last_peered":"2024-07-26T07:49:20.314854+0000","last_clean":"2024-07-26T07:49:20.314854+0000","last_became_active":"2024-07-26T07:47:41.786942+0000","last_became_peered":"2024-07-26T07:47:41.786942+0000","last_unstale":"2024-07-26T07:49:20.314854+0000","last_undegraded":"2024-07-26T07:49:20.314854+0000","last_fullsized":"2024-07-26T07:49:20.314854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:11:31.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111571+0000","last_change":"2024-07-26T07:47:45.854832+0000","last_active":"2024-07-26T07:49:48.111571+0000","last_peered":"2024-07-26T07:49:48.111571+0000","last_clean":"2024-07-26T07:49:48.111571+0000","last_became_active":"2024-07-26T07:47:45.853756+0000","last_became_peered":"2024-07-26T07:47:45.853756+0000","last_unstale":"2024-07-26T07:49:48.111571+0000","last_undegraded":"2024-07-26T07:49:48.111571+0000","last_fullsized":"2024-07-26T07:49:48.111571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:42:14.579177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314889+0000","last_change":"2024-07-26T07:47:41.787839+0000","last_active":"2024-07-26T07:49:20.314889+0000","last_peered":"2024-07-26T07:49:20.314889+0000","last_clean":"2024-07-26T07:49:20.314889+0000","last_became_active":"2024-07-26T07:47:41.785951+0000","last_became_peered":"2024-07-26T07:47:41.785951+0000","last_unstale":"2024-07-26T07:49:20.314889+0000","last_undegraded":"2024-07-26T07:49:20.314889+0000","last_fullsized":"2024-07-26T07:49:20.314889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:58.699056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104028+0000","last_change":"2024-07-26T07:47:45.841451+0000","last_active":"2024-07-26T07:49:48.104028+0000","last_peered":"2024-07-26T07:49:48.104028+0000","last_clean":"2024-07-26T07:49:48.104028+0000","last_became_active":"2024-07-26T07:47:45.840803+0000","last_became_peered":"2024-07-26T07:47:45.840803+0000","last_unstale":"2024-07-26T07:49:48.104028+0000","last_undegraded":"2024-07-26T07:49:48.104028+0000","last_fullsized":"2024-07-26T07:49:48.104028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:24:21.832317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:44.189574+0000","last_change":"2024-07-26T07:47:42.558775+0000","last_active":"2024-07-26T07:49:44.189574+0000","last_peered":"2024-07-26T07:49:44.189574+0000","last_clean":"2024-07-26T07:49:44.189574+0000","last_became_active":"2024-07-26T07:47:42.558300+0000","last_became_peered":"2024-07-26T07:47:42.558300+0000","last_unstale":"2024-07-26T07:49:44.189574+0000","last_undegraded":"2024-07-26T07:49:44.189574+0000","last_fullsized":"2024-07-26T07:49:44.189574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:59.643709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091602+0000","last_change":"2024-07-26T07:47:45.852401+0000","last_active":"2024-07-26T07:49:48.091602+0000","last_peered":"2024-07-26T07:49:48.091602+0000","last_clean":"2024-07-26T07:49:48.091602+0000","last_became_active":"2024-07-26T07:47:45.852054+0000","last_became_peered":"2024-07-26T07:47:45.852054+0000","last_unstale":"2024-07-26T07:49:48.091602+0000","last_undegraded":"2024-07-26T07:49:48.091602+0000","last_fullsized":"2024-07-26T07:49:48.091602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:25.347770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"86'14","reported_seq":284,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:42.531152+0000","last_change":"2024-07-26T07:48:19.961760+0000","last_active":"2024-07-26T07:51:42.531152+0000","last_peered":"2024-07-26T07:51:42.531152+0000","last_clean":"2024-07-26T07:51:42.531152+0000","last_became_active":"2024-07-26T07:48:19.937694+0000","last_became_peered":"2024-07-26T07:48:19.937694+0000","last_unstale":"2024-07-26T07:51:42.531152+0000","last_undegraded":"2024-07-26T07:51:42.531152+0000","last_fullsized":"2024-07-26T07:51:42.531152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:05:10.571086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":211,"num_read_kb":211,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74917,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2195,"num_read_kb":1169,"num_write":274,"num_write_kb":161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1372,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":229791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579972,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":466032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776202,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1662976,"data_stored":1120154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972433,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":462962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168665,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":462556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397598,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1055865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561125,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":463172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757358,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1589248,"data_stored":1055638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953588,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":463424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":462504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411457,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":533450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":529966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":465434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":70646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T07:51:48.848 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T07:51:48.849 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T07:51:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:49 smithi115 ceph-mon[35364]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:49 smithi181 ceph-mon[35869]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:49 smithi097 ceph-mon[32231]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:50 smithi115 ceph-mon[35364]: pgmap v511: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-07-26T07:51:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:50 smithi181 ceph-mon[35869]: pgmap v511: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-07-26T07:51:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:50 smithi097 ceph-mon[32231]: pgmap v511: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-07-26T07:51:51.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 -- 172.21.15.97:0/2976782237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4102790 msgr2=0x7effb410f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:51.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 --2- 172.21.15.97:0/2976782237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4102790 0x7effb410f4b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7effa0009900 tx=0x7effa002a480 comp rx=0 tx=0).stop 2024-07-26T07:51:51.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 -- 172.21.15.97:0/2976782237 shutdown_connections 2024-07-26T07:51:51.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 --2- 172.21.15.97:0/2976782237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7effb410f9f0 0x7effb4111e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 --2- 172.21.15.97:0/2976782237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4102790 0x7effb410f4b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 --2- 172.21.15.97:0/2976782237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb4101e40 0x7effb4102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 -- 172.21.15.97:0/2976782237 >> 172.21.15.97:0/2976782237 conn(0x7effb40fdd60 msgr2=0x7effb41001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:51.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 -- 172.21.15.97:0/2976782237 shutdown_connections 2024-07-26T07:51:51.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.319+0000 7effbaa00640 1 -- 172.21.15.97:0/2976782237 wait complete. 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.320+0000 7effbaa00640 1 Processor -- start 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.320+0000 7effbaa00640 1 -- start start 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effbaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4101e40 0x7effb419be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effbaa00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7effb4102790 0x7effb419c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effbaa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb410f9f0 0x7effb41a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effbaa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7effb41153c0 con 0x7effb4101e40 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effbaa00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7effb4115240 con 0x7effb4102790 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effbaa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7effb4115540 con 0x7effb410f9f0 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effb9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4101e40 0x7effb419be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effb9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4101e40 0x7effb419be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42322/0 (socket says 172.21.15.97:42322) 2024-07-26T07:51:51.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effb9600640 1 -- 172.21.15.97:0/2203362887 learned_addr learned my addr 172.21.15.97:0/2203362887 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:51.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effba000640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb410f9f0 0x7effb41a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.321+0000 7effb8c00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7effb4102790 0x7effb419c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb9600640 1 -- 172.21.15.97:0/2203362887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb410f9f0 msgr2=0x7effb41a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb9600640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb410f9f0 0x7effb41a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb9600640 1 -- 172.21.15.97:0/2203362887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7effb4102790 msgr2=0x7effb419c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb9600640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7effb4102790 0x7effb419c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb9600640 1 -- 172.21.15.97:0/2203362887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb41a1d80 con 0x7effb4101e40 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb9600640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4101e40 0x7effb419be10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7effa800ea80 tx=0x7effa800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb2000640 1 -- 172.21.15.97:0/2203362887 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa800ce40 con 0x7effb4101e40 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb2000640 1 -- 172.21.15.97:0/2203362887 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effa8005020 con 0x7effb4101e40 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effb2000640 1 -- 172.21.15.97:0/2203362887 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa80116d0 con 0x7effb4101e40 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb41a2070 con 0x7effb4101e40 2024-07-26T07:51:51.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.322+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb41a87d0 con 0x7effb4101e40 2024-07-26T07:51:51.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.324+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb4102250 con 0x7effb4101e40 2024-07-26T07:51:51.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.326+0000 7effb2000640 1 -- 172.21.15.97:0/2203362887 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7effa8004a40 con 0x7effb4101e40 2024-07-26T07:51:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.328+0000 7effb2000640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff9c0b3c60 0x7eff9c0b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.328+0000 7effb2000640 1 -- 172.21.15.97:0/2203362887 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7effa80e44c0 con 0x7effb4101e40 2024-07-26T07:51:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.328+0000 7effb8c00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff9c0b3c60 0x7eff9c0b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:51.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.329+0000 7effb8c00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff9c0b3c60 0x7eff9c0b6170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7effa0006010 tx=0x7effa0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:51.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.332+0000 7effb2000640 1 -- 172.21.15.97:0/2203362887 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7effa8016030 con 0x7effb4101e40 2024-07-26T07:51:51.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.530+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 --> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7effb4063230 con 0x7eff9c0b3c60 2024-07-26T07:51:51.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.543+0000 7effb2000640 1 -- 172.21.15.97:0/2203362887 <== mgr.14150 v2:172.21.15.97:6800/2663280026 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258932 (secure 0 0 0) 0x7effb4063230 con 0x7eff9c0b3c60 2024-07-26T07:51:51.547 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:51.549 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.547+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff9c0b3c60 msgr2=0x7eff9c0b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.547+0000 7effbaa00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff9c0b3c60 0x7eff9c0b6170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7effa0006010 tx=0x7effa0000bc0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.547+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4101e40 msgr2=0x7effb419be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.547+0000 7effbaa00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4101e40 0x7effb419be10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7effa800ea80 tx=0x7effa800ef40 comp rx=0 tx=0).stop 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.548+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 shutdown_connections 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.548+0000 7effbaa00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7eff9c0b3c60 0x7eff9c0b6170 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.548+0000 7effbaa00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb410f9f0 0x7effb41a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.548+0000 7effbaa00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7effb4102790 0x7effb419c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.548+0000 7effbaa00640 1 --2- 172.21.15.97:0/2203362887 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effb4101e40 0x7effb419be10 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:51.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.548+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 >> 172.21.15.97:0/2203362887 conn(0x7effb40fdd60 msgr2=0x7effb4100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:51.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.548+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 shutdown_connections 2024-07-26T07:51:51.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:51.548+0000 7effbaa00640 1 -- 172.21.15.97:0/2203362887 wait complete. 2024-07-26T07:51:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:52 smithi097 ceph-mon[32231]: pgmap v512: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-07-26T07:51:52.381 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-07-26T07:51:50.226927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668039,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2313,"num_read_kb":1266,"num_write":400,"num_write_kb":2316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2953212,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":828,"ondisk_log_size":828,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":404180,"kb_used_data":13428,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537644,"statfs":{"total":1151940427776,"available":1151526547456,"internally_reserved":0,"allocated":13750272,"data_stored":7546197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10472,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":129,"num_read_kb":75,"num_write":24,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005313"},"pg_stats":[{"pgid":"3.3f","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090598+0000","last_change":"2024-07-26T07:47:45.843670+0000","last_active":"2024-07-26T07:49:48.090598+0000","last_peered":"2024-07-26T07:49:48.090598+0000","last_clean":"2024-07-26T07:49:48.090598+0000","last_became_active":"2024-07-26T07:47:45.841137+0000","last_became_peered":"2024-07-26T07:47:45.841137+0000","last_unstale":"2024-07-26T07:49:48.090598+0000","last_undegraded":"2024-07-26T07:49:48.090598+0000","last_fullsized":"2024-07-26T07:49:48.090598+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:38.063503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085904+0000","last_change":"2024-07-26T07:47:45.849326+0000","last_active":"2024-07-26T07:49:48.085904+0000","last_peered":"2024-07-26T07:49:48.085904+0000","last_clean":"2024-07-26T07:49:48.085904+0000","last_became_active":"2024-07-26T07:47:45.847761+0000","last_became_peered":"2024-07-26T07:47:45.847761+0000","last_unstale":"2024-07-26T07:49:48.085904+0000","last_undegraded":"2024-07-26T07:49:48.085904+0000","last_fullsized":"2024-07-26T07:49:48.085904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:31.098794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092147+0000","last_change":"2024-07-26T07:47:45.852343+0000","last_active":"2024-07-26T07:49:48.092147+0000","last_peered":"2024-07-26T07:49:48.092147+0000","last_clean":"2024-07-26T07:49:48.092147+0000","last_became_active":"2024-07-26T07:47:45.851950+0000","last_became_peered":"2024-07-26T07:47:45.851950+0000","last_unstale":"2024-07-26T07:49:48.092147+0000","last_undegraded":"2024-07-26T07:49:48.092147+0000","last_fullsized":"2024-07-26T07:49:48.092147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:41.030903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091199+0000","last_change":"2024-07-26T07:47:45.857042+0000","last_active":"2024-07-26T07:49:48.091199+0000","last_peered":"2024-07-26T07:49:48.091199+0000","last_clean":"2024-07-26T07:49:48.091199+0000","last_became_active":"2024-07-26T07:47:45.855145+0000","last_became_peered":"2024-07-26T07:47:45.855145+0000","last_unstale":"2024-07-26T07:49:48.091199+0000","last_undegraded":"2024-07-26T07:49:48.091199+0000","last_fullsized":"2024-07-26T07:49:48.091199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:32.233019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092139+0000","last_change":"2024-07-26T07:47:45.838560+0000","last_active":"2024-07-26T07:49:48.092139+0000","last_peered":"2024-07-26T07:49:48.092139+0000","last_clean":"2024-07-26T07:49:48.092139+0000","last_became_active":"2024-07-26T07:47:45.835578+0000","last_became_peered":"2024-07-26T07:47:45.835578+0000","last_unstale":"2024-07-26T07:49:48.092139+0000","last_undegraded":"2024-07-26T07:49:48.092139+0000","last_fullsized":"2024-07-26T07:49:48.092139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:09:40.445625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.089953+0000","last_change":"2024-07-26T07:47:45.843980+0000","last_active":"2024-07-26T07:49:48.089953+0000","last_peered":"2024-07-26T07:49:48.089953+0000","last_clean":"2024-07-26T07:49:48.089953+0000","last_became_active":"2024-07-26T07:47:45.843546+0000","last_became_peered":"2024-07-26T07:47:45.843546+0000","last_unstale":"2024-07-26T07:49:48.089953+0000","last_undegraded":"2024-07-26T07:49:48.089953+0000","last_fullsized":"2024-07-26T07:49:48.089953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:06.839136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093383+0000","last_change":"2024-07-26T07:47:45.854427+0000","last_active":"2024-07-26T07:49:48.093383+0000","last_peered":"2024-07-26T07:49:48.093383+0000","last_clean":"2024-07-26T07:49:48.093383+0000","last_became_active":"2024-07-26T07:47:45.853495+0000","last_became_peered":"2024-07-26T07:47:45.853495+0000","last_unstale":"2024-07-26T07:49:48.093383+0000","last_undegraded":"2024-07-26T07:49:48.093383+0000","last_fullsized":"2024-07-26T07:49:48.093383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:34:34.110581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096799+0000","last_change":"2024-07-26T07:47:45.847732+0000","last_active":"2024-07-26T07:49:48.096799+0000","last_peered":"2024-07-26T07:49:48.096799+0000","last_clean":"2024-07-26T07:49:48.096799+0000","last_became_active":"2024-07-26T07:47:45.845583+0000","last_became_peered":"2024-07-26T07:47:45.845583+0000","last_unstale":"2024-07-26T07:49:48.096799+0000","last_undegraded":"2024-07-26T07:49:48.096799+0000","last_fullsized":"2024-07-26T07:49:48.096799+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:07:09.370076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.108141+0000","last_change":"2024-07-26T07:47:45.849962+0000","last_active":"2024-07-26T07:49:48.108141+0000","last_peered":"2024-07-26T07:49:48.108141+0000","last_clean":"2024-07-26T07:49:48.108141+0000","last_became_active":"2024-07-26T07:47:45.845766+0000","last_became_peered":"2024-07-26T07:47:45.845766+0000","last_unstale":"2024-07-26T07:49:48.108141+0000","last_undegraded":"2024-07-26T07:49:48.108141+0000","last_fullsized":"2024-07-26T07:49:48.108141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:41:56.602825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086656+0000","last_change":"2024-07-26T07:47:45.857287+0000","last_active":"2024-07-26T07:49:48.086656+0000","last_peered":"2024-07-26T07:49:48.086656+0000","last_clean":"2024-07-26T07:49:48.086656+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.086656+0000","last_undegraded":"2024-07-26T07:49:48.086656+0000","last_fullsized":"2024-07-26T07:49:48.086656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:38:24.318403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.107214+0000","last_change":"2024-07-26T07:47:45.858369+0000","last_active":"2024-07-26T07:49:48.107214+0000","last_peered":"2024-07-26T07:49:48.107214+0000","last_clean":"2024-07-26T07:49:48.107214+0000","last_became_active":"2024-07-26T07:47:45.856685+0000","last_became_peered":"2024-07-26T07:47:45.856685+0000","last_unstale":"2024-07-26T07:49:48.107214+0000","last_undegraded":"2024-07-26T07:49:48.107214+0000","last_fullsized":"2024-07-26T07:49:48.107214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:44.877953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.083667+0000","last_change":"2024-07-26T07:48:20.001394+0000","last_active":"2024-07-26T07:49:48.083667+0000","last_peered":"2024-07-26T07:49:48.083667+0000","last_clean":"2024-07-26T07:49:48.083667+0000","last_became_active":"2024-07-26T07:48:20.000751+0000","last_became_peered":"2024-07-26T07:48:20.000751+0000","last_unstale":"2024-07-26T07:49:48.083667+0000","last_undegraded":"2024-07-26T07:49:48.083667+0000","last_fullsized":"2024-07-26T07:49:48.083667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:26:24.513652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.097228+0000","last_change":"2024-07-26T07:47:45.847931+0000","last_active":"2024-07-26T07:49:48.097228+0000","last_peered":"2024-07-26T07:49:48.097228+0000","last_clean":"2024-07-26T07:49:48.097228+0000","last_became_active":"2024-07-26T07:47:45.847484+0000","last_became_peered":"2024-07-26T07:47:45.847484+0000","last_unstale":"2024-07-26T07:49:48.097228+0000","last_undegraded":"2024-07-26T07:49:48.097228+0000","last_fullsized":"2024-07-26T07:49:48.097228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:04.498360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.087242+0000","last_change":"2024-07-26T07:47:45.846484+0000","last_active":"2024-07-26T07:49:48.087242+0000","last_peered":"2024-07-26T07:49:48.087242+0000","last_clean":"2024-07-26T07:49:48.087242+0000","last_became_active":"2024-07-26T07:47:45.844772+0000","last_became_peered":"2024-07-26T07:47:45.844772+0000","last_unstale":"2024-07-26T07:49:48.087242+0000","last_undegraded":"2024-07-26T07:49:48.087242+0000","last_fullsized":"2024-07-26T07:49:48.087242+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:54.318026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088688+0000","last_change":"2024-07-26T07:47:45.849840+0000","last_active":"2024-07-26T07:49:48.088688+0000","last_peered":"2024-07-26T07:49:48.088688+0000","last_clean":"2024-07-26T07:49:48.088688+0000","last_became_active":"2024-07-26T07:47:45.848744+0000","last_became_peered":"2024-07-26T07:47:45.848744+0000","last_unstale":"2024-07-26T07:49:48.088688+0000","last_undegraded":"2024-07-26T07:49:48.088688+0000","last_fullsized":"2024-07-26T07:49:48.088688+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:44.675382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111006+0000","last_change":"2024-07-26T07:47:45.847176+0000","last_active":"2024-07-26T07:49:48.111006+0000","last_peered":"2024-07-26T07:49:48.111006+0000","last_clean":"2024-07-26T07:49:48.111006+0000","last_became_active":"2024-07-26T07:47:45.845936+0000","last_became_peered":"2024-07-26T07:47:45.845936+0000","last_unstale":"2024-07-26T07:49:48.111006+0000","last_undegraded":"2024-07-26T07:49:48.111006+0000","last_fullsized":"2024-07-26T07:49:48.111006+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:02.245098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090942+0000","last_change":"2024-07-26T07:47:45.853210+0000","last_active":"2024-07-26T07:49:48.090942+0000","last_peered":"2024-07-26T07:49:48.090942+0000","last_clean":"2024-07-26T07:49:48.090942+0000","last_became_active":"2024-07-26T07:47:45.852766+0000","last_became_peered":"2024-07-26T07:47:45.852766+0000","last_unstale":"2024-07-26T07:49:48.090942+0000","last_undegraded":"2024-07-26T07:49:48.090942+0000","last_fullsized":"2024-07-26T07:49:48.090942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:57.406748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.103824+0000","last_change":"2024-07-26T07:47:45.850699+0000","last_active":"2024-07-26T07:49:48.103824+0000","last_peered":"2024-07-26T07:49:48.103824+0000","last_clean":"2024-07-26T07:49:48.103824+0000","last_became_active":"2024-07-26T07:47:45.850314+0000","last_became_peered":"2024-07-26T07:47:45.850314+0000","last_unstale":"2024-07-26T07:49:48.103824+0000","last_undegraded":"2024-07-26T07:49:48.103824+0000","last_fullsized":"2024-07-26T07:49:48.103824+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:09.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090575+0000","last_change":"2024-07-26T07:47:45.851063+0000","last_active":"2024-07-26T07:49:48.090575+0000","last_peered":"2024-07-26T07:49:48.090575+0000","last_clean":"2024-07-26T07:49:48.090575+0000","last_became_active":"2024-07-26T07:47:45.850164+0000","last_became_peered":"2024-07-26T07:47:45.850164+0000","last_unstale":"2024-07-26T07:49:48.090575+0000","last_undegraded":"2024-07-26T07:49:48.090575+0000","last_fullsized":"2024-07-26T07:49:48.090575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:31.597551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088724+0000","last_change":"2024-07-26T07:47:45.861207+0000","last_active":"2024-07-26T07:49:48.088724+0000","last_peered":"2024-07-26T07:49:48.088724+0000","last_clean":"2024-07-26T07:49:48.088724+0000","last_became_active":"2024-07-26T07:47:45.860179+0000","last_became_peered":"2024-07-26T07:47:45.860179+0000","last_unstale":"2024-07-26T07:49:48.088724+0000","last_undegraded":"2024-07-26T07:49:48.088724+0000","last_fullsized":"2024-07-26T07:49:48.088724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:12:51.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101706+0000","last_change":"2024-07-26T07:47:45.846197+0000","last_active":"2024-07-26T07:49:48.101706+0000","last_peered":"2024-07-26T07:49:48.101706+0000","last_clean":"2024-07-26T07:49:48.101706+0000","last_became_active":"2024-07-26T07:47:45.845238+0000","last_became_peered":"2024-07-26T07:47:45.845238+0000","last_unstale":"2024-07-26T07:49:48.101706+0000","last_undegraded":"2024-07-26T07:49:48.101706+0000","last_fullsized":"2024-07-26T07:49:48.101706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:29.900684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112421+0000","last_change":"2024-07-26T07:47:45.847963+0000","last_active":"2024-07-26T07:49:48.112421+0000","last_peered":"2024-07-26T07:49:48.112421+0000","last_clean":"2024-07-26T07:49:48.112421+0000","last_became_active":"2024-07-26T07:47:45.847513+0000","last_became_peered":"2024-07-26T07:47:45.847513+0000","last_unstale":"2024-07-26T07:49:48.112421+0000","last_undegraded":"2024-07-26T07:49:48.112421+0000","last_fullsized":"2024-07-26T07:49:48.112421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:51.082816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092151+0000","last_change":"2024-07-26T07:47:45.853273+0000","last_active":"2024-07-26T07:49:48.092151+0000","last_peered":"2024-07-26T07:49:48.092151+0000","last_clean":"2024-07-26T07:49:48.092151+0000","last_became_active":"2024-07-26T07:47:45.852894+0000","last_became_peered":"2024-07-26T07:47:45.852894+0000","last_unstale":"2024-07-26T07:49:48.092151+0000","last_undegraded":"2024-07-26T07:49:48.092151+0000","last_fullsized":"2024-07-26T07:49:48.092151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:57:12.945228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093387+0000","last_change":"2024-07-26T07:48:19.931938+0000","last_active":"2024-07-26T07:49:48.093387+0000","last_peered":"2024-07-26T07:49:48.093387+0000","last_clean":"2024-07-26T07:49:48.093387+0000","last_became_active":"2024-07-26T07:48:19.930976+0000","last_became_peered":"2024-07-26T07:48:19.930976+0000","last_unstale":"2024-07-26T07:49:48.093387+0000","last_undegraded":"2024-07-26T07:49:48.093387+0000","last_fullsized":"2024-07-26T07:49:48.093387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:59.610464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090184+0000","last_change":"2024-07-26T07:47:45.849736+0000","last_active":"2024-07-26T07:49:48.090184+0000","last_peered":"2024-07-26T07:49:48.090184+0000","last_clean":"2024-07-26T07:49:48.090184+0000","last_became_active":"2024-07-26T07:47:45.849335+0000","last_became_peered":"2024-07-26T07:47:45.849335+0000","last_unstale":"2024-07-26T07:49:48.090184+0000","last_undegraded":"2024-07-26T07:49:48.090184+0000","last_fullsized":"2024-07-26T07:49:48.090184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:20:12.758331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104678+0000","last_change":"2024-07-26T07:47:45.847250+0000","last_active":"2024-07-26T07:49:48.104678+0000","last_peered":"2024-07-26T07:49:48.104678+0000","last_clean":"2024-07-26T07:49:48.104678+0000","last_became_active":"2024-07-26T07:47:45.845564+0000","last_became_peered":"2024-07-26T07:47:45.845564+0000","last_unstale":"2024-07-26T07:49:48.104678+0000","last_undegraded":"2024-07-26T07:49:48.104678+0000","last_fullsized":"2024-07-26T07:49:48.104678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:03.616782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086629+0000","last_change":"2024-07-26T07:47:45.850462+0000","last_active":"2024-07-26T07:49:48.086629+0000","last_peered":"2024-07-26T07:49:48.086629+0000","last_clean":"2024-07-26T07:49:48.086629+0000","last_became_active":"2024-07-26T07:47:45.850036+0000","last_became_peered":"2024-07-26T07:47:45.850036+0000","last_unstale":"2024-07-26T07:49:48.086629+0000","last_undegraded":"2024-07-26T07:49:48.086629+0000","last_fullsized":"2024-07-26T07:49:48.086629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:07:38.374782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.094143+0000","last_change":"2024-07-26T07:47:45.859294+0000","last_active":"2024-07-26T07:49:48.094143+0000","last_peered":"2024-07-26T07:49:48.094143+0000","last_clean":"2024-07-26T07:49:48.094143+0000","last_became_active":"2024-07-26T07:47:45.858812+0000","last_became_peered":"2024-07-26T07:47:45.858812+0000","last_unstale":"2024-07-26T07:49:48.094143+0000","last_undegraded":"2024-07-26T07:49:48.094143+0000","last_fullsized":"2024-07-26T07:49:48.094143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:11.689717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.085416+0000","last_change":"2024-07-26T07:48:19.930943+0000","last_active":"2024-07-26T07:49:48.085416+0000","last_peered":"2024-07-26T07:49:48.085416+0000","last_clean":"2024-07-26T07:49:48.085416+0000","last_became_active":"2024-07-26T07:48:19.929758+0000","last_became_peered":"2024-07-26T07:48:19.929758+0000","last_unstale":"2024-07-26T07:49:48.085416+0000","last_undegraded":"2024-07-26T07:49:48.085416+0000","last_fullsized":"2024-07-26T07:49:48.085416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:29:27.657376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090071+0000","last_change":"2024-07-26T07:47:45.846867+0000","last_active":"2024-07-26T07:49:48.090071+0000","last_peered":"2024-07-26T07:49:48.090071+0000","last_clean":"2024-07-26T07:49:48.090071+0000","last_became_active":"2024-07-26T07:47:45.846455+0000","last_became_peered":"2024-07-26T07:47:45.846455+0000","last_unstale":"2024-07-26T07:49:48.090071+0000","last_undegraded":"2024-07-26T07:49:48.090071+0000","last_fullsized":"2024-07-26T07:49:48.090071+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:14.997452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092573+0000","last_change":"2024-07-26T07:47:45.846179+0000","last_active":"2024-07-26T07:49:48.092573+0000","last_peered":"2024-07-26T07:49:48.092573+0000","last_clean":"2024-07-26T07:49:48.092573+0000","last_became_active":"2024-07-26T07:47:45.844880+0000","last_became_peered":"2024-07-26T07:47:45.844880+0000","last_unstale":"2024-07-26T07:49:48.092573+0000","last_undegraded":"2024-07-26T07:49:48.092573+0000","last_fullsized":"2024-07-26T07:49:48.092573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:00:50.899573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091429+0000","last_change":"2024-07-26T07:48:19.935196+0000","last_active":"2024-07-26T07:49:48.091429+0000","last_peered":"2024-07-26T07:49:48.091429+0000","last_clean":"2024-07-26T07:49:48.091429+0000","last_became_active":"2024-07-26T07:48:19.933730+0000","last_became_peered":"2024-07-26T07:48:19.933730+0000","last_unstale":"2024-07-26T07:49:48.091429+0000","last_undegraded":"2024-07-26T07:49:48.091429+0000","last_fullsized":"2024-07-26T07:49:48.091429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:10.305122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.090542+0000","last_change":"2024-07-26T07:47:45.843360+0000","last_active":"2024-07-26T07:49:48.090542+0000","last_peered":"2024-07-26T07:49:48.090542+0000","last_clean":"2024-07-26T07:49:48.090542+0000","last_became_active":"2024-07-26T07:47:45.842971+0000","last_became_peered":"2024-07-26T07:47:45.842971+0000","last_unstale":"2024-07-26T07:49:48.090542+0000","last_undegraded":"2024-07-26T07:49:48.090542+0000","last_fullsized":"2024-07-26T07:49:48.090542+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:58.687638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:43.556551+0000","last_change":"2024-07-26T07:47:41.787760+0000","last_active":"2024-07-26T07:51:43.556551+0000","last_peered":"2024-07-26T07:51:43.556551+0000","last_clean":"2024-07-26T07:51:43.556551+0000","last_became_active":"2024-07-26T07:47:41.787197+0000","last_became_peered":"2024-07-26T07:47:41.787197+0000","last_unstale":"2024-07-26T07:51:43.556551+0000","last_undegraded":"2024-07-26T07:51:43.556551+0000","last_fullsized":"2024-07-26T07:51:43.556551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:14.597759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092722+0000","last_change":"2024-07-26T07:47:45.846822+0000","last_active":"2024-07-26T07:49:48.092722+0000","last_peered":"2024-07-26T07:49:48.092722+0000","last_clean":"2024-07-26T07:49:48.092722+0000","last_became_active":"2024-07-26T07:47:45.846398+0000","last_became_peered":"2024-07-26T07:47:45.846398+0000","last_unstale":"2024-07-26T07:49:48.092722+0000","last_undegraded":"2024-07-26T07:49:48.092722+0000","last_fullsized":"2024-07-26T07:49:48.092722+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:33:34.727843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"82'7","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.311933+0000","last_change":"2024-07-26T07:47:41.782617+0000","last_active":"2024-07-26T07:49:20.311933+0000","last_peered":"2024-07-26T07:49:20.311933+0000","last_clean":"2024-07-26T07:49:20.311933+0000","last_became_active":"2024-07-26T07:47:41.782077+0000","last_became_peered":"2024-07-26T07:47:41.782077+0000","last_unstale":"2024-07-26T07:49:20.311933+0000","last_undegraded":"2024-07-26T07:49:20.311933+0000","last_fullsized":"2024-07-26T07:49:20.311933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:39:42.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"82'3","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429076+0000","last_change":"2024-07-26T07:47:41.784615+0000","last_active":"2024-07-26T07:49:20.429076+0000","last_peered":"2024-07-26T07:49:20.429076+0000","last_clean":"2024-07-26T07:49:20.429076+0000","last_became_active":"2024-07-26T07:47:41.783945+0000","last_became_peered":"2024-07-26T07:47:41.783945+0000","last_unstale":"2024-07-26T07:49:20.429076+0000","last_undegraded":"2024-07-26T07:49:20.429076+0000","last_fullsized":"2024-07-26T07:49:20.429076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:42.618524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095869+0000","last_change":"2024-07-26T07:47:45.838774+0000","last_active":"2024-07-26T07:49:48.095869+0000","last_peered":"2024-07-26T07:49:48.095869+0000","last_clean":"2024-07-26T07:49:48.095869+0000","last_became_active":"2024-07-26T07:47:45.837707+0000","last_became_peered":"2024-07-26T07:47:45.837707+0000","last_unstale":"2024-07-26T07:49:48.095869+0000","last_undegraded":"2024-07-26T07:49:48.095869+0000","last_fullsized":"2024-07-26T07:49:48.095869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:59.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.297619+0000","last_change":"2024-07-26T07:47:41.792031+0000","last_active":"2024-07-26T07:49:20.297619+0000","last_peered":"2024-07-26T07:49:20.297619+0000","last_clean":"2024-07-26T07:49:20.297619+0000","last_became_active":"2024-07-26T07:47:41.785955+0000","last_became_peered":"2024-07-26T07:47:41.785955+0000","last_unstale":"2024-07-26T07:49:20.297619+0000","last_undegraded":"2024-07-26T07:49:20.297619+0000","last_fullsized":"2024-07-26T07:49:20.297619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:01:47.073970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.094145+0000","last_change":"2024-07-26T07:47:45.844595+0000","last_active":"2024-07-26T07:49:48.094145+0000","last_peered":"2024-07-26T07:49:48.094145+0000","last_clean":"2024-07-26T07:49:48.094145+0000","last_became_active":"2024-07-26T07:47:45.843381+0000","last_became_peered":"2024-07-26T07:47:45.843381+0000","last_unstale":"2024-07-26T07:49:48.094145+0000","last_undegraded":"2024-07-26T07:49:48.094145+0000","last_fullsized":"2024-07-26T07:49:48.094145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:39.692463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'7","reported_seq":477,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:46.563618+0000","last_change":"2024-07-26T07:47:41.788239+0000","last_active":"2024-07-26T07:51:46.563618+0000","last_peered":"2024-07-26T07:51:46.563618+0000","last_clean":"2024-07-26T07:51:46.563618+0000","last_became_active":"2024-07-26T07:47:41.787470+0000","last_became_peered":"2024-07-26T07:47:41.787470+0000","last_unstale":"2024-07-26T07:51:46.563618+0000","last_undegraded":"2024-07-26T07:51:46.563618+0000","last_fullsized":"2024-07-26T07:51:46.563618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:03.129563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":438,"num_read_kb":438,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.112494+0000","last_change":"2024-07-26T07:47:45.852794+0000","last_active":"2024-07-26T07:49:48.112494+0000","last_peered":"2024-07-26T07:49:48.112494+0000","last_clean":"2024-07-26T07:49:48.112494+0000","last_became_active":"2024-07-26T07:47:45.849699+0000","last_became_peered":"2024-07-26T07:47:45.849699+0000","last_unstale":"2024-07-26T07:49:48.112494+0000","last_undegraded":"2024-07-26T07:49:48.112494+0000","last_fullsized":"2024-07-26T07:49:48.112494+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:06:31.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'4","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.315536+0000","last_change":"2024-07-26T07:47:41.789414+0000","last_active":"2024-07-26T07:49:20.315536+0000","last_peered":"2024-07-26T07:49:20.315536+0000","last_clean":"2024-07-26T07:49:20.315536+0000","last_became_active":"2024-07-26T07:47:41.788679+0000","last_became_peered":"2024-07-26T07:47:41.788679+0000","last_unstale":"2024-07-26T07:49:20.315536+0000","last_undegraded":"2024-07-26T07:49:20.315536+0000","last_fullsized":"2024-07-26T07:49:20.315536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:51.809997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099375+0000","last_change":"2024-07-26T07:48:19.937603+0000","last_active":"2024-07-26T07:49:48.099375+0000","last_peered":"2024-07-26T07:49:48.099375+0000","last_clean":"2024-07-26T07:49:48.099375+0000","last_became_active":"2024-07-26T07:48:19.936889+0000","last_became_peered":"2024-07-26T07:48:19.936889+0000","last_unstale":"2024-07-26T07:49:48.099375+0000","last_undegraded":"2024-07-26T07:49:48.099375+0000","last_fullsized":"2024-07-26T07:49:48.099375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:46:35.138825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298028+0000","last_change":"2024-07-26T07:47:42.564288+0000","last_active":"2024-07-26T07:49:20.298028+0000","last_peered":"2024-07-26T07:49:20.298028+0000","last_clean":"2024-07-26T07:49:20.298028+0000","last_became_active":"2024-07-26T07:47:42.561766+0000","last_became_peered":"2024-07-26T07:47:42.561766+0000","last_unstale":"2024-07-26T07:49:20.298028+0000","last_undegraded":"2024-07-26T07:49:20.298028+0000","last_fullsized":"2024-07-26T07:49:20.298028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:16.414030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.113511+0000","last_change":"2024-07-26T07:47:45.861895+0000","last_active":"2024-07-26T07:49:48.113511+0000","last_peered":"2024-07-26T07:49:48.113511+0000","last_clean":"2024-07-26T07:49:48.113511+0000","last_became_active":"2024-07-26T07:47:45.861439+0000","last_became_peered":"2024-07-26T07:47:45.861439+0000","last_unstale":"2024-07-26T07:49:48.113511+0000","last_undegraded":"2024-07-26T07:49:48.113511+0000","last_fullsized":"2024-07-26T07:49:48.113511+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:26.276453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'38","reported_seq":521,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:46.566082+0000","last_change":"2024-07-26T07:47:41.789977+0000","last_active":"2024-07-26T07:51:46.566082+0000","last_peered":"2024-07-26T07:51:46.566082+0000","last_clean":"2024-07-26T07:51:46.566082+0000","last_became_active":"2024-07-26T07:47:41.789488+0000","last_became_peered":"2024-07-26T07:47:41.789488+0000","last_unstale":"2024-07-26T07:51:46.566082+0000","last_undegraded":"2024-07-26T07:51:46.566082+0000","last_fullsized":"2024-07-26T07:51:46.566082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:33.059279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67729,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":451,"num_read_kb":291,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098321+0000","last_change":"2024-07-26T07:47:45.854772+0000","last_active":"2024-07-26T07:49:48.098321+0000","last_peered":"2024-07-26T07:49:48.098321+0000","last_clean":"2024-07-26T07:49:48.098321+0000","last_became_active":"2024-07-26T07:47:45.854354+0000","last_became_peered":"2024-07-26T07:47:45.854354+0000","last_unstale":"2024-07-26T07:49:48.098321+0000","last_undegraded":"2024-07-26T07:49:48.098321+0000","last_fullsized":"2024-07-26T07:49:48.098321+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:31:54.856407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'9","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313468+0000","last_change":"2024-07-26T07:47:41.789752+0000","last_active":"2024-07-26T07:49:20.313468+0000","last_peered":"2024-07-26T07:49:20.313468+0000","last_clean":"2024-07-26T07:49:20.313468+0000","last_became_active":"2024-07-26T07:47:41.788952+0000","last_became_peered":"2024-07-26T07:47:41.788952+0000","last_unstale":"2024-07-26T07:49:20.313468+0000","last_undegraded":"2024-07-26T07:49:20.313468+0000","last_fullsized":"2024-07-26T07:49:20.313468+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:45.141693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.101283+0000","last_change":"2024-07-26T07:47:45.845120+0000","last_active":"2024-07-26T07:49:48.101283+0000","last_peered":"2024-07-26T07:49:48.101283+0000","last_clean":"2024-07-26T07:49:48.101283+0000","last_became_active":"2024-07-26T07:47:45.844699+0000","last_became_peered":"2024-07-26T07:47:45.844699+0000","last_unstale":"2024-07-26T07:49:48.101283+0000","last_undegraded":"2024-07-26T07:49:48.101283+0000","last_fullsized":"2024-07-26T07:49:48.101283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:23:42.130846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298579+0000","last_change":"2024-07-26T07:47:41.790765+0000","last_active":"2024-07-26T07:49:20.298579+0000","last_peered":"2024-07-26T07:49:20.298579+0000","last_clean":"2024-07-26T07:49:20.298579+0000","last_became_active":"2024-07-26T07:47:41.790288+0000","last_became_peered":"2024-07-26T07:47:41.790288+0000","last_unstale":"2024-07-26T07:49:20.298579+0000","last_undegraded":"2024-07-26T07:49:20.298579+0000","last_fullsized":"2024-07-26T07:49:20.298579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:05.734468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.102627+0000","last_change":"2024-07-26T07:47:45.854648+0000","last_active":"2024-07-26T07:49:48.102627+0000","last_peered":"2024-07-26T07:49:48.102627+0000","last_clean":"2024-07-26T07:49:48.102627+0000","last_became_active":"2024-07-26T07:47:45.854240+0000","last_became_peered":"2024-07-26T07:47:45.854240+0000","last_unstale":"2024-07-26T07:49:48.102627+0000","last_undegraded":"2024-07-26T07:49:48.102627+0000","last_fullsized":"2024-07-26T07:49:48.102627+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:41.775760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'7","reported_seq":280,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:48.806013+0000","last_change":"2024-07-26T07:49:20.331153+0000","last_active":"2024-07-26T07:51:48.806013+0000","last_peered":"2024-07-26T07:51:48.806013+0000","last_clean":"2024-07-26T07:51:48.806013+0000","last_became_active":"2024-07-26T07:49:20.315423+0000","last_became_peered":"2024-07-26T07:49:20.315423+0000","last_unstale":"2024-07-26T07:51:48.806013+0000","last_undegraded":"2024-07-26T07:51:48.806013+0000","last_fullsized":"2024-07-26T07:51:48.806013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:24.476818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1472,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":225,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.100845+0000","last_change":"2024-07-26T07:47:45.848731+0000","last_active":"2024-07-26T07:49:48.100845+0000","last_peered":"2024-07-26T07:49:48.100845+0000","last_clean":"2024-07-26T07:49:48.100845+0000","last_became_active":"2024-07-26T07:47:45.846663+0000","last_became_peered":"2024-07-26T07:47:45.846663+0000","last_unstale":"2024-07-26T07:49:48.100845+0000","last_undegraded":"2024-07-26T07:49:48.100845+0000","last_fullsized":"2024-07-26T07:49:48.100845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:05.251423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'17","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:23.742120+0000","last_change":"2024-07-26T07:49:20.355238+0000","last_active":"2024-07-26T07:49:23.742120+0000","last_peered":"2024-07-26T07:49:23.742120+0000","last_clean":"2024-07-26T07:49:23.742120+0000","last_became_active":"2024-07-26T07:49:20.316591+0000","last_became_peered":"2024-07-26T07:49:20.316591+0000","last_unstale":"2024-07-26T07:49:23.742120+0000","last_undegraded":"2024-07-26T07:49:23.742120+0000","last_fullsized":"2024-07-26T07:49:23.742120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:12.739473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.088622+0000","last_change":"2024-07-26T07:47:45.842572+0000","last_active":"2024-07-26T07:49:48.088622+0000","last_peered":"2024-07-26T07:49:48.088622+0000","last_clean":"2024-07-26T07:49:48.088622+0000","last_became_active":"2024-07-26T07:47:45.842116+0000","last_became_peered":"2024-07-26T07:47:45.842116+0000","last_unstale":"2024-07-26T07:49:48.088622+0000","last_undegraded":"2024-07-26T07:49:48.088622+0000","last_fullsized":"2024-07-26T07:49:48.088622+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:06.913737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:59.385245+0000","last_change":"2024-07-26T07:47:41.790215+0000","last_active":"2024-07-26T07:49:59.385245+0000","last_peered":"2024-07-26T07:49:59.385245+0000","last_clean":"2024-07-26T07:49:59.385245+0000","last_became_active":"2024-07-26T07:47:41.789691+0000","last_became_peered":"2024-07-26T07:47:41.789691+0000","last_unstale":"2024-07-26T07:49:59.385245+0000","last_undegraded":"2024-07-26T07:49:59.385245+0000","last_fullsized":"2024-07-26T07:49:59.385245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:01:47.002764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.086806+0000","last_change":"2024-07-26T07:47:45.860887+0000","last_active":"2024-07-26T07:49:48.086806+0000","last_peered":"2024-07-26T07:49:48.086806+0000","last_clean":"2024-07-26T07:49:48.086806+0000","last_became_active":"2024-07-26T07:47:45.860315+0000","last_became_peered":"2024-07-26T07:47:45.860315+0000","last_unstale":"2024-07-26T07:49:48.086806+0000","last_undegraded":"2024-07-26T07:49:48.086806+0000","last_fullsized":"2024-07-26T07:49:48.086806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:17:02.231206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"82'5","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.429700+0000","last_change":"2024-07-26T07:48:20.022885+0000","last_active":"2024-07-26T07:49:20.429700+0000","last_peered":"2024-07-26T07:49:20.429700+0000","last_clean":"2024-07-26T07:49:20.429700+0000","last_became_active":"2024-07-26T07:48:20.001946+0000","last_became_peered":"2024-07-26T07:48:20.001946+0000","last_unstale":"2024-07-26T07:49:20.429700+0000","last_undegraded":"2024-07-26T07:49:20.429700+0000","last_fullsized":"2024-07-26T07:49:20.429700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:45.892886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.098632+0000","last_change":"2024-07-26T07:47:45.858267+0000","last_active":"2024-07-26T07:49:48.098632+0000","last_peered":"2024-07-26T07:49:48.098632+0000","last_clean":"2024-07-26T07:49:48.098632+0000","last_became_active":"2024-07-26T07:47:45.854736+0000","last_became_peered":"2024-07-26T07:47:45.854736+0000","last_unstale":"2024-07-26T07:49:48.098632+0000","last_undegraded":"2024-07-26T07:49:48.098632+0000","last_fullsized":"2024-07-26T07:49:48.098632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:07:32.055977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'3","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.298413+0000","last_change":"2024-07-26T07:47:41.789846+0000","last_active":"2024-07-26T07:49:20.298413+0000","last_peered":"2024-07-26T07:49:20.298413+0000","last_clean":"2024-07-26T07:49:20.298413+0000","last_became_active":"2024-07-26T07:47:41.789237+0000","last_became_peered":"2024-07-26T07:47:41.789237+0000","last_unstale":"2024-07-26T07:49:20.298413+0000","last_undegraded":"2024-07-26T07:49:20.298413+0000","last_fullsized":"2024-07-26T07:49:20.298413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:39.031047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.095798+0000","last_change":"2024-07-26T07:47:45.846294+0000","last_active":"2024-07-26T07:49:48.095798+0000","last_peered":"2024-07-26T07:49:48.095798+0000","last_clean":"2024-07-26T07:49:48.095798+0000","last_became_active":"2024-07-26T07:47:45.845861+0000","last_became_peered":"2024-07-26T07:47:45.845861+0000","last_unstale":"2024-07-26T07:49:48.095798+0000","last_undegraded":"2024-07-26T07:49:48.095798+0000","last_fullsized":"2024-07-26T07:49:48.095798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:56:10.594430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'9","reported_seq":265,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:44.558871+0000","last_change":"2024-07-26T07:47:42.563412+0000","last_active":"2024-07-26T07:51:44.558871+0000","last_peered":"2024-07-26T07:51:44.558871+0000","last_clean":"2024-07-26T07:51:44.558871+0000","last_became_active":"2024-07-26T07:47:42.562364+0000","last_became_peered":"2024-07-26T07:47:42.562364+0000","last_unstale":"2024-07-26T07:51:44.558871+0000","last_undegraded":"2024-07-26T07:51:44.558871+0000","last_fullsized":"2024-07-26T07:51:44.558871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:14.796738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2586,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":224,"num_read_kb":6,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.108677+0000","last_change":"2024-07-26T07:47:45.856170+0000","last_active":"2024-07-26T07:49:48.108677+0000","last_peered":"2024-07-26T07:49:48.108677+0000","last_clean":"2024-07-26T07:49:48.108677+0000","last_became_active":"2024-07-26T07:47:45.855782+0000","last_became_peered":"2024-07-26T07:47:45.855782+0000","last_unstale":"2024-07-26T07:49:48.108677+0000","last_undegraded":"2024-07-26T07:49:48.108677+0000","last_fullsized":"2024-07-26T07:49:48.108677+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:22.690910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.313258+0000","last_change":"2024-07-26T07:45:54.310557+0000","last_active":"2024-07-26T07:49:20.313258+0000","last_peered":"2024-07-26T07:49:20.313258+0000","last_clean":"2024-07-26T07:49:20.313258+0000","last_became_active":"2024-07-26T07:45:54.290906+0000","last_became_peered":"2024-07-26T07:45:54.290906+0000","last_unstale":"2024-07-26T07:49:20.313258+0000","last_undegraded":"2024-07-26T07:49:20.313258+0000","last_fullsized":"2024-07-26T07:49:20.313258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:41:16.689463+0000","last_clean_scrub_stamp":"2024-07-26T07:41:16.689463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:51.196560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"82'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.305857+0000","last_change":"2024-07-26T07:47:41.776420+0000","last_active":"2024-07-26T07:49:20.305857+0000","last_peered":"2024-07-26T07:49:20.305857+0000","last_clean":"2024-07-26T07:49:20.305857+0000","last_became_active":"2024-07-26T07:47:41.775858+0000","last_became_peered":"2024-07-26T07:47:41.775858+0000","last_unstale":"2024-07-26T07:49:20.305857+0000","last_undegraded":"2024-07-26T07:49:20.305857+0000","last_fullsized":"2024-07-26T07:49:20.305857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:18.880331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.109733+0000","last_change":"2024-07-26T07:48:19.933318+0000","last_active":"2024-07-26T07:49:48.109733+0000","last_peered":"2024-07-26T07:49:48.109733+0000","last_clean":"2024-07-26T07:49:48.109733+0000","last_became_active":"2024-07-26T07:48:19.932683+0000","last_became_peered":"2024-07-26T07:48:19.932683+0000","last_unstale":"2024-07-26T07:49:48.109733+0000","last_undegraded":"2024-07-26T07:49:48.109733+0000","last_fullsized":"2024-07-26T07:49:48.109733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:25:23.387428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"82'4","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:43.375132+0000","last_change":"2024-07-26T07:47:41.786698+0000","last_active":"2024-07-26T07:49:43.375132+0000","last_peered":"2024-07-26T07:49:43.375132+0000","last_clean":"2024-07-26T07:49:43.375132+0000","last_became_active":"2024-07-26T07:47:41.784491+0000","last_became_peered":"2024-07-26T07:47:41.784491+0000","last_unstale":"2024-07-26T07:49:43.375132+0000","last_undegraded":"2024-07-26T07:49:43.375132+0000","last_fullsized":"2024-07-26T07:49:43.375132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:09:08.470956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.093849+0000","last_change":"2024-07-26T07:47:45.849698+0000","last_active":"2024-07-26T07:49:48.093849+0000","last_peered":"2024-07-26T07:49:48.093849+0000","last_clean":"2024-07-26T07:49:48.093849+0000","last_became_active":"2024-07-26T07:47:45.849266+0000","last_became_peered":"2024-07-26T07:47:45.849266+0000","last_unstale":"2024-07-26T07:49:48.093849+0000","last_undegraded":"2024-07-26T07:49:48.093849+0000","last_fullsized":"2024-07-26T07:49:48.093849+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:57.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":260,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:47.801073+0000","last_change":"2024-07-26T07:47:42.558680+0000","last_active":"2024-07-26T07:51:47.801073+0000","last_peered":"2024-07-26T07:51:47.801073+0000","last_clean":"2024-07-26T07:51:47.801073+0000","last_became_active":"2024-07-26T07:47:42.558197+0000","last_became_peered":"2024-07-26T07:47:42.558197+0000","last_unstale":"2024-07-26T07:51:47.801073+0000","last_undegraded":"2024-07-26T07:51:47.801073+0000","last_fullsized":"2024-07-26T07:51:47.801073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:52:52.505455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":222,"num_read_kb":222,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.078802+0000","last_change":"2024-07-26T07:48:19.929897+0000","last_active":"2024-07-26T07:49:48.078802+0000","last_peered":"2024-07-26T07:49:48.078802+0000","last_clean":"2024-07-26T07:49:48.078802+0000","last_became_active":"2024-07-26T07:48:19.927946+0000","last_became_peered":"2024-07-26T07:48:19.927946+0000","last_unstale":"2024-07-26T07:49:48.078802+0000","last_undegraded":"2024-07-26T07:49:48.078802+0000","last_fullsized":"2024-07-26T07:49:48.078802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:11.157824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'6","reported_seq":260,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:46.544368+0000","last_change":"2024-07-26T07:47:41.785513+0000","last_active":"2024-07-26T07:51:46.544368+0000","last_peered":"2024-07-26T07:51:46.544368+0000","last_clean":"2024-07-26T07:51:46.544368+0000","last_became_active":"2024-07-26T07:47:41.784783+0000","last_became_peered":"2024-07-26T07:47:41.784783+0000","last_unstale":"2024-07-26T07:51:46.544368+0000","last_undegraded":"2024-07-26T07:51:46.544368+0000","last_fullsized":"2024-07-26T07:51:46.544368+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:54.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":222,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.082468+0000","last_change":"2024-07-26T07:47:45.856580+0000","last_active":"2024-07-26T07:49:48.082468+0000","last_peered":"2024-07-26T07:49:48.082468+0000","last_clean":"2024-07-26T07:49:48.082468+0000","last_became_active":"2024-07-26T07:47:45.852330+0000","last_became_peered":"2024-07-26T07:47:45.852330+0000","last_unstale":"2024-07-26T07:49:48.082468+0000","last_undegraded":"2024-07-26T07:49:48.082468+0000","last_fullsized":"2024-07-26T07:49:48.082468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:09.479563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"82'1","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.428822+0000","last_change":"2024-07-26T07:47:42.559255+0000","last_active":"2024-07-26T07:49:20.428822+0000","last_peered":"2024-07-26T07:49:20.428822+0000","last_clean":"2024-07-26T07:49:20.428822+0000","last_became_active":"2024-07-26T07:47:42.558123+0000","last_became_peered":"2024-07-26T07:47:42.558123+0000","last_unstale":"2024-07-26T07:49:20.428822+0000","last_undegraded":"2024-07-26T07:49:20.428822+0000","last_fullsized":"2024-07-26T07:49:20.428822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:49:46.576674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092093+0000","last_change":"2024-07-26T07:47:45.856256+0000","last_active":"2024-07-26T07:49:48.092093+0000","last_peered":"2024-07-26T07:49:48.092093+0000","last_clean":"2024-07-26T07:49:48.092093+0000","last_became_active":"2024-07-26T07:47:45.855696+0000","last_became_peered":"2024-07-26T07:47:45.855696+0000","last_unstale":"2024-07-26T07:49:48.092093+0000","last_undegraded":"2024-07-26T07:49:48.092093+0000","last_fullsized":"2024-07-26T07:49:48.092093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:24.451225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"82'4","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314969+0000","last_change":"2024-07-26T07:47:42.562631+0000","last_active":"2024-07-26T07:49:20.314969+0000","last_peered":"2024-07-26T07:49:20.314969+0000","last_clean":"2024-07-26T07:49:20.314969+0000","last_became_active":"2024-07-26T07:47:42.561379+0000","last_became_peered":"2024-07-26T07:47:42.561379+0000","last_unstale":"2024-07-26T07:49:20.314969+0000","last_undegraded":"2024-07-26T07:49:20.314969+0000","last_fullsized":"2024-07-26T07:49:20.314969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:03:30.471694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092440+0000","last_change":"2024-07-26T07:47:45.849580+0000","last_active":"2024-07-26T07:49:48.092440+0000","last_peered":"2024-07-26T07:49:48.092440+0000","last_clean":"2024-07-26T07:49:48.092440+0000","last_became_active":"2024-07-26T07:47:45.849107+0000","last_became_peered":"2024-07-26T07:47:45.849107+0000","last_unstale":"2024-07-26T07:49:48.092440+0000","last_undegraded":"2024-07-26T07:49:48.092440+0000","last_fullsized":"2024-07-26T07:49:48.092440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:57:13.694059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'18","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:38.899550+0000","last_change":"2024-07-26T07:47:41.774083+0000","last_active":"2024-07-26T07:51:38.899550+0000","last_peered":"2024-07-26T07:51:38.899550+0000","last_clean":"2024-07-26T07:51:38.899550+0000","last_became_active":"2024-07-26T07:47:41.773502+0000","last_became_peered":"2024-07-26T07:47:41.773502+0000","last_unstale":"2024-07-26T07:51:38.899550+0000","last_undegraded":"2024-07-26T07:51:38.899550+0000","last_fullsized":"2024-07-26T07:51:38.899550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:11:44.083944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.099359+0000","last_change":"2024-07-26T07:47:45.845894+0000","last_active":"2024-07-26T07:49:48.099359+0000","last_peered":"2024-07-26T07:49:48.099359+0000","last_clean":"2024-07-26T07:49:48.099359+0000","last_became_active":"2024-07-26T07:47:45.845534+0000","last_became_peered":"2024-07-26T07:47:45.845534+0000","last_unstale":"2024-07-26T07:49:48.099359+0000","last_undegraded":"2024-07-26T07:49:48.099359+0000","last_fullsized":"2024-07-26T07:49:48.099359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:52:42.209561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'7","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.306134+0000","last_change":"2024-07-26T07:47:41.904220+0000","last_active":"2024-07-26T07:49:20.306134+0000","last_peered":"2024-07-26T07:49:20.306134+0000","last_clean":"2024-07-26T07:49:20.306134+0000","last_became_active":"2024-07-26T07:47:41.902349+0000","last_became_peered":"2024-07-26T07:47:41.902349+0000","last_unstale":"2024-07-26T07:49:20.306134+0000","last_undegraded":"2024-07-26T07:49:20.306134+0000","last_fullsized":"2024-07-26T07:49:20.306134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:08.520461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.092199+0000","last_change":"2024-07-26T07:47:45.846920+0000","last_active":"2024-07-26T07:49:48.092199+0000","last_peered":"2024-07-26T07:49:48.092199+0000","last_clean":"2024-07-26T07:49:48.092199+0000","last_became_active":"2024-07-26T07:47:45.846458+0000","last_became_peered":"2024-07-26T07:47:45.846458+0000","last_unstale":"2024-07-26T07:49:48.092199+0000","last_undegraded":"2024-07-26T07:49:48.092199+0000","last_fullsized":"2024-07-26T07:49:48.092199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:06.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.295007+0000","last_change":"2024-07-26T07:47:42.560346+0000","last_active":"2024-07-26T07:49:20.295007+0000","last_peered":"2024-07-26T07:49:20.295007+0000","last_clean":"2024-07-26T07:49:20.295007+0000","last_became_active":"2024-07-26T07:47:42.558937+0000","last_became_peered":"2024-07-26T07:47:42.558937+0000","last_unstale":"2024-07-26T07:49:20.295007+0000","last_undegraded":"2024-07-26T07:49:20.295007+0000","last_fullsized":"2024-07-26T07:49:20.295007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.676625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.081730+0000","last_change":"2024-07-26T07:47:45.845553+0000","last_active":"2024-07-26T07:49:48.081730+0000","last_peered":"2024-07-26T07:49:48.081730+0000","last_clean":"2024-07-26T07:49:48.081730+0000","last_became_active":"2024-07-26T07:47:45.845125+0000","last_became_peered":"2024-07-26T07:47:45.845125+0000","last_unstale":"2024-07-26T07:49:48.081730+0000","last_undegraded":"2024-07-26T07:49:48.081730+0000","last_fullsized":"2024-07-26T07:49:48.081730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:10.896159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'7","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.292557+0000","last_change":"2024-07-26T07:47:41.779688+0000","last_active":"2024-07-26T07:49:20.292557+0000","last_peered":"2024-07-26T07:49:20.292557+0000","last_clean":"2024-07-26T07:49:20.292557+0000","last_became_active":"2024-07-26T07:47:41.779115+0000","last_became_peered":"2024-07-26T07:47:41.779115+0000","last_unstale":"2024-07-26T07:49:20.292557+0000","last_undegraded":"2024-07-26T07:49:20.292557+0000","last_fullsized":"2024-07-26T07:49:20.292557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:20.708336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.091558+0000","last_change":"2024-07-26T07:47:45.853869+0000","last_active":"2024-07-26T07:49:48.091558+0000","last_peered":"2024-07-26T07:49:48.091558+0000","last_clean":"2024-07-26T07:49:48.091558+0000","last_became_active":"2024-07-26T07:47:45.852509+0000","last_became_peered":"2024-07-26T07:47:45.852509+0000","last_unstale":"2024-07-26T07:49:48.091558+0000","last_undegraded":"2024-07-26T07:49:48.091558+0000","last_fullsized":"2024-07-26T07:49:48.091558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:01:31.764025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'5","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:38.797885+0000","last_change":"2024-07-26T07:47:41.792331+0000","last_active":"2024-07-26T07:51:38.797885+0000","last_peered":"2024-07-26T07:51:38.797885+0000","last_clean":"2024-07-26T07:51:38.797885+0000","last_became_active":"2024-07-26T07:47:41.790668+0000","last_became_peered":"2024-07-26T07:47:41.790668+0000","last_unstale":"2024-07-26T07:51:38.797885+0000","last_undegraded":"2024-07-26T07:51:38.797885+0000","last_fullsized":"2024-07-26T07:51:38.797885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:59.430692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096588+0000","last_change":"2024-07-26T07:47:45.843583+0000","last_active":"2024-07-26T07:49:48.096588+0000","last_peered":"2024-07-26T07:49:48.096588+0000","last_clean":"2024-07-26T07:49:48.096588+0000","last_became_active":"2024-07-26T07:47:45.843185+0000","last_became_peered":"2024-07-26T07:47:45.843185+0000","last_unstale":"2024-07-26T07:49:48.096588+0000","last_undegraded":"2024-07-26T07:49:48.096588+0000","last_fullsized":"2024-07-26T07:49:48.096588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:01.119065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314240+0000","last_change":"2024-07-26T07:48:20.014411+0000","last_active":"2024-07-26T07:49:20.314240+0000","last_peered":"2024-07-26T07:49:20.314240+0000","last_clean":"2024-07-26T07:49:20.314240+0000","last_became_active":"2024-07-26T07:48:20.002085+0000","last_became_peered":"2024-07-26T07:48:20.002085+0000","last_unstale":"2024-07-26T07:49:20.314240+0000","last_undegraded":"2024-07-26T07:49:20.314240+0000","last_fullsized":"2024-07-26T07:49:20.314240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:46:59.335128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.110858+0000","last_change":"2024-07-26T07:47:45.851963+0000","last_active":"2024-07-26T07:49:48.110858+0000","last_peered":"2024-07-26T07:49:48.110858+0000","last_clean":"2024-07-26T07:49:48.110858+0000","last_became_active":"2024-07-26T07:47:45.850459+0000","last_became_peered":"2024-07-26T07:47:45.850459+0000","last_unstale":"2024-07-26T07:49:48.110858+0000","last_undegraded":"2024-07-26T07:49:48.110858+0000","last_fullsized":"2024-07-26T07:49:48.110858+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:18.092186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.096201+0000","last_change":"2024-07-26T07:47:45.848744+0000","last_active":"2024-07-26T07:49:48.096201+0000","last_peered":"2024-07-26T07:49:48.096201+0000","last_clean":"2024-07-26T07:49:48.096201+0000","last_became_active":"2024-07-26T07:47:45.848292+0000","last_became_peered":"2024-07-26T07:47:45.848292+0000","last_unstale":"2024-07-26T07:49:48.096201+0000","last_undegraded":"2024-07-26T07:49:48.096201+0000","last_fullsized":"2024-07-26T07:49:48.096201+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:54.549508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"82'5","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314854+0000","last_change":"2024-07-26T07:47:41.788875+0000","last_active":"2024-07-26T07:49:20.314854+0000","last_peered":"2024-07-26T07:49:20.314854+0000","last_clean":"2024-07-26T07:49:20.314854+0000","last_became_active":"2024-07-26T07:47:41.786942+0000","last_became_peered":"2024-07-26T07:47:41.786942+0000","last_unstale":"2024-07-26T07:49:20.314854+0000","last_undegraded":"2024-07-26T07:49:20.314854+0000","last_fullsized":"2024-07-26T07:49:20.314854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:11:31.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.111571+0000","last_change":"2024-07-26T07:47:45.854832+0000","last_active":"2024-07-26T07:49:48.111571+0000","last_peered":"2024-07-26T07:49:48.111571+0000","last_clean":"2024-07-26T07:49:48.111571+0000","last_became_active":"2024-07-26T07:47:45.853756+0000","last_became_peered":"2024-07-26T07:47:45.853756+0000","last_unstale":"2024-07-26T07:49:48.111571+0000","last_undegraded":"2024-07-26T07:49:48.111571+0000","last_fullsized":"2024-07-26T07:49:48.111571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:42:14.579177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"82'12","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:20.314889+0000","last_change":"2024-07-26T07:47:41.787839+0000","last_active":"2024-07-26T07:49:20.314889+0000","last_peered":"2024-07-26T07:49:20.314889+0000","last_clean":"2024-07-26T07:49:20.314889+0000","last_became_active":"2024-07-26T07:47:41.785951+0000","last_became_peered":"2024-07-26T07:47:41.785951+0000","last_unstale":"2024-07-26T07:49:20.314889+0000","last_undegraded":"2024-07-26T07:49:20.314889+0000","last_fullsized":"2024-07-26T07:49:20.314889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:58.699056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:48.104028+0000","last_change":"2024-07-26T07:47:45.841451+0000","last_active":"2024-07-26T07:49:48.104028+0000","last_peered":"2024-07-26T07:49:48.104028+0000","last_clean":"2024-07-26T07:49:48.104028+0000","last_became_active":"2024-07-26T07:47:45.840803+0000","last_became_peered":"2024-07-26T07:47:45.840803+0000","last_unstale":"2024-07-26T07:49:48.104028+0000","last_undegraded":"2024-07-26T07:49:48.104028+0000","last_fullsized":"2024-07-26T07:49:48.104028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:24:21.832317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:44.189574+0000","last_change":"2024-07-26T07:47:42.558775+0000","last_active":"2024-07-26T07:49:44.189574+0000","last_peered":"2024-07-26T07:49:44.189574+0000","last_clean":"2024-07-26T07:49:44.189574+0000","last_became_active":"2024-07-26T07:47:42.558300+0000","last_became_peered":"2024-07-26T07:47:42.558300+0000","last_unstale":"2024-07-26T07:49:44.189574+0000","last_undegraded":"2024-07-26T07:49:44.189574+0000","last_fullsized":"2024-07-26T07:49:44.189574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:59.643709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:49:49.190570+0000","last_change":"2024-07-26T07:47:45.852401+0000","last_active":"2024-07-26T07:49:49.190570+0000","last_peered":"2024-07-26T07:49:49.190570+0000","last_clean":"2024-07-26T07:49:49.190570+0000","last_became_active":"2024-07-26T07:47:45.852054+0000","last_became_peered":"2024-07-26T07:47:45.852054+0000","last_unstale":"2024-07-26T07:49:49.190570+0000","last_undegraded":"2024-07-26T07:49:49.190570+0000","last_fullsized":"2024-07-26T07:49:49.190570+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:44.773439+0000","last_clean_scrub_stamp":"2024-07-26T07:47:44.773439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:25.347770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"86'14","reported_seq":289,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-07-26T07:51:47.546372+0000","last_change":"2024-07-26T07:48:19.961760+0000","last_active":"2024-07-26T07:51:47.546372+0000","last_peered":"2024-07-26T07:51:47.546372+0000","last_clean":"2024-07-26T07:51:47.546372+0000","last_became_active":"2024-07-26T07:48:19.937694+0000","last_became_peered":"2024-07-26T07:48:19.937694+0000","last_unstale":"2024-07-26T07:51:47.546372+0000","last_undegraded":"2024-07-26T07:51:47.546372+0000","last_fullsized":"2024-07-26T07:51:47.546372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T07:47:40.719835+0000","last_clean_scrub_stamp":"2024-07-26T07:47:40.719835+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:05:10.571086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":216,"num_read_kb":216,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77437,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2237,"num_read_kb":1202,"num_write":275,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1372,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":232311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579973,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":466032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776203,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1667072,"data_stored":1122674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972434,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":462962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168666,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":462556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397599,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1055865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561126,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":463172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757358,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1589248,"data_stored":1055638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953589,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":463424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":462504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411457,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":533450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":532486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738513,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":465434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":70646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":70349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T07:51:52.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:51:52.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:52 smithi115 ceph-mon[35364]: pgmap v512: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-07-26T07:51:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:52 smithi181 ceph-mon[35869]: pgmap v512: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-07-26T07:51:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:53.376 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:53 smithi097 ceph-mon[32231]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:53 smithi115 ceph-mon[35364]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:53 smithi181 ceph-mon[35869]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T07:51:54.402 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:54 smithi097 ceph-mon[32231]: pgmap v513: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:51:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:54 smithi115 ceph-mon[35364]: pgmap v513: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:51:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:54 smithi181 ceph-mon[35869]: pgmap v513: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:51:54.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.878+0000 7fbc41400640 1 -- 172.21.15.97:0/1191443970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 msgr2=0x7fbc3c109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:54.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.878+0000 7fbc41400640 1 --2- 172.21.15.97:0/1191443970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 0x7fbc3c109300 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009900 tx=0x7fbc2802a410 comp rx=0 tx=0).stop 2024-07-26T07:51:54.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.879+0000 7fbc41400640 1 -- 172.21.15.97:0/1191443970 shutdown_connections 2024-07-26T07:51:54.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.879+0000 7fbc41400640 1 --2- 172.21.15.97:0/1191443970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c109840 0x7fbc3c110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:54.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.879+0000 7fbc41400640 1 --2- 172.21.15.97:0/1191443970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 0x7fbc3c109300 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:54.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.879+0000 7fbc41400640 1 --2- 172.21.15.97:0/1191443970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc3c1035b0 0x7fbc3c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:54.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.879+0000 7fbc41400640 1 -- 172.21.15.97:0/1191443970 >> 172.21.15.97:0/1191443970 conn(0x7fbc3c0fefc0 msgr2=0x7fbc3c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:54.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.879+0000 7fbc41400640 1 -- 172.21.15.97:0/1191443970 shutdown_connections 2024-07-26T07:51:54.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.879+0000 7fbc41400640 1 -- 172.21.15.97:0/1191443970 wait complete. 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.880+0000 7fbc41400640 1 Processor -- start 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.880+0000 7fbc41400640 1 -- start start 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc41400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c1035b0 0x7fbc3c19c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc41400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 0x7fbc3c19c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc41400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc3c109840 0x7fbc3c1a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc41400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3c113380 con 0x7fbc3c108ef0 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc41400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3c113200 con 0x7fbc3c109840 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc41400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3c113500 con 0x7fbc3c1035b0 2024-07-26T07:51:54.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc39e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 0x7fbc3c19c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:54.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc39e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 0x7fbc3c19c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42350/0 (socket says 172.21.15.97:42350) 2024-07-26T07:51:54.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc39e00640 1 -- 172.21.15.97:0/157999363 learned_addr learned my addr 172.21.15.97:0/157999363 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:54.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.881+0000 7fbc3a800640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c1035b0 0x7fbc3c19c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:54.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.882+0000 7fbc3b200640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc3c109840 0x7fbc3c1a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:54.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.882+0000 7fbc39e00640 1 -- 172.21.15.97:0/157999363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c1035b0 msgr2=0x7fbc3c19c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:54.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.882+0000 7fbc39e00640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c1035b0 0x7fbc3c19c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.882+0000 7fbc39e00640 1 -- 172.21.15.97:0/157999363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc3c109840 msgr2=0x7fbc3c1a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.882+0000 7fbc39e00640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc3c109840 0x7fbc3c1a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.882+0000 7fbc39e00640 1 -- 172.21.15.97:0/157999363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3c1a1fe0 con 0x7fbc3c108ef0 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.882+0000 7fbc39e00640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 0x7fbc3c19c610 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbc28005940 tx=0x7fbc280040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.882+0000 7fbc33400640 1 -- 172.21.15.97:0/157999363 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2803f030 con 0x7fbc3c108ef0 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.883+0000 7fbc33400640 1 -- 172.21.15.97:0/157999363 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc28003a80 con 0x7fbc3c108ef0 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.883+0000 7fbc33400640 1 -- 172.21.15.97:0/157999363 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc28035070 con 0x7fbc3c108ef0 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.883+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc3c1a2270 con 0x7fbc3c108ef0 2024-07-26T07:51:54.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.883+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc3c1a8a30 con 0x7fbc3c108ef0 2024-07-26T07:51:54.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.884+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc3c1039c0 con 0x7fbc3c108ef0 2024-07-26T07:51:54.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.887+0000 7fbc33400640 1 -- 172.21.15.97:0/157999363 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7fbc28042030 con 0x7fbc3c108ef0 2024-07-26T07:51:54.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.890+0000 7fbc33400640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbc200b3c60 0x7fbc200b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:54.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.890+0000 7fbc33400640 1 -- 172.21.15.97:0/157999363 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fbc281047d0 con 0x7fbc3c108ef0 2024-07-26T07:51:54.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.890+0000 7fbc3a800640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbc200b3c60 0x7fbc200b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:54.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.891+0000 7fbc33400640 1 -- 172.21.15.97:0/157999363 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbc280cbc60 con 0x7fbc3c108ef0 2024-07-26T07:51:54.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:54.891+0000 7fbc3a800640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbc200b3c60 0x7fbc200b6170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c005fd0 tx=0x7fbc2c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:55.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:54.999+0000 7fe7caa00640 1 -- 172.21.15.97:0/2984028378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4104940 msgr2=0x7fe7c410b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:54.999+0000 7fe7caa00640 1 --2- 172.21.15.97:0/2984028378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4104940 0x7fe7c410b2e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009ca0 tx=0x7fe7bc02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:54.999+0000 7fe7caa00640 1 -- 172.21.15.97:0/2984028378 shutdown_connections 2024-07-26T07:51:55.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:54.999+0000 7fe7caa00640 1 --2- 172.21.15.97:0/2984028378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4104940 0x7fe7c410b2e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:54.999+0000 7fe7caa00640 1 --2- 172.21.15.97:0/2984028378 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c4103fa0 0x7fe7c41043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:54.999+0000 7fe7caa00640 1 --2- 172.21.15.97:0/2984028378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c4102f20 0x7fe7c4103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:54.999+0000 7fe7caa00640 1 -- 172.21.15.97:0/2984028378 >> 172.21.15.97:0/2984028378 conn(0x7fe7c40fe860 msgr2=0x7fe7c4100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:55.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.000+0000 7fe7caa00640 1 -- 172.21.15.97:0/2984028378 shutdown_connections 2024-07-26T07:51:55.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.000+0000 7fe7caa00640 1 -- 172.21.15.97:0/2984028378 wait complete. 2024-07-26T07:51:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.001+0000 7fe7caa00640 1 Processor -- start 2024-07-26T07:51:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.001+0000 7fe7caa00640 1 -- start start 2024-07-26T07:51:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.001+0000 7fe7caa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4102f20 0x7fe7c419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.001+0000 7fe7caa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c4103fa0 0x7fe7c419bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.001+0000 7fe7caa00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c4104940 0x7fe7c419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.001+0000 7fe7caa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c410eaa0 con 0x7fe7c4102f20 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.001+0000 7fe7caa00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c410e920 con 0x7fe7c4104940 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.001+0000 7fe7caa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c410ec20 con 0x7fe7c4103fa0 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4102f20 0x7fe7c419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4102f20 0x7fe7c419b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42368/0 (socket says 172.21.15.97:42368) 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 -- 172.21.15.97:0/1473504188 learned_addr learned my addr 172.21.15.97:0/1473504188 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7ca000640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c4104940 0x7fe7c419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c4103fa0 0x7fe7c419bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 -- 172.21.15.97:0/1473504188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c4103fa0 msgr2=0x7fe7c419bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c4103fa0 0x7fe7c419bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 -- 172.21.15.97:0/1473504188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c4104940 msgr2=0x7fe7c419cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c4104940 0x7fe7c419cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 -- 172.21.15.97:0/1473504188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c419d2e0 con 0x7fe7c4102f20 2024-07-26T07:51:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c4103fa0 0x7fe7c419bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.002+0000 7fe7c9600640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4102f20 0x7fe7c419b8a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800e9f0 tx=0x7fe7b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.003+0000 7fe7c2000640 1 -- 172.21.15.97:0/1473504188 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b800cd80 con 0x7fe7c4102f20 2024-07-26T07:51:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.003+0000 7fe7c2000640 1 -- 172.21.15.97:0/1473504188 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7b80054b0 con 0x7fe7c4102f20 2024-07-26T07:51:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.003+0000 7fe7caa00640 1 -- 172.21.15.97:0/1473504188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c419d5d0 con 0x7fe7c4102f20 2024-07-26T07:51:55.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.003+0000 7fe7c2000640 1 -- 172.21.15.97:0/1473504188 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8010430 con 0x7fe7c4102f20 2024-07-26T07:51:55.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.003+0000 7fe7caa00640 1 -- 172.21.15.97:0/1473504188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c41a3cc0 con 0x7fe7c4102f20 2024-07-26T07:51:55.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.007+0000 7fe7d3082740 1 Processor -- start 2024-07-26T07:51:55.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.008+0000 7fe7d3082740 1 -- start start 2024-07-26T07:51:55.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.008+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.008+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c270c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.008+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c270d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.008+0000 7fe7d3082740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a3c267e270 con 0x55a3c23454d0 2024-07-26T07:51:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.008+0000 7fe7d3082740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a3c266d770 con 0x55a3c233d2f0 2024-07-26T07:51:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.008+0000 7fe7d3082740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a3c26717b0 con 0x55a3c23661b0 2024-07-26T07:51:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c270d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c2000640 1 -- 172.21.15.97:0/1473504188 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7fe7b80046c0 con 0x7fe7c4102f20 2024-07-26T07:51:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c270d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42376/0 (socket says 172.21.15.97:42376) 2024-07-26T07:51:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c8c00640 1 -- 172.21.15.97:0/4187081578 learned_addr learned my addr 172.21.15.97:0/4187081578 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7ca000640 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c8c00640 1 -- 172.21.15.97:0/4187081578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 msgr2=0x55a3c270c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c8c00640 1 -- 172.21.15.97:0/4187081578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 msgr2=0x55a3c270c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c270c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7c8c00640 1 -- 172.21.15.97:0/4187081578 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c23d1790 con 0x55a3c23454d0 2024-07-26T07:51:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.009+0000 7fe7ca000640 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.010+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c270d2a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0005fd0 tx=0x7fe7b000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927e36f56e9de546 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.010+0000 7fe7b6600640 1 -- 172.21.15.97:0/4187081578 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0016070 con 0x55a3c23454d0 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.010+0000 7fe7b6600640 1 -- 172.21.15.97:0/4187081578 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7b00054b0 con 0x55a3c23454d0 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.010+0000 7fe7b6600640 1 -- 172.21.15.97:0/4187081578 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0011470 con 0x55a3c23454d0 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.010+0000 7fe7c2000640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe7ac0b3ce0 0x7fe7ac0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7c2000640 1 -- 172.21.15.97:0/1473504188 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fe7b80e42b0 con 0x7fe7c4102f20 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7ca000640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe7ac0b3ce0 0x7fe7ac0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 -- 172.21.15.97:0/4187081578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 msgr2=0x55a3c270d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c270d2a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0005fd0 tx=0x7fe7b000e6c0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 -- 172.21.15.97:0/4187081578 shutdown_connections 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c270d2a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c270c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 --2- 172.21.15.97:0/4187081578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 -- 172.21.15.97:0/4187081578 >> 172.21.15.97:0/4187081578 conn(0x55a3c235a300 msgr2=0x55a3c2707b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 -- 172.21.15.97:0/4187081578 shutdown_connections 2024-07-26T07:51:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.011+0000 7fe7d3082740 1 -- 172.21.15.97:0/4187081578 wait complete. 2024-07-26T07:51:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.012+0000 7fe7ca000640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe7ac0b3ce0 0x7fe7ac0b61f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0319e0 tx=0x7fe7bc009410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.012+0000 7fe7d3082740 1 -- start start 2024-07-26T07:51:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.013+0000 7fe7d3082740 1 Processor -- start 2024-07-26T07:51:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.013+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c2715bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.013+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c27162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.013+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c2716e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.013+0000 7fe7d3082740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a3c25a7ab0 con 0x55a3c23454d0 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.013+0000 7fe7d3082740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a3c260ad50 con 0x55a3c233d2f0 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.013+0000 7fe7d3082740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a3c270ddc0 con 0x55a3c23661b0 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7d3082740 10 client.0 _ll_register_callbacks cb 0x55a3c237da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c27162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c27162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42390/0 (socket says 172.21.15.97:42390) 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c8c00640 1 -- 172.21.15.97:0/3247727339 learned_addr learned my addr 172.21.15.97:0/3247727339 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c9600640 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c2715bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7ca000640 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c2716e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c8c00640 1 -- 172.21.15.97:0/3247727339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 msgr2=0x55a3c2716e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c2716e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c8c00640 1 -- 172.21.15.97:0/3247727339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 msgr2=0x55a3c2715bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c2715bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7c8c00640 1 -- 172.21.15.97:0/3247727339 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c244a620 con 0x55a3c23454d0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.014+0000 7fe7ca000640 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c2716e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.015+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c27162b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009a10 tx=0x7fe7b0009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535470da4c29cabc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.015+0000 7fe7aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.015+0000 7fe7aaa00640 1 -- 172.21.15.97:0/3247727339 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0016040 con 0x55a3c23454d0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.015+0000 7fe7d3082740 1 -- 172.21.15.97:0/3247727339 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7b00054b0 con 0x55a3c23454d0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.015+0000 7fe7aaa00640 1 -- 172.21.15.97:0/3247727339 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fe7b0004ea0 con 0x55a3c23454d0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.015+0000 7fe7aaa00640 1 -- 172.21.15.97:0/3247727339 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe7b00050a0 con 0x55a3c23454d0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.015+0000 7fe7d3082740 10 client.15414 fetch_fsmap learned FSMap version 28 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.015+0000 7fe7d3082740 1 -- 172.21.15.97:0/3247727339 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a3c271fd70 con 0x55a3c23454d0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.016+0000 7fe7aaa00640 1 -- 172.21.15.97:0/3247727339 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7b00054b0 con 0x55a3c23454d0 2024-07-26T07:51:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.016+0000 7fe7aaa00640 1 -- 172.21.15.97:0/3247727339 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe7b000f040 con 0x55a3c23454d0 2024-07-26T07:51:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.016+0000 7fe7d3082740 10 client.15414 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:51:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.016+0000 7fe7d3082740 10 client.15414 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:51:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.016+0000 7fe7d3082740 20 client.15414 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.016+0000 7fe7d3082740 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a3c2324d80 0x55a3c27231e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.016+0000 7fe7d3082740 4 client.15414 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:51:55.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.016+0000 7fe7d3082740 1 -- 172.21.15.97:0/3247727339 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3c26801c0 con 0x55a3c2324d80 2024-07-26T07:51:55.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.017+0000 7fe7c9600640 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a3c2324d80 0x55a3c27231e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.017+0000 7fe7c9600640 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a3c2324d80 0x55a3c27231e0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86e262f2d615077f server_cookie=176508adf6a5e5f2 in_seq=0 out_seq=0 2024-07-26T07:51:55.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.017+0000 7fe7aaa00640 10 client.15414 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:51:55.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.019+0000 7fe7aaa00640 1 -- 172.21.15.97:0/3247727339 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a3c26801c0 con 0x55a3c2324d80 2024-07-26T07:51:55.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.019+0000 7fe7aaa00640 10 client.15414 handle_command_reply: tid=0 2024-07-26T07:51:55.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.019+0000 7fe7d3082740 1 client.15414 shutdown 2024-07-26T07:51:55.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 -- 172.21.15.97:0/3247727339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 msgr2=0x55a3c27162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c27162b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009a10 tx=0x7fe7b0009ea0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 -- 172.21.15.97:0/3247727339 shutdown_connections 2024-07-26T07:51:55.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a3c2324d80 0x55a3c27231e0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c2716e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c27162b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7aaa00640 0 client.15414 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:51:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 --2- 172.21.15.97:0/3247727339 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c2715bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 -- 172.21.15.97:0/3247727339 >> 172.21.15.97:0/3247727339 conn(0x55a3c235a300 msgr2=0x55a3c27060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 -- 172.21.15.97:0/3247727339 shutdown_connections 2024-07-26T07:51:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 1 -- 172.21.15.97:0/3247727339 wait complete. 2024-07-26T07:51:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.020+0000 7fe7d3082740 20 client.15414 trim_cache size 0 max 16384 2024-07-26T07:51:55.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.036+0000 7fe7d3082740 1 Processor -- start 2024-07-26T07:51:55.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.036+0000 7fe7d3082740 1 -- start start 2024-07-26T07:51:55.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c270c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7d3082740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c41acad0 con 0x55a3c23454d0 2024-07-26T07:51:55.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7d3082740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c41ae610 con 0x55a3c233d2f0 2024-07-26T07:51:55.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7d3082740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c41ad1a0 con 0x55a3c23661b0 2024-07-26T07:51:55.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42400/0 (socket says 172.21.15.97:42400) 2024-07-26T07:51:55.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.037+0000 7fe7c8c00640 1 -- 172.21.15.97:0/432655933 learned_addr learned my addr 172.21.15.97:0/432655933 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:55.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.038+0000 7fe7ca000640 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.038+0000 7fe7c9600640 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c270c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.038+0000 7fe7c8c00640 1 -- 172.21.15.97:0/432655933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 msgr2=0x55a3c270c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:55.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.038+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.038+0000 7fe7c8c00640 1 -- 172.21.15.97:0/432655933 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 msgr2=0x55a3c270c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.038+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c270c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.038+0000 7fe7c8c00640 1 -- 172.21.15.97:0/432655933 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c2713d70 con 0x55a3c23454d0 2024-07-26T07:51:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.038+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271dd00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0006010 tx=0x7fe7b0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269012addf34d981 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.039+0000 7fe7b5c00640 1 -- 172.21.15.97:0/432655933 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b00235d0 con 0x55a3c23454d0 2024-07-26T07:51:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.039+0000 7fe7b5c00640 1 -- 172.21.15.97:0/432655933 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7b0023770 con 0x55a3c23454d0 2024-07-26T07:51:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.039+0000 7fe7b5c00640 1 -- 172.21.15.97:0/432655933 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b00116d0 con 0x55a3c23454d0 2024-07-26T07:51:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 -- 172.21.15.97:0/432655933 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 msgr2=0x55a3c271dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271dd00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0006010 tx=0x7fe7b0005030 comp rx=0 tx=0).stop 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 -- 172.21.15.97:0/432655933 shutdown_connections 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c23661b0 0x55a3c270c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271dd00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 --2- 172.21.15.97:0/432655933 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c233d2f0 0x55a3c270c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 -- 172.21.15.97:0/432655933 >> 172.21.15.97:0/432655933 conn(0x55a3c235a300 msgr2=0x55a3c27068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 -- 172.21.15.97:0/432655933 shutdown_connections 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 -- 172.21.15.97:0/432655933 wait complete. 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 -- start start 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.040+0000 7fe7d3082740 1 Processor -- start 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c233d2f0 0x55a3c271a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7d3082740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c23661b0 0x55a3c271b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7d3082740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c41adf40 con 0x55a3c23454d0 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7d3082740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c41ad870 con 0x55a3c23661b0 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7d3082740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c41aaab0 con 0x55a3c233d2f0 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7d3082740 10 client.0 _ll_register_callbacks cb 0x7fe7b0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42408/0 (socket says 172.21.15.97:42408) 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7c8c00640 1 -- 172.21.15.97:0/210463039 learned_addr learned my addr 172.21.15.97:0/210463039 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.041+0000 7fe7c9600640 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c233d2f0 0x55a3c271a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7ca000640 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c23661b0 0x55a3c271b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7c8c00640 1 -- 172.21.15.97:0/210463039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c233d2f0 msgr2=0x55a3c271a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c233d2f0 0x55a3c271a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7c8c00640 1 -- 172.21.15.97:0/210463039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c23661b0 msgr2=0x55a3c271b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c23661b0 0x55a3c271b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7c8c00640 1 -- 172.21.15.97:0/210463039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c271c140 con 0x55a3c23454d0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7c9600640 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c233d2f0 0x55a3c271a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7c8c00640 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271ae70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0004a10 tx=0x7fe7b0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e0f8ee746d172e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7aaa00640 1 -- 172.21.15.97:0/210463039 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0016030 con 0x55a3c23454d0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7aaa00640 1 -- 172.21.15.97:0/210463039 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fe7b0011470 con 0x55a3c23454d0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.042+0000 7fe7d3082740 1 -- 172.21.15.97:0/210463039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7b0023770 con 0x55a3c23454d0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7aaa00640 1 -- 172.21.15.97:0/210463039 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe7b0045050 con 0x55a3c23454d0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7d3082740 10 client.15420 fetch_fsmap learned FSMap version 28 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7d3082740 1 -- 172.21.15.97:0/210463039 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a3c271d460 con 0x55a3c23454d0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7aaa00640 1 -- 172.21.15.97:0/210463039 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7b0023770 con 0x55a3c23454d0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7aaa00640 1 -- 172.21.15.97:0/210463039 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe7b0036550 con 0x55a3c23454d0 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7d3082740 10 client.15420 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:51:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7d3082740 10 client.15420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:51:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7d3082740 20 client.15420 populate_metadata read hostname 'smithi097' 2024-07-26T07:51:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7d3082740 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a3c22fa1c0 0x55a3c278d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7d3082740 4 client.15420 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:51:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.043+0000 7fe7d3082740 1 -- 172.21.15.97:0/210463039 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a3c267bc60 con 0x55a3c22fa1c0 2024-07-26T07:51:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.044+0000 7fe7c9600640 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a3c22fa1c0 0x55a3c278d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.045+0000 7fe7c9600640 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a3c22fa1c0 0x55a3c278d910 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d42882870e843a22 server_cookie=6ce47273e09c2979 in_seq=0 out_seq=0 2024-07-26T07:51:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.045+0000 7fe7aaa00640 10 client.15420 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:51:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.082+0000 7fe7aaa00640 1 -- 172.21.15.97:0/210463039 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a3c267bc60 con 0x55a3c22fa1c0 2024-07-26T07:51:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.082+0000 7fe7aaa00640 10 client.15420 handle_command_reply: tid=0 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.082+0000 7fe7d3082740 1 client.15420 shutdown 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 -- 172.21.15.97:0/210463039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 msgr2=0x55a3c271ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271ae70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0004a10 tx=0x7fe7b0005c50 comp rx=0 tx=0).stop 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 -- 172.21.15.97:0/210463039 shutdown_connections 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a3c22fa1c0 0x55a3c278d910 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a3c23661b0 0x55a3c271b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a3c23454d0 0x55a3c271ae70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7aaa00640 0 client.15420 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 --2- 172.21.15.97:0/210463039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a3c233d2f0 0x55a3c271a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 -- 172.21.15.97:0/210463039 >> 172.21.15.97:0/210463039 conn(0x55a3c235a300 msgr2=0x55a3c270c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 -- 172.21.15.97:0/210463039 shutdown_connections 2024-07-26T07:51:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 1 -- 172.21.15.97:0/210463039 wait complete. 2024-07-26T07:51:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.083+0000 7fe7d3082740 20 client.15420 trim_cache size 0 max 16384 2024-07-26T07:51:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:51:55.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:51:55.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:51:55.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.084+0000 7fe7caa00640 1 -- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe7ac0b3ce0 msgr2=0x7fe7ac0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.084+0000 7fe7caa00640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe7ac0b3ce0 0x7fe7ac0b61f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0319e0 tx=0x7fe7bc009410 comp rx=0 tx=0).stop 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.084+0000 7fe7caa00640 1 -- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4102f20 msgr2=0x7fe7c419b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.084+0000 7fe7caa00640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4102f20 0x7fe7c419b8a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800e9f0 tx=0x7fe7b800edf0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.085+0000 7fe7caa00640 1 -- 172.21.15.97:0/1473504188 shutdown_connections 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.085+0000 7fe7caa00640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe7ac0b3ce0 0x7fe7ac0b61f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.085+0000 7fe7caa00640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c4104940 0x7fe7c419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.085+0000 7fe7caa00640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c4103fa0 0x7fe7c419bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.085+0000 7fe7caa00640 1 --2- 172.21.15.97:0/1473504188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe7c4102f20 0x7fe7c419b8a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.085+0000 7fe7caa00640 1 -- 172.21.15.97:0/1473504188 >> 172.21.15.97:0/1473504188 conn(0x7fe7c40fe860 msgr2=0x7fe7c4075020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.085+0000 7fe7caa00640 1 -- 172.21.15.97:0/1473504188 shutdown_connections 2024-07-26T07:51:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:51:55.085+0000 7fe7caa00640 1 -- 172.21.15.97:0/1473504188 wait complete. 2024-07-26T07:51:55.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.143+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc3c1a8c00 con 0x7fbc3c108ef0 2024-07-26T07:51:55.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.145+0000 7fbc33400640 1 -- 172.21.15.97:0/157999363 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11990 (secure 0 0 0) 0x7fbc280cb560 con 0x7fbc3c108ef0 2024-07-26T07:51:55.147 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:55.147 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"btime":"2024-07-26T07:51:41:000369+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:51:40.647454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:51:55.147 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-07-26T07:51:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbc200b3c60 msgr2=0x7fbc200b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbc200b3c60 0x7fbc200b6170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c005fd0 tx=0x7fbc2c005850 comp rx=0 tx=0).stop 2024-07-26T07:51:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 msgr2=0x7fbc3c19c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 0x7fbc3c19c610 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbc28005940 tx=0x7fbc280040e0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 shutdown_connections 2024-07-26T07:51:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbc200b3c60 0x7fbc200b6170 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc3c109840 0x7fbc3c1a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc3c108ef0 0x7fbc3c19c610 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 --2- 172.21.15.97:0/157999363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c1035b0 0x7fbc3c19c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:55.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 >> 172.21.15.97:0/157999363 conn(0x7fbc3c0fefc0 msgr2=0x7fbc3c1076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:55.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 shutdown_connections 2024-07-26T07:51:55.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:55.148+0000 7fbc41400640 1 -- 172.21.15.97:0/157999363 wait complete. 2024-07-26T07:51:55.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:51:56.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:56 smithi097 ceph-mon[32231]: pgmap v514: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-07-26T07:51:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:56 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/157999363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:56.400 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:51:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:56 smithi115 ceph-mon[35364]: pgmap v514: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-07-26T07:51:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:56 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/157999363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:56 smithi181 ceph-mon[35869]: pgmap v514: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-07-26T07:51:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:56 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/157999363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:58.336 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:58 smithi097 ceph-mon[32231]: pgmap v515: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T07:51:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:58 smithi115 ceph-mon[35364]: pgmap v515: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T07:51:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:58 smithi181 ceph-mon[35869]: pgmap v515: 97 pgs: 97 active+clean; 652 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T07:51:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 -- 172.21.15.97:0/3093872971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 msgr2=0x7f539806b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 --2- 172.21.15.97:0/3093872971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 0x7f539806b6a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5388009900 tx=0x7f538802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 -- 172.21.15.97:0/3093872971 shutdown_connections 2024-07-26T07:51:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 --2- 172.21.15.97:0/3093872971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5398118470 0x7f539811a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 --2- 172.21.15.97:0/3093872971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f539806bbe0 0x7f539806bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 --2- 172.21.15.97:0/3093872971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 0x7f539806b6a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 -- 172.21.15.97:0/3093872971 >> 172.21.15.97:0/3093872971 conn(0x7f539806e360 msgr2=0x7f5398070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 -- 172.21.15.97:0/3093872971 shutdown_connections 2024-07-26T07:51:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.537+0000 7f539f200640 1 -- 172.21.15.97:0/3093872971 wait complete. 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.538+0000 7f539f200640 1 Processor -- start 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f539f200640 1 -- start start 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f539f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f539806bbe0 0x7f53981a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f539f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 0x7f53981a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f539f200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5398118470 0x7f53981aa030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f539f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f539811dd80 con 0x7f539810c860 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f539f200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f539811dc00 con 0x7f5398118470 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f539f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f539811df00 con 0x7f539806bbe0 2024-07-26T07:51:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f5397400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 0x7f53981a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f5397400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 0x7f53981a4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32828/0 (socket says 172.21.15.97:32828) 2024-07-26T07:51:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.539+0000 7f5397400640 1 -- 172.21.15.97:0/1131523042 learned_addr learned my addr 172.21.15.97:0/1131523042 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f5397e00640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f539806bbe0 0x7f53981a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f539d200640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5398118470 0x7f53981aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f5397400640 1 -- 172.21.15.97:0/1131523042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f539806bbe0 msgr2=0x7f53981a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f5397400640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f539806bbe0 0x7f53981a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f5397400640 1 -- 172.21.15.97:0/1131523042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5398118470 msgr2=0x7f53981aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f5397400640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5398118470 0x7f53981aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f5397400640 1 -- 172.21.15.97:0/1131523042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53981aa6a0 con 0x7f539810c860 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f5397e00640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f539806bbe0 0x7f53981a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.540+0000 7f5397400640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 0x7f53981a4d90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f538400c8c0 tx=0x7f538400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.541+0000 7f5394c00640 1 -- 172.21.15.97:0/1131523042 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384007bb0 con 0x7f539810c860 2024-07-26T07:51:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.541+0000 7f5394c00640 1 -- 172.21.15.97:0/1131523042 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5384007d50 con 0x7f539810c860 2024-07-26T07:51:58.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.541+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53981aa990 con 0x7f539810c860 2024-07-26T07:51:58.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.541+0000 7f5394c00640 1 -- 172.21.15.97:0/1131523042 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53840186a0 con 0x7f539810c860 2024-07-26T07:51:58.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.541+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53981b11b0 con 0x7f539810c860 2024-07-26T07:51:58.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.542+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f539806bff0 con 0x7f539810c860 2024-07-26T07:51:58.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.544+0000 7f5394c00640 1 -- 172.21.15.97:0/1131523042 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f538400fa00 con 0x7f539810c860 2024-07-26T07:51:58.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.546+0000 7f5394c00640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f537c0b3d10 0x7f537c0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:51:58.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.546+0000 7f5394c00640 1 -- 172.21.15.97:0/1131523042 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f53840e3f20 con 0x7f539810c860 2024-07-26T07:51:58.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.546+0000 7f5397e00640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f537c0b3d10 0x7f537c0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:51:58.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.547+0000 7f5397e00640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f537c0b3d10 0x7f537c0b6220 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5388000c00 tx=0x7f5388005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:51:58.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.550+0000 7f5394c00640 1 -- 172.21.15.97:0/1131523042 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5384014030 con 0x7f539810c860 2024-07-26T07:51:58.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.787+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53981a5920 con 0x7f539810c860 2024-07-26T07:51:58.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.788+0000 7f5394c00640 1 -- 172.21.15.97:0/1131523042 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11990 (secure 0 0 0) 0x7f53840ab350 con 0x7f539810c860 2024-07-26T07:51:58.790 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:51:58.790 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"btime":"2024-07-26T07:51:41:000369+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:51:40.647454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:51:58.790 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-07-26T07:51:58.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.791+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f537c0b3d10 msgr2=0x7f537c0b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.791+0000 7f539f200640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f537c0b3d10 0x7f537c0b6220 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5388000c00 tx=0x7f5388005a20 comp rx=0 tx=0).stop 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.791+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 msgr2=0x7f53981a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.791+0000 7f539f200640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 0x7f53981a4d90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f538400c8c0 tx=0x7f538400cd80 comp rx=0 tx=0).stop 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.792+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 shutdown_connections 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.792+0000 7f539f200640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f537c0b3d10 0x7f537c0b6220 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.792+0000 7f539f200640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5398118470 0x7f53981aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.792+0000 7f539f200640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f539810c860 0x7f53981a4d90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.792+0000 7f539f200640 1 --2- 172.21.15.97:0/1131523042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f539806bbe0 0x7f53981a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:51:58.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.792+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 >> 172.21.15.97:0/1131523042 conn(0x7f539806e360 msgr2=0x7f5398118880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:51:58.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.792+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 shutdown_connections 2024-07-26T07:51:58.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:51:58.792+0000 7f539f200640 1 -- 172.21.15.97:0/1131523042 wait complete. 2024-07-26T07:51:59.389 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:51:59 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1131523042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:59.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T07:51:59.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:51:59 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1131523042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:51:59 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1131523042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:51:59.930 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:00.376 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:00 smithi097 ceph-mon[32231]: pgmap v516: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T07:52:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:00 smithi115 ceph-mon[35364]: pgmap v516: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T07:52:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:00 smithi181 ceph-mon[35869]: pgmap v516: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T07:52:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 -- 172.21.15.97:0/2138126964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb604102060 msgr2=0x7fb604102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 --2- 172.21.15.97:0/2138126964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb604102060 0x7fb604102470 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009900 tx=0x7fb5f802a480 comp rx=0 tx=0).stop 2024-07-26T07:52:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 -- 172.21.15.97:0/2138126964 shutdown_connections 2024-07-26T07:52:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 --2- 172.21.15.97:0/2138126964 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb60410fc10 0x7fb604112030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 --2- 172.21.15.97:0/2138126964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6041029b0 0x7fb60410f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 --2- 172.21.15.97:0/2138126964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb604102060 0x7fb604102470 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 -- 172.21.15.97:0/2138126964 >> 172.21.15.97:0/2138126964 conn(0x7fb604079160 msgr2=0x7fb604079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 -- 172.21.15.97:0/2138126964 shutdown_connections 2024-07-26T07:52:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.034+0000 7fb60b400640 1 -- 172.21.15.97:0/2138126964 wait complete. 2024-07-26T07:52:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.035+0000 7fb60b400640 1 Processor -- start 2024-07-26T07:52:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.035+0000 7fb60b400640 1 -- start start 2024-07-26T07:52:02.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.036+0000 7fb60b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb604102060 0x7fb60419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:02.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.036+0000 7fb60b400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6041029b0 0x7fb60419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:02.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.036+0000 7fb60b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb60410fc10 0x7fb6041a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:02.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.036+0000 7fb60b400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb604115540 con 0x7fb60410fc10 2024-07-26T07:52:02.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.036+0000 7fb60b400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041153c0 con 0x7fb6041029b0 2024-07-26T07:52:02.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.036+0000 7fb60b400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041156c0 con 0x7fb604102060 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.036+0000 7fb609400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb60410fc10 0x7fb6041a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.036+0000 7fb608a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb604102060 0x7fb60419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb608a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb604102060 0x7fb60419bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:59902/0 (socket says 172.21.15.97:59902) 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb608a00640 1 -- 172.21.15.97:0/4126891527 learned_addr learned my addr 172.21.15.97:0/4126891527 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb609400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb60410fc10 0x7fb6041a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32854/0 (socket says 172.21.15.97:32854) 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb603e00640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6041029b0 0x7fb60419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb609400640 1 -- 172.21.15.97:0/4126891527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb604102060 msgr2=0x7fb60419bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb609400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb604102060 0x7fb60419bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb609400640 1 -- 172.21.15.97:0/4126891527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6041029b0 msgr2=0x7fb60419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb609400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6041029b0 0x7fb60419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb609400640 1 -- 172.21.15.97:0/4126891527 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6041a1f00 con 0x7fb60410fc10 2024-07-26T07:52:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.037+0000 7fb608a00640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb604102060 0x7fb60419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:52:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.038+0000 7fb609400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb60410fc10 0x7fb6041a17d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e940 tx=0x7fb5fc00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.038+0000 7fb601600640 1 -- 172.21.15.97:0/4126891527 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc004d20 con 0x7fb60410fc10 2024-07-26T07:52:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.038+0000 7fb601600640 1 -- 172.21.15.97:0/4126891527 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb5fc005020 con 0x7fb60410fc10 2024-07-26T07:52:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.038+0000 7fb601600640 1 -- 172.21.15.97:0/4126891527 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0116d0 con 0x7fb60410fc10 2024-07-26T07:52:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.038+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6041a21f0 con 0x7fb60410fc10 2024-07-26T07:52:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.038+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6041a8950 con 0x7fb60410fc10 2024-07-26T07:52:02.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.040+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb604102470 con 0x7fb60410fc10 2024-07-26T07:52:02.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.042+0000 7fb601600640 1 -- 172.21.15.97:0/4126891527 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7fb5fc00cb20 con 0x7fb60410fc10 2024-07-26T07:52:02.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.044+0000 7fb601600640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb5ec0b3ce0 0x7fb5ec0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:02.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.045+0000 7fb601600640 1 -- 172.21.15.97:0/4126891527 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fb5fc0e4280 con 0x7fb60410fc10 2024-07-26T07:52:02.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.045+0000 7fb608a00640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb5ec0b3ce0 0x7fb5ec0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:02.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.046+0000 7fb608a00640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb5ec0b3ce0 0x7fb5ec0b61f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8000c00 tx=0x7fb5f8005e60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:02.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.047+0000 7fb601600640 1 -- 172.21.15.97:0/4126891527 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb5fc016030 con 0x7fb60410fc10 2024-07-26T07:52:02.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.272+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb60419d0c0 con 0x7fb60410fc10 2024-07-26T07:52:02.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.274+0000 7fb601600640 1 -- 172.21.15.97:0/4126891527 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22853 (secure 0 0 0) 0x7fb60419d0c0 con 0x7fb60410fc10 2024-07-26T07:52:02.275 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:52:02.275 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":86,"fsid":"9621a072-4b21-11ef-bcb6-c7b262605968","created":"2024-07-26T07:36:12.782822+0000","modified":"2024-07-26T07:49:20.276546+0000","last_up_change":"2024-07-26T07:46:27.786610+0000","last_in_change":"2024-07-26T07:46:01.501557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T07:41:16.123688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T07:47:40.641206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T07:47:44.357412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d860fc33-42a6-4123-af2c-5adff0e9b852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6803","nonce":706181072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6805","nonce":706181072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6809","nonce":706181072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":706181072},{"type":"v1","addr":"172.21.15.97:6807","nonce":706181072}]},"public_addr":"172.21.15.97:6803/706181072","cluster_addr":"172.21.15.97:6805/706181072","heartbeat_back_addr":"172.21.15.97:6809/706181072","heartbeat_front_addr":"172.21.15.97:6807/706181072","state":["exists","up"]},{"osd":1,"uuid":"b2130819-8356-471c-b82d-89ce0ac47dc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6801","nonce":2733242052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6803","nonce":2733242052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6807","nonce":2733242052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2733242052},{"type":"v1","addr":"172.21.15.115:6805","nonce":2733242052}]},"public_addr":"172.21.15.115:6801/2733242052","cluster_addr":"172.21.15.115:6803/2733242052","heartbeat_back_addr":"172.21.15.115:6807/2733242052","heartbeat_front_addr":"172.21.15.115:6805/2733242052","state":["exists","up"]},{"osd":2,"uuid":"c7f693f0-de0d-45f7-8b94-ce3bfd72b2d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6801","nonce":1600494363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6803","nonce":1600494363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6807","nonce":1600494363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1600494363},{"type":"v1","addr":"172.21.15.181:6805","nonce":1600494363}]},"public_addr":"172.21.15.181:6801/1600494363","cluster_addr":"172.21.15.181:6803/1600494363","heartbeat_back_addr":"172.21.15.181:6807/1600494363","heartbeat_front_addr":"172.21.15.181:6805/1600494363","state":["exists","up"]},{"osd":3,"uuid":"144e0dff-895b-471b-bd0c-8fcf3f51ac36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6811","nonce":461651027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6813","nonce":461651027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6817","nonce":461651027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":461651027},{"type":"v1","addr":"172.21.15.97:6815","nonce":461651027}]},"public_addr":"172.21.15.97:6811/461651027","cluster_addr":"172.21.15.97:6813/461651027","heartbeat_back_addr":"172.21.15.97:6817/461651027","heartbeat_front_addr":"172.21.15.97:6815/461651027","state":["exists","up"]},{"osd":4,"uuid":"7fddb4b8-44aa-42d7-8df4-c304b85c69b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6809","nonce":1719675205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6811","nonce":1719675205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6815","nonce":1719675205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1719675205},{"type":"v1","addr":"172.21.15.115:6813","nonce":1719675205}]},"public_addr":"172.21.15.115:6809/1719675205","cluster_addr":"172.21.15.115:6811/1719675205","heartbeat_back_addr":"172.21.15.115:6815/1719675205","heartbeat_front_addr":"172.21.15.115:6813/1719675205","state":["exists","up"]},{"osd":5,"uuid":"1272d9a0-8173-47b6-bfe2-017da6dc526d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6809","nonce":1061806763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6811","nonce":1061806763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6815","nonce":1061806763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1061806763},{"type":"v1","addr":"172.21.15.181:6813","nonce":1061806763}]},"public_addr":"172.21.15.181:6809/1061806763","cluster_addr":"172.21.15.181:6811/1061806763","heartbeat_back_addr":"172.21.15.181:6815/1061806763","heartbeat_front_addr":"172.21.15.181:6813/1061806763","state":["exists","up"]},{"osd":6,"uuid":"9567a0ec-9e3d-4b09-afc6-050d6ef59a83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6819","nonce":3873314113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6821","nonce":3873314113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6825","nonce":3873314113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3873314113},{"type":"v1","addr":"172.21.15.97:6823","nonce":3873314113}]},"public_addr":"172.21.15.97:6819/3873314113","cluster_addr":"172.21.15.97:6821/3873314113","heartbeat_back_addr":"172.21.15.97:6825/3873314113","heartbeat_front_addr":"172.21.15.97:6823/3873314113","state":["exists","up"]},{"osd":7,"uuid":"256172cc-f069-4599-bd65-14deca123b42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6817","nonce":2846701605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6819","nonce":2846701605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6823","nonce":2846701605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":2846701605},{"type":"v1","addr":"172.21.15.115:6821","nonce":2846701605}]},"public_addr":"172.21.15.115:6817/2846701605","cluster_addr":"172.21.15.115:6819/2846701605","heartbeat_back_addr":"172.21.15.115:6823/2846701605","heartbeat_front_addr":"172.21.15.115:6821/2846701605","state":["exists","up"]},{"osd":8,"uuid":"63b664e5-d348-4a9a-afb9-fe1fd1892a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6817","nonce":2477056065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6819","nonce":2477056065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6823","nonce":2477056065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2477056065},{"type":"v1","addr":"172.21.15.181:6821","nonce":2477056065}]},"public_addr":"172.21.15.181:6817/2477056065","cluster_addr":"172.21.15.181:6819/2477056065","heartbeat_back_addr":"172.21.15.181:6823/2477056065","heartbeat_front_addr":"172.21.15.181:6821/2477056065","state":["exists","up"]},{"osd":9,"uuid":"774b752e-42fc-4201-9051-9a9cdb1f495c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6827","nonce":2688123967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6829","nonce":2688123967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6833","nonce":2688123967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2688123967},{"type":"v1","addr":"172.21.15.97:6831","nonce":2688123967}]},"public_addr":"172.21.15.97:6827/2688123967","cluster_addr":"172.21.15.97:6829/2688123967","heartbeat_back_addr":"172.21.15.97:6833/2688123967","heartbeat_front_addr":"172.21.15.97:6831/2688123967","state":["exists","up"]},{"osd":10,"uuid":"3a43a0bf-0592-4daf-ae3f-a9b5c51f7ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6825","nonce":2791495352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6827","nonce":2791495352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6831","nonce":2791495352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":2791495352},{"type":"v1","addr":"172.21.15.115:6829","nonce":2791495352}]},"public_addr":"172.21.15.115:6825/2791495352","cluster_addr":"172.21.15.115:6827/2791495352","heartbeat_back_addr":"172.21.15.115:6831/2791495352","heartbeat_front_addr":"172.21.15.115:6829/2791495352","state":["exists","up"]},{"osd":11,"uuid":"0b5736e9-0d7b-40cc-94a3-c3a02c43dbb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6825","nonce":2026606552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6827","nonce":2026606552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6831","nonce":2026606552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2026606552},{"type":"v1","addr":"172.21.15.181:6829","nonce":2026606552}]},"public_addr":"172.21.15.181:6825/2026606552","cluster_addr":"172.21.15.181:6827/2026606552","heartbeat_back_addr":"172.21.15.181:6831/2026606552","heartbeat_front_addr":"172.21.15.181:6829/2026606552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:05.142514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:40:39.665985+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:13.358283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:41:47.076054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:22.038283+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:42:57.109937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:43:30.921214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:06.042295+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:44:40.751929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:15.448552+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:45:49.729053+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T07:46:25.100644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4135605223":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6800/4230832806":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2767087495":"2024-07-27T07:37:17.917351+0000","172.21.15.97:0/2403991941":"2024-07-27T07:37:17.917351+0000","172.21.15.97:6801/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:6800/3845319015":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1962341976":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/2498853950":"2024-07-27T07:36:48.864300+0000","172.21.15.97:0/1642500127":"2024-07-27T07:36:48.864300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T07:52:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb5ec0b3ce0 msgr2=0x7fb5ec0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:02.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb5ec0b3ce0 0x7fb5ec0b61f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8000c00 tx=0x7fb5f8005e60 comp rx=0 tx=0).stop 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb60410fc10 msgr2=0x7fb6041a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb60410fc10 0x7fb6041a17d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e940 tx=0x7fb5fc00ee00 comp rx=0 tx=0).stop 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 shutdown_connections 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb5ec0b3ce0 0x7fb5ec0b61f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb60410fc10 0x7fb6041a17d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6041029b0 0x7fb60419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 --2- 172.21.15.97:0/4126891527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb604102060 0x7fb60419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 >> 172.21.15.97:0/4126891527 conn(0x7fb604079160 msgr2=0x7fb6041086a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.277+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 shutdown_connections 2024-07-26T07:52:02.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:02.278+0000 7fb60b400640 1 -- 172.21.15.97:0/4126891527 wait complete. 2024-07-26T07:52:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:02 smithi115 ceph-mon[35364]: pgmap v517: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-07-26T07:52:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:52:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:02 smithi181 ceph-mon[35869]: pgmap v517: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-07-26T07:52:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:52:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:02 smithi097 ceph-mon[32231]: pgmap v517: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-07-26T07:52:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:52:03.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-07-26T07:52:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:52:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:52:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:03 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/4126891527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:52:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:52:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:52:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:52:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:03 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/4126891527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:52:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:52:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:52:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:52:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:03 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/4126891527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T07:52:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:52:03.510 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:04 smithi097 ceph-mon[32231]: pgmap v518: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-07-26T07:52:04.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:04 smithi115 ceph-mon[35364]: pgmap v518: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-07-26T07:52:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:04 smithi181 ceph-mon[35869]: pgmap v518: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-07-26T07:52:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:06 smithi115 ceph-mon[35364]: pgmap v519: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-07-26T07:52:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:06 smithi181 ceph-mon[35869]: pgmap v519: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-07-26T07:52:06.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:06 smithi097 ceph-mon[32231]: pgmap v519: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-07-26T07:52:06.542 INFO:teuthology.orchestra.run.smithi097.stdout:{"run_dir":"/var/run/ceph"} 2024-07-26T07:52:06.543 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-07-26T07:52:06.543 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 5532 2024-07-26T07:52:06.543 DEBUG:tasks.quiescer:Yielding 2024-07-26T07:52:06.543 INFO:teuthology.run_tasks:Running task workunit... 2024-07-26T07:52:06.694 INFO:tasks.workunit:Pulling workunits from ref e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:52:06.695 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-26T07:52:06.695 INFO:tasks.workunit:timeout=3h 2024-07-26T07:52:06.695 INFO:tasks.workunit:cleanup=True 2024-07-26T07:52:06.695 DEBUG:teuthology.orchestra.run.smithi097:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-26T07:52:06.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:52:06.752 INFO:teuthology.orchestra.run.smithi097.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-07-26T07:52:06.752 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-07-26T07:52:06.752 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-07-26T07:52:06.752 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T07:52:06.752 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:fusefs_t:s0 2024-07-26T07:52:06.752 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-07-26 07:48:19.380078664 +0000 2024-07-26T07:52:06.752 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-07-26 07:48:19.380078664 +0000 2024-07-26T07:52:06.752 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-07-26 07:51:40.085712345 +0000 2024-07-26T07:52:06.753 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2024-07-26T07:52:06.753 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-07-26T07:52:06.753 DEBUG:teuthology.orchestra.run.smithi097:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-07-26T07:52:06.881 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T07:52:06.926 INFO:tasks.workunit.client.0.smithi097.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-26T07:52:07.145 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.306+0000 7ff24e000640 1 -- 172.21.15.97:0/1123841011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 msgr2=0x7ff24810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.306+0000 7ff24e000640 1 --2- 172.21.15.97:0/1123841011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 0x7ff24810d4e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff240009900 tx=0x7ff24002a410 comp rx=0 tx=0).stop 2024-07-26T07:52:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.306+0000 7ff24e000640 1 -- 172.21.15.97:0/1123841011 shutdown_connections 2024-07-26T07:52:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.306+0000 7ff24e000640 1 --2- 172.21.15.97:0/1123841011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 0x7ff24810d4e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.306+0000 7ff24e000640 1 --2- 172.21.15.97:0/1123841011 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff2481061a0 0x7ff2481065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.306+0000 7ff24e000640 1 --2- 172.21.15.97:0/1123841011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff248105120 0x7ff248105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.306+0000 7ff24e000640 1 -- 172.21.15.97:0/1123841011 >> 172.21.15.97:0/1123841011 conn(0x7ff248100a80 msgr2=0x7ff248102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.307+0000 7ff24e000640 1 -- 172.21.15.97:0/1123841011 shutdown_connections 2024-07-26T07:52:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.307+0000 7ff24e000640 1 -- 172.21.15.97:0/1123841011 wait complete. 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.308+0000 7ff24e000640 1 Processor -- start 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.308+0000 7ff24e000640 1 -- start start 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.308+0000 7ff24e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff248105120 0x7ff24819b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.308+0000 7ff24e000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff2481061a0 0x7ff24819bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.308+0000 7ff24e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 0x7ff24819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.308+0000 7ff24e000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff248110ca0 con 0x7ff248106b40 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.308+0000 7ff24e000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff248110b20 con 0x7ff2481061a0 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.308+0000 7ff24e000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff248110e20 con 0x7ff248105120 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 0x7ff24819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 0x7ff24819cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33144/0 (socket says 172.21.15.97:33144) 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24d600640 1 -- 172.21.15.97:0/4178640073 learned_addr learned my addr 172.21.15.97:0/4178640073 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24cc00640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff248105120 0x7ff24819b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff247e00640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff2481061a0 0x7ff24819bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24d600640 1 -- 172.21.15.97:0/4178640073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff248105120 msgr2=0x7ff24819b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24d600640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff248105120 0x7ff24819b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24d600640 1 -- 172.21.15.97:0/4178640073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff2481061a0 msgr2=0x7ff24819bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24d600640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff2481061a0 0x7ff24819bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.309+0000 7ff24d600640 1 -- 172.21.15.97:0/4178640073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24819d320 con 0x7ff248106b40 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.310+0000 7ff247e00640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff2481061a0 0x7ff24819bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.310+0000 7ff24d600640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 0x7ff24819cb00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff24002a930 tx=0x7ff240003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.310+0000 7ff245600640 1 -- 172.21.15.97:0/4178640073 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24003f070 con 0x7ff248106b40 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.310+0000 7ff245600640 1 -- 172.21.15.97:0/4178640073 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff240031b90 con 0x7ff248106b40 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.310+0000 7ff245600640 1 -- 172.21.15.97:0/4178640073 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240035030 con 0x7ff248106b40 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.310+0000 7ff24e000640 1 -- 172.21.15.97:0/4178640073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24819d5b0 con 0x7ff248106b40 2024-07-26T07:52:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.311+0000 7ff24e000640 1 -- 172.21.15.97:0/4178640073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2481a3d00 con 0x7ff248106b40 2024-07-26T07:52:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.314+0000 7ff256603740 1 Processor -- start 2024-07-26T07:52:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.314+0000 7ff245600640 1 -- 172.21.15.97:0/4178640073 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7ff240042070 con 0x7ff248106b40 2024-07-26T07:52:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.315+0000 7ff256603740 1 -- start start 2024-07-26T07:52:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.315+0000 7ff256603740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565edc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.315+0000 7ff256603740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 0x55a565edc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.315+0000 7ff256603740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b154d0 0x55a565edd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.315+0000 7ff256603740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a565e4e270 con 0x55a565b154d0 2024-07-26T07:52:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.315+0000 7ff256603740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a565e3d770 con 0x55a565b361b0 2024-07-26T07:52:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.315+0000 7ff256603740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a565e417b0 con 0x55a565b0d2f0 2024-07-26T07:52:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b154d0 0x55a565edd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b154d0 0x55a565edd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33146/0 (socket says 172.21.15.97:33146) 2024-07-26T07:52:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 -- 172.21.15.97:0/4148177527 learned_addr learned my addr 172.21.15.97:0/4148177527 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565edc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff24cc00640 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 0x55a565edc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 -- 172.21.15.97:0/4148177527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 msgr2=0x55a565edc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 0x55a565edc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 -- 172.21.15.97:0/4148177527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 msgr2=0x55a565edc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565edc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 -- 172.21.15.97:0/4148177527 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a565ba1790 con 0x55a565b154d0 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.316+0000 7ff247e00640 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b154d0 0x55a565edd280 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff23000cf10 tx=0x7ff23000c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18ec8e3c41635c3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.317+0000 7ff23dc00640 1 -- 172.21.15.97:0/4148177527 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230019020 con 0x55a565b154d0 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.317+0000 7ff23dc00640 1 -- 172.21.15.97:0/4148177527 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff23000fcf0 con 0x55a565b154d0 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.317+0000 7ff23dc00640 1 -- 172.21.15.97:0/4148177527 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230003970 con 0x55a565b154d0 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.317+0000 7ff245600640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff22c0b3ce0 0x7ff22c0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.317+0000 7ff245600640 1 -- 172.21.15.97:0/4178640073 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff240104710 con 0x7ff248106b40 2024-07-26T07:52:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.317+0000 7ff24cc00640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff22c0b3ce0 0x7ff22c0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 -- 172.21.15.97:0/4148177527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b154d0 msgr2=0x55a565edd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b154d0 0x55a565edd280 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff23000cf10 tx=0x7ff23000c5a0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 -- 172.21.15.97:0/4148177527 shutdown_connections 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b154d0 0x55a565edd280 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 0x55a565edc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 --2- 172.21.15.97:0/4148177527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565edc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 -- 172.21.15.97:0/4148177527 >> 172.21.15.97:0/4148177527 conn(0x55a565b2a300 msgr2=0x55a565ed7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff24cc00640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff22c0b3ce0 0x7ff22c0b61f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff238005fd0 tx=0x7ff238009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 -- 172.21.15.97:0/4148177527 shutdown_connections 2024-07-26T07:52:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.318+0000 7ff256603740 1 -- 172.21.15.97:0/4148177527 wait complete. 2024-07-26T07:52:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 1 -- start start 2024-07-26T07:52:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 1 Processor -- start 2024-07-26T07:52:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 0x55a565ee6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565ee6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a565d77ab0 con 0x55a565b0d2f0 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55a565ddad50 con 0x55a565b361b0 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a565eddda0 con 0x55a565b154d0 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.319+0000 7ff256603740 10 client.0 _ll_register_callbacks cb 0x55a565b4da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33150/0 (socket says 172.21.15.97:33150) 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 -- 172.21.15.97:0/3792861071 learned_addr learned my addr 172.21.15.97:0/3792861071 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24d600640 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565ee6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff247e00640 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 0x55a565ee6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 -- 172.21.15.97:0/3792861071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 msgr2=0x55a565ee6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 0x55a565ee6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 -- 172.21.15.97:0/3792861071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 msgr2=0x55a565ee6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565ee6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 -- 172.21.15.97:0/3792861071 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a565c1a620 con 0x55a565b0d2f0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff24cc00640 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee5ba0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff238005f40 tx=0x7ff238004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d660b40774174d0e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.320+0000 7ff236000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.321+0000 7ff236000640 1 -- 172.21.15.97:0/3792861071 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238004d20 con 0x55a565b0d2f0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.321+0000 7ff236000640 1 -- 172.21.15.97:0/3792861071 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff23801d710 con 0x55a565b0d2f0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.321+0000 7ff256603740 1 -- 172.21.15.97:0/3792861071 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff23000fcf0 con 0x55a565b0d2f0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.321+0000 7ff236000640 1 -- 172.21.15.97:0/3792861071 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff23801d950 con 0x55a565b0d2f0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.321+0000 7ff256603740 10 client.15444 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.321+0000 7ff256603740 1 -- 172.21.15.97:0/3792861071 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a565eefdf0 con 0x55a565b0d2f0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.321+0000 7ff236000640 1 -- 172.21.15.97:0/3792861071 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff23000fcf0 con 0x55a565b0d2f0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.322+0000 7ff236000640 1 -- 172.21.15.97:0/3792861071 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff23800f040 con 0x55a565b0d2f0 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.322+0000 7ff256603740 10 client.15444 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.322+0000 7ff256603740 10 client.15444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:52:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.322+0000 7ff256603740 20 client.15444 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.322+0000 7ff256603740 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a565af4d80 0x55a565ef3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.322+0000 7ff256603740 4 client.15444 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.322+0000 7ff256603740 1 -- 172.21.15.97:0/3792861071 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a565e501c0 con 0x55a565af4d80 2024-07-26T07:52:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.323+0000 7ff247e00640 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a565af4d80 0x55a565ef3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.323+0000 7ff247e00640 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a565af4d80 0x55a565ef3530 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a029ae83a06d193 server_cookie=df89bbec18fdcda6 in_seq=0 out_seq=0 2024-07-26T07:52:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.323+0000 7ff236000640 10 client.15444 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff236000640 1 -- 172.21.15.97:0/3792861071 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a565e501c0 con 0x55a565af4d80 2024-07-26T07:52:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff236000640 10 client.15444 handle_command_reply: tid=0 2024-07-26T07:52:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 client.15444 shutdown 2024-07-26T07:52:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 -- 172.21.15.97:0/3792861071 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 msgr2=0x55a565ee5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee5ba0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff238005f40 tx=0x7ff238004a40 comp rx=0 tx=0).stop 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 -- 172.21.15.97:0/3792861071 shutdown_connections 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a565af4d80 0x55a565ef3530 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565ee6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 0x55a565ee6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 --2- 172.21.15.97:0/3792861071 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee5ba0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 -- 172.21.15.97:0/3792861071 >> 172.21.15.97:0/3792861071 conn(0x55a565b2a300 msgr2=0x55a565ed60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff236000640 0 client.15444 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 -- 172.21.15.97:0/3792861071 shutdown_connections 2024-07-26T07:52:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.326+0000 7ff256603740 1 -- 172.21.15.97:0/3792861071 wait complete. 2024-07-26T07:52:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.327+0000 7ff256603740 20 client.15444 trim_cache size 0 max 16384 2024-07-26T07:52:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.344+0000 7ff256603740 1 Processor -- start 2024-07-26T07:52:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.344+0000 7ff256603740 1 -- start start 2024-07-26T07:52:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.344+0000 7ff256603740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.344+0000 7ff256603740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 0x55a565ee4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.344+0000 7ff256603740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565ee4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.344+0000 7ff256603740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481acad0 con 0x55a565b0d2f0 2024-07-26T07:52:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.344+0000 7ff256603740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481ae610 con 0x55a565b361b0 2024-07-26T07:52:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.344+0000 7ff256603740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481ad1a0 con 0x55a565b154d0 2024-07-26T07:52:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33158/0 (socket says 172.21.15.97:33158) 2024-07-26T07:52:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 -- 172.21.15.97:0/654373801 learned_addr learned my addr 172.21.15.97:0/654373801 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 -- 172.21.15.97:0/654373801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 msgr2=0x55a565ee4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 --2- 172.21.15.97:0/654373801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 0x55a565ee4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 -- 172.21.15.97:0/654373801 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 msgr2=0x55a565ee4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 --2- 172.21.15.97:0/654373801 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565ee4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 -- 172.21.15.97:0/654373801 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a565ed8280 con 0x55a565b0d2f0 2024-07-26T07:52:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.345+0000 7ff24cc00640 1 --2- 172.21.15.97:0/654373801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee0b30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff238009e90 tx=0x7ff23801d4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfe279959bdbe89 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.346+0000 7ff23d200640 1 -- 172.21.15.97:0/654373801 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23803e070 con 0x55a565b0d2f0 2024-07-26T07:52:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.346+0000 7ff23d200640 1 -- 172.21.15.97:0/654373801 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2380303a0 con 0x55a565b0d2f0 2024-07-26T07:52:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.346+0000 7ff23d200640 1 -- 172.21.15.97:0/654373801 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238035050 con 0x55a565b0d2f0 2024-07-26T07:52:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 -- 172.21.15.97:0/654373801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 msgr2=0x55a565ee0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 --2- 172.21.15.97:0/654373801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee0b30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff238009e90 tx=0x7ff23801d4c0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 -- 172.21.15.97:0/654373801 shutdown_connections 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 --2- 172.21.15.97:0/654373801 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b361b0 0x55a565ee4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 --2- 172.21.15.97:0/654373801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b154d0 0x55a565ee4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 --2- 172.21.15.97:0/654373801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b0d2f0 0x55a565ee0b30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 -- 172.21.15.97:0/654373801 >> 172.21.15.97:0/654373801 conn(0x55a565b2a300 msgr2=0x55a565ee5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 -- 172.21.15.97:0/654373801 shutdown_connections 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 -- 172.21.15.97:0/654373801 wait complete. 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 -- start start 2024-07-26T07:52:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.347+0000 7ff256603740 1 Processor -- start 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff256603740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 0x55a565eeac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff256603740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b154d0 0x55a565eeb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff256603740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b361b0 0x55a565eebe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff256603740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481adf40 con 0x55a565b361b0 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff256603740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481ad870 con 0x55a565b154d0 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff256603740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481aaab0 con 0x55a565b0d2f0 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff256603740 10 client.0 _ll_register_callbacks cb 0x7ff238009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff24d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b361b0 0x55a565eebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff24d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b361b0 0x55a565eebe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33170/0 (socket says 172.21.15.97:33170) 2024-07-26T07:52:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff24d600640 1 -- 172.21.15.97:0/921130938 learned_addr learned my addr 172.21.15.97:0/921130938 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.348+0000 7ff24cc00640 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 0x55a565eeac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff247e00640 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b154d0 0x55a565eeb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff24d600640 1 -- 172.21.15.97:0/921130938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 msgr2=0x55a565eeac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff24d600640 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 0x55a565eeac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff24d600640 1 -- 172.21.15.97:0/921130938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b154d0 msgr2=0x55a565eeb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff24d600640 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b154d0 0x55a565eeb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff24d600640 1 -- 172.21.15.97:0/921130938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a565eec590 con 0x55a565b361b0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff24d600640 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b361b0 0x55a565eebe30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff240102030 tx=0x7ff2400cac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7154d40a2b082562 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff236000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff236000640 1 -- 172.21.15.97:0/921130938 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240049030 con 0x55a565b361b0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff236000640 1 -- 172.21.15.97:0/921130938 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff24003b460 con 0x55a565b361b0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff256603740 1 -- 172.21.15.97:0/921130938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2380303a0 con 0x55a565b361b0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff236000640 1 -- 172.21.15.97:0/921130938 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff2400cb440 con 0x55a565b361b0 2024-07-26T07:52:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff256603740 10 client.15450 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.349+0000 7ff256603740 1 -- 172.21.15.97:0/921130938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a565eed950 con 0x55a565b361b0 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.350+0000 7ff236000640 1 -- 172.21.15.97:0/921130938 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2380303a0 con 0x55a565b361b0 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.350+0000 7ff236000640 1 -- 172.21.15.97:0/921130938 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff2400cb730 con 0x55a565b361b0 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.350+0000 7ff256603740 10 client.15450 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.350+0000 7ff256603740 10 client.15450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.350+0000 7ff256603740 20 client.15450 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.350+0000 7ff256603740 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a565aca1c0 0x55a565f5df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.350+0000 7ff256603740 4 client.15450 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.350+0000 7ff256603740 1 -- 172.21.15.97:0/921130938 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a565e4bc60 con 0x55a565aca1c0 2024-07-26T07:52:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.351+0000 7ff24cc00640 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a565aca1c0 0x55a565f5df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.352+0000 7ff236000640 10 client.15450 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.352+0000 7ff24cc00640 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a565aca1c0 0x55a565f5df20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e41c3082dcfc8957 server_cookie=8f9c922ba210e1ed in_seq=0 out_seq=0 2024-07-26T07:52:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.380+0000 7ff236000640 1 -- 172.21.15.97:0/921130938 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a565e4bc60 con 0x55a565aca1c0 2024-07-26T07:52:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.380+0000 7ff236000640 10 client.15450 handle_command_reply: tid=0 2024-07-26T07:52:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.380+0000 7ff256603740 1 client.15450 shutdown 2024-07-26T07:52:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 -- 172.21.15.97:0/921130938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b361b0 msgr2=0x55a565eebe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b361b0 0x55a565eebe30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff240102030 tx=0x7ff2400cac70 comp rx=0 tx=0).stop 2024-07-26T07:52:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 -- 172.21.15.97:0/921130938 shutdown_connections 2024-07-26T07:52:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55a565aca1c0 0x55a565f5df20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a565b361b0 0x55a565eebe30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a565b154d0 0x55a565eeb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 --2- 172.21.15.97:0/921130938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a565b0d2f0 0x55a565eeac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff236000640 0 client.15450 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 -- 172.21.15.97:0/921130938 >> 172.21.15.97:0/921130938 conn(0x55a565b2a300 msgr2=0x55a565ee56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 -- 172.21.15.97:0/921130938 shutdown_connections 2024-07-26T07:52:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.381+0000 7ff256603740 1 -- 172.21.15.97:0/921130938 wait complete. 2024-07-26T07:52:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.382+0000 7ff256603740 20 client.15450 trim_cache size 0 max 16384 2024-07-26T07:52:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:52:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:52:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:52:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 -- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff22c0b3ce0 msgr2=0x7ff22c0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff22c0b3ce0 0x7ff22c0b61f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff238005fd0 tx=0x7ff238009680 comp rx=0 tx=0).stop 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 -- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 msgr2=0x7ff24819cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 0x7ff24819cb00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff24002a930 tx=0x7ff240003b90 comp rx=0 tx=0).stop 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 -- 172.21.15.97:0/4178640073 shutdown_connections 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff22c0b3ce0 0x7ff22c0b61f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff248106b40 0x7ff24819cb00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff2481061a0 0x7ff24819bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 --2- 172.21.15.97:0/4178640073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff248105120 0x7ff24819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 -- 172.21.15.97:0/4178640073 >> 172.21.15.97:0/4178640073 conn(0x7ff248100a80 msgr2=0x7ff2481094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 -- 172.21.15.97:0/4178640073 shutdown_connections 2024-07-26T07:52:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:08.383+0000 7ff24e000640 1 -- 172.21.15.97:0/4178640073 wait complete. 2024-07-26T07:52:08.418 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:08 smithi097 ceph-mon[32231]: pgmap v520: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-07-26T07:52:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:08 smithi115 ceph-mon[35364]: pgmap v520: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-07-26T07:52:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:08 smithi181 ceph-mon[35869]: pgmap v520: 97 pgs: 97 active+clean; 659 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-07-26T07:52:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.277+0000 7fc973200640 1 -- 172.21.15.97:0/1176453897 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc974109670 msgr2=0x7fc97410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.277+0000 7fc973200640 1 --2- 172.21.15.97:0/1176453897 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc974109670 0x7fc97410fea0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009ca0 tx=0x7fc96c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.277+0000 7fc973200640 1 -- 172.21.15.97:0/1176453897 shutdown_connections 2024-07-26T07:52:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.277+0000 7fc973200640 1 --2- 172.21.15.97:0/1176453897 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc974109670 0x7fc97410fea0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.277+0000 7fc973200640 1 --2- 172.21.15.97:0/1176453897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc974108d20 0x7fc974109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.277+0000 7fc973200640 1 --2- 172.21.15.97:0/1176453897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9741033e0 0x7fc9741037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.277+0000 7fc973200640 1 -- 172.21.15.97:0/1176453897 >> 172.21.15.97:0/1176453897 conn(0x7fc9740fee70 msgr2=0x7fc9741012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.278+0000 7fc973200640 1 -- 172.21.15.97:0/1176453897 shutdown_connections 2024-07-26T07:52:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.278+0000 7fc973200640 1 -- 172.21.15.97:0/1176453897 wait complete. 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc973200640 1 Processor -- start 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc973200640 1 -- start start 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc973200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9741033e0 0x7fc97419be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc973200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc974108d20 0x7fc97419c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc973200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc974109670 0x7fc9741a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc973200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc974113240 con 0x7fc9741033e0 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc973200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741130c0 con 0x7fc974108d20 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc973200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741133c0 con 0x7fc974109670 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.279+0000 7fc971e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9741033e0 0x7fc97419be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:09.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc971e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9741033e0 0x7fc97419be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33196/0 (socket says 172.21.15.97:33196) 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc971e00640 1 -- 172.21.15.97:0/2292263229 learned_addr learned my addr 172.21.15.97:0/2292263229 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc971400640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc974108d20 0x7fc97419c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc972800640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc974109670 0x7fc9741a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc971e00640 1 -- 172.21.15.97:0/2292263229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc974109670 msgr2=0x7fc9741a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc971e00640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc974109670 0x7fc9741a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc971e00640 1 -- 172.21.15.97:0/2292263229 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc974108d20 msgr2=0x7fc97419c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc971e00640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc974108d20 0x7fc97419c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.280+0000 7fc971e00640 1 -- 172.21.15.97:0/2292263229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9741a1d80 con 0x7fc9741033e0 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.281+0000 7fc971e00640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9741033e0 0x7fc97419be10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc96400e9f0 tx=0x7fc96400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.281+0000 7fc96aa00640 1 -- 172.21.15.97:0/2292263229 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96400cd80 con 0x7fc9741033e0 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.281+0000 7fc96aa00640 1 -- 172.21.15.97:0/2292263229 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9640054b0 con 0x7fc9741033e0 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.281+0000 7fc96aa00640 1 -- 172.21.15.97:0/2292263229 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964010430 con 0x7fc9741033e0 2024-07-26T07:52:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.281+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9741a2070 con 0x7fc9741033e0 2024-07-26T07:52:09.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.281+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9741a87d0 con 0x7fc9741033e0 2024-07-26T07:52:09.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.282+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9741037f0 con 0x7fc9741033e0 2024-07-26T07:52:09.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.285+0000 7fc96aa00640 1 -- 172.21.15.97:0/2292263229 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7fc9640046c0 con 0x7fc9741033e0 2024-07-26T07:52:09.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.286+0000 7fc96aa00640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc9580b3c60 0x7fc9580b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:09.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.286+0000 7fc96aa00640 1 -- 172.21.15.97:0/2292263229 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fc9640e4260 con 0x7fc9741033e0 2024-07-26T07:52:09.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.286+0000 7fc971400640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc9580b3c60 0x7fc9580b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:09.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.287+0000 7fc971400640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc9580b3c60 0x7fc9580b6170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc974108780 tx=0x7fc96000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:09.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.290+0000 7fc96aa00640 1 -- 172.21.15.97:0/2292263229 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc9640ab690 con 0x7fc9741033e0 2024-07-26T07:52:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:09 smithi115 ceph-mon[35364]: pgmap v521: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-07-26T07:52:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:09 smithi181 ceph-mon[35869]: pgmap v521: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-07-26T07:52:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:09 smithi097 ceph-mon[32231]: pgmap v521: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-07-26T07:52:09.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.527+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc97419cf40 con 0x7fc9741033e0 2024-07-26T07:52:09.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.529+0000 7fc96aa00640 1 -- 172.21.15.97:0/2292263229 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11990 (secure 0 0 0) 0x7fc9640aaf90 con 0x7fc9741033e0 2024-07-26T07:52:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:52:09.530 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"btime":"2024-07-26T07:51:41:000369+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:51:40.647454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:52:09.531 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-07-26T07:52:09.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc9580b3c60 msgr2=0x7fc9580b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:09.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc9580b3c60 0x7fc9580b6170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc974108780 tx=0x7fc96000b410 comp rx=0 tx=0).stop 2024-07-26T07:52:09.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9741033e0 msgr2=0x7fc97419be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:09.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9741033e0 0x7fc97419be10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc96400e9f0 tx=0x7fc96400edf0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 shutdown_connections 2024-07-26T07:52:09.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc9580b3c60 0x7fc9580b6170 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc974109670 0x7fc9741a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc974108d20 0x7fc97419c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 --2- 172.21.15.97:0/2292263229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9741033e0 0x7fc97419be10 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:09.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 >> 172.21.15.97:0/2292263229 conn(0x7fc9740fee70 msgr2=0x7fc97410e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:09.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.532+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 shutdown_connections 2024-07-26T07:52:09.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:09.533+0000 7fc973200640 1 -- 172.21.15.97:0/2292263229 wait complete. 2024-07-26T07:52:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:10 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2292263229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:52:10.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T07:47:48.482950+0000', 'modified': '2024-07-26T07:51:40.647454+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 34583, 'mds_1': 34556, 'mds_2': 34535, 'mds_3': 34517, 'mds_4': 24562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14748': {'gid': 14748, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6839/4174464953', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6838', 'nonce': 4174464953}, {'type': 'v1', 'addr': '172.21.15.97:6839', 'nonce': 4174464953}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14760': {'gid': 14760, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6841/4022202837', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6840', 'nonce': 4022202837}, {'type': 'v1', 'addr': '172.21.15.97:6841', 'nonce': 4022202837}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6833/1511551515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6832', 'nonce': 1511551515}, {'type': 'v1', 'addr': '172.21.15.181:6833', 'nonce': 1511551515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6835/4001900111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 4001900111}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 4001900111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6837/3053625422', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 3053625422}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 3053625422}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'i', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.181:6839/4258683328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 4258683328}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 4258683328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34517': {'gid': 34517, 'name': 'k', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.115:6833/1028321962', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6832', 'nonce': 1028321962}, {'type': 'v1', 'addr': '172.21.15.115:6833', 'nonce': 1028321962}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34535': {'gid': 34535, 'name': 'e', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.115:6835/4018170367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6834', 'nonce': 4018170367}, {'type': 'v1', 'addr': '172.21.15.115:6835', 'nonce': 4018170367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34556': {'gid': 34556, 'name': 'h', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.115:6837/3478625369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6836', 'nonce': 3478625369}, {'type': 'v1', 'addr': '172.21.15.115:6837', 'nonce': 3478625369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34583': {'gid': 34583, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.115:6839/1903638498', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6838', 'nonce': 1903638498}, {'type': 'v1', 'addr': '172.21.15.115:6839', 'nonce': 1903638498}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34583, 'qdb_cluster': [34583, 34535, 34556, 34517, 24562]} 2024-07-26T07:52:10.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T07:52:10.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.i status 2024-07-26T07:52:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:10 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2292263229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:52:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:10 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2292263229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:52:10.862 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:11 smithi097 ceph-mon[32231]: pgmap v522: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T07:52:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:11 smithi115 ceph-mon[35364]: pgmap v522: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T07:52:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:11 smithi181 ceph-mon[35869]: pgmap v522: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.880+0000 7fc69be00640 1 -- 172.21.15.97:0/1893071278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c1099f0 msgr2=0x7fc69c109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.880+0000 7fc69be00640 1 --2- 172.21.15.97:0/1893071278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c1099f0 0x7fc69c109e00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009900 tx=0x7fc68c02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.880+0000 7fc69be00640 1 -- 172.21.15.97:0/1893071278 shutdown_connections 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.880+0000 7fc69be00640 1 --2- 172.21.15.97:0/1893071278 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc69c10fc80 0x7fc69c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.880+0000 7fc69be00640 1 --2- 172.21.15.97:0/1893071278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc69c10f330 0x7fc69c10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.880+0000 7fc69be00640 1 --2- 172.21.15.97:0/1893071278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c1099f0 0x7fc69c109e00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.880+0000 7fc69be00640 1 -- 172.21.15.97:0/1893071278 >> 172.21.15.97:0/1893071278 conn(0x7fc69c078ff0 msgr2=0x7fc69c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.880+0000 7fc69be00640 1 -- 172.21.15.97:0/1893071278 shutdown_connections 2024-07-26T07:52:12.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.881+0000 7fc69be00640 1 -- 172.21.15.97:0/1893071278 wait complete. 2024-07-26T07:52:12.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.882+0000 7fc69be00640 1 Processor -- start 2024-07-26T07:52:12.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.882+0000 7fc69be00640 1 -- start start 2024-07-26T07:52:12.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.882+0000 7fc69be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc69c1099f0 0x7fc69c1a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.882+0000 7fc69be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c10f330 0x7fc69c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.882+0000 7fc69be00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc69c10fc80 0x7fc69c1a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.882+0000 7fc69be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1197f0 con 0x7fc69c10f330 2024-07-26T07:52:12.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.882+0000 7fc69be00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c119670 con 0x7fc69c10fc80 2024-07-26T07:52:12.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.882+0000 7fc69be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c119970 con 0x7fc69c1099f0 2024-07-26T07:52:12.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c10f330 0x7fc69c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c10f330 0x7fc69c1a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33218/0 (socket says 172.21.15.97:33218) 2024-07-26T07:52:12.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69a000640 1 -- 172.21.15.97:0/800751152 learned_addr learned my addr 172.21.15.97:0/800751152 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:12.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69aa00640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc69c1099f0 0x7fc69c1a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69b400640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc69c10fc80 0x7fc69c1a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69a000640 1 -- 172.21.15.97:0/800751152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc69c1099f0 msgr2=0x7fc69c1a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.885 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69a000640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc69c1099f0 0x7fc69c1a02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69a000640 1 -- 172.21.15.97:0/800751152 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc69c10fc80 msgr2=0x7fc69c1a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69a000640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc69c10fc80 0x7fc69c1a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69a000640 1 -- 172.21.15.97:0/800751152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c1a6150 con 0x7fc69c10f330 2024-07-26T07:52:12.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.883+0000 7fc69aa00640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc69c1099f0 0x7fc69c1a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:52:12.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.884+0000 7fc69a000640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c10f330 0x7fc69c1a0840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d750 tx=0x7fc68800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:12.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.884+0000 7fc693400640 1 -- 172.21.15.97:0/800751152 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688017070 con 0x7fc69c10f330 2024-07-26T07:52:12.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.884+0000 7fc693400640 1 -- 172.21.15.97:0/800751152 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6880054b0 con 0x7fc69c10f330 2024-07-26T07:52:12.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.884+0000 7fc69be00640 1 -- 172.21.15.97:0/800751152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c1a6440 con 0x7fc69c10f330 2024-07-26T07:52:12.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.884+0000 7fc693400640 1 -- 172.21.15.97:0/800751152 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688004bc0 con 0x7fc69c10f330 2024-07-26T07:52:12.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.884+0000 7fc69be00640 1 -- 172.21.15.97:0/800751152 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c1acc60 con 0x7fc69c10f330 2024-07-26T07:52:12.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.888+0000 7fc6a843f740 1 Processor -- start 2024-07-26T07:52:12.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.889+0000 7fc6a843f740 1 -- start start 2024-07-26T07:52:12.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.889+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487d9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.889+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e4840b0a0 0x555e487d9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.889+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 0x555e487da200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.889+0000 7fc6a843f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555e4874f970 con 0x555e48433f60 2024-07-26T07:52:12.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.889+0000 7fc6a843f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555e4873ee70 con 0x555e48413280 2024-07-26T07:52:12.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.889+0000 7fc6a843f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555e48742eb0 con 0x555e4840b0a0 2024-07-26T07:52:12.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.889+0000 7fc69b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487d9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487d9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33226/0 (socket says 172.21.15.97:33226) 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69b400640 1 -- 172.21.15.97:0/736931673 learned_addr learned my addr 172.21.15.97:0/736931673 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69aa00640 1 --2- 172.21.15.97:0/736931673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e4840b0a0 0x555e487d9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69b400640 1 -- 172.21.15.97:0/736931673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e4840b0a0 msgr2=0x555e487d9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69b400640 1 --2- 172.21.15.97:0/736931673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e4840b0a0 0x555e487d9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69b400640 1 -- 172.21.15.97:0/736931673 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 msgr2=0x555e487da200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69b400640 1 --2- 172.21.15.97:0/736931673 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 0x555e487da200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69b400640 1 -- 172.21.15.97:0/736931673 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e486e6110 con 0x555e48433f60 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc693400640 1 -- 172.21.15.97:0/800751152 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7fc688009c30 con 0x7fc69c10f330 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.890+0000 7fc69b400640 1 --2- 172.21.15.97:0/736931673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487d9780 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc694007e50 tx=0x7fc69400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916dec33c9a743c0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.891+0000 7fc687e00640 1 -- 172.21.15.97:0/736931673 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694014070 con 0x555e48433f60 2024-07-26T07:52:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.891+0000 7fc687e00640 1 -- 172.21.15.97:0/736931673 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc69400ee90 con 0x555e48433f60 2024-07-26T07:52:12.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.891+0000 7fc687e00640 1 -- 172.21.15.97:0/736931673 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694005330 con 0x555e48433f60 2024-07-26T07:52:12.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 -- 172.21.15.97:0/736931673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 msgr2=0x555e487d9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 --2- 172.21.15.97:0/736931673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487d9780 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc694007e50 tx=0x7fc69400e720 comp rx=0 tx=0).stop 2024-07-26T07:52:12.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc693400640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc6800b3d10 0x7fc6800b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 -- 172.21.15.97:0/736931673 shutdown_connections 2024-07-26T07:52:12.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 --2- 172.21.15.97:0/736931673 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 0x555e487da200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 --2- 172.21.15.97:0/736931673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e4840b0a0 0x555e487d9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 --2- 172.21.15.97:0/736931673 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487d9780 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 -- 172.21.15.97:0/736931673 >> 172.21.15.97:0/736931673 conn(0x555e48428350 msgr2=0x555e487d7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 -- 172.21.15.97:0/736931673 shutdown_connections 2024-07-26T07:52:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc693400640 1 -- 172.21.15.97:0/800751152 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fc6880e3520 con 0x7fc69c10f330 2024-07-26T07:52:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc6a843f740 1 -- 172.21.15.97:0/736931673 wait complete. 2024-07-26T07:52:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.892+0000 7fc69b400640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc6800b3d10 0x7fc6800b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.893+0000 7fc6a843f740 1 -- start start 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.893+0000 7fc69b400640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc6800b3d10 0x7fc6800b6220 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc694010e50 tx=0x7fc694000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.893+0000 7fc6a843f740 1 Processor -- start 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 0x555e487e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48413280 0x555e487e5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487f1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc6a843f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555e485a25f0 con 0x555e48433f60 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc6a843f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555e486db540 con 0x555e4840b0a0 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc6a843f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555e487e7f60 con 0x555e48413280 2024-07-26T07:52:12.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc6a843f740 10 client.0 _ll_register_callbacks cb 0x555e486eaf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc69b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487f1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc69b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487f1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33230/0 (socket says 172.21.15.97:33230) 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc69b400640 1 -- 172.21.15.97:0/1410545963 learned_addr learned my addr 172.21.15.97:0/1410545963 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.894+0000 7fc69a000640 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48413280 0x555e487e5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.895+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 0x555e487e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.895+0000 7fc69b400640 1 -- 172.21.15.97:0/1410545963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48413280 msgr2=0x555e487e5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.895+0000 7fc69b400640 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48413280 0x555e487e5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.895+0000 7fc69b400640 1 -- 172.21.15.97:0/1410545963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 msgr2=0x555e487e52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.895+0000 7fc69b400640 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 0x555e487e52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.895+0000 7fc69b400640 1 -- 172.21.15.97:0/1410545963 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e4849f540 con 0x555e48433f60 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.895+0000 7fc69b400640 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487f1280 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc694010670 tx=0x7fc69400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42ca3ba6cb39002 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.895+0000 7fc684c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.896+0000 7fc684c00640 1 -- 172.21.15.97:0/1410545963 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694005820 con 0x555e48433f60 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.896+0000 7fc684c00640 1 -- 172.21.15.97:0/1410545963 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fc694003970 con 0x555e48433f60 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.896+0000 7fc6a843f740 1 -- 172.21.15.97:0/1410545963 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc69400ee90 con 0x555e48433f60 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.896+0000 7fc684c00640 1 -- 172.21.15.97:0/1410545963 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc6940059c0 con 0x555e48433f60 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.896+0000 7fc6a843f740 10 client.15468 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.896+0000 7fc6a843f740 1 -- 172.21.15.97:0/1410545963 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555e487f3ef0 con 0x555e48433f60 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.896+0000 7fc684c00640 1 -- 172.21.15.97:0/1410545963 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc69400ee90 con 0x555e48433f60 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.897+0000 7fc684c00640 1 -- 172.21.15.97:0/1410545963 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc69402e020 con 0x555e48433f60 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.897+0000 7fc6a843f740 10 client.15468 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:12.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.897+0000 7fc6a843f740 10 client.15468 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T07:52:12.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.897+0000 7fc6a843f740 20 client.15468 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:12.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.897+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x555e483f28e0 0x555e487f7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.897+0000 7fc6a843f740 4 client.15468 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:12.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.897+0000 7fc6a843f740 1 -- 172.21.15.97:0/1410545963 --> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e487518c0 con 0x555e483f28e0 2024-07-26T07:52:12.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.898+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x555e483f28e0 0x555e487f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.899+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x555e483f28e0 0x555e487f7730 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4d41718a52060761 server_cookie=782215a92af1c2f8 in_seq=0 out_seq=0 2024-07-26T07:52:12.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.899+0000 7fc684c00640 10 client.15468 ms_handle_connect on v2:172.21.15.181:6838/4258683328 2024-07-26T07:52:12.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc684c00640 1 -- 172.21.15.97:0/1410545963 <== mds.4 v2:172.21.15.181:6838/4258683328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555e487518c0 con 0x555e483f28e0 2024-07-26T07:52:12.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc684c00640 10 client.15468 handle_command_reply: tid=0 2024-07-26T07:52:12.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 client.15468 shutdown 2024-07-26T07:52:12.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 -- 172.21.15.97:0/1410545963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 msgr2=0x555e487f1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487f1280 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc694010670 tx=0x7fc69400e720 comp rx=0 tx=0).stop 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 -- 172.21.15.97:0/1410545963 shutdown_connections 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x555e483f28e0 0x555e487f7730 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48433f60 0x555e487f1280 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48413280 0x555e487e5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1410545963 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 0x555e487e52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc6a843f740 1 -- 172.21.15.97:0/1410545963 >> 172.21.15.97:0/1410545963 conn(0x555e48428350 msgr2=0x555e487dc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.901+0000 7fc684c00640 0 client.15468 ms_handle_reset on v2:172.21.15.181:6838/4258683328 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.902+0000 7fc6a843f740 1 -- 172.21.15.97:0/1410545963 shutdown_connections 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.902+0000 7fc6a843f740 1 -- 172.21.15.97:0/1410545963 wait complete. 2024-07-26T07:52:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.902+0000 7fc6a843f740 20 client.15468 trim_cache size 0 max 16384 2024-07-26T07:52:12.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.922+0000 7fc6a843f740 1 Processor -- start 2024-07-26T07:52:12.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.922+0000 7fc6a843f740 1 -- start start 2024-07-26T07:52:12.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.922+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e4840b0a0 0x555e487f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.922+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 0x555e487f1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.922+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487e3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.922+0000 7fc6a843f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555e487f5a70 con 0x555e4840b0a0 2024-07-26T07:52:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.922+0000 7fc6a843f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555e487f6140 con 0x555e48413280 2024-07-26T07:52:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.922+0000 7fc6a843f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555e487f6810 con 0x555e48433f60 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.923+0000 7fc69aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e4840b0a0 0x555e487f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.923+0000 7fc69aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e4840b0a0 0x555e487f11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33244/0 (socket says 172.21.15.97:33244) 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.923+0000 7fc69aa00640 1 -- 172.21.15.97:0/1618761236 learned_addr learned my addr 172.21.15.97:0/1618761236 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.923+0000 7fc69a000640 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 0x555e487f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.923+0000 7fc69b400640 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487e3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.924+0000 7fc69aa00640 1 -- 172.21.15.97:0/1618761236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 msgr2=0x555e487e3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.924+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487e3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.924+0000 7fc69aa00640 1 -- 172.21.15.97:0/1618761236 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 msgr2=0x555e487f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.924+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 0x555e487f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.924+0000 7fc69aa00640 1 -- 172.21.15.97:0/1618761236 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e487d27f0 con 0x555e4840b0a0 2024-07-26T07:52:12.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.924+0000 7fc69b400640 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487e3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:52:12.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.924+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e4840b0a0 0x555e487f11f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0096c0 tx=0x7fc68c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0d4f4d9f6e4770 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.924+0000 7fc687400640 1 -- 172.21.15.97:0/1618761236 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c035040 con 0x555e4840b0a0 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.925+0000 7fc687400640 1 -- 172.21.15.97:0/1618761236 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc68c049070 con 0x555e4840b0a0 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.925+0000 7fc687400640 1 -- 172.21.15.97:0/1618761236 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c0046c0 con 0x555e4840b0a0 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 -- 172.21.15.97:0/1618761236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e4840b0a0 msgr2=0x555e487f11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e4840b0a0 0x555e487f11f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0096c0 tx=0x7fc68c005a20 comp rx=0 tx=0).stop 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 -- 172.21.15.97:0/1618761236 shutdown_connections 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487e3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e48413280 0x555e487f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1618761236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e4840b0a0 0x555e487f11f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 -- 172.21.15.97:0/1618761236 >> 172.21.15.97:0/1618761236 conn(0x555e48428350 msgr2=0x555e487e0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 -- 172.21.15.97:0/1618761236 shutdown_connections 2024-07-26T07:52:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.926+0000 7fc6a843f740 1 -- 172.21.15.97:0/1618761236 wait complete. 2024-07-26T07:52:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.927+0000 7fc6a843f740 1 -- start start 2024-07-26T07:52:12.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.927+0000 7fc6a843f740 1 Processor -- start 2024-07-26T07:52:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 0x555e487f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48413280 0x555e487f90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc6a843f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487fdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc6a843f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555e487f75b0 con 0x555e48413280 2024-07-26T07:52:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc6a843f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x555e487f6ee0 con 0x555e4840b0a0 2024-07-26T07:52:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc6a843f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555e487f7c80 con 0x555e48433f60 2024-07-26T07:52:12.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc6a843f740 10 client.0 _ll_register_callbacks cb 0x7fc694004ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc69a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48413280 0x555e487f90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc69a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48413280 0x555e487f90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33246/0 (socket says 172.21.15.97:33246) 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc69a000640 1 -- 172.21.15.97:0/1564564246 learned_addr learned my addr 172.21.15.97:0/1564564246 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.928+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 0x555e487f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc69b400640 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487fdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc69a000640 1 -- 172.21.15.97:0/1564564246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 msgr2=0x555e487fdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc69a000640 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487fdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc69a000640 1 -- 172.21.15.97:0/1564564246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 msgr2=0x555e487f8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc69a000640 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 0x555e487f8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc69a000640 1 -- 172.21.15.97:0/1564564246 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e487fe5d0 con 0x555e48413280 2024-07-26T07:52:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc69a000640 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48413280 0x555e487f90f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc68800b470 tx=0x7fc6880a9ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69fa7863ce3ad05 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc684c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.929+0000 7fc684c00640 1 -- 172.21.15.97:0/1564564246 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6880c9060 con 0x555e48413280 2024-07-26T07:52:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.930+0000 7fc684c00640 1 -- 172.21.15.97:0/1564564246 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fc6880a6840 con 0x555e48413280 2024-07-26T07:52:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.930+0000 7fc6a843f740 1 -- 172.21.15.97:0/1564564246 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc68c049070 con 0x555e48413280 2024-07-26T07:52:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.930+0000 7fc684c00640 1 -- 172.21.15.97:0/1564564246 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc6880c3030 con 0x555e48413280 2024-07-26T07:52:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.930+0000 7fc6a843f740 10 client.15474 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.930+0000 7fc6a843f740 1 -- 172.21.15.97:0/1564564246 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555e487ff9f0 con 0x555e48413280 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.930+0000 7fc684c00640 1 -- 172.21.15.97:0/1564564246 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc68c049070 con 0x555e48413280 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.930+0000 7fc684c00640 1 -- 172.21.15.97:0/1564564246 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc6880b1c60 con 0x555e48413280 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.931+0000 7fc6a843f740 10 client.15474 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.931+0000 7fc6a843f740 10 client.15474 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.931+0000 7fc6a843f740 20 client.15474 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.931+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x555e483c7c90 0x555e48863440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.931+0000 7fc6a843f740 4 client.15474 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.931+0000 7fc6a843f740 1 -- 172.21.15.97:0/1564564246 --> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] -- command(tid 0: {"prefix": "status"}) -- 0x555e4874d360 con 0x555e483c7c90 2024-07-26T07:52:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.931+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x555e483c7c90 0x555e48863440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:12.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.932+0000 7fc69aa00640 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x555e483c7c90 0x555e48863440 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7e7a88394322971a server_cookie=74b9ba1f47efab95 in_seq=0 out_seq=0 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.932+0000 7fc684c00640 10 client.15474 ms_handle_connect on v2:172.21.15.181:6838/4258683328 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.933+0000 7fc684c00640 1 -- 172.21.15.97:0/1564564246 <== mds.4 v2:172.21.15.181:6838/4258683328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555e4874d360 con 0x555e483c7c90 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.933+0000 7fc684c00640 10 client.15474 handle_command_reply: tid=0 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.933+0000 7fc6a843f740 1 client.15474 shutdown 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.933+0000 7fc6a843f740 1 -- 172.21.15.97:0/1564564246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48413280 msgr2=0x555e487f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.933+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48413280 0x555e487f90f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc68800b470 tx=0x7fc6880a9ec0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.933+0000 7fc6a843f740 1 -- 172.21.15.97:0/1564564246 shutdown_connections 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.933+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.181:6838/4258683328,v1:172.21.15.181:6839/4258683328] conn(0x555e483c7c90 0x555e48863440 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.934+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e48433f60 0x555e487fdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.934+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555e48413280 0x555e487f90f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.934+0000 7fc684c00640 0 client.15474 ms_handle_reset on v2:172.21.15.181:6838/4258683328 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.934+0000 7fc6a843f740 1 --2- 172.21.15.97:0/1564564246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x555e4840b0a0 0x555e487f8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.934+0000 7fc6a843f740 1 -- 172.21.15.97:0/1564564246 >> 172.21.15.97:0/1564564246 conn(0x555e48428350 msgr2=0x555e487de890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:12.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.934+0000 7fc6a843f740 1 -- 172.21.15.97:0/1564564246 shutdown_connections 2024-07-26T07:52:12.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.934+0000 7fc6a843f740 1 -- 172.21.15.97:0/1564564246 wait complete. 2024-07-26T07:52:12.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.934+0000 7fc6a843f740 20 client.15474 trim_cache size 0 max 16384 2024-07-26T07:52:12.936 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:52:12.936 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:52:12.936 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 4, 2024-07-26T07:52:12.936 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24562, 2024-07-26T07:52:12.936 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 246.13556856700001, 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 86, 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 297.52508004999999 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:52:12.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 -- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc6800b3d10 msgr2=0x7fc6800b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc6800b3d10 0x7fc6800b6220 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc694010e50 tx=0x7fc694000f40 comp rx=0 tx=0).stop 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 -- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c10f330 msgr2=0x7fc69c1a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c10f330 0x7fc69c1a0840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d750 tx=0x7fc68800dc10 comp rx=0 tx=0).stop 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 -- 172.21.15.97:0/800751152 shutdown_connections 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fc6800b3d10 0x7fc6800b6220 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc69c10fc80 0x7fc69c1a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc69c10f330 0x7fc69c1a0840 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 --2- 172.21.15.97:0/800751152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc69c1099f0 0x7fc69c1a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 -- 172.21.15.97:0/800751152 >> 172.21.15.97:0/800751152 conn(0x7fc69c078ff0 msgr2=0x7fc69c079950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 -- 172.21.15.97:0/800751152 shutdown_connections 2024-07-26T07:52:12.938 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:12.935+0000 7fc69be00640 1 -- 172.21.15.97:0/800751152 wait complete. 2024-07-26T07:52:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:13 smithi115 ceph-mon[35364]: pgmap v523: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T07:52:13.725 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:13 smithi097 ceph-mon[32231]: pgmap v523: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T07:52:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:13 smithi181 ceph-mon[35869]: pgmap v523: 97 pgs: 97 active+clean; 664 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T07:52:13.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.k status 2024-07-26T07:52:14.255 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:16.033 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:15 smithi097 ceph-mon[32231]: pgmap v524: 97 pgs: 97 active+clean; 668 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 13 op/s 2024-07-26T07:52:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:15 smithi115 ceph-mon[35364]: pgmap v524: 97 pgs: 97 active+clean; 668 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 13 op/s 2024-07-26T07:52:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:15 smithi181 ceph-mon[35869]: pgmap v524: 97 pgs: 97 active+clean; 668 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 13 op/s 2024-07-26T07:52:16.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 -- 172.21.15.97:0/3161647466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 msgr2=0x7f220c1059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 --2- 172.21.15.97:0/3161647466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 0x7f220c1059e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009900 tx=0x7f21fc02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 -- 172.21.15.97:0/3161647466 shutdown_connections 2024-07-26T07:52:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 --2- 172.21.15.97:0/3161647466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f220c10b860 0x7f220c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 --2- 172.21.15.97:0/3161647466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f220c10af10 0x7f220c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 --2- 172.21.15.97:0/3161647466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 0x7f220c1059e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 -- 172.21.15.97:0/3161647466 >> 172.21.15.97:0/3161647466 conn(0x7f220c101040 msgr2=0x7f220c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 -- 172.21.15.97:0/3161647466 shutdown_connections 2024-07-26T07:52:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.307+0000 7f220b200640 1 -- 172.21.15.97:0/3161647466 wait complete. 2024-07-26T07:52:16.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.308+0000 7f220b200640 1 Processor -- start 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.308+0000 7f220b200640 1 -- start start 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f220b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 0x7f220c19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f220b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f220c10af10 0x7f220c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f220b200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f220c10b860 0x7f220c1a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f220b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f220c115410 con 0x7f220c1055d0 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f220b200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f220c115290 con 0x7f220c10b860 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f220b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f220c115590 con 0x7f220c10af10 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f2209e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 0x7f220c19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f2209e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 0x7f220c19be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56520/0 (socket says 172.21.15.97:56520) 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f2209e00640 1 -- 172.21.15.97:0/3293603204 learned_addr learned my addr 172.21.15.97:0/3293603204 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f2203e00640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f220c10af10 0x7f220c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.309+0000 7f220a800640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f220c10b860 0x7f220c1a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2209e00640 1 -- 172.21.15.97:0/3293603204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f220c10af10 msgr2=0x7f220c19c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2209e00640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f220c10af10 0x7f220c19c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2209e00640 1 -- 172.21.15.97:0/3293603204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f220c10b860 msgr2=0x7f220c1a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2209e00640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f220c10b860 0x7f220c1a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2209e00640 1 -- 172.21.15.97:0/3293603204 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c1a1d30 con 0x7f220c1055d0 2024-07-26T07:52:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f220a800640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f220c10b860 0x7f220c1a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:52:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2209e00640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 0x7f220c19be80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f21fc02a910 tx=0x7f21fc003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2202a00640 1 -- 172.21.15.97:0/3293603204 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc03f070 con 0x7f220c1055d0 2024-07-26T07:52:16.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2202a00640 1 -- 172.21.15.97:0/3293603204 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21fc03bdc0 con 0x7f220c1055d0 2024-07-26T07:52:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f220b200640 1 -- 172.21.15.97:0/3293603204 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220c1a1fc0 con 0x7f220c1055d0 2024-07-26T07:52:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f2202a00640 1 -- 172.21.15.97:0/3293603204 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc035030 con 0x7f220c1055d0 2024-07-26T07:52:16.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.310+0000 7f220b200640 1 -- 172.21.15.97:0/3293603204 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220c1a24f0 con 0x7f220c1055d0 2024-07-26T07:52:16.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2217851740 1 Processor -- start 2024-07-26T07:52:16.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2217851740 1 -- start start 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2202a00640 1 -- 172.21.15.97:0/3293603204 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f21fc042070 con 0x7f220c1055d0 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2217851740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a90b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2217851740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a53d0a0 0x55ae8a90bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2217851740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a545280 0x55ae8a90c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2217851740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a881970 con 0x55ae8a53d0a0 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2217851740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a870e70 con 0x55ae8a565f60 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.314+0000 7f2217851740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a874eb0 con 0x55ae8a545280 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2209e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a53d0a0 0x55ae8a90bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2209e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a53d0a0 0x55ae8a90bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56526/0 (socket says 172.21.15.97:56526) 2024-07-26T07:52:16.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2209e00640 1 -- 172.21.15.97:0/1253592903 learned_addr learned my addr 172.21.15.97:0/1253592903 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f220a800640 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a90b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2203e00640 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a545280 0x55ae8a90c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2209e00640 1 -- 172.21.15.97:0/1253592903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a545280 msgr2=0x55ae8a90c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2209e00640 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a545280 0x55ae8a90c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2209e00640 1 -- 172.21.15.97:0/1253592903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 msgr2=0x55ae8a90b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2209e00640 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a90b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.315+0000 7f2209e00640 1 -- 172.21.15.97:0/1253592903 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae8a818110 con 0x55ae8a53d0a0 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.316+0000 7f2209e00640 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a53d0a0 0x55ae8a90bcc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f21fc10abf0 tx=0x7f21fc10ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984ecc67d962b5d6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.316+0000 7f21fb000640 1 -- 172.21.15.97:0/1253592903 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc113070 con 0x55ae8a53d0a0 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.316+0000 7f21fb000640 1 -- 172.21.15.97:0/1253592903 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21fc107e50 con 0x55ae8a53d0a0 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.317+0000 7f21fb000640 1 -- 172.21.15.97:0/1253592903 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc10f420 con 0x55ae8a53d0a0 2024-07-26T07:52:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.317+0000 7f2202a00640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f21ec0b3d10 0x7f21ec0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.317+0000 7f2202a00640 1 -- 172.21.15.97:0/3293603204 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f21fc104870 con 0x7f220c1055d0 2024-07-26T07:52:16.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.317+0000 7f220a800640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f21ec0b3d10 0x7f21ec0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.317+0000 7f2217851740 1 -- 172.21.15.97:0/1253592903 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a53d0a0 msgr2=0x55ae8a90bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.317+0000 7f2217851740 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a53d0a0 0x55ae8a90bcc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f21fc10abf0 tx=0x7f21fc10ace0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.318+0000 7f2217851740 1 -- 172.21.15.97:0/1253592903 shutdown_connections 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.318+0000 7f2217851740 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a545280 0x55ae8a90c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.318+0000 7f2217851740 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a53d0a0 0x55ae8a90bcc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.318+0000 7f2217851740 1 --2- 172.21.15.97:0/1253592903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a90b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.318+0000 7f2217851740 1 -- 172.21.15.97:0/1253592903 >> 172.21.15.97:0/1253592903 conn(0x55ae8a55a350 msgr2=0x55ae8a909670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.318+0000 7f2217851740 1 -- 172.21.15.97:0/1253592903 shutdown_connections 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.318+0000 7f220a800640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f21ec0b3d10 0x7f21ec0b6220 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2204005fd0 tx=0x7f2204008470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.318+0000 7f2217851740 1 -- 172.21.15.97:0/1253592903 wait complete. 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.319+0000 7f2217851740 1 -- start start 2024-07-26T07:52:16.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.319+0000 7f2217851740 1 Processor -- start 2024-07-26T07:52:16.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2217851740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a916fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2217851740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a917520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2217851740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a922fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2217851740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a6d45f0 con 0x55ae8a545280 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2217851740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a80d540 con 0x55ae8a565f60 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2217851740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a919d20 con 0x55ae8a53d0a0 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2217851740 10 client.0 _ll_register_callbacks cb 0x55ae8a81cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2203e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a917520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2203e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a917520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56536/0 (socket says 172.21.15.97:56536) 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2203e00640 1 -- 172.21.15.97:0/2326765858 learned_addr learned my addr 172.21.15.97:0/2326765858 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2209e00640 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a916fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f220a800640 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a922fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2203e00640 1 -- 172.21.15.97:0/2326765858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 msgr2=0x55ae8a916fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.320+0000 7f2203e00640 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a916fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f2203e00640 1 -- 172.21.15.97:0/2326765858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 msgr2=0x55ae8a922fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:16.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f2203e00640 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a922fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f2203e00640 1 -- 172.21.15.97:0/2326765858 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae8a5d1540 con 0x55ae8a545280 2024-07-26T07:52:16.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f2209e00640 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a916fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:52:16.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f2203e00640 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a917520 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f21f4010720 tx=0x7f21f4010be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d298b401068fb7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:16.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f21f3400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:16.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f21f3400640 1 -- 172.21.15.97:0/2326765858 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f400cb10 con 0x55ae8a545280 2024-07-26T07:52:16.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f21f3400640 1 -- 172.21.15.97:0/2326765858 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f21f400ccb0 con 0x55ae8a545280 2024-07-26T07:52:16.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.321+0000 7f2217851740 1 -- 172.21.15.97:0/2326765858 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21fc107e50 con 0x55ae8a545280 2024-07-26T07:52:16.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f21f3400640 1 -- 172.21.15.97:0/2326765858 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21f4005020 con 0x55ae8a545280 2024-07-26T07:52:16.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f2217851740 10 client.15486 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:16.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f2217851740 1 -- 172.21.15.97:0/2326765858 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ae8a925ba0 con 0x55ae8a545280 2024-07-26T07:52:16.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f21f3400640 1 -- 172.21.15.97:0/2326765858 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21fc107e50 con 0x55ae8a545280 2024-07-26T07:52:16.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f21f3400640 1 -- 172.21.15.97:0/2326765858 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f21f4011440 con 0x55ae8a545280 2024-07-26T07:52:16.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f2217851740 10 client.15486 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:16.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f2217851740 10 client.15486 resolve_mds: resolved name 'k' to daemon mds.k 2024-07-26T07:52:16.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f2217851740 20 client.15486 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:16.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f2217851740 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55ae8a5248e0 0x55ae8a929650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f2217851740 4 client.15486 mds_command: new command op to 34517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:16.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.322+0000 7f2217851740 1 -- 172.21.15.97:0/2326765858 --> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae8a8838c0 con 0x55ae8a5248e0 2024-07-26T07:52:16.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.323+0000 7f2209e00640 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55ae8a5248e0 0x55ae8a929650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.324+0000 7f2209e00640 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55ae8a5248e0 0x55ae8a929650 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f011a8bdee77c94f server_cookie=667b3396ab08358 in_seq=0 out_seq=0 2024-07-26T07:52:16.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.324+0000 7f21f3400640 10 client.15486 ms_handle_connect on v2:172.21.15.115:6832/1028321962 2024-07-26T07:52:16.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f21f3400640 1 -- 172.21.15.97:0/2326765858 <== mds.3 v2:172.21.15.115:6832/1028321962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ae8a8838c0 con 0x55ae8a5248e0 2024-07-26T07:52:16.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f21f3400640 10 client.15486 handle_command_reply: tid=0 2024-07-26T07:52:16.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 client.15486 shutdown 2024-07-26T07:52:16.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 -- 172.21.15.97:0/2326765858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 msgr2=0x55ae8a917520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a917520 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f21f4010720 tx=0x7f21f4010be0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 -- 172.21.15.97:0/2326765858 shutdown_connections 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55ae8a5248e0 0x55ae8a929650 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a922fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a917520 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 --2- 172.21.15.97:0/2326765858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a916fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f21f3400640 0 client.15486 ms_handle_reset on v2:172.21.15.115:6832/1028321962 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.326+0000 7f2217851740 1 -- 172.21.15.97:0/2326765858 >> 172.21.15.97:0/2326765858 conn(0x55ae8a55a350 msgr2=0x55ae8a90e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.327+0000 7f2217851740 1 -- 172.21.15.97:0/2326765858 shutdown_connections 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.327+0000 7f2217851740 1 -- 172.21.15.97:0/2326765858 wait complete. 2024-07-26T07:52:16.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.327+0000 7f2217851740 20 client.15486 trim_cache size 0 max 16384 2024-07-26T07:52:16.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.346+0000 7f2217851740 1 Processor -- start 2024-07-26T07:52:16.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.346+0000 7f2217851740 1 -- start start 2024-07-26T07:52:16.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.346+0000 7f2217851740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a922f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.346+0000 7f2217851740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a916d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.346+0000 7f2217851740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a917290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.346+0000 7f2217851740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a927990 con 0x55ae8a545280 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.346+0000 7f2217851740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a928060 con 0x55ae8a565f60 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.346+0000 7f2217851740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a928730 con 0x55ae8a53d0a0 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a916d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a916d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56550/0 (socket says 172.21.15.97:56550) 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 -- 172.21.15.97:0/1062648765 learned_addr learned my addr 172.21.15.97:0/1062648765 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 -- 172.21.15.97:0/1062648765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 msgr2=0x55ae8a922f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2209e00640 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a922f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a922f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f220a800640 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a917290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 -- 172.21.15.97:0/1062648765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 msgr2=0x55ae8a917290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a917290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 -- 172.21.15.97:0/1062648765 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae8a9047f0 con 0x55ae8a545280 2024-07-26T07:52:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.347+0000 7f2203e00640 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a916d50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f21f40058c0 tx=0x7f21f400e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50eb7afde402e106 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.348+0000 7f21fa600640 1 -- 172.21.15.97:0/1062648765 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f402a070 con 0x55ae8a545280 2024-07-26T07:52:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.348+0000 7f21fa600640 1 -- 172.21.15.97:0/1062648765 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21f4011aa0 con 0x55ae8a545280 2024-07-26T07:52:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.348+0000 7f21fa600640 1 -- 172.21.15.97:0/1062648765 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4000e60 con 0x55ae8a545280 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 -- 172.21.15.97:0/1062648765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 msgr2=0x55ae8a916d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a916d50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f21f40058c0 tx=0x7f21f400e660 comp rx=0 tx=0).stop 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 -- 172.21.15.97:0/1062648765 shutdown_connections 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a565f60 0x55ae8a917290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a916d50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 --2- 172.21.15.97:0/1062648765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a53d0a0 0x55ae8a922f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 -- 172.21.15.97:0/1062648765 >> 172.21.15.97:0/1062648765 conn(0x55ae8a55a350 msgr2=0x55ae8a915820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 -- 172.21.15.97:0/1062648765 shutdown_connections 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.349+0000 7f2217851740 1 -- 172.21.15.97:0/1062648765 wait complete. 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.350+0000 7f2217851740 1 -- start start 2024-07-26T07:52:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.350+0000 7f2217851740 1 Processor -- start 2024-07-26T07:52:16.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.350+0000 7f2217851740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a53d0a0 0x55ae8a92af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2217851740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a92b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2217851740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a565f60 0x55ae8a9303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2217851740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a9294d0 con 0x55ae8a545280 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2217851740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a928e00 con 0x55ae8a53d0a0 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2217851740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ae8a929ba0 con 0x55ae8a565f60 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2217851740 10 client.0 _ll_register_callbacks cb 0x7f21f400eee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2203e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a92b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2203e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a92b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56558/0 (socket says 172.21.15.97:56558) 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2203e00640 1 -- 172.21.15.97:0/1363888267 learned_addr learned my addr 172.21.15.97:0/1363888267 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2209e00640 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a53d0a0 0x55ae8a92af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2203e00640 1 -- 172.21.15.97:0/1363888267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a565f60 msgr2=0x55ae8a9303b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2203e00640 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a565f60 0x55ae8a9303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2203e00640 1 -- 172.21.15.97:0/1363888267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a53d0a0 msgr2=0x55ae8a92af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2203e00640 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a53d0a0 0x55ae8a92af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.351+0000 7f2203e00640 1 -- 172.21.15.97:0/1363888267 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae8a930a20 con 0x55ae8a545280 2024-07-26T07:52:16.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.352+0000 7f2203e00640 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a92b530 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f21f4010410 tx=0x7f21f400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26df03bd5b8e98f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:16.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.352+0000 7f21f3400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.352+0000 7f21f3400640 1 -- 172.21.15.97:0/1363888267 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f403b060 con 0x55ae8a545280 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.352+0000 7f21f3400640 1 -- 172.21.15.97:0/1363888267 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f21f40051c0 con 0x55ae8a545280 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.352+0000 7f2217851740 1 -- 172.21.15.97:0/1363888267 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21f4011aa0 con 0x55ae8a545280 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.352+0000 7f21f3400640 1 -- 172.21.15.97:0/1363888267 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21f4011e60 con 0x55ae8a545280 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f2217851740 10 client.15492 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f2217851740 1 -- 172.21.15.97:0/1363888267 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ae8a931de0 con 0x55ae8a545280 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f21f3400640 1 -- 172.21.15.97:0/1363888267 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21f4011aa0 con 0x55ae8a545280 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f21f3400640 1 -- 172.21.15.97:0/1363888267 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f21f403f030 con 0x55ae8a545280 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f2217851740 10 client.15492 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f2217851740 10 client.15492 resolve_mds: resolved name 'k' to daemon mds.k 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f2217851740 20 client.15492 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f2217851740 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55ae8a4f9c90 0x55ae8a995540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:16.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f2217851740 4 client.15492 mds_command: new command op to 34517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:52:16.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.353+0000 7f2217851740 1 -- 172.21.15.97:0/1363888267 --> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] -- command(tid 0: {"prefix": "status"}) -- 0x55ae8a87f360 con 0x55ae8a4f9c90 2024-07-26T07:52:16.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.354+0000 7f2209e00640 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55ae8a4f9c90 0x55ae8a995540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.355+0000 7f2209e00640 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55ae8a4f9c90 0x55ae8a995540 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c581a3b2ee4145d3 server_cookie=1b4b0d8b4ba0aa79 in_seq=0 out_seq=0 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.355+0000 7f21f3400640 10 client.15492 ms_handle_connect on v2:172.21.15.115:6832/1028321962 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.355+0000 7f21f3400640 1 -- 172.21.15.97:0/1363888267 <== mds.3 v2:172.21.15.115:6832/1028321962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ae8a87f360 con 0x55ae8a4f9c90 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.355+0000 7f21f3400640 10 client.15492 handle_command_reply: tid=0 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.355+0000 7f2217851740 1 client.15492 shutdown 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 -- 172.21.15.97:0/1363888267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 msgr2=0x55ae8a92b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a92b530 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f21f4010410 tx=0x7f21f400cf00 comp rx=0 tx=0).stop 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 -- 172.21.15.97:0/1363888267 shutdown_connections 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae8a565f60 0x55ae8a9303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ae8a545280 0x55ae8a92b530 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.115:6832/1028321962,v1:172.21.15.115:6833/1028321962] conn(0x55ae8a4f9c90 0x55ae8a995540 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 --2- 172.21.15.97:0/1363888267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae8a53d0a0 0x55ae8a92af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 -- 172.21.15.97:0/1363888267 >> 172.21.15.97:0/1363888267 conn(0x55ae8a55a350 msgr2=0x55ae8a9157e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:16.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f21f3400640 0 client.15492 ms_handle_reset on v2:172.21.15.115:6832/1028321962 2024-07-26T07:52:16.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 -- 172.21.15.97:0/1363888267 shutdown_connections 2024-07-26T07:52:16.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 1 -- 172.21.15.97:0/1363888267 wait complete. 2024-07-26T07:52:16.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.356+0000 7f2217851740 20 client.15492 trim_cache size 0 max 16384 2024-07-26T07:52:16.358 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:52:16.358 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 3, 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34517, 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 250.58136365199999, 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 86, 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 328.174283737 2024-07-26T07:52:16.359 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:52:16.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 -- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f21ec0b3d10 msgr2=0x7f21ec0b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f21ec0b3d10 0x7f21ec0b6220 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2204005fd0 tx=0x7f2204008470 comp rx=0 tx=0).stop 2024-07-26T07:52:16.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 -- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 msgr2=0x7f220c19be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:16.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 0x7f220c19be80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f21fc02a910 tx=0x7f21fc003d60 comp rx=0 tx=0).stop 2024-07-26T07:52:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 -- 172.21.15.97:0/3293603204 shutdown_connections 2024-07-26T07:52:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f21ec0b3d10 0x7f21ec0b6220 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f220c10b860 0x7f220c1a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f220c10af10 0x7f220c19c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 --2- 172.21.15.97:0/3293603204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f220c1055d0 0x7f220c19be80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 -- 172.21.15.97:0/3293603204 >> 172.21.15.97:0/3293603204 conn(0x7f220c101040 msgr2=0x7f220c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.357+0000 7f220b200640 1 -- 172.21.15.97:0/3293603204 shutdown_connections 2024-07-26T07:52:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:16.358+0000 7f220b200640 1 -- 172.21.15.97:0/3293603204 wait complete. 2024-07-26T07:52:17.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.e status 2024-07-26T07:52:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:17.986 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:17 smithi097 ceph-mon[32231]: pgmap v525: 97 pgs: 97 active+clean; 668 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T07:52:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:17 smithi115 ceph-mon[35364]: pgmap v525: 97 pgs: 97 active+clean; 668 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T07:52:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:17 smithi181 ceph-mon[35869]: pgmap v525: 97 pgs: 97 active+clean; 668 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T07:52:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 -- 172.21.15.97:0/3609456564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4102760 msgr2=0x7f1cb410f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 --2- 172.21.15.97:0/3609456564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4102760 0x7f1cb410f480 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4009900 tx=0x7f1ca402a480 comp rx=0 tx=0).stop 2024-07-26T07:52:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 -- 172.21.15.97:0/3609456564 shutdown_connections 2024-07-26T07:52:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 --2- 172.21.15.97:0/3609456564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb410f9c0 0x7f1cb4111de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 --2- 172.21.15.97:0/3609456564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4102760 0x7f1cb410f480 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 --2- 172.21.15.97:0/3609456564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1cb4101e10 0x7f1cb4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 -- 172.21.15.97:0/3609456564 >> 172.21.15.97:0/3609456564 conn(0x7f1cb40fdcf0 msgr2=0x7f1cb4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:19.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 -- 172.21.15.97:0/3609456564 shutdown_connections 2024-07-26T07:52:19.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.555+0000 7f1cba600640 1 -- 172.21.15.97:0/3609456564 wait complete. 2024-07-26T07:52:19.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.556+0000 7f1cba600640 1 Processor -- start 2024-07-26T07:52:19.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.556+0000 7f1cba600640 1 -- start start 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.556+0000 7f1cba600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4101e10 0x7f1cb419bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cba600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1cb4102760 0x7f1cb419c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cba600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb410f9c0 0x7f1cb41a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cba600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb4115350 con 0x7f1cb4101e10 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cba600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41151d0 con 0x7f1cb4102760 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cba600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41154d0 con 0x7f1cb410f9c0 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4101e10 0x7f1cb419bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4101e10 0x7f1cb419bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56578/0 (socket says 172.21.15.97:56578) 2024-07-26T07:52:19.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cb9200640 1 -- 172.21.15.97:0/932317898 learned_addr learned my addr 172.21.15.97:0/932317898 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:19.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cb9c00640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb410f9c0 0x7f1cb41a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cabe00640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1cb4102760 0x7f1cb419c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cb9200640 1 -- 172.21.15.97:0/932317898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb410f9c0 msgr2=0x7f1cb41a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cb9200640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb410f9c0 0x7f1cb41a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cb9200640 1 -- 172.21.15.97:0/932317898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1cb4102760 msgr2=0x7f1cb419c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.557+0000 7f1cb9200640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1cb4102760 0x7f1cb419c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.558+0000 7f1cb9200640 1 -- 172.21.15.97:0/932317898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb41a1d50 con 0x7f1cb4101e10 2024-07-26T07:52:19.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.558+0000 7f1cabe00640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1cb4102760 0x7f1cb419c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:52:19.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.558+0000 7f1cb9200640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4101e10 0x7f1cb419bde0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d950 tx=0x7f1c9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:19.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.558+0000 7f1cb2000640 1 -- 172.21.15.97:0/932317898 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c017070 con 0x7f1cb4101e10 2024-07-26T07:52:19.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.558+0000 7f1cb2000640 1 -- 172.21.15.97:0/932317898 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c9c009c30 con 0x7f1cb4101e10 2024-07-26T07:52:19.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.558+0000 7f1cb2000640 1 -- 172.21.15.97:0/932317898 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c00f620 con 0x7f1cb4101e10 2024-07-26T07:52:19.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.558+0000 7f1cba600640 1 -- 172.21.15.97:0/932317898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb41a2040 con 0x7f1cb4101e10 2024-07-26T07:52:19.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.558+0000 7f1cba600640 1 -- 172.21.15.97:0/932317898 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb41a87a0 con 0x7f1cb4101e10 2024-07-26T07:52:19.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.563+0000 7f1cc2a2b740 1 Processor -- start 2024-07-26T07:52:19.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.563+0000 7f1cc2a2b740 1 -- start start 2024-07-26T07:52:19.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.563+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b961780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.563+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b961cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.563+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b962200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.563+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x559b0b8d7970 con 0x559b0b5930a0 2024-07-26T07:52:19.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.563+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559b0b8c6e70 con 0x559b0b5bbf60 2024-07-26T07:52:19.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.563+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559b0b8caeb0 con 0x559b0b59b280 2024-07-26T07:52:19.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.564+0000 7f1cb2000640 1 -- 172.21.15.97:0/932317898 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f1c9c010460 con 0x7f1cb4101e10 2024-07-26T07:52:19.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.564+0000 7f1cb9c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b961780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.564+0000 7f1cb9c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b961780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:51614/0 (socket says 172.21.15.97:51614) 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.564+0000 7f1cb9c00640 1 -- 172.21.15.97:0/4100552164 learned_addr learned my addr 172.21.15.97:0/4100552164 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.564+0000 7f1cb9200640 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b961cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.564+0000 7f1cabe00640 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b962200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.565+0000 7f1cb9200640 1 -- 172.21.15.97:0/4100552164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 msgr2=0x559b0b962200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.565+0000 7f1cb9200640 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b962200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.565+0000 7f1cb9200640 1 -- 172.21.15.97:0/4100552164 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 msgr2=0x559b0b961780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.565+0000 7f1cb9200640 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b961780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.565+0000 7f1cb9200640 1 -- 172.21.15.97:0/4100552164 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b0b86e110 con 0x559b0b5930a0 2024-07-26T07:52:19.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.566+0000 7f1cb9200640 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b961cc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0e1380 tx=0x7f1c9c0ec560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d251f6603cc8019f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:19.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.566+0000 7f1cb2000640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1c940b3ce0 0x7f1c940b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.566+0000 7f1cb2000640 1 -- 172.21.15.97:0/932317898 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f1c9c0e3a30 con 0x7f1cb4101e10 2024-07-26T07:52:19.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.566+0000 7f1caa000640 1 -- 172.21.15.97:0/4100552164 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0f5020 con 0x559b0b5930a0 2024-07-26T07:52:19.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.566+0000 7f1cb9c00640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1c940b3ce0 0x7f1c940b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.566+0000 7f1caa000640 1 -- 172.21.15.97:0/4100552164 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c9c0aa2d0 con 0x559b0b5930a0 2024-07-26T07:52:19.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.566+0000 7f1caa000640 1 -- 172.21.15.97:0/4100552164 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0ea070 con 0x559b0b5930a0 2024-07-26T07:52:19.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.567+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/4100552164 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 msgr2=0x559b0b961cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.567+0000 7f1cb9c00640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1c940b3ce0 0x7f1c940b61f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1cac005c40 tx=0x7f1cac0097e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:19.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.567+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b961cc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0e1380 tx=0x7f1c9c0ec560 comp rx=0 tx=0).stop 2024-07-26T07:52:19.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.568+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/4100552164 shutdown_connections 2024-07-26T07:52:19.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.568+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b962200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.568+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b961cc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.568+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/4100552164 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b961780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.568+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/4100552164 >> 172.21.15.97:0/4100552164 conn(0x559b0b5b0350 msgr2=0x559b0b95f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:19.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.568+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/4100552164 shutdown_connections 2024-07-26T07:52:19.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.568+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/4100552164 wait complete. 2024-07-26T07:52:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.568+0000 7f1cc2a2b740 1 -- start start 2024-07-26T07:52:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.569+0000 7f1cc2a2b740 1 Processor -- start 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.569+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b96cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.569+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b96d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.569+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b9792b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.569+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x559b0b72a5f0 con 0x559b0b5930a0 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.569+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559b0b863540 con 0x559b0b5bbf60 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.569+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559b0b96ff60 con 0x559b0b59b280 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.569+0000 7f1cc2a2b740 10 client.0 _ll_register_callbacks cb 0x559b0b872f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b96cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b96cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56592/0 (socket says 172.21.15.97:56592) 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 -- 172.21.15.97:0/3795727769 learned_addr learned my addr 172.21.15.97:0/3795727769 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cabe00640 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b96d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9c00640 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b9792b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 -- 172.21.15.97:0/3795727769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 msgr2=0x559b0b96d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b96d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 -- 172.21.15.97:0/3795727769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 msgr2=0x559b0b9792b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b9792b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 -- 172.21.15.97:0/3795727769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b0b627540 con 0x559b0b5930a0 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1cb9200640 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b96cfe0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0e1380 tx=0x7f1c9c00df40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8ef2c94592c08d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.570+0000 7f1ca2a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.571+0000 7f1ca2a00640 1 -- 172.21.15.97:0/3795727769 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0f5070 con 0x559b0b5930a0 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.571+0000 7f1ca2a00640 1 -- 172.21.15.97:0/3795727769 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f1c9c0a6a50 con 0x559b0b5930a0 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.571+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3795727769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c9c0aa2d0 con 0x559b0b5930a0 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.571+0000 7f1ca2a00640 1 -- 172.21.15.97:0/3795727769 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c9c0bfe60 con 0x559b0b5930a0 2024-07-26T07:52:19.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.571+0000 7f1cc2a2b740 10 client.15504 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.571+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3795727769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b0b97be20 con 0x559b0b5930a0 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.571+0000 7f1ca2a00640 1 -- 172.21.15.97:0/3795727769 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c9c0aa2d0 con 0x559b0b5930a0 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.571+0000 7f1ca2a00640 1 -- 172.21.15.97:0/3795727769 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1c9c0ecc40 con 0x559b0b5930a0 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.572+0000 7f1cc2a2b740 10 client.15504 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.572+0000 7f1cc2a2b740 10 client.15504 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.572+0000 7f1cc2a2b740 20 client.15504 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.572+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x559b0b57a8e0 0x559b0b97f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.572+0000 7f1cc2a2b740 4 client.15504 mds_command: new command op to 34535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:19.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.572+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3795727769 --> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b0b8d98c0 con 0x559b0b57a8e0 2024-07-26T07:52:19.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.573+0000 7f1cabe00640 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x559b0b57a8e0 0x559b0b97f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.574+0000 7f1cabe00640 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x559b0b57a8e0 0x559b0b97f320 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=51af13dc33c6abfa server_cookie=503846effc92aa42 in_seq=0 out_seq=0 2024-07-26T07:52:19.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.574+0000 7f1ca2a00640 10 client.15504 ms_handle_connect on v2:172.21.15.115:6834/4018170367 2024-07-26T07:52:19.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.575+0000 7f1ca2a00640 1 -- 172.21.15.97:0/3795727769 <== mds.2 v2:172.21.15.115:6834/4018170367 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559b0b8d98c0 con 0x559b0b57a8e0 2024-07-26T07:52:19.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.575+0000 7f1ca2a00640 10 client.15504 handle_command_reply: tid=0 2024-07-26T07:52:19.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 client.15504 shutdown 2024-07-26T07:52:19.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3795727769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 msgr2=0x559b0b96cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b96cfe0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0e1380 tx=0x7f1c9c00df40 comp rx=0 tx=0).stop 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3795727769 shutdown_connections 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x559b0b57a8e0 0x559b0b97f320 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b9792b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b96d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3795727769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b96cfe0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3795727769 >> 172.21.15.97:0/3795727769 conn(0x559b0b5b0350 msgr2=0x559b0b964910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1ca2a00640 0 client.15504 ms_handle_reset on v2:172.21.15.115:6834/4018170367 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3795727769 shutdown_connections 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3795727769 wait complete. 2024-07-26T07:52:19.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.576+0000 7f1cc2a2b740 20 client.15504 trim_cache size 0 max 16384 2024-07-26T07:52:19.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.600+0000 7f1cc2a2b740 1 Processor -- start 2024-07-26T07:52:19.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.601+0000 7f1cc2a2b740 1 -- start start 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.601+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5930a0 0x559b0b97f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.601+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b9678a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.601+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5bbf60 0x559b0b96ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.601+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x559b0b97d660 con 0x559b0b5bbf60 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.601+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559b0b97dd30 con 0x559b0b5930a0 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.601+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559b0b97e400 con 0x559b0b59b280 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.602+0000 7f1cb9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5bbf60 0x559b0b96ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.602+0000 7f1cb9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5bbf60 0x559b0b96ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56600/0 (socket says 172.21.15.97:56600) 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.602+0000 7f1cb9c00640 1 -- 172.21.15.97:0/3849613094 learned_addr learned my addr 172.21.15.97:0/3849613094 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.602+0000 7f1cb9200640 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5930a0 0x559b0b97f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.602+0000 7f1cabe00640 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b9678a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.602+0000 7f1cb9c00640 1 -- 172.21.15.97:0/3849613094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 msgr2=0x559b0b9678a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:19.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.603+0000 7f1cb9c00640 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b9678a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.603+0000 7f1cb9c00640 1 -- 172.21.15.97:0/3849613094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5930a0 msgr2=0x559b0b97f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:19.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.603+0000 7f1cb9c00640 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5930a0 0x559b0b97f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.603+0000 7f1cb9c00640 1 -- 172.21.15.97:0/3849613094 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b0b95a7f0 con 0x559b0b5bbf60 2024-07-26T07:52:19.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.603+0000 7f1cb9c00640 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5bbf60 0x559b0b96ec10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1cac010620 tx=0x7f1cac015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e29392b502c29aa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:19.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.603+0000 7f1ca9600640 1 -- 172.21.15.97:0/3849613094 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac01f620 con 0x559b0b5bbf60 2024-07-26T07:52:19.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.604+0000 7f1ca9600640 1 -- 172.21.15.97:0/3849613094 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cac01fc50 con 0x559b0b5bbf60 2024-07-26T07:52:19.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.604+0000 7f1ca9600640 1 -- 172.21.15.97:0/3849613094 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac01ecf0 con 0x559b0b5bbf60 2024-07-26T07:52:19.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3849613094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5bbf60 msgr2=0x559b0b96ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5bbf60 0x559b0b96ec10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1cac010620 tx=0x7f1cac015000 comp rx=0 tx=0).stop 2024-07-26T07:52:19.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3849613094 shutdown_connections 2024-07-26T07:52:19.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5bbf60 0x559b0b96ec10 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b9678a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/3849613094 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5930a0 0x559b0b97f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3849613094 >> 172.21.15.97:0/3849613094 conn(0x559b0b5b0350 msgr2=0x559b0b96e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:19.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3849613094 shutdown_connections 2024-07-26T07:52:19.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.605+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/3849613094 wait complete. 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.606+0000 7f1cc2a2b740 1 -- start start 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.606+0000 7f1cc2a2b740 1 Processor -- start 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b9809f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b980fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cc2a2b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b985dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x559b0b97f1a0 con 0x559b0b5930a0 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x559b0b97ead0 con 0x559b0b5bbf60 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cc2a2b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559b0b97f870 con 0x559b0b59b280 2024-07-26T07:52:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cc2a2b740 10 client.0 _ll_register_callbacks cb 0x7f1c9c0e7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:19.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b9809f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b9809f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56610/0 (socket says 172.21.15.97:56610) 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.607+0000 7f1cb9200640 1 -- 172.21.15.97:0/420192648 learned_addr learned my addr 172.21.15.97:0/420192648 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.608+0000 7f1cabe00640 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b980fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.608+0000 7f1cb9c00640 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b985dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.608+0000 7f1cb9200640 1 -- 172.21.15.97:0/420192648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 msgr2=0x559b0b980fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.608+0000 7f1cb9200640 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b980fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.608+0000 7f1cb9200640 1 -- 172.21.15.97:0/420192648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 msgr2=0x559b0b985dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.608+0000 7f1cb9200640 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b985dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.608+0000 7f1cb9200640 1 -- 172.21.15.97:0/420192648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b0b986440 con 0x559b0b5930a0 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.609+0000 7f1cb9200640 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b9809f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0058c0 tx=0x7f1c9c0b3190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5705f6a1b4ef90b1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:19.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.609+0000 7f1ca2a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:19.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.609+0000 7f1ca2a00640 1 -- 172.21.15.97:0/420192648 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0f5070 con 0x559b0b5930a0 2024-07-26T07:52:19.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.609+0000 7f1ca2a00640 1 -- 172.21.15.97:0/420192648 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f1c9c0cd0f0 con 0x559b0b5930a0 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.609+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/420192648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cac01fc50 con 0x559b0b5930a0 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.609+0000 7f1ca2a00640 1 -- 172.21.15.97:0/420192648 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c9c0ea030 con 0x559b0b5930a0 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.610+0000 7f1cc2a2b740 10 client.15510 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.610+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/420192648 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b0b987800 con 0x559b0b5930a0 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.610+0000 7f1ca2a00640 1 -- 172.21.15.97:0/420192648 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cac01fc50 con 0x559b0b5930a0 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.610+0000 7f1ca2a00640 1 -- 172.21.15.97:0/420192648 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1c9c0b33e0 con 0x559b0b5930a0 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.610+0000 7f1cc2a2b740 10 client.15510 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.610+0000 7f1cc2a2b740 10 client.15510 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.610+0000 7f1cc2a2b740 20 client.15510 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.611+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x559b0b54fc90 0x559b0b9eb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.611+0000 7f1cc2a2b740 4 client.15510 mds_command: new command op to 34535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:52:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.611+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/420192648 --> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] -- command(tid 0: {"prefix": "status"}) -- 0x559b0b8d5360 con 0x559b0b54fc90 2024-07-26T07:52:19.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.611+0000 7f1cabe00640 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x559b0b54fc90 0x559b0b9eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:19.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.612+0000 7f1cabe00640 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x559b0b54fc90 0x559b0b9eb520 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2caec6fada58a6b0 server_cookie=fe5a131a1dd3ec33 in_seq=0 out_seq=0 2024-07-26T07:52:19.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.612+0000 7f1ca2a00640 10 client.15510 ms_handle_connect on v2:172.21.15.115:6834/4018170367 2024-07-26T07:52:19.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1ca2a00640 1 -- 172.21.15.97:0/420192648 <== mds.2 v2:172.21.15.115:6834/4018170367 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559b0b8d5360 con 0x559b0b54fc90 2024-07-26T07:52:19.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1ca2a00640 10 client.15510 handle_command_reply: tid=0 2024-07-26T07:52:19.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 client.15510 shutdown 2024-07-26T07:52:19.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/420192648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 msgr2=0x559b0b9809f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b9809f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0058c0 tx=0x7f1c9c0b3190 comp rx=0 tx=0).stop 2024-07-26T07:52:19.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/420192648 shutdown_connections 2024-07-26T07:52:19.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.115:6834/4018170367,v1:172.21.15.115:6835/4018170367] conn(0x559b0b54fc90 0x559b0b9eb520 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x559b0b5bbf60 0x559b0b985dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559b0b59b280 0x559b0b980fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 --2- 172.21.15.97:0/420192648 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559b0b5930a0 0x559b0b9809f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/420192648 >> 172.21.15.97:0/420192648 conn(0x559b0b5b0350 msgr2=0x559b0b962680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:19.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.613+0000 7f1ca2a00640 0 client.15510 ms_handle_reset on v2:172.21.15.115:6834/4018170367 2024-07-26T07:52:19.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.614+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/420192648 shutdown_connections 2024-07-26T07:52:19.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.614+0000 7f1cc2a2b740 1 -- 172.21.15.97:0/420192648 wait complete. 2024-07-26T07:52:19.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.614+0000 7f1cc2a2b740 20 client.15510 trim_cache size 0 max 16384 2024-07-26T07:52:19.616 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:52:19.616 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:52:19.616 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 2, 2024-07-26T07:52:19.616 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34535, 2024-07-26T07:52:19.616 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:52:19.616 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:52:19.617 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:52:19.617 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 254.85572412100001, 2024-07-26T07:52:19.617 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-07-26T07:52:19.617 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:52:19.617 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 86, 2024-07-26T07:52:19.617 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 318.85810989300001 2024-07-26T07:52:19.617 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 -- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1c940b3ce0 msgr2=0x7f1c940b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1c940b3ce0 0x7f1c940b61f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1cac005c40 tx=0x7f1cac0097e0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 -- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4101e10 msgr2=0x7f1cb419bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4101e10 0x7f1cb419bde0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d950 tx=0x7f1c9c00de10 comp rx=0 tx=0).stop 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 -- 172.21.15.97:0/932317898 shutdown_connections 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1c940b3ce0 0x7f1c940b61f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb410f9c0 0x7f1cb41a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1cb4102760 0x7f1cb419c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 --2- 172.21.15.97:0/932317898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1cb4101e10 0x7f1cb419bde0 secure :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d950 tx=0x7f1c9c00de10 comp rx=0 tx=0).stop 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.615+0000 7f1cba600640 1 -- 172.21.15.97:0/932317898 >> 172.21.15.97:0/932317898 conn(0x7f1cb40fdcf0 msgr2=0x7f1cb4111bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:19.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.617+0000 7f1cba600640 1 -- 172.21.15.97:0/932317898 shutdown_connections 2024-07-26T07:52:19.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:19.617+0000 7f1cba600640 1 -- 172.21.15.97:0/932317898 wait complete. 2024-07-26T07:52:20.189 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:19 smithi097 ceph-mon[32231]: pgmap v526: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-07-26T07:52:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:19 smithi115 ceph-mon[35364]: pgmap v526: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-07-26T07:52:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:19 smithi181 ceph-mon[35869]: pgmap v526: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-07-26T07:52:20.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.h status 2024-07-26T07:52:20.702 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:21.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 -- 172.21.15.97:0/1391140543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a78104070 msgr2=0x7f6a781064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 --2- 172.21.15.97:0/1391140543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a78104070 0x7f6a781064f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009900 tx=0x7f6a6002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 -- 172.21.15.97:0/1391140543 shutdown_connections 2024-07-26T07:52:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 --2- 172.21.15.97:0/1391140543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a78106c30 0x7f6a78107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 --2- 172.21.15.97:0/1391140543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a78104070 0x7f6a781064f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 --2- 172.21.15.97:0/1391140543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a7806bee0 0x7f6a78103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 -- 172.21.15.97:0/1391140543 >> 172.21.15.97:0/1391140543 conn(0x7f6a780fd960 msgr2=0x7f6a780ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 -- 172.21.15.97:0/1391140543 shutdown_connections 2024-07-26T07:52:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.604+0000 7f6a7d200640 1 -- 172.21.15.97:0/1391140543 wait complete. 2024-07-26T07:52:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.606+0000 7f6a7d200640 1 Processor -- start 2024-07-26T07:52:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.606+0000 7f6a7d200640 1 -- start start 2024-07-26T07:52:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.606+0000 7f6a7d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a7806bee0 0x7f6a7819fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.606+0000 7f6a7d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78104070 0x7f6a781a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.606+0000 7f6a7d200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a78106c30 0x7f6a781a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.606+0000 7f6a7d200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78115210 con 0x7f6a7806bee0 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.606+0000 7f6a7d200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78115090 con 0x7f6a78106c30 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.606+0000 7f6a7d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78115390 con 0x7f6a78104070 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a7806bee0 0x7f6a7819fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a7806bee0 0x7f6a7819fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56614/0 (socket says 172.21.15.97:56614) 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a75e00640 1 -- 172.21.15.97:0/932140763 learned_addr learned my addr 172.21.15.97:0/932140763 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a6fe00640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78104070 0x7f6a781a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a76800640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a78106c30 0x7f6a781a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a75e00640 1 -- 172.21.15.97:0/932140763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78104070 msgr2=0x7f6a781a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a75e00640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78104070 0x7f6a781a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a75e00640 1 -- 172.21.15.97:0/932140763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a78106c30 msgr2=0x7f6a781a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a75e00640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a78106c30 0x7f6a781a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a75e00640 1 -- 172.21.15.97:0/932140763 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a781a1920 con 0x7f6a7806bee0 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.607+0000 7f6a76800640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a78106c30 0x7f6a781a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.608+0000 7f6a75e00640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a7806bee0 0x7f6a7819fee0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800d770 tx=0x7f6a6800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.608+0000 7f6a6ea00640 1 -- 172.21.15.97:0/932140763 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68017070 con 0x7f6a7806bee0 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.608+0000 7f6a6ea00640 1 -- 172.21.15.97:0/932140763 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a680054b0 con 0x7f6a7806bee0 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.608+0000 7f6a7d200640 1 -- 172.21.15.97:0/932140763 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a781a1c10 con 0x7f6a7806bee0 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.608+0000 7f6a6ea00640 1 -- 172.21.15.97:0/932140763 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68004c60 con 0x7f6a7806bee0 2024-07-26T07:52:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.608+0000 7f6a7d200640 1 -- 172.21.15.97:0/932140763 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a781a8300 con 0x7f6a7806bee0 2024-07-26T07:52:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.611+0000 7f6a841dc740 1 Processor -- start 2024-07-26T07:52:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.612+0000 7f6a841dc740 1 -- start start 2024-07-26T07:52:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.612+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b305ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.612+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af372f0 0x562c4b306550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.612+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af3f4d0 0x562c4b3070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.612+0000 7f6a841dc740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562c4b278270 con 0x562c4af372f0 2024-07-26T07:52:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.612+0000 7f6a841dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562c4b267770 con 0x562c4af601b0 2024-07-26T07:52:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.612+0000 7f6a841dc740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562c4b26b7b0 con 0x562c4af3f4d0 2024-07-26T07:52:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.612+0000 7f6a6fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af3f4d0 0x562c4b3070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a76800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b305ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a6fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af3f4d0 0x562c4b3070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:52058/0 (socket says 172.21.15.97:52058) 2024-07-26T07:52:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a6fe00640 1 -- 172.21.15.97:0/278356997 learned_addr learned my addr 172.21.15.97:0/278356997 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a6fe00640 1 -- 172.21.15.97:0/278356997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 msgr2=0x562c4b305ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a6fe00640 1 --2- 172.21.15.97:0/278356997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b305ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a6fe00640 1 -- 172.21.15.97:0/278356997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af372f0 msgr2=0x562c4b306550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:52:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a6fe00640 1 --2- 172.21.15.97:0/278356997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af372f0 0x562c4b306550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a6fe00640 1 -- 172.21.15.97:0/278356997 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c4afcb790 con 0x562c4af3f4d0 2024-07-26T07:52:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.613+0000 7f6a76800640 1 --2- 172.21.15.97:0/278356997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b305ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:52:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.614+0000 7f6a6fe00640 1 --2- 172.21.15.97:0/278356997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af3f4d0 0x562c4b3070a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6a600096c0 tx=0x7f6a60031650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb421a5454989d61 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.614+0000 7f6a67400640 1 -- 172.21.15.97:0/278356997 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60035030 con 0x562c4af3f4d0 2024-07-26T07:52:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.614+0000 7f6a6ea00640 1 -- 172.21.15.97:0/932140763 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f6a68010460 con 0x7f6a7806bee0 2024-07-26T07:52:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.614+0000 7f6a67400640 1 -- 172.21.15.97:0/278356997 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a600044c0 con 0x562c4af3f4d0 2024-07-26T07:52:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.614+0000 7f6a67400640 1 -- 172.21.15.97:0/278356997 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60003ac0 con 0x562c4af3f4d0 2024-07-26T07:52:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 -- 172.21.15.97:0/278356997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af3f4d0 msgr2=0x562c4b3070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 --2- 172.21.15.97:0/278356997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af3f4d0 0x562c4b3070a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6a600096c0 tx=0x7f6a60031650 comp rx=0 tx=0).stop 2024-07-26T07:52:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 -- 172.21.15.97:0/278356997 shutdown_connections 2024-07-26T07:52:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 --2- 172.21.15.97:0/278356997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af3f4d0 0x562c4b3070a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 --2- 172.21.15.97:0/278356997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af372f0 0x562c4b306550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 --2- 172.21.15.97:0/278356997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b305ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 -- 172.21.15.97:0/278356997 >> 172.21.15.97:0/278356997 conn(0x562c4af54300 msgr2=0x562c4b301970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 -- 172.21.15.97:0/278356997 shutdown_connections 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.615+0000 7f6a841dc740 1 -- 172.21.15.97:0/278356997 wait complete. 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.616+0000 7f6a6ea00640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a5c0b3c00 0x7f6a5c0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.616+0000 7f6a841dc740 1 -- start start 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.616+0000 7f6a6ea00640 1 -- 172.21.15.97:0/932140763 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f6a680e38e0 con 0x7f6a7806bee0 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.616+0000 7f6a76800640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a5c0b3c00 0x7f6a5c0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.616+0000 7f6a841dc740 1 Processor -- start 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b30f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 0x562c4b310080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b310bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a841dc740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562c4b1a1ab0 con 0x562c4af601b0 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a841dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562c4b204d50 con 0x562c4af3f4d0 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a841dc740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562c4b307b60 con 0x562c4af372f0 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a841dc740 10 client.0 _ll_register_callbacks cb 0x562c4af77a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a76800640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a5c0b3c00 0x7f6a5c0b6110 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000aec0 tx=0x7f6a7000a730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b30f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.617+0000 7f6a76800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b310bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a6fe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 0x562c4b310080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a76800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b310bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56630/0 (socket says 172.21.15.97:56630) 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b30f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:52060/0 (socket says 172.21.15.97:52060) 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a76800640 1 -- 172.21.15.97:0/3442371188 learned_addr learned my addr 172.21.15.97:0/3442371188 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a76800640 1 -- 172.21.15.97:0/3442371188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 msgr2=0x562c4b30f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a76800640 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b30f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a76800640 1 -- 172.21.15.97:0/3442371188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 msgr2=0x562c4b310080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:21.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a76800640 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 0x562c4b310080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.618+0000 7f6a76800640 1 -- 172.21.15.97:0/3442371188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c4b044620 con 0x562c4af601b0 2024-07-26T07:52:21.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.619+0000 7f6a76800640 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b310bf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000edf0 tx=0x7f6a70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3761c31489169d88 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:21.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.619+0000 7f6a5be00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.619+0000 7f6a5be00640 1 -- 172.21.15.97:0/3442371188 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7001cd30 con 0x562c4af601b0 2024-07-26T07:52:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.619+0000 7f6a5be00640 1 -- 172.21.15.97:0/3442371188 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f6a70028480 con 0x562c4af601b0 2024-07-26T07:52:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.619+0000 7f6a841dc740 1 -- 172.21.15.97:0/3442371188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562c4b27a1c0 con 0x562c4af601b0 2024-07-26T07:52:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.619+0000 7f6a5be00640 1 -- 172.21.15.97:0/3442371188 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6a70028680 con 0x562c4af601b0 2024-07-26T07:52:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a841dc740 10 client.15519 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a841dc740 1 -- 172.21.15.97:0/3442371188 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562c4b319c60 con 0x562c4af601b0 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a5be00640 1 -- 172.21.15.97:0/3442371188 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x562c4b27a1c0 con 0x562c4af601b0 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a5be00640 1 -- 172.21.15.97:0/3442371188 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6a70031050 con 0x562c4af601b0 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a841dc740 10 client.15519 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a841dc740 10 client.15519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a841dc740 20 client.15519 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562c4af1ed80 0x562c4b31d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a841dc740 4 client.15519 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.620+0000 7f6a841dc740 1 -- 172.21.15.97:0/3442371188 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c4b275c60 con 0x562c4af1ed80 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.621+0000 7f6a75e00640 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562c4af1ed80 0x562c4b31d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.622+0000 7f6a75e00640 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562c4af1ed80 0x562c4b31d340 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee9c591ffad0ea24 server_cookie=ba0bff3c6b168904 in_seq=0 out_seq=0 2024-07-26T07:52:21.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.622+0000 7f6a5be00640 10 client.15519 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:21.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a5be00640 1 -- 172.21.15.97:0/3442371188 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562c4b275c60 con 0x562c4af1ed80 2024-07-26T07:52:21.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a5be00640 10 client.15519 handle_command_reply: tid=0 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 client.15519 shutdown 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 -- 172.21.15.97:0/3442371188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 msgr2=0x562c4b310bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b310bf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000edf0 tx=0x7f6a70004a40 comp rx=0 tx=0).stop 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 -- 172.21.15.97:0/3442371188 shutdown_connections 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562c4af1ed80 0x562c4b31d340 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b310bf0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 0x562c4b310080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3442371188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b30f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 -- 172.21.15.97:0/3442371188 >> 172.21.15.97:0/3442371188 conn(0x562c4af54300 msgr2=0x562c4b3000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a5be00640 0 client.15519 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 -- 172.21.15.97:0/3442371188 shutdown_connections 2024-07-26T07:52:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.624+0000 7f6a841dc740 1 -- 172.21.15.97:0/3442371188 wait complete. 2024-07-26T07:52:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.625+0000 7f6a841dc740 20 client.15519 trim_cache size 0 max 16384 2024-07-26T07:52:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.641+0000 7f6a841dc740 1 Processor -- start 2024-07-26T07:52:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.641+0000 7f6a841dc740 1 -- start start 2024-07-26T07:52:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3007c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 0x562c4b300d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b322b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a841dc740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6a781b1010 con 0x562c4af601b0 2024-07-26T07:52:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a841dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6a781b2b50 con 0x562c4af3f4d0 2024-07-26T07:52:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a841dc740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6a781b16e0 con 0x562c4af372f0 2024-07-26T07:52:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a76800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b322b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a76800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b322b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56636/0 (socket says 172.21.15.97:56636) 2024-07-26T07:52:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a76800640 1 -- 172.21.15.97:0/1291105853 learned_addr learned my addr 172.21.15.97:0/1291105853 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a75e00640 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.642+0000 7f6a6fe00640 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 0x562c4b300d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.643+0000 7f6a75e00640 1 -- 172.21.15.97:0/1291105853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 msgr2=0x562c4b300d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:21.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.643+0000 7f6a75e00640 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 0x562c4b300d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.643+0000 7f6a75e00640 1 -- 172.21.15.97:0/1291105853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 msgr2=0x562c4b322b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:21.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.643+0000 7f6a75e00640 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b322b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.643+0000 7f6a75e00640 1 -- 172.21.15.97:0/1291105853 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c4b30f750 con 0x562c4af372f0 2024-07-26T07:52:21.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.643+0000 7f6a76800640 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b322b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:52:21.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.644+0000 7f6a75e00640 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3007c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6a680e1230 tx=0x7f6a68005bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ff8817fe81a5172 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.644+0000 7f6a66a00640 1 -- 172.21.15.97:0/1291105853 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68014070 con 0x562c4af372f0 2024-07-26T07:52:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.644+0000 7f6a66a00640 1 -- 172.21.15.97:0/1291105853 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a680aa670 con 0x562c4af372f0 2024-07-26T07:52:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.644+0000 7f6a66a00640 1 -- 172.21.15.97:0/1291105853 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68020040 con 0x562c4af372f0 2024-07-26T07:52:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.645+0000 7f6a841dc740 1 -- 172.21.15.97:0/1291105853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 msgr2=0x562c4b3007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.645+0000 7f6a841dc740 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3007c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6a680e1230 tx=0x7f6a68005bb0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.645+0000 7f6a841dc740 1 -- 172.21.15.97:0/1291105853 shutdown_connections 2024-07-26T07:52:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.645+0000 7f6a841dc740 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af601b0 0x562c4b322b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.645+0000 7f6a841dc740 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af3f4d0 0x562c4b300d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.645+0000 7f6a841dc740 1 --2- 172.21.15.97:0/1291105853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3007c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.645+0000 7f6a841dc740 1 -- 172.21.15.97:0/1291105853 >> 172.21.15.97:0/1291105853 conn(0x562c4af54300 msgr2=0x562c4b3000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.646+0000 7f6a841dc740 1 -- 172.21.15.97:0/1291105853 shutdown_connections 2024-07-26T07:52:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.646+0000 7f6a841dc740 1 -- 172.21.15.97:0/1291105853 wait complete. 2024-07-26T07:52:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.646+0000 7f6a841dc740 1 -- start start 2024-07-26T07:52:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.647+0000 7f6a841dc740 1 Processor -- start 2024-07-26T07:52:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.647+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3178d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.647+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af3f4d0 0x562c4b381af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.647+0000 7f6a841dc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b384150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.647+0000 7f6a841dc740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6a781b2480 con 0x562c4af3f4d0 2024-07-26T07:52:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.647+0000 7f6a841dc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6a781b1db0 con 0x562c4af601b0 2024-07-26T07:52:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.647+0000 7f6a841dc740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6a781aeff0 con 0x562c4af372f0 2024-07-26T07:52:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a841dc740 10 client.0 _ll_register_callbacks cb 0x7f6a7000ab10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af3f4d0 0x562c4b381af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af3f4d0 0x562c4b381af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56646/0 (socket says 172.21.15.97:56646) 2024-07-26T07:52:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a6fe00640 1 -- 172.21.15.97:0/3195251626 learned_addr learned my addr 172.21.15.97:0/3195251626 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:21.650 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a75e00640 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3178d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a76800640 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b384150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a6fe00640 1 -- 172.21.15.97:0/3195251626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 msgr2=0x562c4b3178d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a6fe00640 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3178d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a6fe00640 1 -- 172.21.15.97:0/3195251626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 msgr2=0x562c4b384150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a6fe00640 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b384150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.648+0000 7f6a6fe00640 1 -- 172.21.15.97:0/3195251626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c4b3848e0 con 0x562c4af3f4d0 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.649+0000 7f6a6fe00640 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af3f4d0 0x562c4b381af0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a60004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb006628e267e392 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.649+0000 7f6a5be00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.649+0000 7f6a5be00640 1 -- 172.21.15.97:0/3195251626 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60053060 con 0x562c4af3f4d0 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.649+0000 7f6a5be00640 1 -- 172.21.15.97:0/3195251626 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f6a60003660 con 0x562c4af3f4d0 2024-07-26T07:52:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.649+0000 7f6a841dc740 1 -- 172.21.15.97:0/3195251626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a680aa670 con 0x562c4af3f4d0 2024-07-26T07:52:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.650+0000 7f6a5be00640 1 -- 172.21.15.97:0/3195251626 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6a60035040 con 0x562c4af3f4d0 2024-07-26T07:52:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.650+0000 7f6a841dc740 10 client.15525 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.650+0000 7f6a841dc740 1 -- 172.21.15.97:0/3195251626 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562c4b385c10 con 0x562c4af3f4d0 2024-07-26T07:52:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.650+0000 7f6a5be00640 1 -- 172.21.15.97:0/3195251626 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6a680aa670 con 0x562c4af3f4d0 2024-07-26T07:52:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.651+0000 7f6a5be00640 1 -- 172.21.15.97:0/3195251626 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6a60057030 con 0x562c4af3f4d0 2024-07-26T07:52:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.651+0000 7f6a841dc740 10 client.15525 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.651+0000 7f6a841dc740 10 client.15525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:52:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.651+0000 7f6a841dc740 20 client.15525 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.651+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562c4aef41c0 0x562c4b3881c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.652+0000 7f6a841dc740 4 client.15525 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:52:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.652+0000 7f6a841dc740 1 -- 172.21.15.97:0/3195251626 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c4b3074b0 con 0x562c4aef41c0 2024-07-26T07:52:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.652+0000 7f6a75e00640 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562c4aef41c0 0x562c4b3881c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.653+0000 7f6a75e00640 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562c4aef41c0 0x562c4b3881c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d658e7ff7491dfe server_cookie=b75a2b7aa4534bd2 in_seq=0 out_seq=0 2024-07-26T07:52:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.653+0000 7f6a5be00640 10 client.15525 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.675+0000 7f6a5be00640 1 -- 172.21.15.97:0/3195251626 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c4b3074b0 con 0x562c4aef41c0 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.675+0000 7f6a5be00640 10 client.15525 handle_command_reply: tid=0 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.675+0000 7f6a841dc740 1 client.15525 shutdown 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 -- 172.21.15.97:0/3195251626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af3f4d0 msgr2=0x562c4b381af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af3f4d0 0x562c4b381af0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a60004260 comp rx=0 tx=0).stop 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 -- 172.21.15.97:0/3195251626 shutdown_connections 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562c4aef41c0 0x562c4b3881c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562c4af601b0 0x562c4b384150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562c4af3f4d0 0x562c4b381af0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 --2- 172.21.15.97:0/3195251626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c4af372f0 0x562c4b3178d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 -- 172.21.15.97:0/3195251626 >> 172.21.15.97:0/3195251626 conn(0x562c4af54300 msgr2=0x562c4b300a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a5be00640 0 client.15525 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 -- 172.21.15.97:0/3195251626 shutdown_connections 2024-07-26T07:52:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 1 -- 172.21.15.97:0/3195251626 wait complete. 2024-07-26T07:52:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.676+0000 7f6a841dc740 20 client.15525 trim_cache size 0 max 16384 2024-07-26T07:52:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:52:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:52:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:52:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:52:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 -- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a5c0b3c00 msgr2=0x7f6a5c0b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a5c0b3c00 0x7f6a5c0b6110 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000aec0 tx=0x7f6a7000a730 comp rx=0 tx=0).stop 2024-07-26T07:52:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 -- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a7806bee0 msgr2=0x7f6a7819fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a7806bee0 0x7f6a7819fee0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800d770 tx=0x7f6a6800dc30 comp rx=0 tx=0).stop 2024-07-26T07:52:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 -- 172.21.15.97:0/932140763 shutdown_connections 2024-07-26T07:52:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6a5c0b3c00 0x7f6a5c0b6110 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a78106c30 0x7f6a781a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78104070 0x7f6a781a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 --2- 172.21.15.97:0/932140763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a7806bee0 0x7f6a7819fee0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.677+0000 7f6a7d200640 1 -- 172.21.15.97:0/932140763 >> 172.21.15.97:0/932140763 conn(0x7f6a780fd960 msgr2=0x7f6a78105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.678+0000 7f6a7d200640 1 -- 172.21.15.97:0/932140763 shutdown_connections 2024-07-26T07:52:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:21.678+0000 7f6a7d200640 1 -- 172.21.15.97:0/932140763 wait complete. 2024-07-26T07:52:21.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:21 smithi097 ceph-mon[32231]: pgmap v527: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-07-26T07:52:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:21 smithi115 ceph-mon[35364]: pgmap v527: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-07-26T07:52:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:21 smithi181 ceph-mon[35869]: pgmap v527: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-07-26T07:52:22.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 -- 172.21.15.97:0/3478707452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec102760 msgr2=0x7f63ec10f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 --2- 172.21.15.97:0/3478707452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec102760 0x7f63ec10f480 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009900 tx=0x7f63d802a480 comp rx=0 tx=0).stop 2024-07-26T07:52:22.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 -- 172.21.15.97:0/3478707452 shutdown_connections 2024-07-26T07:52:22.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 --2- 172.21.15.97:0/3478707452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec10f9c0 0x7f63ec111de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 --2- 172.21.15.97:0/3478707452 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec102760 0x7f63ec10f480 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 --2- 172.21.15.97:0/3478707452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63ec101e10 0x7f63ec102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 -- 172.21.15.97:0/3478707452 >> 172.21.15.97:0/3478707452 conn(0x7f63ec0fdcf0 msgr2=0x7f63ec100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:22.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 -- 172.21.15.97:0/3478707452 shutdown_connections 2024-07-26T07:52:22.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.905+0000 7f63ebe00640 1 -- 172.21.15.97:0/3478707452 wait complete. 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.906+0000 7f63ebe00640 1 Processor -- start 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63ebe00640 1 -- start start 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63ebe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec101e10 0x7f63ec19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63ebe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec102760 0x7f63ec19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63ebe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63ec10f9c0 0x7f63ec1a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63ebe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec115380 con 0x7f63ec101e10 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63ebe00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec115200 con 0x7f63ec102760 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63ebe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec115500 con 0x7f63ec10f9c0 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63eaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec101e10 0x7f63ec19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63eaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec101e10 0x7f63ec19bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56676/0 (socket says 172.21.15.97:56676) 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.907+0000 7f63eaa00640 1 -- 172.21.15.97:0/1769327065 learned_addr learned my addr 172.21.15.97:0/1769327065 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.908+0000 7f63ea000640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec102760 0x7f63ec19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.908+0000 7f63eb400640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63ec10f9c0 0x7f63ec1a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.908+0000 7f63eaa00640 1 -- 172.21.15.97:0/1769327065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63ec10f9c0 msgr2=0x7f63ec1a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.908+0000 7f63eaa00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63ec10f9c0 0x7f63ec1a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.908+0000 7f63eaa00640 1 -- 172.21.15.97:0/1769327065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec102760 msgr2=0x7f63ec19c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.908+0000 7f63eaa00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec102760 0x7f63ec19c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.908+0000 7f63eaa00640 1 -- 172.21.15.97:0/1769327065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ec1a1c40 con 0x7f63ec101e10 2024-07-26T07:52:22.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.908+0000 7f63eaa00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec101e10 0x7f63ec19bd90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f63dc00c920 tx=0x7f63dc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:22.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.909+0000 7f63e3400640 1 -- 172.21.15.97:0/1769327065 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63dc007c70 con 0x7f63ec101e10 2024-07-26T07:52:22.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.909+0000 7f63e3400640 1 -- 172.21.15.97:0/1769327065 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63dc007e10 con 0x7f63ec101e10 2024-07-26T07:52:22.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.909+0000 7f63e3400640 1 -- 172.21.15.97:0/1769327065 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63dc018710 con 0x7f63ec101e10 2024-07-26T07:52:22.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.909+0000 7f63ebe00640 1 -- 172.21.15.97:0/1769327065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ec1a1f30 con 0x7f63ec101e10 2024-07-26T07:52:22.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.909+0000 7f63ebe00640 1 -- 172.21.15.97:0/1769327065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ec1a23f0 con 0x7f63ec101e10 2024-07-26T07:52:22.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.912+0000 7f63f83f1740 1 Processor -- start 2024-07-26T07:52:22.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.913+0000 7f63f83f1740 1 -- start start 2024-07-26T07:52:22.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.913+0000 7f63e3400640 1 -- 172.21.15.97:0/1769327065 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f63dc00f450 con 0x7f63ec101e10 2024-07-26T07:52:22.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.913+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b9592b7f60 0x55b95965d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.913+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b95928f0a0 0x55b95965dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.913+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95965e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.913+0000 7f63f83f1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b9595d3970 con 0x55b959297280 2024-07-26T07:52:22.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.913+0000 7f63f83f1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b9595c2e70 con 0x55b9592b7f60 2024-07-26T07:52:22.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.913+0000 7f63f83f1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b9595c6eb0 con 0x55b95928f0a0 2024-07-26T07:52:22.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.914+0000 7f63ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95965e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.914+0000 7f63ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95965e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56678/0 (socket says 172.21.15.97:56678) 2024-07-26T07:52:22.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.914+0000 7f63ea000640 1 -- 172.21.15.97:0/1386982794 learned_addr learned my addr 172.21.15.97:0/1386982794 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:22.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.914+0000 7f63eaa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b95928f0a0 0x55b95965dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.914+0000 7f63eb400640 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b9592b7f60 0x55b95965d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.915+0000 7f63ea000640 1 -- 172.21.15.97:0/1386982794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b95928f0a0 msgr2=0x55b95965dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.915+0000 7f63ea000640 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b95928f0a0 0x55b95965dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.915+0000 7f63ea000640 1 -- 172.21.15.97:0/1386982794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b9592b7f60 msgr2=0x55b95965d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.915+0000 7f63ea000640 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b9592b7f60 0x55b95965d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.915+0000 7f63ea000640 1 -- 172.21.15.97:0/1386982794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95956a110 con 0x55b959297280 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.915+0000 7f63eb400640 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b9592b7f60 0x55b95965d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.916+0000 7f63e3400640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63d00b3c90 0x7f63d00b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.916+0000 7f63e3400640 1 -- 172.21.15.97:0/1769327065 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f63dc0e3f90 con 0x7f63ec101e10 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.916+0000 7f63ea000640 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95965e200 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009900 tx=0x7f63d8004390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddb79c5d0c2dbc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.916+0000 7f63eb400640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63d00b3c90 0x7f63d00b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.916+0000 7f63d7e00640 1 -- 172.21.15.97:0/1386982794 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8035030 con 0x55b959297280 2024-07-26T07:52:22.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.916+0000 7f63d7e00640 1 -- 172.21.15.97:0/1386982794 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63d8040070 con 0x55b959297280 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.916+0000 7f63d7e00640 1 -- 172.21.15.97:0/1386982794 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8045ab0 con 0x55b959297280 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 -- 172.21.15.97:0/1386982794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 msgr2=0x55b95965e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63eb400640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63d00b3c90 0x7f63d00b61a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f63e400ae90 tx=0x7f63e400a440 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95965e200 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009900 tx=0x7f63d8004390 comp rx=0 tx=0).stop 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 -- 172.21.15.97:0/1386982794 shutdown_connections 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95965e200 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b95928f0a0 0x55b95965dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1386982794 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b9592b7f60 0x55b95965d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 -- 172.21.15.97:0/1386982794 >> 172.21.15.97:0/1386982794 conn(0x55b9592ac350 msgr2=0x55b95965b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 -- 172.21.15.97:0/1386982794 shutdown_connections 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.918+0000 7f63f83f1740 1 -- 172.21.15.97:0/1386982794 wait complete. 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.919+0000 7f63f83f1740 1 -- start start 2024-07-26T07:52:22.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.919+0000 7f63f83f1740 1 Processor -- start 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.920+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b959669270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.920+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b9596697b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.920+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959675220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.920+0000 7f63f83f1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b9594265f0 con 0x55b959297280 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.920+0000 7f63f83f1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b95955f540 con 0x55b95928f0a0 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.920+0000 7f63f83f1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b95966bf00 con 0x55b9592b7f60 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.920+0000 7f63f83f1740 10 client.0 _ll_register_callbacks cb 0x55b95956ef10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.920+0000 7f63ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b9596697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b9596697b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56684/0 (socket says 172.21.15.97:56684) 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63ea000640 1 -- 172.21.15.97:0/1299780730 learned_addr learned my addr 172.21.15.97:0/1299780730 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63eaa00640 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b959669270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63eb400640 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959675220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63ea000640 1 -- 172.21.15.97:0/1299780730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 msgr2=0x55b959675220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63ea000640 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959675220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63ea000640 1 -- 172.21.15.97:0/1299780730 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 msgr2=0x55b959669270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63ea000640 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b959669270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.921+0000 7f63ea000640 1 -- 172.21.15.97:0/1299780730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b959323540 con 0x55b959297280 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.922+0000 7f63ea000640 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b9596697b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f63d802a4c0 tx=0x7f63d8045ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84e7a1d8f2fc2fb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.922+0000 7f63d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.922+0000 7f63d4c00640 1 -- 172.21.15.97:0/1299780730 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8035070 con 0x55b959297280 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.922+0000 7f63d4c00640 1 -- 172.21.15.97:0/1299780730 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f63d8003a00 con 0x55b959297280 2024-07-26T07:52:22.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.922+0000 7f63f83f1740 1 -- 172.21.15.97:0/1299780730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d8040070 con 0x55b959297280 2024-07-26T07:52:22.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.922+0000 7f63d4c00640 1 -- 172.21.15.97:0/1299780730 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f63d8003c00 con 0x55b959297280 2024-07-26T07:52:22.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.922+0000 7f63f83f1740 10 client.15537 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:22.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.922+0000 7f63f83f1740 1 -- 172.21.15.97:0/1299780730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b959677de0 con 0x55b959297280 2024-07-26T07:52:22.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.923+0000 7f63d4c00640 1 -- 172.21.15.97:0/1299780730 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63d8040070 con 0x55b959297280 2024-07-26T07:52:22.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.923+0000 7f63d4c00640 1 -- 172.21.15.97:0/1299780730 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f63d803ccc0 con 0x55b959297280 2024-07-26T07:52:22.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.924+0000 7f63f83f1740 10 client.15537 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:22.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.924+0000 7f63f83f1740 10 client.15537 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T07:52:22.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.924+0000 7f63f83f1740 20 client.15537 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.924+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x55b9592768e0 0x55b95967b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.924+0000 7f63f83f1740 4 client.15537 mds_command: new command op to 34556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.924+0000 7f63f83f1740 1 -- 172.21.15.97:0/1299780730 --> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9595d58c0 con 0x55b9592768e0 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.924+0000 7f63eaa00640 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x55b9592768e0 0x55b95967b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.925+0000 7f63eaa00640 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x55b9592768e0 0x55b95967b320 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d8c817010db72fd server_cookie=5800ac1748432629 in_seq=0 out_seq=0 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.925+0000 7f63d4c00640 10 client.15537 ms_handle_connect on v2:172.21.15.115:6836/3478625369 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.927+0000 7f63d4c00640 1 -- 172.21.15.97:0/1299780730 <== mds.1 v2:172.21.15.115:6836/3478625369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b9595d58c0 con 0x55b9592768e0 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.927+0000 7f63d4c00640 10 client.15537 handle_command_reply: tid=0 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.927+0000 7f63f83f1740 1 client.15537 shutdown 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 -- 172.21.15.97:0/1299780730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 msgr2=0x55b9596697b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b9596697b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f63d802a4c0 tx=0x7f63d8045ae0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 -- 172.21.15.97:0/1299780730 shutdown_connections 2024-07-26T07:52:22.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x55b9592768e0 0x55b95967b320 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959675220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b9596697b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63d4c00640 0 client.15537 ms_handle_reset on v2:172.21.15.115:6836/3478625369 2024-07-26T07:52:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1299780730 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b959669270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 -- 172.21.15.97:0/1299780730 >> 172.21.15.97:0/1299780730 conn(0x55b9592ac350 msgr2=0x55b959660910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 -- 172.21.15.97:0/1299780730 shutdown_connections 2024-07-26T07:52:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 1 -- 172.21.15.97:0/1299780730 wait complete. 2024-07-26T07:52:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.928+0000 7f63f83f1740 20 client.15537 trim_cache size 0 max 16384 2024-07-26T07:52:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.962+0000 7f63f83f1740 1 Processor -- start 2024-07-26T07:52:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.962+0000 7f63f83f1740 1 -- start start 2024-07-26T07:52:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b9596797e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959663870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63f83f1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b959679660 con 0x55b959297280 2024-07-26T07:52:22.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63f83f1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b959679d30 con 0x55b95928f0a0 2024-07-26T07:52:22.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63f83f1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b95967a400 con 0x55b9592b7f60 2024-07-26T07:52:22.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56694/0 (socket says 172.21.15.97:56694) 2024-07-26T07:52:22.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.963+0000 7f63ea000640 1 -- 172.21.15.97:0/1217771827 learned_addr learned my addr 172.21.15.97:0/1217771827 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:22.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63eb400640 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959663870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63eaa00640 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b9596797e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63ea000640 1 -- 172.21.15.97:0/1217771827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 msgr2=0x55b959663870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63ea000640 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959663870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63ea000640 1 -- 172.21.15.97:0/1217771827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 msgr2=0x55b9596797e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63ea000640 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b9596797e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63ea000640 1 -- 172.21.15.97:0/1217771827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9596567f0 con 0x55b959297280 2024-07-26T07:52:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63eb400640 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959663870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:52:22.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.964+0000 7f63ea000640 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967b320 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f63d8004650 tx=0x7f63d8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20000ad3c58dffc2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:22.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.965+0000 7f63d7400640 1 -- 172.21.15.97:0/1217771827 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8035040 con 0x55b959297280 2024-07-26T07:52:22.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.965+0000 7f63d7400640 1 -- 172.21.15.97:0/1217771827 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63d8040070 con 0x55b959297280 2024-07-26T07:52:22.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.965+0000 7f63d7400640 1 -- 172.21.15.97:0/1217771827 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8003a60 con 0x55b959297280 2024-07-26T07:52:22.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.966+0000 7f63f83f1740 1 -- 172.21.15.97:0/1217771827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 msgr2=0x55b95967b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.966+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967b320 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f63d8004650 tx=0x7f63d8000bc0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.966+0000 7f63f83f1740 1 -- 172.21.15.97:0/1217771827 shutdown_connections 2024-07-26T07:52:22.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.966+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959663870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.966+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967b320 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.966+0000 7f63f83f1740 1 --2- 172.21.15.97:0/1217771827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b9596797e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.966+0000 7f63f83f1740 1 -- 172.21.15.97:0/1217771827 >> 172.21.15.97:0/1217771827 conn(0x55b9592ac350 msgr2=0x55b95966a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:22.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.967+0000 7f63f83f1740 1 -- 172.21.15.97:0/1217771827 shutdown_connections 2024-07-26T07:52:22.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.967+0000 7f63f83f1740 1 -- 172.21.15.97:0/1217771827 wait complete. 2024-07-26T07:52:22.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.967+0000 7f63f83f1740 1 -- start start 2024-07-26T07:52:22.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.967+0000 7f63f83f1740 1 Processor -- start 2024-07-26T07:52:22.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b95967ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63f83f1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959682130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63f83f1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b95967b1a0 con 0x55b959297280 2024-07-26T07:52:22.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63f83f1740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b95967aad0 con 0x55b95928f0a0 2024-07-26T07:52:22.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63f83f1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b95967b870 con 0x55b9592b7f60 2024-07-26T07:52:22.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63f83f1740 10 client.0 _ll_register_callbacks cb 0x7f63d8004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:22.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63ea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56698/0 (socket says 172.21.15.97:56698) 2024-07-26T07:52:22.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.968+0000 7f63ea000640 1 -- 172.21.15.97:0/2553698695 learned_addr learned my addr 172.21.15.97:0/2553698695 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:22.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63eaa00640 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b95967ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63ea000640 1 -- 172.21.15.97:0/2553698695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 msgr2=0x55b959682130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:22.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63ea000640 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959682130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63ea000640 1 -- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 msgr2=0x55b95967ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:22.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63ea000640 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b95967ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63ea000640 1 -- 172.21.15.97:0/2553698695 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b959659a30 con 0x55b959297280 2024-07-26T07:52:22.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63eaa00640 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b95967ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:52:22.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63ea000640 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967d280 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f63d8005e50 tx=0x7f63d8003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b99f1804140c88 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:22.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.969+0000 7f63d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:22.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.970+0000 7f63d4c00640 1 -- 172.21.15.97:0/2553698695 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8035070 con 0x55b959297280 2024-07-26T07:52:22.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.970+0000 7f63d4c00640 1 -- 172.21.15.97:0/2553698695 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f63d804e820 con 0x55b959297280 2024-07-26T07:52:22.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.970+0000 7f63f83f1740 1 -- 172.21.15.97:0/2553698695 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d8040070 con 0x55b959297280 2024-07-26T07:52:22.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.970+0000 7f63d4c00640 1 -- 172.21.15.97:0/2553698695 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f63d804ea20 con 0x55b959297280 2024-07-26T07:52:22.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.970+0000 7f63f83f1740 10 client.15543 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:22.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.970+0000 7f63f83f1740 1 -- 172.21.15.97:0/2553698695 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b959683990 con 0x55b959297280 2024-07-26T07:52:22.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.971+0000 7f63d4c00640 1 -- 172.21.15.97:0/2553698695 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63d8040070 con 0x55b959297280 2024-07-26T07:52:22.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.971+0000 7f63d4c00640 1 -- 172.21.15.97:0/2553698695 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f63d803cc00 con 0x55b959297280 2024-07-26T07:52:22.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.971+0000 7f63f83f1740 10 client.15543 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:22.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.971+0000 7f63f83f1740 10 client.15543 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T07:52:22.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.971+0000 7f63f83f1740 20 client.15543 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:22.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.971+0000 7f63f83f1740 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x55b95924bc90 0x55b9596e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:22.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.971+0000 7f63f83f1740 4 client.15543 mds_command: new command op to 34556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:52:22.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.971+0000 7f63f83f1740 1 -- 172.21.15.97:0/2553698695 --> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] -- command(tid 0: {"prefix": "status"}) -- 0x55b9595d1360 con 0x55b95924bc90 2024-07-26T07:52:22.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.972+0000 7f63eaa00640 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x55b95924bc90 0x55b9596e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:22.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.973+0000 7f63eaa00640 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x55b95924bc90 0x55b9596e6e70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aec062fbf6465149 server_cookie=24ec4c7c17723904 in_seq=0 out_seq=0 2024-07-26T07:52:22.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.973+0000 7f63d4c00640 10 client.15543 ms_handle_connect on v2:172.21.15.115:6836/3478625369 2024-07-26T07:52:22.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.973+0000 7f63d4c00640 1 -- 172.21.15.97:0/2553698695 <== mds.1 v2:172.21.15.115:6836/3478625369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55b9595d1360 con 0x55b95924bc90 2024-07-26T07:52:22.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.973+0000 7f63d4c00640 10 client.15543 handle_command_reply: tid=0 2024-07-26T07:52:22.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.973+0000 7f63f83f1740 1 client.15543 shutdown 2024-07-26T07:52:22.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 -- 172.21.15.97:0/2553698695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 msgr2=0x55b95967d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967d280 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f63d8005e50 tx=0x7f63d8003fd0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 -- 172.21.15.97:0/2553698695 shutdown_connections 2024-07-26T07:52:22.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:6836/3478625369,v1:172.21.15.115:6837/3478625369] conn(0x55b95924bc90 0x55b9596e6e70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9592b7f60 0x55b959682130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b959297280 0x55b95967d280 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 --2- 172.21.15.97:0/2553698695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b95928f0a0 0x55b95967ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63d4c00640 0 client.15543 ms_handle_reset on v2:172.21.15.115:6836/3478625369 2024-07-26T07:52:22.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 -- 172.21.15.97:0/2553698695 >> 172.21.15.97:0/2553698695 conn(0x55b9592ac350 msgr2=0x55b959659c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:22.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.974+0000 7f63f83f1740 1 -- 172.21.15.97:0/2553698695 shutdown_connections 2024-07-26T07:52:22.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.975+0000 7f63f83f1740 1 -- 172.21.15.97:0/2553698695 wait complete. 2024-07-26T07:52:22.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.975+0000 7f63f83f1740 20 client.15543 trim_cache size 0 max 16384 2024-07-26T07:52:22.980 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 1, 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34556, 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 259.230346553, 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 86, 2024-07-26T07:52:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 311.157615959 2024-07-26T07:52:22.982 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:52:22.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.975+0000 7f63ebe00640 1 -- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63d00b3c90 msgr2=0x7f63d00b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.975+0000 7f63ebe00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63d00b3c90 0x7f63d00b61a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f63e400ae90 tx=0x7f63e400a440 comp rx=0 tx=0).stop 2024-07-26T07:52:22.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.975+0000 7f63ebe00640 1 -- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec101e10 msgr2=0x7f63ec19bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:22.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.975+0000 7f63ebe00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec101e10 0x7f63ec19bd90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f63dc00c920 tx=0x7f63dc00cde0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.976+0000 7f63ebe00640 1 -- 172.21.15.97:0/1769327065 shutdown_connections 2024-07-26T07:52:22.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.976+0000 7f63ebe00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f63d00b3c90 0x7f63d00b61a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.976+0000 7f63ebe00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63ec10f9c0 0x7f63ec1a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.976+0000 7f63ebe00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec102760 0x7f63ec19c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.976+0000 7f63ebe00640 1 --2- 172.21.15.97:0/1769327065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63ec101e10 0x7f63ec19bd90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:22.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.976+0000 7f63ebe00640 1 -- 172.21.15.97:0/1769327065 >> 172.21.15.97:0/1769327065 conn(0x7f63ec0fdcf0 msgr2=0x7f63ec111b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:22.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.977+0000 7f63ebe00640 1 -- 172.21.15.97:0/1769327065 shutdown_connections 2024-07-26T07:52:22.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:22.977+0000 7f63ebe00640 1 -- 172.21.15.97:0/1769327065 wait complete. 2024-07-26T07:52:23.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph tell mds.b status 2024-07-26T07:52:24.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:23 smithi097 ceph-mon[32231]: pgmap v528: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-07-26T07:52:24.175 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:23 smithi115 ceph-mon[35364]: pgmap v528: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-07-26T07:52:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:23 smithi181 ceph-mon[35869]: pgmap v528: 97 pgs: 97 active+clean; 677 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-07-26T07:52:26.194 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:25 smithi097 ceph-mon[32231]: pgmap v529: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-07-26T07:52:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:25 smithi115 ceph-mon[35364]: pgmap v529: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-07-26T07:52:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:25 smithi181 ceph-mon[35869]: pgmap v529: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-07-26T07:52:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 -- 172.21.15.97:0/3851181609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c104220 msgr2=0x7f045c104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 --2- 172.21.15.97:0/3851181609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c104220 0x7f045c104630 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f044c009900 tx=0x7f044c02a410 comp rx=0 tx=0).stop 2024-07-26T07:52:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 -- 172.21.15.97:0/3851181609 shutdown_connections 2024-07-26T07:52:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 --2- 172.21.15.97:0/3851181609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f045c10d680 0x7f045c10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 --2- 172.21.15.97:0/3851181609 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f045c104b70 0x7f045c10d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 --2- 172.21.15.97:0/3851181609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c104220 0x7f045c104630 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 -- 172.21.15.97:0/3851181609 >> 172.21.15.97:0/3851181609 conn(0x7f045c0fdd20 msgr2=0x7f045c100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:26.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 -- 172.21.15.97:0/3851181609 shutdown_connections 2024-07-26T07:52:26.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.398+0000 7f0461c00640 1 -- 172.21.15.97:0/3851181609 wait complete. 2024-07-26T07:52:26.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.399+0000 7f0461c00640 1 Processor -- start 2024-07-26T07:52:26.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.399+0000 7f0461c00640 1 -- start start 2024-07-26T07:52:26.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f0461c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f045c104220 0x7f045c101f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f0461c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f045c104b70 0x7f045c1024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f0461c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c10d680 0x7f045c1029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f0461c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f045c113210 con 0x7f045c10d680 2024-07-26T07:52:26.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f0461c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f045c113090 con 0x7f045c104220 2024-07-26T07:52:26.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f0461c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f045c113390 con 0x7f045c104b70 2024-07-26T07:52:26.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f045b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c10d680 0x7f045c1029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f045b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c10d680 0x7f045c1029e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54184/0 (socket says 172.21.15.97:54184) 2024-07-26T07:52:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f045b400640 1 -- 172.21.15.97:0/929361373 learned_addr learned my addr 172.21.15.97:0/929361373 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f045a000640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f045c104b70 0x7f045c1024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.400+0000 7f045aa00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f045c104220 0x7f045c101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.401+0000 7f045b400640 1 -- 172.21.15.97:0/929361373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f045c104b70 msgr2=0x7f045c1024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.401+0000 7f045b400640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f045c104b70 0x7f045c1024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.401+0000 7f045b400640 1 -- 172.21.15.97:0/929361373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f045c104220 msgr2=0x7f045c101f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.401+0000 7f045b400640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f045c104220 0x7f045c101f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.401+0000 7f045b400640 1 -- 172.21.15.97:0/929361373 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c1a7f40 con 0x7f045c10d680 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.401+0000 7f045b400640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c10d680 0x7f045c1029e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f045400eaa0 tx=0x7f045400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.402+0000 7f0453400640 1 -- 172.21.15.97:0/929361373 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045400cdb0 con 0x7f045c10d680 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.402+0000 7f0453400640 1 -- 172.21.15.97:0/929361373 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0454005020 con 0x7f045c10d680 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.402+0000 7f0453400640 1 -- 172.21.15.97:0/929361373 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04540116d0 con 0x7f045c10d680 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.402+0000 7f0461c00640 1 -- 172.21.15.97:0/929361373 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045c1a8230 con 0x7f045c10d680 2024-07-26T07:52:26.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.402+0000 7f0461c00640 1 -- 172.21.15.97:0/929361373 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f045c1a8720 con 0x7f045c10d680 2024-07-26T07:52:26.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.406+0000 7f0468863740 1 Processor -- start 2024-07-26T07:52:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.407+0000 7f0453400640 1 -- 172.21.15.97:0/929361373 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f04540051d0 con 0x7f045c10d680 2024-07-26T07:52:26.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.407+0000 7f0468863740 1 -- start start 2024-07-26T07:52:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.407+0000 7f0468863740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acd0f60 0x55855b076760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f0468863740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 0x55855b076ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f0468863740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b0771e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f0468863740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55855afec950 con 0x55855acd0f60 2024-07-26T07:52:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f0468863740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55855afdbe50 con 0x55855acb0280 2024-07-26T07:52:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f0468863740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55855afdfe90 con 0x55855aca80a0 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acd0f60 0x55855b076760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acd0f60 0x55855b076760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54194/0 (socket says 172.21.15.97:54194) 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045a000640 1 -- 172.21.15.97:0/627740220 learned_addr learned my addr 172.21.15.97:0/627740220 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045aa00640 1 --2- 172.21.15.97:0/627740220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 0x55855b076ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045a000640 1 -- 172.21.15.97:0/627740220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 msgr2=0x55855b076ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045a000640 1 --2- 172.21.15.97:0/627740220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 0x55855b076ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045a000640 1 -- 172.21.15.97:0/627740220 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 msgr2=0x55855b0771e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045a000640 1 --2- 172.21.15.97:0/627740220 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b0771e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.408+0000 7f045a000640 1 -- 172.21.15.97:0/627740220 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55855af830f0 con 0x55855acd0f60 2024-07-26T07:52:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.409+0000 7f0453400640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f04400b3ce0 0x7f04400b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.409+0000 7f0453400640 1 -- 172.21.15.97:0/929361373 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f04540e3f00 con 0x7f045c10d680 2024-07-26T07:52:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.409+0000 7f045a000640 1 --2- 172.21.15.97:0/627740220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acd0f60 0x55855b076760 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f044800bcf0 tx=0x7f04480107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12044f6c69e3ceff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.409+0000 7f045aa00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f04400b3ce0 0x7f04400b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.409+0000 7f0447e00640 1 -- 172.21.15.97:0/627740220 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448004030 con 0x55855acd0f60 2024-07-26T07:52:26.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.409+0000 7f0447e00640 1 -- 172.21.15.97:0/627740220 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0448005020 con 0x55855acd0f60 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.410+0000 7f0447e00640 1 -- 172.21.15.97:0/627740220 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448004c40 con 0x55855acd0f60 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.410+0000 7f0468863740 1 -- 172.21.15.97:0/627740220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acd0f60 msgr2=0x55855b076760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.410+0000 7f045aa00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f04400b3ce0 0x7f04400b61f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f045c1039b0 tx=0x7f044c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.410+0000 7f0468863740 1 --2- 172.21.15.97:0/627740220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acd0f60 0x55855b076760 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f044800bcf0 tx=0x7f04480107a0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.411+0000 7f0468863740 1 -- 172.21.15.97:0/627740220 shutdown_connections 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.411+0000 7f0468863740 1 --2- 172.21.15.97:0/627740220 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b0771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.411+0000 7f0468863740 1 --2- 172.21.15.97:0/627740220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 0x55855b076ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.411+0000 7f0468863740 1 --2- 172.21.15.97:0/627740220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acd0f60 0x55855b076760 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.411+0000 7f0468863740 1 -- 172.21.15.97:0/627740220 >> 172.21.15.97:0/627740220 conn(0x55855acc5350 msgr2=0x55855b074650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.411+0000 7f0468863740 1 -- 172.21.15.97:0/627740220 shutdown_connections 2024-07-26T07:52:26.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.411+0000 7f0468863740 1 -- 172.21.15.97:0/627740220 wait complete. 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.411+0000 7f0468863740 1 -- start start 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.412+0000 7f0468863740 1 Processor -- start 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.412+0000 7f0468863740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b0822e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.412+0000 7f0468863740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b082820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.412+0000 7f0468863740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b08e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.412+0000 7f0468863740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55855ae3f5f0 con 0x55855aca80a0 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.412+0000 7f0468863740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55855af78520 con 0x55855acb0280 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.412+0000 7f0468863740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55855b084f40 con 0x55855acd0f60 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.412+0000 7f0468863740 10 client.0 _ll_register_callbacks cb 0x55855af87ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b0822e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b0822e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54204/0 (socket says 172.21.15.97:54204) 2024-07-26T07:52:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045aa00640 1 -- 172.21.15.97:0/880155691 learned_addr learned my addr 172.21.15.97:0/880155691 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045a000640 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b082820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045b400640 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b08e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045aa00640 1 -- 172.21.15.97:0/880155691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 msgr2=0x55855b08e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045aa00640 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b08e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045aa00640 1 -- 172.21.15.97:0/880155691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 msgr2=0x55855b082820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045aa00640 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b082820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045aa00640 1 -- 172.21.15.97:0/880155691 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55855ad3c540 con 0x55855aca80a0 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.413+0000 7f045b400640 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b08e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.414+0000 7f045aa00640 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b0822e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f044c009900 tx=0x7f044c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d560cfc0384efa82 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.414+0000 7f0444c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.414+0000 7f0444c00640 1 -- 172.21.15.97:0/880155691 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c04d070 con 0x55855aca80a0 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.414+0000 7f0444c00640 1 -- 172.21.15.97:0/880155691 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f044c049450 con 0x55855aca80a0 2024-07-26T07:52:26.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.414+0000 7f0468863740 1 -- 172.21.15.97:0/880155691 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0448005020 con 0x55855aca80a0 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.414+0000 7f0444c00640 1 -- 172.21.15.97:0/880155691 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f044c059050 con 0x55855aca80a0 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.414+0000 7f0468863740 10 client.15555 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.414+0000 7f0468863740 1 -- 172.21.15.97:0/880155691 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55855b090f20 con 0x55855aca80a0 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.415+0000 7f0444c00640 1 -- 172.21.15.97:0/880155691 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0448005020 con 0x55855aca80a0 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.415+0000 7f0444c00640 1 -- 172.21.15.97:0/880155691 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f044c03c670 con 0x55855aca80a0 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.415+0000 7f0468863740 10 client.15555 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.415+0000 7f0468863740 10 client.15555 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.415+0000 7f0468863740 20 client.15555 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.415+0000 7f0468863740 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855ac8f8e0 0x55855b0949e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.415+0000 7f0468863740 4 client.15555 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.415+0000 7f0468863740 1 -- 172.21.15.97:0/880155691 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55855afee8a0 con 0x55855ac8f8e0 2024-07-26T07:52:26.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.416+0000 7f045a000640 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855ac8f8e0 0x55855b0949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.417+0000 7f045a000640 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855ac8f8e0 0x55855b0949e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb80c780b953cde0 server_cookie=c2fbedc34780dbdc in_seq=0 out_seq=0 2024-07-26T07:52:26.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.417+0000 7f0444c00640 10 client.15555 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.419+0000 7f0444c00640 1 -- 172.21.15.97:0/880155691 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55855afee8a0 con 0x55855ac8f8e0 2024-07-26T07:52:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.419+0000 7f0444c00640 10 client.15555 handle_command_reply: tid=0 2024-07-26T07:52:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.419+0000 7f0468863740 1 client.15555 shutdown 2024-07-26T07:52:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.419+0000 7f0468863740 1 -- 172.21.15.97:0/880155691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 msgr2=0x55855b0822e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.419+0000 7f0468863740 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b0822e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f044c009900 tx=0x7f044c02a410 comp rx=0 tx=0).stop 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.419+0000 7f0468863740 1 -- 172.21.15.97:0/880155691 shutdown_connections 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.419+0000 7f0468863740 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855ac8f8e0 0x55855b0949e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.420+0000 7f0468863740 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b08e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.420+0000 7f0468863740 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b082820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.420+0000 7f0468863740 1 --2- 172.21.15.97:0/880155691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b0822e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.420+0000 7f0468863740 1 -- 172.21.15.97:0/880155691 >> 172.21.15.97:0/880155691 conn(0x55855acc5350 msgr2=0x55855b079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.420+0000 7f0444c00640 0 client.15555 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.420+0000 7f0468863740 1 -- 172.21.15.97:0/880155691 shutdown_connections 2024-07-26T07:52:26.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.420+0000 7f0468863740 1 -- 172.21.15.97:0/880155691 wait complete. 2024-07-26T07:52:26.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.420+0000 7f0468863740 20 client.15555 trim_cache size 0 max 16384 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.439+0000 7f0468863740 1 Processor -- start 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.439+0000 7f0468863740 1 -- start start 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f0468863740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 0x55855b0815d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f0468863740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acb0280 0x55855b08e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f0468863740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acd0f60 0x55855b08e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f0468863740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55855b092d20 con 0x55855acb0280 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f0468863740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55855b0933f0 con 0x55855acd0f60 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f0468863740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55855b093ac0 con 0x55855aca80a0 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acb0280 0x55855b08e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acb0280 0x55855b08e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54214/0 (socket says 172.21.15.97:54214) 2024-07-26T07:52:26.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045a000640 1 -- 172.21.15.97:0/1507148040 learned_addr learned my addr 172.21.15.97:0/1507148040 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045aa00640 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 0x55855b0815d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045b400640 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acd0f60 0x55855b08e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045a000640 1 -- 172.21.15.97:0/1507148040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 msgr2=0x55855b0815d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045a000640 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 0x55855b0815d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045a000640 1 -- 172.21.15.97:0/1507148040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acd0f60 msgr2=0x55855b08e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.440+0000 7f045a000640 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acd0f60 0x55855b08e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.441+0000 7f045a000640 1 -- 172.21.15.97:0/1507148040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55855b06f7d0 con 0x55855acb0280 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.441+0000 7f045b400640 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acd0f60 0x55855b08e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:52:26.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.441+0000 7f045a000640 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acb0280 0x55855b08e1a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0448010ca0 tx=0x7f0448005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71696763b898edf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:26.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.441+0000 7f0447400640 1 -- 172.21.15.97:0/1507148040 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448004300 con 0x55855acb0280 2024-07-26T07:52:26.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.441+0000 7f0447400640 1 -- 172.21.15.97:0/1507148040 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f04480044a0 con 0x55855acb0280 2024-07-26T07:52:26.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.441+0000 7f0447400640 1 -- 172.21.15.97:0/1507148040 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04480118a0 con 0x55855acb0280 2024-07-26T07:52:26.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.442+0000 7f0468863740 1 -- 172.21.15.97:0/1507148040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acb0280 msgr2=0x55855b08e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.442+0000 7f0468863740 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acb0280 0x55855b08e1a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0448010ca0 tx=0x7f0448005850 comp rx=0 tx=0).stop 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.443+0000 7f0468863740 1 -- 172.21.15.97:0/1507148040 shutdown_connections 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.443+0000 7f0468863740 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acd0f60 0x55855b08e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.443+0000 7f0468863740 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855acb0280 0x55855b08e1a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.443+0000 7f0468863740 1 --2- 172.21.15.97:0/1507148040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855aca80a0 0x55855b0815d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.443+0000 7f0468863740 1 -- 172.21.15.97:0/1507148040 >> 172.21.15.97:0/1507148040 conn(0x55855acc5350 msgr2=0x55855b07d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.443+0000 7f0468863740 1 -- 172.21.15.97:0/1507148040 shutdown_connections 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.443+0000 7f0468863740 1 -- 172.21.15.97:0/1507148040 wait complete. 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 1 -- start start 2024-07-26T07:52:26.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 1 Processor -- start 2024-07-26T07:52:26.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b09d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b095be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b096120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55855b094860 con 0x55855aca80a0 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55855b094190 con 0x55855acb0280 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55855b094f30 con 0x55855acd0f60 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.444+0000 7f0468863740 10 client.0 _ll_register_callbacks cb 0x7f044c004500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b09d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b09d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54230/0 (socket says 172.21.15.97:54230) 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045aa00640 1 -- 172.21.15.97:0/3969842971 learned_addr learned my addr 172.21.15.97:0/3969842971 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045a000640 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b095be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045b400640 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b096120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045aa00640 1 -- 172.21.15.97:0/3969842971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 msgr2=0x55855b096120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:26.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045aa00640 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b096120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045aa00640 1 -- 172.21.15.97:0/3969842971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 msgr2=0x55855b095be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:26.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045aa00640 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b095be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.445+0000 7f045aa00640 1 -- 172.21.15.97:0/3969842971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55855b096810 con 0x55855aca80a0 2024-07-26T07:52:26.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.446+0000 7f045b400640 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b096120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:52:26.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.446+0000 7f045aa00640 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b09d2e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f044c0096c0 tx=0x7f044c003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d500c746f8b486 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:26.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.446+0000 7f0444c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:26.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.446+0000 7f0444c00640 1 -- 172.21.15.97:0/3969842971 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c04d070 con 0x55855aca80a0 2024-07-26T07:52:26.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.446+0000 7f0444c00640 1 -- 172.21.15.97:0/3969842971 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f044c049450 con 0x55855aca80a0 2024-07-26T07:52:26.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.446+0000 7f0468863740 1 -- 172.21.15.97:0/3969842971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04480044a0 con 0x55855aca80a0 2024-07-26T07:52:26.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.447+0000 7f0444c00640 1 -- 172.21.15.97:0/3969842971 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f044c035040 con 0x55855aca80a0 2024-07-26T07:52:26.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.447+0000 7f0468863740 10 client.15561 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:26.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.447+0000 7f0468863740 1 -- 172.21.15.97:0/3969842971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55855b0fca70 con 0x55855aca80a0 2024-07-26T07:52:26.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.448+0000 7f0444c00640 1 -- 172.21.15.97:0/3969842971 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f04480044a0 con 0x55855aca80a0 2024-07-26T07:52:26.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.448+0000 7f0444c00640 1 -- 172.21.15.97:0/3969842971 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f044c052440 con 0x55855aca80a0 2024-07-26T07:52:26.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.448+0000 7f0468863740 10 client.15561 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:26.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.448+0000 7f0468863740 10 client.15561 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T07:52:26.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.448+0000 7f0468863740 20 client.15561 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:26.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.448+0000 7f0468863740 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855ac64c90 0x55855b1003b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:26.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.448+0000 7f0468863740 4 client.15561 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T07:52:26.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.448+0000 7f0468863740 1 -- 172.21.15.97:0/3969842971 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "status"}) -- 0x55855afea340 con 0x55855ac64c90 2024-07-26T07:52:26.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.449+0000 7f045a000640 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855ac64c90 0x55855b1003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:26.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.450+0000 7f0444c00640 10 client.15561 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:26.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.450+0000 7f045a000640 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855ac64c90 0x55855b1003b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad45be26c679698d server_cookie=189bac594631f944 in_seq=0 out_seq=0 2024-07-26T07:52:26.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.450+0000 7f0444c00640 1 -- 172.21.15.97:0/3969842971 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55855afea340 con 0x55855ac64c90 2024-07-26T07:52:26.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.450+0000 7f0444c00640 10 client.15561 handle_command_reply: tid=0 2024-07-26T07:52:26.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.450+0000 7f0468863740 1 client.15561 shutdown 2024-07-26T07:52:26.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.450+0000 7f0468863740 1 -- 172.21.15.97:0/3969842971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 msgr2=0x55855b09d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.450+0000 7f0468863740 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b09d2e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f044c0096c0 tx=0x7f044c003f40 comp rx=0 tx=0).stop 2024-07-26T07:52:26.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 1 -- 172.21.15.97:0/3969842971 shutdown_connections 2024-07-26T07:52:26.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855ac64c90 0x55855b1003b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855acd0f60 0x55855b096120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55855acb0280 0x55855b095be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 1 --2- 172.21.15.97:0/3969842971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855aca80a0 0x55855b09d2e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 1 -- 172.21.15.97:0/3969842971 >> 172.21.15.97:0/3969842971 conn(0x55855acc5350 msgr2=0x55855b0822c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:26.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0444c00640 0 client.15561 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:26.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 1 -- 172.21.15.97:0/3969842971 shutdown_connections 2024-07-26T07:52:26.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 1 -- 172.21.15.97:0/3969842971 wait complete. 2024-07-26T07:52:26.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.451+0000 7f0468863740 20 client.15561 trim_cache size 0 max 16384 2024-07-26T07:52:26.458 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:52:26.458 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "9621a072-4b21-11ef-bcb6-c7b262605968", 2024-07-26T07:52:26.458 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 34583, 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 277.60705658000001, 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 86, 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 86, 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 298.24091837499998 2024-07-26T07:52:26.459 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:52:26.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 -- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f04400b3ce0 msgr2=0x7f04400b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f04400b3ce0 0x7f04400b61f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f045c1039b0 tx=0x7f044c000bc0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 -- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c10d680 msgr2=0x7f045c1029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:26.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c10d680 0x7f045c1029e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f045400eaa0 tx=0x7f045400ef60 comp rx=0 tx=0).stop 2024-07-26T07:52:26.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 -- 172.21.15.97:0/929361373 shutdown_connections 2024-07-26T07:52:26.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f04400b3ce0 0x7f04400b61f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f045c10d680 0x7f045c1029e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f045c104b70 0x7f045c1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 --2- 172.21.15.97:0/929361373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f045c104220 0x7f045c101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:26.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 -- 172.21.15.97:0/929361373 >> 172.21.15.97:0/929361373 conn(0x7f045c0fdd20 msgr2=0x7f045c107200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:26.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 -- 172.21.15.97:0/929361373 shutdown_connections 2024-07-26T07:52:26.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:26.452+0000 7f0461c00640 1 -- 172.21.15.97:0/929361373 wait complete. 2024-07-26T07:52:27.171 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-07-26T07:52:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:27 smithi115 ceph-mon[35364]: pgmap v530: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:52:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:27 smithi181 ceph-mon[35869]: pgmap v530: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:52:28.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:27 smithi097 ceph-mon[32231]: pgmap v530: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:52:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:29 smithi115 ceph-mon[35364]: pgmap v531: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:52:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:29 smithi181 ceph-mon[35869]: pgmap v531: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:52:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:29 smithi097 ceph-mon[32231]: pgmap v531: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T07:52:32.005 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:31 smithi097 ceph-mon[32231]: pgmap v532: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T07:52:32.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:31 smithi115 ceph-mon[35364]: pgmap v532: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T07:52:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:31 smithi181 ceph-mon[35869]: pgmap v532: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T07:52:34.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:33 smithi115 ceph-mon[35364]: pgmap v533: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T07:52:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:33 smithi181 ceph-mon[35869]: pgmap v533: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T07:52:34.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:33 smithi097 ceph-mon[32231]: pgmap v533: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T07:52:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.982+0000 7f11a6800640 1 -- 172.21.15.97:0/382713376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 msgr2=0x7f11a0105580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.982+0000 7f11a6800640 1 --2- 172.21.15.97:0/382713376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 0x7f11a0105580 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1194009900 tx=0x7f119402a410 comp rx=0 tx=0).stop 2024-07-26T07:52:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.983+0000 7f11a6800640 1 -- 172.21.15.97:0/382713376 shutdown_connections 2024-07-26T07:52:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.983+0000 7f11a6800640 1 --2- 172.21.15.97:0/382713376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11a0106b30 0x7f11a010d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.983+0000 7f11a6800640 1 --2- 172.21.15.97:0/382713376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11a0106190 0x7f11a01065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.983+0000 7f11a6800640 1 --2- 172.21.15.97:0/382713376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 0x7f11a0105580 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.983+0000 7f11a6800640 1 -- 172.21.15.97:0/382713376 >> 172.21.15.97:0/382713376 conn(0x7f11a0100a30 msgr2=0x7f11a0102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.983+0000 7f11a6800640 1 -- 172.21.15.97:0/382713376 shutdown_connections 2024-07-26T07:52:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.983+0000 7f11a6800640 1 -- 172.21.15.97:0/382713376 wait complete. 2024-07-26T07:52:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.984+0000 7f11a6800640 1 Processor -- start 2024-07-26T07:52:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a6800640 1 -- start start 2024-07-26T07:52:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 0x7f11a019b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a6800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11a0106190 0x7f11a019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11a0106b30 0x7f11a019cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a6800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f11a0110c90 con 0x7f11a0105110 2024-07-26T07:52:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a6800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11a0110b10 con 0x7f11a0106190 2024-07-26T07:52:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f11a0110e10 con 0x7f11a0106b30 2024-07-26T07:52:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 0x7f11a019b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 0x7f11a019b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54240/0 (socket says 172.21.15.97:54240) 2024-07-26T07:52:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.985+0000 7f11a5400640 1 -- 172.21.15.97:0/54167224 learned_addr learned my addr 172.21.15.97:0/54167224 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.986+0000 7f11a4a00640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11a0106190 0x7f11a019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.986+0000 7f11a5e00640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11a0106b30 0x7f11a019cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.986+0000 7f11a5400640 1 -- 172.21.15.97:0/54167224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11a0106b30 msgr2=0x7f11a019cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.986+0000 7f11a5400640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11a0106b30 0x7f11a019cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.986+0000 7f11a5400640 1 -- 172.21.15.97:0/54167224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11a0106190 msgr2=0x7f11a019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.986+0000 7f11a5400640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11a0106190 0x7f11a019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.986+0000 7f11a5400640 1 -- 172.21.15.97:0/54167224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a019d320 con 0x7f11a0105110 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.986+0000 7f11a5400640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 0x7f11a019b8e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1194005760 tx=0x7f1194003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.987+0000 7f119e000640 1 -- 172.21.15.97:0/54167224 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119403f070 con 0x7f11a0105110 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.987+0000 7f119e000640 1 -- 172.21.15.97:0/54167224 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1194031da0 con 0x7f11a0105110 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.987+0000 7f11a6800640 1 -- 172.21.15.97:0/54167224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a019d5b0 con 0x7f11a0105110 2024-07-26T07:52:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.987+0000 7f11a6800640 1 -- 172.21.15.97:0/54167224 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a019d8c0 con 0x7f11a0105110 2024-07-26T07:52:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.987+0000 7f119e000640 1 -- 172.21.15.97:0/54167224 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1194035030 con 0x7f11a0105110 2024-07-26T07:52:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.991+0000 7f119e000640 1 -- 172.21.15.97:0/54167224 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f1194042070 con 0x7f11a0105110 2024-07-26T07:52:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.991+0000 7f11aeb22740 1 Processor -- start 2024-07-26T07:52:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.991+0000 7f11aeb22740 1 -- start start 2024-07-26T07:52:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb931b0 0x5600abf418f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf41f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb724d0 0x5600abf42ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11aeb22740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5600abeab270 con 0x5600abb931b0 2024-07-26T07:52:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11aeb22740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5600abe9a770 con 0x5600abb724d0 2024-07-26T07:52:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11aeb22740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5600abe9e7b0 con 0x5600abb6a2f0 2024-07-26T07:52:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11a5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb931b0 0x5600abf418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11a5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb931b0 0x5600abf418f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54250/0 (socket says 172.21.15.97:54250) 2024-07-26T07:52:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.992+0000 7f11a5e00640 1 -- 172.21.15.97:0/744461030 learned_addr learned my addr 172.21.15.97:0/744461030 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.993+0000 7f11a5400640 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf41f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:34.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.993+0000 7f11a4a00640 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb724d0 0x5600abf42ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.993+0000 7f11a5400640 1 -- 172.21.15.97:0/744461030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb724d0 msgr2=0x5600abf42ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.993+0000 7f11a5400640 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb724d0 0x5600abf42ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.993+0000 7f11a5400640 1 -- 172.21.15.97:0/744461030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb931b0 msgr2=0x5600abf418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.993+0000 7f11a5400640 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb931b0 0x5600abf418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.993+0000 7f11a5400640 1 -- 172.21.15.97:0/744461030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600abbfe790 con 0x5600abb6a2f0 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.994+0000 7f11a5e00640 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb931b0 0x5600abf418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.994+0000 7f119e000640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11880b3c60 0x7f11880b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.994+0000 7f119e000640 1 -- 172.21.15.97:0/54167224 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f1194104820 con 0x7f11a0105110 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.994+0000 7f11a5e00640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11880b3c60 0x7f11880b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.994+0000 7f11a5400640 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf41f60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f11941020e0 tx=0x7f119410acb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6befbe4ae1b061f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.994+0000 7f1192600640 1 -- 172.21.15.97:0/744461030 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1194049070 con 0x5600abb6a2f0 2024-07-26T07:52:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.995+0000 7f1192600640 1 -- 172.21.15.97:0/744461030 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11940cb060 con 0x5600abb6a2f0 2024-07-26T07:52:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.995+0000 7f1192600640 1 -- 172.21.15.97:0/744461030 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11940cacc0 con 0x5600abb6a2f0 2024-07-26T07:52:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 -- 172.21.15.97:0/744461030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 msgr2=0x5600abf41f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11a5e00640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11880b3c60 0x7f11880b6170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1198005c40 tx=0x7f11980097e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf41f60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f11941020e0 tx=0x7f119410acb0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 -- 172.21.15.97:0/744461030 shutdown_connections 2024-07-26T07:52:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb724d0 0x5600abf42ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf41f60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 --2- 172.21.15.97:0/744461030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb931b0 0x5600abf418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 -- 172.21.15.97:0/744461030 >> 172.21.15.97:0/744461030 conn(0x5600abb87300 msgr2=0x5600abf34b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 -- 172.21.15.97:0/744461030 shutdown_connections 2024-07-26T07:52:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.996+0000 7f11aeb22740 1 -- 172.21.15.97:0/744461030 wait complete. 2024-07-26T07:52:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.997+0000 7f11aeb22740 1 -- start start 2024-07-26T07:52:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.997+0000 7f11aeb22740 1 Processor -- start 2024-07-26T07:52:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf4b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb931b0 0x5600abf4c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11aeb22740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5600abdd4ab0 con 0x5600abb724d0 2024-07-26T07:52:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11aeb22740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5600abe37d50 con 0x5600abb931b0 2024-07-26T07:52:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11aeb22740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5600abf43390 con 0x5600abb6a2f0 2024-07-26T07:52:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11aeb22740 10 client.0 _ll_register_callbacks cb 0x5600abbaaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11a4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11a4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54252/0 (socket says 172.21.15.97:54252) 2024-07-26T07:52:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.998+0000 7f11a4a00640 1 -- 172.21.15.97:0/692798566 learned_addr learned my addr 172.21.15.97:0/692798566 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f11a5400640 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf4b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f11a5e00640 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb931b0 0x5600abf4c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f11a4a00640 1 -- 172.21.15.97:0/692798566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 msgr2=0x5600abf4b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f11a4a00640 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf4b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f11a4a00640 1 -- 172.21.15.97:0/692798566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb931b0 msgr2=0x5600abf4c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f11a4a00640 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb931b0 0x5600abf4c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f11a4a00640 1 -- 172.21.15.97:0/692798566 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600abc77620 con 0x5600abb724d0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f11a4a00640 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b7b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f118c00bfc0 tx=0x7f118c0128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2294129d44545f26 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:34.999+0000 7f1186a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.000+0000 7f1186a00640 1 -- 172.21.15.97:0/692798566 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c01f060 con 0x5600abb724d0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.000+0000 7f11aeb22740 1 -- 172.21.15.97:0/692798566 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11940cb060 con 0x5600abb724d0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.000+0000 7f1186a00640 1 -- 172.21.15.97:0/692798566 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f118c010900 con 0x5600abb724d0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.000+0000 7f1186a00640 1 -- 172.21.15.97:0/692798566 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f118c010b00 con 0x5600abb724d0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.000+0000 7f11aeb22740 10 client.15573 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.000+0000 7f11aeb22740 1 -- 172.21.15.97:0/692798566 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5600abf55370 con 0x5600abb724d0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.000+0000 7f1186a00640 1 -- 172.21.15.97:0/692798566 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11940cb060 con 0x5600abb724d0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.001+0000 7f1186a00640 1 -- 172.21.15.97:0/692798566 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f118c023030 con 0x5600abb724d0 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.001+0000 7f11aeb22740 10 client.15573 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.001+0000 7f11aeb22740 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:52:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.001+0000 7f11aeb22740 20 client.15573 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.001+0000 7f11aeb22740 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5600abb51d80 0x5600abf58b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.001+0000 7f11aeb22740 4 client.15573 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.001+0000 7f11aeb22740 1 -- 172.21.15.97:0/692798566 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600abead1c0 con 0x5600abb51d80 2024-07-26T07:52:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.002+0000 7f11a5400640 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5600abb51d80 0x5600abf58b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.002+0000 7f11a5400640 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5600abb51d80 0x5600abf58b90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eed4c2f1e951814 server_cookie=baafa15356552952 in_seq=0 out_seq=0 2024-07-26T07:52:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.002+0000 7f1186a00640 10 client.15573 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.004+0000 7f1186a00640 1 -- 172.21.15.97:0/692798566 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5600abead1c0 con 0x5600abb51d80 2024-07-26T07:52:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.004+0000 7f1186a00640 10 client.15573 handle_command_reply: tid=0 2024-07-26T07:52:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 client.15573 shutdown 2024-07-26T07:52:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 -- 172.21.15.97:0/692798566 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 msgr2=0x5600abf4b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b7b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f118c00bfc0 tx=0x7f118c0128e0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 -- 172.21.15.97:0/692798566 shutdown_connections 2024-07-26T07:52:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5600abb51d80 0x5600abf58b90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb931b0 0x5600abf4c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f1186a00640 0 client.15573 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b7b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 --2- 172.21.15.97:0/692798566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf4b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.005+0000 7f11aeb22740 1 -- 172.21.15.97:0/692798566 >> 172.21.15.97:0/692798566 conn(0x5600abb87300 msgr2=0x5600abf42960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.006+0000 7f11aeb22740 1 -- 172.21.15.97:0/692798566 shutdown_connections 2024-07-26T07:52:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.006+0000 7f11aeb22740 1 -- 172.21.15.97:0/692798566 wait complete. 2024-07-26T07:52:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.006+0000 7f11aeb22740 20 client.15573 trim_cache size 0 max 16384 2024-07-26T07:52:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.036+0000 7f11aeb22740 1 Processor -- start 2024-07-26T07:52:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.036+0000 7f11aeb22740 1 -- start start 2024-07-26T07:52:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 0x5600abf49680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf49bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11aeb22740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f118000e630 con 0x5600abb724d0 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11aeb22740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1180010170 con 0x5600abb51d80 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11aeb22740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f118000ed00 con 0x5600abb6a2f0 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11a5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11a5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54268/0 (socket says 172.21.15.97:54268) 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.037+0000 7f11a5e00640 1 -- 172.21.15.97:0/359652720 learned_addr learned my addr 172.21.15.97:0/359652720 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.038+0000 7f11a5400640 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 0x5600abf49680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.038+0000 7f11a4a00640 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf49bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.038+0000 7f11a5e00640 1 -- 172.21.15.97:0/359652720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 msgr2=0x5600abf49bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.038+0000 7f11a5e00640 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf49bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.038+0000 7f11a5e00640 1 -- 172.21.15.97:0/359652720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 msgr2=0x5600abf49680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.038+0000 7f11a5e00640 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 0x5600abf49680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.038+0000 7f11a5e00640 1 -- 172.21.15.97:0/359652720 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600abf42730 con 0x5600abb724d0 2024-07-26T07:52:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.038+0000 7f11a5e00640 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b0f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1198010670 tx=0x7f1198015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10728122592e4dff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.039+0000 7f1191c00640 1 -- 172.21.15.97:0/359652720 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11980207d0 con 0x5600abb724d0 2024-07-26T07:52:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.039+0000 7f1191c00640 1 -- 172.21.15.97:0/359652720 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1198020e00 con 0x5600abb724d0 2024-07-26T07:52:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.039+0000 7f1191c00640 1 -- 172.21.15.97:0/359652720 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119801e400 con 0x5600abb724d0 2024-07-26T07:52:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 -- 172.21.15.97:0/359652720 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 msgr2=0x5600abf4b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b0f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1198010670 tx=0x7f1198015000 comp rx=0 tx=0).stop 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 -- 172.21.15.97:0/359652720 shutdown_connections 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb724d0 0x5600abf4b0f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb6a2f0 0x5600abf49bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 --2- 172.21.15.97:0/359652720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 0x5600abf49680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 -- 172.21.15.97:0/359652720 >> 172.21.15.97:0/359652720 conn(0x5600abb87300 msgr2=0x5600abf45f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 -- 172.21.15.97:0/359652720 shutdown_connections 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.040+0000 7f11aeb22740 1 -- 172.21.15.97:0/359652720 wait complete. 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.041+0000 7f11aeb22740 1 -- start start 2024-07-26T07:52:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.041+0000 7f11aeb22740 1 Processor -- start 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 0x5600abf500d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb6a2f0 0x5600abf50790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11aeb22740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb724d0 0x5600abf51300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11aeb22740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f118000faa0 con 0x5600abb6a2f0 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11aeb22740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f118000f3d0 con 0x5600abb51d80 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11aeb22740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f118000c610 con 0x5600abb724d0 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11aeb22740 10 client.0 _ll_register_callbacks cb 0x7f118c00b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11a4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb6a2f0 0x5600abf50790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11a4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb6a2f0 0x5600abf50790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54276/0 (socket says 172.21.15.97:54276) 2024-07-26T07:52:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11a4a00640 1 -- 172.21.15.97:0/1852654395 learned_addr learned my addr 172.21.15.97:0/1852654395 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11a5400640 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 0x5600abf500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.042+0000 7f11a5e00640 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb724d0 0x5600abf51300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.043+0000 7f11a4a00640 1 -- 172.21.15.97:0/1852654395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb724d0 msgr2=0x5600abf51300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.043+0000 7f11a4a00640 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb724d0 0x5600abf51300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.043+0000 7f11a4a00640 1 -- 172.21.15.97:0/1852654395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 msgr2=0x5600abf500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.043+0000 7f11a4a00640 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 0x5600abf500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.043+0000 7f11a4a00640 1 -- 172.21.15.97:0/1852654395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600abf43480 con 0x5600abb6a2f0 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.043+0000 7f11a4a00640 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb6a2f0 0x5600abf50790 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f118c000c00 tx=0x7f118c010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1acf558cd8b63ee9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.043+0000 7f1186a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.044+0000 7f1186a00640 1 -- 172.21.15.97:0/1852654395 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c031060 con 0x5600abb6a2f0 2024-07-26T07:52:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.044+0000 7f1186a00640 1 -- 172.21.15.97:0/1852654395 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f118c004030 con 0x5600abb6a2f0 2024-07-26T07:52:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.044+0000 7f11aeb22740 1 -- 172.21.15.97:0/1852654395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1198020e00 con 0x5600abb6a2f0 2024-07-26T07:52:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.044+0000 7f1186a00640 1 -- 172.21.15.97:0/1852654395 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f118c00bb40 con 0x5600abb6a2f0 2024-07-26T07:52:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.044+0000 7f11aeb22740 10 client.15579 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.044+0000 7f11aeb22740 1 -- 172.21.15.97:0/1852654395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5600abf52bb0 con 0x5600abb6a2f0 2024-07-26T07:52:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.045+0000 7f1186a00640 1 -- 172.21.15.97:0/1852654395 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1198020e00 con 0x5600abb6a2f0 2024-07-26T07:52:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.045+0000 7f1186a00640 1 -- 172.21.15.97:0/1852654395 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f118c035030 con 0x5600abb6a2f0 2024-07-26T07:52:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.045+0000 7f11aeb22740 10 client.15579 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.045+0000 7f11aeb22740 10 client.15579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:52:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.045+0000 7f11aeb22740 20 client.15579 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.045+0000 7f11aeb22740 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5600abb931b0 0x5600abfc3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.045+0000 7f11aeb22740 4 client.15579 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:52:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.045+0000 7f11aeb22740 1 -- 172.21.15.97:0/1852654395 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600abea8c60 con 0x5600abb931b0 2024-07-26T07:52:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.046+0000 7f11a5400640 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5600abb931b0 0x5600abfc3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.047+0000 7f1186a00640 10 client.15579 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.047+0000 7f11a5400640 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5600abb931b0 0x5600abfc3330 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e12f1a35b20aad15 server_cookie=5cbacb83e424c5df in_seq=0 out_seq=0 2024-07-26T07:52:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f1186a00640 1 -- 172.21.15.97:0/1852654395 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600abea8c60 con 0x5600abb931b0 2024-07-26T07:52:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f1186a00640 10 client.15579 handle_command_reply: tid=0 2024-07-26T07:52:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 client.15579 shutdown 2024-07-26T07:52:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 -- 172.21.15.97:0/1852654395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb6a2f0 msgr2=0x5600abf50790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb6a2f0 0x5600abf50790 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f118c000c00 tx=0x7f118c010990 comp rx=0 tx=0).stop 2024-07-26T07:52:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 -- 172.21.15.97:0/1852654395 shutdown_connections 2024-07-26T07:52:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5600abb931b0 0x5600abfc3330 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5600abb724d0 0x5600abf51300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600abb6a2f0 0x5600abf50790 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 --2- 172.21.15.97:0/1852654395 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5600abb51d80 0x5600abf500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 -- 172.21.15.97:0/1852654395 >> 172.21.15.97:0/1852654395 conn(0x5600abb87300 msgr2=0x5600abf49410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f1186a00640 0 client.15579 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 -- 172.21.15.97:0/1852654395 shutdown_connections 2024-07-26T07:52:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.064+0000 7f11aeb22740 1 -- 172.21.15.97:0/1852654395 wait complete. 2024-07-26T07:52:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.065+0000 7f11aeb22740 20 client.15579 trim_cache size 0 max 16384 2024-07-26T07:52:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:52:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:52:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:52:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:52:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.065+0000 7f11a6800640 1 -- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11880b3c60 msgr2=0x7f11880b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.065+0000 7f11a6800640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11880b3c60 0x7f11880b6170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1198005c40 tx=0x7f11980097e0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.065+0000 7f11a6800640 1 -- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 msgr2=0x7f11a019b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.065+0000 7f11a6800640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 0x7f11a019b8e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1194005760 tx=0x7f1194003d50 comp rx=0 tx=0).stop 2024-07-26T07:52:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.066+0000 7f11a6800640 1 -- 172.21.15.97:0/54167224 shutdown_connections 2024-07-26T07:52:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.066+0000 7f11a6800640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11880b3c60 0x7f11880b6170 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.066+0000 7f11a6800640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11a0106b30 0x7f11a019cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.066+0000 7f11a6800640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11a0106190 0x7f11a019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.066+0000 7f11a6800640 1 --2- 172.21.15.97:0/54167224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11a0105110 0x7f11a019b8e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.066+0000 7f11a6800640 1 -- 172.21.15.97:0/54167224 >> 172.21.15.97:0/54167224 conn(0x7f11a0100a30 msgr2=0x7f11a01094b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.066+0000 7f11a6800640 1 -- 172.21.15.97:0/54167224 shutdown_connections 2024-07-26T07:52:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:35.066+0000 7f11a6800640 1 -- 172.21.15.97:0/54167224 wait complete. 2024-07-26T07:52:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:35 smithi115 ceph-mon[35364]: pgmap v534: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T07:52:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:35 smithi181 ceph-mon[35869]: pgmap v534: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T07:52:36.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:35 smithi097 ceph-mon[32231]: pgmap v534: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T07:52:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:37 smithi115 ceph-mon[35364]: pgmap v535: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:52:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:37 smithi181 ceph-mon[35869]: pgmap v535: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:52:38.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:37 smithi097 ceph-mon[32231]: pgmap v535: 97 pgs: 97 active+clean; 679 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:52:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:39 smithi115 ceph-mon[35364]: pgmap v536: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:39 smithi181 ceph-mon[35869]: pgmap v536: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:40.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:39 smithi097 ceph-mon[32231]: pgmap v536: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:42.172 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-07-26T07:52:42.172 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-07-26T07:52:42.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:52:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:41 smithi115 ceph-mon[35364]: pgmap v537: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:42.226 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:41 smithi097 ceph-mon[32231]: pgmap v537: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:41 smithi181 ceph-mon[35869]: pgmap v537: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:42.615 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:43 smithi115 ceph-mon[35364]: pgmap v538: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T07:52:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:43 smithi181 ceph-mon[35869]: pgmap v538: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T07:52:44.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:43 smithi097 ceph-mon[32231]: pgmap v538: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T07:52:44.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.765+0000 7f8e8be00640 1 -- 172.21.15.97:0/3233910777 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c1040c0 msgr2=0x7f8e8c1044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:44.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.765+0000 7f8e8be00640 1 --2- 172.21.15.97:0/3233910777 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c1040c0 0x7f8e8c1044d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009900 tx=0x7f8e7c02a410 comp rx=0 tx=0).stop 2024-07-26T07:52:44.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.765+0000 7f8e8be00640 1 -- 172.21.15.97:0/3233910777 shutdown_connections 2024-07-26T07:52:44.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.765+0000 7f8e8be00640 1 --2- 172.21.15.97:0/3233910777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c10d670 0x7f8e8c10fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:44.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.765+0000 7f8e8be00640 1 --2- 172.21.15.97:0/3233910777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e8c104af0 0x7f8e8c10d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:44.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.765+0000 7f8e8be00640 1 --2- 172.21.15.97:0/3233910777 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c1040c0 0x7f8e8c1044d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:44.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.765+0000 7f8e8be00640 1 -- 172.21.15.97:0/3233910777 >> 172.21.15.97:0/3233910777 conn(0x7f8e8c0fdcf0 msgr2=0x7f8e8c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:44.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.766+0000 7f8e8be00640 1 -- 172.21.15.97:0/3233910777 shutdown_connections 2024-07-26T07:52:44.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.766+0000 7f8e8be00640 1 -- 172.21.15.97:0/3233910777 wait complete. 2024-07-26T07:52:44.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.768+0000 7f8e8be00640 1 Processor -- start 2024-07-26T07:52:44.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.768+0000 7f8e8be00640 1 -- start start 2024-07-26T07:52:44.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.768+0000 7f8e8be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1040c0 0x7f8e8c19be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:44.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.768+0000 7f8e8be00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e8c104af0 0x7f8e8c19c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:44.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.768+0000 7f8e8be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c10d670 0x7f8e8c1a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:44.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.769+0000 7f8e8be00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c1131c0 con 0x7f8e8c10d670 2024-07-26T07:52:44.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.769+0000 7f8e8be00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c113040 con 0x7f8e8c104af0 2024-07-26T07:52:44.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.769+0000 7f8e8be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c113340 con 0x7f8e8c1040c0 2024-07-26T07:52:44.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.769+0000 7f8e8aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1040c0 0x7f8e8c19be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:44.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.769+0000 7f8e8aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1040c0 0x7f8e8c19be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:55154/0 (socket says 172.21.15.97:55154) 2024-07-26T07:52:44.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.769+0000 7f8e8aa00640 1 -- 172.21.15.97:0/2349056357 learned_addr learned my addr 172.21.15.97:0/2349056357 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:44.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.769+0000 7f8e8a000640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e8c104af0 0x7f8e8c19c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:44.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.769+0000 7f8e8b400640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c10d670 0x7f8e8c1a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:44.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.770+0000 7f8e8b400640 1 -- 172.21.15.97:0/2349056357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1040c0 msgr2=0x7f8e8c19be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.770+0000 7f8e8b400640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1040c0 0x7f8e8c19be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.770+0000 7f8e8b400640 1 -- 172.21.15.97:0/2349056357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e8c104af0 msgr2=0x7f8e8c19c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.770+0000 7f8e8b400640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e8c104af0 0x7f8e8c19c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.770+0000 7f8e8b400640 1 -- 172.21.15.97:0/2349056357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c1a1ce0 con 0x7f8e8c10d670 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.770+0000 7f8e8b400640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c10d670 0x7f8e8c1a1670 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400e940 tx=0x7f8e8400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.771+0000 7f8e90a00640 1 -- 172.21.15.97:0/2349056357 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e84004d20 con 0x7f8e8c10d670 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.771+0000 7f8e90a00640 1 -- 172.21.15.97:0/2349056357 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8e84005020 con 0x7f8e8c10d670 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.771+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8c1a1fd0 con 0x7f8e8c10d670 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.771+0000 7f8e90a00640 1 -- 172.21.15.97:0/2349056357 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e840116d0 con 0x7f8e8c10d670 2024-07-26T07:52:44.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.771+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8c1a87f0 con 0x7f8e8c10d670 2024-07-26T07:52:44.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.772+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e6c0052c0 con 0x7f8e8c10d670 2024-07-26T07:52:44.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.776+0000 7f8e90a00640 1 -- 172.21.15.97:0/2349056357 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f8e8400ca60 con 0x7f8e8c10d670 2024-07-26T07:52:44.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.779+0000 7f8e90a00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8e700b3d10 0x7f8e700b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.779+0000 7f8e90a00640 1 -- 172.21.15.97:0/2349056357 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f8e840e3cb0 con 0x7f8e8c10d670 2024-07-26T07:52:44.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.779+0000 7f8e8aa00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8e700b3d10 0x7f8e700b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:44.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.780+0000 7f8e8aa00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8e700b3d10 0x7f8e700b6220 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0096c0 tx=0x7f8e7c005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:44.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:44.784+0000 7f8e90a00640 1 -- 172.21.15.97:0/2349056357 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8e84016030 con 0x7f8e8c10d670 2024-07-26T07:52:45.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.236+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e6c005880 con 0x7f8e8c10d670 2024-07-26T07:52:45.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.237+0000 7f8e90a00640 1 -- 172.21.15.97:0/2349056357 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11990 (secure 0 0 0) 0x7f8e840ab0e0 con 0x7f8e8c10d670 2024-07-26T07:52:45.238 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:52:45.239 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"btime":"2024-07-26T07:51:41:000369+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:51:40.647454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:52:45.239 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-07-26T07:52:45.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.240+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8e700b3d10 msgr2=0x7f8e700b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:45.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.240+0000 7f8e8be00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8e700b3d10 0x7f8e700b6220 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0096c0 tx=0x7f8e7c005e80 comp rx=0 tx=0).stop 2024-07-26T07:52:45.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.240+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c10d670 msgr2=0x7f8e8c1a1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:45.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.240+0000 7f8e8be00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c10d670 0x7f8e8c1a1670 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400e940 tx=0x7f8e8400ed40 comp rx=0 tx=0).stop 2024-07-26T07:52:45.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.241+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 shutdown_connections 2024-07-26T07:52:45.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.241+0000 7f8e8be00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f8e700b3d10 0x7f8e700b6220 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:45.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.241+0000 7f8e8be00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e8c10d670 0x7f8e8c1a1670 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:45.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.241+0000 7f8e8be00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e8c104af0 0x7f8e8c19c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:45.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.241+0000 7f8e8be00640 1 --2- 172.21.15.97:0/2349056357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1040c0 0x7f8e8c19be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:45.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.241+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 >> 172.21.15.97:0/2349056357 conn(0x7f8e8c0fdcf0 msgr2=0x7f8e8c101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:45.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.241+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 shutdown_connections 2024-07-26T07:52:45.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:45.241+0000 7f8e8be00640 1 -- 172.21.15.97:0/2349056357 wait complete. 2024-07-26T07:52:45.994 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0' 2024-07-26T07:52:45.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0 2024-07-26T07:52:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:45 smithi115 ceph-mon[35364]: pgmap v539: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T07:52:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:45 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2349056357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:52:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:45 smithi181 ceph-mon[35869]: pgmap v539: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T07:52:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:45 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2349056357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:52:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:45 smithi097 ceph-mon[32231]: pgmap v539: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T07:52:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:45 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2349056357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:52:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:52:48.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:47 smithi097 ceph-mon[32231]: pgmap v540: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:47 smithi115 ceph-mon[35364]: pgmap v540: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:47 smithi181 ceph-mon[35869]: pgmap v540: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 -- 172.21.15.97:0/842405439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524102f10 msgr2=0x7f5524103380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 --2- 172.21.15.97:0/842405439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524102f10 0x7f5524103380 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5514009900 tx=0x7f551402a3f0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 -- 172.21.15.97:0/842405439 shutdown_connections 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 --2- 172.21.15.97:0/842405439 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5524104930 0x7f552410b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 --2- 172.21.15.97:0/842405439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5524103f90 0x7f55241043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 --2- 172.21.15.97:0/842405439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524102f10 0x7f5524103380 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 -- 172.21.15.97:0/842405439 >> 172.21.15.97:0/842405439 conn(0x7f55240fe850 msgr2=0x7f5524100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 -- 172.21.15.97:0/842405439 shutdown_connections 2024-07-26T07:52:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.334+0000 7f5529c00640 1 -- 172.21.15.97:0/842405439 wait complete. 2024-07-26T07:52:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.335+0000 7f5529c00640 1 Processor -- start 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.335+0000 7f5529c00640 1 -- start start 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.335+0000 7f5529c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5524102f10 0x7f55241996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.335+0000 7f5529c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524103f90 0x7f5524199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.335+0000 7f5529c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5524104930 0x7f552419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.335+0000 7f5529c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f552410ea90 con 0x7f5524103f90 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.335+0000 7f5529c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f552410e910 con 0x7f5524102f10 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.335+0000 7f5529c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f552410ec10 con 0x7f5524104930 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524103f90 0x7f5524199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524103f90 0x7f5524199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46294/0 (socket says 172.21.15.97:46294) 2024-07-26T07:52:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523400640 1 -- 172.21.15.97:0/4053649080 learned_addr learned my addr 172.21.15.97:0/4053649080 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523e00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5524102f10 0x7f55241996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523400640 1 -- 172.21.15.97:0/4053649080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5524104930 msgr2=0x7f552419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523400640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5524104930 0x7f552419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523400640 1 -- 172.21.15.97:0/4053649080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5524102f10 msgr2=0x7f55241996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523400640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5524102f10 0x7f55241996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.336+0000 7f5523400640 1 -- 172.21.15.97:0/4053649080 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552419b120 con 0x7f5524103f90 2024-07-26T07:52:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.337+0000 7f5523400640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524103f90 0x7f5524199d50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f551000d950 tx=0x7f551000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.337+0000 7f5520c00640 1 -- 172.21.15.97:0/4053649080 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510017070 con 0x7f5524103f90 2024-07-26T07:52:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.337+0000 7f5520c00640 1 -- 172.21.15.97:0/4053649080 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5510009c30 con 0x7f5524103f90 2024-07-26T07:52:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.337+0000 7f5529c00640 1 -- 172.21.15.97:0/4053649080 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552419b410 con 0x7f5524103f90 2024-07-26T07:52:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.337+0000 7f5520c00640 1 -- 172.21.15.97:0/4053649080 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551000f620 con 0x7f5524103f90 2024-07-26T07:52:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.337+0000 7f5529c00640 1 -- 172.21.15.97:0/4053649080 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55241a1b00 con 0x7f5524103f90 2024-07-26T07:52:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.342+0000 7f5520c00640 1 -- 172.21.15.97:0/4053649080 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f5510010460 con 0x7f5524103f90 2024-07-26T07:52:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.343+0000 7f5532194740 1 Processor -- start 2024-07-26T07:52:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.343+0000 7f5532194740 1 -- start start 2024-07-26T07:52:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.343+0000 7f5532194740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5613452461b0 0x5613455ec0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.343+0000 7f5532194740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56134521d2f0 0x5613455ec730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.343+0000 7f5532194740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 0x5613455ed280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.343+0000 7f5532194740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56134555e270 con 0x56134521d2f0 2024-07-26T07:52:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.343+0000 7f5532194740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56134554d770 con 0x5613452461b0 2024-07-26T07:52:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.343+0000 7f5532194740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5613455517b0 con 0x5613452254d0 2024-07-26T07:52:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.344+0000 7f5523e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56134521d2f0 0x5613455ec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.344+0000 7f5523e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56134521d2f0 0x5613455ec730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46310/0 (socket says 172.21.15.97:46310) 2024-07-26T07:52:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.344+0000 7f5523e00640 1 -- 172.21.15.97:0/3612156998 learned_addr learned my addr 172.21.15.97:0/3612156998 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.344+0000 7f5529200640 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5613452461b0 0x5613455ec0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.344+0000 7f5523400640 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 0x5613455ed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.345+0000 7f5529200640 1 -- 172.21.15.97:0/3612156998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 msgr2=0x5613455ed280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.345+0000 7f5529200640 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 0x5613455ed280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.345+0000 7f5529200640 1 -- 172.21.15.97:0/3612156998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56134521d2f0 msgr2=0x5613455ec730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.345+0000 7f5529200640 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56134521d2f0 0x5613455ec730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.345+0000 7f5529200640 1 -- 172.21.15.97:0/3612156998 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613452b1790 con 0x5613452461b0 2024-07-26T07:52:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.345+0000 7f5529200640 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5613452461b0 0x5613455ec0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f551c009cc0 tx=0x7f551c00e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=196e640fd71d785d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.345+0000 7f5520c00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f55080b3c60 0x7f55080b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.346+0000 7f5520c00640 1 -- 172.21.15.97:0/4053649080 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f55100e3a80 con 0x7f5524103f90 2024-07-26T07:52:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.346+0000 7f5519600640 1 -- 172.21.15.97:0/3612156998 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c016070 con 0x5613452461b0 2024-07-26T07:52:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.346+0000 7f5523e00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f55080b3c60 0x7f55080b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.346+0000 7f5519600640 1 -- 172.21.15.97:0/3612156998 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f551c005020 con 0x5613452461b0 2024-07-26T07:52:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.346+0000 7f5519600640 1 -- 172.21.15.97:0/3612156998 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c0116d0 con 0x5613452461b0 2024-07-26T07:52:48.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.347+0000 7f5532194740 1 -- 172.21.15.97:0/3612156998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5613452461b0 msgr2=0x5613455ec0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.347+0000 7f5532194740 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5613452461b0 0x5613455ec0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f551c009cc0 tx=0x7f551c00e6c0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.347+0000 7f5523e00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f55080b3c60 0x7f55080b6170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f55140096c0 tx=0x7f5514031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.348+0000 7f5532194740 1 -- 172.21.15.97:0/3612156998 shutdown_connections 2024-07-26T07:52:48.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.348+0000 7f5532194740 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 0x5613455ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.348+0000 7f5532194740 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56134521d2f0 0x5613455ec730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.348+0000 7f5532194740 1 --2- 172.21.15.97:0/3612156998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5613452461b0 0x5613455ec0c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.348+0000 7f5532194740 1 -- 172.21.15.97:0/3612156998 >> 172.21.15.97:0/3612156998 conn(0x56134523a300 msgr2=0x5613455e7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.348+0000 7f5532194740 1 -- 172.21.15.97:0/3612156998 shutdown_connections 2024-07-26T07:52:48.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.348+0000 7f5532194740 1 -- 172.21.15.97:0/3612156998 wait complete. 2024-07-26T07:52:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.349+0000 7f5532194740 1 -- start start 2024-07-26T07:52:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.349+0000 7f5532194740 1 Processor -- start 2024-07-26T07:52:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5532194740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5532194740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x5613455f6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5532194740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452461b0 0x5613455f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5523400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x5613455f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5523400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x5613455f6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46314/0 (socket says 172.21.15.97:46314) 2024-07-26T07:52:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5523400640 1 -- 172.21.15.97:0/3681558967 learned_addr learned my addr 172.21.15.97:0/3681558967 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561345487ab0 con 0x5613452254d0 2024-07-26T07:52:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5613454ead50 con 0x56134521d2f0 2024-07-26T07:52:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5613455edda0 con 0x5613452461b0 2024-07-26T07:52:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.350+0000 7f5523e00640 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.351+0000 7f5523400640 1 -- 172.21.15.97:0/3681558967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452461b0 msgr2=0x5613455f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.351+0000 7f5532194740 10 client.0 _ll_register_callbacks cb 0x56134525da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.351+0000 7f5529200640 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452461b0 0x5613455f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.351+0000 7f5523400640 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452461b0 0x5613455f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.351+0000 7f5523400640 1 -- 172.21.15.97:0/3681558967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 msgr2=0x5613455f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.351+0000 7f5523400640 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455f5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.351+0000 7f5523400640 1 -- 172.21.15.97:0/3681558967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56134532a620 con 0x5613452254d0 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.351+0000 7f5529200640 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452461b0 0x5613455f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.352+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f551c005020 con 0x5613452254d0 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.352+0000 7f5523400640 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x5613455f6260 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f55100a6830 tx=0x7f55100aa1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d61b5ecde79483 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.352+0000 7f550e000640 10 client.15597 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.352+0000 7f550e000640 1 -- 172.21.15.97:0/3681558967 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510020030 con 0x5613452254d0 2024-07-26T07:52:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.352+0000 7f550e000640 1 -- 172.21.15.97:0/3681558967 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f55100a6da0 con 0x5613452254d0 2024-07-26T07:52:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.352+0000 7f550e000640 1 -- 172.21.15.97:0/3681558967 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5510020030 con 0x5613452254d0 2024-07-26T07:52:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.352+0000 7f5532194740 10 client.15597 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.352+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5613455ffcb0 con 0x5613452254d0 2024-07-26T07:52:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.353+0000 7f550e000640 1 -- 172.21.15.97:0/3681558967 <== mon.0 v2:172.21.15.97:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f55100a9a10 con 0x5613452254d0 2024-07-26T07:52:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.353+0000 7f5532194740 10 client.15597 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.353+0000 7f5532194740 10 client.15597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:52:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.353+0000 7f5532194740 20 client.15597 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.353+0000 7f5532194740 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561345204d80 0x561345603310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.353+0000 7f5532194740 4 client.15597 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.353+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613455601c0 con 0x561345204d80 2024-07-26T07:52:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.354+0000 7f5523e00640 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561345204d80 0x561345603310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.355+0000 7f5523e00640 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561345204d80 0x561345603310 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fd7d101c45724f4 server_cookie=f55643620f6e6e1e in_seq=0 out_seq=0 2024-07-26T07:52:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.355+0000 7f550e000640 10 client.15597 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:48.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f550e000640 1 -- 172.21.15.97:0/3681558967 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5613455601c0 con 0x561345204d80 2024-07-26T07:52:48.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f550e000640 10 client.15597 handle_command_reply: tid=0 2024-07-26T07:52:48.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 client.15597 shutdown 2024-07-26T07:52:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 msgr2=0x5613455f6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x5613455f6260 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f55100a6830 tx=0x7f55100aa1c0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 shutdown_connections 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561345204d80 0x561345603310 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f550e000640 0 client.15597 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452461b0 0x5613455f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x5613455f6260 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 --2- 172.21.15.97:0/3681558967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.357+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 >> 172.21.15.97:0/3681558967 conn(0x56134523a300 msgr2=0x5613455e60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.358+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 shutdown_connections 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.358+0000 7f5532194740 1 -- 172.21.15.97:0/3681558967 wait complete. 2024-07-26T07:52:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.358+0000 7f5532194740 20 client.15597 trim_cache size 0 max 16384 2024-07-26T07:52:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.389+0000 7f5532194740 1 Processor -- start 2024-07-26T07:52:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.389+0000 7f5532194740 1 -- start start 2024-07-26T07:52:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5532194740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561345204d80 0x5613455e6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5532194740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5532194740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x561345607aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5532194740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f55241acaf0 con 0x5613452254d0 2024-07-26T07:52:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5532194740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ae630 con 0x56134521d2f0 2024-07-26T07:52:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5532194740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ad1c0 con 0x561345204d80 2024-07-26T07:52:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5529200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x561345607aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5523e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561345204d80 0x5613455e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5529200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x561345607aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46326/0 (socket says 172.21.15.97:46326) 2024-07-26T07:52:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.390+0000 7f5529200640 1 -- 172.21.15.97:0/4225916577 learned_addr learned my addr 172.21.15.97:0/4225916577 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.391+0000 7f5523400640 1 --2- 172.21.15.97:0/4225916577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.391+0000 7f5529200640 1 -- 172.21.15.97:0/4225916577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561345204d80 msgr2=0x5613455e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.391+0000 7f5529200640 1 --2- 172.21.15.97:0/4225916577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561345204d80 0x5613455e6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.391+0000 7f5529200640 1 -- 172.21.15.97:0/4225916577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 msgr2=0x5613455e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.391+0000 7f5529200640 1 --2- 172.21.15.97:0/4225916577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455e6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.391+0000 7f5529200640 1 -- 172.21.15.97:0/4225916577 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613455e72a0 con 0x5613452254d0 2024-07-26T07:52:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.392+0000 7f5529200640 1 --2- 172.21.15.97:0/4225916577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x561345607aa0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f551c009a70 tx=0x7f551c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb68bc6564b1c6fa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.392+0000 7f5518c00640 1 -- 172.21.15.97:0/4225916577 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c028070 con 0x5613452254d0 2024-07-26T07:52:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.392+0000 7f5518c00640 1 -- 172.21.15.97:0/4225916577 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f551c009e30 con 0x5613452254d0 2024-07-26T07:52:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.392+0000 7f5518c00640 1 -- 172.21.15.97:0/4225916577 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c016040 con 0x5613452254d0 2024-07-26T07:52:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 -- 172.21.15.97:0/4225916577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 msgr2=0x561345607aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 --2- 172.21.15.97:0/4225916577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x561345607aa0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f551c009a70 tx=0x7f551c00ef10 comp rx=0 tx=0).stop 2024-07-26T07:52:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 -- 172.21.15.97:0/4225916577 shutdown_connections 2024-07-26T07:52:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 --2- 172.21.15.97:0/4225916577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5613452254d0 0x561345607aa0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 --2- 172.21.15.97:0/4225916577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 --2- 172.21.15.97:0/4225916577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561345204d80 0x5613455e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 -- 172.21.15.97:0/4225916577 >> 172.21.15.97:0/4225916577 conn(0x56134523a300 msgr2=0x5613455f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 -- 172.21.15.97:0/4225916577 shutdown_connections 2024-07-26T07:52:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.394+0000 7f5532194740 1 -- 172.21.15.97:0/4225916577 wait complete. 2024-07-26T07:52:48.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.395+0000 7f5532194740 1 -- start start 2024-07-26T07:52:48.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.395+0000 7f5532194740 1 Processor -- start 2024-07-26T07:52:48.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5532194740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561345204d80 0x5613455fcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5532194740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455fd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5532194740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 0x5613455fb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5532194740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f55241adf60 con 0x561345204d80 2024-07-26T07:52:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5532194740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ad890 con 0x56134521d2f0 2024-07-26T07:52:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5532194740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f55241aaad0 con 0x5613452254d0 2024-07-26T07:52:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5532194740 10 client.0 _ll_register_callbacks cb 0x7f551000df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5523400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455fd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5523e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561345204d80 0x5613455fcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5523e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561345204d80 0x5613455fcd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46336/0 (socket says 172.21.15.97:46336) 2024-07-26T07:52:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5523e00640 1 -- 172.21.15.97:0/905719271 learned_addr learned my addr 172.21.15.97:0/905719271 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.396+0000 7f5529200640 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 0x5613455fb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.397+0000 7f5523e00640 1 -- 172.21.15.97:0/905719271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 msgr2=0x5613455fb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.397+0000 7f5523e00640 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 0x5613455fb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.397+0000 7f5523e00640 1 -- 172.21.15.97:0/905719271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 msgr2=0x5613455fd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.397+0000 7f5523e00640 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455fd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.397+0000 7f5523e00640 1 -- 172.21.15.97:0/905719271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613455fbcf0 con 0x561345204d80 2024-07-26T07:52:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.398+0000 7f5523e00640 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561345204d80 0x5613455fcd20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f55140057c0 tx=0x7f5514009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a910674043aef16 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.399+0000 7f550e000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.399+0000 7f550e000640 1 -- 172.21.15.97:0/905719271 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514035040 con 0x561345204d80 2024-07-26T07:52:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.399+0000 7f550e000640 1 -- 172.21.15.97:0/905719271 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f5514054750 con 0x561345204d80 2024-07-26T07:52:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.399+0000 7f5532194740 1 -- 172.21.15.97:0/905719271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f551c009e30 con 0x561345204d80 2024-07-26T07:52:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.400+0000 7f550e000640 1 -- 172.21.15.97:0/905719271 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5514054950 con 0x561345204d80 2024-07-26T07:52:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.400+0000 7f5532194740 10 client.15603 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.400+0000 7f5532194740 1 -- 172.21.15.97:0/905719271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561345674c70 con 0x561345204d80 2024-07-26T07:52:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.400+0000 7f550e000640 1 -- 172.21.15.97:0/905719271 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f551c009e30 con 0x561345204d80 2024-07-26T07:52:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.401+0000 7f550e000640 1 -- 172.21.15.97:0/905719271 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5514044070 con 0x561345204d80 2024-07-26T07:52:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.401+0000 7f5532194740 10 client.15603 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.401+0000 7f5532194740 10 client.15603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:52:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.401+0000 7f5532194740 20 client.15603 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.401+0000 7f5532194740 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5613452461b0 0x561345677560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.401+0000 7f5532194740 4 client.15603 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:52:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.401+0000 7f5532194740 1 -- 172.21.15.97:0/905719271 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613455fdea0 con 0x5613452461b0 2024-07-26T07:52:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.401+0000 7f5523400640 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5613452461b0 0x561345677560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.403+0000 7f550e000640 10 client.15603 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.403+0000 7f5523400640 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5613452461b0 0x561345677560 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=889ac91231a4acde server_cookie=e3c5ebe39eef5609 in_seq=0 out_seq=0 2024-07-26T07:52:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.419+0000 7f550e000640 1 -- 172.21.15.97:0/905719271 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613455fdea0 con 0x5613452461b0 2024-07-26T07:52:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.419+0000 7f550e000640 10 client.15603 handle_command_reply: tid=0 2024-07-26T07:52:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.419+0000 7f5532194740 1 client.15603 shutdown 2024-07-26T07:52:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.419+0000 7f5532194740 1 -- 172.21.15.97:0/905719271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561345204d80 msgr2=0x5613455fcd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.419+0000 7f5532194740 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561345204d80 0x5613455fcd20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f55140057c0 tx=0x7f5514009260 comp rx=0 tx=0).stop 2024-07-26T07:52:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 1 -- 172.21.15.97:0/905719271 shutdown_connections 2024-07-26T07:52:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5613452461b0 0x561345677560 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5613452254d0 0x5613455fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56134521d2f0 0x5613455fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 1 --2- 172.21.15.97:0/905719271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561345204d80 0x5613455fcd20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 1 -- 172.21.15.97:0/905719271 >> 172.21.15.97:0/905719271 conn(0x56134523a300 msgr2=0x5613455f3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f550e000640 0 client.15603 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 1 -- 172.21.15.97:0/905719271 shutdown_connections 2024-07-26T07:52:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 1 -- 172.21.15.97:0/905719271 wait complete. 2024-07-26T07:52:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.420+0000 7f5532194740 20 client.15603 trim_cache size 0 max 16384 2024-07-26T07:52:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:52:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:52:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:52:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:52:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.421+0000 7f5529c00640 1 -- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f55080b3c60 msgr2=0x7f55080b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.421+0000 7f5529c00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f55080b3c60 0x7f55080b6170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f55140096c0 tx=0x7f5514031400 comp rx=0 tx=0).stop 2024-07-26T07:52:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.421+0000 7f5529c00640 1 -- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524103f90 msgr2=0x7f5524199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.421+0000 7f5529c00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524103f90 0x7f5524199d50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f551000d950 tx=0x7f551000de10 comp rx=0 tx=0).stop 2024-07-26T07:52:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.422+0000 7f5529c00640 1 -- 172.21.15.97:0/4053649080 shutdown_connections 2024-07-26T07:52:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.422+0000 7f5529c00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f55080b3c60 0x7f55080b6170 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.422+0000 7f5529c00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5524104930 0x7f552419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.422+0000 7f5529c00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5524103f90 0x7f5524199d50 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.422+0000 7f5529c00640 1 --2- 172.21.15.97:0/4053649080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5524102f10 0x7f55241996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.422+0000 7f5529c00640 1 -- 172.21.15.97:0/4053649080 >> 172.21.15.97:0/4053649080 conn(0x7f55240fe850 msgr2=0x7f55241072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.422+0000 7f5529c00640 1 -- 172.21.15.97:0/4053649080 shutdown_connections 2024-07-26T07:52:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:52:48.422+0000 7f5529c00640 1 -- 172.21.15.97:0/4053649080 wait complete. 2024-07-26T07:52:48.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.591+0000 7f2579400640 1 -- 172.21.15.97:0/1028040910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 msgr2=0x7f25741120e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.591+0000 7f2579400640 1 --2- 172.21.15.97:0/1028040910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 0x7f25741120e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f256c009ca0 tx=0x7f256c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.592+0000 7f2579400640 1 -- 172.21.15.97:0/1028040910 shutdown_connections 2024-07-26T07:52:48.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.592+0000 7f2579400640 1 --2- 172.21.15.97:0/1028040910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 0x7f25741120e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.592+0000 7f2579400640 1 --2- 172.21.15.97:0/1028040910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f257410af60 0x7f257410b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.592+0000 7f2579400640 1 --2- 172.21.15.97:0/1028040910 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574105620 0x7f2574105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.592+0000 7f2579400640 1 -- 172.21.15.97:0/1028040910 >> 172.21.15.97:0/1028040910 conn(0x7f25741010b0 msgr2=0x7f2574103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.592+0000 7f2579400640 1 -- 172.21.15.97:0/1028040910 shutdown_connections 2024-07-26T07:52:48.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.592+0000 7f2579400640 1 -- 172.21.15.97:0/1028040910 wait complete. 2024-07-26T07:52:48.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.593+0000 7f2579400640 1 Processor -- start 2024-07-26T07:52:48.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.593+0000 7f2579400640 1 -- start start 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.594+0000 7f2579400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574105620 0x7f257419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.594+0000 7f2579400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f257410af60 0x7f257419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.594+0000 7f2579400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 0x7f25741a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.594+0000 7f2579400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2574115480 con 0x7f257410b8b0 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.594+0000 7f2579400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f2574115300 con 0x7f2574105620 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.594+0000 7f2579400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2574115600 con 0x7f257410af60 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.594+0000 7f2573e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574105620 0x7f257419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2573400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f257410af60 0x7f257419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2578a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 0x7f25741a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2573e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574105620 0x7f257419be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:45954/0 (socket says 172.21.15.97:45954) 2024-07-26T07:52:48.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2573e00640 1 -- 172.21.15.97:0/1591259406 learned_addr learned my addr 172.21.15.97:0/1591259406 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2578a00640 1 -- 172.21.15.97:0/1591259406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f257410af60 msgr2=0x7f257419c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2578a00640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f257410af60 0x7f257419c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2578a00640 1 -- 172.21.15.97:0/1591259406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574105620 msgr2=0x7f257419be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2578a00640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574105620 0x7f257419be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.595+0000 7f2578a00640 1 -- 172.21.15.97:0/1591259406 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25741a1e00 con 0x7f257410b8b0 2024-07-26T07:52:48.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.596+0000 7f2578a00640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 0x7f25741a16d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f256c031040 tx=0x7f256c032bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.596+0000 7f2570c00640 1 -- 172.21.15.97:0/1591259406 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c004040 con 0x7f257410b8b0 2024-07-26T07:52:48.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.596+0000 7f2570c00640 1 -- 172.21.15.97:0/1591259406 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f256c004670 con 0x7f257410b8b0 2024-07-26T07:52:48.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.596+0000 7f2570c00640 1 -- 172.21.15.97:0/1591259406 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c030c00 con 0x7f257410b8b0 2024-07-26T07:52:48.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.596+0000 7f2579400640 1 -- 172.21.15.97:0/1591259406 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25741a2090 con 0x7f257410b8b0 2024-07-26T07:52:48.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.597+0000 7f2579400640 1 -- 172.21.15.97:0/1591259406 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25741a8850 con 0x7f257410b8b0 2024-07-26T07:52:48.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.601+0000 7f258168d740 1 Processor -- start 2024-07-26T07:52:48.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.603+0000 7f2570c00640 1 -- 172.21.15.97:0/1591259406 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 19) ==== 151110+0+0 (secure 0 0 0) 0x7f256c0041e0 con 0x7f257410b8b0 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.603+0000 7f258168d740 1 -- start start 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.603+0000 7f258168d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552547060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.603+0000 7f258168d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552177990 0x5585525475a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.603+0000 7f258168d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x558552547ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.603+0000 7f258168d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5585524bb970 con 0x5585521a0850 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.603+0000 7f258168d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5585524aae70 con 0x558552177990 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.603+0000 7f258168d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5585524aeeb0 con 0x55855217fb70 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.604+0000 7f2573e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552177990 0x5585525475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.604+0000 7f2573e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552177990 0x5585525475a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:45956/0 (socket says 172.21.15.97:45956) 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.604+0000 7f2573e00640 1 -- 172.21.15.97:0/2814064961 learned_addr learned my addr 172.21.15.97:0/2814064961 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.604+0000 7f2573400640 1 --2- 172.21.15.97:0/2814064961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552547060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.605+0000 7f2573e00640 1 -- 172.21.15.97:0/2814064961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 msgr2=0x558552547ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.605+0000 7f2573e00640 1 --2- 172.21.15.97:0/2814064961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x558552547ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.605+0000 7f2573e00640 1 -- 172.21.15.97:0/2814064961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 msgr2=0x558552547060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.605+0000 7f2573e00640 1 --2- 172.21.15.97:0/2814064961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552547060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.605+0000 7f2573e00640 1 -- 172.21.15.97:0/2814064961 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585524338e0 con 0x558552177990 2024-07-26T07:52:48.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.605+0000 7f2570c00640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f25580b3cd0 0x7f25580b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.606+0000 7f2570c00640 1 -- 172.21.15.97:0/1591259406 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f256c105200 con 0x7f257410b8b0 2024-07-26T07:52:48.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.606+0000 7f2573400640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f25580b3cd0 0x7f25580b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.606+0000 7f2573e00640 1 --2- 172.21.15.97:0/2814064961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552177990 0x5585525475a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f256400bda0 tx=0x7f25640107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa6e87fd538c8224 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.607+0000 7f2569200640 1 -- 172.21.15.97:0/2814064961 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256401c680 con 0x558552177990 2024-07-26T07:52:48.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.607+0000 7f2569200640 1 -- 172.21.15.97:0/2814064961 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25640118d0 con 0x558552177990 2024-07-26T07:52:48.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.607+0000 7f2569200640 1 -- 172.21.15.97:0/2814064961 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256401a780 con 0x558552177990 2024-07-26T07:52:48.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.608+0000 7f258168d740 1 -- 172.21.15.97:0/2814064961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552177990 msgr2=0x5585525475a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.608+0000 7f258168d740 1 --2- 172.21.15.97:0/2814064961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552177990 0x5585525475a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f256400bda0 tx=0x7f25640107a0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.608+0000 7f2573400640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f25580b3cd0 0x7f25580b61e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f25741047b0 tx=0x7f25600042c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.609+0000 7f258168d740 1 -- 172.21.15.97:0/2814064961 shutdown_connections 2024-07-26T07:52:48.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.609+0000 7f258168d740 1 --2- 172.21.15.97:0/2814064961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x558552547ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.609+0000 7f258168d740 1 --2- 172.21.15.97:0/2814064961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552177990 0x5585525475a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.609+0000 7f258168d740 1 --2- 172.21.15.97:0/2814064961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552547060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.609+0000 7f258168d740 1 -- 172.21.15.97:0/2814064961 >> 172.21.15.97:0/2814064961 conn(0x558552194c40 msgr2=0x558552544f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.609+0000 7f258168d740 1 -- 172.21.15.97:0/2814064961 shutdown_connections 2024-07-26T07:52:48.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.609+0000 7f258168d740 1 -- 172.21.15.97:0/2814064961 wait complete. 2024-07-26T07:52:48.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.610+0000 7f258168d740 1 -- start start 2024-07-26T07:52:48.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.610+0000 7f258168d740 1 Processor -- start 2024-07-26T07:52:48.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.611+0000 7f258168d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558552177990 0x55855255b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.611+0000 7f258168d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855217fb70 0x55855255b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.611+0000 7f258168d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 0x5585525672d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.611+0000 7f258168d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55855230e7b0 con 0x55855217fb70 2024-07-26T07:52:48.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.611+0000 7f258168d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558552447420 con 0x5585521a0850 2024-07-26T07:52:48.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.611+0000 7f258168d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55855255df70 con 0x558552177990 2024-07-26T07:52:48.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.611+0000 7f258168d740 10 client.0 _ll_register_callbacks cb 0x5585521b80a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:48.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.612+0000 7f2573400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855217fb70 0x55855255b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.612+0000 7f2573e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558552177990 0x55855255b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.612+0000 7f2573400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855217fb70 0x55855255b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46390/0 (socket says 172.21.15.97:46390) 2024-07-26T07:52:48.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.612+0000 7f2573400640 1 -- 172.21.15.97:0/49280401 learned_addr learned my addr 172.21.15.97:0/49280401 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.612+0000 7f2578a00640 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 0x5585525672d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.613+0000 7f2578a00640 1 -- 172.21.15.97:0/49280401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558552177990 msgr2=0x55855255b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.613+0000 7f2578a00640 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558552177990 0x55855255b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.613+0000 7f2578a00640 1 -- 172.21.15.97:0/49280401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855217fb70 msgr2=0x55855255b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.613+0000 7f2578a00640 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855217fb70 0x55855255b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.613+0000 7f2578a00640 1 -- 172.21.15.97:0/49280401 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55855243fc50 con 0x5585521a0850 2024-07-26T07:52:48.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.613+0000 7f2578a00640 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 0x5585525672d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f256c102c00 tx=0x7f256c0cb7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=560308bcea5a9c75 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.613+0000 7f255d600640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:52:48.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.614+0000 7f255d600640 1 -- 172.21.15.97:0/49280401 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c0d0420 con 0x5585521a0850 2024-07-26T07:52:48.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.614+0000 7f258168d740 1 -- 172.21.15.97:0/49280401 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25640118d0 con 0x5585521a0850 2024-07-26T07:52:48.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.614+0000 7f255d600640 1 -- 172.21.15.97:0/49280401 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f256c0c8630 con 0x5585521a0850 2024-07-26T07:52:48.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.614+0000 7f255d600640 1 -- 172.21.15.97:0/49280401 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f256c0ed050 con 0x5585521a0850 2024-07-26T07:52:48.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.614+0000 7f258168d740 10 client.35192 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:48.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.614+0000 7f258168d740 1 -- 172.21.15.97:0/49280401 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55855221aba0 con 0x5585521a0850 2024-07-26T07:52:48.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.615+0000 7f255d600640 1 -- 172.21.15.97:0/49280401 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25640118d0 con 0x5585521a0850 2024-07-26T07:52:48.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.616+0000 7f255d600640 1 -- 172.21.15.97:0/49280401 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f256c0c82d0 con 0x5585521a0850 2024-07-26T07:52:48.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.616+0000 7f258168d740 10 client.35192 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:48.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.616+0000 7f258168d740 10 client.35192 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:52:48.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.616+0000 7f258168d740 20 client.35192 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:48.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.616+0000 7f258168d740 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855215f1d0 0x55855256d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.616+0000 7f258168d740 4 client.35192 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:52:48.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.616+0000 7f258168d740 1 -- 172.21.15.97:0/49280401 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585524bd8c0 con 0x55855215f1d0 2024-07-26T07:52:48.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.617+0000 7f2573e00640 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855215f1d0 0x55855256d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.618+0000 7f2573e00640 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855215f1d0 0x55855256d260 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f0b34695510dd25 server_cookie=fcefcdf4d3b9bd68 in_seq=0 out_seq=0 2024-07-26T07:52:48.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.618+0000 7f255d600640 10 client.35192 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:48.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.620+0000 7f255d600640 1 -- 172.21.15.97:0/49280401 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5585524bd8c0 con 0x55855215f1d0 2024-07-26T07:52:48.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.620+0000 7f255d600640 10 client.35192 handle_command_reply: tid=0 2024-07-26T07:52:48.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.620+0000 7f258168d740 1 client.35192 shutdown 2024-07-26T07:52:48.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.620+0000 7f258168d740 1 -- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 msgr2=0x5585525672d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.620+0000 7f258168d740 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 0x5585525672d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f256c102c00 tx=0x7f256c0cb7b0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 1 -- 172.21.15.97:0/49280401 shutdown_connections 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55855215f1d0 0x55855256d260 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 0x5585525672d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55855217fb70 0x55855255b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f255d600640 0 client.35192 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 1 --2- 172.21.15.97:0/49280401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558552177990 0x55855255b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 1 -- 172.21.15.97:0/49280401 >> 172.21.15.97:0/49280401 conn(0x558552194c40 msgr2=0x558552543590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 1 -- 172.21.15.97:0/49280401 shutdown_connections 2024-07-26T07:52:48.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 1 -- 172.21.15.97:0/49280401 wait complete. 2024-07-26T07:52:48.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.621+0000 7f258168d740 20 client.35192 trim_cache size 0 max 16384 2024-07-26T07:52:48.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.656+0000 7f258168d740 1 Processor -- start 2024-07-26T07:52:48.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.656+0000 7f258168d740 1 -- start start 2024-07-26T07:52:48.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f258168d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558552177990 0x5585525670c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f258168d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x558552567600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f2573e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558552177990 0x5585525670c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f258168d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 0x558552572a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f258168d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f25741acb20 con 0x558552177990 2024-07-26T07:52:48.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f258168d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f25741ae660 con 0x5585521a0850 2024-07-26T07:52:48.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f258168d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f25741ad1f0 con 0x55855217fb70 2024-07-26T07:52:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f2573e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558552177990 0x5585525670c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46392/0 (socket says 172.21.15.97:46392) 2024-07-26T07:52:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.657+0000 7f2573e00640 1 -- 172.21.15.97:0/3644346175 learned_addr learned my addr 172.21.15.97:0/3644346175 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.658+0000 7f2573400640 1 --2- 172.21.15.97:0/3644346175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x558552567600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.658+0000 7f2573e00640 1 -- 172.21.15.97:0/3644346175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 msgr2=0x558552567600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.658+0000 7f2573e00640 1 --2- 172.21.15.97:0/3644346175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x558552567600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.658+0000 7f2573e00640 1 -- 172.21.15.97:0/3644346175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 msgr2=0x558552572a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:52:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.658+0000 7f2573e00640 1 --2- 172.21.15.97:0/3644346175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 0x558552572a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.658+0000 7f2573e00640 1 -- 172.21.15.97:0/3644346175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558552559600 con 0x558552177990 2024-07-26T07:52:48.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.659+0000 7f2573e00640 1 --2- 172.21.15.97:0/3644346175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558552177990 0x5585525670c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2564000c00 tx=0x7f2564005610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ff460e72a77922 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.659+0000 7f255fe00640 1 -- 172.21.15.97:0/3644346175 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25640052d0 con 0x558552177990 2024-07-26T07:52:48.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.660+0000 7f255fe00640 1 -- 172.21.15.97:0/3644346175 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2564003a80 con 0x558552177990 2024-07-26T07:52:48.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.660+0000 7f255fe00640 1 -- 172.21.15.97:0/3644346175 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25640119a0 con 0x558552177990 2024-07-26T07:52:48.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.661+0000 7f258168d740 1 -- 172.21.15.97:0/3644346175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558552177990 msgr2=0x5585525670c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.661+0000 7f258168d740 1 --2- 172.21.15.97:0/3644346175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558552177990 0x5585525670c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2564000c00 tx=0x7f2564005610 comp rx=0 tx=0).stop 2024-07-26T07:52:48.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.661+0000 7f258168d740 1 -- 172.21.15.97:0/3644346175 shutdown_connections 2024-07-26T07:52:48.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.661+0000 7f258168d740 1 --2- 172.21.15.97:0/3644346175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5585521a0850 0x558552572a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.661+0000 7f258168d740 1 --2- 172.21.15.97:0/3644346175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x558552567600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.661+0000 7f258168d740 1 --2- 172.21.15.97:0/3644346175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558552177990 0x5585525670c0 secure :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2564000c00 tx=0x7f2564005610 comp rx=0 tx=0).stop 2024-07-26T07:52:48.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.661+0000 7f258168d740 1 -- 172.21.15.97:0/3644346175 >> 172.21.15.97:0/3644346175 conn(0x558552194c40 msgr2=0x55855255c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.662+0000 7f258168d740 1 -- 172.21.15.97:0/3644346175 shutdown_connections 2024-07-26T07:52:48.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.662+0000 7f258168d740 1 -- 172.21.15.97:0/3644346175 wait complete. 2024-07-26T07:52:48.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.662+0000 7f258168d740 1 -- start start 2024-07-26T07:52:48.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.663+0000 7f258168d740 1 Processor -- start 2024-07-26T07:52:48.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f258168d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x55855255fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f258168d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552560020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f258168d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552134580 0x5585525cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f258168d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f25741adf90 con 0x5585521a0850 2024-07-26T07:52:48.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f258168d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f25741ad8c0 con 0x558552134580 2024-07-26T07:52:48.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f258168d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f25741aab00 con 0x55855217fb70 2024-07-26T07:52:48.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f258168d740 10 client.0 _ll_register_callbacks cb 0x7f256c009410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:52:48.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f2573400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552560020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f2573400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552560020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46402/0 (socket says 172.21.15.97:46402) 2024-07-26T07:52:48.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f2573400640 1 -- 172.21.15.97:0/3091202646 learned_addr learned my addr 172.21.15.97:0/3091202646 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:52:48.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.664+0000 7f2573e00640 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x55855255fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.665+0000 7f2573400640 1 -- 172.21.15.97:0/3091202646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 msgr2=0x55855255fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.665+0000 7f2573400640 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x55855255fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.665+0000 7f2573400640 1 -- 172.21.15.97:0/3091202646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552134580 msgr2=0x5585525cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:52:48.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.665+0000 7f2578a00640 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552134580 0x5585525cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.665+0000 7f2573400640 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552134580 0x5585525cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.665+0000 7f2573400640 1 -- 172.21.15.97:0/3091202646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585525cf0e0 con 0x5585521a0850 2024-07-26T07:52:48.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.666+0000 7f2573400640 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552560020 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2560003e30 tx=0x7f2560014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3cd4ffce94b104a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:52:48.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.666+0000 7f255d600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:52:48.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.667+0000 7f255d600640 1 -- 172.21.15.97:0/3091202646 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256002b060 con 0x5585521a0850 2024-07-26T07:52:48.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.667+0000 7f255d600640 1 -- 172.21.15.97:0/3091202646 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f256001b870 con 0x5585521a0850 2024-07-26T07:52:48.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.667+0000 7f258168d740 1 -- 172.21.15.97:0/3091202646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2564003a80 con 0x5585521a0850 2024-07-26T07:52:48.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.667+0000 7f255d600640 1 -- 172.21.15.97:0/3091202646 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2560003e60 con 0x5585521a0850 2024-07-26T07:52:48.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.667+0000 7f258168d740 10 client.15621 fetch_fsmap learned FSMap version 28 2024-07-26T07:52:48.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.667+0000 7f258168d740 1 -- 172.21.15.97:0/3091202646 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5585525d6460 con 0x5585521a0850 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.668+0000 7f255d600640 1 -- 172.21.15.97:0/3091202646 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2564003a80 con 0x5585521a0850 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.668+0000 7f255d600640 1 -- 172.21.15.97:0/3091202646 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f256002f030 con 0x5585521a0850 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.669+0000 7f258168d740 10 client.15621 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.669+0000 7f258168d740 10 client.15621 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.669+0000 7f258168d740 20 client.15621 populate_metadata read hostname 'smithi097' 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.669+0000 7f258168d740 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5585525d6cb0 0x5585525d90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.669+0000 7f258168d740 4 client.15621 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}] 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.669+0000 7f258168d740 1 -- 172.21.15.97:0/3091202646 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}) -- 0x5585524b9360 con 0x5585525d6cb0 2024-07-26T07:52:48.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.670+0000 7f2573e00640 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5585525d6cb0 0x5585525d90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:52:48.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.671+0000 7f2573e00640 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5585525d6cb0 0x5585525d90d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9338bda91776b068 server_cookie=f2da3633dcd41246 in_seq=0 out_seq=0 2024-07-26T07:52:48.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.671+0000 7f255d600640 10 client.15621 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:48.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.697+0000 7f255d600640 1 -- 172.21.15.97:0/3091202646 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5585524b9360 con 0x5585525d6cb0 2024-07-26T07:52:48.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.697+0000 7f255d600640 10 client.15621 handle_command_reply: tid=0 2024-07-26T07:52:48.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f258168d740 1 client.15621 shutdown 2024-07-26T07:52:48.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f258168d740 1 -- 172.21.15.97:0/3091202646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 msgr2=0x558552560020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f258168d740 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552560020 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2560003e30 tx=0x7f2560014040 comp rx=0 tx=0).stop 2024-07-26T07:52:48.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f258168d740 1 -- 172.21.15.97:0/3091202646 shutdown_connections 2024-07-26T07:52:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f258168d740 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5585525d6cb0 0x5585525d90d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f258168d740 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558552134580 0x5585525cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f258168d740 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5585521a0850 0x558552560020 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f255d600640 0 client.15621 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:52:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.698+0000 7f258168d740 1 --2- 172.21.15.97:0/3091202646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55855217fb70 0x55855255fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.699+0000 7f258168d740 1 -- 172.21.15.97:0/3091202646 >> 172.21.15.97:0/3091202646 conn(0x558552194c40 msgr2=0x558552548090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.699+0000 7f258168d740 1 -- 172.21.15.97:0/3091202646 shutdown_connections 2024-07-26T07:52:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.699+0000 7f258168d740 1 -- 172.21.15.97:0/3091202646 wait complete. 2024-07-26T07:52:48.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.699+0000 7f258168d740 20 client.15621 trim_cache size 0 max 16384 2024-07-26T07:52:48.701 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:52:48.701 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 28, 2024-07-26T07:52:48.701 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:52:48.701 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 2, 2024-07-26T07:52:48.701 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:52:48.701 INFO:teuthology.orchestra.run.smithi097.stdout: "1506616e": { 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 2, 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 142.6, 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:52:48.702 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:52:48.703 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:52:48.703 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:52:48.703 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:52:48.703 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:52:48.703 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:52:48.703 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:52:48.703 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:52:48.703 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.700+0000 7f2579400640 1 -- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f25580b3cd0 msgr2=0x7f25580b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.700+0000 7f2579400640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f25580b3cd0 0x7f25580b61e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f25741047b0 tx=0x7f25600042c0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.700+0000 7f2579400640 1 -- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 msgr2=0x7f25741a16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.700+0000 7f2579400640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 0x7f25741a16d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f256c031040 tx=0x7f256c032bc0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.701+0000 7f2579400640 1 -- 172.21.15.97:0/1591259406 shutdown_connections 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.701+0000 7f2579400640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f25580b3cd0 0x7f25580b61e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.701+0000 7f2579400640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f257410b8b0 0x7f25741a16d0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.701+0000 7f2579400640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f257410af60 0x7f257419c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.701+0000 7f2579400640 1 --2- 172.21.15.97:0/1591259406 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2574105620 0x7f257419be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.701+0000 7f2579400640 1 -- 172.21.15.97:0/1591259406 >> 172.21.15.97:0/1591259406 conn(0x7f25741010b0 msgr2=0x7f257410a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:52:48.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.701+0000 7f2579400640 1 -- 172.21.15.97:0/1591259406 shutdown_connections 2024-07-26T07:52:48.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:52:48.701+0000 7f2579400640 1 -- 172.21.15.97:0/1591259406 wait complete. 2024-07-26T07:52:49.509 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1506616e', quiesce took 7.3 seconds. Will release after: 15.25 2024-07-26T07:52:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:49 smithi097 ceph-mon[32231]: pgmap v541: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:49 smithi097 ceph-mon[32231]: mgrmap e20: x(active, since 15m), standbys: z, y 2024-07-26T07:52:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:49 smithi115 ceph-mon[35364]: pgmap v541: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:50.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:49 smithi115 ceph-mon[35364]: mgrmap e20: x(active, since 15m), standbys: z, y 2024-07-26T07:52:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:49 smithi181 ceph-mon[35869]: pgmap v541: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:49 smithi181 ceph-mon[35869]: mgrmap e20: x(active, since 15m), standbys: z, y 2024-07-26T07:52:52.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:51 smithi097 ceph-mon[32231]: pgmap v542: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:52:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:51 smithi115 ceph-mon[35364]: pgmap v542: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:52:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:51 smithi181 ceph-mon[35869]: pgmap v542: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:52:54.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:53 smithi097 ceph-mon[32231]: pgmap v543: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:53 smithi115 ceph-mon[35364]: pgmap v543: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:53 smithi181 ceph-mon[35869]: pgmap v543: 97 pgs: 97 active+clean; 680 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T07:52:56.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:55 smithi097 ceph-mon[32231]: pgmap v544: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-07-26T07:52:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:55 smithi115 ceph-mon[35364]: pgmap v544: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-07-26T07:52:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:55 smithi181 ceph-mon[35869]: pgmap v544: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-07-26T07:52:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:52:58 smithi115 ceph-mon[35364]: pgmap v545: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:52:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:52:58 smithi181 ceph-mon[35869]: pgmap v545: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:52:58.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:52:58 smithi097 ceph-mon[32231]: pgmap v545: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:53:00.303 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:00 smithi097 ceph-mon[32231]: pgmap v546: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:53:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:00 smithi115 ceph-mon[35364]: pgmap v546: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:53:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:00 smithi181 ceph-mon[35869]: pgmap v546: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:53:01.698 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 -- 172.21.15.97:0/1674273276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9640076ac0 msgr2=0x7f9640075180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.698 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 --2- 172.21.15.97:0/1674273276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9640076ac0 0x7f9640075180 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9630009900 tx=0x7f963002a410 comp rx=0 tx=0).stop 2024-07-26T07:53:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 -- 172.21.15.97:0/1674273276 shutdown_connections 2024-07-26T07:53:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 --2- 172.21.15.97:0/1674273276 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964010b0f0 0x7f964010d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 --2- 172.21.15.97:0/1674273276 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96400756c0 0x7f9640075ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 --2- 172.21.15.97:0/1674273276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9640076ac0 0x7f9640075180 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 -- 172.21.15.97:0/1674273276 >> 172.21.15.97:0/1674273276 conn(0x7f96400fea60 msgr2=0x7f9640100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 -- 172.21.15.97:0/1674273276 shutdown_connections 2024-07-26T07:53:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.697+0000 7f9645600640 1 -- 172.21.15.97:0/1674273276 wait complete. 2024-07-26T07:53:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.698+0000 7f9645600640 1 Processor -- start 2024-07-26T07:53:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f9645600640 1 -- start start 2024-07-26T07:53:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f9645600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96400756c0 0x7f964019bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f9645600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9640076ac0 0x7f964019c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f9645600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f964010b0f0 0x7f964019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f9645600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9640110e60 con 0x7f964010b0f0 2024-07-26T07:53:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f9645600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9640110ce0 con 0x7f9640076ac0 2024-07-26T07:53:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f9645600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9640110fe0 con 0x7f96400756c0 2024-07-26T07:53:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f963f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f964010b0f0 0x7f964019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f963f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f964010b0f0 0x7f964019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51892/0 (socket says 172.21.15.97:51892) 2024-07-26T07:53:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.699+0000 7f963f400640 1 -- 172.21.15.97:0/3384518600 learned_addr learned my addr 172.21.15.97:0/3384518600 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963ea00640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96400756c0 0x7f964019bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963f400640 1 -- 172.21.15.97:0/3384518600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96400756c0 msgr2=0x7f964019bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963f400640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96400756c0 0x7f964019bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963f400640 1 -- 172.21.15.97:0/3384518600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9640076ac0 msgr2=0x7f964019c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963e000640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9640076ac0 0x7f964019c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963f400640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9640076ac0 0x7f964019c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963f400640 1 -- 172.21.15.97:0/3384518600 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964019d520 con 0x7f964010b0f0 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963e000640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9640076ac0 0x7f964019c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.700+0000 7f963f400640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f964010b0f0 0x7f964019cd00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f963800e9d0 tx=0x7f963800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.701+0000 7f9637400640 1 -- 172.21.15.97:0/3384518600 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963800cd80 con 0x7f964010b0f0 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.701+0000 7f9637400640 1 -- 172.21.15.97:0/3384518600 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9638005020 con 0x7f964010b0f0 2024-07-26T07:53:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.701+0000 7f9637400640 1 -- 172.21.15.97:0/3384518600 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96380116d0 con 0x7f964010b0f0 2024-07-26T07:53:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.701+0000 7f9645600640 1 -- 172.21.15.97:0/3384518600 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f964019d810 con 0x7f964010b0f0 2024-07-26T07:53:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.701+0000 7f9645600640 1 -- 172.21.15.97:0/3384518600 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96401a3f00 con 0x7f964010b0f0 2024-07-26T07:53:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.705+0000 7f964c684740 1 Processor -- start 2024-07-26T07:53:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.707+0000 7f9637400640 1 -- 172.21.15.97:0/3384518600 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f9638004a40 con 0x7f964010b0f0 2024-07-26T07:53:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.707+0000 7f964c684740 1 -- start start 2024-07-26T07:53:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f964c684740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b952ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f964c684740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5842f0 0x562d7b953550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f964c684740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 0x562d7b9540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f964c684740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562d7b8c5270 con 0x562d7b5ad1b0 2024-07-26T07:53:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f964c684740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d7b8b4770 con 0x562d7b58c4d0 2024-07-26T07:53:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f964c684740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562d7b8b87b0 con 0x562d7b5842f0 2024-07-26T07:53:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f963ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5842f0 0x562d7b953550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f963e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b952ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f963ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5842f0 0x562d7b953550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:35220/0 (socket says 172.21.15.97:35220) 2024-07-26T07:53:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.708+0000 7f963ea00640 1 -- 172.21.15.97:0/886738174 learned_addr learned my addr 172.21.15.97:0/886738174 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f9637400640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f96240b3c50 0x7f96240b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f963e000640 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 0x562d7b9540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f9637400640 1 -- 172.21.15.97:0/3384518600 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f96380e41a0 con 0x7f964010b0f0 2024-07-26T07:53:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f963e000640 1 -- 172.21.15.97:0/886738174 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5842f0 msgr2=0x562d7b953550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f963e000640 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5842f0 0x562d7b953550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f963e000640 1 -- 172.21.15.97:0/886738174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 msgr2=0x562d7b9540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f963e000640 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 0x562d7b9540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f963e000640 1 -- 172.21.15.97:0/886738174 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d7b618790 con 0x562d7b5ad1b0 2024-07-26T07:53:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f963ea00640 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5842f0 0x562d7b953550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:53:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.709+0000 7f963f400640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f96240b3c50 0x7f96240b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.710+0000 7f963e000640 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b952ee0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f962c013500 tx=0x7f962c00b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1422f303b475a177 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.710+0000 7f962b400640 1 -- 172.21.15.97:0/886738174 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c01d600 con 0x562d7b5ad1b0 2024-07-26T07:53:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.710+0000 7f962b400640 1 -- 172.21.15.97:0/886738174 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f962c011870 con 0x562d7b5ad1b0 2024-07-26T07:53:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.710+0000 7f962b400640 1 -- 172.21.15.97:0/886738174 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c00e7d0 con 0x562d7b5ad1b0 2024-07-26T07:53:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.710+0000 7f963f400640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f96240b3c50 0x7f96240b6160 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f96380a82a0 tx=0x7f9638005b40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.711+0000 7f964c684740 1 -- 172.21.15.97:0/886738174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 msgr2=0x562d7b952ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.711+0000 7f964c684740 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b952ee0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f962c013500 tx=0x7f962c00b3a0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.711+0000 7f964c684740 1 -- 172.21.15.97:0/886738174 shutdown_connections 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.711+0000 7f964c684740 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 0x562d7b9540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.711+0000 7f964c684740 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5842f0 0x562d7b953550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.711+0000 7f964c684740 1 --2- 172.21.15.97:0/886738174 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b952ee0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.711+0000 7f964c684740 1 -- 172.21.15.97:0/886738174 >> 172.21.15.97:0/886738174 conn(0x562d7b5a1300 msgr2=0x562d7b94e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.712+0000 7f964c684740 1 -- 172.21.15.97:0/886738174 shutdown_connections 2024-07-26T07:53:01.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.712+0000 7f964c684740 1 -- 172.21.15.97:0/886738174 wait complete. 2024-07-26T07:53:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.713+0000 7f964c684740 1 -- start start 2024-07-26T07:53:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.713+0000 7f964c684740 1 Processor -- start 2024-07-26T07:53:01.715 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f964c684740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5842f0 0x562d7b965290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.715 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f964c684740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b965950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.715 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f964c684740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b9664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.715 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f964c684740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562d7b7eeab0 con 0x562d7b5ad1b0 2024-07-26T07:53:01.715 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f964c684740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562d7b851d50 con 0x562d7b5842f0 2024-07-26T07:53:01.715 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f964c684740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562d7b954bc0 con 0x562d7b58c4d0 2024-07-26T07:53:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f964c684740 10 client.0 _ll_register_callbacks cb 0x562d7b5c4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f963ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b965950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.714+0000 7f963ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b965950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:35222/0 (socket says 172.21.15.97:35222) 2024-07-26T07:53:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963ea00640 1 -- 172.21.15.97:0/4058483882 learned_addr learned my addr 172.21.15.97:0/4058483882 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963e000640 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b9664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963e000640 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5842f0 0x562d7b965290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963e000640 1 -- 172.21.15.97:0/4058483882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 msgr2=0x562d7b965950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963e000640 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b965950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963e000640 1 -- 172.21.15.97:0/4058483882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5842f0 msgr2=0x562d7b965290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963e000640 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5842f0 0x562d7b965290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963e000640 1 -- 172.21.15.97:0/4058483882 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d7b691620 con 0x562d7b5ad1b0 2024-07-26T07:53:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f963e000640 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b9664c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f962c0057c0 tx=0x7f962c00bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e914d03d06a0239 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.715+0000 7f961fe00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.716+0000 7f961fe00640 1 -- 172.21.15.97:0/4058483882 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c011eb0 con 0x562d7b5ad1b0 2024-07-26T07:53:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.716+0000 7f961fe00640 1 -- 172.21.15.97:0/4058483882 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f962c004020 con 0x562d7b5ad1b0 2024-07-26T07:53:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.716+0000 7f964c684740 1 -- 172.21.15.97:0/4058483882 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f962c011870 con 0x562d7b5ad1b0 2024-07-26T07:53:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.716+0000 7f961fe00640 1 -- 172.21.15.97:0/4058483882 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f962c004220 con 0x562d7b5ad1b0 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.716+0000 7f964c684740 10 client.15633 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.716+0000 7f964c684740 1 -- 172.21.15.97:0/4058483882 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562d7b96f410 con 0x562d7b5ad1b0 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.717+0000 7f961fe00640 1 -- 172.21.15.97:0/4058483882 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f962c011870 con 0x562d7b5ad1b0 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.717+0000 7f961fe00640 1 -- 172.21.15.97:0/4058483882 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f962c02b020 con 0x562d7b5ad1b0 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.717+0000 7f964c684740 10 client.15633 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.717+0000 7f964c684740 10 client.15633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.717+0000 7f964c684740 20 client.15633 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.717+0000 7f964c684740 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562d7b56bd80 0x562d7b972930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.717+0000 7f964c684740 4 client.15633 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:53:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.717+0000 7f964c684740 1 -- 172.21.15.97:0/4058483882 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d7b8c71c0 con 0x562d7b56bd80 2024-07-26T07:53:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.718+0000 7f963ea00640 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562d7b56bd80 0x562d7b972930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.719+0000 7f963ea00640 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562d7b56bd80 0x562d7b972930 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18581f302199c1d server_cookie=82013d5036bdfa05 in_seq=0 out_seq=0 2024-07-26T07:53:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.719+0000 7f961fe00640 10 client.15633 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f961fe00640 1 -- 172.21.15.97:0/4058483882 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562d7b8c71c0 con 0x562d7b56bd80 2024-07-26T07:53:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f961fe00640 10 client.15633 handle_command_reply: tid=0 2024-07-26T07:53:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 client.15633 shutdown 2024-07-26T07:53:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 -- 172.21.15.97:0/4058483882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 msgr2=0x562d7b9664c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b9664c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f962c0057c0 tx=0x7f962c00bac0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 -- 172.21.15.97:0/4058483882 shutdown_connections 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562d7b56bd80 0x562d7b972930 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5ad1b0 0x562d7b9664c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b965950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 --2- 172.21.15.97:0/4058483882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5842f0 0x562d7b965290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 -- 172.21.15.97:0/4058483882 >> 172.21.15.97:0/4058483882 conn(0x562d7b5a1300 msgr2=0x562d7b94d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f961fe00640 0 client.15633 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 -- 172.21.15.97:0/4058483882 shutdown_connections 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.721+0000 7f964c684740 1 -- 172.21.15.97:0/4058483882 wait complete. 2024-07-26T07:53:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.722+0000 7f964c684740 20 client.15633 trim_cache size 0 max 16384 2024-07-26T07:53:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.737+0000 7f964c684740 1 Processor -- start 2024-07-26T07:53:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.738+0000 7f964c684740 1 -- start start 2024-07-26T07:53:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.738+0000 7f964c684740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b9770e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.738+0000 7f964c684740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b977620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.738+0000 7f964c684740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5ad1b0 0x562d7b977b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.738+0000 7f964c684740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f96401a8a70 con 0x562d7b5842f0 2024-07-26T07:53:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.738+0000 7f964c684740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f96401aa5b0 con 0x562d7b5ad1b0 2024-07-26T07:53:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.738+0000 7f964c684740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f96401a9140 con 0x562d7b58c4d0 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.738+0000 7f963e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b9770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b9770e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51926/0 (socket says 172.21.15.97:51926) 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 -- 172.21.15.97:0/2083704830 learned_addr learned my addr 172.21.15.97:0/2083704830 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963ea00640 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b977620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5ad1b0 0x562d7b977b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 -- 172.21.15.97:0/2083704830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 msgr2=0x562d7b977620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b977620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 -- 172.21.15.97:0/2083704830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5ad1b0 msgr2=0x562d7b977b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5ad1b0 0x562d7b977b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 -- 172.21.15.97:0/2083704830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d7b94f0a0 con 0x562d7b5842f0 2024-07-26T07:53:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.739+0000 7f963e000640 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b9770e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f962c011be0 tx=0x7f962c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f58b4b81422b03 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.740+0000 7f962aa00640 1 -- 172.21.15.97:0/2083704830 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c011eb0 con 0x562d7b5842f0 2024-07-26T07:53:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.740+0000 7f962aa00640 1 -- 172.21.15.97:0/2083704830 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f962c00bbd0 con 0x562d7b5842f0 2024-07-26T07:53:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.740+0000 7f962aa00640 1 -- 172.21.15.97:0/2083704830 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c0043c0 con 0x562d7b5842f0 2024-07-26T07:53:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 -- 172.21.15.97:0/2083704830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 msgr2=0x562d7b9770e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b9770e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f962c011be0 tx=0x7f962c000bc0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 -- 172.21.15.97:0/2083704830 shutdown_connections 2024-07-26T07:53:01.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b5ad1b0 0x562d7b977b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b58c4d0 0x562d7b977620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 --2- 172.21.15.97:0/2083704830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b9770e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 -- 172.21.15.97:0/2083704830 >> 172.21.15.97:0/2083704830 conn(0x562d7b5a1300 msgr2=0x562d7b94fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:01.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 -- 172.21.15.97:0/2083704830 shutdown_connections 2024-07-26T07:53:01.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.741+0000 7f964c684740 1 -- 172.21.15.97:0/2083704830 wait complete. 2024-07-26T07:53:01.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 1 -- start start 2024-07-26T07:53:01.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 1 Processor -- start 2024-07-26T07:53:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b96a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 0x562d7b96a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5ad1b0 0x562d7b96b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f96401a9ee0 con 0x562d7b5842f0 2024-07-26T07:53:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f96401a9810 con 0x562d7b58c4d0 2024-07-26T07:53:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f96401a6a50 con 0x562d7b5ad1b0 2024-07-26T07:53:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.742+0000 7f964c684740 10 client.0 _ll_register_callbacks cb 0x7f9630009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b96a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b96a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51934/0 (socket says 172.21.15.97:51934) 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 -- 172.21.15.97:0/257350996 learned_addr learned my addr 172.21.15.97:0/257350996 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963ea00640 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 0x562d7b96a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5ad1b0 0x562d7b96b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 -- 172.21.15.97:0/257350996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5ad1b0 msgr2=0x562d7b96b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5ad1b0 0x562d7b96b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 -- 172.21.15.97:0/257350996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 msgr2=0x562d7b96a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 0x562d7b96a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.743+0000 7f963e000640 1 -- 172.21.15.97:0/257350996 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d7b96bc90 con 0x562d7b5842f0 2024-07-26T07:53:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.744+0000 7f963e000640 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b96a300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f962c005940 tx=0x7f962c00e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce16a64729cfd7c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.744+0000 7f961fe00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.744+0000 7f961fe00640 1 -- 172.21.15.97:0/257350996 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c011eb0 con 0x562d7b5842f0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.744+0000 7f961fe00640 1 -- 172.21.15.97:0/257350996 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f962c0043c0 con 0x562d7b5842f0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.744+0000 7f964c684740 1 -- 172.21.15.97:0/257350996 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f962c00bbd0 con 0x562d7b5842f0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.744+0000 7f961fe00640 1 -- 172.21.15.97:0/257350996 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f962c00e8a0 con 0x562d7b5842f0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f964c684740 10 client.15639 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f964c684740 1 -- 172.21.15.97:0/257350996 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562d7b96cf80 con 0x562d7b5842f0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f961fe00640 1 -- 172.21.15.97:0/257350996 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f962c00bbd0 con 0x562d7b5842f0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f961fe00640 1 -- 172.21.15.97:0/257350996 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f962c02b070 con 0x562d7b5842f0 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f964c684740 10 client.15639 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f964c684740 10 client.15639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f964c684740 20 client.15639 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f964c684740 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562d7b5411c0 0x562d7b9dd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f964c684740 4 client.15639 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:53:01.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.745+0000 7f964c684740 1 -- 172.21.15.97:0/257350996 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d7b8c2c60 con 0x562d7b5411c0 2024-07-26T07:53:01.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.746+0000 7f963ea00640 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562d7b5411c0 0x562d7b9dd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:01.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.747+0000 7f961fe00640 10 client.15639 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:01.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.747+0000 7f963ea00640 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562d7b5411c0 0x562d7b9dd0d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dda25ad4256c72c2 server_cookie=97b67c523c608e83 in_seq=0 out_seq=0 2024-07-26T07:53:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.756+0000 7f961fe00640 1 -- 172.21.15.97:0/257350996 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d7b8c2c60 con 0x562d7b5411c0 2024-07-26T07:53:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.756+0000 7f961fe00640 10 client.15639 handle_command_reply: tid=0 2024-07-26T07:53:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.756+0000 7f964c684740 1 client.15639 shutdown 2024-07-26T07:53:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 -- 172.21.15.97:0/257350996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 msgr2=0x562d7b96a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b96a300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f962c005940 tx=0x7f962c00e6e0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 -- 172.21.15.97:0/257350996 shutdown_connections 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562d7b5411c0 0x562d7b9dd0d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d7b5ad1b0 0x562d7b96b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562d7b58c4d0 0x562d7b96a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 --2- 172.21.15.97:0/257350996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562d7b5842f0 0x562d7b96a300 secure :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f962c005940 tx=0x7f962c00e6e0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f961fe00640 0 client.15639 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 -- 172.21.15.97:0/257350996 >> 172.21.15.97:0/257350996 conn(0x562d7b5a1300 msgr2=0x562d7b9638e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 -- 172.21.15.97:0/257350996 shutdown_connections 2024-07-26T07:53:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 1 -- 172.21.15.97:0/257350996 wait complete. 2024-07-26T07:53:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.757+0000 7f964c684740 20 client.15639 trim_cache size 0 max 16384 2024-07-26T07:53:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:53:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:53:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:53:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 -- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f96240b3c50 msgr2=0x7f96240b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f96240b3c50 0x7f96240b6160 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f96380a82a0 tx=0x7f9638005b40 comp rx=0 tx=0).stop 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 -- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f964010b0f0 msgr2=0x7f964019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f964010b0f0 0x7f964019cd00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f963800e9d0 tx=0x7f963800ee90 comp rx=0 tx=0).stop 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 -- 172.21.15.97:0/3384518600 shutdown_connections 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f96240b3c50 0x7f96240b6160 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f964010b0f0 0x7f964019cd00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9640076ac0 0x7f964019c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 --2- 172.21.15.97:0/3384518600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96400756c0 0x7f964019bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:01.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.758+0000 7f9645600640 1 -- 172.21.15.97:0/3384518600 >> 172.21.15.97:0/3384518600 conn(0x7f96400fea60 msgr2=0x7f96401070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.759+0000 7f9645600640 1 -- 172.21.15.97:0/3384518600 shutdown_connections 2024-07-26T07:53:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:01.759+0000 7f9645600640 1 -- 172.21.15.97:0/3384518600 wait complete. 2024-07-26T07:53:02.285 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:02 smithi181 ceph-mon[35869]: pgmap v547: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:53:02.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:02 smithi097 ceph-mon[32231]: pgmap v547: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:53:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:02 smithi115 ceph-mon[35364]: pgmap v547: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T07:53:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:53:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:53:03.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:53:04.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:04 smithi115 ceph-mon[35364]: pgmap v548: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T07:53:04.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:53:04.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:53:04.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:53:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:04 smithi181 ceph-mon[35869]: pgmap v548: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T07:53:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:53:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:53:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:53:04.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:04 smithi097 ceph-mon[32231]: pgmap v548: 97 pgs: 97 active+clean; 681 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T07:53:04.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:53:04.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:53:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:53:04.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:53:05.310 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:53:06.032 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:06 smithi097 ceph-mon[32231]: pgmap v549: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T07:53:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:06 smithi115 ceph-mon[35364]: pgmap v549: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T07:53:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:06 smithi181 ceph-mon[35869]: pgmap v549: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T07:53:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.344+0000 7f1933e00640 1 -- 172.21.15.97:0/3252748040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 msgr2=0x7f1934109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.344+0000 7f1933e00640 1 --2- 172.21.15.97:0/3252748040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 0x7f1934109130 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1920009900 tx=0x7f192002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.344+0000 7f1933e00640 1 -- 172.21.15.97:0/3252748040 shutdown_connections 2024-07-26T07:53:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.344+0000 7f1933e00640 1 --2- 172.21.15.97:0/3252748040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1934109670 0x7f193410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.344+0000 7f1933e00640 1 --2- 172.21.15.97:0/3252748040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 0x7f1934109130 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.344+0000 7f1933e00640 1 --2- 172.21.15.97:0/3252748040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19341033e0 0x7f19341037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.344+0000 7f1933e00640 1 -- 172.21.15.97:0/3252748040 >> 172.21.15.97:0/3252748040 conn(0x7f19340fee30 msgr2=0x7f1934101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.344+0000 7f1933e00640 1 -- 172.21.15.97:0/3252748040 shutdown_connections 2024-07-26T07:53:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.345+0000 7f1933e00640 1 -- 172.21.15.97:0/3252748040 wait complete. 2024-07-26T07:53:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.346+0000 7f1933e00640 1 Processor -- start 2024-07-26T07:53:07.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1933e00640 1 -- start start 2024-07-26T07:53:07.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1933e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19341033e0 0x7f193419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:07.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1933e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 0x7f193419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1933e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1934109670 0x7f19341a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1933e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1934113200 con 0x7f1934108d20 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1933e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1934113080 con 0x7f1934109670 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1933e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1934113380 con 0x7f19341033e0 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1932000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 0x7f193419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1932000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 0x7f193419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46628/0 (socket says 172.21.15.97:46628) 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.347+0000 7f1932000640 1 -- 172.21.15.97:0/1013191985 learned_addr learned my addr 172.21.15.97:0/1013191985 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.348+0000 7f1932a00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19341033e0 0x7f193419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:07.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.348+0000 7f1933400640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1934109670 0x7f19341a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.348+0000 7f1932000640 1 -- 172.21.15.97:0/1013191985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19341033e0 msgr2=0x7f193419be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.348+0000 7f1932000640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19341033e0 0x7f193419be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.348+0000 7f1932000640 1 -- 172.21.15.97:0/1013191985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1934109670 msgr2=0x7f19341a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.348+0000 7f1932000640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1934109670 0x7f19341a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.348+0000 7f1932000640 1 -- 172.21.15.97:0/1013191985 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19341a1d40 con 0x7f1934108d20 2024-07-26T07:53:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.348+0000 7f1932000640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 0x7f193419c430 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f192002a910 tx=0x7f1920003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:07.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.349+0000 7f192b400640 1 -- 172.21.15.97:0/1013191985 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192003f070 con 0x7f1934108d20 2024-07-26T07:53:07.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.349+0000 7f192b400640 1 -- 172.21.15.97:0/1013191985 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f192003bdc0 con 0x7f1934108d20 2024-07-26T07:53:07.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.349+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19341a1fd0 con 0x7f1934108d20 2024-07-26T07:53:07.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.349+0000 7f192b400640 1 -- 172.21.15.97:0/1013191985 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920035030 con 0x7f1934108d20 2024-07-26T07:53:07.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.349+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19341a2500 con 0x7f1934108d20 2024-07-26T07:53:07.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.350+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19341037f0 con 0x7f1934108d20 2024-07-26T07:53:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.353+0000 7f192b400640 1 -- 172.21.15.97:0/1013191985 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f1920042070 con 0x7f1934108d20 2024-07-26T07:53:07.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.354+0000 7f192b400640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19180b3c80 0x7f19180b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:07.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.354+0000 7f192b400640 1 -- 172.21.15.97:0/1013191985 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f1920104a20 con 0x7f1934108d20 2024-07-26T07:53:07.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.354+0000 7f1932a00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19180b3c80 0x7f19180b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:07.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.355+0000 7f1932a00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19180b3c80 0x7f19180b6190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1924005fd0 tx=0x7f1924005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:07.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.356+0000 7f192b400640 1 -- 172.21.15.97:0/1013191985 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f19341a2500 con 0x7f1934108d20 2024-07-26T07:53:07.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.590+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f193419cfc0 con 0x7f1934108d20 2024-07-26T07:53:07.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.592+0000 7f192b400640 1 -- 172.21.15.97:0/1013191985 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11990 (secure 0 0 0) 0x7f19200cc040 con 0x7f1934108d20 2024-07-26T07:53:07.594 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:53:07.594 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"btime":"2024-07-26T07:51:41:000369+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:51:40.647454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:53:07.594 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-07-26T07:53:07.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19180b3c80 msgr2=0x7f19180b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:07.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19180b3c80 0x7f19180b6190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1924005fd0 tx=0x7f1924005e20 comp rx=0 tx=0).stop 2024-07-26T07:53:07.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 msgr2=0x7f193419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:07.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 0x7f193419c430 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f192002a910 tx=0x7f1920003d60 comp rx=0 tx=0).stop 2024-07-26T07:53:07.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 shutdown_connections 2024-07-26T07:53:07.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19180b3c80 0x7f19180b6190 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1934109670 0x7f19341a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1934108d20 0x7f193419c430 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 --2- 172.21.15.97:0/1013191985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19341033e0 0x7f193419be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:07.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 >> 172.21.15.97:0/1013191985 conn(0x7f19340fee30 msgr2=0x7f19341063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:07.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 shutdown_connections 2024-07-26T07:53:07.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:07.595+0000 7f1933e00640 1 -- 172.21.15.97:0/1013191985 wait complete. 2024-07-26T07:53:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:08 smithi115 ceph-mon[35364]: pgmap v550: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:08 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1013191985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:08 smithi181 ceph-mon[35869]: pgmap v550: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:08 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1013191985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:08 smithi097 ceph-mon[32231]: pgmap v550: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:08 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1013191985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:08.581 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db --set-id 1506616e --release --await' 2024-07-26T07:53:08.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db --set-id 1506616e --release --await 2024-07-26T07:53:09.022 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:53:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:10 smithi115 ceph-mon[35364]: pgmap v551: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:10 smithi181 ceph-mon[35869]: pgmap v551: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:10 smithi097 ceph-mon[32231]: pgmap v551: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:12 smithi097 ceph-mon[32231]: pgmap v552: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:12 smithi115 ceph-mon[35364]: pgmap v552: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:12 smithi181 ceph-mon[35869]: pgmap v552: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T07:53:13.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 -- 172.21.15.97:0/4084846070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c78075b30 msgr2=0x7f0c7810f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/4084846070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c78075b30 0x7f0c7810f5c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009900 tx=0x7f0c6002a410 comp rx=0 tx=0).stop 2024-07-26T07:53:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 -- 172.21.15.97:0/4084846070 shutdown_connections 2024-07-26T07:53:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/4084846070 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c7810fb00 0x7f0c78111f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/4084846070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c78075b30 0x7f0c7810f5c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/4084846070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c780751e0 0x7f0c780755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 -- 172.21.15.97:0/4084846070 >> 172.21.15.97:0/4084846070 conn(0x7f0c780ff010 msgr2=0x7f0c78101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:13.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 -- 172.21.15.97:0/4084846070 shutdown_connections 2024-07-26T07:53:13.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.819+0000 7f0c7ec00640 1 -- 172.21.15.97:0/4084846070 wait complete. 2024-07-26T07:53:13.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.820+0000 7f0c7ec00640 1 Processor -- start 2024-07-26T07:53:13.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.820+0000 7f0c7ec00640 1 -- start start 2024-07-26T07:53:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c780751e0 0x7f0c7819c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7ec00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c78075b30 0x7f0c7819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7ec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c7810fb00 0x7f0c781a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7ec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0c781155a0 con 0x7f0c7810fb00 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7ec00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0c78115420 con 0x7f0c78075b30 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0c78115720 con 0x7f0c780751e0 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c7810fb00 0x7f0c781a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c7810fb00 0x7f0c781a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46646/0 (socket says 172.21.15.97:46646) 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c7cc00640 1 -- 172.21.15.97:0/3757624150 learned_addr learned my addr 172.21.15.97:0/3757624150 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.821+0000 7f0c77e00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c780751e0 0x7f0c7819c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.822+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c78075b30 0x7f0c7819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.822+0000 7f0c7cc00640 1 -- 172.21.15.97:0/3757624150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c780751e0 msgr2=0x7f0c7819c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.822+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c780751e0 0x7f0c7819c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.822+0000 7f0c7cc00640 1 -- 172.21.15.97:0/3757624150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c78075b30 msgr2=0x7f0c7819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.822+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c78075b30 0x7f0c7819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.822+0000 7f0c7cc00640 1 -- 172.21.15.97:0/3757624150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c781a2000 con 0x7f0c7810fb00 2024-07-26T07:53:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.822+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c7810fb00 0x7f0c781a18d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000eaa0 tx=0x7f0c7000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.822+0000 7f0c75600640 1 -- 172.21.15.97:0/3757624150 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7000cdb0 con 0x7f0c7810fb00 2024-07-26T07:53:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.823+0000 7f0c7ec00640 1 -- 172.21.15.97:0/3757624150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c781a22f0 con 0x7f0c7810fb00 2024-07-26T07:53:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.823+0000 7f0c7ec00640 1 -- 172.21.15.97:0/3757624150 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c781a8a50 con 0x7f0c7810fb00 2024-07-26T07:53:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.823+0000 7f0c75600640 1 -- 172.21.15.97:0/3757624150 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c70005020 con 0x7f0c7810fb00 2024-07-26T07:53:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.823+0000 7f0c75600640 1 -- 172.21.15.97:0/3757624150 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c700116d0 con 0x7f0c7810fb00 2024-07-26T07:53:13.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.826+0000 7f0c85522740 1 Processor -- start 2024-07-26T07:53:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.828+0000 7f0c75600640 1 -- 172.21.15.97:0/3757624150 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f0c70004a40 con 0x7f0c7810fb00 2024-07-26T07:53:13.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.828+0000 7f0c85522740 1 -- start start 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c85522740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb4979bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c85522740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493ce660 0x55eb4979c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c85522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493d6840 0x55eb4979ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c85522740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55eb49712f30 con 0x55eb493f7520 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c85522740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55eb49702430 con 0x55eb493ce660 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c85522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55eb49706470 con 0x55eb493d6840 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb4979bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb4979bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46662/0 (socket says 172.21.15.97:46662) 2024-07-26T07:53:13.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c6fe00640 1 -- 172.21.15.97:0/2522846586 learned_addr learned my addr 172.21.15.97:0/2522846586 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:13.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.829+0000 7f0c77e00640 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493ce660 0x55eb4979c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.830+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493d6840 0x55eb4979ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.830+0000 7f0c6fe00640 1 -- 172.21.15.97:0/2522846586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493d6840 msgr2=0x55eb4979ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.830+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493d6840 0x55eb4979ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.830+0000 7f0c6fe00640 1 -- 172.21.15.97:0/2522846586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493ce660 msgr2=0x55eb4979c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.830+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493ce660 0x55eb4979c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.830+0000 7f0c6fe00640 1 -- 172.21.15.97:0/2522846586 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb49462b00 con 0x55eb493f7520 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.830+0000 7f0c75600640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c5c0b3c50 0x7f0c5c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.830+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb4979bfc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0c600096c0 tx=0x7f0c60009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2093f3e720cab80 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.831+0000 7f0c75600640 1 -- 172.21.15.97:0/3757624150 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f0c700e3fa0 con 0x7f0c7810fb00 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.831+0000 7f0c77e00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c5c0b3c50 0x7f0c5c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.831+0000 7f0c6d200640 1 -- 172.21.15.97:0/2522846586 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c60035070 con 0x55eb493f7520 2024-07-26T07:53:13.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.831+0000 7f0c6d200640 1 -- 172.21.15.97:0/2522846586 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c600046c0 con 0x55eb493f7520 2024-07-26T07:53:13.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.831+0000 7f0c6d200640 1 -- 172.21.15.97:0/2522846586 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c60003850 con 0x55eb493f7520 2024-07-26T07:53:13.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 -- 172.21.15.97:0/2522846586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 msgr2=0x55eb4979bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c77e00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c5c0b3c50 0x7f0c5c0b6160 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55eb4979ac40 tx=0x7f0c6800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb4979bfc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0c600096c0 tx=0x7f0c60009560 comp rx=0 tx=0).stop 2024-07-26T07:53:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 -- 172.21.15.97:0/2522846586 shutdown_connections 2024-07-26T07:53:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493d6840 0x55eb4979ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493ce660 0x55eb4979c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 --2- 172.21.15.97:0/2522846586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb4979bfc0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 -- 172.21.15.97:0/2522846586 >> 172.21.15.97:0/2522846586 conn(0x55eb493eb910 msgr2=0x55eb49799eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:13.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 -- 172.21.15.97:0/2522846586 shutdown_connections 2024-07-26T07:53:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.832+0000 7f0c85522740 1 -- 172.21.15.97:0/2522846586 wait complete. 2024-07-26T07:53:13.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.833+0000 7f0c85522740 1 -- start start 2024-07-26T07:53:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.833+0000 7f0c85522740 1 Processor -- start 2024-07-26T07:53:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c85522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c85522740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c85522740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb497b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c85522740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55eb49565bb0 con 0x55eb493d6840 2024-07-26T07:53:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c85522740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55eb4969e910 con 0x55eb493f7520 2024-07-26T07:53:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c85522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55eb497aa6e0 con 0x55eb493ce660 2024-07-26T07:53:13.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c85522740 10 client.0 _ll_register_callbacks cb 0x55eb4940ed70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497a7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46668/0 (socket says 172.21.15.97:46668) 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.834+0000 7f0c6fe00640 1 -- 172.21.15.97:0/421666233 learned_addr learned my addr 172.21.15.97:0/421666233 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb497b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c77e00640 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497a76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c6fe00640 1 -- 172.21.15.97:0/421666233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 msgr2=0x55eb497a76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497a76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c6fe00640 1 -- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 msgr2=0x55eb497b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb497b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c6fe00640 1 -- 172.21.15.97:0/421666233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb494d9960 con 0x55eb493d6840 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb497b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497a7be0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0c60006010 tx=0x7f0c600055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b073f93e5ca2fe server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.835+0000 7f0c65600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.836+0000 7f0c65600640 1 -- 172.21.15.97:0/421666233 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c60035070 con 0x55eb493d6840 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.836+0000 7f0c65600640 1 -- 172.21.15.97:0/421666233 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f0c60004c80 con 0x55eb493d6840 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.836+0000 7f0c85522740 1 -- 172.21.15.97:0/421666233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c600046c0 con 0x55eb493d6840 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.836+0000 7f0c65600640 1 -- 172.21.15.97:0/421666233 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0c600035f0 con 0x55eb493d6840 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.836+0000 7f0c85522740 10 client.15657 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.836+0000 7f0c85522740 1 -- 172.21.15.97:0/421666233 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb497b6480 con 0x55eb493d6840 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.836+0000 7f0c65600640 1 -- 172.21.15.97:0/421666233 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c600046c0 con 0x55eb493d6840 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.837+0000 7f0c65600640 1 -- 172.21.15.97:0/421666233 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0c600395a0 con 0x55eb493d6840 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.837+0000 7f0c85522740 10 client.15657 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.837+0000 7f0c85522740 10 client.15657 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:53:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.837+0000 7f0c85522740 20 client.15657 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:13.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.837+0000 7f0c85522740 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55eb493b5ea0 0x55eb497b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.837+0000 7f0c85522740 4 client.15657 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:53:13.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.837+0000 7f0c85522740 1 -- 172.21.15.97:0/421666233 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb49714e80 con 0x55eb493b5ea0 2024-07-26T07:53:13.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.837+0000 7f0c77e00640 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55eb493b5ea0 0x55eb497b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.838+0000 7f0c77e00640 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55eb493b5ea0 0x55eb497b97e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97c796b301d18ff9 server_cookie=4955bde912f8722c in_seq=0 out_seq=0 2024-07-26T07:53:13.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.838+0000 7f0c65600640 10 client.15657 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:13.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.840+0000 7f0c65600640 1 -- 172.21.15.97:0/421666233 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eb49714e80 con 0x55eb493b5ea0 2024-07-26T07:53:13.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.840+0000 7f0c65600640 10 client.15657 handle_command_reply: tid=0 2024-07-26T07:53:13.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.840+0000 7f0c85522740 1 client.15657 shutdown 2024-07-26T07:53:13.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 -- 172.21.15.97:0/421666233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 msgr2=0x55eb497a7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497a7be0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0c60006010 tx=0x7f0c600055d0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 -- 172.21.15.97:0/421666233 shutdown_connections 2024-07-26T07:53:13.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55eb493b5ea0 0x55eb497b97e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb497b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497a7be0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 --2- 172.21.15.97:0/421666233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497a76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 -- 172.21.15.97:0/421666233 >> 172.21.15.97:0/421666233 conn(0x55eb493eb910 msgr2=0x55eb4979f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:13.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c65600640 0 client.15657 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:13.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 -- 172.21.15.97:0/421666233 shutdown_connections 2024-07-26T07:53:13.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 1 -- 172.21.15.97:0/421666233 wait complete. 2024-07-26T07:53:13.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.841+0000 7f0c85522740 20 client.15657 trim_cache size 0 max 16384 2024-07-26T07:53:13.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.861+0000 7f0c85522740 1 Processor -- start 2024-07-26T07:53:13.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.862+0000 7f0c85522740 1 -- start start 2024-07-26T07:53:13.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.862+0000 7f0c85522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.862+0000 7f0c85522740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493d6840 0x55eb497a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.862+0000 7f0c85522740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb497a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.862+0000 7f0c85522740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5400e600 con 0x55eb493f7520 2024-07-26T07:53:13.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.862+0000 7f0c85522740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0c54010140 con 0x55eb493d6840 2024-07-26T07:53:13.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.862+0000 7f0c85522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5400ecd0 con 0x55eb493ce660 2024-07-26T07:53:13.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.863+0000 7f0c7cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb497a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.863+0000 7f0c7cc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb497a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46670/0 (socket says 172.21.15.97:46670) 2024-07-26T07:53:13.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.863+0000 7f0c7cc00640 1 -- 172.21.15.97:0/95097137 learned_addr learned my addr 172.21.15.97:0/95097137 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:13.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.863+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493d6840 0x55eb497a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.863+0000 7f0c77e00640 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.864+0000 7f0c7cc00640 1 -- 172.21.15.97:0/95097137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 msgr2=0x55eb497b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:13.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.864+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.864+0000 7f0c7cc00640 1 -- 172.21.15.97:0/95097137 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493d6840 msgr2=0x55eb497a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:13.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.864+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493d6840 0x55eb497a92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.864+0000 7f0c7cc00640 1 -- 172.21.15.97:0/95097137 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb497a8960 con 0x55eb493f7520 2024-07-26T07:53:13.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.864+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb497a1050 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0c700e19e0 tx=0x7f0c700ab670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1547166f8707e235 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:13.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.864+0000 7f0c67e00640 1 -- 172.21.15.97:0/95097137 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c700c5070 con 0x55eb493f7520 2024-07-26T07:53:13.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.865+0000 7f0c67e00640 1 -- 172.21.15.97:0/95097137 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c700b3ee0 con 0x55eb493f7520 2024-07-26T07:53:13.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.865+0000 7f0c67e00640 1 -- 172.21.15.97:0/95097137 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7001f040 con 0x55eb493f7520 2024-07-26T07:53:13.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.866+0000 7f0c85522740 1 -- 172.21.15.97:0/95097137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 msgr2=0x55eb497a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.866+0000 7f0c85522740 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb497a1050 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0c700e19e0 tx=0x7f0c700ab670 comp rx=0 tx=0).stop 2024-07-26T07:53:13.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.866+0000 7f0c85522740 1 -- 172.21.15.97:0/95097137 shutdown_connections 2024-07-26T07:53:13.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.866+0000 7f0c85522740 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493f7520 0x55eb497a1050 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.866+0000 7f0c85522740 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493d6840 0x55eb497a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.866+0000 7f0c85522740 1 --2- 172.21.15.97:0/95097137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.866+0000 7f0c85522740 1 -- 172.21.15.97:0/95097137 >> 172.21.15.97:0/95097137 conn(0x55eb493eb910 msgr2=0x55eb497a8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:13.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.867+0000 7f0c85522740 1 -- 172.21.15.97:0/95097137 shutdown_connections 2024-07-26T07:53:13.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.867+0000 7f0c85522740 1 -- 172.21.15.97:0/95097137 wait complete. 2024-07-26T07:53:13.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.867+0000 7f0c85522740 1 -- start start 2024-07-26T07:53:13.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.868+0000 7f0c85522740 1 Processor -- start 2024-07-26T07:53:13.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.868+0000 7f0c85522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.868+0000 7f0c85522740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497b9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.868+0000 7f0c85522740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb49820d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.868+0000 7f0c85522740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5400fa70 con 0x55eb493d6840 2024-07-26T07:53:13.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.868+0000 7f0c85522740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5400f3a0 con 0x55eb493f7520 2024-07-26T07:53:13.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.868+0000 7f0c85522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5400c5e0 con 0x55eb493ce660 2024-07-26T07:53:13.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.868+0000 7f0c85522740 10 client.0 _ll_register_callbacks cb 0x7f0c68004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:13.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497b9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497b9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46686/0 (socket says 172.21.15.97:46686) 2024-07-26T07:53:13.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c6fe00640 1 -- 172.21.15.97:0/1650755991 learned_addr learned my addr 172.21.15.97:0/1650755991 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:13.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c7cc00640 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb49820d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c77e00640 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c6fe00640 1 -- 172.21.15.97:0/1650755991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 msgr2=0x55eb497b8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:13.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c6fe00640 1 -- 172.21.15.97:0/1650755991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 msgr2=0x55eb49820d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:13.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb49820d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.869+0000 7f0c6fe00640 1 -- 172.21.15.97:0/1650755991 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb49821260 con 0x55eb493d6840 2024-07-26T07:53:13.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.870+0000 7f0c77e00640 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:13.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.870+0000 7f0c6fe00640 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497b9050 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009340 tx=0x7f0c6002ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8e102931590c3d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:13.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.870+0000 7f0c65600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:13.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.871+0000 7f0c65600640 1 -- 172.21.15.97:0/1650755991 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c60035040 con 0x55eb493d6840 2024-07-26T07:53:13.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.872+0000 7f0c65600640 1 -- 172.21.15.97:0/1650755991 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f0c600395f0 con 0x55eb493d6840 2024-07-26T07:53:13.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.872+0000 7f0c85522740 1 -- 172.21.15.97:0/1650755991 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c700b3ee0 con 0x55eb493d6840 2024-07-26T07:53:13.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.872+0000 7f0c65600640 1 -- 172.21.15.97:0/1650755991 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0c600046c0 con 0x55eb493d6840 2024-07-26T07:53:13.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.872+0000 7f0c85522740 10 client.15663 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:13.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.872+0000 7f0c85522740 1 -- 172.21.15.97:0/1650755991 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb498223f0 con 0x55eb493d6840 2024-07-26T07:53:13.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.873+0000 7f0c65600640 1 -- 172.21.15.97:0/1650755991 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c700b3ee0 con 0x55eb493d6840 2024-07-26T07:53:13.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.873+0000 7f0c65600640 1 -- 172.21.15.97:0/1650755991 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0c60059070 con 0x55eb493d6840 2024-07-26T07:53:13.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.873+0000 7f0c85522740 10 client.15663 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:13.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.873+0000 7f0c85522740 10 client.15663 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:53:13.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.873+0000 7f0c85522740 20 client.15663 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:13.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.873+0000 7f0c85522740 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55eb4938b250 0x55eb49824b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:13.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.873+0000 7f0c85522740 4 client.15663 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1506616e", "release": true, "await": true}] 2024-07-26T07:53:13.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.873+0000 7f0c85522740 1 -- 172.21.15.97:0/1650755991 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1506616e", "release": true, "await": true}) -- 0x55eb49710920 con 0x55eb4938b250 2024-07-26T07:53:13.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.874+0000 7f0c77e00640 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55eb4938b250 0x55eb49824b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:13.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.876+0000 7f0c77e00640 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55eb4938b250 0x55eb49824b80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60965c085c905da1 server_cookie=dfae6a39e8c9d8b2 in_seq=0 out_seq=0 2024-07-26T07:53:13.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.876+0000 7f0c65600640 10 client.15663 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:13.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.884+0000 7f0c65600640 1 -- 172.21.15.97:0/1650755991 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55eb49710920 con 0x55eb4938b250 2024-07-26T07:53:13.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.884+0000 7f0c65600640 10 client.15663 handle_command_reply: tid=0 2024-07-26T07:53:13.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.884+0000 7f0c85522740 1 client.15663 shutdown 2024-07-26T07:53:13.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.884+0000 7f0c85522740 1 -- 172.21.15.97:0/1650755991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 msgr2=0x55eb497b9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.884+0000 7f0c85522740 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497b9050 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009340 tx=0x7f0c6002ab80 comp rx=0 tx=0).stop 2024-07-26T07:53:13.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 1 -- 172.21.15.97:0/1650755991 shutdown_connections 2024-07-26T07:53:13.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55eb4938b250 0x55eb49824b80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55eb493f7520 0x55eb49820d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55eb493d6840 0x55eb497b9050 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 1 --2- 172.21.15.97:0/1650755991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb493ce660 0x55eb497b8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 1 -- 172.21.15.97:0/1650755991 >> 172.21.15.97:0/1650755991 conn(0x55eb493eb910 msgr2=0x55eb497a94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:13.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c65600640 0 client.15663 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:13.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 1 -- 172.21.15.97:0/1650755991 shutdown_connections 2024-07-26T07:53:13.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 1 -- 172.21.15.97:0/1650755991 wait complete. 2024-07-26T07:53:13.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.885+0000 7f0c85522740 20 client.15663 trim_cache size 0 max 16384 2024-07-26T07:53:13.889 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:53:13.889 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 28, 2024-07-26T07:53:13.890 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:53:13.890 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 4, 2024-07-26T07:53:13.890 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:53:13.890 INFO:teuthology.orchestra.run.smithi097.stdout: "1506616e": { 2024-07-26T07:53:13.890 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 4, 2024-07-26T07:53:13.891 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:53:13.891 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:53:13.891 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:53:13.891 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:53:13.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:53:13.891 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:53:13.892 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 142.6, 2024-07-26T07:53:13.892 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:53:13.892 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:53:13.892 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:53:13.892 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:53:13.893 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:53:13.893 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:53:13.893 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:13.893 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:13.893 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:13.893 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:13.894 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:13.894 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:53:13.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 -- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c5c0b3c50 msgr2=0x7f0c5c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c5c0b3c50 0x7f0c5c0b6160 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55eb4979ac40 tx=0x7f0c6800b390 comp rx=0 tx=0).stop 2024-07-26T07:53:13.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 -- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c7810fb00 msgr2=0x7f0c781a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:13.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c7810fb00 0x7f0c781a18d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000eaa0 tx=0x7f0c7000ef60 comp rx=0 tx=0).stop 2024-07-26T07:53:13.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 -- 172.21.15.97:0/3757624150 shutdown_connections 2024-07-26T07:53:13.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0c5c0b3c50 0x7f0c5c0b6160 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0c7810fb00 0x7f0c781a18d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0c78075b30 0x7f0c7819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 --2- 172.21.15.97:0/3757624150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c780751e0 0x7f0c7819c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:13.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 -- 172.21.15.97:0/3757624150 >> 172.21.15.97:0/3757624150 conn(0x7f0c780ff010 msgr2=0x7f0c781096a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:13.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 -- 172.21.15.97:0/3757624150 shutdown_connections 2024-07-26T07:53:13.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:13.886+0000 7f0c7ec00640 1 -- 172.21.15.97:0/3757624150 wait complete. 2024-07-26T07:53:14.027 INFO:tasks.workunit.client.0.smithi097.stderr:Updating files: 59% (7525/12697) Updating files: 60% (7619/12697) Updating files: 61% (7746/12697) Updating files: 62% (7873/12697) Updating files: 63% (8000/12697) Updating files: 64% (8127/12697) Updating files: 65% (8254/12697) Updating files: 66% (8381/12697) Updating files: 67% (8507/12697) Updating files: 68% (8634/12697) Updating files: 69% (8761/12697) Updating files: 70% (8888/12697) Updating files: 71% (9015/12697) Updating files: 72% (9142/12697) Updating files: 73% (9269/12697) Updating files: 74% (9396/12697) Updating files: 75% (9523/12697) Updating files: 76% (9650/12697) Updating files: 77% (9777/12697) Updating files: 78% (9904/12697) Updating files: 79% (10031/12697) Updating files: 80% (10158/12697) Updating files: 81% (10285/12697) Updating files: 82% (10412/12697) Updating files: 83% (10539/12697) Updating files: 84% (10666/12697) Updating files: 85% (10793/12697) Updating files: 86% (10920/12697) Updating files: 87% (11047/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 97% (12381/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-07-26T07:53:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:14 smithi097 ceph-mon[32231]: pgmap v553: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr:Note: switching to 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'. 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T07:53:14.362 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr: git switch -c 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr:Or undo this operation with: 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr: git switch - 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:14.363 INFO:tasks.workunit.client.0.smithi097.stderr:HEAD is now at e8f88bb1fe0 Merge PR #54435 into wip-vshankar-testing-20240725.121331-debug 2024-07-26T07:53:14.371 DEBUG:teuthology.orchestra.run.smithi097:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-26T07:53:14.458 INFO:tasks.workunit.client.0.smithi097.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-26T07:53:14.467 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T07:53:14.467 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-26T07:53:14.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:14 smithi115 ceph-mon[35364]: pgmap v553: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T07:53:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:14 smithi181 ceph-mon[35869]: pgmap v553: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T07:53:14.859 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1506616e', total seconds elapsed: 32.7 2024-07-26T07:53:14.859 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-07-26T07:53:14.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:53:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 -- 172.21.15.97:0/3702869068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744103f90 msgr2=0x7f77441043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 --2- 172.21.15.97:0/3702869068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744103f90 0x7f77441043a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7730009900 tx=0x7f773002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 -- 172.21.15.97:0/3702869068 shutdown_connections 2024-07-26T07:53:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 --2- 172.21.15.97:0/3702869068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7744104930 0x7f774410b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 --2- 172.21.15.97:0/3702869068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744103f90 0x7f77441043a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 --2- 172.21.15.97:0/3702869068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7744102f10 0x7f7744103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 -- 172.21.15.97:0/3702869068 >> 172.21.15.97:0/3702869068 conn(0x7f77440fe850 msgr2=0x7f7744100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 -- 172.21.15.97:0/3702869068 shutdown_connections 2024-07-26T07:53:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.968+0000 7f7743200640 1 -- 172.21.15.97:0/3702869068 wait complete. 2024-07-26T07:53:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.969+0000 7f7743200640 1 Processor -- start 2024-07-26T07:53:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.969+0000 7f7743200640 1 -- start start 2024-07-26T07:53:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.969+0000 7f7743200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744102f10 0x7f7744199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7743200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7744103f90 0x7f7744199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7743200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7744104930 0x7f774419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7743200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f774410ea40 con 0x7f7744102f10 2024-07-26T07:53:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7743200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f774410e8c0 con 0x7f7744104930 2024-07-26T07:53:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7743200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f774410ebc0 con 0x7f7744103f90 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744102f10 0x7f7744199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744102f10 0x7f7744199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46706/0 (socket says 172.21.15.97:46706) 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741e00640 1 -- 172.21.15.97:0/273417243 learned_addr learned my addr 172.21.15.97:0/273417243 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741400640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7744103f90 0x7f7744199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741e00640 1 -- 172.21.15.97:0/273417243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7744103f90 msgr2=0x7f7744199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741e00640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7744103f90 0x7f7744199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741e00640 1 -- 172.21.15.97:0/273417243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7744104930 msgr2=0x7f774419a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741e00640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7744104930 0x7f774419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.970+0000 7f7741e00640 1 -- 172.21.15.97:0/273417243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774419b0c0 con 0x7f7744102f10 2024-07-26T07:53:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.971+0000 7f7741e00640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744102f10 0x7f7744199710 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f773400d770 tx=0x7f773400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.971+0000 7f773aa00640 1 -- 172.21.15.97:0/273417243 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7734017070 con 0x7f7744102f10 2024-07-26T07:53:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.971+0000 7f773aa00640 1 -- 172.21.15.97:0/273417243 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77340054b0 con 0x7f7744102f10 2024-07-26T07:53:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.971+0000 7f7743200640 1 -- 172.21.15.97:0/273417243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774419b3b0 con 0x7f7744102f10 2024-07-26T07:53:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.971+0000 7f773aa00640 1 -- 172.21.15.97:0/273417243 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7734004c60 con 0x7f7744102f10 2024-07-26T07:53:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.971+0000 7f7743200640 1 -- 172.21.15.97:0/273417243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77441a1950 con 0x7f7744102f10 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.974+0000 7f774ffd6740 1 Processor -- start 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.974+0000 7f774ffd6740 1 -- start start 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f773aa00640 1 -- 172.21.15.97:0/273417243 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f7734004e00 con 0x7f7744102f10 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758621b0 0x564675c0c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c0cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c0d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f774ffd6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x564675b7a270 con 0x5646758414d0 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f774ffd6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564675b69770 con 0x5646758392f0 2024-07-26T07:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f774ffd6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564675b6d7b0 con 0x5646758621b0 2024-07-26T07:53:14.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f7741400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c0d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f7741400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c0d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46720/0 (socket says 172.21.15.97:46720) 2024-07-26T07:53:14.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f7741400640 1 -- 172.21.15.97:0/2499157305 learned_addr learned my addr 172.21.15.97:0/2499157305 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:14.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.975+0000 7f7742800640 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758621b0 0x564675c0c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f7741e00640 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c0cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f7741400640 1 -- 172.21.15.97:0/2499157305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758621b0 msgr2=0x564675c0c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f7741400640 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758621b0 0x564675c0c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f7741400640 1 -- 172.21.15.97:0/2499157305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 msgr2=0x564675c0cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f7741400640 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c0cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f7741400640 1 -- 172.21.15.97:0/2499157305 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646758cd790 con 0x5646758414d0 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f7741400640 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c0d690 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7730006010 tx=0x7f77300315d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d015a2b33c275d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f772f400640 1 -- 172.21.15.97:0/2499157305 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730035030 con 0x5646758414d0 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f772f400640 1 -- 172.21.15.97:0/2499157305 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77300044c0 con 0x5646758414d0 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.976+0000 7f772f400640 1 -- 172.21.15.97:0/2499157305 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730003ae0 con 0x5646758414d0 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.977+0000 7f774ffd6740 1 -- 172.21.15.97:0/2499157305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 msgr2=0x564675c0d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.977+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c0d690 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7730006010 tx=0x7f77300315d0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.978+0000 7f773aa00640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f77280b3bd0 0x7f77280b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.978+0000 7f774ffd6740 1 -- 172.21.15.97:0/2499157305 shutdown_connections 2024-07-26T07:53:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.978+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c0d690 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.978+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c0cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.978+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2499157305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758621b0 0x564675c0c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.978+0000 7f774ffd6740 1 -- 172.21.15.97:0/2499157305 >> 172.21.15.97:0/2499157305 conn(0x564675856300 msgr2=0x564675c03b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.978+0000 7f773aa00640 1 -- 172.21.15.97:0/273417243 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f77340e37b0 con 0x7f7744102f10 2024-07-26T07:53:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.979+0000 7f7742800640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f77280b3bd0 0x7f77280b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.979+0000 7f774ffd6740 1 -- 172.21.15.97:0/2499157305 shutdown_connections 2024-07-26T07:53:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.979+0000 7f774ffd6740 1 -- 172.21.15.97:0/2499157305 wait complete. 2024-07-26T07:53:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.979+0000 7f7742800640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f77280b3bd0 0x7f77280b60e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f773c005fd0 tx=0x7f773c007360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 1 -- start start 2024-07-26T07:53:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 1 Processor -- start 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758392f0 0x564675c15f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c16600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758621b0 0x564675c17170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x564675aa3ab0 con 0x5646758414d0 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x564675b06d50 con 0x5646758621b0 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564675c0e150 con 0x5646758392f0 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.980+0000 7f774ffd6740 10 client.0 _ll_register_callbacks cb 0x564675879a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c16600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c16600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46722/0 (socket says 172.21.15.97:46722) 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 -- 172.21.15.97:0/272653097 learned_addr learned my addr 172.21.15.97:0/272653097 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7742800640 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758621b0 0x564675c17170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741e00640 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758392f0 0x564675c15f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 -- 172.21.15.97:0/272653097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758392f0 msgr2=0x564675c15f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758392f0 0x564675c15f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 -- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758621b0 msgr2=0x564675c17170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758621b0 0x564675c17170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 -- 172.21.15.97:0/272653097 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564675946620 con 0x5646758414d0 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7742800640 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758621b0 0x564675c17170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.981+0000 7f7741400640 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c16600 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f773002a430 tx=0x7f7730004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f313585f29ed572 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.982+0000 7f7723e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.982+0000 7f7723e00640 1 -- 172.21.15.97:0/272653097 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773004d020 con 0x5646758414d0 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.982+0000 7f774ffd6740 1 -- 172.21.15.97:0/272653097 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564675b7c1c0 con 0x5646758414d0 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.982+0000 7f7723e00640 1 -- 172.21.15.97:0/272653097 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f7730003860 con 0x5646758414d0 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.982+0000 7f7723e00640 1 -- 172.21.15.97:0/272653097 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7730058050 con 0x5646758414d0 2024-07-26T07:53:14.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.982+0000 7f774ffd6740 10 client.15675 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.982+0000 7f774ffd6740 1 -- 172.21.15.97:0/272653097 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564675c200c0 con 0x5646758414d0 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.982+0000 7f7723e00640 1 -- 172.21.15.97:0/272653097 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x564675b7c1c0 con 0x5646758414d0 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.983+0000 7f7723e00640 1 -- 172.21.15.97:0/272653097 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f773003a040 con 0x5646758414d0 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.983+0000 7f774ffd6740 10 client.15675 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.983+0000 7f774ffd6740 10 client.15675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.983+0000 7f774ffd6740 20 client.15675 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.983+0000 7f774ffd6740 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564675820d80 0x564675c23420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.983+0000 7f774ffd6740 4 client.15675 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:53:14.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.983+0000 7f774ffd6740 1 -- 172.21.15.97:0/272653097 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564675b77c60 con 0x564675820d80 2024-07-26T07:53:14.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.984+0000 7f7741e00640 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564675820d80 0x564675c23420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:14.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.984+0000 7f7741e00640 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564675820d80 0x564675c23420 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a315b1873e393c1 server_cookie=cdbf86dcfb63766b in_seq=0 out_seq=0 2024-07-26T07:53:14.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.984+0000 7f7723e00640 10 client.15675 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:14.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.986+0000 7f7723e00640 1 -- 172.21.15.97:0/272653097 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564675b77c60 con 0x564675820d80 2024-07-26T07:53:14.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.986+0000 7f7723e00640 10 client.15675 handle_command_reply: tid=0 2024-07-26T07:53:14.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 client.15675 shutdown 2024-07-26T07:53:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 -- 172.21.15.97:0/272653097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 msgr2=0x564675c16600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c16600 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f773002a430 tx=0x7f7730004260 comp rx=0 tx=0).stop 2024-07-26T07:53:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 -- 172.21.15.97:0/272653097 shutdown_connections 2024-07-26T07:53:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564675820d80 0x564675c23420 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758621b0 0x564675c17170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758414d0 0x564675c16600 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f7723e00640 0 client.15675 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 --2- 172.21.15.97:0/272653097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758392f0 0x564675c15f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 -- 172.21.15.97:0/272653097 >> 172.21.15.97:0/272653097 conn(0x564675856300 msgr2=0x564675c020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 -- 172.21.15.97:0/272653097 shutdown_connections 2024-07-26T07:53:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 1 -- 172.21.15.97:0/272653097 wait complete. 2024-07-26T07:53:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:14.987+0000 7f774ffd6740 20 client.15675 trim_cache size 0 max 16384 2024-07-26T07:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.003+0000 7f774ffd6740 1 Processor -- start 2024-07-26T07:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.003+0000 7f774ffd6740 1 -- start start 2024-07-26T07:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.003+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c05080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.003+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 0x564675c1e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.003+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c14490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.003+0000 7f774ffd6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f77441a8620 con 0x5646758621b0 2024-07-26T07:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.003+0000 7f774ffd6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f77441aa160 con 0x5646758392f0 2024-07-26T07:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.003+0000 7f774ffd6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f77441a8cf0 con 0x5646758414d0 2024-07-26T07:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7742800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c14490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7742800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c14490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46730/0 (socket says 172.21.15.97:46730) 2024-07-26T07:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7742800640 1 -- 172.21.15.97:0/418818871 learned_addr learned my addr 172.21.15.97:0/418818871 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7741e00640 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c05080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7741400640 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 0x564675c1e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7742800640 1 -- 172.21.15.97:0/418818871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 msgr2=0x564675c1e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7742800640 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 0x564675c1e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7742800640 1 -- 172.21.15.97:0/418818871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 msgr2=0x564675c05080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7742800640 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c05080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7742800640 1 -- 172.21.15.97:0/418818871 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564675c14110 con 0x5646758621b0 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.004+0000 7f7741e00640 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c05080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.005+0000 7f7742800640 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c14490 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f773c010510 tx=0x7f773c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cad7a2e8fabbda8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.005+0000 7f772ea00640 1 -- 172.21.15.97:0/418818871 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c01d8f0 con 0x5646758621b0 2024-07-26T07:53:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.005+0000 7f772ea00640 1 -- 172.21.15.97:0/418818871 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f773c025400 con 0x5646758621b0 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.005+0000 7f772ea00640 1 -- 172.21.15.97:0/418818871 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c01c610 con 0x5646758621b0 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 -- 172.21.15.97:0/418818871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 msgr2=0x564675c14490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c14490 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f773c010510 tx=0x7f773c011000 comp rx=0 tx=0).stop 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 -- 172.21.15.97:0/418818871 shutdown_connections 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c14490 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 0x564675c1e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 --2- 172.21.15.97:0/418818871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c05080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 -- 172.21.15.97:0/418818871 >> 172.21.15.97:0/418818871 conn(0x564675856300 msgr2=0x564675c02860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 -- 172.21.15.97:0/418818871 shutdown_connections 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.006+0000 7f774ffd6740 1 -- 172.21.15.97:0/418818871 wait complete. 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 1 -- start start 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 1 Processor -- start 2024-07-26T07:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c1acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 0x564675c1b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c1bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f77441a9a90 con 0x5646758621b0 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f77441a93c0 con 0x5646758392f0 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f77441a6600 con 0x5646758414d0 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.007+0000 7f774ffd6740 10 client.0 _ll_register_callbacks cb 0x7f7730031500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c1bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c1bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46740/0 (socket says 172.21.15.97:46740) 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 -- 172.21.15.97:0/2449088870 learned_addr learned my addr 172.21.15.97:0/2449088870 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7741e00640 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c1acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 -- 172.21.15.97:0/2449088870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 msgr2=0x564675c1b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 0x564675c1b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 -- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 msgr2=0x564675c1acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c1acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 -- 172.21.15.97:0/2449088870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564675c1c680 con 0x5646758621b0 2024-07-26T07:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7741e00640 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c1acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7742800640 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c1bf20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f773c005ed0 tx=0x7f773c01d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f863f6fef71345 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.008+0000 7f7723e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.009+0000 7f7723e00640 1 -- 172.21.15.97:0/2449088870 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c036060 con 0x5646758621b0 2024-07-26T07:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.009+0000 7f774ffd6740 1 -- 172.21.15.97:0/2449088870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f773c025400 con 0x5646758621b0 2024-07-26T07:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.009+0000 7f7723e00640 1 -- 172.21.15.97:0/2449088870 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f773c01da90 con 0x5646758621b0 2024-07-26T07:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.009+0000 7f7723e00640 1 -- 172.21.15.97:0/2449088870 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f773c01dc90 con 0x5646758621b0 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.009+0000 7f774ffd6740 10 client.15681 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.009+0000 7f774ffd6740 1 -- 172.21.15.97:0/2449088870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564675c1d9a0 con 0x5646758621b0 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.009+0000 7f7723e00640 1 -- 172.21.15.97:0/2449088870 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f773c025400 con 0x5646758621b0 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.010+0000 7f7723e00640 1 -- 172.21.15.97:0/2449088870 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f773c03a030 con 0x5646758621b0 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.010+0000 7f774ffd6740 10 client.15681 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.010+0000 7f774ffd6740 10 client.15681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.010+0000 7f774ffd6740 20 client.15681 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.010+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5646757f61c0 0x564675c8db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.010+0000 7f774ffd6740 4 client.15681 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.010+0000 7f774ffd6740 1 -- 172.21.15.97:0/2449088870 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564675b753a0 con 0x5646757f61c0 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.010+0000 7f7741e00640 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5646757f61c0 0x564675c8db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.011+0000 7f7741e00640 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5646757f61c0 0x564675c8db90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d96c42af3f87e7f server_cookie=5cfe1062e3d4cb9a in_seq=0 out_seq=0 2024-07-26T07:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.011+0000 7f7723e00640 10 client.15681 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f7723e00640 1 -- 172.21.15.97:0/2449088870 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564675b753a0 con 0x5646757f61c0 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f7723e00640 10 client.15681 handle_command_reply: tid=0 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 client.15681 shutdown 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 -- 172.21.15.97:0/2449088870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 msgr2=0x564675c1bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c1bf20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f773c005ed0 tx=0x7f773c01d920 comp rx=0 tx=0).stop 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 -- 172.21.15.97:0/2449088870 shutdown_connections 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5646757f61c0 0x564675c8db90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5646758621b0 0x564675c1bf20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646758414d0 0x564675c1b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 --2- 172.21.15.97:0/2449088870 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5646758392f0 0x564675c1acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 -- 172.21.15.97:0/2449088870 >> 172.21.15.97:0/2449088870 conn(0x564675856300 msgr2=0x564675c0c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f7723e00640 0 client.15681 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.021+0000 7f774ffd6740 1 -- 172.21.15.97:0/2449088870 shutdown_connections 2024-07-26T07:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f774ffd6740 1 -- 172.21.15.97:0/2449088870 wait complete. 2024-07-26T07:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f774ffd6740 20 client.15681 trim_cache size 0 max 16384 2024-07-26T07:53:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:53:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:53:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:53:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 -- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f77280b3bd0 msgr2=0x7f77280b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f77280b3bd0 0x7f77280b60e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f773c005fd0 tx=0x7f773c007360 comp rx=0 tx=0).stop 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 -- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744102f10 msgr2=0x7f7744199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744102f10 0x7f7744199710 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f773400d770 tx=0x7f773400dc30 comp rx=0 tx=0).stop 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 -- 172.21.15.97:0/273417243 shutdown_connections 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f77280b3bd0 0x7f77280b60e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7744104930 0x7f774419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7744103f90 0x7f7744199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 --2- 172.21.15.97:0/273417243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7744102f10 0x7f7744199710 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.022+0000 7f7743200640 1 -- 172.21.15.97:0/273417243 >> 172.21.15.97:0/273417243 conn(0x7f77440fe850 msgr2=0x7f77441072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.023+0000 7f7743200640 1 -- 172.21.15.97:0/273417243 shutdown_connections 2024-07-26T07:53:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:15.023+0000 7f7743200640 1 -- 172.21.15.97:0/273417243 wait complete. 2024-07-26T07:53:15.315 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:53:15.689 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-26T07:53:15.791 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-26T07:53:15.869 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T07:53:15.875 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T07:53:15.875 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-26T07:53:15.956 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T07:53:15.964 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-07-26T07:53:15.964 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-26T07:53:16.003 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-07-26T07:53:16.005 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-07-26T07:53:16.005 DEBUG:teuthology.orchestra.run.smithi097:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-07-26T07:53:16.152 INFO:tasks.workunit.client.0.smithi097.stdout:getting blogbench 2024-07-26T07:53:16.152 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'getting blogbench' 2024-07-26T07:53:16.152 INFO:tasks.workunit.client.0.smithi097.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-07-26T07:53:16.254 INFO:tasks.workunit.client.0.smithi097.stderr:--2024-07-26 07:53:16-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-07-26T07:53:16.256 INFO:tasks.workunit.client.0.smithi097.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-07-26T07:53:16.277 INFO:tasks.workunit.client.0.smithi097.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-07-26T07:53:16.298 INFO:tasks.workunit.client.0.smithi097.stderr:HTTP request sent, awaiting response... 200 OK 2024-07-26T07:53:16.299 INFO:tasks.workunit.client.0.smithi097.stderr:Length: 103355 (101K) [application/octet-stream] 2024-07-26T07:53:16.308 INFO:tasks.workunit.client.0.smithi097.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-07-26T07:53:16.318 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:16.341 INFO:tasks.workunit.client.0.smithi097.stderr: 0K .......... .......... .......... .......... .......... 49% 1.52M 0s 2024-07-26T07:53:16.361 INFO:tasks.workunit.client.0.smithi097.stderr: 50K .......... .......... .......... .......... .......... 99% 2.51M 0s 2024-07-26T07:53:16.361 INFO:tasks.workunit.client.0.smithi097.stderr: 100K 100% 1.74T=0.05s 2024-07-26T07:53:16.362 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:16.362 INFO:tasks.workunit.client.0.smithi097.stderr:2024-07-26 07:53:16 (1.91 MB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-07-26T07:53:16.362 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-07-26T07:53:16.363 INFO:tasks.workunit.client.0.smithi097.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-07-26T07:53:16.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:16 smithi097 ceph-mon[32231]: pgmap v554: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T07:53:16.444 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/ 2024-07-26T07:53:16.449 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/m4/ 2024-07-26T07:53:16.455 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-07-26T07:53:16.462 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/man/ 2024-07-26T07:53:16.469 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/man/blogbench.8 2024-07-26T07:53:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:16 smithi115 ceph-mon[35364]: pgmap v554: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T07:53:16.478 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/man/Makefile.am 2024-07-26T07:53:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:16 smithi181 ceph-mon[35869]: pgmap v554: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T07:53:16.485 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/man/Makefile.in 2024-07-26T07:53:16.493 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/ 2024-07-26T07:53:16.500 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/mysnprintf.c 2024-07-26T07:53:16.510 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/mysnprintf.h 2024-07-26T07:53:16.518 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/blogbench.c 2024-07-26T07:53:16.528 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/blogbench.h 2024-07-26T07:53:16.535 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/fakesnprintf.c 2024-07-26T07:53:16.543 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/fakesnprintf.h 2024-07-26T07:53:16.549 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/blogbench_p.h 2024-07-26T07:53:16.557 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-07-26T07:53:16.569 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-07-26T07:53:16.581 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/Makefile.am 2024-07-26T07:53:16.588 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/Makefile.in 2024-07-26T07:53:16.598 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/process.c 2024-07-26T07:53:16.609 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/globals.h 2024-07-26T07:53:16.618 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/commenter.c 2024-07-26T07:53:16.627 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/rewriter.c 2024-07-26T07:53:16.633 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/reader.c 2024-07-26T07:53:16.640 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/writer.c 2024-07-26T07:53:16.648 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/src/helpers.c 2024-07-26T07:53:16.656 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/NEWS 2024-07-26T07:53:16.665 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/compile 2024-07-26T07:53:16.673 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/depcomp 2024-07-26T07:53:16.682 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/aclocal.m4 2024-07-26T07:53:16.693 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/README 2024-07-26T07:53:16.702 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/configure 2024-07-26T07:53:16.741 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/configure.ac 2024-07-26T07:53:16.749 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/config.guess 2024-07-26T07:53:16.759 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/install-sh 2024-07-26T07:53:16.764 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/config.sub 2024-07-26T07:53:16.772 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/missing 2024-07-26T07:53:16.779 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/mkinstalldirs 2024-07-26T07:53:16.785 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/Makefile.am 2024-07-26T07:53:16.791 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/Makefile.in 2024-07-26T07:53:16.798 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/config.h.in 2024-07-26T07:53:16.805 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/AUTHORS 2024-07-26T07:53:16.812 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/INSTALL 2024-07-26T07:53:16.822 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/ChangeLog 2024-07-26T07:53:16.830 INFO:tasks.workunit.client.0.smithi097.stdout:blogbench-1.0/COPYING 2024-07-26T07:53:16.840 INFO:tasks.workunit.client.0.smithi097.stderr:+ cd blogbench-1.0/ 2024-07-26T07:53:16.846 INFO:tasks.workunit.client.0.smithi097.stdout:making blogbench 2024-07-26T07:53:16.847 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'making blogbench' 2024-07-26T07:53:16.847 INFO:tasks.workunit.client.0.smithi097.stderr:+ ./configure 2024-07-26T07:53:17.227 INFO:tasks.workunit.client.0.smithi097.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-07-26T07:53:17.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 -- 172.21.15.97:0/2309714931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 msgr2=0x7f4b2c116180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:17.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 --2- 172.21.15.97:0/2309714931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 0x7f4b2c116180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009900 tx=0x7f4b1c02a410 comp rx=0 tx=0).stop 2024-07-26T07:53:17.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 -- 172.21.15.97:0/2309714931 shutdown_connections 2024-07-26T07:53:17.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 --2- 172.21.15.97:0/2309714931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 0x7f4b2c116180 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 --2- 172.21.15.97:0/2309714931 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b2c075800 0x7f4b2c075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 --2- 172.21.15.97:0/2309714931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b2c076db0 0x7f4b2c0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 -- 172.21.15.97:0/2309714931 >> 172.21.15.97:0/2309714931 conn(0x7f4b2c0fee70 msgr2=0x7f4b2c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:17.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 -- 172.21.15.97:0/2309714931 shutdown_connections 2024-07-26T07:53:17.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.584+0000 7f4b30c00640 1 -- 172.21.15.97:0/2309714931 wait complete. 2024-07-26T07:53:17.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.585+0000 7f4b30c00640 1 Processor -- start 2024-07-26T07:53:17.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b30c00640 1 -- start start 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b30c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b2c075800 0x7f4b2c1a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b30c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b2c076db0 0x7f4b2c1a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b30c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 0x7f4b2c1a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b30c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2c1197a0 con 0x7f4b2c113cb0 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b30c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2c119620 con 0x7f4b2c076db0 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b30c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2c119920 con 0x7f4b2c075800 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b2be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 0x7f4b2c1a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b2be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 0x7f4b2c1a5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52124/0 (socket says 172.21.15.97:52124) 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.586+0000 7f4b2be00640 1 -- 172.21.15.97:0/178889705 learned_addr learned my addr 172.21.15.97:0/178889705 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.587+0000 7f4b2b400640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b2c075800 0x7f4b2c1a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:17.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.587+0000 7f4b2aa00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b2c076db0 0x7f4b2c1a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:17.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.587+0000 7f4b2be00640 1 -- 172.21.15.97:0/178889705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b2c075800 msgr2=0x7f4b2c1a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:17.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.587+0000 7f4b2be00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b2c075800 0x7f4b2c1a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.587+0000 7f4b2be00640 1 -- 172.21.15.97:0/178889705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b2c076db0 msgr2=0x7f4b2c1a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.587+0000 7f4b2be00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b2c076db0 0x7f4b2c1a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.587+0000 7f4b2be00640 1 -- 172.21.15.97:0/178889705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2c1a6140 con 0x7f4b2c113cb0 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.587+0000 7f4b2be00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 0x7f4b2c1a5a10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c005ab0 tx=0x7f4b1c0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.588+0000 7f4b23e00640 1 -- 172.21.15.97:0/178889705 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c03f030 con 0x7f4b2c113cb0 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.588+0000 7f4b23e00640 1 -- 172.21.15.97:0/178889705 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4b1c003a80 con 0x7f4b2c113cb0 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.588+0000 7f4b23e00640 1 -- 172.21.15.97:0/178889705 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c035070 con 0x7f4b2c113cb0 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.588+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b2c1a63d0 con 0x7f4b2c113cb0 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.588+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b2c1acb90 con 0x7f4b2c113cb0 2024-07-26T07:53:17.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.589+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b2c075c10 con 0x7f4b2c113cb0 2024-07-26T07:53:17.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.591+0000 7f4b23e00640 1 -- 172.21.15.97:0/178889705 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f4b1c042030 con 0x7f4b2c113cb0 2024-07-26T07:53:17.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.594+0000 7f4b23e00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4b100b3c50 0x7f4b100b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:17.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.594+0000 7f4b23e00640 1 -- 172.21.15.97:0/178889705 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f4b1c104620 con 0x7f4b2c113cb0 2024-07-26T07:53:17.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.594+0000 7f4b2b400640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4b100b3c50 0x7f4b100b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:17.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.595+0000 7f4b2b400640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4b100b3c50 0x7f4b100b6160 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4b24005fd0 tx=0x7f4b24009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:17.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.596+0000 7f4b23e00640 1 -- 172.21.15.97:0/178889705 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4b1c0cbbb0 con 0x7f4b2c113cb0 2024-07-26T07:53:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.834+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b2c1129e0 con 0x7f4b2c113cb0 2024-07-26T07:53:17.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.836+0000 7f4b23e00640 1 -- 172.21.15.97:0/178889705 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11990 (secure 0 0 0) 0x7f4b1c0cb4b0 con 0x7f4b2c113cb0 2024-07-26T07:53:17.838 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:53:17.838 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"btime":"2024-07-26T07:51:41:000369+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:51:40.647454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:53:17.838 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.839+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4b100b3c50 msgr2=0x7f4b100b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.839+0000 7f4b30c00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4b100b3c50 0x7f4b100b6160 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4b24005fd0 tx=0x7f4b24009870 comp rx=0 tx=0).stop 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.839+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 msgr2=0x7f4b2c1a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.839+0000 7f4b30c00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 0x7f4b2c1a5a10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c005ab0 tx=0x7f4b1c0040e0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.840+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 shutdown_connections 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.840+0000 7f4b30c00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4b100b3c50 0x7f4b100b6160 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.840+0000 7f4b30c00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4b2c113cb0 0x7f4b2c1a5a10 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.840+0000 7f4b30c00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b2c076db0 0x7f4b2c1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.840+0000 7f4b30c00640 1 --2- 172.21.15.97:0/178889705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b2c075800 0x7f4b2c1a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.840+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 >> 172.21.15.97:0/178889705 conn(0x7f4b2c0fee70 msgr2=0x7f4b2c10fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.840+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 shutdown_connections 2024-07-26T07:53:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:17.840+0000 7f4b30c00640 1 -- 172.21.15.97:0/178889705 wait complete. 2024-07-26T07:53:18.238 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:18 smithi097 ceph-mon[32231]: pgmap v555: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:53:18.239 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:18 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/178889705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:18.263 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether build environment is sane... yes 2024-07-26T07:53:18.310 INFO:tasks.workunit.client.0.smithi097.stdout:checking for gawk... gawk 2024-07-26T07:53:18.385 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether make sets $(MAKE)... yes 2024-07-26T07:53:18.458 INFO:tasks.workunit.client.0.smithi097.stdout:checking for gcc... gcc 2024-07-26T07:53:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:18 smithi115 ceph-mon[35364]: pgmap v555: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:53:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:18 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/178889705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:18 smithi181 ceph-mon[35869]: pgmap v555: 97 pgs: 97 active+clean; 682 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T07:53:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:18 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/178889705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:18.718 INFO:tasks.workunit.client.0.smithi097.stdout:checking for C compiler default output file name... a.out 2024-07-26T07:53:18.719 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0' 2024-07-26T07:53:18.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0 2024-07-26T07:53:18.731 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether the C compiler works... yes 2024-07-26T07:53:18.740 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether we are cross compiling... no 2024-07-26T07:53:18.881 INFO:tasks.workunit.client.0.smithi097.stdout:checking for suffix of executables... 2024-07-26T07:53:19.022 INFO:tasks.workunit.client.0.smithi097.stdout:checking for suffix of object files... o 2024-07-26T07:53:19.146 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether we are using the GNU C compiler... yes 2024-07-26T07:53:19.184 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:53:19.272 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether gcc accepts -g... yes 2024-07-26T07:53:19.582 INFO:tasks.workunit.client.0.smithi097.stdout:checking for gcc option to accept ANSI C... none needed 2024-07-26T07:53:19.795 INFO:tasks.workunit.client.0.smithi097.stdout:checking for style of include used by make... GNU 2024-07-26T07:53:20.362 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:20 smithi097 ceph-mon[32231]: pgmap v556: 97 pgs: 97 active+clean; 816 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-07-26T07:53:20.390 INFO:tasks.workunit.client.0.smithi097.stdout:checking dependency style of gcc... gcc3 2024-07-26T07:53:20.392 INFO:tasks.workunit.client.0.smithi097.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-07-26T07:53:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:20 smithi115 ceph-mon[35364]: pgmap v556: 97 pgs: 97 active+clean; 816 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-07-26T07:53:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:20 smithi181 ceph-mon[35869]: pgmap v556: 97 pgs: 97 active+clean; 816 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-07-26T07:53:20.760 INFO:tasks.workunit.client.0.smithi097.stdout:checking how to run the C preprocessor... gcc -E 2024-07-26T07:53:20.934 INFO:tasks.workunit.client.0.smithi097.stdout:checking for egrep... grep -E 2024-07-26T07:53:20.968 INFO:tasks.workunit.client.0.smithi097.stdout:checking for AIX... no 2024-07-26T07:53:21.256 INFO:tasks.workunit.client.0.smithi097.stdout:checking for library containing strerror... none required 2024-07-26T07:53:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 -- 172.21.15.97:0/572973864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c109680 msgr2=0x7f9b1c10feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 --2- 172.21.15.97:0/572973864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c109680 0x7f9b1c10feb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009ca0 tx=0x7f9b1402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 -- 172.21.15.97:0/572973864 shutdown_connections 2024-07-26T07:53:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 --2- 172.21.15.97:0/572973864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c109680 0x7f9b1c10feb0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 --2- 172.21.15.97:0/572973864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b1c108d30 0x7f9b1c109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 --2- 172.21.15.97:0/572973864 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b1c1033f0 0x7f9b1c103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 -- 172.21.15.97:0/572973864 >> 172.21.15.97:0/572973864 conn(0x7f9b1c0fee60 msgr2=0x7f9b1c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:21.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 -- 172.21.15.97:0/572973864 shutdown_connections 2024-07-26T07:53:21.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.500+0000 7f9b21800640 1 -- 172.21.15.97:0/572973864 wait complete. 2024-07-26T07:53:21.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.502+0000 7f9b21800640 1 Processor -- start 2024-07-26T07:53:21.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.502+0000 7f9b21800640 1 -- start start 2024-07-26T07:53:21.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b21800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c1033f0 0x7f9b1c19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b21800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b1c108d30 0x7f9b1c19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b21800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b1c109680 0x7f9b1c1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b21800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c113250 con 0x7f9b1c1033f0 2024-07-26T07:53:21.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b21800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c1130d0 con 0x7f9b1c108d30 2024-07-26T07:53:21.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b21800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c1133d0 con 0x7f9b1c109680 2024-07-26T07:53:21.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b1be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c1033f0 0x7f9b1c19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b1be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c1033f0 0x7f9b1c19be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52140/0 (socket says 172.21.15.97:52140) 2024-07-26T07:53:21.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b1be00640 1 -- 172.21.15.97:0/3448520528 learned_addr learned my addr 172.21.15.97:0/3448520528 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:21.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b1b400640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b1c108d30 0x7f9b1c19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.503+0000 7f9b20e00640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b1c109680 0x7f9b1c1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.504+0000 7f9b1be00640 1 -- 172.21.15.97:0/3448520528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b1c109680 msgr2=0x7f9b1c1a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.504+0000 7f9b1be00640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b1c109680 0x7f9b1c1a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.504+0000 7f9b1be00640 1 -- 172.21.15.97:0/3448520528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b1c108d30 msgr2=0x7f9b1c19c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.504+0000 7f9b1be00640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b1c108d30 0x7f9b1c19c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.504+0000 7f9b1be00640 1 -- 172.21.15.97:0/3448520528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1c1a1db0 con 0x7f9b1c1033f0 2024-07-26T07:53:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.504+0000 7f9b1b400640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b1c108d30 0x7f9b1c19c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:53:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.504+0000 7f9b1be00640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c1033f0 0x7f9b1c19be40 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00d950 tx=0x7f9b0c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:21.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.505+0000 7f9b18c00640 1 -- 172.21.15.97:0/3448520528 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c017070 con 0x7f9b1c1033f0 2024-07-26T07:53:21.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.505+0000 7f9b18c00640 1 -- 172.21.15.97:0/3448520528 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b0c009c30 con 0x7f9b1c1033f0 2024-07-26T07:53:21.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.505+0000 7f9b21800640 1 -- 172.21.15.97:0/3448520528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b1c1a20a0 con 0x7f9b1c1033f0 2024-07-26T07:53:21.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.505+0000 7f9b18c00640 1 -- 172.21.15.97:0/3448520528 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c00f620 con 0x7f9b1c1033f0 2024-07-26T07:53:21.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.505+0000 7f9b21800640 1 -- 172.21.15.97:0/3448520528 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b1c1a8800 con 0x7f9b1c1033f0 2024-07-26T07:53:21.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.509+0000 7f9b29a5c740 1 Processor -- start 2024-07-26T07:53:21.510 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.509+0000 7f9b29a5c740 1 -- start start 2024-07-26T07:53:21.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b18c00640 1 -- 172.21.15.97:0/3448520528 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f9b0c00f7c0 con 0x7f9b1c1033f0 2024-07-26T07:53:21.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668eef060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668eef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668eefae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560668e63970 con 0x560668b48850 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560668e52e70 con 0x560668b27b70 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560668e56eb0 con 0x560668b1f990 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b20e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668eef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.510+0000 7f9b20e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668eef060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52144/0 (socket says 172.21.15.97:52144) 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.511+0000 7f9b20e00640 1 -- 172.21.15.97:0/4240038142 learned_addr learned my addr 172.21.15.97:0/4240038142 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.511+0000 7f9b1be00640 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668eef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.511+0000 7f9b1b400640 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668eefae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.512+0000 7f9b20e00640 1 -- 172.21.15.97:0/4240038142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 msgr2=0x560668eef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.512+0000 7f9b20e00640 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668eef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.512+0000 7f9b20e00640 1 -- 172.21.15.97:0/4240038142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 msgr2=0x560668eefae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.512+0000 7f9b20e00640 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668eefae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.512+0000 7f9b20e00640 1 -- 172.21.15.97:0/4240038142 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560668ddb8e0 con 0x560668b48850 2024-07-26T07:53:21.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.513+0000 7f9b20e00640 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668eef060 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9b14005cc0 tx=0x7f9b14009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa114ff204ecad97 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:21.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.513+0000 7f9b11600640 1 -- 172.21.15.97:0/4240038142 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14031980 con 0x560668b48850 2024-07-26T07:53:21.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.513+0000 7f9b11600640 1 -- 172.21.15.97:0/4240038142 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b14036040 con 0x560668b48850 2024-07-26T07:53:21.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.513+0000 7f9b11600640 1 -- 172.21.15.97:0/4240038142 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14004310 con 0x560668b48850 2024-07-26T07:53:21.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.513+0000 7f9b18c00640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9b000b3d10 0x7f9b000b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.514+0000 7f9b18c00640 1 -- 172.21.15.97:0/3448520528 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f9b0c0e38e0 con 0x7f9b1c1033f0 2024-07-26T07:53:21.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.514+0000 7f9b29a5c740 1 -- 172.21.15.97:0/4240038142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 msgr2=0x560668eef060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:21.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.514+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668eef060 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9b14005cc0 tx=0x7f9b14009a20 comp rx=0 tx=0).stop 2024-07-26T07:53:21.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.515+0000 7f9b29a5c740 1 -- 172.21.15.97:0/4240038142 shutdown_connections 2024-07-26T07:53:21.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.515+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668eefae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.515+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668eef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.515+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/4240038142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668eef060 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.515+0000 7f9b29a5c740 1 -- 172.21.15.97:0/4240038142 >> 172.21.15.97:0/4240038142 conn(0x560668b3cc40 msgr2=0x560668eecf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:21.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.515+0000 7f9b1be00640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9b000b3d10 0x7f9b000b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.515+0000 7f9b29a5c740 1 -- 172.21.15.97:0/4240038142 shutdown_connections 2024-07-26T07:53:21.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.515+0000 7f9b29a5c740 1 -- 172.21.15.97:0/4240038142 wait complete. 2024-07-26T07:53:21.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.516+0000 7f9b1be00640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9b000b3d10 0x7f9b000b6220 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x560668eedce0 tx=0x7f9b0c005880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:21.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.516+0000 7f9b29a5c740 1 -- start start 2024-07-26T07:53:21.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.516+0000 7f9b29a5c740 1 Processor -- start 2024-07-26T07:53:21.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668efabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668efb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668f0f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560668cb67b0 con 0x560668b48850 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560668def420 con 0x560668b27b70 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560668efd7e0 con 0x560668b1f990 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b29a5c740 10 client.0 _ll_register_callbacks cb 0x560668b600a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b20e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668f0f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b20e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668f0f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52146/0 (socket says 172.21.15.97:52146) 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.517+0000 7f9b20e00640 1 -- 172.21.15.97:0/773732925 learned_addr learned my addr 172.21.15.97:0/773732925 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.518+0000 7f9b20e00640 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668efabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.518+0000 7f9b1b400640 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668efb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.518+0000 7f9b20e00640 1 -- 172.21.15.97:0/773732925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 msgr2=0x560668efabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.518+0000 7f9b20e00640 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668efabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.518+0000 7f9b20e00640 1 -- 172.21.15.97:0/773732925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 msgr2=0x560668efb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.518+0000 7f9b20e00640 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668efb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.518+0000 7f9b20e00640 1 -- 172.21.15.97:0/773732925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560668de7c50 con 0x560668b48850 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.518+0000 7f9b20e00640 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668f0f2e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x560668efd230 tx=0x7f9b1403e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec705f7dd9654f8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.519+0000 7f9b06000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.519+0000 7f9b06000640 1 -- 172.21.15.97:0/773732925 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14059060 con 0x560668b48850 2024-07-26T07:53:21.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.519+0000 7f9b29a5c740 1 -- 172.21.15.97:0/773732925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b14036040 con 0x560668b48850 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.519+0000 7f9b06000640 1 -- 172.21.15.97:0/773732925 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f9b14003970 con 0x560668b48850 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.519+0000 7f9b06000640 1 -- 172.21.15.97:0/773732925 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b14004340 con 0x560668b48850 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.519+0000 7f9b29a5c740 10 client.15699 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.519+0000 7f9b29a5c740 1 -- 172.21.15.97:0/773732925 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560668bc2ba0 con 0x560668b48850 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.520+0000 7f9b06000640 1 -- 172.21.15.97:0/773732925 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b14036040 con 0x560668b48850 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.520+0000 7f9b06000640 1 -- 172.21.15.97:0/773732925 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9b140305c0 con 0x560668b48850 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.520+0000 7f9b29a5c740 10 client.15699 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.520+0000 7f9b29a5c740 10 client.15699 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.520+0000 7f9b29a5c740 20 client.15699 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.520+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560668b071d0 0x560668f15160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.520+0000 7f9b29a5c740 4 client.15699 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:53:21.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.520+0000 7f9b29a5c740 1 -- 172.21.15.97:0/773732925 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560668e658c0 con 0x560668b071d0 2024-07-26T07:53:21.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.521+0000 7f9b1b400640 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560668b071d0 0x560668f15160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.522+0000 7f9b1b400640 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560668b071d0 0x560668f15160 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff69541eba820616 server_cookie=e4d6beb089ab2d63 in_seq=0 out_seq=0 2024-07-26T07:53:21.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.522+0000 7f9b06000640 10 client.15699 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:21.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b06000640 1 -- 172.21.15.97:0/773732925 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560668e658c0 con 0x560668b071d0 2024-07-26T07:53:21.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b06000640 10 client.15699 handle_command_reply: tid=0 2024-07-26T07:53:21.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 client.15699 shutdown 2024-07-26T07:53:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 -- 172.21.15.97:0/773732925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 msgr2=0x560668f0f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668f0f2e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x560668efd230 tx=0x7f9b1403e040 comp rx=0 tx=0).stop 2024-07-26T07:53:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 -- 172.21.15.97:0/773732925 shutdown_connections 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560668b071d0 0x560668f15160 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b48850 0x560668f0f2e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b06000640 0 client.15699 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668efb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/773732925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668efabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 -- 172.21.15.97:0/773732925 >> 172.21.15.97:0/773732925 conn(0x560668b3cc40 msgr2=0x560668ef2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.524+0000 7f9b29a5c740 1 -- 172.21.15.97:0/773732925 shutdown_connections 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.525+0000 7f9b29a5c740 1 -- 172.21.15.97:0/773732925 wait complete. 2024-07-26T07:53:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.525+0000 7f9b29a5c740 20 client.15699 trim_cache size 0 max 16384 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b29a5c740 1 Processor -- start 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b29a5c740 1 -- start start 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b1f990 0x560668ef0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668eeb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b48850 0x560668f0f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c1acb70 con 0x560668b1f990 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c1ae6b0 con 0x560668b27b70 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c1ad240 con 0x560668b48850 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.544+0000 7f9b20e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b1f990 0x560668ef0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b20e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b1f990 0x560668ef0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52154/0 (socket says 172.21.15.97:52154) 2024-07-26T07:53:21.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b20e00640 1 -- 172.21.15.97:0/2896494614 learned_addr learned my addr 172.21.15.97:0/2896494614 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b1b400640 1 --2- 172.21.15.97:0/2896494614 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668eeb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b20e00640 1 -- 172.21.15.97:0/2896494614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b48850 msgr2=0x560668f0f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b20e00640 1 --2- 172.21.15.97:0/2896494614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b48850 0x560668f0f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b20e00640 1 -- 172.21.15.97:0/2896494614 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 msgr2=0x560668eeb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b20e00640 1 --2- 172.21.15.97:0/2896494614 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668eeb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b20e00640 1 -- 172.21.15.97:0/2896494614 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560668eed8c0 con 0x560668b1f990 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.545+0000 7f9b20e00640 1 --2- 172.21.15.97:0/2896494614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b1f990 0x560668ef0a50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9b1402ade0 tx=0x7f9b14009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc635d950d9fbe9f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.546+0000 7f9b10c00640 1 -- 172.21.15.97:0/2896494614 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14059070 con 0x560668b1f990 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.546+0000 7f9b10c00640 1 -- 172.21.15.97:0/2896494614 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b14036070 con 0x560668b1f990 2024-07-26T07:53:21.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.546+0000 7f9b10c00640 1 -- 172.21.15.97:0/2896494614 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14003f20 con 0x560668b1f990 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2896494614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b1f990 msgr2=0x560668ef0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2896494614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b1f990 0x560668ef0a50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9b1402ade0 tx=0x7f9b14009ba0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2896494614 shutdown_connections 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2896494614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b48850 0x560668f0f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2896494614 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b27b70 0x560668eeb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2896494614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b1f990 0x560668ef0a50 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2896494614 >> 172.21.15.97:0/2896494614 conn(0x560668b3cc40 msgr2=0x560668efbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2896494614 shutdown_connections 2024-07-26T07:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.547+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2896494614 wait complete. 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.548+0000 7f9b29a5c740 1 -- start start 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.548+0000 7f9b29a5c740 1 Processor -- start 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668f189c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b27b70 0x560668f15e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b29a5c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b48850 0x560668f16530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c1adfe0 con 0x560668b27b70 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c1ad910 con 0x560668b48850 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b29a5c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c1aabd0 con 0x560668b1f990 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b29a5c740 10 client.0 _ll_register_callbacks cb 0x7f9b0800b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b1b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b27b70 0x560668f15e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b1b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b27b70 0x560668f15e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52164/0 (socket says 172.21.15.97:52164) 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b1b400640 1 -- 172.21.15.97:0/2201658467 learned_addr learned my addr 172.21.15.97:0/2201658467 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b20e00640 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668f189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b20e00640 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b48850 0x560668f16530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b1b400640 1 -- 172.21.15.97:0/2201658467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 msgr2=0x560668f189c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.549+0000 7f9b1b400640 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668f189c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b1b400640 1 -- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b48850 msgr2=0x560668f16530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b1b400640 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b48850 0x560668f16530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b1b400640 1 -- 172.21.15.97:0/2201658467 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560668f16cc0 con 0x560668b27b70 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b20e00640 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b48850 0x560668f16530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b1b400640 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b27b70 0x560668f15e90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c105430 tx=0x7f9b08011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab88967c7a96421 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:21.552 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b06000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b06000640 1 -- 172.21.15.97:0/2201658467 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b08022020 con 0x560668b27b70 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b06000640 1 -- 172.21.15.97:0/2201658467 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f9b0801d8f0 con 0x560668b27b70 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.550+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2201658467 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b14036070 con 0x560668b27b70 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b06000640 1 -- 172.21.15.97:0/2201658467 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b0802d050 con 0x560668b27b70 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b29a5c740 10 client.15705 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2201658467 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560668f7d7a0 con 0x560668b27b70 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b06000640 1 -- 172.21.15.97:0/2201658467 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b14036070 con 0x560668b27b70 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b06000640 1 -- 172.21.15.97:0/2201658467 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9b0801c3a0 con 0x560668b27b70 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b29a5c740 10 client.15705 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b29a5c740 10 client.15705 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:53:21.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b29a5c740 20 client.15705 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:21.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560668adc580 0x560668f80050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:21.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b29a5c740 4 client.15705 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}] 2024-07-26T07:53:21.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.551+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2201658467 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}) -- 0x560668e61360 con 0x560668adc580 2024-07-26T07:53:21.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.552+0000 7f9b20e00640 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560668adc580 0x560668f80050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:21.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.553+0000 7f9b06000640 10 client.15705 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:21.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:21.553+0000 7f9b20e00640 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560668adc580 0x560668f80050 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a4301eca15b9492 server_cookie=4a21ea03c88ec226 in_seq=0 out_seq=0 2024-07-26T07:53:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:22 smithi115 ceph-mon[35364]: pgmap v557: 97 pgs: 97 active+clean; 1.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 KiB/s wr, 17 op/s 2024-07-26T07:53:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:22 smithi181 ceph-mon[35869]: pgmap v557: 97 pgs: 97 active+clean; 1.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 KiB/s wr, 17 op/s 2024-07-26T07:53:22.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:22 smithi097 ceph-mon[32231]: pgmap v557: 97 pgs: 97 active+clean; 1.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 KiB/s wr, 17 op/s 2024-07-26T07:53:23.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.366+0000 7f9b06000640 1 -- 172.21.15.97:0/2201658467 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x560668e61360 con 0x560668adc580 2024-07-26T07:53:23.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.366+0000 7f9b06000640 10 client.15705 handle_command_reply: tid=0 2024-07-26T07:53:23.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.366+0000 7f9b29a5c740 1 client.15705 shutdown 2024-07-26T07:53:23.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2201658467 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b27b70 msgr2=0x560668f15e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:23.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b27b70 0x560668f15e90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c105430 tx=0x7f9b08011a00 comp rx=0 tx=0).stop 2024-07-26T07:53:23.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2201658467 shutdown_connections 2024-07-26T07:53:23.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560668adc580 0x560668f80050 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:23.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560668b48850 0x560668f16530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:23.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560668b27b70 0x560668f15e90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:23.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 --2- 172.21.15.97:0/2201658467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560668b1f990 0x560668f189c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:23.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2201658467 >> 172.21.15.97:0/2201658467 conn(0x560668b3cc40 msgr2=0x560668ef0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:23.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b06000640 0 client.15705 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:23.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.367+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2201658467 shutdown_connections 2024-07-26T07:53:23.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.368+0000 7f9b29a5c740 1 -- 172.21.15.97:0/2201658467 wait complete. 2024-07-26T07:53:23.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.368+0000 7f9b29a5c740 20 client.15705 trim_cache size 0 max 16384 2024-07-26T07:53:23.370 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:53:23.370 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 28, 2024-07-26T07:53:23.370 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:53:23.370 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 6, 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "9b1e79da": { 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 6, 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 142.6, 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:53:23.371 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:53:23.372 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:53:23.372 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:23.372 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:23.372 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:23.372 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:23.372 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:23.372 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:53:23.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.369+0000 7f9b21800640 1 -- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9b000b3d10 msgr2=0x7f9b000b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.369+0000 7f9b21800640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9b000b3d10 0x7f9b000b6220 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x560668eedce0 tx=0x7f9b0c005880 comp rx=0 tx=0).stop 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.369+0000 7f9b21800640 1 -- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c1033f0 msgr2=0x7f9b1c19be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.369+0000 7f9b21800640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c1033f0 0x7f9b1c19be40 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00d950 tx=0x7f9b0c00de10 comp rx=0 tx=0).stop 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.370+0000 7f9b21800640 1 -- 172.21.15.97:0/3448520528 shutdown_connections 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.370+0000 7f9b21800640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f9b000b3d10 0x7f9b000b6220 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.370+0000 7f9b21800640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b1c109680 0x7f9b1c1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.370+0000 7f9b21800640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9b1c108d30 0x7f9b1c19c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.370+0000 7f9b21800640 1 --2- 172.21.15.97:0/3448520528 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b1c1033f0 0x7f9b1c19be40 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.370+0000 7f9b21800640 1 -- 172.21.15.97:0/3448520528 >> 172.21.15.97:0/3448520528 conn(0x7f9b1c0fee60 msgr2=0x7f9b1c108370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.370+0000 7f9b21800640 1 -- 172.21.15.97:0/3448520528 shutdown_connections 2024-07-26T07:53:23.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:23.370+0000 7f9b21800640 1 -- 172.21.15.97:0/3448520528 wait complete. 2024-07-26T07:53:24.180 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9b1e79da', quiesce took 9.3 seconds. Will release after: 13.25 2024-07-26T07:53:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:24 smithi115 ceph-mon[35364]: pgmap v558: 97 pgs: 97 active+clean; 1.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 75 KiB/s wr, 17 op/s 2024-07-26T07:53:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:24 smithi181 ceph-mon[35869]: pgmap v558: 97 pgs: 97 active+clean; 1.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 75 KiB/s wr, 17 op/s 2024-07-26T07:53:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:24 smithi097 ceph-mon[32231]: pgmap v558: 97 pgs: 97 active+clean; 1.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 75 KiB/s wr, 17 op/s 2024-07-26T07:53:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:26 smithi115 ceph-mon[35364]: pgmap v559: 97 pgs: 97 active+clean; 2.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 29 op/s 2024-07-26T07:53:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:26 smithi181 ceph-mon[35869]: pgmap v559: 97 pgs: 97 active+clean; 2.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 29 op/s 2024-07-26T07:53:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:26 smithi097 ceph-mon[32231]: pgmap v559: 97 pgs: 97 active+clean; 2.1 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 29 op/s 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 -- 172.21.15.97:0/473115845 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634106b40 msgr2=0x7fd63410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 --2- 172.21.15.97:0/473115845 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634106b40 0x7fd63410d4e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009ca0 tx=0x7fd62c02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 -- 172.21.15.97:0/473115845 shutdown_connections 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 --2- 172.21.15.97:0/473115845 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634106b40 0x7fd63410d4e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 --2- 172.21.15.97:0/473115845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6341061a0 0x7fd6341065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 --2- 172.21.15.97:0/473115845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd634105120 0x7fd634105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 -- 172.21.15.97:0/473115845 >> 172.21.15.97:0/473115845 conn(0x7fd634100a80 msgr2=0x7fd634102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 -- 172.21.15.97:0/473115845 shutdown_connections 2024-07-26T07:53:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.237+0000 7fd639600640 1 -- 172.21.15.97:0/473115845 wait complete. 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.238+0000 7fd639600640 1 Processor -- start 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.238+0000 7fd639600640 1 -- start start 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd639600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634105120 0x7fd63419b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd639600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6341061a0 0x7fd63419bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd639600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd634106b40 0x7fd63419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd639600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd634110c00 con 0x7fd634105120 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd639600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd634110a80 con 0x7fd634106b40 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd639600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd634110d80 con 0x7fd6341061a0 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634105120 0x7fd63419b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634105120 0x7fd63419b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36548/0 (socket says 172.21.15.97:36548) 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd633e00640 1 -- 172.21.15.97:0/3594739911 learned_addr learned my addr 172.21.15.97:0/3594739911 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd633400640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6341061a0 0x7fd63419bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd638c00640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd634106b40 0x7fd63419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd633e00640 1 -- 172.21.15.97:0/3594739911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6341061a0 msgr2=0x7fd63419bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.239+0000 7fd633e00640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6341061a0 0x7fd63419bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd633e00640 1 -- 172.21.15.97:0/3594739911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd634106b40 msgr2=0x7fd63419ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd633e00640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd634106b40 0x7fd63419ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd633e00640 1 -- 172.21.15.97:0/3594739911 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63419d220 con 0x7fd634105120 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd638c00640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd634106b40 0x7fd63419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd633e00640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634105120 0x7fd63419b870 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd62400d950 tx=0x7fd62400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd630c00640 1 -- 172.21.15.97:0/3594739911 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624017070 con 0x7fd634105120 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd639600640 1 -- 172.21.15.97:0/3594739911 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63419d510 con 0x7fd634105120 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd630c00640 1 -- 172.21.15.97:0/3594739911 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd624009c30 con 0x7fd634105120 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.240+0000 7fd639600640 1 -- 172.21.15.97:0/3594739911 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6341a3c90 con 0x7fd634105120 2024-07-26T07:53:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.241+0000 7fd630c00640 1 -- 172.21.15.97:0/3594739911 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624020060 con 0x7fd634105120 2024-07-26T07:53:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd641c45740 1 Processor -- start 2024-07-26T07:53:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd641c45740 1 -- start start 2024-07-26T07:53:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd641c45740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af43b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd641c45740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af43b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd641c45740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af43c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd641c45740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5579af3ad270 con 0x5579af06c2f0 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd641c45740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5579af39c770 con 0x5579af0951b0 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd641c45740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5579af3a07b0 con 0x5579af0744d0 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.244+0000 7fd630c00640 1 -- 172.21.15.97:0/3594739911 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fd624010460 con 0x7fd634105120 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af43b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af43b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36558/0 (socket says 172.21.15.97:36558) 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 -- 172.21.15.97:0/3738055386 learned_addr learned my addr 172.21.15.97:0/3738055386 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd638c00640 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af43b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633400640 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af43c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 -- 172.21.15.97:0/3738055386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 msgr2=0x5579af43c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af43c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 -- 172.21.15.97:0/3738055386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 msgr2=0x5579af43b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af43b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 -- 172.21.15.97:0/3738055386 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579af100790 con 0x5579af06c2f0 2024-07-26T07:53:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.245+0000 7fd633e00640 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af43b750 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd6240e7e20 tx=0x7fd6240e7f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d93daa335d2878 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.246+0000 7fd629200640 1 -- 172.21.15.97:0/3738055386 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6240ee450 con 0x5579af06c2f0 2024-07-26T07:53:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.246+0000 7fd629200640 1 -- 172.21.15.97:0/3738055386 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd6240eea80 con 0x5579af06c2f0 2024-07-26T07:53:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.246+0000 7fd630c00640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd6180b3bd0 0x7fd6180b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.246+0000 7fd629200640 1 -- 172.21.15.97:0/3738055386 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6240f69a0 con 0x5579af06c2f0 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.246+0000 7fd630c00640 1 -- 172.21.15.97:0/3594739911 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd6240e3cf0 con 0x7fd634105120 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.246+0000 7fd638c00640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd6180b3bd0 0x7fd6180b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 -- 172.21.15.97:0/3738055386 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 msgr2=0x5579af43b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af43b750 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd6240e7e20 tx=0x7fd6240e7f10 comp rx=0 tx=0).stop 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 -- 172.21.15.97:0/3738055386 shutdown_connections 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af43c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af43b750 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 --2- 172.21.15.97:0/3738055386 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af43b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd638c00640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd6180b3bd0 0x7fd6180b60e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd62c031920 tx=0x7fd62c031530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 -- 172.21.15.97:0/3738055386 >> 172.21.15.97:0/3738055386 conn(0x5579af089300 msgr2=0x5579af436b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 -- 172.21.15.97:0/3738055386 shutdown_connections 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.247+0000 7fd641c45740 1 -- 172.21.15.97:0/3738055386 wait complete. 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 1 -- start start 2024-07-26T07:53:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 1 Processor -- start 2024-07-26T07:53:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af06c2f0 0x5579af444c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af445340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0951b0 0x5579af445eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5579af2d6ab0 con 0x5579af0951b0 2024-07-26T07:53:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5579af339d50 con 0x5579af06c2f0 2024-07-26T07:53:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5579af43cdc0 con 0x5579af0744d0 2024-07-26T07:53:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.248+0000 7fd641c45740 10 client.0 _ll_register_callbacks cb 0x5579af0aca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd638c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0951b0 0x5579af445eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd638c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0951b0 0x5579af445eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36570/0 (socket says 172.21.15.97:36570) 2024-07-26T07:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd638c00640 1 -- 172.21.15.97:0/117238294 learned_addr learned my addr 172.21.15.97:0/117238294 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd633e00640 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af06c2f0 0x5579af444c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd633400640 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af445340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd638c00640 1 -- 172.21.15.97:0/117238294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 msgr2=0x5579af445340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd638c00640 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af445340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd638c00640 1 -- 172.21.15.97:0/117238294 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af06c2f0 msgr2=0x5579af444c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd638c00640 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af06c2f0 0x5579af444c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.249+0000 7fd638c00640 1 -- 172.21.15.97:0/117238294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579af179620 con 0x5579af0951b0 2024-07-26T07:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd638c00640 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0951b0 0x5579af445eb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd62c031f20 tx=0x7fd62c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f017763c684ceb66 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd61d600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd61d600640 1 -- 172.21.15.97:0/117238294 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c04e070 con 0x5579af0951b0 2024-07-26T07:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd61d600640 1 -- 172.21.15.97:0/117238294 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd62c003e00 con 0x5579af0951b0 2024-07-26T07:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd641c45740 1 -- 172.21.15.97:0/117238294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6240eea80 con 0x5579af0951b0 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd61d600640 1 -- 172.21.15.97:0/117238294 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd62c05a050 con 0x5579af0951b0 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd641c45740 10 client.15717 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd641c45740 1 -- 172.21.15.97:0/117238294 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5579af44ee00 con 0x5579af0951b0 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.250+0000 7fd61d600640 1 -- 172.21.15.97:0/117238294 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd6240eea80 con 0x5579af0951b0 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.251+0000 7fd61d600640 1 -- 172.21.15.97:0/117238294 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd62c043040 con 0x5579af0951b0 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.251+0000 7fd641c45740 10 client.15717 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.251+0000 7fd641c45740 10 client.15717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.251+0000 7fd641c45740 20 client.15717 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.251+0000 7fd641c45740 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5579af053d80 0x5579af452680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.251+0000 7fd641c45740 4 client.15717 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.251+0000 7fd641c45740 1 -- 172.21.15.97:0/117238294 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579af3af1c0 con 0x5579af053d80 2024-07-26T07:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.252+0000 7fd633e00640 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5579af053d80 0x5579af452680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.253+0000 7fd633e00640 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5579af053d80 0x5579af452680 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5128f5f36bb40cbc server_cookie=4c0f14df340b7a13 in_seq=0 out_seq=0 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.253+0000 7fd61d600640 10 client.15717 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd61d600640 1 -- 172.21.15.97:0/117238294 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5579af3af1c0 con 0x5579af053d80 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd61d600640 10 client.15717 handle_command_reply: tid=0 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 client.15717 shutdown 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 -- 172.21.15.97:0/117238294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0951b0 msgr2=0x5579af445eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0951b0 0x5579af445eb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd62c031f20 tx=0x7fd62c0095c0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 -- 172.21.15.97:0/117238294 shutdown_connections 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5579af053d80 0x5579af452680 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0951b0 0x5579af445eb0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af445340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 --2- 172.21.15.97:0/117238294 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af06c2f0 0x5579af444c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 -- 172.21.15.97:0/117238294 >> 172.21.15.97:0/117238294 conn(0x5579af089300 msgr2=0x5579af43c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd61d600640 0 client.15717 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 -- 172.21.15.97:0/117238294 shutdown_connections 2024-07-26T07:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 1 -- 172.21.15.97:0/117238294 wait complete. 2024-07-26T07:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.255+0000 7fd641c45740 20 client.15717 trim_cache size 0 max 16384 2024-07-26T07:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd641c45740 1 Processor -- start 2024-07-26T07:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd641c45740 1 -- start start 2024-07-26T07:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd641c45740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af06c2f0 0x5579af4317f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd641c45740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0744d0 0x5579af443190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd641c45740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af4436d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd641c45740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341ac9f0 con 0x5579af0744d0 2024-07-26T07:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd641c45740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341ae530 con 0x5579af0951b0 2024-07-26T07:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd641c45740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341ad0c0 con 0x5579af06c2f0 2024-07-26T07:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd633400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0744d0 0x5579af443190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd633400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0744d0 0x5579af443190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36572/0 (socket says 172.21.15.97:36572) 2024-07-26T07:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.272+0000 7fd633400640 1 -- 172.21.15.97:0/2385893274 learned_addr learned my addr 172.21.15.97:0/2385893274 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd633e00640 1 --2- 172.21.15.97:0/2385893274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af06c2f0 0x5579af4317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd633400640 1 -- 172.21.15.97:0/2385893274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af06c2f0 msgr2=0x5579af4317f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd633400640 1 --2- 172.21.15.97:0/2385893274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af06c2f0 0x5579af4317f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd633400640 1 -- 172.21.15.97:0/2385893274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 msgr2=0x5579af4436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd633400640 1 --2- 172.21.15.97:0/2385893274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af4436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd633400640 1 -- 172.21.15.97:0/2385893274 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579af43b4e0 con 0x5579af0744d0 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd633400640 1 --2- 172.21.15.97:0/2385893274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0744d0 0x5579af443190 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd62000edb0 tx=0x7fd620012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaab562982a1e758 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd61fe00640 1 -- 172.21.15.97:0/2385893274 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620005220 con 0x5579af0744d0 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd61fe00640 1 -- 172.21.15.97:0/2385893274 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd620004c30 con 0x5579af0744d0 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.273+0000 7fd61fe00640 1 -- 172.21.15.97:0/2385893274 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620010a50 con 0x5579af0744d0 2024-07-26T07:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 -- 172.21.15.97:0/2385893274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0744d0 msgr2=0x5579af443190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 --2- 172.21.15.97:0/2385893274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0744d0 0x5579af443190 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd62000edb0 tx=0x7fd620012820 comp rx=0 tx=0).stop 2024-07-26T07:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 -- 172.21.15.97:0/2385893274 shutdown_connections 2024-07-26T07:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 --2- 172.21.15.97:0/2385893274 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af4436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 --2- 172.21.15.97:0/2385893274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af0744d0 0x5579af443190 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 --2- 172.21.15.97:0/2385893274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af06c2f0 0x5579af4317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 -- 172.21.15.97:0/2385893274 >> 172.21.15.97:0/2385893274 conn(0x5579af089300 msgr2=0x5579af43fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 -- 172.21.15.97:0/2385893274 shutdown_connections 2024-07-26T07:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.274+0000 7fd641c45740 1 -- 172.21.15.97:0/2385893274 wait complete. 2024-07-26T07:53:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.275+0000 7fd641c45740 1 -- start start 2024-07-26T07:53:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.275+0000 7fd641c45740 1 Processor -- start 2024-07-26T07:53:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.275+0000 7fd641c45740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af448410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd641c45740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af448ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd641c45740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af44c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd641c45740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341ade60 con 0x5579af06c2f0 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd641c45740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341ad790 con 0x5579af0951b0 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd641c45740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341aa9d0 con 0x5579af0744d0 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd641c45740 10 client.0 _ll_register_callbacks cb 0x7fd62c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af448410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af448410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36582/0 (socket says 172.21.15.97:36582) 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633e00640 1 -- 172.21.15.97:0/3199967024 learned_addr learned my addr 172.21.15.97:0/3199967024 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633e00640 1 -- 172.21.15.97:0/3199967024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 msgr2=0x5579af448ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633400640 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af448ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633e00640 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af448ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633e00640 1 -- 172.21.15.97:0/3199967024 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 msgr2=0x5579af44c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633e00640 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af44c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.276+0000 7fd633e00640 1 -- 172.21.15.97:0/3199967024 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579af44cdd0 con 0x5579af06c2f0 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd633e00640 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af448410 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd6240a7df0 tx=0x7fd6240ecf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfde9f3261da9aa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd61d600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd61d600640 1 -- 172.21.15.97:0/3199967024 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624014020 con 0x5579af06c2f0 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd61d600640 1 -- 172.21.15.97:0/3199967024 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd6240e17b0 con 0x5579af06c2f0 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 1 -- 172.21.15.97:0/3199967024 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd620004c30 con 0x5579af06c2f0 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd61d600640 1 -- 172.21.15.97:0/3199967024 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd6240a6a40 con 0x5579af06c2f0 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 10 client.15723 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 1 -- 172.21.15.97:0/3199967024 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5579af44a310 con 0x5579af06c2f0 2024-07-26T07:53:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd61d600640 1 -- 172.21.15.97:0/3199967024 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd620004c30 con 0x5579af06c2f0 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd61d600640 1 -- 172.21.15.97:0/3199967024 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd6240eb040 con 0x5579af06c2f0 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 10 client.15723 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 20 client.15723 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5579af0291c0 0x5579af4bd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 4 client.15723 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.277+0000 7fd641c45740 1 -- 172.21.15.97:0/3199967024 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5579af3aac60 con 0x5579af0291c0 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.278+0000 7fd633400640 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5579af0291c0 0x5579af4bd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.279+0000 7fd61d600640 10 client.15723 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.279+0000 7fd633400640 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5579af0291c0 0x5579af4bd100 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=760de05139ce4d66 server_cookie=611f52a41a597b46 in_seq=0 out_seq=0 2024-07-26T07:53:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd61d600640 1 -- 172.21.15.97:0/3199967024 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5579af3aac60 con 0x5579af0291c0 2024-07-26T07:53:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd61d600640 10 client.15723 handle_command_reply: tid=0 2024-07-26T07:53:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 client.15723 shutdown 2024-07-26T07:53:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 -- 172.21.15.97:0/3199967024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 msgr2=0x5579af448410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af448410 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd6240a7df0 tx=0x7fd6240ecf60 comp rx=0 tx=0).stop 2024-07-26T07:53:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 -- 172.21.15.97:0/3199967024 shutdown_connections 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5579af0291c0 0x5579af4bd100 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5579af0951b0 0x5579af44c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd61d600640 0 client.15723 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579af0744d0 0x5579af448ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 --2- 172.21.15.97:0/3199967024 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5579af06c2f0 0x5579af448410 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.331+0000 7fd641c45740 1 -- 172.21.15.97:0/3199967024 >> 172.21.15.97:0/3199967024 conn(0x5579af089300 msgr2=0x5579af443710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.332+0000 7fd641c45740 1 -- 172.21.15.97:0/3199967024 shutdown_connections 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.332+0000 7fd641c45740 1 -- 172.21.15.97:0/3199967024 wait complete. 2024-07-26T07:53:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.332+0000 7fd641c45740 20 client.15723 trim_cache size 0 max 16384 2024-07-26T07:53:28.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:53:28.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:53:28.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:53:28.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:53:28.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 -- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd6180b3bd0 msgr2=0x7fd6180b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd6180b3bd0 0x7fd6180b60e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd62c031920 tx=0x7fd62c031530 comp rx=0 tx=0).stop 2024-07-26T07:53:28.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 -- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634105120 msgr2=0x7fd63419b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:28.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634105120 0x7fd63419b870 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd62400d950 tx=0x7fd62400de10 comp rx=0 tx=0).stop 2024-07-26T07:53:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 -- 172.21.15.97:0/3594739911 shutdown_connections 2024-07-26T07:53:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd6180b3bd0 0x7fd6180b60e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd634106b40 0x7fd63419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6341061a0 0x7fd63419bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 --2- 172.21.15.97:0/3594739911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd634105120 0x7fd63419b870 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 -- 172.21.15.97:0/3594739911 >> 172.21.15.97:0/3594739911 conn(0x7fd634100a80 msgr2=0x7fd6341094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 -- 172.21.15.97:0/3594739911 shutdown_connections 2024-07-26T07:53:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:28.333+0000 7fd639600640 1 -- 172.21.15.97:0/3594739911 wait complete. 2024-07-26T07:53:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:28 smithi115 ceph-mon[35364]: pgmap v560: 97 pgs: 97 active+clean; 2.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 138 KiB/s wr, 34 op/s 2024-07-26T07:53:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:28 smithi181 ceph-mon[35869]: pgmap v560: 97 pgs: 97 active+clean; 2.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 138 KiB/s wr, 34 op/s 2024-07-26T07:53:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:28 smithi097 ceph-mon[32231]: pgmap v560: 97 pgs: 97 active+clean; 2.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 138 KiB/s wr, 34 op/s 2024-07-26T07:53:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:30 smithi115 ceph-mon[35364]: pgmap v561: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 149 KiB/s wr, 36 op/s 2024-07-26T07:53:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:30 smithi181 ceph-mon[35869]: pgmap v561: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 149 KiB/s wr, 36 op/s 2024-07-26T07:53:30.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:30 smithi097 ceph-mon[32231]: pgmap v561: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 149 KiB/s wr, 36 op/s 2024-07-26T07:53:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:32 smithi097 ceph-mon[32231]: pgmap v562: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 142 KiB/s wr, 46 op/s 2024-07-26T07:53:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:32 smithi115 ceph-mon[35364]: pgmap v562: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 142 KiB/s wr, 46 op/s 2024-07-26T07:53:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:32 smithi181 ceph-mon[35869]: pgmap v562: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 142 KiB/s wr, 46 op/s 2024-07-26T07:53:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:34 smithi115 ceph-mon[35364]: pgmap v563: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 89 KiB/s wr, 41 op/s 2024-07-26T07:53:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:34 smithi181 ceph-mon[35869]: pgmap v563: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 89 KiB/s wr, 41 op/s 2024-07-26T07:53:34.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:34 smithi097 ceph-mon[32231]: pgmap v563: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 89 KiB/s wr, 41 op/s 2024-07-26T07:53:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:36 smithi115 ceph-mon[35364]: pgmap v564: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 81 KiB/s wr, 42 op/s 2024-07-26T07:53:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:36 smithi181 ceph-mon[35869]: pgmap v564: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 81 KiB/s wr, 42 op/s 2024-07-26T07:53:36.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:36 smithi097 ceph-mon[32231]: pgmap v564: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 81 KiB/s wr, 42 op/s 2024-07-26T07:53:37.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:53:37.909 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:53:38.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:38 smithi097 ceph-mon[32231]: pgmap v565: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-07-26T07:53:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:38 smithi115 ceph-mon[35364]: pgmap v565: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-07-26T07:53:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:38 smithi181 ceph-mon[35869]: pgmap v565: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-07-26T07:53:40.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.097+0000 7f055b200640 1 -- 172.21.15.97:0/4057580743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c109650 msgr2=0x7f055c10fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:40.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.097+0000 7f055b200640 1 --2- 172.21.15.97:0/4057580743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c109650 0x7f055c10fe80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0554009ca0 tx=0x7f055402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.098+0000 7f055b200640 1 -- 172.21.15.97:0/4057580743 shutdown_connections 2024-07-26T07:53:40.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.098+0000 7f055b200640 1 --2- 172.21.15.97:0/4057580743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c109650 0x7f055c10fe80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.098+0000 7f055b200640 1 --2- 172.21.15.97:0/4057580743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f055c108d00 0x7f055c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.098+0000 7f055b200640 1 --2- 172.21.15.97:0/4057580743 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f055c1033c0 0x7f055c1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.098+0000 7f055b200640 1 -- 172.21.15.97:0/4057580743 >> 172.21.15.97:0/4057580743 conn(0x7f055c0fee10 msgr2=0x7f055c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:40.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.098+0000 7f055b200640 1 -- 172.21.15.97:0/4057580743 shutdown_connections 2024-07-26T07:53:40.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.098+0000 7f055b200640 1 -- 172.21.15.97:0/4057580743 wait complete. 2024-07-26T07:53:40.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.099+0000 7f055b200640 1 Processor -- start 2024-07-26T07:53:40.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.099+0000 7f055b200640 1 -- start start 2024-07-26T07:53:40.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.099+0000 7f055b200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f055c1033c0 0x7f055c19be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:40.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.099+0000 7f055b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c108d00 0x7f055c19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:40.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.099+0000 7f055b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f055c109650 0x7f055c1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:40.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.099+0000 7f055b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f055c113220 con 0x7f055c108d00 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.099+0000 7f055b200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1130a0 con 0x7f055c1033c0 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.099+0000 7f055b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1133a0 con 0x7f055c109650 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0553e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c108d00 0x7f055c19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0553e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c108d00 0x7f055c19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48648/0 (socket says 172.21.15.97:48648) 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0553e00640 1 -- 172.21.15.97:0/3440343404 learned_addr learned my addr 172.21.15.97:0/3440343404 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0559e00640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f055c1033c0 0x7f055c19be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0553e00640 1 -- 172.21.15.97:0/3440343404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f055c109650 msgr2=0x7f055c1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0553e00640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f055c109650 0x7f055c1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0553e00640 1 -- 172.21.15.97:0/3440343404 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f055c1033c0 msgr2=0x7f055c19be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0553e00640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f055c1033c0 0x7f055c19be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0553e00640 1 -- 172.21.15.97:0/3440343404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055c1a1d40 con 0x7f055c108d00 2024-07-26T07:53:40.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.100+0000 7f0559e00640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f055c1033c0 0x7f055c19be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:53:40.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.101+0000 7f0553e00640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c108d00 0x7f055c19c430 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f054400c970 tx=0x7f054400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:40.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.101+0000 7f0552a00640 1 -- 172.21.15.97:0/3440343404 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544004220 con 0x7f055c108d00 2024-07-26T07:53:40.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.101+0000 7f0552a00640 1 -- 172.21.15.97:0/3440343404 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0544007e40 con 0x7f055c108d00 2024-07-26T07:53:40.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.101+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055c1a2030 con 0x7f055c108d00 2024-07-26T07:53:40.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.101+0000 7f0552a00640 1 -- 172.21.15.97:0/3440343404 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05440047a0 con 0x7f055c108d00 2024-07-26T07:53:40.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.101+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f055c1a8850 con 0x7f055c108d00 2024-07-26T07:53:40.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.103+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f055c1037d0 con 0x7f055c108d00 2024-07-26T07:53:40.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.105+0000 7f0552a00640 1 -- 172.21.15.97:0/3440343404 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f05440079b0 con 0x7f055c108d00 2024-07-26T07:53:40.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.109+0000 7f0552a00640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05400b3c80 0x7f05400b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:40.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.109+0000 7f0552a00640 1 -- 172.21.15.97:0/3440343404 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f05440e3d00 con 0x7f055c108d00 2024-07-26T07:53:40.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.110+0000 7f0559e00640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05400b3c80 0x7f05400b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:40.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.110+0000 7f0559e00640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05400b3c80 0x7f05400b6190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f054c00abe0 tx=0x7f054c00a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:40.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.111+0000 7f0552a00640 1 -- 172.21.15.97:0/3440343404 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0544014030 con 0x7f055c108d00 2024-07-26T07:53:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:40 smithi097 ceph-mon[32231]: pgmap v566: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 26 op/s 2024-07-26T07:53:40.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.348+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f055c19cfc0 con 0x7f055c108d00 2024-07-26T07:53:40.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.350+0000 7f0552a00640 1 -- 172.21.15.97:0/3440343404 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11990 (secure 0 0 0) 0x7f05440ab310 con 0x7f055c108d00 2024-07-26T07:53:40.352 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:53:40.352 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-07-26T07:53:40.352 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"btime":"2024-07-26T07:51:41:000369+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14691,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:51:40.647454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3053625422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3053625422},{"type":"v1","addr":"172.21.15.181:6837","nonce":3053625422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:53:40.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05400b3c80 msgr2=0x7f05400b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:40.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05400b3c80 0x7f05400b6190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f054c00abe0 tx=0x7f054c00a5c0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c108d00 msgr2=0x7f055c19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c108d00 0x7f055c19c430 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f054400c970 tx=0x7f054400ce30 comp rx=0 tx=0).stop 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 shutdown_connections 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f05400b3c80 0x7f05400b6190 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f055c109650 0x7f055c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f055c108d00 0x7f055c19c430 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 --2- 172.21.15.97:0/3440343404 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f055c1033c0 0x7f055c19be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.353+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 >> 172.21.15.97:0/3440343404 conn(0x7f055c0fee10 msgr2=0x7f055c1083a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.354+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 shutdown_connections 2024-07-26T07:53:40.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:40.354+0000 7f055b200640 1 -- 172.21.15.97:0/3440343404 wait complete. 2024-07-26T07:53:40.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:40 smithi115 ceph-mon[35364]: pgmap v566: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 26 op/s 2024-07-26T07:53:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:40 smithi181 ceph-mon[35869]: pgmap v566: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 26 op/s 2024-07-26T07:53:41.087 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db --set-id 9b1e79da --release --await' 2024-07-26T07:53:41.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db --set-id 9b1e79da --release --await 2024-07-26T07:53:41.377 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:41 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3440343404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:41 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3440343404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:41 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3440343404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:53:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 -- 172.21.15.97:0/1546181840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 msgr2=0x7f933c10d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 --2- 172.21.15.97:0/1546181840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 0x7f933c10d520 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9334009ca0 tx=0x7f933402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.549 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:53:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 -- 172.21.15.97:0/1546181840 shutdown_connections 2024-07-26T07:53:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 --2- 172.21.15.97:0/1546181840 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 0x7f933c10d520 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 --2- 172.21.15.97:0/1546181840 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933c075720 0x7f933c075b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 --2- 172.21.15.97:0/1546181840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c076b20 0x7f933c0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 -- 172.21.15.97:0/1546181840 >> 172.21.15.97:0/1546181840 conn(0x7f933c0fea20 msgr2=0x7f933c100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 -- 172.21.15.97:0/1546181840 shutdown_connections 2024-07-26T07:53:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.547+0000 7f9343c00640 1 -- 172.21.15.97:0/1546181840 wait complete. 2024-07-26T07:53:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9343c00640 1 Processor -- start 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9343c00640 1 -- start start 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9343c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c075720 0x7f933c19bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9343c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933c076b20 0x7f933c19c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9343c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 0x7f933c19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9343c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f933c110e30 con 0x7f933c10b070 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9343c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f933c110cb0 con 0x7f933c076b20 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9343c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f933c110fb0 con 0x7f933c075720 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9341c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 0x7f933c19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9341c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 0x7f933c19cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48656/0 (socket says 172.21.15.97:48656) 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.549+0000 7f9341c00640 1 -- 172.21.15.97:0/158163825 learned_addr learned my addr 172.21.15.97:0/158163825 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.550+0000 7f933be00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933c076b20 0x7f933c19c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.550+0000 7f9341200640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c075720 0x7f933c19bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.550+0000 7f9341c00640 1 -- 172.21.15.97:0/158163825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c075720 msgr2=0x7f933c19bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.550+0000 7f9341c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c075720 0x7f933c19bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.550+0000 7f9341c00640 1 -- 172.21.15.97:0/158163825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933c076b20 msgr2=0x7f933c19c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.550+0000 7f9341c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933c076b20 0x7f933c19c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.550+0000 7f9341c00640 1 -- 172.21.15.97:0/158163825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933c19d420 con 0x7f933c10b070 2024-07-26T07:53:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.550+0000 7f9341c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 0x7f933c19cc90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f93340098d0 tx=0x7f9334003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.551+0000 7f9339600640 1 -- 172.21.15.97:0/158163825 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9334003d90 con 0x7f933c10b070 2024-07-26T07:53:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.551+0000 7f9343c00640 1 -- 172.21.15.97:0/158163825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f933c19d6b0 con 0x7f933c10b070 2024-07-26T07:53:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.551+0000 7f9339600640 1 -- 172.21.15.97:0/158163825 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93340044d0 con 0x7f933c10b070 2024-07-26T07:53:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.551+0000 7f9339600640 1 -- 172.21.15.97:0/158163825 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9334030d10 con 0x7f933c10b070 2024-07-26T07:53:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.551+0000 7f9343c00640 1 -- 172.21.15.97:0/158163825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f933c1a3e90 con 0x7f933c10b070 2024-07-26T07:53:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.554+0000 7f934ac7a740 1 Processor -- start 2024-07-26T07:53:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.556+0000 7f9339600640 1 -- 172.21.15.97:0/158163825 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f9334004040 con 0x7f933c10b070 2024-07-26T07:53:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.556+0000 7f934ac7a740 1 -- start start 2024-07-26T07:53:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.556+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296be71b0 0x556296f95710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.556+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f95d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.556+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bc64d0 0x556296f968d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.556+0000 7f934ac7a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x556296eff270 con 0x556296bc64d0 2024-07-26T07:53:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.556+0000 7f934ac7a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556296eee770 con 0x556296be71b0 2024-07-26T07:53:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.556+0000 7f934ac7a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556296ef27b0 con 0x556296bbe2f0 2024-07-26T07:53:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bc64d0 0x556296f968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bc64d0 0x556296f968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48662/0 (socket says 172.21.15.97:48662) 2024-07-26T07:53:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 -- 172.21.15.97:0/3304879484 learned_addr learned my addr 172.21.15.97:0/3304879484 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296be71b0 0x556296f95710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f9341200640 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f95d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 -- 172.21.15.97:0/3304879484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 msgr2=0x556296f95d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f95d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 -- 172.21.15.97:0/3304879484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296be71b0 msgr2=0x556296f95710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296be71b0 0x556296f95710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.557+0000 7f933be00640 1 -- 172.21.15.97:0/3304879484 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556296c52790 con 0x556296bc64d0 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.558+0000 7f9339600640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f93240b3c50 0x7f93240b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.558+0000 7f933be00640 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bc64d0 0x556296f968d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f93280149f0 tx=0x7f932800c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df11084c3239848 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.558+0000 7f9339600640 1 -- 172.21.15.97:0/158163825 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f9334105180 con 0x7f933c10b070 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.558+0000 7f932e000640 1 -- 172.21.15.97:0/3304879484 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93280142d0 con 0x556296bc64d0 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.558+0000 7f9341200640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f93240b3c50 0x7f93240b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.558+0000 7f932e000640 1 -- 172.21.15.97:0/3304879484 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f932800fbf0 con 0x556296bc64d0 2024-07-26T07:53:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.558+0000 7f932e000640 1 -- 172.21.15.97:0/3304879484 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93280129a0 con 0x556296bc64d0 2024-07-26T07:53:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.559+0000 7f934ac7a740 1 -- 172.21.15.97:0/3304879484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bc64d0 msgr2=0x556296f968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.559+0000 7f934ac7a740 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bc64d0 0x556296f968d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f93280149f0 tx=0x7f932800c4f0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.560+0000 7f9341200640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f93240b3c50 0x7f93240b6160 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9330005e10 tx=0x7f9330005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.560+0000 7f934ac7a740 1 -- 172.21.15.97:0/3304879484 shutdown_connections 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.560+0000 7f934ac7a740 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bc64d0 0x556296f968d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.560+0000 7f934ac7a740 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f95d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.560+0000 7f934ac7a740 1 --2- 172.21.15.97:0/3304879484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296be71b0 0x556296f95710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.560+0000 7f934ac7a740 1 -- 172.21.15.97:0/3304879484 >> 172.21.15.97:0/3304879484 conn(0x556296bdb300 msgr2=0x556296f88990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.560+0000 7f934ac7a740 1 -- 172.21.15.97:0/3304879484 shutdown_connections 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.560+0000 7f934ac7a740 1 -- 172.21.15.97:0/3304879484 wait complete. 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.561+0000 7f934ac7a740 1 -- start start 2024-07-26T07:53:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.561+0000 7f934ac7a740 1 Processor -- start 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.561+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.561+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296f9f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.561+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fa0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.561+0000 7f934ac7a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x556296e28ab0 con 0x556296be71b0 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.561+0000 7f934ac7a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556296e8bd50 con 0x556296bc64d0 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.561+0000 7f934ac7a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556296f973f0 con 0x556296bbe2f0 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f934ac7a740 10 client.0 _ll_register_callbacks cb 0x556296bfea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fa0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fa0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48668/0 (socket says 172.21.15.97:48668) 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341c00640 1 -- 172.21.15.97:0/121889444 learned_addr learned my addr 172.21.15.97:0/121889444 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341200640 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f933be00640 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296f9f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341c00640 1 -- 172.21.15.97:0/121889444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 msgr2=0x556296f9f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341c00640 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341c00640 1 -- 172.21.15.97:0/121889444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 msgr2=0x556296f9f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341c00640 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296f9f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.562+0000 7f9341c00640 1 -- 172.21.15.97:0/121889444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556296ccb620 con 0x556296be71b0 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.563+0000 7f9341c00640 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fa0510 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f93340c80f0 tx=0x7f93340cb8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8631296572673d08 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.563+0000 7f9322a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.563+0000 7f9322a00640 1 -- 172.21.15.97:0/121889444 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93340cc070 con 0x556296be71b0 2024-07-26T07:53:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.563+0000 7f9322a00640 1 -- 172.21.15.97:0/121889444 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f93340305c0 con 0x556296be71b0 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.563+0000 7f934ac7a740 1 -- 172.21.15.97:0/121889444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f932800fbf0 con 0x556296be71b0 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.563+0000 7f9322a00640 1 -- 172.21.15.97:0/121889444 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f93340d0670 con 0x556296be71b0 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.563+0000 7f934ac7a740 10 client.15741 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.563+0000 7f934ac7a740 1 -- 172.21.15.97:0/121889444 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556296fa9460 con 0x556296be71b0 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.564+0000 7f9322a00640 1 -- 172.21.15.97:0/121889444 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f932800fbf0 con 0x556296be71b0 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.564+0000 7f9322a00640 1 -- 172.21.15.97:0/121889444 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f93340d4c90 con 0x556296be71b0 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.564+0000 7f934ac7a740 10 client.15741 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.564+0000 7f934ac7a740 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.564+0000 7f934ac7a740 20 client.15741 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.564+0000 7f934ac7a740 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556296ba5d80 0x556296facc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.564+0000 7f934ac7a740 4 client.15741 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:53:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.564+0000 7f934ac7a740 1 -- 172.21.15.97:0/121889444 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556296f011c0 con 0x556296ba5d80 2024-07-26T07:53:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.565+0000 7f9341200640 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556296ba5d80 0x556296facc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.566+0000 7f9341200640 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556296ba5d80 0x556296facc40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9f2c3c87a5efd62 server_cookie=6d7d490bc299b860 in_seq=0 out_seq=0 2024-07-26T07:53:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.566+0000 7f9322a00640 10 client.15741 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f9322a00640 1 -- 172.21.15.97:0/121889444 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556296f011c0 con 0x556296ba5d80 2024-07-26T07:53:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f9322a00640 10 client.15741 handle_command_reply: tid=0 2024-07-26T07:53:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 client.15741 shutdown 2024-07-26T07:53:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 -- 172.21.15.97:0/121889444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 msgr2=0x556296fa0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fa0510 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f93340c80f0 tx=0x7f93340cb8a0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 -- 172.21.15.97:0/121889444 shutdown_connections 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556296ba5d80 0x556296facc40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fa0510 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296f9f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 --2- 172.21.15.97:0/121889444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 -- 172.21.15.97:0/121889444 >> 172.21.15.97:0/121889444 conn(0x556296bdb300 msgr2=0x556296f870d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f9322a00640 0 client.15741 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 -- 172.21.15.97:0/121889444 shutdown_connections 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 1 -- 172.21.15.97:0/121889444 wait complete. 2024-07-26T07:53:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.568+0000 7f934ac7a740 20 client.15741 trim_cache size 0 max 16384 2024-07-26T07:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.585+0000 7f934ac7a740 1 Processor -- start 2024-07-26T07:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.585+0000 7f934ac7a740 1 -- start start 2024-07-26T07:53:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.585+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.585+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296f870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.585+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fb13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f934ac7a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f933c1a8740 con 0x556296be71b0 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f934ac7a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f933c1aa280 con 0x556296bc64d0 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f934ac7a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f933c1a8e10 con 0x556296bbe2f0 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fb13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fb13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48684/0 (socket says 172.21.15.97:48684) 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341c00640 1 -- 172.21.15.97:0/2302449352 learned_addr learned my addr 172.21.15.97:0/2302449352 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f933be00640 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296f870d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341200640 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341c00640 1 -- 172.21.15.97:0/2302449352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 msgr2=0x556296f9e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341c00640 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341c00640 1 -- 172.21.15.97:0/2302449352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 msgr2=0x556296f870d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341c00640 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296f870d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341c00640 1 -- 172.21.15.97:0/2302449352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556296f9f070 con 0x556296be71b0 2024-07-26T07:53:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.586+0000 7f9341200640 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.587+0000 7f9341c00640 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fb13f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9334004890 tx=0x7f933402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41ed6e7d8e7aa4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.587+0000 7f932d600640 1 -- 172.21.15.97:0/2302449352 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93340cbe40 con 0x556296be71b0 2024-07-26T07:53:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.587+0000 7f932d600640 1 -- 172.21.15.97:0/2302449352 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9334036040 con 0x556296be71b0 2024-07-26T07:53:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.587+0000 7f932d600640 1 -- 172.21.15.97:0/2302449352 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9334030690 con 0x556296be71b0 2024-07-26T07:53:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 -- 172.21.15.97:0/2302449352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 msgr2=0x556296fb13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fb13f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9334004890 tx=0x7f933402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 -- 172.21.15.97:0/2302449352 shutdown_connections 2024-07-26T07:53:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296be71b0 0x556296fb13f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296f870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2302449352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296bbe2f0 0x556296f9e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 -- 172.21.15.97:0/2302449352 >> 172.21.15.97:0/2302449352 conn(0x556296bdb300 msgr2=0x556296facbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 -- 172.21.15.97:0/2302449352 shutdown_connections 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.588+0000 7f934ac7a740 1 -- 172.21.15.97:0/2302449352 wait complete. 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 1 -- start start 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 1 Processor -- start 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bbe2f0 0x556296fa6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296fa65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296be71b0 0x5562970133c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f933c1a9bb0 con 0x556296bbe2f0 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f933c1a94e0 con 0x556296bc64d0 2024-07-26T07:53:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f933c1a6720 con 0x556296be71b0 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.589+0000 7f934ac7a740 10 client.0 _ll_register_callbacks cb 0x7f93340095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bbe2f0 0x556296fa6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bbe2f0 0x556296fa6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48696/0 (socket says 172.21.15.97:48696) 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341200640 1 -- 172.21.15.97:0/2504971429 learned_addr learned my addr 172.21.15.97:0/2504971429 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f933be00640 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296fa65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341200640 1 -- 172.21.15.97:0/2504971429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296be71b0 msgr2=0x5562970133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341c00640 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296be71b0 0x5562970133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341200640 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296be71b0 0x5562970133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341200640 1 -- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 msgr2=0x556296fa65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341200640 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296fa65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f9341200640 1 -- 172.21.15.97:0/2504971429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556297013900 con 0x556296bbe2f0 2024-07-26T07:53:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.590+0000 7f933be00640 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296fa65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.591+0000 7f9341200640 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bbe2f0 0x556296fa6080 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9330005940 tx=0x7f93300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde72cc770c07c46 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.591+0000 7f9322a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.591+0000 7f9322a00640 1 -- 172.21.15.97:0/2504971429 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9330005220 con 0x556296bbe2f0 2024-07-26T07:53:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.591+0000 7f9322a00640 1 -- 172.21.15.97:0/2504971429 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f93300053c0 con 0x556296bbe2f0 2024-07-26T07:53:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.591+0000 7f934ac7a740 1 -- 172.21.15.97:0/2504971429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9334036040 con 0x556296bbe2f0 2024-07-26T07:53:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.591+0000 7f9322a00640 1 -- 172.21.15.97:0/2504971429 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9330038050 con 0x556296bbe2f0 2024-07-26T07:53:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.591+0000 7f934ac7a740 10 client.15747 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.591+0000 7f934ac7a740 1 -- 172.21.15.97:0/2504971429 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556297014aa0 con 0x556296bbe2f0 2024-07-26T07:53:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.592+0000 7f9322a00640 1 -- 172.21.15.97:0/2504971429 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9334036040 con 0x556296bbe2f0 2024-07-26T07:53:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.592+0000 7f9322a00640 1 -- 172.21.15.97:0/2504971429 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9330004b30 con 0x556296bbe2f0 2024-07-26T07:53:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.592+0000 7f934ac7a740 10 client.15747 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.592+0000 7f934ac7a740 10 client.15747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.592+0000 7f934ac7a740 20 client.15747 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.592+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556296b7b1c0 0x556297017470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.592+0000 7f934ac7a740 4 client.15747 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:53:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.592+0000 7f934ac7a740 1 -- 172.21.15.97:0/2504971429 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556296efcc60 con 0x556296b7b1c0 2024-07-26T07:53:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.593+0000 7f933be00640 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556296b7b1c0 0x556297017470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.594+0000 7f933be00640 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556296b7b1c0 0x556297017470 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b708b83fded0cb70 server_cookie=96db30d5455bc480 in_seq=0 out_seq=0 2024-07-26T07:53:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.594+0000 7f9322a00640 10 client.15747 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f9322a00640 1 -- 172.21.15.97:0/2504971429 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556296efcc60 con 0x556296b7b1c0 2024-07-26T07:53:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f9322a00640 10 client.15747 handle_command_reply: tid=0 2024-07-26T07:53:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 client.15747 shutdown 2024-07-26T07:53:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 -- 172.21.15.97:0/2504971429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bbe2f0 msgr2=0x556296fa6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bbe2f0 0x556296fa6080 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9330005940 tx=0x7f93300046c0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 -- 172.21.15.97:0/2504971429 shutdown_connections 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556296b7b1c0 0x556297017470 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556296be71b0 0x5562970133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556296bc64d0 0x556296fa65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 --2- 172.21.15.97:0/2504971429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556296bbe2f0 0x556296fa6080 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f9322a00640 0 client.15747 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 -- 172.21.15.97:0/2504971429 >> 172.21.15.97:0/2504971429 conn(0x556296bdb300 msgr2=0x556296facbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 -- 172.21.15.97:0/2504971429 shutdown_connections 2024-07-26T07:53:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.625+0000 7f934ac7a740 1 -- 172.21.15.97:0/2504971429 wait complete. 2024-07-26T07:53:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f934ac7a740 20 client.15747 trim_cache size 0 max 16384 2024-07-26T07:53:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:53:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:53:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:53:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 -- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f93240b3c50 msgr2=0x7f93240b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f93240b3c50 0x7f93240b6160 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9330005e10 tx=0x7f9330005da0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 -- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 msgr2=0x7f933c19cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 0x7f933c19cc90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f93340098d0 tx=0x7f9334003ba0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 -- 172.21.15.97:0/158163825 shutdown_connections 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f93240b3c50 0x7f93240b6160 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f933c10b070 0x7f933c19cc90 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933c076b20 0x7f933c19c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 --2- 172.21.15.97:0/158163825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c075720 0x7f933c19bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.626+0000 7f9343c00640 1 -- 172.21.15.97:0/158163825 >> 172.21.15.97:0/158163825 conn(0x7f933c0fea20 msgr2=0x7f933c107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.627+0000 7f9343c00640 1 -- 172.21.15.97:0/158163825 shutdown_connections 2024-07-26T07:53:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:41.627+0000 7f9343c00640 1 -- 172.21.15.97:0/158163825 wait complete. 2024-07-26T07:53:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:42 smithi097 ceph-mon[32231]: pgmap v567: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 24 op/s 2024-07-26T07:53:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:42 smithi115 ceph-mon[35364]: pgmap v567: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 24 op/s 2024-07-26T07:53:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:42 smithi181 ceph-mon[35869]: pgmap v567: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 24 op/s 2024-07-26T07:53:44.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 -- 172.21.15.97:0/2427261195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d0075ad0 msgr2=0x7f11d010f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 --2- 172.21.15.97:0/2427261195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d0075ad0 0x7f11d010f460 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009900 tx=0x7f11bc02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 -- 172.21.15.97:0/2427261195 shutdown_connections 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 --2- 172.21.15.97:0/2427261195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11d010f9a0 0x7f11d0111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 --2- 172.21.15.97:0/2427261195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d0075ad0 0x7f11d010f460 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 --2- 172.21.15.97:0/2427261195 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11d0075180 0x7f11d0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 -- 172.21.15.97:0/2427261195 >> 172.21.15.97:0/2427261195 conn(0x7f11d00fee30 msgr2=0x7f11d0101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 -- 172.21.15.97:0/2427261195 shutdown_connections 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.162+0000 7f11d6400640 1 -- 172.21.15.97:0/2427261195 wait complete. 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.163+0000 7f11d6400640 1 Processor -- start 2024-07-26T07:53:44.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.164+0000 7f11d6400640 1 -- start start 2024-07-26T07:53:44.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.164+0000 7f11d6400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11d0075180 0x7f11d019be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.164+0000 7f11d6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11d0075ad0 0x7f11d019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.164+0000 7f11d6400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d010f9a0 0x7f11d01a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.164+0000 7f11d6400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01153c0 con 0x7f11d010f9a0 2024-07-26T07:53:44.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.164+0000 7f11d6400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11d0115240 con 0x7f11d0075180 2024-07-26T07:53:44.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.164+0000 7f11d6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f11d0115540 con 0x7f11d0075ad0 2024-07-26T07:53:44.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d010f9a0 0x7f11d01a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d010f9a0 0x7f11d01a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48718/0 (socket says 172.21.15.97:48718) 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 -- 172.21.15.97:0/4268806299 learned_addr learned my addr 172.21.15.97:0/4268806299 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5000640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11d0075180 0x7f11d019be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11cfe00640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11d0075ad0 0x7f11d019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 -- 172.21.15.97:0/4268806299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11d0075ad0 msgr2=0x7f11d019c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11d0075ad0 0x7f11d019c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 -- 172.21.15.97:0/4268806299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11d0075180 msgr2=0x7f11d019be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11d0075180 0x7f11d019be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 -- 172.21.15.97:0/4268806299 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d01a1d20 con 0x7f11d010f9a0 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.165+0000 7f11d5a00640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d010f9a0 0x7f11d01a16b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f11c800cce0 tx=0x7f11c800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.166+0000 7f11cd600640 1 -- 172.21.15.97:0/4268806299 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c800f450 con 0x7f11d010f9a0 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.166+0000 7f11d6400640 1 -- 172.21.15.97:0/4268806299 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d01a2010 con 0x7f11d010f9a0 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.166+0000 7f11cd600640 1 -- 172.21.15.97:0/4268806299 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11c800fa80 con 0x7f11d010f9a0 2024-07-26T07:53:44.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.166+0000 7f11cd600640 1 -- 172.21.15.97:0/4268806299 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8018ad0 con 0x7f11d010f9a0 2024-07-26T07:53:44.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.166+0000 7f11d6400640 1 -- 172.21.15.97:0/4268806299 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d01a8830 con 0x7f11d010f9a0 2024-07-26T07:53:44.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.170+0000 7f11de66e740 1 Processor -- start 2024-07-26T07:53:44.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.170+0000 7f11cd600640 1 -- 172.21.15.97:0/4268806299 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f11c800f5f0 con 0x7f11d010f9a0 2024-07-26T07:53:44.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.170+0000 7f11de66e740 1 -- start start 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.170+0000 7f11de66e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 0x5566b5004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.170+0000 7f11de66e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b50046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.170+0000 7f11de66e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b5004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11de66e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5566b4f7af30 con 0x5566b4c36660 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11de66e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5566b4f6a430 con 0x5566b4c3e840 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11de66e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5566b4f6e470 con 0x5566b4c5f520 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11d5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b50046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11d5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b50046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48726/0 (socket says 172.21.15.97:48726) 2024-07-26T07:53:44.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11d5000640 1 -- 172.21.15.97:0/4255051280 learned_addr learned my addr 172.21.15.97:0/4255051280 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:44.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11cfe00640 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 0x5566b5004180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11cfe00640 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b5004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.171+0000 7f11d5000640 1 -- 172.21.15.97:0/4255051280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 msgr2=0x5566b5004180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11d5000640 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 0x5566b5004180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11d5000640 1 -- 172.21.15.97:0/4255051280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 msgr2=0x5566b5004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11d5000640 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b5004c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11d5000640 1 -- 172.21.15.97:0/4255051280 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566b4ccab00 con 0x5566b4c36660 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11d5000640 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b50046c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f11c400ac20 tx=0x7f11c400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ab96dac7ce1059 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11cd600640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11b80b3c00 0x7f11b80b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11c2000640 1 -- 172.21.15.97:0/4255051280 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c401d070 con 0x5566b4c36660 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11cd600640 1 -- 172.21.15.97:0/4268806299 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f11c80e4440 con 0x7f11d010f9a0 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11c2000640 1 -- 172.21.15.97:0/4255051280 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11c4018ad0 con 0x5566b4c36660 2024-07-26T07:53:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11c2000640 1 -- 172.21.15.97:0/4255051280 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c40109d0 con 0x5566b4c36660 2024-07-26T07:53:44.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.172+0000 7f11cfe00640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11b80b3c00 0x7f11b80b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 -- 172.21.15.97:0/4255051280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 msgr2=0x5566b50046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b50046c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f11c400ac20 tx=0x7f11c400f790 comp rx=0 tx=0).stop 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11cfe00640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11b80b3c00 0x7f11b80b6110 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f11bc02a910 tx=0x7f11bc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 -- 172.21.15.97:0/4255051280 shutdown_connections 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b5004c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b50046c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 --2- 172.21.15.97:0/4255051280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 0x5566b5004180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 -- 172.21.15.97:0/4255051280 >> 172.21.15.97:0/4255051280 conn(0x5566b4c53910 msgr2=0x5566b5002070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 -- 172.21.15.97:0/4255051280 shutdown_connections 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 -- 172.21.15.97:0/4255051280 wait complete. 2024-07-26T07:53:44.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.174+0000 7f11de66e740 1 -- start start 2024-07-26T07:53:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.175+0000 7f11de66e740 1 Processor -- start 2024-07-26T07:53:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.175+0000 7f11de66e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c36660 0x5566b500f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.175+0000 7f11de66e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b500fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.175+0000 7f11de66e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c5f520 0x5566b5010260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.175+0000 7f11de66e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5566b4dcdbb0 con 0x5566b4c5f520 2024-07-26T07:53:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.175+0000 7f11de66e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5566b4f06910 con 0x5566b4c36660 2024-07-26T07:53:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.175+0000 7f11de66e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5566b5012960 con 0x5566b4c3e840 2024-07-26T07:53:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.175+0000 7f11de66e740 10 client.0 _ll_register_callbacks cb 0x5566b4c76d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:44.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c5f520 0x5566b5010260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c5f520 0x5566b5010260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48738/0 (socket says 172.21.15.97:48738) 2024-07-26T07:53:44.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 -- 172.21.15.97:0/2917071472 learned_addr learned my addr 172.21.15.97:0/2917071472 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:44.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5000640 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c36660 0x5566b500f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11cfe00640 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b500fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 -- 172.21.15.97:0/2917071472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 msgr2=0x5566b500fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:44.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b500fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 -- 172.21.15.97:0/2917071472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c36660 msgr2=0x5566b500f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c36660 0x5566b500f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 -- 172.21.15.97:0/2917071472 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566b4d41960 con 0x5566b4c5f520 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11d5a00640 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c5f520 0x5566b5010260 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f11c800afd0 tx=0x7f11c80aab00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456fc63a44c04896 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.176+0000 7f11b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.177+0000 7f11b6a00640 1 -- 172.21.15.97:0/2917071472 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80c4020 con 0x5566b4c5f520 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.177+0000 7f11b6a00640 1 -- 172.21.15.97:0/2917071472 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f11c80b3ef0 con 0x5566b4c5f520 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.177+0000 7f11de66e740 1 -- 172.21.15.97:0/2917071472 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c4018ad0 con 0x5566b4c5f520 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.177+0000 7f11b6a00640 1 -- 172.21.15.97:0/2917071472 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11c80ab2d0 con 0x5566b4c5f520 2024-07-26T07:53:44.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.177+0000 7f11de66e740 10 client.15759 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.177+0000 7f11de66e740 1 -- 172.21.15.97:0/2917071472 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5566b501e670 con 0x5566b4c5f520 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.177+0000 7f11b6a00640 1 -- 172.21.15.97:0/2917071472 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11c4018ad0 con 0x5566b4c5f520 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.177+0000 7f11b6a00640 1 -- 172.21.15.97:0/2917071472 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f11c80a6dd0 con 0x5566b4c5f520 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.178+0000 7f11de66e740 10 client.15759 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.178+0000 7f11de66e740 10 client.15759 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.178+0000 7f11de66e740 20 client.15759 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.178+0000 7f11de66e740 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5566b4c1dea0 0x5566b5021c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.178+0000 7f11de66e740 4 client.15759 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.178+0000 7f11de66e740 1 -- 172.21.15.97:0/2917071472 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566b4f7ce80 con 0x5566b4c1dea0 2024-07-26T07:53:44.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.178+0000 7f11d5000640 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5566b4c1dea0 0x5566b5021c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.179+0000 7f11d5000640 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5566b4c1dea0 0x5566b5021c10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c833e7cfeea3cff9 server_cookie=e1ba11a3f6d37d6f in_seq=0 out_seq=0 2024-07-26T07:53:44.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.179+0000 7f11b6a00640 10 client.15759 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.181+0000 7f11b6a00640 1 -- 172.21.15.97:0/2917071472 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5566b4f7ce80 con 0x5566b4c1dea0 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.181+0000 7f11b6a00640 10 client.15759 handle_command_reply: tid=0 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.181+0000 7f11de66e740 1 client.15759 shutdown 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 -- 172.21.15.97:0/2917071472 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c5f520 msgr2=0x5566b5010260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c5f520 0x5566b5010260 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f11c800afd0 tx=0x7f11c80aab00 comp rx=0 tx=0).stop 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 -- 172.21.15.97:0/2917071472 shutdown_connections 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5566b4c1dea0 0x5566b5021c10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c5f520 0x5566b5010260 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b500fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 --2- 172.21.15.97:0/2917071472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c36660 0x5566b500f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 -- 172.21.15.97:0/2917071472 >> 172.21.15.97:0/2917071472 conn(0x5566b4c53910 msgr2=0x5566b5007310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:44.184 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11b6a00640 0 client.15759 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:44.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 -- 172.21.15.97:0/2917071472 shutdown_connections 2024-07-26T07:53:44.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 1 -- 172.21.15.97:0/2917071472 wait complete. 2024-07-26T07:53:44.185 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.182+0000 7f11de66e740 20 client.15759 trim_cache size 0 max 16384 2024-07-26T07:53:44.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.201+0000 7f11de66e740 1 Processor -- start 2024-07-26T07:53:44.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.202+0000 7f11de66e740 1 -- start start 2024-07-26T07:53:44.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:44 smithi097 ceph-mon[32231]: pgmap v568: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-07-26T07:53:44.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.202+0000 7f11de66e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b500afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.202+0000 7f11de66e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b50100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.202+0000 7f11de66e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 0x5566b5001620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.202+0000 7f11de66e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01b0df0 con 0x5566b4c36660 2024-07-26T07:53:44.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.202+0000 7f11de66e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01b2930 con 0x5566b4c3e840 2024-07-26T07:53:44.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.202+0000 7f11de66e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01b14c0 con 0x5566b4c5f520 2024-07-26T07:53:44.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.203+0000 7f11d5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b500afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.203+0000 7f11d5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b500afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48742/0 (socket says 172.21.15.97:48742) 2024-07-26T07:53:44.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.203+0000 7f11d5000640 1 -- 172.21.15.97:0/2528152167 learned_addr learned my addr 172.21.15.97:0/2528152167 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:44.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.203+0000 7f11cfe00640 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b50100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.203+0000 7f11d5a00640 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 0x5566b5001620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.204+0000 7f11cfe00640 1 -- 172.21.15.97:0/2528152167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 msgr2=0x5566b5001620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.204+0000 7f11cfe00640 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 0x5566b5001620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.204+0000 7f11cfe00640 1 -- 172.21.15.97:0/2528152167 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 msgr2=0x5566b500afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.204+0000 7f11cfe00640 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b500afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.204+0000 7f11cfe00640 1 -- 172.21.15.97:0/2528152167 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566b5006fe0 con 0x5566b4c3e840 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.204+0000 7f11d5000640 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b500afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.204+0000 7f11cfe00640 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b50100f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f11bc02a9e0 tx=0x7f11bc040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88309aabbd34cf50 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.205+0000 7f11c1600640 1 -- 172.21.15.97:0/2528152167 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc035070 con 0x5566b4c3e840 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.205+0000 7f11c1600640 1 -- 172.21.15.97:0/2528152167 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11bc03dae0 con 0x5566b4c3e840 2024-07-26T07:53:44.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.205+0000 7f11c1600640 1 -- 172.21.15.97:0/2528152167 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc048a00 con 0x5566b4c3e840 2024-07-26T07:53:44.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 -- 172.21.15.97:0/2528152167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 msgr2=0x5566b50100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b50100f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f11bc02a9e0 tx=0x7f11bc040000 comp rx=0 tx=0).stop 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 -- 172.21.15.97:0/2528152167 shutdown_connections 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c5f520 0x5566b5001620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c3e840 0x5566b50100f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 --2- 172.21.15.97:0/2528152167 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b500afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 -- 172.21.15.97:0/2528152167 >> 172.21.15.97:0/2528152167 conn(0x5566b4c53910 msgr2=0x5566b5005a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 -- 172.21.15.97:0/2528152167 shutdown_connections 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.206+0000 7f11de66e740 1 -- 172.21.15.97:0/2528152167 wait complete. 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.207+0000 7f11de66e740 1 -- start start 2024-07-26T07:53:44.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.207+0000 7f11de66e740 1 Processor -- start 2024-07-26T07:53:44.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.207+0000 7f11de66e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b5017f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.207+0000 7f11de66e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b50184c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.207+0000 7f11de66e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c5f520 0x5566b5013560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11de66e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01b2260 con 0x5566b4c36660 2024-07-26T07:53:44.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11de66e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01b1b90 con 0x5566b4c5f520 2024-07-26T07:53:44.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11de66e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f11d01aedd0 con 0x5566b4c3e840 2024-07-26T07:53:44.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11de66e740 10 client.0 _ll_register_callbacks cb 0x7f11c8018510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:44.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b5017f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b5017f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48758/0 (socket says 172.21.15.97:48758) 2024-07-26T07:53:44.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5000640 1 -- 172.21.15.97:0/939074175 learned_addr learned my addr 172.21.15.97:0/939074175 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:44.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11cfe00640 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b50184c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5000640 1 -- 172.21.15.97:0/939074175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 msgr2=0x5566b50184c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5000640 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b50184c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5000640 1 -- 172.21.15.97:0/939074175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c5f520 msgr2=0x5566b5013560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5a00640 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c5f520 0x5566b5013560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5000640 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c5f520 0x5566b5013560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11d5000640 1 -- 172.21.15.97:0/939074175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566b5013cd0 con 0x5566b4c36660 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.208+0000 7f11cfe00640 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b50184c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.209+0000 7f11d5000640 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b5017f00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f11c400ac20 tx=0x7f11c400aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25403f24a51a3d5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.209+0000 7f11b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.209+0000 7f11b6a00640 1 -- 172.21.15.97:0/939074175 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4031030 con 0x5566b4c36660 2024-07-26T07:53:44.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.209+0000 7f11b6a00640 1 -- 172.21.15.97:0/939074175 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f11c400a730 con 0x5566b4c36660 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.209+0000 7f11de66e740 1 -- 172.21.15.97:0/939074175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11bc03dae0 con 0x5566b4c36660 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11b6a00640 1 -- 172.21.15.97:0/939074175 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11c403b050 con 0x5566b4c36660 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11de66e740 10 client.15765 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11de66e740 1 -- 172.21.15.97:0/939074175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5566b5089fe0 con 0x5566b4c36660 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11b6a00640 1 -- 172.21.15.97:0/939074175 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11bc03dae0 con 0x5566b4c36660 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11b6a00640 1 -- 172.21.15.97:0/939074175 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f11c4018440 con 0x5566b4c36660 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11de66e740 10 client.15765 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11de66e740 10 client.15765 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11de66e740 20 client.15765 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11de66e740 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5566b4bf3250 0x5566b508c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11de66e740 4 client.15765 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9b1e79da", "release": true, "await": true}] 2024-07-26T07:53:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.210+0000 7f11de66e740 1 -- 172.21.15.97:0/939074175 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9b1e79da", "release": true, "await": true}) -- 0x5566b4f78920 con 0x5566b4bf3250 2024-07-26T07:53:44.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.211+0000 7f11cfe00640 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5566b4bf3250 0x5566b508c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:44.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.212+0000 7f11cfe00640 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5566b4bf3250 0x5566b508c570 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=332193bef65029f5 server_cookie=44f4fa54caeae741 in_seq=0 out_seq=0 2024-07-26T07:53:44.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.212+0000 7f11b6a00640 10 client.15765 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:44.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.229+0000 7f11b6a00640 1 -- 172.21.15.97:0/939074175 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5566b4f78920 con 0x5566b4bf3250 2024-07-26T07:53:44.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.229+0000 7f11b6a00640 10 client.15765 handle_command_reply: tid=0 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 client.15765 shutdown 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 -- 172.21.15.97:0/939074175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 msgr2=0x5566b5017f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b5017f00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f11c400ac20 tx=0x7f11c400aba0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 -- 172.21.15.97:0/939074175 shutdown_connections 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5566b4bf3250 0x5566b508c570 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566b4c5f520 0x5566b5013560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11b6a00640 0 client.15765 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5566b4c3e840 0x5566b50184c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 --2- 172.21.15.97:0/939074175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5566b4c36660 0x5566b5017f00 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.230+0000 7f11de66e740 1 -- 172.21.15.97:0/939074175 >> 172.21.15.97:0/939074175 conn(0x5566b4c53910 msgr2=0x5566b5010c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.231+0000 7f11de66e740 1 -- 172.21.15.97:0/939074175 shutdown_connections 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.231+0000 7f11de66e740 1 -- 172.21.15.97:0/939074175 wait complete. 2024-07-26T07:53:44.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.231+0000 7f11de66e740 20 client.15765 trim_cache size 0 max 16384 2024-07-26T07:53:44.233 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:53:44.233 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 28, 2024-07-26T07:53:44.233 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:53:44.233 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 8, 2024-07-26T07:53:44.233 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:53:44.233 INFO:teuthology.orchestra.run.smithi097.stdout: "9b1e79da": { 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 8, 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 142.6, 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:53:44.234 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:53:44.235 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 -- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11b80b3c00 msgr2=0x7f11b80b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11b80b3c00 0x7f11b80b6110 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f11bc02a910 tx=0x7f11bc009260 comp rx=0 tx=0).stop 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 -- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d010f9a0 msgr2=0x7f11d01a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d010f9a0 0x7f11d01a16b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f11c800cce0 tx=0x7f11c800a620 comp rx=0 tx=0).stop 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 -- 172.21.15.97:0/4268806299 shutdown_connections 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f11b80b3c00 0x7f11b80b6110 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d010f9a0 0x7f11d01a16b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11d0075ad0 0x7f11d019c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 --2- 172.21.15.97:0/4268806299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11d0075180 0x7f11d019be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:44.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 -- 172.21.15.97:0/4268806299 >> 172.21.15.97:0/4268806299 conn(0x7f11d00fee30 msgr2=0x7f11d0108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:44.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 -- 172.21.15.97:0/4268806299 shutdown_connections 2024-07-26T07:53:44.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:53:44.232+0000 7f11d6400640 1 -- 172.21.15.97:0/4268806299 wait complete. 2024-07-26T07:53:44.347 INFO:tasks.workunit.client.0.smithi097.stdout:checking for ANSI C header files... yes 2024-07-26T07:53:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:44 smithi115 ceph-mon[35364]: pgmap v568: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-07-26T07:53:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:44 smithi181 ceph-mon[35869]: pgmap v568: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-07-26T07:53:44.588 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-07-26T07:53:44.837 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sys/types.h... yes 2024-07-26T07:53:45.101 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sys/stat.h... yes 2024-07-26T07:53:45.124 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9b1e79da', total seconds elapsed: 30.3 2024-07-26T07:53:45.337 INFO:tasks.workunit.client.0.smithi097.stdout:checking for stdlib.h... yes 2024-07-26T07:53:45.613 INFO:tasks.workunit.client.0.smithi097.stdout:checking for string.h... yes 2024-07-26T07:53:45.892 INFO:tasks.workunit.client.0.smithi097.stdout:checking for memory.h... yes 2024-07-26T07:53:46.154 INFO:tasks.workunit.client.0.smithi097.stdout:checking for strings.h... yes 2024-07-26T07:53:46.373 INFO:tasks.workunit.client.0.smithi097.stdout:checking for inttypes.h... yes 2024-07-26T07:53:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:46 smithi115 ceph-mon[35364]: pgmap v569: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-07-26T07:53:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:46 smithi181 ceph-mon[35869]: pgmap v569: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-07-26T07:53:46.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:46 smithi097 ceph-mon[32231]: pgmap v569: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-07-26T07:53:46.632 INFO:tasks.workunit.client.0.smithi097.stdout:checking for stdint.h... yes 2024-07-26T07:53:46.846 INFO:tasks.workunit.client.0.smithi097.stdout:checking for unistd.h... yes 2024-07-26T07:53:46.905 INFO:tasks.workunit.client.0.smithi097.stdout:checking for memory.h... (cached) yes 2024-07-26T07:53:46.939 INFO:tasks.workunit.client.0.smithi097.stdout:checking for strings.h... (cached) yes 2024-07-26T07:53:46.970 INFO:tasks.workunit.client.0.smithi097.stdout:checking for unistd.h... (cached) yes 2024-07-26T07:53:47.182 INFO:tasks.workunit.client.0.smithi097.stdout:checking locale.h usability... yes 2024-07-26T07:53:47.283 INFO:tasks.workunit.client.0.smithi097.stdout:checking locale.h presence... yes 2024-07-26T07:53:47.289 INFO:tasks.workunit.client.0.smithi097.stdout:checking for locale.h... yes 2024-07-26T07:53:47.381 INFO:tasks.workunit.client.0.smithi097.stdout:checking for stdlib.h... (cached) yes 2024-07-26T07:53:47.412 INFO:tasks.workunit.client.0.smithi097.stdout:checking for string.h... (cached) yes 2024-07-26T07:53:47.617 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/socket.h usability... yes 2024-07-26T07:53:47.713 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/socket.h presence... yes 2024-07-26T07:53:47.716 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sys/socket.h... yes 2024-07-26T07:53:47.974 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/time.h usability... yes 2024-07-26T07:53:48.092 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/time.h presence... yes 2024-07-26T07:53:48.100 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sys/time.h... yes 2024-07-26T07:53:48.383 INFO:tasks.workunit.client.0.smithi097.stdout:checking fcntl.h usability... yes 2024-07-26T07:53:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:48 smithi115 ceph-mon[35364]: pgmap v570: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-07-26T07:53:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:48 smithi181 ceph-mon[35869]: pgmap v570: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-07-26T07:53:48.483 INFO:tasks.workunit.client.0.smithi097.stdout:checking fcntl.h presence... yes 2024-07-26T07:53:48.489 INFO:tasks.workunit.client.0.smithi097.stdout:checking for fcntl.h... yes 2024-07-26T07:53:48.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:48 smithi097 ceph-mon[32231]: pgmap v570: 97 pgs: 97 active+clean; 2.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-07-26T07:53:49.150 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/fcntl.h usability... yes 2024-07-26T07:53:49.246 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/fcntl.h presence... yes 2024-07-26T07:53:49.251 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sys/fcntl.h... yes 2024-07-26T07:53:49.579 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/param.h usability... yes 2024-07-26T07:53:49.681 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/param.h presence... yes 2024-07-26T07:53:49.687 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sys/param.h... yes 2024-07-26T07:53:49.946 INFO:tasks.workunit.client.0.smithi097.stdout:checking ioctl.h usability... no 2024-07-26T07:53:50.063 INFO:tasks.workunit.client.0.smithi097.stdout:checking ioctl.h presence... no 2024-07-26T07:53:50.070 INFO:tasks.workunit.client.0.smithi097.stdout:checking for ioctl.h... no 2024-07-26T07:53:50.354 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/ioctl.h usability... yes 2024-07-26T07:53:50.451 INFO:tasks.workunit.client.0.smithi097.stdout:checking sys/ioctl.h presence... yes 2024-07-26T07:53:50.455 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sys/ioctl.h... yes 2024-07-26T07:53:50.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:50 smithi115 ceph-mon[35364]: pgmap v571: 97 pgs: 97 active+clean; 2.7 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-07-26T07:53:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:50 smithi181 ceph-mon[35869]: pgmap v571: 97 pgs: 97 active+clean; 2.7 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-07-26T07:53:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:50 smithi097 ceph-mon[32231]: pgmap v571: 97 pgs: 97 active+clean; 2.7 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 49 KiB/s wr, 17 op/s 2024-07-26T07:53:50.713 INFO:tasks.workunit.client.0.smithi097.stdout:checking return type of signal handlers... void 2024-07-26T07:53:50.909 INFO:tasks.workunit.client.0.smithi097.stdout:checking for size_t... yes 2024-07-26T07:53:51.094 INFO:tasks.workunit.client.0.smithi097.stdout:checking for dev_t... yes 2024-07-26T07:53:51.277 INFO:tasks.workunit.client.0.smithi097.stdout:checking for short... yes 2024-07-26T07:53:51.660 INFO:tasks.workunit.client.0.smithi097.stdout:checking size of short... 2 2024-07-26T07:53:51.869 INFO:tasks.workunit.client.0.smithi097.stdout:checking for int... yes 2024-07-26T07:53:52.212 INFO:tasks.workunit.client.0.smithi097.stdout:checking size of int... 4 2024-07-26T07:53:52.458 INFO:tasks.workunit.client.0.smithi097.stdout:checking for long... yes 2024-07-26T07:53:52.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:52 smithi115 ceph-mon[35364]: pgmap v572: 97 pgs: 97 active+clean; 2.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 KiB/s wr, 31 op/s 2024-07-26T07:53:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:52 smithi181 ceph-mon[35869]: pgmap v572: 97 pgs: 97 active+clean; 2.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 KiB/s wr, 31 op/s 2024-07-26T07:53:52.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:52 smithi097 ceph-mon[32231]: pgmap v572: 97 pgs: 97 active+clean; 2.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 KiB/s wr, 31 op/s 2024-07-26T07:53:52.802 INFO:tasks.workunit.client.0.smithi097.stdout:checking size of long... 8 2024-07-26T07:53:53.019 INFO:tasks.workunit.client.0.smithi097.stdout:checking for long long... yes 2024-07-26T07:53:53.361 INFO:tasks.workunit.client.0.smithi097.stdout:checking size of long long... 8 2024-07-26T07:53:53.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:53 smithi097 ceph-mon[32231]: pgmap v573: 97 pgs: 97 active+clean; 2.7 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 62 KiB/s wr, 31 op/s 2024-07-26T07:53:53.568 INFO:tasks.workunit.client.0.smithi097.stdout:checking for ssize_t... yes 2024-07-26T07:53:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:53 smithi115 ceph-mon[35364]: pgmap v573: 97 pgs: 97 active+clean; 2.7 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 62 KiB/s wr, 31 op/s 2024-07-26T07:53:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:53 smithi181 ceph-mon[35869]: pgmap v573: 97 pgs: 97 active+clean; 2.7 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 62 KiB/s wr, 31 op/s 2024-07-26T07:53:53.741 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sig_atomic_t... yes 2024-07-26T07:53:53.933 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether gcc needs -traditional... no 2024-07-26T07:53:54.140 INFO:tasks.workunit.client.0.smithi097.stdout:checking for an ANSI C-conforming const... yes 2024-07-26T07:53:54.277 INFO:tasks.workunit.client.0.smithi097.stdout:checking for inline... inline 2024-07-26T07:53:54.613 INFO:tasks.workunit.client.0.smithi097.stdout:checking for working alloca.h... yes 2024-07-26T07:53:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 -- 172.21.15.97:0/2760846173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e24076aa0 msgr2=0x7f7e24075160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 --2- 172.21.15.97:0/2760846173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e24076aa0 0x7f7e24075160 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009900 tx=0x7f7e1402a410 comp rx=0 tx=0).stop 2024-07-26T07:53:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 -- 172.21.15.97:0/2760846173 shutdown_connections 2024-07-26T07:53:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 --2- 172.21.15.97:0/2760846173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e2410af20 0x7f7e2410d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 --2- 172.21.15.97:0/2760846173 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e240756a0 0x7f7e24075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 --2- 172.21.15.97:0/2760846173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e24076aa0 0x7f7e24075160 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 -- 172.21.15.97:0/2760846173 >> 172.21.15.97:0/2760846173 conn(0x7f7e240fe850 msgr2=0x7f7e24100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 -- 172.21.15.97:0/2760846173 shutdown_connections 2024-07-26T07:53:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.855+0000 7f7e23200640 1 -- 172.21.15.97:0/2760846173 wait complete. 2024-07-26T07:53:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.856+0000 7f7e23200640 1 Processor -- start 2024-07-26T07:53:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.856+0000 7f7e23200640 1 -- start start 2024-07-26T07:53:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e23200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e240756a0 0x7f7e2419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e23200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e24076aa0 0x7f7e2419bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e23200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2410af20 0x7f7e2419cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e23200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7e24110c70 con 0x7f7e2410af20 2024-07-26T07:53:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e23200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7e24110af0 con 0x7f7e24076aa0 2024-07-26T07:53:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e23200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e24110df0 con 0x7f7e240756a0 2024-07-26T07:53:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e22800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2410af20 0x7f7e2419cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e22800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2410af20 0x7f7e2419cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38866/0 (socket says 172.21.15.97:38866) 2024-07-26T07:53:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e22800640 1 -- 172.21.15.97:0/1222731676 learned_addr learned my addr 172.21.15.97:0/1222731676 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.857+0000 7f7e21e00640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e240756a0 0x7f7e2419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e22800640 1 -- 172.21.15.97:0/1222731676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e240756a0 msgr2=0x7f7e2419b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e22800640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e240756a0 0x7f7e2419b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e22800640 1 -- 172.21.15.97:0/1222731676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e24076aa0 msgr2=0x7f7e2419bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e22800640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e24076aa0 0x7f7e2419bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e22800640 1 -- 172.21.15.97:0/1222731676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2419d390 con 0x7f7e2410af20 2024-07-26T07:53:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e22800640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2410af20 0x7f7e2419cb70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00cab0 tx=0x7f7e1c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e1aa00640 1 -- 172.21.15.97:0/1222731676 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c00ae20 con 0x7f7e2410af20 2024-07-26T07:53:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e1aa00640 1 -- 172.21.15.97:0/1222731676 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e1c00f8e0 con 0x7f7e2410af20 2024-07-26T07:53:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e23200640 1 -- 172.21.15.97:0/1222731676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e2419d680 con 0x7f7e2410af20 2024-07-26T07:53:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.858+0000 7f7e1aa00640 1 -- 172.21.15.97:0/1222731676 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c018990 con 0x7f7e2410af20 2024-07-26T07:53:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.859+0000 7f7e23200640 1 -- 172.21.15.97:0/1222731676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e241a3d70 con 0x7f7e2410af20 2024-07-26T07:53:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.862+0000 7f7e30163740 1 Processor -- start 2024-07-26T07:53:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.863+0000 7f7e1aa00640 1 -- 172.21.15.97:0/1222731676 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f7e1c00f450 con 0x7f7e2410af20 2024-07-26T07:53:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.863+0000 7f7e30163740 1 -- start start 2024-07-26T07:53:54.864 INFO:tasks.workunit.client.0.smithi097.stdout:checking for alloca... yes 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e30163740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x5615749ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e30163740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x5615749ff730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e30163740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a00280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e30163740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561574971270 con 0x5615746591b0 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e30163740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561574960770 con 0x5615746384d0 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e30163740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5615749647b0 con 0x5615746302f0 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e21400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x5615749ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e21400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x5615749ff0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38882/0 (socket says 172.21.15.97:38882) 2024-07-26T07:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e21400640 1 -- 172.21.15.97:0/2110855411 learned_addr learned my addr 172.21.15.97:0/2110855411 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.864+0000 7f7e21e00640 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x5615749ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e21400640 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a00280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e21400640 1 -- 172.21.15.97:0/2110855411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 msgr2=0x5615749ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e21400640 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x5615749ff730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e21400640 1 -- 172.21.15.97:0/2110855411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 msgr2=0x561574a00280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e21400640 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a00280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e21400640 1 -- 172.21.15.97:0/2110855411 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615746c4790 con 0x5615746591b0 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e21e00640 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x5615749ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e21400640 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x5615749ff0c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000afd0 tx=0x7f7e1000a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5426f1a840ec7120 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.865+0000 7f7e1aa00640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7e080b3bd0 0x7f7e080b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.866+0000 7f7e0f400640 1 -- 172.21.15.97:0/2110855411 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100115f0 con 0x5615746591b0 2024-07-26T07:53:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.866+0000 7f7e0f400640 1 -- 172.21.15.97:0/2110855411 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e10011c20 con 0x5615746591b0 2024-07-26T07:53:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.866+0000 7f7e1aa00640 1 -- 172.21.15.97:0/1222731676 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f7e1c0e40f0 con 0x7f7e2410af20 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.866+0000 7f7e0f400640 1 -- 172.21.15.97:0/2110855411 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10012c80 con 0x5615746591b0 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.866+0000 7f7e21e00640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7e080b3bd0 0x7f7e080b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 -- 172.21.15.97:0/2110855411 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 msgr2=0x5615749ff0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x5615749ff0c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000afd0 tx=0x7f7e1000a420 comp rx=0 tx=0).stop 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e21e00640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7e080b3bd0 0x7f7e080b60e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e14006010 tx=0x7f7e14031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 -- 172.21.15.97:0/2110855411 shutdown_connections 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a00280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x5615749ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 --2- 172.21.15.97:0/2110855411 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x5615749ff0c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 -- 172.21.15.97:0/2110855411 >> 172.21.15.97:0/2110855411 conn(0x56157464d300 msgr2=0x5615749fab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 -- 172.21.15.97:0/2110855411 shutdown_connections 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 -- 172.21.15.97:0/2110855411 wait complete. 2024-07-26T07:53:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.867+0000 7f7e30163740 1 -- start start 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e30163740 1 Processor -- start 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e30163740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x561574a08c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e30163740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e30163740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x561574a09e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e30163740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56157489aab0 con 0x5615746591b0 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e30163740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5615748fdd50 con 0x5615746384d0 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e30163740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561574a00da0 con 0x5615746302f0 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e30163740 10 client.0 _ll_register_callbacks cb 0x561574670a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.868+0000 7f7e22800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x561574a09e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e22800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x561574a09e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38886/0 (socket says 172.21.15.97:38886) 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e22800640 1 -- 172.21.15.97:0/3439634133 learned_addr learned my addr 172.21.15.97:0/3439634133 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e21e00640 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x561574a08c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e21400640 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e22800640 1 -- 172.21.15.97:0/3439634133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 msgr2=0x561574a08c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e22800640 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x561574a08c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e22800640 1 -- 172.21.15.97:0/3439634133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 msgr2=0x561574a092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e22800640 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e22800640 1 -- 172.21.15.97:0/3439634133 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56157473d620 con 0x5615746591b0 2024-07-26T07:53:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.869+0000 7f7e21e00640 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x561574a08c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.870+0000 7f7e22800640 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x561574a09e60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0aac80 tx=0x7f7e1c0aabd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83bc40a569bdb4c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.870+0000 7f7e03e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.870+0000 7f7e03e00640 1 -- 172.21.15.97:0/3439634133 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0b4030 con 0x5615746591b0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.870+0000 7f7e03e00640 1 -- 172.21.15.97:0/3439634133 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f7e1c00fa90 con 0x5615746591b0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.870+0000 7f7e30163740 1 -- 172.21.15.97:0/3439634133 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e10011c20 con 0x5615746591b0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.870+0000 7f7e03e00640 1 -- 172.21.15.97:0/3439634133 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e1c070080 con 0x5615746591b0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.870+0000 7f7e30163740 10 client.15777 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.870+0000 7f7e30163740 1 -- 172.21.15.97:0/3439634133 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561574a12e20 con 0x5615746591b0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.871+0000 7f7e03e00640 1 -- 172.21.15.97:0/3439634133 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e10011c20 con 0x5615746591b0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.871+0000 7f7e03e00640 1 -- 172.21.15.97:0/3439634133 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7e1c020030 con 0x5615746591b0 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.871+0000 7f7e30163740 10 client.15777 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.871+0000 7f7e30163740 10 client.15777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.871+0000 7f7e30163740 20 client.15777 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.871+0000 7f7e30163740 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561574617d80 0x561574a165a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.871+0000 7f7e30163740 4 client.15777 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.871+0000 7f7e30163740 1 -- 172.21.15.97:0/3439634133 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615749731c0 con 0x561574617d80 2024-07-26T07:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.872+0000 7f7e21e00640 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561574617d80 0x561574a165a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.873+0000 7f7e21e00640 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561574617d80 0x561574a165a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b3cb2da6f12e6d8 server_cookie=a3ca806a7842537c in_seq=0 out_seq=0 2024-07-26T07:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.873+0000 7f7e03e00640 10 client.15777 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.877+0000 7f7e03e00640 1 -- 172.21.15.97:0/3439634133 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5615749731c0 con 0x561574617d80 2024-07-26T07:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e03e00640 10 client.15777 handle_command_reply: tid=0 2024-07-26T07:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 client.15777 shutdown 2024-07-26T07:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 -- 172.21.15.97:0/3439634133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 msgr2=0x561574a09e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x561574a09e60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0aac80 tx=0x7f7e1c0aabd0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 -- 172.21.15.97:0/3439634133 shutdown_connections 2024-07-26T07:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561574617d80 0x561574a165a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746591b0 0x561574a09e60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e03e00640 0 client.15777 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 --2- 172.21.15.97:0/3439634133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746302f0 0x561574a08c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 -- 172.21.15.97:0/3439634133 >> 172.21.15.97:0/3439634133 conn(0x56157464d300 msgr2=0x5615749f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 -- 172.21.15.97:0/3439634133 shutdown_connections 2024-07-26T07:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 1 -- 172.21.15.97:0/3439634133 wait complete. 2024-07-26T07:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.878+0000 7f7e30163740 20 client.15777 trim_cache size 0 max 16384 2024-07-26T07:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e30163740 1 Processor -- start 2024-07-26T07:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e30163740 1 -- start start 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e30163740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746302f0 0x5615749fffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e30163740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746384d0 0x561574a07e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e30163740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746591b0 0x561574a03b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e30163740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241acb20 con 0x5615746302f0 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e30163740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241ae660 con 0x5615746591b0 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e30163740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241ad1f0 con 0x5615746384d0 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.897+0000 7f7e21e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746302f0 0x5615749fffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e21e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746302f0 0x5615749fffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38902/0 (socket says 172.21.15.97:38902) 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e21e00640 1 -- 172.21.15.97:0/2071073901 learned_addr learned my addr 172.21.15.97:0/2071073901 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e21400640 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746384d0 0x561574a07e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e22800640 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746591b0 0x561574a03b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e21e00640 1 -- 172.21.15.97:0/2071073901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746384d0 msgr2=0x561574a07e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e21e00640 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746384d0 0x561574a07e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e21e00640 1 -- 172.21.15.97:0/2071073901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746591b0 msgr2=0x561574a03b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e21e00640 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746591b0 0x561574a03b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.898+0000 7f7e21e00640 1 -- 172.21.15.97:0/2071073901 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561574a00510 con 0x5615746302f0 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.899+0000 7f7e21e00640 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746302f0 0x5615749fffd0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7e14005760 tx=0x7f7e1403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b1c127593acff7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.899+0000 7f7e0ea00640 1 -- 172.21.15.97:0/2071073901 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14057070 con 0x5615746302f0 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.899+0000 7f7e0ea00640 1 -- 172.21.15.97:0/2071073901 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e14003bf0 con 0x5615746302f0 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.899+0000 7f7e0ea00640 1 -- 172.21.15.97:0/2071073901 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14035050 con 0x5615746302f0 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.900+0000 7f7e30163740 1 -- 172.21.15.97:0/2071073901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746302f0 msgr2=0x5615749fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.900+0000 7f7e30163740 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746302f0 0x5615749fffd0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7e14005760 tx=0x7f7e1403e000 comp rx=0 tx=0).stop 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.900+0000 7f7e30163740 1 -- 172.21.15.97:0/2071073901 shutdown_connections 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.900+0000 7f7e30163740 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746591b0 0x561574a03b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.900+0000 7f7e30163740 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746384d0 0x561574a07e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.900+0000 7f7e30163740 1 --2- 172.21.15.97:0/2071073901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615746302f0 0x5615749fffd0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.900+0000 7f7e30163740 1 -- 172.21.15.97:0/2071073901 >> 172.21.15.97:0/2071073901 conn(0x56157464d300 msgr2=0x5615749f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.901+0000 7f7e30163740 1 -- 172.21.15.97:0/2071073901 shutdown_connections 2024-07-26T07:53:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.901+0000 7f7e30163740 1 -- 172.21.15.97:0/2071073901 wait complete. 2024-07-26T07:53:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.901+0000 7f7e30163740 1 -- start start 2024-07-26T07:53:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.901+0000 7f7e30163740 1 Processor -- start 2024-07-26T07:53:54.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e30163740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a0dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e30163740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746591b0 0x561574a0e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e30163740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615745ed1c0 0x561574a0edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e30163740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241adf90 con 0x5615745ed1c0 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e30163740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241ad8c0 con 0x5615746384d0 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e30163740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241aab00 con 0x5615746591b0 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e30163740 10 client.0 _ll_register_callbacks cb 0x7f7e1c0a6ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e22800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615745ed1c0 0x561574a0edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e22800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615745ed1c0 0x561574a0edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38914/0 (socket says 172.21.15.97:38914) 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.902+0000 7f7e22800640 1 -- 172.21.15.97:0/1596909021 learned_addr learned my addr 172.21.15.97:0/1596909021 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e21e00640 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a0dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e21400640 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746591b0 0x561574a0e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e22800640 1 -- 172.21.15.97:0/1596909021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746591b0 msgr2=0x561574a0e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:54.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e22800640 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746591b0 0x561574a0e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e22800640 1 -- 172.21.15.97:0/1596909021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 msgr2=0x561574a0dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e22800640 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a0dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e22800640 1 -- 172.21.15.97:0/1596909021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561574a0f550 con 0x5615745ed1c0 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e22800640 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615745ed1c0 0x561574a0edf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0aaf10 tx=0x7f7e1c070450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62612a77a8d31a33 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e03e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e03e00640 1 -- 172.21.15.97:0/1596909021 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0aad50 con 0x5615745ed1c0 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e03e00640 1 -- 172.21.15.97:0/1596909021 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f7e1c0aa1e0 con 0x5615745ed1c0 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.903+0000 7f7e30163740 1 -- 172.21.15.97:0/1596909021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e14003bf0 con 0x5615745ed1c0 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e03e00640 1 -- 172.21.15.97:0/1596909021 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e1c0b4030 con 0x5615745ed1c0 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e30163740 10 client.15783 fetch_fsmap learned FSMap version 28 2024-07-26T07:53:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e30163740 1 -- 172.21.15.97:0/1596909021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561574a10870 con 0x5615745ed1c0 2024-07-26T07:53:54.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e03e00640 1 -- 172.21.15.97:0/1596909021 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e14003bf0 con 0x5615745ed1c0 2024-07-26T07:53:54.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e03e00640 1 -- 172.21.15.97:0/1596909021 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7e1c0c5ac0 con 0x5615745ed1c0 2024-07-26T07:53:54.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e30163740 10 client.15783 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:53:54.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e30163740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:53:54.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e30163740 20 client.15783 populate_metadata read hostname 'smithi097' 2024-07-26T07:53:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e30163740 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561574a7ecd0 0x561574a810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:53:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e30163740 4 client.15783 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:53:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.904+0000 7f7e30163740 1 -- 172.21.15.97:0/1596909021 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56157496ec60 con 0x561574a7ecd0 2024-07-26T07:53:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.905+0000 7f7e21e00640 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561574a7ecd0 0x561574a810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:53:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.906+0000 7f7e21e00640 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561574a7ecd0 0x561574a810f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cad7694add3fc25 server_cookie=dfb184a9003649cb in_seq=0 out_seq=0 2024-07-26T07:53:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.906+0000 7f7e03e00640 10 client.15783 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e03e00640 1 -- 172.21.15.97:0/1596909021 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56157496ec60 con 0x561574a7ecd0 2024-07-26T07:53:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e03e00640 10 client.15783 handle_command_reply: tid=0 2024-07-26T07:53:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 client.15783 shutdown 2024-07-26T07:53:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 -- 172.21.15.97:0/1596909021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615745ed1c0 msgr2=0x561574a0edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615745ed1c0 0x561574a0edf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0aaf10 tx=0x7f7e1c070450 comp rx=0 tx=0).stop 2024-07-26T07:53:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 -- 172.21.15.97:0/1596909021 shutdown_connections 2024-07-26T07:53:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561574a7ecd0 0x561574a810f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615745ed1c0 0x561574a0edf0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615746591b0 0x561574a0e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 --2- 172.21.15.97:0/1596909021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5615746384d0 0x561574a0dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e03e00640 0 client.15783 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:53:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.952+0000 7f7e30163740 1 -- 172.21.15.97:0/1596909021 >> 172.21.15.97:0/1596909021 conn(0x56157464d300 msgr2=0x5615749f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:54.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.953+0000 7f7e30163740 1 -- 172.21.15.97:0/1596909021 shutdown_connections 2024-07-26T07:53:54.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.953+0000 7f7e30163740 1 -- 172.21.15.97:0/1596909021 wait complete. 2024-07-26T07:53:54.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.953+0000 7f7e30163740 20 client.15783 trim_cache size 0 max 16384 2024-07-26T07:53:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:53:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:53:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:53:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:53:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 -- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7e080b3bd0 msgr2=0x7f7e080b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7e080b3bd0 0x7f7e080b60e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e14006010 tx=0x7f7e14031400 comp rx=0 tx=0).stop 2024-07-26T07:53:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 -- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2410af20 msgr2=0x7f7e2419cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:53:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2410af20 0x7f7e2419cb70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00cab0 tx=0x7f7e1c00cf70 comp rx=0 tx=0).stop 2024-07-26T07:53:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 -- 172.21.15.97:0/1222731676 shutdown_connections 2024-07-26T07:53:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f7e080b3bd0 0x7f7e080b60e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2410af20 0x7f7e2419cb70 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e24076aa0 0x7f7e2419bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 --2- 172.21.15.97:0/1222731676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e240756a0 0x7f7e2419b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:53:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 -- 172.21.15.97:0/1222731676 >> 172.21.15.97:0/1222731676 conn(0x7f7e240fe850 msgr2=0x7f7e24106f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:53:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 -- 172.21.15.97:0/1222731676 shutdown_connections 2024-07-26T07:53:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:53:54.954+0000 7f7e23200640 1 -- 172.21.15.97:0/1222731676 wait complete. 2024-07-26T07:53:55.126 INFO:tasks.workunit.client.0.smithi097.stdout:checking for working memcmp... yes 2024-07-26T07:53:55.463 INFO:tasks.workunit.client.0.smithi097.stdout:checking for strftime... yes 2024-07-26T07:53:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:55 smithi115 ceph-mon[35364]: pgmap v574: 97 pgs: 97 active+clean; 3.5 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 143 KiB/s wr, 50 op/s 2024-07-26T07:53:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:55 smithi181 ceph-mon[35869]: pgmap v574: 97 pgs: 97 active+clean; 3.5 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 143 KiB/s wr, 50 op/s 2024-07-26T07:53:55.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:55 smithi097 ceph-mon[32231]: pgmap v574: 97 pgs: 97 active+clean; 3.5 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 143 KiB/s wr, 50 op/s 2024-07-26T07:53:55.855 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-07-26T07:53:56.174 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether stat accepts an empty string... no 2024-07-26T07:53:56.504 INFO:tasks.workunit.client.0.smithi097.stdout:checking for vprintf... yes 2024-07-26T07:53:56.829 INFO:tasks.workunit.client.0.smithi097.stdout:checking for _doprnt... no 2024-07-26T07:53:56.936 INFO:tasks.workunit.client.0.smithi097.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-07-26T07:53:56.989 INFO:tasks.workunit.client.0.smithi097.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-07-26T07:53:57.194 INFO:tasks.workunit.client.0.smithi097.stdout:checking for the pthreads library -lpthreads... no 2024-07-26T07:53:57.462 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether pthreads work without any flags... yes 2024-07-26T07:53:57.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:57 smithi097 ceph-mon[32231]: pgmap v575: 97 pgs: 97 active+clean; 3.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 149 KiB/s wr, 57 op/s 2024-07-26T07:53:57.705 INFO:tasks.workunit.client.0.smithi097.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-07-26T07:53:57.706 INFO:tasks.workunit.client.0.smithi097.stdout:checking if more special flags are required for pthreads... no 2024-07-26T07:53:57.708 INFO:tasks.workunit.client.0.smithi097.stdout:checking for cc_r... gcc 2024-07-26T07:53:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:57 smithi115 ceph-mon[35364]: pgmap v575: 97 pgs: 97 active+clean; 3.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 149 KiB/s wr, 57 op/s 2024-07-26T07:53:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:57 smithi181 ceph-mon[35869]: pgmap v575: 97 pgs: 97 active+clean; 3.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 149 KiB/s wr, 57 op/s 2024-07-26T07:53:58.045 INFO:tasks.workunit.client.0.smithi097.stdout:checking for setrlimit... yes 2024-07-26T07:53:58.377 INFO:tasks.workunit.client.0.smithi097.stdout:checking for setproctitle... no 2024-07-26T07:53:58.709 INFO:tasks.workunit.client.0.smithi097.stdout:checking for getopt_long... yes 2024-07-26T07:53:59.058 INFO:tasks.workunit.client.0.smithi097.stdout:checking for strtoull... yes 2024-07-26T07:53:59.418 INFO:tasks.workunit.client.0.smithi097.stdout:checking for strtoq... yes 2024-07-26T07:53:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:53:59 smithi115 ceph-mon[35364]: pgmap v576: 97 pgs: 97 active+clean; 4.2 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 202 KiB/s wr, 60 op/s 2024-07-26T07:53:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:53:59 smithi181 ceph-mon[35869]: pgmap v576: 97 pgs: 97 active+clean; 4.2 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 202 KiB/s wr, 60 op/s 2024-07-26T07:53:59.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:53:59 smithi097 ceph-mon[32231]: pgmap v576: 97 pgs: 97 active+clean; 4.2 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 202 KiB/s wr, 60 op/s 2024-07-26T07:53:59.769 INFO:tasks.workunit.client.0.smithi097.stdout:checking for snprintf... yes 2024-07-26T07:54:00.124 INFO:tasks.workunit.client.0.smithi097.stdout:checking for vsnprintf... yes 2024-07-26T07:54:00.491 INFO:tasks.workunit.client.0.smithi097.stdout:checking for vfprintf... yes 2024-07-26T07:54:00.869 INFO:tasks.workunit.client.0.smithi097.stdout:checking for gethostname... yes 2024-07-26T07:54:01.240 INFO:tasks.workunit.client.0.smithi097.stdout:checking for setlocale... yes 2024-07-26T07:54:01.600 INFO:tasks.workunit.client.0.smithi097.stdout:checking for sync... yes 2024-07-26T07:54:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:01 smithi115 ceph-mon[35364]: pgmap v577: 97 pgs: 97 active+clean; 4.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 183 KiB/s wr, 85 op/s 2024-07-26T07:54:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:01 smithi181 ceph-mon[35869]: pgmap v577: 97 pgs: 97 active+clean; 4.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 183 KiB/s wr, 85 op/s 2024-07-26T07:54:01.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:01 smithi097 ceph-mon[32231]: pgmap v577: 97 pgs: 97 active+clean; 4.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 183 KiB/s wr, 85 op/s 2024-07-26T07:54:01.940 INFO:tasks.workunit.client.0.smithi097.stdout:checking whether snprintf is C99 conformant... done 2024-07-26T07:54:02.009 INFO:tasks.workunit.client.0.smithi097.stdout:configure: creating ./config.status 2024-07-26T07:54:02.934 INFO:tasks.workunit.client.0.smithi097.stdout:config.status: creating Makefile 2024-07-26T07:54:03.029 INFO:tasks.workunit.client.0.smithi097.stdout:config.status: creating src/Makefile 2024-07-26T07:54:03.123 INFO:tasks.workunit.client.0.smithi097.stdout:config.status: creating man/Makefile 2024-07-26T07:54:03.174 INFO:tasks.workunit.client.0.smithi097.stdout:config.status: creating config.h 2024-07-26T07:54:03.443 INFO:tasks.workunit.client.0.smithi097.stdout:config.status: executing depfiles commands 2024-07-26T07:54:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:03 smithi115 ceph-mon[35364]: pgmap v578: 97 pgs: 97 active+clean; 4.2 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 175 KiB/s wr, 74 op/s 2024-07-26T07:54:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:03 smithi181 ceph-mon[35869]: pgmap v578: 97 pgs: 97 active+clean; 4.2 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 175 KiB/s wr, 74 op/s 2024-07-26T07:54:03.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:03 smithi097 ceph-mon[32231]: pgmap v578: 97 pgs: 97 active+clean; 4.2 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 175 KiB/s wr, 74 op/s 2024-07-26T07:54:04.062 INFO:tasks.workunit.client.0.smithi097.stderr:+ make 2024-07-26T07:54:04.111 INFO:tasks.workunit.client.0.smithi097.stdout:make all-recursive 2024-07-26T07:54:04.125 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-07-26T07:54:04.135 INFO:tasks.workunit.client.0.smithi097.stdout:Making all in src 2024-07-26T07:54:04.192 INFO:tasks.workunit.client.0.smithi097.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-07-26T07:54:04.192 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-07-26T07:54:04.192 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-07-26T07:54:04.448 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-07-26T07:54:04.448 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-07-26T07:54:04.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:54:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:54:04.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:54:04.909 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-07-26T07:54:04.909 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-07-26T07:54:05.411 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-07-26T07:54:05.412 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-07-26T07:54:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:05 smithi115 ceph-mon[35364]: pgmap v579: 97 pgs: 97 active+clean; 5.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 270 KiB/s wr, 100 op/s 2024-07-26T07:54:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:54:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:54:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:54:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:05 smithi181 ceph-mon[35869]: pgmap v579: 97 pgs: 97 active+clean; 5.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 270 KiB/s wr, 100 op/s 2024-07-26T07:54:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:54:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:54:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:54:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:05 smithi097 ceph-mon[32231]: pgmap v579: 97 pgs: 97 active+clean; 5.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 270 KiB/s wr, 100 op/s 2024-07-26T07:54:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:54:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:54:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:54:05.992 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-07-26T07:54:05.993 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-07-26T07:54:06.596 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-07-26T07:54:06.596 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-07-26T07:54:07.193 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-07-26T07:54:07.193 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-07-26T07:54:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:07 smithi115 ceph-mon[35364]: pgmap v580: 97 pgs: 97 active+clean; 5.0 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 199 KiB/s wr, 89 op/s 2024-07-26T07:54:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:07 smithi181 ceph-mon[35869]: pgmap v580: 97 pgs: 97 active+clean; 5.0 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 199 KiB/s wr, 89 op/s 2024-07-26T07:54:07.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:07 smithi097 ceph-mon[32231]: pgmap v580: 97 pgs: 97 active+clean; 5.0 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 199 KiB/s wr, 89 op/s 2024-07-26T07:54:07.838 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-07-26T07:54:07.838 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-07-26T07:54:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 -- 172.21.15.97:0/1369072539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08106d20 msgr2=0x7fee0810d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 --2- 172.21.15.97:0/1369072539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08106d20 0x7fee0810d6c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fee00009900 tx=0x7fee0002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 -- 172.21.15.97:0/1369072539 shutdown_connections 2024-07-26T07:54:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 --2- 172.21.15.97:0/1369072539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08106d20 0x7fee0810d6c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 --2- 172.21.15.97:0/1369072539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fee08106380 0x7fee08106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 --2- 172.21.15.97:0/1369072539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee08105300 0x7fee08105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 -- 172.21.15.97:0/1369072539 >> 172.21.15.97:0/1369072539 conn(0x7fee08100c60 msgr2=0x7fee081030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:08.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 -- 172.21.15.97:0/1369072539 shutdown_connections 2024-07-26T07:54:08.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.174+0000 7fee0ce00640 1 -- 172.21.15.97:0/1369072539 wait complete. 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.175+0000 7fee0ce00640 1 Processor -- start 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.175+0000 7fee0ce00640 1 -- start start 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee0ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08105300 0x7fee0819ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee0ce00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee08106380 0x7fee0819c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee0ce00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fee08106d20 0x7fee0819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee0ce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fee08110e30 con 0x7fee08105300 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee0ce00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fee08110cb0 con 0x7fee08106d20 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee0ce00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fee08110fb0 con 0x7fee08106380 2024-07-26T07:54:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee05e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08105300 0x7fee0819ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee05e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08105300 0x7fee0819ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59984/0 (socket says 172.21.15.97:59984) 2024-07-26T07:54:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee05e00640 1 -- 172.21.15.97:0/3447543107 learned_addr learned my addr 172.21.15.97:0/3447543107 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee05400640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee08106380 0x7fee0819c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.176+0000 7fee06800640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fee08106d20 0x7fee0819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fee05e00640 1 -- 172.21.15.97:0/3447543107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee08106380 msgr2=0x7fee0819c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fee05e00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee08106380 0x7fee0819c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fee05e00640 1 -- 172.21.15.97:0/3447543107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fee08106d20 msgr2=0x7fee0819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fee05e00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fee08106d20 0x7fee0819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fee05e00640 1 -- 172.21.15.97:0/3447543107 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0819d460 con 0x7fee08105300 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fee05e00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08105300 0x7fee0819ba20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fedf800d950 tx=0x7fedf800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fedfea00640 1 -- 172.21.15.97:0/3447543107 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf8017070 con 0x7fee08105300 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fedfea00640 1 -- 172.21.15.97:0/3447543107 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fedf8009c30 con 0x7fee08105300 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fee0ce00640 1 -- 172.21.15.97:0/3447543107 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0819d750 con 0x7fee08105300 2024-07-26T07:54:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fedfea00640 1 -- 172.21.15.97:0/3447543107 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf800f620 con 0x7fee08105300 2024-07-26T07:54:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.177+0000 7fee0ce00640 1 -- 172.21.15.97:0/3447543107 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee081a3e40 con 0x7fee08105300 2024-07-26T07:54:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.182+0000 7fee13fe5740 1 Processor -- start 2024-07-26T07:54:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.182+0000 7fee13fe5740 1 -- start start 2024-07-26T07:54:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.182+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c341b0 0x55f9a9fd9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.182+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fda550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.182+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c134d0 0x55f9a9fdb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.182+0000 7fee13fe5740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55f9a9f4c270 con 0x55f9a9c134d0 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.182+0000 7fee13fe5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f9a9f3b770 con 0x55f9a9c0b2f0 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.182+0000 7fee13fe5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f9a9f3f7b0 con 0x55f9a9c341b0 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee05400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c134d0 0x55f9a9fdb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee05400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c134d0 0x55f9a9fdb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59990/0 (socket says 172.21.15.97:59990) 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee05400640 1 -- 172.21.15.97:0/1071007319 learned_addr learned my addr 172.21.15.97:0/1071007319 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fedfea00640 1 -- 172.21.15.97:0/3447543107 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fedf800f7c0 con 0x7fee08105300 2024-07-26T07:54:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee06800640 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c341b0 0x55f9a9fd9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee05400640 1 -- 172.21.15.97:0/1071007319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c341b0 msgr2=0x55f9a9fd9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee05400640 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c341b0 0x55f9a9fd9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee05400640 1 -- 172.21.15.97:0/1071007319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 msgr2=0x55f9a9fda550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee05400640 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fda550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee05400640 1 -- 172.21.15.97:0/1071007319 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9a9c9f790 con 0x55f9a9c134d0 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.183+0000 7fee06800640 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c341b0 0x55f9a9fd9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.184+0000 7fee05400640 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c134d0 0x55f9a9fdb0a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009f60 tx=0x7fedf4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eead7f554653562d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.184+0000 7fedf3400640 1 -- 172.21.15.97:0/1071007319 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4016070 con 0x55f9a9c134d0 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.184+0000 7fedf3400640 1 -- 172.21.15.97:0/1071007319 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fedf401baf0 con 0x55f9a9c134d0 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.184+0000 7fedf3400640 1 -- 172.21.15.97:0/1071007319 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4012b00 con 0x55f9a9c134d0 2024-07-26T07:54:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fedfea00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fedec0b3bd0 0x7fedec0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fee13fe5740 1 -- 172.21.15.97:0/1071007319 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c134d0 msgr2=0x55f9a9fdb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c134d0 0x55f9a9fdb0a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009f60 tx=0x7fedf4009f90 comp rx=0 tx=0).stop 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fedfea00640 1 -- 172.21.15.97:0/3447543107 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fedf80e3a00 con 0x7fee08105300 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fee06800640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fedec0b3bd0 0x7fedec0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fee13fe5740 1 -- 172.21.15.97:0/1071007319 shutdown_connections 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c134d0 0x55f9a9fdb0a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fda550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1071007319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c341b0 0x55f9a9fd9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.185+0000 7fee13fe5740 1 -- 172.21.15.97:0/1071007319 >> 172.21.15.97:0/1071007319 conn(0x55f9a9c28300 msgr2=0x55f9a9fd5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.186+0000 7fee13fe5740 1 -- 172.21.15.97:0/1071007319 shutdown_connections 2024-07-26T07:54:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.186+0000 7fee13fe5740 1 -- 172.21.15.97:0/1071007319 wait complete. 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.186+0000 7fee06800640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fedec0b3bd0 0x7fedec0b60e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fee00005760 tx=0x7fee00031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.186+0000 7fee13fe5740 1 -- start start 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee13fe5740 1 Processor -- start 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fe3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 0x55f9a9fe40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9fe4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee13fe5740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55f9a9e75ab0 con 0x55f9a9c341b0 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee13fe5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55f9a9ed8d50 con 0x55f9a9c0b2f0 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee13fe5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f9a9fdbbc0 con 0x55f9a9c134d0 2024-07-26T07:54:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee13fe5740 10 client.0 _ll_register_callbacks cb 0x55f9a9c4ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:54:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee06800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9fe4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee06800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9fe4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59994/0 (socket says 172.21.15.97:59994) 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.187+0000 7fee06800640 1 -- 172.21.15.97:0/1648750379 learned_addr learned my addr 172.21.15.97:0/1648750379 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fee05e00640 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fe3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fee05400640 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 0x55f9a9fe40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fee06800640 1 -- 172.21.15.97:0/1648750379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 msgr2=0x55f9a9fe40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fee06800640 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 0x55f9a9fe40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fee06800640 1 -- 172.21.15.97:0/1648750379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 msgr2=0x55f9a9fe3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fee06800640 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fe3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fee06800640 1 -- 172.21.15.97:0/1648750379 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9a9d18620 con 0x55f9a9c341b0 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fee06800640 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9fe4c50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fee00033000 tx=0x7fee00009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4af7e4a685d1581 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.188+0000 7fede7e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fede7e00640 1 -- 172.21.15.97:0/1648750379 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0004c030 con 0x55f9a9c341b0 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fede7e00640 1 -- 172.21.15.97:0/1648750379 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fee000035f0 con 0x55f9a9c341b0 2024-07-26T07:54:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fee13fe5740 1 -- 172.21.15.97:0/1648750379 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedf401baf0 con 0x55f9a9c341b0 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fede7e00640 1 -- 172.21.15.97:0/1648750379 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fee00035050 con 0x55f9a9c341b0 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fee13fe5740 10 client.15795 fetch_fsmap learned FSMap version 28 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fee13fe5740 1 -- 172.21.15.97:0/1648750379 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f9a9fedc10 con 0x55f9a9c341b0 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fede7e00640 1 -- 172.21.15.97:0/1648750379 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fedf401baf0 con 0x55f9a9c341b0 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fede7e00640 1 -- 172.21.15.97:0/1648750379 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fee000487b0 con 0x55f9a9c341b0 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.189+0000 7fee13fe5740 10 client.15795 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.190+0000 7fee13fe5740 10 client.15795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.190+0000 7fee13fe5740 20 client.15795 populate_metadata read hostname 'smithi097' 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.190+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55f9a9bf2d80 0x55f9a9ff13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.190+0000 7fee13fe5740 4 client.15795 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.190+0000 7fee13fe5740 1 -- 172.21.15.97:0/1648750379 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9a9f4e1c0 con 0x55f9a9bf2d80 2024-07-26T07:54:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.190+0000 7fee05e00640 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55f9a9bf2d80 0x55f9a9ff13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.191+0000 7fee05e00640 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55f9a9bf2d80 0x55f9a9ff13a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b21bed79c69ad727 server_cookie=d278f37a3b91794b in_seq=0 out_seq=0 2024-07-26T07:54:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.191+0000 7fede7e00640 10 client.15795 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.193+0000 7fede7e00640 1 -- 172.21.15.97:0/1648750379 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f9a9f4e1c0 con 0x55f9a9bf2d80 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.193+0000 7fede7e00640 10 client.15795 handle_command_reply: tid=0 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 client.15795 shutdown 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 -- 172.21.15.97:0/1648750379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 msgr2=0x55f9a9fe4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9fe4c50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fee00033000 tx=0x7fee00009260 comp rx=0 tx=0).stop 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 -- 172.21.15.97:0/1648750379 shutdown_connections 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55f9a9bf2d80 0x55f9a9ff13a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9fe4c50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 0x55f9a9fe40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 --2- 172.21.15.97:0/1648750379 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fe3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fede7e00640 0 client.15795 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 -- 172.21.15.97:0/1648750379 >> 172.21.15.97:0/1648750379 conn(0x55f9a9c28300 msgr2=0x55f9a9fd40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 -- 172.21.15.97:0/1648750379 shutdown_connections 2024-07-26T07:54:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 1 -- 172.21.15.97:0/1648750379 wait complete. 2024-07-26T07:54:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.194+0000 7fee13fe5740 20 client.15795 trim_cache size 0 max 16384 2024-07-26T07:54:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.215+0000 7fee13fe5740 1 Processor -- start 2024-07-26T07:54:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee13fe5740 1 -- start start 2024-07-26T07:54:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fe2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 0x55f9a9fe51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9ff5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee13fe5740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fee081accb0 con 0x55f9a9c341b0 2024-07-26T07:54:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee13fe5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fee081ae7f0 con 0x55f9a9c0b2f0 2024-07-26T07:54:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee13fe5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fee081ad380 con 0x55f9a9c134d0 2024-07-26T07:54:08.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee06800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9ff5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee06800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9ff5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60000/0 (socket says 172.21.15.97:60000) 2024-07-26T07:54:08.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.216+0000 7fee06800640 1 -- 172.21.15.97:0/116481425 learned_addr learned my addr 172.21.15.97:0/116481425 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fee05400640 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 0x55f9a9fe51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fee06800640 1 -- 172.21.15.97:0/116481425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 msgr2=0x55f9a9fe51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fee06800640 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 0x55f9a9fe51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fee05e00640 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fe2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fee06800640 1 -- 172.21.15.97:0/116481425 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 msgr2=0x55f9a9fe2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fee06800640 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fe2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fee06800640 1 -- 172.21.15.97:0/116481425 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9a9fd50c0 con 0x55f9a9c341b0 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fee06800640 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9ff5b90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fee0002a430 tx=0x7fee00032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8c4e43cc9d33a8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.217+0000 7fedf2a00640 1 -- 172.21.15.97:0/116481425 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0004c040 con 0x55f9a9c341b0 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.218+0000 7fedf2a00640 1 -- 172.21.15.97:0/116481425 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee0005f070 con 0x55f9a9c341b0 2024-07-26T07:54:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.218+0000 7fedf2a00640 1 -- 172.21.15.97:0/116481425 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00003e30 con 0x55f9a9c341b0 2024-07-26T07:54:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 -- 172.21.15.97:0/116481425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 msgr2=0x55f9a9ff5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9ff5b90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fee0002a430 tx=0x7fee00032f90 comp rx=0 tx=0).stop 2024-07-26T07:54:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 -- 172.21.15.97:0/116481425 shutdown_connections 2024-07-26T07:54:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9a9ff5b90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c134d0 0x55f9a9fe51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 --2- 172.21.15.97:0/116481425 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fe2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 -- 172.21.15.97:0/116481425 >> 172.21.15.97:0/116481425 conn(0x55f9a9c28300 msgr2=0x55f9a9fe1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 -- 172.21.15.97:0/116481425 shutdown_connections 2024-07-26T07:54:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 -- 172.21.15.97:0/116481425 wait complete. 2024-07-26T07:54:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.219+0000 7fee13fe5740 1 -- start start 2024-07-26T07:54:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee13fe5740 1 Processor -- start 2024-07-26T07:54:08.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fead50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c134d0 0x55f9a9feb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee13fe5740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9aa057b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee13fe5740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fee081ae120 con 0x55f9a9c341b0 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee13fe5740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fee081ada50 con 0x55f9a9c134d0 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee13fe5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fee081aac90 con 0x55f9a9c0b2f0 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee13fe5740 10 client.0 _ll_register_callbacks cb 0x7fee00009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee06800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9aa057b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee06800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9aa057b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60002/0 (socket says 172.21.15.97:60002) 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.220+0000 7fee06800640 1 -- 172.21.15.97:0/4104391605 learned_addr learned my addr 172.21.15.97:0/4104391605 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee05e00640 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee05400640 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c134d0 0x55f9a9feb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee06800640 1 -- 172.21.15.97:0/4104391605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c0b2f0 msgr2=0x55f9a9fead50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee06800640 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fead50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee06800640 1 -- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c134d0 msgr2=0x55f9a9feb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee06800640 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c134d0 0x55f9a9feb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee06800640 1 -- 172.21.15.97:0/4104391605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9aa058310 con 0x55f9a9c341b0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee05400640 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c134d0 0x55f9a9feb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee06800640 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9aa057b80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fee0002abe0 tx=0x7fee00009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f318ca77f4d12a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fede7e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fede7e00640 1 -- 172.21.15.97:0/4104391605 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0004c040 con 0x55f9a9c341b0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fede7e00640 1 -- 172.21.15.97:0/4104391605 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fee000037f0 con 0x55f9a9c341b0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.221+0000 7fee13fe5740 1 -- 172.21.15.97:0/4104391605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee0005f070 con 0x55f9a9c341b0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fede7e00640 1 -- 172.21.15.97:0/4104391605 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fee00048580 con 0x55f9a9c341b0 2024-07-26T07:54:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fee13fe5740 10 client.15801 fetch_fsmap learned FSMap version 28 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fee13fe5740 1 -- 172.21.15.97:0/4104391605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f9aa0595a0 con 0x55f9a9c341b0 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fede7e00640 1 -- 172.21.15.97:0/4104391605 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee0005f070 con 0x55f9a9c341b0 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fede7e00640 1 -- 172.21.15.97:0/4104391605 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fee00075020 con 0x55f9a9c341b0 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fee13fe5740 10 client.15801 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fee13fe5740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fee13fe5740 20 client.15801 populate_metadata read hostname 'smithi097' 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fee13fe5740 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55f9a9bc81c0 0x55f9aa05be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fee13fe5740 4 client.15801 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.222+0000 7fee13fe5740 1 -- 172.21.15.97:0/4104391605 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f9a9fdb520 con 0x55f9a9bc81c0 2024-07-26T07:54:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.223+0000 7fee05e00640 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55f9a9bc81c0 0x55f9aa05be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.224+0000 7fee05e00640 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55f9a9bc81c0 0x55f9aa05be80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74f5b02e082e3783 server_cookie=eb4d4cd092aaa944 in_seq=0 out_seq=0 2024-07-26T07:54:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.224+0000 7fede7e00640 10 client.15801 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.346+0000 7fede7e00640 1 -- 172.21.15.97:0/4104391605 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f9a9fdb520 con 0x55f9a9bc81c0 2024-07-26T07:54:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.346+0000 7fede7e00640 10 client.15801 handle_command_reply: tid=0 2024-07-26T07:54:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.346+0000 7fee13fe5740 1 client.15801 shutdown 2024-07-26T07:54:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 -- 172.21.15.97:0/4104391605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 msgr2=0x55f9aa057b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9aa057b80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fee0002abe0 tx=0x7fee00009620 comp rx=0 tx=0).stop 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 -- 172.21.15.97:0/4104391605 shutdown_connections 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55f9a9bc81c0 0x55f9aa05be80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f9a9c341b0 0x55f9aa057b80 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55f9a9c134d0 0x55f9a9feb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 --2- 172.21.15.97:0/4104391605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9a9c0b2f0 0x55f9a9fead50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fede7e00640 0 client.15801 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 -- 172.21.15.97:0/4104391605 >> 172.21.15.97:0/4104391605 conn(0x55f9a9c28300 msgr2=0x55f9a9fe1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.347+0000 7fee13fe5740 1 -- 172.21.15.97:0/4104391605 shutdown_connections 2024-07-26T07:54:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.348+0000 7fee13fe5740 1 -- 172.21.15.97:0/4104391605 wait complete. 2024-07-26T07:54:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.348+0000 7fee13fe5740 20 client.15801 trim_cache size 0 max 16384 2024-07-26T07:54:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:54:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:54:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:54:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 -- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fedec0b3bd0 msgr2=0x7fedec0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fedec0b3bd0 0x7fedec0b60e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fee00005760 tx=0x7fee00031400 comp rx=0 tx=0).stop 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 -- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08105300 msgr2=0x7fee0819ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08105300 0x7fee0819ba20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fedf800d950 tx=0x7fedf800de10 comp rx=0 tx=0).stop 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 -- 172.21.15.97:0/3447543107 shutdown_connections 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fedec0b3bd0 0x7fedec0b60e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fee08106d20 0x7fee0819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee08106380 0x7fee0819c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 --2- 172.21.15.97:0/3447543107 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee08105300 0x7fee0819ba20 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.349+0000 7fee0ce00640 1 -- 172.21.15.97:0/3447543107 >> 172.21.15.97:0/3447543107 conn(0x7fee08100c60 msgr2=0x7fee081096a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.350+0000 7fee0ce00640 1 -- 172.21.15.97:0/3447543107 shutdown_connections 2024-07-26T07:54:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:08.350+0000 7fee0ce00640 1 -- 172.21.15.97:0/3447543107 wait complete. 2024-07-26T07:54:08.376 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-07-26T07:54:08.376 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-07-26T07:54:08.912 INFO:tasks.workunit.client.0.smithi097.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-07-26T07:54:08.912 INFO:tasks.workunit.client.0.smithi097.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-07-26T07:54:09.292 INFO:tasks.workunit.client.0.smithi097.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-07-26T07:54:09.671 INFO:tasks.workunit.client.0.smithi097.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-07-26T07:54:09.675 INFO:tasks.workunit.client.0.smithi097.stdout:Making all in man 2024-07-26T07:54:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:09 smithi115 ceph-mon[35364]: pgmap v581: 97 pgs: 97 active+clean; 5.4 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 223 KiB/s wr, 82 op/s 2024-07-26T07:54:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:09 smithi181 ceph-mon[35869]: pgmap v581: 97 pgs: 97 active+clean; 5.4 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 223 KiB/s wr, 82 op/s 2024-07-26T07:54:09.742 INFO:tasks.workunit.client.0.smithi097.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-07-26T07:54:09.743 INFO:tasks.workunit.client.0.smithi097.stdout:make[2]: Nothing to be done for 'all'. 2024-07-26T07:54:09.743 INFO:tasks.workunit.client.0.smithi097.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-07-26T07:54:09.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:09 smithi097 ceph-mon[32231]: pgmap v581: 97 pgs: 97 active+clean; 5.4 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 223 KiB/s wr, 82 op/s 2024-07-26T07:54:09.787 INFO:tasks.workunit.client.0.smithi097.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-07-26T07:54:09.801 INFO:tasks.workunit.client.0.smithi097.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-07-26T07:54:09.805 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-07-26T07:54:09.809 INFO:tasks.workunit.client.0.smithi097.stderr:+ cd src 2024-07-26T07:54:09.817 INFO:tasks.workunit.client.0.smithi097.stderr:+ mkdir blogtest_in 2024-07-26T07:54:09.828 INFO:tasks.workunit.client.0.smithi097.stdout:running blogbench 2024-07-26T07:54:09.828 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'running blogbench' 2024-07-26T07:54:09.828 INFO:tasks.workunit.client.0.smithi097.stderr:+ ./blogbench -d blogtest_in 2024-07-26T07:54:09.832 INFO:tasks.workunit.client.0.smithi097.stdout: 2024-07-26T07:54:09.832 INFO:tasks.workunit.client.0.smithi097.stdout:Frequency = 10 secs 2024-07-26T07:54:09.832 INFO:tasks.workunit.client.0.smithi097.stdout:Scratch dir = [blogtest_in] 2024-07-26T07:54:10.192 INFO:tasks.workunit.client.0.smithi097.stdout:Spawning 3 writers... 2024-07-26T07:54:10.193 INFO:tasks.workunit.client.0.smithi097.stdout:Spawning 1 rewriters... 2024-07-26T07:54:10.194 INFO:tasks.workunit.client.0.smithi097.stdout:Spawning 5 commenters... 2024-07-26T07:54:10.194 INFO:tasks.workunit.client.0.smithi097.stdout:Spawning 100 readers... 2024-07-26T07:54:10.200 INFO:tasks.workunit.client.0.smithi097.stdout:Benchmarking for 30 iterations. 2024-07-26T07:54:10.200 INFO:tasks.workunit.client.0.smithi097.stdout:The test will run during 5 minutes. 2024-07-26T07:54:10.200 INFO:tasks.workunit.client.0.smithi097.stdout: 2024-07-26T07:54:10.200 INFO:tasks.workunit.client.0.smithi097.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-07-26T07:54:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:11 smithi115 ceph-mon[35364]: pgmap v582: 97 pgs: 97 active+clean; 5.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 185 KiB/s wr, 91 op/s 2024-07-26T07:54:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:11 smithi181 ceph-mon[35869]: pgmap v582: 97 pgs: 97 active+clean; 5.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 185 KiB/s wr, 91 op/s 2024-07-26T07:54:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:11 smithi097 ceph-mon[32231]: pgmap v582: 97 pgs: 97 active+clean; 5.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 185 KiB/s wr, 91 op/s 2024-07-26T07:54:13.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:13 smithi097 ceph-mon[32231]: pgmap v583: 97 pgs: 97 active+clean; 5.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 184 KiB/s wr, 66 op/s 2024-07-26T07:54:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:13 smithi115 ceph-mon[35364]: pgmap v583: 97 pgs: 97 active+clean; 5.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 184 KiB/s wr, 66 op/s 2024-07-26T07:54:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:13 smithi181 ceph-mon[35869]: pgmap v583: 97 pgs: 97 active+clean; 5.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 184 KiB/s wr, 66 op/s 2024-07-26T07:54:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:15 smithi115 ceph-mon[35364]: pgmap v584: 97 pgs: 97 active+clean; 5.7 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 539 KiB/s wr, 91 op/s 2024-07-26T07:54:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:15 smithi181 ceph-mon[35869]: pgmap v584: 97 pgs: 97 active+clean; 5.7 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 539 KiB/s wr, 91 op/s 2024-07-26T07:54:15.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:15 smithi097 ceph-mon[32231]: pgmap v584: 97 pgs: 97 active+clean; 5.7 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 539 KiB/s wr, 91 op/s 2024-07-26T07:54:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:17 smithi115 ceph-mon[35364]: pgmap v585: 97 pgs: 97 active+clean; 12 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 961 KiB/s wr, 84 op/s 2024-07-26T07:54:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:17 smithi181 ceph-mon[35869]: pgmap v585: 97 pgs: 97 active+clean; 12 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 961 KiB/s wr, 84 op/s 2024-07-26T07:54:17.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:17 smithi097 ceph-mon[32231]: pgmap v585: 97 pgs: 97 active+clean; 12 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 961 KiB/s wr, 84 op/s 2024-07-26T07:54:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:19 smithi115 ceph-mon[35364]: pgmap v586: 97 pgs: 97 active+clean; 12 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 986 KiB/s wr, 80 op/s 2024-07-26T07:54:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:19 smithi181 ceph-mon[35869]: pgmap v586: 97 pgs: 97 active+clean; 12 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 986 KiB/s wr, 80 op/s 2024-07-26T07:54:19.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:19 smithi097 ceph-mon[32231]: pgmap v586: 97 pgs: 97 active+clean; 12 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 986 KiB/s wr, 80 op/s 2024-07-26T07:54:20.200 INFO:tasks.workunit.client.0.smithi097.stdout: 4 7183 319 3540 193 3861 332 2024-07-26T07:54:21.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 -- 172.21.15.97:0/1093181293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104170 msgr2=0x7fcbdc104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 --2- 172.21.15.97:0/1093181293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104170 0x7fcbdc104580 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8009900 tx=0x7fcbc802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 -- 172.21.15.97:0/1093181293 shutdown_connections 2024-07-26T07:54:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 --2- 172.21.15.97:0/1093181293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbdc104b10 0x7fcbdc10b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 --2- 172.21.15.97:0/1093181293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104170 0x7fcbdc104580 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 --2- 172.21.15.97:0/1093181293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbdc1030f0 0x7fcbdc103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 -- 172.21.15.97:0/1093181293 >> 172.21.15.97:0/1093181293 conn(0x7fcbdc0fea10 msgr2=0x7fcbdc100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 -- 172.21.15.97:0/1093181293 shutdown_connections 2024-07-26T07:54:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.625+0000 7fcbe2800640 1 -- 172.21.15.97:0/1093181293 wait complete. 2024-07-26T07:54:21.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.627+0000 7fcbe2800640 1 Processor -- start 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.627+0000 7fcbe2800640 1 -- start start 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.627+0000 7fcbe2800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbdc1030f0 0x7fcbdc1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.627+0000 7fcbe2800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbdc104170 0x7fcbdc199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.627+0000 7fcbe2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104b10 0x7fcbdc19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.627+0000 7fcbe2800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc10ec70 con 0x7fcbdc104b10 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.627+0000 7fcbe2800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc10eaf0 con 0x7fcbdc104170 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.627+0000 7fcbe2800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc10edf0 con 0x7fcbdc1030f0 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104b10 0x7fcbdc19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104b10 0x7fcbdc19aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51936/0 (socket says 172.21.15.97:51936) 2024-07-26T07:54:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdbe00640 1 -- 172.21.15.97:0/3968618620 learned_addr learned my addr 172.21.15.97:0/3968618620 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdb400640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbdc1030f0 0x7fcbdc1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdbe00640 1 -- 172.21.15.97:0/3968618620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbdc1030f0 msgr2=0x7fcbdc1998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbdc1030f0 0x7fcbdc1998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdbe00640 1 -- 172.21.15.97:0/3968618620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbdc104170 msgr2=0x7fcbdc199f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbdc104170 0x7fcbdc199f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.628+0000 7fcbdbe00640 1 -- 172.21.15.97:0/3968618620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc19b260 con 0x7fcbdc104b10 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.629+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104b10 0x7fcbdc19aad0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400eb10 tx=0x7fcbd400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.629+0000 7fcbd3e00640 1 -- 172.21.15.97:0/3968618620 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400ce20 con 0x7fcbdc104b10 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.630+0000 7fcbd3e00640 1 -- 172.21.15.97:0/3968618620 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbd4005020 con 0x7fcbdc104b10 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.630+0000 7fcbe2800640 1 -- 172.21.15.97:0/3968618620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbdc19b550 con 0x7fcbdc104b10 2024-07-26T07:54:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.630+0000 7fcbe2800640 1 -- 172.21.15.97:0/3968618620 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbdc06c9f0 con 0x7fcbdc104b10 2024-07-26T07:54:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.631+0000 7fcbd3e00640 1 -- 172.21.15.97:0/3968618620 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4010690 con 0x7fcbdc104b10 2024-07-26T07:54:21.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.634+0000 7fcbe99a2740 1 Processor -- start 2024-07-26T07:54:21.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.635+0000 7fcbd3e00640 1 -- 172.21.15.97:0/3968618620 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fcbd4004a40 con 0x7fcbdc104b10 2024-07-26T07:54:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.635+0000 7fcbe99a2740 1 -- start start 2024-07-26T07:54:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.635+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee402201b0 0x55ee405ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.635+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401f72f0 0x55ee405ca960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.636+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401ff4d0 0x55ee405cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.636+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ee40538270 con 0x55ee401ff4d0 2024-07-26T07:54:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.636+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ee40527770 con 0x55ee402201b0 2024-07-26T07:54:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.636+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ee4052b7b0 con 0x55ee401f72f0 2024-07-26T07:54:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.636+0000 7fcbdaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401ff4d0 0x55ee405cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.636+0000 7fcbdaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401ff4d0 0x55ee405cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51950/0 (socket says 172.21.15.97:51950) 2024-07-26T07:54:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.636+0000 7fcbdaa00640 1 -- 172.21.15.97:0/667026514 learned_addr learned my addr 172.21.15.97:0/667026514 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.636+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee402201b0 0x55ee405ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.637+0000 7fcbdb400640 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401f72f0 0x55ee405ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.637+0000 7fcbdaa00640 1 -- 172.21.15.97:0/667026514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401f72f0 msgr2=0x55ee405ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.637+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401f72f0 0x55ee405ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.637+0000 7fcbdaa00640 1 -- 172.21.15.97:0/667026514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee402201b0 msgr2=0x55ee405ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.637+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee402201b0 0x55ee405ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.637+0000 7fcbdaa00640 1 -- 172.21.15.97:0/667026514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee4028b790 con 0x55ee401ff4d0 2024-07-26T07:54:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.637+0000 7fcbd3e00640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcbc00b3c50 0x7fcbc00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.637+0000 7fcbd3e00640 1 -- 172.21.15.97:0/3968618620 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fcbd40e42f0 con 0x7fcbdc104b10 2024-07-26T07:54:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.638+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401ff4d0 0x55ee405cb3c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8009230 tx=0x7fcbc8005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c70cb9610c94c3e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.638+0000 7fcbdb400640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcbc00b3c50 0x7fcbc00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.638+0000 7fcbd0c00640 1 -- 172.21.15.97:0/667026514 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8041070 con 0x55ee401ff4d0 2024-07-26T07:54:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.638+0000 7fcbd0c00640 1 -- 172.21.15.97:0/667026514 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbc8031c30 con 0x55ee401ff4d0 2024-07-26T07:54:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.639+0000 7fcbd0c00640 1 -- 172.21.15.97:0/667026514 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8035040 con 0x55ee401ff4d0 2024-07-26T07:54:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.640+0000 7fcbe99a2740 1 -- 172.21.15.97:0/667026514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401ff4d0 msgr2=0x55ee405cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.640+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401ff4d0 0x55ee405cb3c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8009230 tx=0x7fcbc8005720 comp rx=0 tx=0).stop 2024-07-26T07:54:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.640+0000 7fcbdb400640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcbc00b3c50 0x7fcbc00b6160 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00ac80 tx=0x7fcbcc00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.640+0000 7fcbe99a2740 1 -- 172.21.15.97:0/667026514 shutdown_connections 2024-07-26T07:54:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.640+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401ff4d0 0x55ee405cb3c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.640+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401f72f0 0x55ee405ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.640+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/667026514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee402201b0 0x55ee405ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.640+0000 7fcbe99a2740 1 -- 172.21.15.97:0/667026514 >> 172.21.15.97:0/667026514 conn(0x55ee40214300 msgr2=0x55ee405c1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.641+0000 7fcbe99a2740 1 -- 172.21.15.97:0/667026514 shutdown_connections 2024-07-26T07:54:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.641+0000 7fcbe99a2740 1 -- 172.21.15.97:0/667026514 wait complete. 2024-07-26T07:54:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.641+0000 7fcbe99a2740 1 -- start start 2024-07-26T07:54:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.642+0000 7fcbe99a2740 1 Processor -- start 2024-07-26T07:54:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.642+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405d81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.642+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 0x55ee405d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405d93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ee40461ab0 con 0x55ee402201b0 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55ee404c4d50 con 0x55ee401f72f0 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ee405cbba0 con 0x55ee401ff4d0 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbe99a2740 10 client.0 _ll_register_callbacks cb 0x55ee40237a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbdbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405d93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbdbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405d93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51956/0 (socket says 172.21.15.97:51956) 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbdbe00640 1 -- 172.21.15.97:0/2578549448 learned_addr learned my addr 172.21.15.97:0/2578549448 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.643+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 0x55ee405d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbdbe00640 1 -- 172.21.15.97:0/2578549448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 msgr2=0x55ee405d8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 0x55ee405d8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbdbe00640 1 -- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 msgr2=0x55ee405d81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbdb400640 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405d81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405d81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbdbe00640 1 -- 172.21.15.97:0/2578549448 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee40304620 con 0x55ee402201b0 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbdb400640 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405d81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405d93d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40a83f0 tx=0x7fcbd40ab880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a0f6285681fd5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbc5600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbc5600640 1 -- 172.21.15.97:0/2578549448 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40c4070 con 0x55ee402201b0 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbc5600640 1 -- 172.21.15.97:0/2578549448 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fcbd40051d0 con 0x55ee402201b0 2024-07-26T07:54:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.644+0000 7fcbe99a2740 1 -- 172.21.15.97:0/2578549448 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbc8031c30 con 0x55ee402201b0 2024-07-26T07:54:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.645+0000 7fcbc5600640 1 -- 172.21.15.97:0/2578549448 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbd40af780 con 0x55ee402201b0 2024-07-26T07:54:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.645+0000 7fcbe99a2740 10 client.15813 fetch_fsmap learned FSMap version 28 2024-07-26T07:54:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.645+0000 7fcbe99a2740 1 -- 172.21.15.97:0/2578549448 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ee405e2390 con 0x55ee402201b0 2024-07-26T07:54:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.645+0000 7fcbc5600640 1 -- 172.21.15.97:0/2578549448 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbc8031c30 con 0x55ee402201b0 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.645+0000 7fcbc5600640 1 -- 172.21.15.97:0/2578549448 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcbd40cc2a0 con 0x55ee402201b0 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.646+0000 7fcbe99a2740 10 client.15813 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.646+0000 7fcbe99a2740 10 client.15813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.646+0000 7fcbe99a2740 20 client.15813 populate_metadata read hostname 'smithi097' 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.646+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55ee401ded80 0x55ee405e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.646+0000 7fcbe99a2740 4 client.15813 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.646+0000 7fcbe99a2740 1 -- 172.21.15.97:0/2578549448 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee4053a1c0 con 0x55ee401ded80 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.647+0000 7fcbdb400640 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55ee401ded80 0x55ee405e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.648+0000 7fcbdb400640 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55ee401ded80 0x55ee405e5bf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=246e59207f820d90 server_cookie=2eb894a5f56d209 in_seq=0 out_seq=0 2024-07-26T07:54:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.648+0000 7fcbc5600640 10 client.15813 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.650+0000 7fcbc5600640 1 -- 172.21.15.97:0/2578549448 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ee4053a1c0 con 0x55ee401ded80 2024-07-26T07:54:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.650+0000 7fcbc5600640 10 client.15813 handle_command_reply: tid=0 2024-07-26T07:54:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.650+0000 7fcbe99a2740 1 client.15813 shutdown 2024-07-26T07:54:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 -- 172.21.15.97:0/2578549448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 msgr2=0x55ee405d93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405d93d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40a83f0 tx=0x7fcbd40ab880 comp rx=0 tx=0).stop 2024-07-26T07:54:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 -- 172.21.15.97:0/2578549448 shutdown_connections 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55ee401ded80 0x55ee405e5bf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405d93d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 0x55ee405d8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbc5600640 0 client.15813 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/2578549448 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405d81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 -- 172.21.15.97:0/2578549448 >> 172.21.15.97:0/2578549448 conn(0x55ee40214300 msgr2=0x55ee405c00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 -- 172.21.15.97:0/2578549448 shutdown_connections 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.651+0000 7fcbe99a2740 1 -- 172.21.15.97:0/2578549448 wait complete. 2024-07-26T07:54:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.652+0000 7fcbe99a2740 20 client.15813 trim_cache size 0 max 16384 2024-07-26T07:54:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.678+0000 7fcbe99a2740 1 Processor -- start 2024-07-26T07:54:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.678+0000 7fcbe99a2740 1 -- start start 2024-07-26T07:54:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.679+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.679+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 0x55ee405c8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.679+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405c8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.679+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc075700 con 0x55ee402201b0 2024-07-26T07:54:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.679+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc1b2b50 con 0x55ee401f72f0 2024-07-26T07:54:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.679+0000 7fcbe99a2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc075dd0 con 0x55ee401ff4d0 2024-07-26T07:54:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.680+0000 7fcbdbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.680+0000 7fcbdbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51970/0 (socket says 172.21.15.97:51970) 2024-07-26T07:54:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.680+0000 7fcbdbe00640 1 -- 172.21.15.97:0/4119374317 learned_addr learned my addr 172.21.15.97:0/4119374317 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.680+0000 7fcbdb400640 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.680+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 0x55ee405c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.681+0000 7fcbdbe00640 1 -- 172.21.15.97:0/4119374317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 msgr2=0x55ee405c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.681+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 0x55ee405c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.681+0000 7fcbdbe00640 1 -- 172.21.15.97:0/4119374317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 msgr2=0x55ee405c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.681+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.681+0000 7fcbdbe00640 1 -- 172.21.15.97:0/4119374317 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee405d8cc0 con 0x55ee402201b0 2024-07-26T07:54:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.682+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405c8560 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40a7200 tx=0x7fcbd40aab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd9881e3d8a072 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.682+0000 7fcbc7e00640 1 -- 172.21.15.97:0/4119374317 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40c4070 con 0x55ee402201b0 2024-07-26T07:54:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.682+0000 7fcbc7e00640 1 -- 172.21.15.97:0/4119374317 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbd40b4230 con 0x55ee402201b0 2024-07-26T07:54:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.682+0000 7fcbc7e00640 1 -- 172.21.15.97:0/4119374317 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40a7410 con 0x55ee402201b0 2024-07-26T07:54:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.684+0000 7fcbe99a2740 1 -- 172.21.15.97:0/4119374317 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 msgr2=0x55ee405c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.684+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405c8560 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40a7200 tx=0x7fcbd40aab60 comp rx=0 tx=0).stop 2024-07-26T07:54:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.684+0000 7fcbe99a2740 1 -- 172.21.15.97:0/4119374317 shutdown_connections 2024-07-26T07:54:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.684+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee402201b0 0x55ee405c8560 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.684+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee401ff4d0 0x55ee405c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.684+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/4119374317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401f72f0 0x55ee405c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.684+0000 7fcbe99a2740 1 -- 172.21.15.97:0/4119374317 >> 172.21.15.97:0/4119374317 conn(0x55ee40214300 msgr2=0x55ee405d6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.685+0000 7fcbe99a2740 1 -- 172.21.15.97:0/4119374317 shutdown_connections 2024-07-26T07:54:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.685+0000 7fcbe99a2740 1 -- 172.21.15.97:0/4119374317 wait complete. 2024-07-26T07:54:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.685+0000 7fcbe99a2740 1 -- start start 2024-07-26T07:54:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.686+0000 7fcbe99a2740 1 Processor -- start 2024-07-26T07:54:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.687+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401f72f0 0x55ee405dfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.687+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401ff4d0 0x55ee405e0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.687+0000 7fcbe99a2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee402201b0 0x55ee406466f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.687+0000 7fcbdb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401f72f0 0x55ee405dfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.687+0000 7fcbdb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401f72f0 0x55ee405dfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51978/0 (socket says 172.21.15.97:51978) 2024-07-26T07:54:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.688+0000 7fcbdb400640 1 -- 172.21.15.97:0/1561373949 learned_addr learned my addr 172.21.15.97:0/1561373949 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.688+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401ff4d0 0x55ee405e0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.688+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc076b70 con 0x55ee401f72f0 2024-07-26T07:54:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.688+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc0764a0 con 0x55ee401ff4d0 2024-07-26T07:54:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.688+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc0736e0 con 0x55ee402201b0 2024-07-26T07:54:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.688+0000 7fcbe99a2740 10 client.0 _ll_register_callbacks cb 0x7fcbd40aa960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:54:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.688+0000 7fcbdbe00640 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee402201b0 0x55ee406466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.689+0000 7fcbdb400640 1 -- 172.21.15.97:0/1561373949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee402201b0 msgr2=0x55ee406466f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.689+0000 7fcbdb400640 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee402201b0 0x55ee406466f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.689+0000 7fcbdb400640 1 -- 172.21.15.97:0/1561373949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401ff4d0 msgr2=0x55ee405e0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.689+0000 7fcbdb400640 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401ff4d0 0x55ee405e0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.689+0000 7fcbdb400640 1 -- 172.21.15.97:0/1561373949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee40646e80 con 0x55ee401f72f0 2024-07-26T07:54:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.690+0000 7fcbdb400640 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401f72f0 0x55ee405dfa50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc01e560 tx=0x7fcbcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfca55127a07e0c4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.690+0000 7fcbc5600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:54:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.690+0000 7fcbc5600640 1 -- 172.21.15.97:0/1561373949 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc010e20 con 0x55ee401f72f0 2024-07-26T07:54:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.690+0000 7fcbc5600640 1 -- 172.21.15.97:0/1561373949 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fcbcc01e020 con 0x55ee401f72f0 2024-07-26T07:54:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.691+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbd40b4230 con 0x55ee401f72f0 2024-07-26T07:54:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.691+0000 7fcbc5600640 1 -- 172.21.15.97:0/1561373949 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbcc01e220 con 0x55ee401f72f0 2024-07-26T07:54:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.691+0000 7fcbe99a2740 10 client.15819 fetch_fsmap learned FSMap version 28 2024-07-26T07:54:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.691+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ee4064e2a0 con 0x55ee401f72f0 2024-07-26T07:54:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.692+0000 7fcbc5600640 1 -- 172.21.15.97:0/1561373949 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbd40b4230 con 0x55ee401f72f0 2024-07-26T07:54:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.693+0000 7fcbc5600640 1 -- 172.21.15.97:0/1561373949 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcbcc004ac0 con 0x55ee401f72f0 2024-07-26T07:54:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.693+0000 7fcbe99a2740 10 client.15819 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T07:54:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.693+0000 7fcbe99a2740 10 client.15819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:54:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.693+0000 7fcbe99a2740 20 client.15819 populate_metadata read hostname 'smithi097' 2024-07-26T07:54:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.693+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55ee401b41c0 0x55ee40650c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.693+0000 7fcbe99a2740 4 client.15819 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:54:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.693+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee40535c60 con 0x55ee401b41c0 2024-07-26T07:54:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.694+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55ee401b41c0 0x55ee40650c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.695+0000 7fcbc5600640 10 client.15819 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:21.695+0000 7fcbdaa00640 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55ee401b41c0 0x55ee40650c30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53644851ec0039d7 server_cookie=105bc5cdb0bfa855 in_seq=0 out_seq=0 2024-07-26T07:54:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:21 smithi115 ceph-mon[35364]: pgmap v587: 97 pgs: 97 active+clean; 18 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-07-26T07:54:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:21 smithi181 ceph-mon[35869]: pgmap v587: 97 pgs: 97 active+clean; 18 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-07-26T07:54:21.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:21 smithi097 ceph-mon[32231]: pgmap v587: 97 pgs: 97 active+clean; 18 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-07-26T07:54:23.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:23 smithi115 ceph-mon[35364]: pgmap v588: 97 pgs: 97 active+clean; 19 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.6 MiB/s wr, 106 op/s 2024-07-26T07:54:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:23 smithi181 ceph-mon[35869]: pgmap v588: 97 pgs: 97 active+clean; 19 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.6 MiB/s wr, 106 op/s 2024-07-26T07:54:23.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:23 smithi097 ceph-mon[32231]: pgmap v588: 97 pgs: 97 active+clean; 19 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.6 MiB/s wr, 106 op/s 2024-07-26T07:54:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.797+0000 7fcbc5600640 1 -- 172.21.15.97:0/1561373949 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee40535c60 con 0x55ee401b41c0 2024-07-26T07:54:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.798+0000 7fcbc5600640 10 client.15819 handle_command_reply: tid=0 2024-07-26T07:54:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.798+0000 7fcbe99a2740 1 client.15819 shutdown 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.798+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401f72f0 msgr2=0x55ee405dfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.798+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401f72f0 0x55ee405dfa50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc01e560 tx=0x7fcbcc0046c0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 shutdown_connections 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55ee401b41c0 0x55ee40650c30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ee402201b0 0x55ee406466f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbc5600640 0 client.15819 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ee401ff4d0 0x55ee405e0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 1 --2- 172.21.15.97:0/1561373949 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ee401f72f0 0x55ee405dfa50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 >> 172.21.15.97:0/1561373949 conn(0x55ee40214300 msgr2=0x55ee405c7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 shutdown_connections 2024-07-26T07:54:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 1 -- 172.21.15.97:0/1561373949 wait complete. 2024-07-26T07:54:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.799+0000 7fcbe99a2740 20 client.15819 trim_cache size 0 max 16384 2024-07-26T07:54:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:54:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:54:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:54:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:54:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.800+0000 7fcbe2800640 1 -- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcbc00b3c50 msgr2=0x7fcbc00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.800+0000 7fcbe2800640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcbc00b3c50 0x7fcbc00b6160 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00ac80 tx=0x7fcbcc00a2d0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.800+0000 7fcbe2800640 1 -- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104b10 msgr2=0x7fcbdc19aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.800+0000 7fcbe2800640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104b10 0x7fcbdc19aad0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400eb10 tx=0x7fcbd400ef10 comp rx=0 tx=0).stop 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.801+0000 7fcbe2800640 1 -- 172.21.15.97:0/3968618620 shutdown_connections 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.801+0000 7fcbe2800640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcbc00b3c50 0x7fcbc00b6160 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.801+0000 7fcbe2800640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcbdc104b10 0x7fcbdc19aad0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.801+0000 7fcbe2800640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbdc104170 0x7fcbdc199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.801+0000 7fcbe2800640 1 --2- 172.21.15.97:0/3968618620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbdc1030f0 0x7fcbdc1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.801+0000 7fcbe2800640 1 -- 172.21.15.97:0/3968618620 >> 172.21.15.97:0/3968618620 conn(0x7fcbdc0fea10 msgr2=0x7fcbdc107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.801+0000 7fcbe2800640 1 -- 172.21.15.97:0/3968618620 shutdown_connections 2024-07-26T07:54:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:23.801+0000 7fcbe2800640 1 -- 172.21.15.97:0/3968618620 wait complete. 2024-07-26T07:54:24.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:24 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:54:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:24 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:54:24.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:24 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:54:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:25 smithi115 ceph-mon[35364]: pgmap v589: 97 pgs: 97 active+clean; 24 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 188 op/s 2024-07-26T07:54:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:25 smithi181 ceph-mon[35869]: pgmap v589: 97 pgs: 97 active+clean; 24 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 188 op/s 2024-07-26T07:54:25.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:25 smithi097 ceph-mon[32231]: pgmap v589: 97 pgs: 97 active+clean; 24 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 2.0 MiB/s wr, 188 op/s 2024-07-26T07:54:27.232 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:26 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[65903]: *** Caught signal (Segmentation fault) ** 2024-07-26T07:54:27.233 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:26 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[65903]: in thread 7f17d9200640 thread_name:MR_Finisher 2024-07-26T07:54:27.233 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:27 smithi181 podman[68785]: 2024-07-26 07:54:27.051437038 +0000 UTC m=+0.037933153 container died 8970b3366a51d2a658e0785f82bacc26d588b2301f13631b09b0d80fea2c206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS) 2024-07-26T07:54:27.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:27 smithi181 ceph-mon[35869]: pgmap v590: 97 pgs: 97 active+clean; 23 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 2.0 MiB/s wr, 215 op/s 2024-07-26T07:54:27.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:27 smithi097 ceph-mon[32231]: pgmap v590: 97 pgs: 97 active+clean; 23 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 2.0 MiB/s wr, 215 op/s 2024-07-26T07:54:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:27 smithi115 ceph-mon[35364]: pgmap v590: 97 pgs: 97 active+clean; 23 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 2.0 MiB/s wr, 215 op/s 2024-07-26T07:54:28.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:27 smithi181 podman[68785]: 2024-07-26 07:54:27.983776794 +0000 UTC m=+0.970272943 container remove 8970b3366a51d2a658e0785f82bacc26d588b2301f13631b09b0d80fea2c206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-07-26T07:54:28.536 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:27 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-07-26T07:54:28.982 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:28 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T07:54:28.982 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:28 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.c.service: Consumed 4.818s CPU time. 2024-07-26T07:54:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:29 smithi181 ceph-mon[35869]: pgmap v591: 97 pgs: 97 active+clean; 24 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-07-26T07:54:29.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:29 smithi097 ceph-mon[32231]: pgmap v591: 97 pgs: 97 active+clean; 24 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-07-26T07:54:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:29 smithi115 ceph-mon[35364]: pgmap v591: 97 pgs: 97 active+clean; 24 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-07-26T07:54:30.201 INFO:tasks.workunit.client.0.smithi097.stdout: 8 12696 169 5227 215 1337 275 2024-07-26T07:54:31.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:31 smithi097 ceph-mon[32231]: pgmap v592: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.1 MiB/s wr, 254 op/s 2024-07-26T07:54:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:31 smithi115 ceph-mon[35364]: pgmap v592: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.1 MiB/s wr, 254 op/s 2024-07-26T07:54:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:31 smithi181 ceph-mon[35869]: pgmap v592: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.1 MiB/s wr, 254 op/s 2024-07-26T07:54:33.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:33 smithi097 ceph-mon[32231]: pgmap v593: 97 pgs: 97 active+clean; 31 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-07-26T07:54:33.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:33 smithi181 ceph-mon[35869]: pgmap v593: 97 pgs: 97 active+clean; 31 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-07-26T07:54:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:33 smithi115 ceph-mon[35364]: pgmap v593: 97 pgs: 97 active+clean; 31 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-07-26T07:54:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:35 smithi115 ceph-mon[35364]: pgmap v594: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.9 MiB/s wr, 244 op/s 2024-07-26T07:54:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:35 smithi181 ceph-mon[35869]: pgmap v594: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.9 MiB/s wr, 244 op/s 2024-07-26T07:54:36.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:35 smithi097 ceph-mon[32231]: pgmap v594: 97 pgs: 97 active+clean; 35 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.9 MiB/s wr, 244 op/s 2024-07-26T07:54:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.091+0000 7ff408a00640 1 -- 172.21.15.97:0/3643626076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 msgr2=0x7ff40410b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.091+0000 7ff408a00640 1 --2- 172.21.15.97:0/3643626076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 0x7ff40410b180 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009ca0 tx=0x7ff3fc02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.091+0000 7ff408a00640 1 -- 172.21.15.97:0/3643626076 shutdown_connections 2024-07-26T07:54:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.091+0000 7ff408a00640 1 --2- 172.21.15.97:0/3643626076 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 0x7ff40410b180 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.091+0000 7ff408a00640 1 --2- 172.21.15.97:0/3643626076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041060b0 0x7ff404108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.091+0000 7ff408a00640 1 --2- 172.21.15.97:0/3643626076 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40406bcc0 0x7ff404105b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.091+0000 7ff408a00640 1 -- 172.21.15.97:0/3643626076 >> 172.21.15.97:0/3643626076 conn(0x7ff4040fd770 msgr2=0x7ff4040ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.092+0000 7ff408a00640 1 -- 172.21.15.97:0/3643626076 shutdown_connections 2024-07-26T07:54:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.092+0000 7ff408a00640 1 -- 172.21.15.97:0/3643626076 wait complete. 2024-07-26T07:54:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.093+0000 7ff408a00640 1 Processor -- start 2024-07-26T07:54:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.093+0000 7ff408a00640 1 -- start start 2024-07-26T07:54:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff408a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40406bcc0 0x7ff404071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff408a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041060b0 0x7ff4040723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff408a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 0x7ff404072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff408a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff404110c90 con 0x7ff404108c70 2024-07-26T07:54:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff408a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff404110b10 con 0x7ff40406bcc0 2024-07-26T07:54:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff408a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff404110e10 con 0x7ff4041060b0 2024-07-26T07:54:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff403200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 0x7ff404072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff403200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 0x7ff404072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39784/0 (socket says 172.21.15.97:39784) 2024-07-26T07:54:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.094+0000 7ff403200640 1 -- 172.21.15.97:0/3870938234 learned_addr learned my addr 172.21.15.97:0/3870938234 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.095+0000 7ff401e00640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041060b0 0x7ff4040723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.095+0000 7ff403200640 1 -- 172.21.15.97:0/3870938234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041060b0 msgr2=0x7ff4040723c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.095+0000 7ff403200640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041060b0 0x7ff4040723c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.095+0000 7ff403200640 1 -- 172.21.15.97:0/3870938234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40406bcc0 msgr2=0x7ff404071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.095+0000 7ff403200640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40406bcc0 0x7ff404071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.095+0000 7ff403200640 1 -- 172.21.15.97:0/3870938234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff404073790 con 0x7ff404108c70 2024-07-26T07:54:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.095+0000 7ff401e00640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041060b0 0x7ff4040723c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:54:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.095+0000 7ff403200640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 0x7ff404072f70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0098d0 tx=0x7ff3fc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.096+0000 7ff3fb400640 1 -- 172.21.15.97:0/3870938234 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc003d90 con 0x7ff404108c70 2024-07-26T07:54:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.096+0000 7ff3fb400640 1 -- 172.21.15.97:0/3870938234 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3fc0044d0 con 0x7ff404108c70 2024-07-26T07:54:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.096+0000 7ff408a00640 1 -- 172.21.15.97:0/3870938234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff404073a20 con 0x7ff404108c70 2024-07-26T07:54:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.096+0000 7ff3fb400640 1 -- 172.21.15.97:0/3870938234 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc030d40 con 0x7ff404108c70 2024-07-26T07:54:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.097+0000 7ff408a00640 1 -- 172.21.15.97:0/3870938234 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff404074140 con 0x7ff404108c70 2024-07-26T07:54:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.102+0000 7ff3fb400640 1 -- 172.21.15.97:0/3870938234 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7ff3fc004040 con 0x7ff404108c70 2024-07-26T07:54:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.104+0000 7ff4105b4740 1 Processor -- start 2024-07-26T07:54:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.104+0000 7ff4105b4740 1 -- start start 2024-07-26T07:54:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59b71b0 0x55b5f5d5d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d5d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59964d0 0x55b5f5d5e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff4105b4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b5f5ccf270 con 0x55b5f59b71b0 2024-07-26T07:54:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff4105b4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b5f5cbe770 con 0x55b5f59964d0 2024-07-26T07:54:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff4105b4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b5f5cc27b0 con 0x55b5f598e2f0 2024-07-26T07:54:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff3fb400640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3c50 0x7ff3e80b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff401e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59b71b0 0x55b5f5d5d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff401e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59b71b0 0x55b5f5d5d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39790/0 (socket says 172.21.15.97:39790) 2024-07-26T07:54:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff401e00640 1 -- 172.21.15.97:0/1195383185 learned_addr learned my addr 172.21.15.97:0/1195383185 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff3fb400640 1 -- 172.21.15.97:0/3870938234 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff3fc108170 con 0x7ff404108c70 2024-07-26T07:54:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.105+0000 7ff402800640 1 --2- 172.21.15.97:0/1195383185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d5d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.106+0000 7ff401e00640 1 -- 172.21.15.97:0/1195383185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 msgr2=0x55b5f5d5d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.106+0000 7ff401e00640 1 --2- 172.21.15.97:0/1195383185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d5d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.106+0000 7ff401e00640 1 -- 172.21.15.97:0/1195383185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59964d0 msgr2=0x55b5f5d5e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.106+0000 7ff401e00640 1 --2- 172.21.15.97:0/1195383185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59964d0 0x55b5f5d5e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.106+0000 7ff401e00640 1 -- 172.21.15.97:0/1195383185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f5a22790 con 0x55b5f59b71b0 2024-07-26T07:54:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.106+0000 7ff403200640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3c50 0x7ff3e80b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.106+0000 7ff401e00640 1 --2- 172.21.15.97:0/1195383185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59b71b0 0x55b5f5d5d0e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000bbe0 tx=0x7ff3f000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b45ce139aa33201 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:37.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.107+0000 7ff3ef400640 1 -- 172.21.15.97:0/1195383185 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0011670 con 0x55b5f59b71b0 2024-07-26T07:54:37.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.107+0000 7ff3ef400640 1 -- 172.21.15.97:0/1195383185 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3f0011ca0 con 0x55b5f59b71b0 2024-07-26T07:54:37.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.107+0000 7ff3ef400640 1 -- 172.21.15.97:0/1195383185 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0019bb0 con 0x55b5f59b71b0 2024-07-26T07:54:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 -- 172.21.15.97:0/1195383185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59b71b0 msgr2=0x55b5f5d5d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff403200640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3c50 0x7ff3e80b6160 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0c7a20 tx=0x7ff3fc0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 --2- 172.21.15.97:0/1195383185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59b71b0 0x55b5f5d5d0e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000bbe0 tx=0x7ff3f000f730 comp rx=0 tx=0).stop 2024-07-26T07:54:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 -- 172.21.15.97:0/1195383185 shutdown_connections 2024-07-26T07:54:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 --2- 172.21.15.97:0/1195383185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59964d0 0x55b5f5d5e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 --2- 172.21.15.97:0/1195383185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d5d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 --2- 172.21.15.97:0/1195383185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59b71b0 0x55b5f5d5d0e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 -- 172.21.15.97:0/1195383185 >> 172.21.15.97:0/1195383185 conn(0x55b5f59ab300 msgr2=0x55b5f5d58b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 -- 172.21.15.97:0/1195383185 shutdown_connections 2024-07-26T07:54:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.108+0000 7ff4105b4740 1 -- 172.21.15.97:0/1195383185 wait complete. 2024-07-26T07:54:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.109+0000 7ff4105b4740 1 -- start start 2024-07-26T07:54:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.109+0000 7ff4105b4740 1 Processor -- start 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.110+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.110+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.110+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f59b71b0 0x55b5f5d70630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.110+0000 7ff4105b4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b5f5bf8ab0 con 0x55b5f59964d0 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.110+0000 7ff4105b4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55b5f5c5bd50 con 0x55b5f598e2f0 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.110+0000 7ff4105b4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b5f5d5edc0 con 0x55b5f59b71b0 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.110+0000 7ff4105b4740 10 client.0 _ll_register_callbacks cb 0x55b5f59cea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff401e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff401e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:56748/0 (socket says 172.21.15.97:56748) 2024-07-26T07:54:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff402800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff401e00640 1 -- 172.21.15.97:0/2442202952 learned_addr learned my addr 172.21.15.97:0/2442202952 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff401e00640 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f59b71b0 0x55b5f5d70630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff402800640 1 -- 172.21.15.97:0/2442202952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f59b71b0 msgr2=0x55b5f5d70630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff402800640 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f59b71b0 0x55b5f5d70630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff402800640 1 -- 172.21.15.97:0/2442202952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f598e2f0 msgr2=0x55b5f5d6f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.111+0000 7ff402800640 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.112+0000 7ff402800640 1 -- 172.21.15.97:0/2442202952 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f5a9b620 con 0x55b5f59964d0 2024-07-26T07:54:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.112+0000 7ff402800640 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6fac0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4003870 tx=0x7ff3f4003d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7afb83c24123ee1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.112+0000 7ff3e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.112+0000 7ff3e3e00640 1 -- 172.21.15.97:0/2442202952 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f401a470 con 0x55b5f59964d0 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.112+0000 7ff3e3e00640 1 -- 172.21.15.97:0/2442202952 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff3f401a610 con 0x55b5f59964d0 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.112+0000 7ff4105b4740 1 -- 172.21.15.97:0/2442202952 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3f0011ca0 con 0x55b5f59964d0 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff3e3e00640 1 -- 172.21.15.97:0/2442202952 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff3f401a810 con 0x55b5f59964d0 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff4105b4740 10 client.15831 fetch_fsmap learned FSMap version 29 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff4105b4740 1 -- 172.21.15.97:0/2442202952 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5f5d68850 con 0x55b5f59964d0 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff3e3e00640 1 -- 172.21.15.97:0/2442202952 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3f0011ca0 con 0x55b5f59964d0 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff3e3e00640 1 -- 172.21.15.97:0/2442202952 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7ff3f4023050 con 0x55b5f59964d0 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff4105b4740 10 client.15831 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff4105b4740 10 client.15831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff4105b4740 20 client.15831 populate_metadata read hostname 'smithi097' 2024-07-26T07:54:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff4105b4740 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55b5f5975d80 0x55b5f5d6c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff4105b4740 4 client.15831 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:54:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.113+0000 7ff4105b4740 1 -- 172.21.15.97:0/2442202952 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5f5cd11c0 con 0x55b5f5975d80 2024-07-26T07:54:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.114+0000 7ff401e00640 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55b5f5975d80 0x55b5f5d6c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.115+0000 7ff401e00640 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55b5f5975d80 0x55b5f5d6c060 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2b442bde645496b server_cookie=f94f613cafb82480 in_seq=0 out_seq=0 2024-07-26T07:54:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.115+0000 7ff3e3e00640 10 client.15831 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.117+0000 7ff3e3e00640 1 -- 172.21.15.97:0/2442202952 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5f5cd11c0 con 0x55b5f5975d80 2024-07-26T07:54:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.117+0000 7ff3e3e00640 10 client.15831 handle_command_reply: tid=0 2024-07-26T07:54:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.117+0000 7ff4105b4740 1 client.15831 shutdown 2024-07-26T07:54:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.118+0000 7ff4105b4740 1 -- 172.21.15.97:0/2442202952 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 msgr2=0x55b5f5d6fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.118+0000 7ff4105b4740 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6fac0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4003870 tx=0x7ff3f4003d30 comp rx=0 tx=0).stop 2024-07-26T07:54:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.118+0000 7ff4105b4740 1 -- 172.21.15.97:0/2442202952 shutdown_connections 2024-07-26T07:54:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.118+0000 7ff4105b4740 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55b5f5975d80 0x55b5f5d6c060 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.118+0000 7ff4105b4740 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f59b71b0 0x55b5f5d70630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.118+0000 7ff4105b4740 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6fac0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.118+0000 7ff4105b4740 1 --2- 172.21.15.97:0/2442202952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.118+0000 7ff4105b4740 1 -- 172.21.15.97:0/2442202952 >> 172.21.15.97:0/2442202952 conn(0x55b5f59ab300 msgr2=0x55b5f5d570d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.119+0000 7ff3e3e00640 0 client.15831 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.119+0000 7ff4105b4740 1 -- 172.21.15.97:0/2442202952 shutdown_connections 2024-07-26T07:54:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.119+0000 7ff4105b4740 1 -- 172.21.15.97:0/2442202952 wait complete. 2024-07-26T07:54:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.119+0000 7ff4105b4740 20 client.15831 trim_cache size 0 max 16384 2024-07-26T07:54:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.151+0000 7ff4105b4740 1 Processor -- start 2024-07-26T07:54:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.152+0000 7ff4105b4740 1 -- start start 2024-07-26T07:54:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.152+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.152+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.152+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 0x55b5f5d57880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.152+0000 7ff4105b4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff4041b0f30 con 0x55b5f59964d0 2024-07-26T07:54:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.152+0000 7ff4105b4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff4041b2aa0 con 0x55b5f59b71b0 2024-07-26T07:54:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.152+0000 7ff4105b4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4041b1600 con 0x55b5f598e2f0 2024-07-26T07:54:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.153+0000 7ff402800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.153+0000 7ff402800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39800/0 (socket says 172.21.15.97:39800) 2024-07-26T07:54:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.153+0000 7ff401e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.153+0000 7ff401e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:41776/0 (socket says 172.21.15.97:41776) 2024-07-26T07:54:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.153+0000 7ff401e00640 1 -- 172.21.15.97:0/214668326 learned_addr learned my addr 172.21.15.97:0/214668326 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.153+0000 7ff401e00640 1 --2- 172.21.15.97:0/214668326 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 0x55b5f5d57880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.154+0000 7ff402800640 1 -- 172.21.15.97:0/214668326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 msgr2=0x55b5f5d6e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.154+0000 7ff402800640 1 --2- 172.21.15.97:0/214668326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.154+0000 7ff402800640 1 -- 172.21.15.97:0/214668326 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 msgr2=0x55b5f5d57880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.154+0000 7ff402800640 1 --2- 172.21.15.97:0/214668326 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 0x55b5f5d57880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.154+0000 7ff402800640 1 -- 172.21.15.97:0/214668326 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f5d5e630 con 0x55b5f59964d0 2024-07-26T07:54:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.155+0000 7ff402800640 1 --2- 172.21.15.97:0/214668326 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6ecf0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4000c00 tx=0x7ff3f400b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95454dec9ceaf085 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.155+0000 7ff3eea00640 1 -- 172.21.15.97:0/214668326 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4031070 con 0x55b5f59964d0 2024-07-26T07:54:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.155+0000 7ff3eea00640 1 -- 172.21.15.97:0/214668326 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3f4015070 con 0x55b5f59964d0 2024-07-26T07:54:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.155+0000 7ff3eea00640 1 -- 172.21.15.97:0/214668326 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4011440 con 0x55b5f59964d0 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 -- 172.21.15.97:0/214668326 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 msgr2=0x55b5f5d6ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 --2- 172.21.15.97:0/214668326 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6ecf0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4000c00 tx=0x7ff3f400b400 comp rx=0 tx=0).stop 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 -- 172.21.15.97:0/214668326 shutdown_connections 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 --2- 172.21.15.97:0/214668326 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 0x55b5f5d57880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 --2- 172.21.15.97:0/214668326 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f59964d0 0x55b5f5d6ecf0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 --2- 172.21.15.97:0/214668326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d6e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 -- 172.21.15.97:0/214668326 >> 172.21.15.97:0/214668326 conn(0x55b5f59ab300 msgr2=0x55b5f5d5df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 -- 172.21.15.97:0/214668326 shutdown_connections 2024-07-26T07:54:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.157+0000 7ff4105b4740 1 -- 172.21.15.97:0/214668326 wait complete. 2024-07-26T07:54:37.159 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.158+0000 7ff4105b4740 1 -- start start 2024-07-26T07:54:37.159 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.158+0000 7ff4105b4740 1 Processor -- start 2024-07-26T07:54:37.159 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.158+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d73ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.158+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 0x55b5f5d74170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.158+0000 7ff4105b4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f594b1c0 0x55b5f5d74c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.158+0000 7ff4105b4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff4041b23a0 con 0x55b5f594b1c0 2024-07-26T07:54:37.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.158+0000 7ff4105b4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff4041b1cd0 con 0x55b5f59b71b0 2024-07-26T07:54:37.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.158+0000 7ff4105b4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4041aef10 con 0x55b5f598e2f0 2024-07-26T07:54:37.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.159+0000 7ff4105b4740 10 client.0 _ll_register_callbacks cb 0x7ff3f4004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:54:37.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.159+0000 7ff401e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f594b1c0 0x55b5f5d74c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.159+0000 7ff401e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f594b1c0 0x55b5f5d74c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39808/0 (socket says 172.21.15.97:39808) 2024-07-26T07:54:37.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.159+0000 7ff401e00640 1 -- 172.21.15.97:0/3885719711 learned_addr learned my addr 172.21.15.97:0/3885719711 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:37.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.159+0000 7ff402800640 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 0x55b5f5d74170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.159+0000 7ff401e00640 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d73ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.160+0000 7ff401e00640 1 -- 172.21.15.97:0/3885719711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 msgr2=0x55b5f5d73ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:37.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.160+0000 7ff401e00640 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d73ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.160+0000 7ff401e00640 1 -- 172.21.15.97:0/3885719711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 msgr2=0x55b5f5d74170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.160+0000 7ff401e00640 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 0x55b5f5d74170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.160+0000 7ff401e00640 1 -- 172.21.15.97:0/3885719711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f5d753d0 con 0x55b5f594b1c0 2024-07-26T07:54:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.160+0000 7ff401e00640 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f594b1c0 0x55b5f5d74c70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000ffd0 tx=0x7ff3f4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a07b1ef569db6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.160+0000 7ff3e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.161+0000 7ff3e3e00640 1 -- 172.21.15.97:0/3885719711 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f003c060 con 0x55b5f594b1c0 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.161+0000 7ff3e3e00640 1 -- 172.21.15.97:0/3885719711 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff3f0011470 con 0x55b5f594b1c0 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.161+0000 7ff4105b4740 1 -- 172.21.15.97:0/3885719711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3f4015070 con 0x55b5f594b1c0 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.161+0000 7ff3e3e00640 1 -- 172.21.15.97:0/3885719711 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff3f0019c30 con 0x55b5f594b1c0 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.161+0000 7ff4105b4740 10 client.15837 fetch_fsmap learned FSMap version 29 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.161+0000 7ff4105b4740 1 -- 172.21.15.97:0/3885719711 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5f5d76750 con 0x55b5f594b1c0 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.162+0000 7ff3e3e00640 1 -- 172.21.15.97:0/3885719711 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3f4015070 con 0x55b5f594b1c0 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.162+0000 7ff3e3e00640 1 -- 172.21.15.97:0/3885719711 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7ff3f0031070 con 0x55b5f594b1c0 2024-07-26T07:54:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.162+0000 7ff4105b4740 10 client.15837 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T07:54:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.162+0000 7ff4105b4740 10 client.15837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:54:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.162+0000 7ff4105b4740 20 client.15837 populate_metadata read hostname 'smithi097' 2024-07-26T07:54:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.162+0000 7ff4105b4740 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55b5f5d76fd0 0x55b5f5df0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.162+0000 7ff4105b4740 4 client.15837 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:54:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.162+0000 7ff4105b4740 1 -- 172.21.15.97:0/3885719711 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5f5cccc60 con 0x55b5f5d76fd0 2024-07-26T07:54:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.163+0000 7ff402800640 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55b5f5d76fd0 0x55b5f5df0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.164+0000 7ff402800640 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55b5f5d76fd0 0x55b5f5df0620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=455e575b53fba90 server_cookie=1c8ce8775bb0b7fb in_seq=0 out_seq=0 2024-07-26T07:54:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:37.164+0000 7ff3e3e00640 10 client.15837 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:37 smithi115 ceph-mon[35364]: pgmap v595: 97 pgs: 97 active+clean; 36 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-07-26T07:54:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:37 smithi181 ceph-mon[35869]: pgmap v595: 97 pgs: 97 active+clean; 36 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-07-26T07:54:38.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:37 smithi097 ceph-mon[32231]: pgmap v595: 97 pgs: 97 active+clean; 36 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-07-26T07:54:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff3e3e00640 1 -- 172.21.15.97:0/3885719711 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5f5cccc60 con 0x55b5f5d76fd0 2024-07-26T07:54:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff3e3e00640 10 client.15837 handle_command_reply: tid=0 2024-07-26T07:54:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 client.15837 shutdown 2024-07-26T07:54:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 -- 172.21.15.97:0/3885719711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f594b1c0 msgr2=0x55b5f5d74c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f594b1c0 0x55b5f5d74c70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000ffd0 tx=0x7ff3f4004470 comp rx=0 tx=0).stop 2024-07-26T07:54:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 -- 172.21.15.97:0/3885719711 shutdown_connections 2024-07-26T07:54:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55b5f5d76fd0 0x55b5f5df0620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5f594b1c0 0x55b5f5d74c70 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b5f59b71b0 0x55b5f5d74170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 --2- 172.21.15.97:0/3885719711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b5f598e2f0 0x55b5f5d73ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff4105b4740 1 -- 172.21.15.97:0/3885719711 >> 172.21.15.97:0/3885719711 conn(0x55b5f59ab300 msgr2=0x55b5f5d5df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.496+0000 7ff3e3e00640 0 client.15837 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.497+0000 7ff4105b4740 1 -- 172.21.15.97:0/3885719711 shutdown_connections 2024-07-26T07:54:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.497+0000 7ff4105b4740 1 -- 172.21.15.97:0/3885719711 wait complete. 2024-07-26T07:54:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.497+0000 7ff4105b4740 20 client.15837 trim_cache size 0 max 16384 2024-07-26T07:54:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:54:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:54:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:54:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:54:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 -- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3c50 msgr2=0x7ff3e80b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3c50 0x7ff3e80b6160 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0c7a20 tx=0x7ff3fc0095c0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 -- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 msgr2=0x7ff404072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 0x7ff404072f70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0098d0 tx=0x7ff3fc003ba0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 -- 172.21.15.97:0/3870938234 shutdown_connections 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3e80b3c50 0x7ff3e80b6160 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff404108c70 0x7ff404072f70 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4041060b0 0x7ff4040723c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 --2- 172.21.15.97:0/3870938234 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff40406bcc0 0x7ff404071d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.498+0000 7ff408a00640 1 -- 172.21.15.97:0/3870938234 >> 172.21.15.97:0/3870938234 conn(0x7ff4040fd770 msgr2=0x7ff4041045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.499+0000 7ff408a00640 1 -- 172.21.15.97:0/3870938234 shutdown_connections 2024-07-26T07:54:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:38.499+0000 7ff408a00640 1 -- 172.21.15.97:0/3870938234 wait complete. 2024-07-26T07:54:39.126 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:38 smithi181 ceph-mon[35869]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T07:54:39.126 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:38 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T07:54:39.126 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:38 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-07-26T07:54:39.126 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:38 smithi181 systemd[1]: Stopped Ceph mds.c for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:54:39.126 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:38 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.c.service: Consumed 4.818s CPU time. 2024-07-26T07:54:39.127 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:38 smithi181 systemd[1]: Starting Ceph mds.c for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:54:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:38 smithi115 ceph-mon[35364]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T07:54:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:38 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T07:54:39.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:38 smithi097 ceph-mon[32231]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T07:54:39.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:38 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T07:54:39.455 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:39 smithi181 podman[68878]: 2024-07-26 07:54:39.031356772 +0000 UTC m=+0.035445111 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:54:39.455 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:39 smithi181 podman[68878]: 2024-07-26 07:54:39.168365987 +0000 UTC m=+0.172454339 container create ab64ea8e8253f2d780396a1fae18ba7aaddfcf3d4311bfb7bbd438f590ecff77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T07:54:39.733 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:39 smithi181 podman[68878]: 2024-07-26 07:54:39.454458243 +0000 UTC m=+0.458546612 container init ab64ea8e8253f2d780396a1fae18ba7aaddfcf3d4311bfb7bbd438f590ecff77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T07:54:39.733 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:39 smithi181 podman[68878]: 2024-07-26 07:54:39.462886996 +0000 UTC m=+0.466975308 container start ab64ea8e8253f2d780396a1fae18ba7aaddfcf3d4311bfb7bbd438f590ecff77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-07-26T07:54:39.733 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:39 smithi181 bash[68878]: ab64ea8e8253f2d780396a1fae18ba7aaddfcf3d4311bfb7bbd438f590ecff77 2024-07-26T07:54:39.734 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:39 smithi181 systemd[1]: Started Ceph mds.c for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:54:39.734 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:39 smithi181 ceph-mds[68898]: -- 172.21.15.181:0/2967718055 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563be9e9b860 con 0x563bee03d180 2024-07-26T07:54:39.734 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:54:39 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: starting mds.c at 2024-07-26T07:54:40.201 INFO:tasks.workunit.client.0.smithi097.stdout: 9 11131 150 5930 107 2878 301 2024-07-26T07:54:40.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:39 smithi115 ceph-mon[35364]: pgmap v596: 97 pgs: 97 active+clean; 38 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-07-26T07:54:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:39 smithi181 ceph-mon[35869]: pgmap v596: 97 pgs: 97 active+clean; 38 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-07-26T07:54:40.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:39 smithi097 ceph-mon[32231]: pgmap v596: 97 pgs: 97 active+clean; 38 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-07-26T07:54:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:40 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:boot 2024-07-26T07:54:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:40 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:54:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T07:54:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:40 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:boot 2024-07-26T07:54:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:40 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:54:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T07:54:41.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:40 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:boot 2024-07-26T07:54:41.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:40 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:54:41.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T07:54:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:41 smithi115 ceph-mon[35364]: pgmap v597: 97 pgs: 97 active+clean; 33 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.7 MiB/s wr, 264 op/s 2024-07-26T07:54:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:41 smithi181 ceph-mon[35869]: pgmap v597: 97 pgs: 97 active+clean; 33 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.7 MiB/s wr, 264 op/s 2024-07-26T07:54:42.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:41 smithi097 ceph-mon[32231]: pgmap v597: 97 pgs: 97 active+clean; 33 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.7 MiB/s wr, 264 op/s 2024-07-26T07:54:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:43 smithi115 ceph-mon[35364]: pgmap v598: 97 pgs: 97 active+clean; 34 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 242 op/s 2024-07-26T07:54:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:43 smithi181 ceph-mon[35869]: pgmap v598: 97 pgs: 97 active+clean; 34 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 242 op/s 2024-07-26T07:54:44.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:43 smithi097 ceph-mon[32231]: pgmap v598: 97 pgs: 97 active+clean; 34 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.2 MiB/s wr, 242 op/s 2024-07-26T07:54:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:45 smithi181 ceph-mon[35869]: pgmap v599: 97 pgs: 97 active+clean; 37 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 1.4 MiB/s wr, 273 op/s 2024-07-26T07:54:46.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:45 smithi097 ceph-mon[32231]: pgmap v599: 97 pgs: 97 active+clean; 37 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 1.4 MiB/s wr, 273 op/s 2024-07-26T07:54:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:45 smithi115 ceph-mon[35364]: pgmap v599: 97 pgs: 97 active+clean; 37 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 1.4 MiB/s wr, 273 op/s 2024-07-26T07:54:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:47 smithi181 ceph-mon[35869]: pgmap v600: 97 pgs: 97 active+clean; 38 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.1 MiB/s wr, 258 op/s 2024-07-26T07:54:48.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:47 smithi097 ceph-mon[32231]: pgmap v600: 97 pgs: 97 active+clean; 38 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.1 MiB/s wr, 258 op/s 2024-07-26T07:54:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:47 smithi115 ceph-mon[35364]: pgmap v600: 97 pgs: 97 active+clean; 38 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.1 MiB/s wr, 258 op/s 2024-07-26T07:54:50.201 INFO:tasks.workunit.client.0.smithi097.stdout: 11 11313 101 5898 98 3355 244 2024-07-26T07:54:50.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:49 smithi097 ceph-mon[32231]: pgmap v601: 97 pgs: 97 active+clean; 39 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.1 MiB/s wr, 246 op/s 2024-07-26T07:54:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:49 smithi115 ceph-mon[35364]: pgmap v601: 97 pgs: 97 active+clean; 39 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.1 MiB/s wr, 246 op/s 2024-07-26T07:54:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:49 smithi181 ceph-mon[35869]: pgmap v601: 97 pgs: 97 active+clean; 39 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.1 MiB/s wr, 246 op/s 2024-07-26T07:54:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.807+0000 7f3adbe00640 1 -- 172.21.15.97:0/3455799477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc106b30 msgr2=0x7f3adc10d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.807+0000 7f3adbe00640 1 --2- 172.21.15.97:0/3455799477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc106b30 0x7f3adc10d4d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009ca0 tx=0x7f3ad402a8e0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.808+0000 7f3adbe00640 1 -- 172.21.15.97:0/3455799477 shutdown_connections 2024-07-26T07:54:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.808+0000 7f3adbe00640 1 --2- 172.21.15.97:0/3455799477 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc106b30 0x7f3adc10d4d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.808+0000 7f3adbe00640 1 --2- 172.21.15.97:0/3455799477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3adc106190 0x7f3adc1065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.808+0000 7f3adbe00640 1 --2- 172.21.15.97:0/3455799477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3adc105110 0x7f3adc105580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.808+0000 7f3adbe00640 1 -- 172.21.15.97:0/3455799477 >> 172.21.15.97:0/3455799477 conn(0x7f3adc100a30 msgr2=0x7f3adc102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.808+0000 7f3adbe00640 1 -- 172.21.15.97:0/3455799477 shutdown_connections 2024-07-26T07:54:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.808+0000 7f3adbe00640 1 -- 172.21.15.97:0/3455799477 wait complete. 2024-07-26T07:54:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.809+0000 7f3adbe00640 1 Processor -- start 2024-07-26T07:54:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.809+0000 7f3adbe00640 1 -- start start 2024-07-26T07:54:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc105110 0x7f3adc19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adbe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3adc106190 0x7f3adc1a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3adc106b30 0x7f3adc1a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc078c50 con 0x7f3adc105110 2024-07-26T07:54:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adbe00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc078ad0 con 0x7f3adc106190 2024-07-26T07:54:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adbe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc078dd0 con 0x7f3adc106b30 2024-07-26T07:54:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc105110 0x7f3adc19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3ada000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3adc106190 0x7f3adc1a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc105110 0x7f3adc19fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48488/0 (socket says 172.21.15.97:48488) 2024-07-26T07:54:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.810+0000 7f3adaa00640 1 -- 172.21.15.97:0/4135908534 learned_addr learned my addr 172.21.15.97:0/4135908534 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.811+0000 7f3adb400640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3adc106b30 0x7f3adc1a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.811+0000 7f3adaa00640 1 -- 172.21.15.97:0/4135908534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3adc106b30 msgr2=0x7f3adc1a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.811+0000 7f3adaa00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3adc106b30 0x7f3adc1a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.811+0000 7f3adaa00640 1 -- 172.21.15.97:0/4135908534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3adc106190 msgr2=0x7f3adc1a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.811+0000 7f3adaa00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3adc106190 0x7f3adc1a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.811+0000 7f3adaa00640 1 -- 172.21.15.97:0/4135908534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3adc1a16c0 con 0x7f3adc105110 2024-07-26T07:54:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.812+0000 7f3adaa00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc105110 0x7f3adc19fc80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00e9f0 tx=0x7f3acc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.813+0000 7f3ad3400640 1 -- 172.21.15.97:0/4135908534 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc00cd80 con 0x7f3adc105110 2024-07-26T07:54:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.813+0000 7f3ad3400640 1 -- 172.21.15.97:0/4135908534 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3acc0108c0 con 0x7f3adc105110 2024-07-26T07:54:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.813+0000 7f3ad3400640 1 -- 172.21.15.97:0/4135908534 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc0187c0 con 0x7f3adc105110 2024-07-26T07:54:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.813+0000 7f3adbe00640 1 -- 172.21.15.97:0/4135908534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3adc1a19b0 con 0x7f3adc105110 2024-07-26T07:54:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.813+0000 7f3adbe00640 1 -- 172.21.15.97:0/4135908534 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3adc1a80a0 con 0x7f3adc105110 2024-07-26T07:54:51.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.818+0000 7f3ae89b2740 1 Processor -- start 2024-07-26T07:54:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.819+0000 7f3ae89b2740 1 -- start start 2024-07-26T07:54:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.819+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b5900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.819+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b590750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.820+0000 7f3ad3400640 1 -- 172.21.15.97:0/4135908534 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f3acc010430 con 0x7f3adc105110 2024-07-26T07:54:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.820+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b5912a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.820+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56207b502270 con 0x56207b1c12f0 2024-07-26T07:54:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.820+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56207b4f1770 con 0x56207b1c94d0 2024-07-26T07:54:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.820+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56207b4f57b0 con 0x56207b1ea1b0 2024-07-26T07:54:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.820+0000 7f3adb400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b5900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.820+0000 7f3adb400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b5900e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:35934/0 (socket says 172.21.15.97:35934) 2024-07-26T07:54:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.820+0000 7f3adb400640 1 -- 172.21.15.97:0/951865527 learned_addr learned my addr 172.21.15.97:0/951865527 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.821+0000 7f3adb400640 1 -- 172.21.15.97:0/951865527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 msgr2=0x56207b5912a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:54:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.821+0000 7f3adb400640 1 --2- 172.21.15.97:0/951865527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b5912a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.821+0000 7f3adb400640 1 -- 172.21.15.97:0/951865527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 msgr2=0x56207b590750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:54:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.821+0000 7f3adb400640 1 --2- 172.21.15.97:0/951865527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b590750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.821+0000 7f3adb400640 1 -- 172.21.15.97:0/951865527 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56207b255790 con 0x56207b1ea1b0 2024-07-26T07:54:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.821+0000 7f3adb400640 1 --2- 172.21.15.97:0/951865527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b5900e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009ca0 tx=0x7f3ad40315b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b52bef0d5a09c96e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.822+0000 7f3ad3400640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3ac00b3cd0 0x7f3ac00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.822+0000 7f3ac7e00640 1 -- 172.21.15.97:0/951865527 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4031b10 con 0x56207b1ea1b0 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.822+0000 7f3ac7e00640 1 -- 172.21.15.97:0/951865527 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ad4036030 con 0x56207b1ea1b0 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.822+0000 7f3ac7e00640 1 -- 172.21.15.97:0/951865527 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4004550 con 0x56207b1ea1b0 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.823+0000 7f3adaa00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3ac00b3cd0 0x7f3ac00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.823+0000 7f3ae89b2740 1 -- 172.21.15.97:0/951865527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 msgr2=0x56207b5900e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.823+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/951865527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b5900e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009ca0 tx=0x7f3ad40315b0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.824+0000 7f3adaa00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3ac00b3cd0 0x7f3ac00b61e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0aaeb0 tx=0x7f3acc009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.824+0000 7f3ae89b2740 1 -- 172.21.15.97:0/951865527 shutdown_connections 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.824+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/951865527 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b5912a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.824+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/951865527 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b590750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.824+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/951865527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b5900e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.824+0000 7f3ae89b2740 1 -- 172.21.15.97:0/951865527 >> 172.21.15.97:0/951865527 conn(0x56207b1de300 msgr2=0x56207b58bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.822+0000 7f3ad3400640 1 -- 172.21.15.97:0/4135908534 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f3acc0e3fe0 con 0x7f3adc105110 2024-07-26T07:54:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.824+0000 7f3ae89b2740 1 -- 172.21.15.97:0/951865527 shutdown_connections 2024-07-26T07:54:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.824+0000 7f3ae89b2740 1 -- 172.21.15.97:0/951865527 wait complete. 2024-07-26T07:54:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.825+0000 7f3ae89b2740 1 -- start start 2024-07-26T07:54:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.825+0000 7f3ae89b2740 1 Processor -- start 2024-07-26T07:54:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.827+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1c12f0 0x56207b599b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.827+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b59a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.827+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.828+0000 7f3adb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.828+0000 7f3ada000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b59a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.828+0000 7f3adb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48490/0 (socket says 172.21.15.97:48490) 2024-07-26T07:54:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.828+0000 7f3ada000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b59a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:33076/0 (socket says 172.21.15.97:33076) 2024-07-26T07:54:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.828+0000 7f3adb400640 1 -- 172.21.15.97:0/1367314850 learned_addr learned my addr 172.21.15.97:0/1367314850 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.828+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56207b42bab0 con 0x56207b1ea1b0 2024-07-26T07:54:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.828+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56207b48ed50 con 0x56207b1c94d0 2024-07-26T07:54:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.829+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56207b591d60 con 0x56207b1c12f0 2024-07-26T07:54:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.829+0000 7f3adb400640 1 -- 172.21.15.97:0/1367314850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1c12f0 msgr2=0x56207b599b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.829+0000 7f3adb400640 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1c12f0 0x56207b599b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.829+0000 7f3adb400640 1 -- 172.21.15.97:0/1367314850 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 msgr2=0x56207b59a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.829+0000 7f3adb400640 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b59a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.829+0000 7f3adb400640 1 -- 172.21.15.97:0/1367314850 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56207b2ce620 con 0x56207b1ea1b0 2024-07-26T07:54:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.830+0000 7f3ada000640 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b59a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:54:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.830+0000 7f3ae89b2740 10 client.0 _ll_register_callbacks cb 0x56207b201a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:54:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.830+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ad4036030 con 0x56207b1ea1b0 2024-07-26T07:54:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.831+0000 7f3adb400640 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59ad80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009570 tx=0x7f3ad40099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffbfcaa3125bf07 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.831+0000 7f3ac4c00640 10 client.15852 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:54:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.831+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1367314850 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4003970 con 0x56207b1ea1b0 2024-07-26T07:54:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.832+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1367314850 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f3ad4003b10 con 0x56207b1ea1b0 2024-07-26T07:54:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.832+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1367314850 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3ad4003d30 con 0x56207b1ea1b0 2024-07-26T07:54:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.832+0000 7f3ae89b2740 10 client.15852 fetch_fsmap learned FSMap version 31 2024-07-26T07:54:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.832+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({fsmap=31}) -- 0x56207b5a3860 con 0x56207b1ea1b0 2024-07-26T07:54:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.832+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1367314850 <== mon.0 v2:172.21.15.97:3300/0 4 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3ad4042070 con 0x56207b1ea1b0 2024-07-26T07:54:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.833+0000 7f3ae89b2740 10 client.15852 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:54:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.833+0000 7f3ae89b2740 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:54:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.833+0000 7f3ae89b2740 20 client.15852 populate_metadata read hostname 'smithi097' 2024-07-26T07:54:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.833+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56207b1a8d80 0x56207b5a6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.833+0000 7f3ae89b2740 4 client.15852 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:54:51.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.833+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56207b5041c0 con 0x56207b1a8d80 2024-07-26T07:54:51.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.833+0000 7f3ada000640 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56207b1a8d80 0x56207b5a6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.834+0000 7f3ac4c00640 10 client.15852 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:51.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.834+0000 7f3ada000640 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56207b1a8d80 0x56207b5a6da0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad44aec9bb4e61aa server_cookie=a2fc540f3bf6adb5 in_seq=0 out_seq=0 2024-07-26T07:54:51.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.837+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1367314850 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56207b5041c0 con 0x56207b1a8d80 2024-07-26T07:54:51.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.837+0000 7f3ac4c00640 10 client.15852 handle_command_reply: tid=0 2024-07-26T07:54:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.837+0000 7f3ae89b2740 1 client.15852 shutdown 2024-07-26T07:54:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.837+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 msgr2=0x56207b59ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.837+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59ad80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009570 tx=0x7f3ad40099a0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 shutdown_connections 2024-07-26T07:54:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56207b1a8d80 0x56207b5a6da0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59ad80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b59a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1367314850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1c12f0 0x56207b599b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 >> 172.21.15.97:0/1367314850 conn(0x56207b1de300 msgr2=0x56207b58a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ac4c00640 0 client.15852 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 shutdown_connections 2024-07-26T07:54:51.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.838+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1367314850 wait complete. 2024-07-26T07:54:51.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.839+0000 7f3ae89b2740 20 client.15852 trim_cache size 0 max 16384 2024-07-26T07:54:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.863+0000 7f3ae89b2740 1 Processor -- start 2024-07-26T07:54:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.863+0000 7f3ae89b2740 1 -- start start 2024-07-26T07:54:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.863+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b58d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.863+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b58af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.863+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b58b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.863+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc1b0e90 con 0x56207b1c12f0 2024-07-26T07:54:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.863+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc1b2a90 con 0x56207b1c94d0 2024-07-26T07:54:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.863+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc1b1560 con 0x56207b1ea1b0 2024-07-26T07:54:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.864+0000 7f3adb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b58d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.864+0000 7f3adb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b58d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48498/0 (socket says 172.21.15.97:48498) 2024-07-26T07:54:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.864+0000 7f3adb400640 1 -- 172.21.15.97:0/472983940 learned_addr learned my addr 172.21.15.97:0/472983940 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.865+0000 7f3ada000640 1 --2- 172.21.15.97:0/472983940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b58af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.865+0000 7f3adb400640 1 -- 172.21.15.97:0/472983940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 msgr2=0x56207b58b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.865+0000 7f3adb400640 1 --2- 172.21.15.97:0/472983940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b58b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.865+0000 7f3adb400640 1 -- 172.21.15.97:0/472983940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 msgr2=0x56207b58af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.865+0000 7f3adb400640 1 --2- 172.21.15.97:0/472983940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b58af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.865+0000 7f3adb400640 1 -- 172.21.15.97:0/472983940 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56207b59a690 con 0x56207b1c12f0 2024-07-26T07:54:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.866+0000 7f3adb400640 1 --2- 172.21.15.97:0/472983940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b58d150 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4002230 tx=0x7f3ad404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6132fa06aeaf4553 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.866+0000 7f3ac7400640 1 -- 172.21.15.97:0/472983940 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4003fa0 con 0x56207b1c12f0 2024-07-26T07:54:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.866+0000 7f3ac7400640 1 -- 172.21.15.97:0/472983940 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ad4036030 con 0x56207b1c12f0 2024-07-26T07:54:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.866+0000 7f3ac7400640 1 -- 172.21.15.97:0/472983940 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4030e10 con 0x56207b1c12f0 2024-07-26T07:54:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.867+0000 7f3ae89b2740 1 -- 172.21.15.97:0/472983940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 msgr2=0x56207b58d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.867+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/472983940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b58d150 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4002230 tx=0x7f3ad404e000 comp rx=0 tx=0).stop 2024-07-26T07:54:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.868+0000 7f3ae89b2740 1 -- 172.21.15.97:0/472983940 shutdown_connections 2024-07-26T07:54:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.868+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/472983940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1ea1b0 0x56207b58b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.868+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/472983940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c94d0 0x56207b58af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.868+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/472983940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1c12f0 0x56207b58d150 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.868+0000 7f3ae89b2740 1 -- 172.21.15.97:0/472983940 >> 172.21.15.97:0/472983940 conn(0x56207b1de300 msgr2=0x56207b58a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.868+0000 7f3ae89b2740 1 -- 172.21.15.97:0/472983940 shutdown_connections 2024-07-26T07:54:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.868+0000 7f3ae89b2740 1 -- 172.21.15.97:0/472983940 wait complete. 2024-07-26T07:54:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.868+0000 7f3ae89b2740 1 -- start start 2024-07-26T07:54:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.869+0000 7f3ae89b2740 1 Processor -- start 2024-07-26T07:54:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.869+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c12f0 0x56207b59e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.869+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1c94d0 0x56207b59e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3ae89b2740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc1b2300 con 0x56207b1ea1b0 2024-07-26T07:54:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc1b1c30 con 0x56207b1c12f0 2024-07-26T07:54:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3ae89b2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc1aee70 con 0x56207b1c94d0 2024-07-26T07:54:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3ae89b2740 10 client.0 _ll_register_callbacks cb 0x7f3ad4031500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:54:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48502/0 (socket says 172.21.15.97:48502) 2024-07-26T07:54:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 -- 172.21.15.97:0/1762489314 learned_addr learned my addr 172.21.15.97:0/1762489314 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:54:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c12f0 0x56207b59e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 -- 172.21.15.97:0/1762489314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1c94d0 msgr2=0x56207b59e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1c94d0 0x56207b59e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 -- 172.21.15.97:0/1762489314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c12f0 msgr2=0x56207b59e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c12f0 0x56207b59e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.870+0000 7f3adb400640 1 -- 172.21.15.97:0/1762489314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56207b59fbd0 con 0x56207b1ea1b0 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.871+0000 7f3adb400640 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59f470 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4030600 tx=0x7f3ad40306e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d5c4033f1b28d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.871+0000 7f3ac4c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.871+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1762489314 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4003b70 con 0x56207b1ea1b0 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.871+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1762489314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ad4036030 con 0x56207b1ea1b0 2024-07-26T07:54:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.871+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1762489314 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f3ad40307f0 con 0x56207b1ea1b0 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.871+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1762489314 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3ad40309f0 con 0x56207b1ea1b0 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ae89b2740 10 client.15858 fetch_fsmap learned FSMap version 31 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1762489314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56207b5a0ec0 con 0x56207b1ea1b0 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1762489314 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ad4036030 con 0x56207b1ea1b0 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1762489314 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3ad4042070 con 0x56207b1ea1b0 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ae89b2740 10 client.15858 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ae89b2740 10 client.15858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ae89b2740 20 client.15858 populate_metadata read hostname 'smithi097' 2024-07-26T07:54:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56207b17e1c0 0x56207b5a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:54:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ae89b2740 4 client.15858 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:54:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.872+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1762489314 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56207b4ffc60 con 0x56207b17e1c0 2024-07-26T07:54:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.873+0000 7f3ada000640 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56207b17e1c0 0x56207b5a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:54:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.874+0000 7f3ac4c00640 10 client.15858 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:51.874+0000 7f3ada000640 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56207b17e1c0 0x56207b5a16c0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd7af466309de8ae server_cookie=e51c27faa29c7b0e in_seq=0 out_seq=0 2024-07-26T07:54:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:51 smithi097 ceph-mon[32231]: pgmap v602: 97 pgs: 97 active+clean; 43 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 271 op/s 2024-07-26T07:54:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.264+0000 7f3ac4c00640 1 -- 172.21.15.97:0/1762489314 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56207b4ffc60 con 0x56207b17e1c0 2024-07-26T07:54:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.264+0000 7f3ac4c00640 10 client.15858 handle_command_reply: tid=0 2024-07-26T07:54:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.264+0000 7f3ae89b2740 1 client.15858 shutdown 2024-07-26T07:54:52.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.266+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1762489314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 msgr2=0x56207b59f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:52.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.266+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59f470 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4030600 tx=0x7f3ad40306e0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.266+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1762489314 shutdown_connections 2024-07-26T07:54:52.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.266+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56207b17e1c0 0x56207b5a16c0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.266+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56207b1ea1b0 0x56207b59f470 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.266+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56207b1c94d0 0x56207b59e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.266+0000 7f3ae89b2740 1 --2- 172.21.15.97:0/1762489314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56207b1c12f0 0x56207b59e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.267+0000 7f3ac4c00640 0 client.15858 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:54:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.267+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1762489314 >> 172.21.15.97:0/1762489314 conn(0x56207b1de300 msgr2=0x56207b58cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.267+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1762489314 shutdown_connections 2024-07-26T07:54:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.267+0000 7f3ae89b2740 1 -- 172.21.15.97:0/1762489314 wait complete. 2024-07-26T07:54:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.267+0000 7f3ae89b2740 20 client.15858 trim_cache size 0 max 16384 2024-07-26T07:54:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:54:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:54:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:54:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.268+0000 7f3adbe00640 1 -- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3ac00b3cd0 msgr2=0x7f3ac00b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.268+0000 7f3adbe00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3ac00b3cd0 0x7f3ac00b61e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0aaeb0 tx=0x7f3acc009310 comp rx=0 tx=0).stop 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.268+0000 7f3adbe00640 1 -- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc105110 msgr2=0x7f3adc19fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.268+0000 7f3adbe00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc105110 0x7f3adc19fc80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00e9f0 tx=0x7f3acc00eeb0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.269+0000 7f3adbe00640 1 -- 172.21.15.97:0/4135908534 shutdown_connections 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.269+0000 7f3adbe00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f3ac00b3cd0 0x7f3ac00b61e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.269+0000 7f3adbe00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3adc106b30 0x7f3adc1a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.269+0000 7f3adbe00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3adc106190 0x7f3adc1a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.269+0000 7f3adbe00640 1 --2- 172.21.15.97:0/4135908534 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3adc105110 0x7f3adc19fc80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:54:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.269+0000 7f3adbe00640 1 -- 172.21.15.97:0/4135908534 >> 172.21.15.97:0/4135908534 conn(0x7f3adc100a30 msgr2=0x7f3adc1074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:54:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.269+0000 7f3adbe00640 1 -- 172.21.15.97:0/4135908534 shutdown_connections 2024-07-26T07:54:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:54:52.269+0000 7f3adbe00640 1 -- 172.21.15.97:0/4135908534 wait complete. 2024-07-26T07:54:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:51 smithi115 ceph-mon[35364]: pgmap v602: 97 pgs: 97 active+clean; 43 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 271 op/s 2024-07-26T07:54:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:51 smithi181 ceph-mon[35869]: pgmap v602: 97 pgs: 97 active+clean; 43 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 271 op/s 2024-07-26T07:54:54.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:53 smithi097 ceph-mon[32231]: pgmap v603: 97 pgs: 97 active+clean; 43 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1015 KiB/s wr, 117 op/s 2024-07-26T07:54:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:53 smithi115 ceph-mon[35364]: pgmap v603: 97 pgs: 97 active+clean; 43 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1015 KiB/s wr, 117 op/s 2024-07-26T07:54:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:53 smithi181 ceph-mon[35869]: pgmap v603: 97 pgs: 97 active+clean; 43 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1015 KiB/s wr, 117 op/s 2024-07-26T07:54:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:56 smithi115 ceph-mon[35364]: pgmap v604: 97 pgs: 97 active+clean; 42 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 1.2 MiB/s wr, 192 op/s 2024-07-26T07:54:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:56 smithi181 ceph-mon[35869]: pgmap v604: 97 pgs: 97 active+clean; 42 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 1.2 MiB/s wr, 192 op/s 2024-07-26T07:54:56.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:56 smithi097 ceph-mon[32231]: pgmap v604: 97 pgs: 97 active+clean; 42 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 1.2 MiB/s wr, 192 op/s 2024-07-26T07:54:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:54:58 smithi115 ceph-mon[35364]: pgmap v605: 97 pgs: 97 active+clean; 43 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1004 KiB/s wr, 212 op/s 2024-07-26T07:54:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:54:58 smithi181 ceph-mon[35869]: pgmap v605: 97 pgs: 97 active+clean; 43 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1004 KiB/s wr, 212 op/s 2024-07-26T07:54:58.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:54:58 smithi097 ceph-mon[32231]: pgmap v605: 97 pgs: 97 active+clean; 43 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 1004 KiB/s wr, 212 op/s 2024-07-26T07:55:00.201 INFO:tasks.workunit.client.0.smithi097.stdout: 12 10944 70 5997 61 3541 231 2024-07-26T07:55:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:00 smithi115 ceph-mon[35364]: pgmap v606: 97 pgs: 97 active+clean; 43 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 918 KiB/s wr, 202 op/s 2024-07-26T07:55:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:00 smithi181 ceph-mon[35869]: pgmap v606: 97 pgs: 97 active+clean; 43 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 918 KiB/s wr, 202 op/s 2024-07-26T07:55:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:00 smithi097 ceph-mon[32231]: pgmap v606: 97 pgs: 97 active+clean; 43 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 918 KiB/s wr, 202 op/s 2024-07-26T07:55:02.302 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:02 smithi097 ceph-mon[32231]: pgmap v607: 97 pgs: 97 active+clean; 46 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.1 MiB/s wr, 227 op/s 2024-07-26T07:55:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:02 smithi115 ceph-mon[35364]: pgmap v607: 97 pgs: 97 active+clean; 46 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.1 MiB/s wr, 227 op/s 2024-07-26T07:55:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:02 smithi181 ceph-mon[35869]: pgmap v607: 97 pgs: 97 active+clean; 46 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.1 MiB/s wr, 227 op/s 2024-07-26T07:55:04.285 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:04 smithi181 ceph-mon[35869]: pgmap v608: 97 pgs: 97 active+clean; 46 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 794 KiB/s wr, 206 op/s 2024-07-26T07:55:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:04 smithi115 ceph-mon[35364]: pgmap v608: 97 pgs: 97 active+clean; 46 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 794 KiB/s wr, 206 op/s 2024-07-26T07:55:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:04 smithi097 ceph-mon[32231]: pgmap v608: 97 pgs: 97 active+clean; 46 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 794 KiB/s wr, 206 op/s 2024-07-26T07:55:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.548+0000 7fef35a00640 1 -- 172.21.15.97:0/2687625896 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef301052f0 msgr2=0x7fef30105760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.548+0000 7fef35a00640 1 --2- 172.21.15.97:0/2687625896 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef301052f0 0x7fef30105760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef20009860 tx=0x7fef2002a360 comp rx=0 tx=0).stop 2024-07-26T07:55:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.549+0000 7fef35a00640 1 -- 172.21.15.97:0/2687625896 shutdown_connections 2024-07-26T07:55:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.549+0000 7fef35a00640 1 --2- 172.21.15.97:0/2687625896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef30106d10 0x7fef3010d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.549+0000 7fef35a00640 1 --2- 172.21.15.97:0/2687625896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef30106370 0x7fef30106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.549+0000 7fef35a00640 1 --2- 172.21.15.97:0/2687625896 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef301052f0 0x7fef30105760 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.549+0000 7fef35a00640 1 -- 172.21.15.97:0/2687625896 >> 172.21.15.97:0/2687625896 conn(0x7fef30100c50 msgr2=0x7fef301030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.549+0000 7fef35a00640 1 -- 172.21.15.97:0/2687625896 shutdown_connections 2024-07-26T07:55:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.549+0000 7fef35a00640 1 -- 172.21.15.97:0/2687625896 wait complete. 2024-07-26T07:55:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.550+0000 7fef35a00640 1 Processor -- start 2024-07-26T07:55:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.551+0000 7fef35a00640 1 -- start start 2024-07-26T07:55:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.551+0000 7fef35a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef301052f0 0x7fef3019fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.551+0000 7fef35a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef30106370 0x7fef301a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef35a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef30106d10 0x7fef301a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef35a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fef30110e20 con 0x7fef30106370 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef35a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fef30110ca0 con 0x7fef301052f0 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef35a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fef30110fa0 con 0x7fef30106d10 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef2e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef30106370 0x7fef301a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef2e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef30106370 0x7fef301a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50406/0 (socket says 172.21.15.97:50406) 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef2e000640 1 -- 172.21.15.97:0/2765837885 learned_addr learned my addr 172.21.15.97:0/2765837885 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef2ea00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef301052f0 0x7fef3019fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.552+0000 7fef2f400640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef30106d10 0x7fef301a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.553+0000 7fef2f400640 1 -- 172.21.15.97:0/2765837885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef301052f0 msgr2=0x7fef3019fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.553+0000 7fef2f400640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef301052f0 0x7fef3019fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.553+0000 7fef2f400640 1 -- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef30106370 msgr2=0x7fef301a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:05.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.553+0000 7fef2f400640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef30106370 0x7fef301a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.553+0000 7fef2f400640 1 -- 172.21.15.97:0/2765837885 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef301a18e0 con 0x7fef30106d10 2024-07-26T07:55:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.553+0000 7fef2e000640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef30106370 0x7fef301a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:55:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.553+0000 7fef2f400640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef30106d10 0x7fef301a10c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef28009fd0 tx=0x7fef2800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.554+0000 7fef27400640 1 -- 172.21.15.97:0/2765837885 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2800ce20 con 0x7fef30106d10 2024-07-26T07:55:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.554+0000 7fef35a00640 1 -- 172.21.15.97:0/2765837885 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef301a1bd0 con 0x7fef30106d10 2024-07-26T07:55:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.554+0000 7fef35a00640 1 -- 172.21.15.97:0/2765837885 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef301a82c0 con 0x7fef30106d10 2024-07-26T07:55:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.554+0000 7fef27400640 1 -- 172.21.15.97:0/2765837885 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef28005020 con 0x7fef30106d10 2024-07-26T07:55:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.554+0000 7fef27400640 1 -- 172.21.15.97:0/2765837885 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28010690 con 0x7fef30106d10 2024-07-26T07:55:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.558+0000 7fef3c9db740 1 Processor -- start 2024-07-26T07:55:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.559+0000 7fef27400640 1 -- 172.21.15.97:0/2765837885 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fef28004b20 con 0x7fef30106d10 2024-07-26T07:55:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.561+0000 7fef3c9db740 1 -- start start 2024-07-26T07:55:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.561+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093ababee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.561+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abac550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.561+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 0x56093abad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.561+0000 7fef3c9db740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56093ab1e270 con 0x56093a7e54d0 2024-07-26T07:55:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.561+0000 7fef3c9db740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56093ab0d770 con 0x56093a7dd2f0 2024-07-26T07:55:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.561+0000 7fef3c9db740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56093ab117b0 con 0x56093a8061b0 2024-07-26T07:55:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.562+0000 7fef2e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093ababee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.562+0000 7fef2ea00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.562+0000 7fef2ea00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abac550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:47814/0 (socket says 172.21.15.97:47814) 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.562+0000 7fef2ea00640 1 -- 172.21.15.97:0/557941157 learned_addr learned my addr 172.21.15.97:0/557941157 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.562+0000 7fef2e000640 1 --2- 172.21.15.97:0/557941157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 0x56093abad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.562+0000 7fef27400640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fef140b3c50 0x7fef140b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.562+0000 7fef2e000640 1 -- 172.21.15.97:0/557941157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 msgr2=0x56093abac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.562+0000 7fef2e000640 1 --2- 172.21.15.97:0/557941157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abac550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.563+0000 7fef2e000640 1 -- 172.21.15.97:0/557941157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 msgr2=0x56093abad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.563+0000 7fef2e000640 1 --2- 172.21.15.97:0/557941157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 0x56093abad0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.563+0000 7fef2e000640 1 -- 172.21.15.97:0/557941157 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56093a871790 con 0x56093a8061b0 2024-07-26T07:55:05.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.563+0000 7fef2f400640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fef140b3c50 0x7fef140b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.563+0000 7fef2e000640 1 --2- 172.21.15.97:0/557941157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093ababee0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fef1c011bf0 tx=0x7fef1c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e56d88845e88bfa0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.563+0000 7fef27400640 1 -- 172.21.15.97:0/2765837885 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fef280e42e0 con 0x7fef30106d10 2024-07-26T07:55:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.564+0000 7fef1be00640 1 -- 172.21.15.97:0/557941157 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c00fc50 con 0x56093a8061b0 2024-07-26T07:55:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.564+0000 7fef2f400640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fef140b3c50 0x7fef140b6160 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fef2801a040 tx=0x7fef28005390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.564+0000 7fef1be00640 1 -- 172.21.15.97:0/557941157 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef1c00fdf0 con 0x56093a8061b0 2024-07-26T07:55:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.564+0000 7fef1be00640 1 -- 172.21.15.97:0/557941157 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c004c90 con 0x56093a8061b0 2024-07-26T07:55:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.565+0000 7fef3c9db740 1 -- 172.21.15.97:0/557941157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 msgr2=0x56093ababee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.565+0000 7fef3c9db740 1 --2- 172.21.15.97:0/557941157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093ababee0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fef1c011bf0 tx=0x7fef1c0046c0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.565+0000 7fef3c9db740 1 -- 172.21.15.97:0/557941157 shutdown_connections 2024-07-26T07:55:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.565+0000 7fef3c9db740 1 --2- 172.21.15.97:0/557941157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 0x56093abad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.565+0000 7fef3c9db740 1 --2- 172.21.15.97:0/557941157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.565+0000 7fef3c9db740 1 --2- 172.21.15.97:0/557941157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093ababee0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.565+0000 7fef3c9db740 1 -- 172.21.15.97:0/557941157 >> 172.21.15.97:0/557941157 conn(0x56093a7fa300 msgr2=0x56093aba7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.566+0000 7fef3c9db740 1 -- 172.21.15.97:0/557941157 shutdown_connections 2024-07-26T07:55:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.566+0000 7fef3c9db740 1 -- 172.21.15.97:0/557941157 wait complete. 2024-07-26T07:55:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.566+0000 7fef3c9db740 1 -- start start 2024-07-26T07:55:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.567+0000 7fef3c9db740 1 Processor -- start 2024-07-26T07:55:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.567+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7dd2f0 0x56093abb5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.567+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7e54d0 0x56093abb6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.567+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a8061b0 0x56093abb6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.567+0000 7fef3c9db740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56093aa47ab0 con 0x56093a7dd2f0 2024-07-26T07:55:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.567+0000 7fef3c9db740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56093aaaad50 con 0x56093a8061b0 2024-07-26T07:55:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.567+0000 7fef3c9db740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56093abadbc0 con 0x56093a7e54d0 2024-07-26T07:55:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.567+0000 7fef3c9db740 10 client.0 _ll_register_callbacks cb 0x56093a81da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.568+0000 7fef2e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7dd2f0 0x56093abb5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.568+0000 7fef2ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7e54d0 0x56093abb6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.568+0000 7fef2ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7e54d0 0x56093abb6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:36854/0 (socket says 172.21.15.97:36854) 2024-07-26T07:55:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.568+0000 7fef2ea00640 1 -- 172.21.15.97:0/1885459355 learned_addr learned my addr 172.21.15.97:0/1885459355 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.568+0000 7fef2e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7dd2f0 0x56093abb5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50420/0 (socket says 172.21.15.97:50420) 2024-07-26T07:55:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.569+0000 7fef2ea00640 1 -- 172.21.15.97:0/1885459355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a8061b0 msgr2=0x56093abb6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.569+0000 7fef2ea00640 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a8061b0 0x56093abb6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.569+0000 7fef2ea00640 1 -- 172.21.15.97:0/1885459355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7dd2f0 msgr2=0x56093abb5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.569+0000 7fef2ea00640 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7dd2f0 0x56093abb5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.569+0000 7fef2ea00640 1 -- 172.21.15.97:0/1885459355 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56093a8ea620 con 0x56093a7e54d0 2024-07-26T07:55:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.569+0000 7fef2e000640 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7dd2f0 0x56093abb5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.569+0000 7fef2ea00640 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7e54d0 0x56093abb6140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fef20035040 tx=0x7fef20030750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f320a5f3fc756c3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.570+0000 7fef18c00640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.570+0000 7fef18c00640 1 -- 172.21.15.97:0/1885459355 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20043070 con 0x56093a7e54d0 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.570+0000 7fef3c9db740 1 -- 172.21.15.97:0/1885459355 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef1c00fdf0 con 0x56093a7e54d0 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.570+0000 7fef18c00640 1 -- 172.21.15.97:0/1885459355 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fef20004450 con 0x56093a7e54d0 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef18c00640 1 -- 172.21.15.97:0/1885459355 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fef2004f050 con 0x56093a7e54d0 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef3c9db740 10 client.25390 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef3c9db740 1 -- 172.21.15.97:0/1885459355 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56093abd0cd0 con 0x56093a7e54d0 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef18c00640 1 -- 172.21.15.97:0/1885459355 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef1c00fdf0 con 0x56093a7e54d0 2024-07-26T07:55:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef18c00640 1 -- 172.21.15.97:0/1885459355 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fef2003d040 con 0x56093a7e54d0 2024-07-26T07:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef3c9db740 10 client.25390 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef3c9db740 10 client.25390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef3c9db740 20 client.25390 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56093a7c4d80 0x56093abd4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef3c9db740 4 client.25390 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.571+0000 7fef3c9db740 1 -- 172.21.15.97:0/1885459355 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56093ab201c0 con 0x56093a7c4d80 2024-07-26T07:55:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.572+0000 7fef2e000640 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56093a7c4d80 0x56093abd4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.573+0000 7fef18c00640 10 client.25390 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.573+0000 7fef2e000640 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56093a7c4d80 0x56093abd4510 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f34d018a3c101d3d server_cookie=a4dbc99f9fb79ce6 in_seq=0 out_seq=0 2024-07-26T07:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.574+0000 7fef18c00640 1 -- 172.21.15.97:0/1885459355 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56093ab201c0 con 0x56093a7c4d80 2024-07-26T07:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.574+0000 7fef18c00640 10 client.25390 handle_command_reply: tid=0 2024-07-26T07:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.574+0000 7fef3c9db740 1 client.25390 shutdown 2024-07-26T07:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef3c9db740 1 -- 172.21.15.97:0/1885459355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7e54d0 msgr2=0x56093abb6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7e54d0 0x56093abb6140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fef20035040 tx=0x7fef20030750 comp rx=0 tx=0).stop 2024-07-26T07:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef3c9db740 1 -- 172.21.15.97:0/1885459355 shutdown_connections 2024-07-26T07:55:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56093a7c4d80 0x56093abd4510 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a8061b0 0x56093abb6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7e54d0 0x56093abb6140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1885459355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7dd2f0 0x56093abb5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef3c9db740 1 -- 172.21.15.97:0/1885459355 >> 172.21.15.97:0/1885459355 conn(0x56093a7fa300 msgr2=0x56093aba60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.575+0000 7fef18c00640 0 client.25390 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.576+0000 7fef3c9db740 1 -- 172.21.15.97:0/1885459355 shutdown_connections 2024-07-26T07:55:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.576+0000 7fef3c9db740 1 -- 172.21.15.97:0/1885459355 wait complete. 2024-07-26T07:55:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.576+0000 7fef3c9db740 20 client.25390 trim_cache size 0 max 16384 2024-07-26T07:55:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.603+0000 7fef3c9db740 1 Processor -- start 2024-07-26T07:55:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.603+0000 7fef3c9db740 1 -- start start 2024-07-26T07:55:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.603+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7dd2f0 0x56093abb6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.603+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7e54d0 0x56093abb7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.603+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a8061b0 0x56093aba7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.603+0000 7fef3c9db740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fef301a87a0 con 0x56093a8061b0 2024-07-26T07:55:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.603+0000 7fef3c9db740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fef301aa3a0 con 0x56093a7e54d0 2024-07-26T07:55:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.603+0000 7fef3c9db740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fef301a8e70 con 0x56093a7dd2f0 2024-07-26T07:55:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.604+0000 7fef2e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a8061b0 0x56093aba7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.604+0000 7fef2e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a8061b0 0x56093aba7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50422/0 (socket says 172.21.15.97:50422) 2024-07-26T07:55:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.604+0000 7fef2e000640 1 -- 172.21.15.97:0/1952500484 learned_addr learned my addr 172.21.15.97:0/1952500484 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.604+0000 7fef2e000640 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7dd2f0 0x56093abb6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.604+0000 7fef2ea00640 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7e54d0 0x56093abb7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.605+0000 7fef2e000640 1 -- 172.21.15.97:0/1952500484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7e54d0 msgr2=0x56093abb7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.605+0000 7fef2e000640 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7e54d0 0x56093abb7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.605+0000 7fef2e000640 1 -- 172.21.15.97:0/1952500484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a8061b0 msgr2=0x56093aba7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.605+0000 7fef2e000640 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a8061b0 0x56093aba7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.605+0000 7fef2e000640 1 -- 172.21.15.97:0/1952500484 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56093aba3350 con 0x56093a7dd2f0 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.605+0000 7fef2e000640 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7dd2f0 0x56093abb6bf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fef1c004870 tx=0x7fef1c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1166d3c42616d628 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.605+0000 7fef1b400640 1 -- 172.21.15.97:0/1952500484 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c00fcd0 con 0x56093a7dd2f0 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.606+0000 7fef1b400640 1 -- 172.21.15.97:0/1952500484 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef1c00fe70 con 0x56093a7dd2f0 2024-07-26T07:55:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.606+0000 7fef1b400640 1 -- 172.21.15.97:0/1952500484 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c003e30 con 0x56093a7dd2f0 2024-07-26T07:55:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.606+0000 7fef3c9db740 1 -- 172.21.15.97:0/1952500484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7dd2f0 msgr2=0x56093abb6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.606+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7dd2f0 0x56093abb6bf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fef1c004870 tx=0x7fef1c0046c0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.607+0000 7fef3c9db740 1 -- 172.21.15.97:0/1952500484 shutdown_connections 2024-07-26T07:55:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.607+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a8061b0 0x56093aba7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.607+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7e54d0 0x56093abb7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.607+0000 7fef3c9db740 1 --2- 172.21.15.97:0/1952500484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a7dd2f0 0x56093abb6bf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.607+0000 7fef3c9db740 1 -- 172.21.15.97:0/1952500484 >> 172.21.15.97:0/1952500484 conn(0x56093a7fa300 msgr2=0x56093abace40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.607+0000 7fef3c9db740 1 -- 172.21.15.97:0/1952500484 shutdown_connections 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.607+0000 7fef3c9db740 1 -- 172.21.15.97:0/1952500484 wait complete. 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.608+0000 7fef3c9db740 1 -- start start 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.608+0000 7fef3c9db740 1 Processor -- start 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.608+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abba340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.608+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 0x56093abbaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.608+0000 7fef3c9db740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093abbb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.608+0000 7fef3c9db740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fef301a9c10 con 0x56093a7e54d0 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.608+0000 7fef3c9db740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fef301a9540 con 0x56093a7dd2f0 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.608+0000 7fef3c9db740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fef30105760 con 0x56093a8061b0 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.609+0000 7fef3c9db740 10 client.0 _ll_register_callbacks cb 0x7fef280093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.609+0000 7fef2e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093abbb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.609+0000 7fef2e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093abbb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:36864/0 (socket says 172.21.15.97:36864) 2024-07-26T07:55:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.609+0000 7fef2e000640 1 -- 172.21.15.97:0/3838772352 learned_addr learned my addr 172.21.15.97:0/3838772352 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.609+0000 7fef2e000640 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abba340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.610+0000 7fef2ea00640 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 0x56093abbaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.611+0000 7fef2e000640 1 -- 172.21.15.97:0/3838772352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 msgr2=0x56093abba340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.611+0000 7fef2e000640 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abba340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.611+0000 7fef2e000640 1 -- 172.21.15.97:0/3838772352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 msgr2=0x56093abbaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.611+0000 7fef2e000640 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 0x56093abbaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.611+0000 7fef2e000640 1 -- 172.21.15.97:0/3838772352 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56093abbbc80 con 0x56093a8061b0 2024-07-26T07:55:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.612+0000 7fef2e000640 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093abbb520 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009340 tx=0x7fef1c009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98f7ef0b97006446 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.612+0000 7fef18c00640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-07-26T07:55:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.613+0000 7fef18c00640 1 -- 172.21.15.97:0/3838772352 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c003d20 con 0x56093a8061b0 2024-07-26T07:55:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.613+0000 7fef18c00640 1 -- 172.21.15.97:0/3838772352 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fef1c03f040 con 0x56093a8061b0 2024-07-26T07:55:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.614+0000 7fef3c9db740 1 -- 172.21.15.97:0/3838772352 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef1c00fe70 con 0x56093a8061b0 2024-07-26T07:55:05.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.615+0000 7fef18c00640 1 -- 172.21.15.97:0/3838772352 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fef1c003d20 con 0x56093a8061b0 2024-07-26T07:55:05.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.615+0000 7fef3c9db740 10 client.25396 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:05.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.615+0000 7fef3c9db740 1 -- 172.21.15.97:0/3838772352 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56093abbcfa0 con 0x56093a8061b0 2024-07-26T07:55:05.617 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.616+0000 7fef18c00640 1 -- 172.21.15.97:0/3838772352 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef1c0035f0 con 0x56093a8061b0 2024-07-26T07:55:05.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.618+0000 7fef18c00640 1 -- 172.21.15.97:0/3838772352 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fef1c042020 con 0x56093a8061b0 2024-07-26T07:55:05.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.618+0000 7fef3c9db740 10 client.25396 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:05.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.618+0000 7fef3c9db740 10 client.25396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:55:05.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.618+0000 7fef3c9db740 20 client.25396 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:05.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.618+0000 7fef3c9db740 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56093a79a1c0 0x56093ac3ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.618+0000 7fef3c9db740 4 client.25396 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:55:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.618+0000 7fef3c9db740 1 -- 172.21.15.97:0/3838772352 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fef1c00fe70 con 0x56093a79a1c0 2024-07-26T07:55:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.619+0000 7fef2ea00640 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56093a79a1c0 0x56093ac3ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.621+0000 7fef2ea00640 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56093a79a1c0 0x56093ac3ecb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc9109d339f6e040 server_cookie=9a39ada7188b52a9 in_seq=0 out_seq=0 2024-07-26T07:55:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:05.621+0000 7fef18c00640 10 client.25396 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:06.056 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.054+0000 7fef18c00640 1 -- 172.21.15.97:0/3838772352 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fef1c00fe70 con 0x56093a79a1c0 2024-07-26T07:55:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.054+0000 7fef18c00640 10 client.25396 handle_command_reply: tid=0 2024-07-26T07:55:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.054+0000 7fef3c9db740 1 client.25396 shutdown 2024-07-26T07:55:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.054+0000 7fef3c9db740 1 -- 172.21.15.97:0/3838772352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 msgr2=0x56093abbb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.054+0000 7fef3c9db740 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093abbb520 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009340 tx=0x7fef1c009f50 comp rx=0 tx=0).stop 2024-07-26T07:55:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.054+0000 7fef3c9db740 1 -- 172.21.15.97:0/3838772352 shutdown_connections 2024-07-26T07:55:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.054+0000 7fef3c9db740 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56093a79a1c0 0x56093ac3ecb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.055+0000 7fef3c9db740 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56093a8061b0 0x56093abbb520 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.055+0000 7fef3c9db740 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56093a7e54d0 0x56093abbaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.055+0000 7fef3c9db740 1 --2- 172.21.15.97:0/3838772352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56093a7dd2f0 0x56093abba340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.055+0000 7fef3c9db740 1 -- 172.21.15.97:0/3838772352 >> 172.21.15.97:0/3838772352 conn(0x56093a7fa300 msgr2=0x56093abace40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.055+0000 7fef18c00640 0 client.25396 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.055+0000 7fef3c9db740 1 -- 172.21.15.97:0/3838772352 shutdown_connections 2024-07-26T07:55:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.055+0000 7fef3c9db740 1 -- 172.21.15.97:0/3838772352 wait complete. 2024-07-26T07:55:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.056+0000 7fef3c9db740 20 client.25396 trim_cache size 0 max 16384 2024-07-26T07:55:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:55:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:55:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:55:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:55:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.057+0000 7fef35a00640 1 -- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fef140b3c50 msgr2=0x7fef140b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.057+0000 7fef35a00640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fef140b3c50 0x7fef140b6160 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fef2801a040 tx=0x7fef28005390 comp rx=0 tx=0).stop 2024-07-26T07:55:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.057+0000 7fef35a00640 1 -- 172.21.15.97:0/2765837885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef30106d10 msgr2=0x7fef301a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.057+0000 7fef35a00640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef30106d10 0x7fef301a10c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef28009fd0 tx=0x7fef2800ef10 comp rx=0 tx=0).stop 2024-07-26T07:55:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.059+0000 7fef35a00640 1 -- 172.21.15.97:0/2765837885 shutdown_connections 2024-07-26T07:55:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.059+0000 7fef35a00640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fef140b3c50 0x7fef140b6160 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.059+0000 7fef35a00640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef30106d10 0x7fef301a10c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.059+0000 7fef35a00640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef30106370 0x7fef301a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.059+0000 7fef35a00640 1 --2- 172.21.15.97:0/2765837885 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef301052f0 0x7fef3019fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.059+0000 7fef35a00640 1 -- 172.21.15.97:0/2765837885 >> 172.21.15.97:0/2765837885 conn(0x7fef30100c50 msgr2=0x7fef30079240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.060+0000 7fef35a00640 1 -- 172.21.15.97:0/2765837885 shutdown_connections 2024-07-26T07:55:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:06.060+0000 7fef35a00640 1 -- 172.21.15.97:0/2765837885 wait complete. 2024-07-26T07:55:06.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:06 smithi115 ceph-mon[35364]: pgmap v609: 97 pgs: 97 active+clean; 48 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 924 KiB/s wr, 221 op/s 2024-07-26T07:55:06.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:55:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:06 smithi181 ceph-mon[35869]: pgmap v609: 97 pgs: 97 active+clean; 48 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 924 KiB/s wr, 221 op/s 2024-07-26T07:55:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:55:06.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:06 smithi097 ceph-mon[32231]: pgmap v609: 97 pgs: 97 active+clean; 48 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 924 KiB/s wr, 221 op/s 2024-07-26T07:55:06.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:55:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:55:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:55:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:55:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:55:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:55:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:55:07.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:55:07.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:55:07.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:55:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:08 smithi115 ceph-mon[35364]: pgmap v610: 97 pgs: 97 active+clean; 50 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 873 KiB/s wr, 135 op/s 2024-07-26T07:55:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:08 smithi181 ceph-mon[35869]: pgmap v610: 97 pgs: 97 active+clean; 50 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 873 KiB/s wr, 135 op/s 2024-07-26T07:55:08.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:08 smithi097 ceph-mon[32231]: pgmap v610: 97 pgs: 97 active+clean; 50 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 873 KiB/s wr, 135 op/s 2024-07-26T07:55:08.883 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 31.1 2024-07-26T07:55:08.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:55:09.479 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:55:10.201 INFO:tasks.workunit.client.0.smithi097.stdout: 13 10474 54 5853 84 3700 224 2024-07-26T07:55:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:10 smithi097 ceph-mon[32231]: pgmap v611: 97 pgs: 97 active+clean; 50 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 793 KiB/s wr, 87 op/s 2024-07-26T07:55:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:10 smithi115 ceph-mon[35364]: pgmap v611: 97 pgs: 97 active+clean; 50 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 793 KiB/s wr, 87 op/s 2024-07-26T07:55:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:10 smithi181 ceph-mon[35869]: pgmap v611: 97 pgs: 97 active+clean; 50 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 793 KiB/s wr, 87 op/s 2024-07-26T07:55:11.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.846+0000 7fcb9fe00640 1 -- 172.21.15.97:0/3275696504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077cc0 msgr2=0x7fcba0113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:11.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.846+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/3275696504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077cc0 0x7fcba0113a70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009900 tx=0x7fcb8c02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:55:11.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.846+0000 7fcb9fe00640 1 -- 172.21.15.97:0/3275696504 shutdown_connections 2024-07-26T07:55:11.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.846+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/3275696504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcba0113fb0 0x7fcba01163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:11.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.846+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/3275696504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077cc0 0x7fcba0113a70 secure :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009900 tx=0x7fcb8c02a3f0 comp rx=0 tx=0).stop 2024-07-26T07:55:11.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.846+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/3275696504 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcba0077370 0x7fcba0077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:11.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.846+0000 7fcb9fe00640 1 -- 172.21.15.97:0/3275696504 >> 172.21.15.97:0/3275696504 conn(0x7fcba0101060 msgr2=0x7fcba01034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:11.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.848+0000 7fcb9fe00640 1 -- 172.21.15.97:0/3275696504 shutdown_connections 2024-07-26T07:55:11.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.848+0000 7fcb9fe00640 1 -- 172.21.15.97:0/3275696504 wait complete. 2024-07-26T07:55:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.849+0000 7fcb9fe00640 1 Processor -- start 2024-07-26T07:55:11.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.850+0000 7fcb9fe00640 1 -- start start 2024-07-26T07:55:11.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.850+0000 7fcb9fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077370 0x7fcba01a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:11.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.850+0000 7fcb9fe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcba0113fb0 0x7fcba01a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:11.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.850+0000 7fcb9fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcba01a5be0 0x7fcba01a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:11.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.850+0000 7fcb9fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcba0119850 con 0x7fcba0077370 2024-07-26T07:55:11.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.850+0000 7fcb9fe00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fcba01196d0 con 0x7fcba0113fb0 2024-07-26T07:55:11.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.850+0000 7fcb9fe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcba01199d0 con 0x7fcba01a5be0 2024-07-26T07:55:11.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077370 0x7fcba01a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:11.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9e000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcba0113fb0 0x7fcba01a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:11.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077370 0x7fcba01a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54904/0 (socket says 172.21.15.97:54904) 2024-07-26T07:55:11.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9ea00640 1 -- 172.21.15.97:0/490439726 learned_addr learned my addr 172.21.15.97:0/490439726 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:11.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9f400640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcba01a5be0 0x7fcba01a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9ea00640 1 -- 172.21.15.97:0/490439726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcba01a5be0 msgr2=0x7fcba01a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9ea00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcba01a5be0 0x7fcba01a5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9ea00640 1 -- 172.21.15.97:0/490439726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcba0113fb0 msgr2=0x7fcba01a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9ea00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcba0113fb0 0x7fcba01a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.851+0000 7fcb9ea00640 1 -- 172.21.15.97:0/490439726 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba01a6730 con 0x7fcba0077370 2024-07-26T07:55:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.852+0000 7fcb9ea00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077370 0x7fcba01a0460 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000c920 tx=0x7fcb9000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.852+0000 7fcba4a00640 1 -- 172.21.15.97:0/490439726 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90007c70 con 0x7fcba0077370 2024-07-26T07:55:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.853+0000 7fcba4a00640 1 -- 172.21.15.97:0/490439726 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb90007e10 con 0x7fcba0077370 2024-07-26T07:55:11.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.853+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba01a6a20 con 0x7fcba0077370 2024-07-26T07:55:11.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.853+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba01075f0 con 0x7fcba0077370 2024-07-26T07:55:11.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.853+0000 7fcba4a00640 1 -- 172.21.15.97:0/490439726 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90018710 con 0x7fcba0077370 2024-07-26T07:55:11.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.854+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb800052c0 con 0x7fcba0077370 2024-07-26T07:55:11.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.858+0000 7fcba4a00640 1 -- 172.21.15.97:0/490439726 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fcb900189b0 con 0x7fcba0077370 2024-07-26T07:55:11.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.859+0000 7fcba4a00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb840b3d20 0x7fcb840b6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:11.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.860+0000 7fcba4a00640 1 -- 172.21.15.97:0/490439726 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fcb900e3ce0 con 0x7fcba0077370 2024-07-26T07:55:11.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.860+0000 7fcb9e000640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb840b3d20 0x7fcb840b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:11.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.861+0000 7fcb9e000640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb840b3d20 0x7fcb840b6230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c008000 tx=0x7fcb8c005a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:11.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:11.863+0000 7fcba4a00640 1 -- 172.21.15.97:0/490439726 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcb90014030 con 0x7fcba0077370 2024-07-26T07:55:12.227 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:12 smithi097 ceph-mon[32231]: pgmap v612: 97 pgs: 97 active+clean; 49 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.1 MiB/s wr, 193 op/s 2024-07-26T07:55:12.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.226+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb800055b0 con 0x7fcba0077370 2024-07-26T07:55:12.231 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:55:12.231 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":31,"btime":"2024-07-26T07:54:39:931393+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":25369,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3957930666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3957930666},{"type":"v1","addr":"172.21.15.181:6837","nonce":3957930666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:54:38.906083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14691":{"gid":14691,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:55:12.232 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 31 2024-07-26T07:55:12.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.227+0000 7fcba4a00640 1 -- 172.21.15.97:0/490439726 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11998 (secure 0 0 0) 0x7fcb900ab460 con 0x7fcba0077370 2024-07-26T07:55:12.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.230+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb840b3d20 msgr2=0x7fcb840b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:12.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.230+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb840b3d20 0x7fcb840b6230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c008000 tx=0x7fcb8c005a40 comp rx=0 tx=0).stop 2024-07-26T07:55:12.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.230+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077370 msgr2=0x7fcba01a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:12.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.230+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077370 0x7fcba01a0460 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000c920 tx=0x7fcb9000cde0 comp rx=0 tx=0).stop 2024-07-26T07:55:12.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.231+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 shutdown_connections 2024-07-26T07:55:12.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.231+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fcb840b3d20 0x7fcb840b6230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:12.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.231+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcba01a5be0 0x7fcba01a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:12.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.231+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcba0113fb0 0x7fcba01a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:12.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.231+0000 7fcb9fe00640 1 --2- 172.21.15.97:0/490439726 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba0077370 0x7fcba01a0460 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:12.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.231+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 >> 172.21.15.97:0/490439726 conn(0x7fcba0101060 msgr2=0x7fcba0103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:12.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.231+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 shutdown_connections 2024-07-26T07:55:12.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:12.231+0000 7fcb9fe00640 1 -- 172.21.15.97:0/490439726 wait complete. 2024-07-26T07:55:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:12 smithi115 ceph-mon[35364]: pgmap v612: 97 pgs: 97 active+clean; 49 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.1 MiB/s wr, 193 op/s 2024-07-26T07:55:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:12 smithi181 ceph-mon[35869]: pgmap v612: 97 pgs: 97 active+clean; 49 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.1 MiB/s wr, 193 op/s 2024-07-26T07:55:13.397 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:13 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/490439726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:55:13.452 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db / --timeout 90 --expiration 151.1 --await-for 45.0' 2024-07-26T07:55:13.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db / --timeout 90 --expiration 151.1 --await-for 45.0 2024-07-26T07:55:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:13 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/490439726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:55:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:13 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/490439726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:55:14.032 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:55:14.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:14 smithi097 ceph-mon[32231]: pgmap v613: 97 pgs: 97 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 792 KiB/s wr, 182 op/s 2024-07-26T07:55:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:14 smithi115 ceph-mon[35364]: pgmap v613: 97 pgs: 97 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 792 KiB/s wr, 182 op/s 2024-07-26T07:55:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:14 smithi181 ceph-mon[35869]: pgmap v613: 97 pgs: 97 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 792 KiB/s wr, 182 op/s 2024-07-26T07:55:16.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.329+0000 7fd820e00640 1 -- 172.21.15.97:0/2275588200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c1141b0 msgr2=0x7fd81c1165d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:16.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.329+0000 7fd820e00640 1 --2- 172.21.15.97:0/2275588200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c1141b0 0x7fd81c1165d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd814009ca0 tx=0x7fd81402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.329+0000 7fd820e00640 1 -- 172.21.15.97:0/2275588200 shutdown_connections 2024-07-26T07:55:16.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.329+0000 7fd820e00640 1 --2- 172.21.15.97:0/2275588200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c1141b0 0x7fd81c1165d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.329+0000 7fd820e00640 1 --2- 172.21.15.97:0/2275588200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd81c077cc0 0x7fd81c113c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.329+0000 7fd820e00640 1 --2- 172.21.15.97:0/2275588200 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd81c077370 0x7fd81c077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.329+0000 7fd820e00640 1 -- 172.21.15.97:0/2275588200 >> 172.21.15.97:0/2275588200 conn(0x7fd81c101200 msgr2=0x7fd81c103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:16.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.329+0000 7fd820e00640 1 -- 172.21.15.97:0/2275588200 shutdown_connections 2024-07-26T07:55:16.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.330+0000 7fd820e00640 1 -- 172.21.15.97:0/2275588200 wait complete. 2024-07-26T07:55:16.332 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:16 smithi097 ceph-mon[32231]: pgmap v614: 97 pgs: 97 active+clean; 51 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 887 KiB/s wr, 193 op/s 2024-07-26T07:55:16.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.331+0000 7fd820e00640 1 Processor -- start 2024-07-26T07:55:16.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.331+0000 7fd820e00640 1 -- start start 2024-07-26T07:55:16.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd820e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd81c077370 0x7fd81c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd820e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c077cc0 0x7fd81c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd820e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd81c1141b0 0x7fd81c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd820e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c119a30 con 0x7fd81c077cc0 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd820e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c1198b0 con 0x7fd81c077370 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd820e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c119bb0 con 0x7fd81c1141b0 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd819400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c077cc0 0x7fd81c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd819400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c077cc0 0x7fd81c06db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58610/0 (socket says 172.21.15.97:58610) 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd819400640 1 -- 172.21.15.97:0/1000171621 learned_addr learned my addr 172.21.15.97:0/1000171621 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd819e00640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd81c077370 0x7fd81c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.332+0000 7fd81a800640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd81c1141b0 0x7fd81c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.333+0000 7fd819400640 1 -- 172.21.15.97:0/1000171621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd81c1141b0 msgr2=0x7fd81c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.333+0000 7fd819400640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd81c1141b0 0x7fd81c06e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.333+0000 7fd819400640 1 -- 172.21.15.97:0/1000171621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd81c077370 msgr2=0x7fd81c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.333+0000 7fd819400640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd81c077370 0x7fd81c0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.333+0000 7fd819400640 1 -- 172.21.15.97:0/1000171621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81c06e8a0 con 0x7fd81c077cc0 2024-07-26T07:55:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.334+0000 7fd819400640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c077cc0 0x7fd81c06db70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd80400d750 tx=0x7fd80400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:16.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.334+0000 7fd812a00640 1 -- 172.21.15.97:0/1000171621 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804017070 con 0x7fd81c077cc0 2024-07-26T07:55:16.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.334+0000 7fd812a00640 1 -- 172.21.15.97:0/1000171621 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8040054b0 con 0x7fd81c077cc0 2024-07-26T07:55:16.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.335+0000 7fd812a00640 1 -- 172.21.15.97:0/1000171621 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804004bc0 con 0x7fd81c077cc0 2024-07-26T07:55:16.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.335+0000 7fd820e00640 1 -- 172.21.15.97:0/1000171621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd81c073570 con 0x7fd81c077cc0 2024-07-26T07:55:16.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.335+0000 7fd820e00640 1 -- 172.21.15.97:0/1000171621 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd81c073a30 con 0x7fd81c077cc0 2024-07-26T07:55:16.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.341+0000 7fd827a7c740 1 Processor -- start 2024-07-26T07:55:16.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.341+0000 7fd812a00640 1 -- 172.21.15.97:0/1000171621 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fd804009c30 con 0x7fd81c077cc0 2024-07-26T07:55:16.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.342+0000 7fd827a7c740 1 -- start start 2024-07-26T07:55:16.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.342+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 0x56421f8b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.342+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.342+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.342+0000 7fd827a7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56421f87ccf0 con 0x56421f4ebb60 2024-07-26T07:55:16.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.342+0000 7fd827a7c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56421f827b40 con 0x56421f4e3980 2024-07-26T07:55:16.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.342+0000 7fd827a7c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56421f817040 con 0x56421f50c840 2024-07-26T07:55:16.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.343+0000 7fd819400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.343+0000 7fd819400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8b2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58620/0 (socket says 172.21.15.97:58620) 2024-07-26T07:55:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.343+0000 7fd819400640 1 -- 172.21.15.97:0/1468708484 learned_addr learned my addr 172.21.15.97:0/1468708484 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.343+0000 7fd81a800640 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 0x56421f8b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.343+0000 7fd812a00640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd8000b3c50 0x7fd8000b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.344+0000 7fd812a00640 1 -- 172.21.15.97:0/1000171621 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd8040e3570 con 0x7fd81c077cc0 2024-07-26T07:55:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.344+0000 7fd819e00640 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.344+0000 7fd81a800640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd8000b3c50 0x7fd8000b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.345+0000 7fd819400640 1 -- 172.21.15.97:0/1468708484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 msgr2=0x56421f8b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:16.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.345+0000 7fd819400640 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 0x56421f8b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.345+0000 7fd819400640 1 -- 172.21.15.97:0/1468708484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 msgr2=0x56421f8b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:16.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.345+0000 7fd819400640 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.345+0000 7fd819400640 1 -- 172.21.15.97:0/1468708484 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421f79eaf0 con 0x56421f4ebb60 2024-07-26T07:55:16.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.345+0000 7fd819400640 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8b2c20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd8040e0f50 tx=0x7fd8040e6ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefb0eda4d2f7f76 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:16.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.346+0000 7fd80b400640 1 -- 172.21.15.97:0/1468708484 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804020030 con 0x56421f4ebb60 2024-07-26T07:55:16.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.346+0000 7fd80b400640 1 -- 172.21.15.97:0/1468708484 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8040a9f70 con 0x56421f4ebb60 2024-07-26T07:55:16.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.346+0000 7fd80b400640 1 -- 172.21.15.97:0/1468708484 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8040b3510 con 0x56421f4ebb60 2024-07-26T07:55:16.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.346+0000 7fd81a800640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd8000b3c50 0x7fd8000b6160 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd814006010 tx=0x7fd81402a8c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:16.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.347+0000 7fd827a7c740 1 -- 172.21.15.97:0/1468708484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 msgr2=0x56421f8b2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:16.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.347+0000 7fd827a7c740 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8b2c20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd8040e0f50 tx=0x7fd8040e6ea0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.348+0000 7fd827a7c740 1 -- 172.21.15.97:0/1468708484 shutdown_connections 2024-07-26T07:55:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.348+0000 7fd827a7c740 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8b2c20 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.348+0000 7fd827a7c740 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.348+0000 7fd827a7c740 1 --2- 172.21.15.97:0/1468708484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 0x56421f8b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.348+0000 7fd827a7c740 1 -- 172.21.15.97:0/1468708484 >> 172.21.15.97:0/1468708484 conn(0x56421f500990 msgr2=0x56421f8b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:16.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.348+0000 7fd827a7c740 1 -- 172.21.15.97:0/1468708484 shutdown_connections 2024-07-26T07:55:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.348+0000 7fd827a7c740 1 -- 172.21.15.97:0/1468708484 wait complete. 2024-07-26T07:55:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.349+0000 7fd827a7c740 1 -- start start 2024-07-26T07:55:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.349+0000 7fd827a7c740 1 Processor -- start 2024-07-26T07:55:16.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.350+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8bdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8be1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 0x56421f8c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd827a7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56421f7b3500 con 0x56421f4ebb60 2024-07-26T07:55:16.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd827a7c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56421f81b080 con 0x56421f4e3980 2024-07-26T07:55:16.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd827a7c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56421f67a7a0 con 0x56421f50c840 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd819400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8be1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd819e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8bdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd827a7c740 10 client.0 _ll_register_callbacks cb 0x56421f80fbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd819e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8bdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:42140/0 (socket says 172.21.15.97:42140) 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.351+0000 7fd819e00640 1 -- 172.21.15.97:0/3618347381 learned_addr learned my addr 172.21.15.97:0/3618347381 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.352+0000 7fd81a800640 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 0x56421f8c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.352+0000 7fd819e00640 1 -- 172.21.15.97:0/3618347381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 msgr2=0x56421f8c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.352+0000 7fd819e00640 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 0x56421f8c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.352+0000 7fd819e00640 1 -- 172.21.15.97:0/3618347381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 msgr2=0x56421f8be1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:16.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.352+0000 7fd819e00640 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8be1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.352+0000 7fd819e00640 1 -- 172.21.15.97:0/3618347381 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421f586b90 con 0x56421f4e3980 2024-07-26T07:55:16.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.352+0000 7fd819400640 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8be1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:55:16.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.353+0000 7fd819e00640 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8bdca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00ce00 tx=0x7fd80c0039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fb716cd5b9928ae server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:16.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.353+0000 7fd7ffe00640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:55:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.353+0000 7fd7ffe00640 1 -- 172.21.15.97:0/3618347381 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c011970 con 0x56421f4e3980 2024-07-26T07:55:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.353+0000 7fd827a7c740 1 -- 172.21.15.97:0/3618347381 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8040a9f70 con 0x56421f4e3980 2024-07-26T07:55:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.354+0000 7fd7ffe00640 1 -- 172.21.15.97:0/3618347381 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd80c011b10 con 0x56421f4e3980 2024-07-26T07:55:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.354+0000 7fd7ffe00640 1 -- 172.21.15.97:0/3618347381 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd80c011d70 con 0x56421f4e3980 2024-07-26T07:55:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.354+0000 7fd827a7c740 10 client.35360 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:16.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.354+0000 7fd827a7c740 1 -- 172.21.15.97:0/3618347381 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56421f8cc7b0 con 0x56421f4e3980 2024-07-26T07:55:16.356 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.355+0000 7fd7ffe00640 1 -- 172.21.15.97:0/3618347381 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8040a9f70 con 0x56421f4e3980 2024-07-26T07:55:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.355+0000 7fd7ffe00640 1 -- 172.21.15.97:0/3618347381 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fd80c018070 con 0x56421f4e3980 2024-07-26T07:55:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.355+0000 7fd827a7c740 10 client.35360 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.355+0000 7fd827a7c740 10 client.35360 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:55:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.355+0000 7fd827a7c740 20 client.35360 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.355+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56421f4cb1c0 0x56421f8cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.355+0000 7fd827a7c740 4 client.35360 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:55:16.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.355+0000 7fd827a7c740 1 -- 172.21.15.97:0/3618347381 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56421f829a90 con 0x56421f4cb1c0 2024-07-26T07:55:16.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.356+0000 7fd819400640 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56421f4cb1c0 0x56421f8cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.357+0000 7fd819400640 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56421f4cb1c0 0x56421f8cfda0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f9caaa4ae9cd94f server_cookie=f558aae5cb18e649 in_seq=0 out_seq=0 2024-07-26T07:55:16.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.357+0000 7fd7ffe00640 10 client.35360 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:16.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.359+0000 7fd7ffe00640 1 -- 172.21.15.97:0/3618347381 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56421f829a90 con 0x56421f4cb1c0 2024-07-26T07:55:16.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.359+0000 7fd7ffe00640 10 client.35360 handle_command_reply: tid=0 2024-07-26T07:55:16.360 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.359+0000 7fd827a7c740 1 client.35360 shutdown 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.359+0000 7fd827a7c740 1 -- 172.21.15.97:0/3618347381 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 msgr2=0x56421f8bdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.359+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8bdca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00ce00 tx=0x7fd80c0039c0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd827a7c740 1 -- 172.21.15.97:0/3618347381 shutdown_connections 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56421f4cb1c0 0x56421f8cfda0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd7ffe00640 0 client.35360 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f50c840 0x56421f8c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8be1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3618347381 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8bdca0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd827a7c740 1 -- 172.21.15.97:0/3618347381 >> 172.21.15.97:0/3618347381 conn(0x56421f500990 msgr2=0x56421f8b5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:16.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd827a7c740 1 -- 172.21.15.97:0/3618347381 shutdown_connections 2024-07-26T07:55:16.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.360+0000 7fd827a7c740 1 -- 172.21.15.97:0/3618347381 wait complete. 2024-07-26T07:55:16.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.361+0000 7fd827a7c740 20 client.35360 trim_cache size 0 max 16384 2024-07-26T07:55:16.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.387+0000 7fd827a7c740 1 Processor -- start 2024-07-26T07:55:16.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.387+0000 7fd827a7c740 1 -- start start 2024-07-26T07:55:16.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8af880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd827a7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c1bdd00 con 0x56421f4ebb60 2024-07-26T07:55:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd827a7c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c1bf900 con 0x56421f4e3980 2024-07-26T07:55:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd827a7c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c1be3d0 con 0x56421f4cb1c0 2024-07-26T07:55:16.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd81a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd81a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8af880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58638/0 (socket says 172.21.15.97:58638) 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.388+0000 7fd81a800640 1 -- 172.21.15.97:0/3316518985 learned_addr learned my addr 172.21.15.97:0/3316518985 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.389+0000 7fd819e00640 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.389+0000 7fd819400640 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.389+0000 7fd81a800640 1 -- 172.21.15.97:0/3316518985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 msgr2=0x56421f8b82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.390+0000 7fd81a800640 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8b82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.390+0000 7fd81a800640 1 -- 172.21.15.97:0/3316518985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 msgr2=0x56421f8aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.390+0000 7fd81a800640 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.390+0000 7fd81a800640 1 -- 172.21.15.97:0/3316518985 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421f8b4490 con 0x56421f4ebb60 2024-07-26T07:55:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.390+0000 7fd819e00640 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8b82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:55:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.390+0000 7fd819400640 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:55:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.390+0000 7fd81a800640 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8af880 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd814009ca0 tx=0x7fd814009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deede57e2d37873a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.391+0000 7fd80aa00640 1 -- 172.21.15.97:0/3316518985 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814045070 con 0x56421f4ebb60 2024-07-26T07:55:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.391+0000 7fd80aa00640 1 -- 172.21.15.97:0/3316518985 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd814036040 con 0x56421f4ebb60 2024-07-26T07:55:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.391+0000 7fd80aa00640 1 -- 172.21.15.97:0/3316518985 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814030820 con 0x56421f4ebb60 2024-07-26T07:55:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 -- 172.21.15.97:0/3316518985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 msgr2=0x56421f8af880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8af880 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd814009ca0 tx=0x7fd814009870 comp rx=0 tx=0).stop 2024-07-26T07:55:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 -- 172.21.15.97:0/3316518985 shutdown_connections 2024-07-26T07:55:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8af880 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 --2- 172.21.15.97:0/3316518985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8b82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 -- 172.21.15.97:0/3316518985 >> 172.21.15.97:0/3316518985 conn(0x56421f500990 msgr2=0x56421f8bc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 -- 172.21.15.97:0/3316518985 shutdown_connections 2024-07-26T07:55:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.393+0000 7fd827a7c740 1 -- 172.21.15.97:0/3316518985 wait complete. 2024-07-26T07:55:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.394+0000 7fd827a7c740 1 -- start start 2024-07-26T07:55:16.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.394+0000 7fd827a7c740 1 Processor -- start 2024-07-26T07:55:16.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.395+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.395+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.395+0000 7fd827a7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.395+0000 7fd827a7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c1bf170 con 0x56421f4ebb60 2024-07-26T07:55:16.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.395+0000 7fd827a7c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c1beaa0 con 0x56421f4e3980 2024-07-26T07:55:16.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.395+0000 7fd827a7c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c1bbce0 con 0x56421f4cb1c0 2024-07-26T07:55:16.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.395+0000 7fd827a7c740 10 client.0 _ll_register_callbacks cb 0x7fd80c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:16.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.395+0000 7fd819e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd819e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:51968/0 (socket says 172.21.15.97:51968) 2024-07-26T07:55:16.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd819e00640 1 -- 172.21.15.97:0/4064735021 learned_addr learned my addr 172.21.15.97:0/4064735021 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd81a800640 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd819400640 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd819e00640 1 -- 172.21.15.97:0/4064735021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 msgr2=0x56421f8c17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd819e00640 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8c17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd819e00640 1 -- 172.21.15.97:0/4064735021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 msgr2=0x56421f8c1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd819e00640 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8c1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:16.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.396+0000 7fd819e00640 1 -- 172.21.15.97:0/4064735021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421f937530 con 0x56421f4cb1c0 2024-07-26T07:55:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.397+0000 7fd819e00640 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8c62d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00ce00 tx=0x7fd80c019d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7166d0e864a3a9d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.397+0000 7fd7ffe00640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-07-26T07:55:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.397+0000 7fd7ffe00640 1 -- 172.21.15.97:0/4064735021 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c029070 con 0x56421f4cb1c0 2024-07-26T07:55:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.397+0000 7fd827a7c740 1 -- 172.21.15.97:0/4064735021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd814036040 con 0x56421f4cb1c0 2024-07-26T07:55:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.397+0000 7fd7ffe00640 1 -- 172.21.15.97:0/4064735021 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd80c005030 con 0x56421f4cb1c0 2024-07-26T07:55:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.398+0000 7fd7ffe00640 1 -- 172.21.15.97:0/4064735021 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd80c000e60 con 0x56421f4cb1c0 2024-07-26T07:55:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.398+0000 7fd827a7c740 10 client.25414 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.398+0000 7fd827a7c740 1 -- 172.21.15.97:0/4064735021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56421f938660 con 0x56421f4cb1c0 2024-07-26T07:55:16.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.398+0000 7fd7ffe00640 1 -- 172.21.15.97:0/4064735021 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd814036040 con 0x56421f4cb1c0 2024-07-26T07:55:16.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.399+0000 7fd7ffe00640 1 -- 172.21.15.97:0/4064735021 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fd80c018070 con 0x56421f4cb1c0 2024-07-26T07:55:16.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.399+0000 7fd827a7c740 10 client.25414 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:16.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.399+0000 7fd827a7c740 10 client.25414 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:55:16.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.399+0000 7fd827a7c740 20 client.25414 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:16.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.399+0000 7fd827a7c740 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56421f50c840 0x56421f93b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:16.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.399+0000 7fd827a7c740 4 client.25414 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 151.1, "await_for": 45.0}] 2024-07-26T07:55:16.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.399+0000 7fd827a7c740 1 -- 172.21.15.97:0/4064735021 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 151.1, "await_for": 45.0}) -- 0x56421f825530 con 0x56421f50c840 2024-07-26T07:55:16.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.401+0000 7fd819400640 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56421f50c840 0x56421f93b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:16.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.402+0000 7fd819400640 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56421f50c840 0x56421f93b000 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1af35fe854aa11c server_cookie=2ba2be48e46e200 in_seq=0 out_seq=0 2024-07-26T07:55:16.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:16.402+0000 7fd7ffe00640 10 client.25414 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:16 smithi115 ceph-mon[35364]: pgmap v614: 97 pgs: 97 active+clean; 51 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 887 KiB/s wr, 193 op/s 2024-07-26T07:55:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:16 smithi181 ceph-mon[35869]: pgmap v614: 97 pgs: 97 active+clean; 51 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 887 KiB/s wr, 193 op/s 2024-07-26T07:55:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:18 smithi115 ceph-mon[35364]: pgmap v615: 97 pgs: 97 active+clean; 53 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 929 KiB/s wr, 190 op/s 2024-07-26T07:55:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:18 smithi181 ceph-mon[35869]: pgmap v615: 97 pgs: 97 active+clean; 53 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 929 KiB/s wr, 190 op/s 2024-07-26T07:55:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:18 smithi097 ceph-mon[32231]: pgmap v615: 97 pgs: 97 active+clean; 53 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 929 KiB/s wr, 190 op/s 2024-07-26T07:55:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 -- 172.21.15.97:0/2779391628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f988c106b40 msgr2=0x7f988c10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 --2- 172.21.15.97:0/2779391628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f988c106b40 0x7f988c10d4e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9884009ca0 tx=0x7f988402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 -- 172.21.15.97:0/2779391628 shutdown_connections 2024-07-26T07:55:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 --2- 172.21.15.97:0/2779391628 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f988c106b40 0x7f988c10d4e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 --2- 172.21.15.97:0/2779391628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988c1061a0 0x7f988c1065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 --2- 172.21.15.97:0/2779391628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f988c105120 0x7f988c105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 -- 172.21.15.97:0/2779391628 >> 172.21.15.97:0/2779391628 conn(0x7f988c100a80 msgr2=0x7f988c102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 -- 172.21.15.97:0/2779391628 shutdown_connections 2024-07-26T07:55:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.293+0000 7f9892600640 1 -- 172.21.15.97:0/2779391628 wait complete. 2024-07-26T07:55:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.295+0000 7f9892600640 1 Processor -- start 2024-07-26T07:55:19.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.295+0000 7f9892600640 1 -- start start 2024-07-26T07:55:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.295+0000 7f9892600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f988c105120 0x7f988c19b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.295+0000 7f9892600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988c1061a0 0x7f988c19bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.295+0000 7f9892600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f988c106b40 0x7f988c19cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.295+0000 7f9892600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f988c110c00 con 0x7f988c105120 2024-07-26T07:55:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.295+0000 7f9892600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f988c110a80 con 0x7f988c106b40 2024-07-26T07:55:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.295+0000 7f9892600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f988c110d80 con 0x7f988c1061a0 2024-07-26T07:55:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f988be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988c1061a0 0x7f988c19bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f9891200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f988c105120 0x7f988c19b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f988be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988c1061a0 0x7f988c19bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:51996/0 (socket says 172.21.15.97:51996) 2024-07-26T07:55:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f988be00640 1 -- 172.21.15.97:0/3642780670 learned_addr learned my addr 172.21.15.97:0/3642780670 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f9891c00640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f988c106b40 0x7f988c19cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f988be00640 1 -- 172.21.15.97:0/3642780670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f988c106b40 msgr2=0x7f988c19cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f988be00640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f988c106b40 0x7f988c19cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f988be00640 1 -- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f988c105120 msgr2=0x7f988c19b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f988be00640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f988c105120 0x7f988c19b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.296+0000 7f988be00640 1 -- 172.21.15.97:0/3642780670 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c19d2f0 con 0x7f988c1061a0 2024-07-26T07:55:19.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.297+0000 7f988be00640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988c1061a0 0x7f988c19bf20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f987800d950 tx=0x7f987800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:19.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.297+0000 7f9889600640 1 -- 172.21.15.97:0/3642780670 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878017070 con 0x7f988c1061a0 2024-07-26T07:55:19.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.297+0000 7f9892600640 1 -- 172.21.15.97:0/3642780670 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c19d5e0 con 0x7f988c1061a0 2024-07-26T07:55:19.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.297+0000 7f9889600640 1 -- 172.21.15.97:0/3642780670 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9878009c30 con 0x7f988c1061a0 2024-07-26T07:55:19.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.297+0000 7f9889600640 1 -- 172.21.15.97:0/3642780670 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987800f620 con 0x7f988c1061a0 2024-07-26T07:55:19.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.297+0000 7f9892600640 1 -- 172.21.15.97:0/3642780670 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c1a3cd0 con 0x7f988c1061a0 2024-07-26T07:55:19.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.300+0000 7f989ada7740 1 Processor -- start 2024-07-26T07:55:19.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f989ada7740 1 -- start start 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f989ada7740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb38f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f989ada7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb38f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f989ada7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 0x562feb3902a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f989ada7740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562feb301270 con 0x562feafe91b0 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f989ada7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562feb2f0770 con 0x562feafc02f0 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f989ada7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562feb2f47b0 con 0x562feafc84d0 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f9891c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb38f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f9891c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb38f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58676/0 (socket says 172.21.15.97:58676) 2024-07-26T07:55:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f9891c00640 1 -- 172.21.15.97:0/2392530635 learned_addr learned my addr 172.21.15.97:0/2392530635 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:19.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.301+0000 7f9891200640 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb38f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f9891c00640 1 -- 172.21.15.97:0/2392530635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 msgr2=0x562feb3902a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:55:19.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f9891c00640 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 0x562feb3902a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f9891c00640 1 -- 172.21.15.97:0/2392530635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 msgr2=0x562feb38f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:19.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f9891c00640 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb38f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f9891c00640 1 -- 172.21.15.97:0/2392530635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562feb054790 con 0x562feafe91b0 2024-07-26T07:55:19.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f9891200640 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb38f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:55:19.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f9891c00640 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb38f0e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9884035000 tx=0x7f9884009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdcfd590d877ea3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:19.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f987e000640 1 -- 172.21.15.97:0/2392530635 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884031a80 con 0x562feafe91b0 2024-07-26T07:55:19.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f987e000640 1 -- 172.21.15.97:0/2392530635 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9884036070 con 0x562feafe91b0 2024-07-26T07:55:19.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.302+0000 7f987e000640 1 -- 172.21.15.97:0/2392530635 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98840045a0 con 0x562feafe91b0 2024-07-26T07:55:19.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 -- 172.21.15.97:0/2392530635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 msgr2=0x562feb38f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb38f0e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9884035000 tx=0x7f9884009930 comp rx=0 tx=0).stop 2024-07-26T07:55:19.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f9889600640 1 -- 172.21.15.97:0/3642780670 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f9878010460 con 0x7f988c1061a0 2024-07-26T07:55:19.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 -- 172.21.15.97:0/2392530635 shutdown_connections 2024-07-26T07:55:19.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 0x562feb3902a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb38f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 --2- 172.21.15.97:0/2392530635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb38f0e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 -- 172.21.15.97:0/2392530635 >> 172.21.15.97:0/2392530635 conn(0x562feafdd300 msgr2=0x562feb38ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 -- 172.21.15.97:0/2392530635 shutdown_connections 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.303+0000 7f989ada7740 1 -- 172.21.15.97:0/2392530635 wait complete. 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.304+0000 7f989ada7740 1 -- start start 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.304+0000 7f989ada7740 1 Processor -- start 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f989ada7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb398aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f989ada7740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb399160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f989ada7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb399cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f989ada7740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562feb22aab0 con 0x562feafc84d0 2024-07-26T07:55:19.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f989ada7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562feb28dd50 con 0x562feafc02f0 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f989ada7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562feb390be0 con 0x562feafe91b0 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f989ada7740 10 client.0 _ll_register_callbacks cb 0x562feb000a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f9891200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb399160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f9891c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb398aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.305+0000 7f9891200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb399160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58682/0 (socket says 172.21.15.97:58682) 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9891200640 1 -- 172.21.15.97:0/1593836457 learned_addr learned my addr 172.21.15.97:0/1593836457 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f988be00640 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb399cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9889600640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f98740b3ae0 0x7f98740b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9889600640 1 -- 172.21.15.97:0/3642780670 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f98780e37b0 con 0x7f988c1061a0 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9891c00640 1 -- 172.21.15.97:0/1593836457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 msgr2=0x562feb399cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9891c00640 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb399cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9891c00640 1 -- 172.21.15.97:0/1593836457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 msgr2=0x562feb399160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9891c00640 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb399160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9891c00640 1 -- 172.21.15.97:0/1593836457 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562feb0cd620 con 0x562feafc02f0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9891200640 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb399160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f988be00640 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb399cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.306+0000 7f9891c00640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f98740b3ae0 0x7f98740b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.307+0000 7f9891c00640 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb398aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9884004af0 tx=0x7f98840043e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6610655fe4a1f0e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.307+0000 7f9872a00640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.307+0000 7f9872a00640 1 -- 172.21.15.97:0/1593836457 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98840045d0 con 0x562feafc02f0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.307+0000 7f9872a00640 1 -- 172.21.15.97:0/1593836457 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f9884003cc0 con 0x562feafc02f0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.307+0000 7f989ada7740 1 -- 172.21.15.97:0/1593836457 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9884036070 con 0x562feafc02f0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.307+0000 7f9891c00640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f98740b3ae0 0x7f98740b5ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x562feb39a380 tx=0x7f9884054000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.307+0000 7f9872a00640 1 -- 172.21.15.97:0/1593836457 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9884060020 con 0x562feafc02f0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.307+0000 7f989ada7740 10 client.35378 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f989ada7740 1 -- 172.21.15.97:0/1593836457 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562feb3a2c20 con 0x562feafc02f0 2024-07-26T07:55:19.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f9872a00640 1 -- 172.21.15.97:0/1593836457 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9884036070 con 0x562feafc02f0 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f9872a00640 1 -- 172.21.15.97:0/1593836457 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f9884043020 con 0x562feafc02f0 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f989ada7740 10 client.35378 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f989ada7740 10 client.35378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f989ada7740 20 client.35378 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f989ada7740 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562feafa7d80 0x562feb3a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f989ada7740 4 client.35378 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f989ada7740 1 -- 172.21.15.97:0/1593836457 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562feb3031c0 con 0x562feafa7d80 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.308+0000 7f9891200640 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562feafa7d80 0x562feb3a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.309+0000 7f9891200640 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562feafa7d80 0x562feb3a61c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4266e975db37e72c server_cookie=ddefb57b512c1c57 in_seq=0 out_seq=0 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.309+0000 7f9872a00640 10 client.35378 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f9872a00640 1 -- 172.21.15.97:0/1593836457 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562feb3031c0 con 0x562feafa7d80 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f9872a00640 10 client.35378 handle_command_reply: tid=0 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 client.35378 shutdown 2024-07-26T07:55:19.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 -- 172.21.15.97:0/1593836457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 msgr2=0x562feb398aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb398aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9884004af0 tx=0x7f98840043e0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 -- 172.21.15.97:0/1593836457 shutdown_connections 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562feafa7d80 0x562feb3a61c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb399cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb399160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f9872a00640 0 client.35378 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 --2- 172.21.15.97:0/1593836457 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb398aa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 -- 172.21.15.97:0/1593836457 >> 172.21.15.97:0/1593836457 conn(0x562feafdd300 msgr2=0x562feb3890d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 -- 172.21.15.97:0/1593836457 shutdown_connections 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 1 -- 172.21.15.97:0/1593836457 wait complete. 2024-07-26T07:55:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.314+0000 7f989ada7740 20 client.35378 trim_cache size 0 max 16384 2024-07-26T07:55:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.331+0000 7f989ada7740 1 Processor -- start 2024-07-26T07:55:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.331+0000 7f989ada7740 1 -- start start 2024-07-26T07:55:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.331+0000 7f989ada7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb385920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.331+0000 7f989ada7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 0x562feb397030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.331+0000 7f989ada7740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb3890d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.331+0000 7f989ada7740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f988c1ac980 con 0x562feafe91b0 2024-07-26T07:55:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.331+0000 7f989ada7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f988c1ae580 con 0x562feafc02f0 2024-07-26T07:55:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.331+0000 7f989ada7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f988c1ad050 con 0x562feafc84d0 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb3890d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f988be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 0x562feb397030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb3890d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58694/0 (socket says 172.21.15.97:58694) 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891c00640 1 -- 172.21.15.97:0/2231666543 learned_addr learned my addr 172.21.15.97:0/2231666543 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891200640 1 --2- 172.21.15.97:0/2231666543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb385920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891c00640 1 -- 172.21.15.97:0/2231666543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 msgr2=0x562feb397030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891c00640 1 --2- 172.21.15.97:0/2231666543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 0x562feb397030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891c00640 1 -- 172.21.15.97:0/2231666543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 msgr2=0x562feb385920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891c00640 1 --2- 172.21.15.97:0/2231666543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb385920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.332+0000 7f9891c00640 1 -- 172.21.15.97:0/2231666543 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562feb39a040 con 0x562feafe91b0 2024-07-26T07:55:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.333+0000 7f9891c00640 1 --2- 172.21.15.97:0/2231666543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb3890d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f98840094e0 tx=0x7f9884009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389f2d68d24098bb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.333+0000 7f987d600640 1 -- 172.21.15.97:0/2231666543 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884003cc0 con 0x562feafe91b0 2024-07-26T07:55:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.333+0000 7f987d600640 1 -- 172.21.15.97:0/2231666543 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f988404a070 con 0x562feafe91b0 2024-07-26T07:55:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.333+0000 7f987d600640 1 -- 172.21.15.97:0/2231666543 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884045e20 con 0x562feafe91b0 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 -- 172.21.15.97:0/2231666543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 msgr2=0x562feb3890d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 --2- 172.21.15.97:0/2231666543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb3890d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f98840094e0 tx=0x7f9884009360 comp rx=0 tx=0).stop 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 -- 172.21.15.97:0/2231666543 shutdown_connections 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 --2- 172.21.15.97:0/2231666543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafe91b0 0x562feb3890d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 --2- 172.21.15.97:0/2231666543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafc84d0 0x562feb397030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 --2- 172.21.15.97:0/2231666543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb385920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 -- 172.21.15.97:0/2231666543 >> 172.21.15.97:0/2231666543 conn(0x562feafdd300 msgr2=0x562feb38bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 -- 172.21.15.97:0/2231666543 shutdown_connections 2024-07-26T07:55:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 -- 172.21.15.97:0/2231666543 wait complete. 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 -- start start 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.334+0000 7f989ada7740 1 Processor -- start 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f989ada7740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb39d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f989ada7740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb39e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f989ada7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb39ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f989ada7740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f988c1addf0 con 0x562feafc84d0 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f989ada7740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f988c1ad720 con 0x562feafc02f0 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f989ada7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f988c1aa960 con 0x562feafe91b0 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f989ada7740 10 client.0 _ll_register_callbacks cb 0x7f9884009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f988be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb39e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f9891200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb39d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f988be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb39e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58704/0 (socket says 172.21.15.97:58704) 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f988be00640 1 -- 172.21.15.97:0/3649300040 learned_addr learned my addr 172.21.15.97:0/3649300040 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f9891200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb39d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:42232/0 (socket says 172.21.15.97:42232) 2024-07-26T07:55:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.335+0000 7f9891c00640 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb39ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f988be00640 1 -- 172.21.15.97:0/3649300040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 msgr2=0x562feb39ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f988be00640 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb39ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f988be00640 1 -- 172.21.15.97:0/3649300040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 msgr2=0x562feb39d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f988be00640 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb39d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f988be00640 1 -- 172.21.15.97:0/3649300040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562feb39f350 con 0x562feafc84d0 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f988be00640 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb39e080 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f98780a6770 tx=0x7f98780aa040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aeaa80e3bd5117d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f9872a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f9872a00640 1 -- 172.21.15.97:0/3649300040 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878020070 con 0x562feafc84d0 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f9872a00640 1 -- 172.21.15.97:0/3649300040 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f98780e3da0 con 0x562feafc84d0 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f989ada7740 1 -- 172.21.15.97:0/3649300040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f988404a070 con 0x562feafc84d0 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f9872a00640 1 -- 172.21.15.97:0/3649300040 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f98780aec40 con 0x562feafc84d0 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f989ada7740 10 client.15918 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.336+0000 7f989ada7740 1 -- 172.21.15.97:0/3649300040 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562feb3a06b0 con 0x562feafc84d0 2024-07-26T07:55:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f9872a00640 1 -- 172.21.15.97:0/3649300040 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98780c6020 con 0x562feafc84d0 2024-07-26T07:55:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f9872a00640 1 -- 172.21.15.97:0/3649300040 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f98780bfce0 con 0x562feafc84d0 2024-07-26T07:55:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f989ada7740 10 client.15918 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f989ada7740 10 client.15918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:55:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f989ada7740 20 client.15918 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f989ada7740 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562feaf7d1c0 0x562feb4109e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f989ada7740 4 client.15918 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:55:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f989ada7740 1 -- 172.21.15.97:0/3649300040 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562feb2fec60 con 0x562feaf7d1c0 2024-07-26T07:55:19.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.337+0000 7f9891200640 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562feaf7d1c0 0x562feb4109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:19.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.338+0000 7f9891200640 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562feaf7d1c0 0x562feb4109e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72430a0a558603fc server_cookie=84b0f071e631e8f5 in_seq=0 out_seq=0 2024-07-26T07:55:19.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.338+0000 7f9872a00640 10 client.15918 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.979+0000 7f9872a00640 1 -- 172.21.15.97:0/3649300040 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562feb2fec60 con 0x562feaf7d1c0 2024-07-26T07:55:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.979+0000 7f9872a00640 10 client.15918 handle_command_reply: tid=0 2024-07-26T07:55:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.979+0000 7f989ada7740 1 client.15918 shutdown 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f989ada7740 1 -- 172.21.15.97:0/3649300040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 msgr2=0x562feb39e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f989ada7740 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb39e080 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f98780a6770 tx=0x7f98780aa040 comp rx=0 tx=0).stop 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f989ada7740 1 -- 172.21.15.97:0/3649300040 shutdown_connections 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f989ada7740 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x562feaf7d1c0 0x562feb4109e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f989ada7740 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feafe91b0 0x562feb39ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f989ada7740 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562feafc84d0 0x562feb39e080 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f989ada7740 1 --2- 172.21.15.97:0/3649300040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x562feafc02f0 0x562feb39d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f9872a00640 0 client.15918 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.980+0000 7f989ada7740 1 -- 172.21.15.97:0/3649300040 >> 172.21.15.97:0/3649300040 conn(0x562feafdd300 msgr2=0x562feb38bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.981+0000 7f989ada7740 1 -- 172.21.15.97:0/3649300040 shutdown_connections 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.981+0000 7f989ada7740 1 -- 172.21.15.97:0/3649300040 wait complete. 2024-07-26T07:55:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.981+0000 7f989ada7740 20 client.15918 trim_cache size 0 max 16384 2024-07-26T07:55:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:55:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:55:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:55:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 -- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f98740b3ae0 msgr2=0x7f98740b5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f98740b3ae0 0x7f98740b5ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x562feb39a380 tx=0x7f9884054000 comp rx=0 tx=0).stop 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 -- 172.21.15.97:0/3642780670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988c1061a0 msgr2=0x7f988c19bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988c1061a0 0x7f988c19bf20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f987800d950 tx=0x7f987800de10 comp rx=0 tx=0).stop 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 -- 172.21.15.97:0/3642780670 shutdown_connections 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f98740b3ae0 0x7f98740b5ff0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f988c106b40 0x7f988c19cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988c1061a0 0x7f988c19bf20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 --2- 172.21.15.97:0/3642780670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f988c105120 0x7f988c19b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.982+0000 7f9892600640 1 -- 172.21.15.97:0/3642780670 >> 172.21.15.97:0/3642780670 conn(0x7f988c100a80 msgr2=0x7f988c1094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.983+0000 7f9892600640 1 -- 172.21.15.97:0/3642780670 shutdown_connections 2024-07-26T07:55:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:19.983+0000 7f9892600640 1 -- 172.21.15.97:0/3642780670 wait complete. 2024-07-26T07:55:20.201 INFO:tasks.workunit.client.0.smithi097.stdout: 14 6445 57 3712 31 3409 137 2024-07-26T07:55:20.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:20 smithi115 ceph-mon[35364]: pgmap v616: 97 pgs: 97 active+clean; 53 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 706 KiB/s wr, 175 op/s 2024-07-26T07:55:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:20 smithi181 ceph-mon[35869]: pgmap v616: 97 pgs: 97 active+clean; 53 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 706 KiB/s wr, 175 op/s 2024-07-26T07:55:20.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:20 smithi097 ceph-mon[32231]: pgmap v616: 97 pgs: 97 active+clean; 53 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 706 KiB/s wr, 175 op/s 2024-07-26T07:55:21.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.346+0000 7fd7ffe00640 1 -- 172.21.15.97:0/4064735021 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56421f825530 con 0x56421f50c840 2024-07-26T07:55:21.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.346+0000 7fd7ffe00640 10 client.25414 handle_command_reply: tid=0 2024-07-26T07:55:21.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.346+0000 7fd827a7c740 1 client.25414 shutdown 2024-07-26T07:55:21.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd827a7c740 1 -- 172.21.15.97:0/4064735021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 msgr2=0x56421f8c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:21.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd827a7c740 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8c62d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00ce00 tx=0x7fd80c019d80 comp rx=0 tx=0).stop 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd827a7c740 1 -- 172.21.15.97:0/4064735021 shutdown_connections 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd827a7c740 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56421f50c840 0x56421f93b000 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd827a7c740 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56421f4ebb60 0x56421f8c1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd827a7c740 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56421f4e3980 0x56421f8c17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd827a7c740 1 --2- 172.21.15.97:0/4064735021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56421f4cb1c0 0x56421f8c62d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd827a7c740 1 -- 172.21.15.97:0/4064735021 >> 172.21.15.97:0/4064735021 conn(0x56421f500990 msgr2=0x56421f8af880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.347+0000 7fd7ffe00640 0 client.25414 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.348+0000 7fd827a7c740 1 -- 172.21.15.97:0/4064735021 shutdown_connections 2024-07-26T07:55:21.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.348+0000 7fd827a7c740 1 -- 172.21.15.97:0/4064735021 wait complete. 2024-07-26T07:55:21.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.348+0000 7fd827a7c740 20 client.25414 trim_cache size 0 max 16384 2024-07-26T07:55:21.350 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:55:21.350 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 30, 2024-07-26T07:55:21.350 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:55:21.350 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 10, 2024-07-26T07:55:21.350 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "36ce223e": { 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 10, 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 151.1, 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:55:21.351 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:21.352 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 -- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd8000b3c50 msgr2=0x7fd8000b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd8000b3c50 0x7fd8000b6160 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd814006010 tx=0x7fd81402a8c0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 -- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c077cc0 msgr2=0x7fd81c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c077cc0 0x7fd81c06db70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd80400d750 tx=0x7fd80400dc10 comp rx=0 tx=0).stop 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 -- 172.21.15.97:0/1000171621 shutdown_connections 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd8000b3c50 0x7fd8000b6160 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd81c1141b0 0x7fd81c06e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd81c077cc0 0x7fd81c06db70 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 --2- 172.21.15.97:0/1000171621 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd81c077370 0x7fd81c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.350+0000 7fd820e00640 1 -- 172.21.15.97:0/1000171621 >> 172.21.15.97:0/1000171621 conn(0x7fd81c101200 msgr2=0x7fd81c103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:21.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.351+0000 7fd820e00640 1 -- 172.21.15.97:0/1000171621 shutdown_connections 2024-07-26T07:55:21.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:21.351+0000 7fd820e00640 1 -- 172.21.15.97:0/1000171621 wait complete. 2024-07-26T07:55:22.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:22 smithi097 ceph-mon[32231]: pgmap v617: 97 pgs: 97 active+clean; 56 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 975 KiB/s wr, 236 op/s 2024-07-26T07:55:22.402 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '36ce223e', quiesce took 13.5 seconds. Will release after: 17.6 2024-07-26T07:55:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:22 smithi115 ceph-mon[35364]: pgmap v617: 97 pgs: 97 active+clean; 56 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 975 KiB/s wr, 236 op/s 2024-07-26T07:55:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:22 smithi181 ceph-mon[35869]: pgmap v617: 97 pgs: 97 active+clean; 56 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 975 KiB/s wr, 236 op/s 2024-07-26T07:55:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:24 smithi115 ceph-mon[35364]: pgmap v618: 97 pgs: 97 active+clean; 56 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 712 KiB/s wr, 156 op/s 2024-07-26T07:55:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:24 smithi181 ceph-mon[35869]: pgmap v618: 97 pgs: 97 active+clean; 56 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 712 KiB/s wr, 156 op/s 2024-07-26T07:55:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:24 smithi097 ceph-mon[32231]: pgmap v618: 97 pgs: 97 active+clean; 56 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 712 KiB/s wr, 156 op/s 2024-07-26T07:55:26.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:26 smithi115 ceph-mon[35364]: pgmap v619: 97 pgs: 97 active+clean; 60 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.0 MiB/s wr, 325 op/s 2024-07-26T07:55:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:26 smithi181 ceph-mon[35869]: pgmap v619: 97 pgs: 97 active+clean; 60 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.0 MiB/s wr, 325 op/s 2024-07-26T07:55:26.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:26 smithi097 ceph-mon[32231]: pgmap v619: 97 pgs: 97 active+clean; 60 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.0 MiB/s wr, 325 op/s 2024-07-26T07:55:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:28 smithi115 ceph-mon[35364]: pgmap v620: 97 pgs: 97 active+clean; 52 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 963 KiB/s wr, 327 op/s 2024-07-26T07:55:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:28 smithi181 ceph-mon[35869]: pgmap v620: 97 pgs: 97 active+clean; 52 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 963 KiB/s wr, 327 op/s 2024-07-26T07:55:28.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:28 smithi097 ceph-mon[32231]: pgmap v620: 97 pgs: 97 active+clean; 52 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 963 KiB/s wr, 327 op/s 2024-07-26T07:55:30.202 INFO:tasks.workunit.client.0.smithi097.stdout: 14 4692 0 3183 0 2862 0 2024-07-26T07:55:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:30 smithi115 ceph-mon[35364]: pgmap v621: 97 pgs: 97 active+clean; 52 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 755 KiB/s wr, 311 op/s 2024-07-26T07:55:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:30 smithi181 ceph-mon[35869]: pgmap v621: 97 pgs: 97 active+clean; 52 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 755 KiB/s wr, 311 op/s 2024-07-26T07:55:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:30 smithi097 ceph-mon[32231]: pgmap v621: 97 pgs: 97 active+clean; 52 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 755 KiB/s wr, 311 op/s 2024-07-26T07:55:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:32 smithi115 ceph-mon[35364]: pgmap v622: 97 pgs: 97 active+clean; 52 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 748 KiB/s wr, 313 op/s 2024-07-26T07:55:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:32 smithi181 ceph-mon[35869]: pgmap v622: 97 pgs: 97 active+clean; 52 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 748 KiB/s wr, 313 op/s 2024-07-26T07:55:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:32 smithi097 ceph-mon[32231]: pgmap v622: 97 pgs: 97 active+clean; 52 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 748 KiB/s wr, 313 op/s 2024-07-26T07:55:33.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 -- 172.21.15.97:0/1110093944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed01061a0 msgr2=0x7fbed01065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 --2- 172.21.15.97:0/1110093944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed01061a0 0x7fbed01065b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009900 tx=0x7fbec402a410 comp rx=0 tx=0).stop 2024-07-26T07:55:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 -- 172.21.15.97:0/1110093944 shutdown_connections 2024-07-26T07:55:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 --2- 172.21.15.97:0/1110093944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbed0106b40 0x7fbed010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 --2- 172.21.15.97:0/1110093944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed01061a0 0x7fbed01065b0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 --2- 172.21.15.97:0/1110093944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbed0105120 0x7fbed0105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 -- 172.21.15.97:0/1110093944 >> 172.21.15.97:0/1110093944 conn(0x7fbed0100a80 msgr2=0x7fbed0102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 -- 172.21.15.97:0/1110093944 shutdown_connections 2024-07-26T07:55:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.214+0000 7fbed6c00640 1 -- 172.21.15.97:0/1110093944 wait complete. 2024-07-26T07:55:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed6c00640 1 Processor -- start 2024-07-26T07:55:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed6c00640 1 -- start start 2024-07-26T07:55:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed6c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed0105120 0x7fbed019b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbed01061a0 0x7fbed019bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed6c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbed0106b40 0x7fbed019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed6c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0110c50 con 0x7fbed0105120 2024-07-26T07:55:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed6c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0110ad0 con 0x7fbed0106b40 2024-07-26T07:55:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0110dd0 con 0x7fbed01061a0 2024-07-26T07:55:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed0105120 0x7fbed019b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed0105120 0x7fbed019b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49772/0 (socket says 172.21.15.97:49772) 2024-07-26T07:55:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.216+0000 7fbed5800640 1 -- 172.21.15.97:0/3627622235 learned_addr learned my addr 172.21.15.97:0/3627622235 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed4e00640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbed01061a0 0x7fbed019bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed6200640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbed0106b40 0x7fbed019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed5800640 1 -- 172.21.15.97:0/3627622235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbed01061a0 msgr2=0x7fbed019bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed5800640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbed01061a0 0x7fbed019bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed5800640 1 -- 172.21.15.97:0/3627622235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbed0106b40 msgr2=0x7fbed019cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed5800640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbed0106b40 0x7fbed019cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed5800640 1 -- 172.21.15.97:0/3627622235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed019d3c0 con 0x7fbed0105120 2024-07-26T07:55:33.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed4e00640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbed01061a0 0x7fbed019bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:55:33.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.217+0000 7fbed5800640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed0105120 0x7fbed019b980 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00d770 tx=0x7fbebc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:33.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.218+0000 7fbece000640 1 -- 172.21.15.97:0/3627622235 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc017070 con 0x7fbed0105120 2024-07-26T07:55:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.218+0000 7fbece000640 1 -- 172.21.15.97:0/3627622235 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbebc0054b0 con 0x7fbed0105120 2024-07-26T07:55:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.218+0000 7fbed6c00640 1 -- 172.21.15.97:0/3627622235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed019d6b0 con 0x7fbed0105120 2024-07-26T07:55:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.218+0000 7fbece000640 1 -- 172.21.15.97:0/3627622235 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc004c60 con 0x7fbed0105120 2024-07-26T07:55:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.218+0000 7fbed6c00640 1 -- 172.21.15.97:0/3627622235 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed01a3da0 con 0x7fbed0105120 2024-07-26T07:55:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.221+0000 7fbedf38e740 1 Processor -- start 2024-07-26T07:55:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.221+0000 7fbedf38e740 1 -- start start 2024-07-26T07:55:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.221+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.221+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa67750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 0x56062fa682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbedf38e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56062f9d9270 con 0x56062f6982f0 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbedf38e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56062f9c8770 con 0x56062f6c11b0 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbedf38e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56062f9cc7b0 con 0x56062f6a04d0 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbece000640 1 -- 172.21.15.97:0/3627622235 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fbebc004e00 con 0x7fbed0105120 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbed5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa67750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbed5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa67750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49786/0 (socket says 172.21.15.97:49786) 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbed5800640 1 -- 172.21.15.97:0/810693494 learned_addr learned my addr 172.21.15.97:0/810693494 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbed6200640 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.222+0000 7fbed4e00640 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 0x56062fa682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.223+0000 7fbed5800640 1 -- 172.21.15.97:0/810693494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 msgr2=0x56062fa682a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.223+0000 7fbed5800640 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 0x56062fa682a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.223+0000 7fbed5800640 1 -- 172.21.15.97:0/810693494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 msgr2=0x56062fa670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.223+0000 7fbed5800640 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.223+0000 7fbed5800640 1 -- 172.21.15.97:0/810693494 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56062f72c790 con 0x56062f6982f0 2024-07-26T07:55:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.223+0000 7fbed5800640 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa67750 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0e9c10 tx=0x7fbebc0eb690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8d162cdd2328a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.223+0000 7fbec2a00640 1 -- 172.21.15.97:0/810693494 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0f2070 con 0x56062f6982f0 2024-07-26T07:55:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.223+0000 7fbece000640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbeb80b3bd0 0x7fbeb80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.224+0000 7fbec2a00640 1 -- 172.21.15.97:0/810693494 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbebc0aa1b0 con 0x56062f6982f0 2024-07-26T07:55:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.224+0000 7fbec2a00640 1 -- 172.21.15.97:0/810693494 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0a9ff0 con 0x56062f6982f0 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.224+0000 7fbece000640 1 -- 172.21.15.97:0/3627622235 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fbebc0e37b0 con 0x7fbed0105120 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.224+0000 7fbed6200640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbeb80b3bd0 0x7fbeb80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.224+0000 7fbedf38e740 1 -- 172.21.15.97:0/810693494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 msgr2=0x56062fa67750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa67750 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0e9c10 tx=0x7fbebc0eb690 comp rx=0 tx=0).stop 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbed6200640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbeb80b3bd0 0x7fbeb80b60e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbec8008e40 tx=0x7fbec80085f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 -- 172.21.15.97:0/810693494 shutdown_connections 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 0x56062fa682a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa67750 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 --2- 172.21.15.97:0/810693494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 -- 172.21.15.97:0/810693494 >> 172.21.15.97:0/810693494 conn(0x56062f6b5300 msgr2=0x56062fa62b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 -- 172.21.15.97:0/810693494 shutdown_connections 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 -- 172.21.15.97:0/810693494 wait complete. 2024-07-26T07:55:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.225+0000 7fbedf38e740 1 -- start start 2024-07-26T07:55:33.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbedf38e740 1 Processor -- start 2024-07-26T07:55:33.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa70bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 0x56062fa71280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa71d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbedf38e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56062f902ab0 con 0x56062f6982f0 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbedf38e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56062f965d50 con 0x56062f6c11b0 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbedf38e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56062fa68dc0 con 0x56062f6a04d0 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbedf38e740 10 client.0 _ll_register_callbacks cb 0x56062f6d8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbed5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa70bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbed5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa70bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49790/0 (socket says 172.21.15.97:49790) 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.226+0000 7fbed5800640 1 -- 172.21.15.97:0/3991585004 learned_addr learned my addr 172.21.15.97:0/3991585004 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed6200640 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa71d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed4e00640 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 0x56062fa71280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed5800640 1 -- 172.21.15.97:0/3991585004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 msgr2=0x56062fa71280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed5800640 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 0x56062fa71280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed5800640 1 -- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 msgr2=0x56062fa71d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed5800640 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa71d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed5800640 1 -- 172.21.15.97:0/3991585004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56062f7a5620 con 0x56062f6982f0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed6200640 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa71d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbed5800640 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa70bc0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0a7910 tx=0x7fbebc009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3782d05bfdd47f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbeb7400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbeb7400640 1 -- 172.21.15.97:0/3991585004 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc073b80 con 0x56062f6982f0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbeb7400640 1 -- 172.21.15.97:0/3991585004 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fbebc0a6c90 con 0x56062f6982f0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.227+0000 7fbedf38e740 1 -- 172.21.15.97:0/3991585004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbebc0aa1b0 con 0x56062f6982f0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbeb7400640 1 -- 172.21.15.97:0/3991585004 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbebc073da0 con 0x56062f6982f0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbedf38e740 10 client.15930 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbedf38e740 1 -- 172.21.15.97:0/3991585004 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56062fa7acd0 con 0x56062f6982f0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbeb7400640 1 -- 172.21.15.97:0/3991585004 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbebc0aa1b0 con 0x56062f6982f0 2024-07-26T07:55:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbeb7400640 1 -- 172.21.15.97:0/3991585004 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fbebc0a6930 con 0x56062f6982f0 2024-07-26T07:55:33.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbedf38e740 10 client.15930 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:33.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbedf38e740 10 client.15930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:55:33.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbedf38e740 20 client.15930 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:33.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbedf38e740 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56062f67fd80 0x56062fa7e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbedf38e740 4 client.15930 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:55:33.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.228+0000 7fbedf38e740 1 -- 172.21.15.97:0/3991585004 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56062f9db1c0 con 0x56062f67fd80 2024-07-26T07:55:33.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.229+0000 7fbed4e00640 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56062f67fd80 0x56062fa7e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.230+0000 7fbed4e00640 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56062f67fd80 0x56062fa7e030 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35e1f0ab31780ea5 server_cookie=280f9be11147de6e in_seq=0 out_seq=0 2024-07-26T07:55:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.230+0000 7fbeb7400640 10 client.15930 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbeb7400640 1 -- 172.21.15.97:0/3991585004 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56062f9db1c0 con 0x56062f67fd80 2024-07-26T07:55:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbeb7400640 10 client.15930 handle_command_reply: tid=0 2024-07-26T07:55:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 client.15930 shutdown 2024-07-26T07:55:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 -- 172.21.15.97:0/3991585004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 msgr2=0x56062fa70bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa70bc0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0a7910 tx=0x7fbebc009f00 comp rx=0 tx=0).stop 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 -- 172.21.15.97:0/3991585004 shutdown_connections 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56062f67fd80 0x56062fa7e030 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa71d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6a04d0 0x56062fa71280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 --2- 172.21.15.97:0/3991585004 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6982f0 0x56062fa70bc0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbeb7400640 0 client.15930 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 -- 172.21.15.97:0/3991585004 >> 172.21.15.97:0/3991585004 conn(0x56062f6b5300 msgr2=0x56062fa63770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 -- 172.21.15.97:0/3991585004 shutdown_connections 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.232+0000 7fbedf38e740 1 -- 172.21.15.97:0/3991585004 wait complete. 2024-07-26T07:55:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.233+0000 7fbedf38e740 20 client.15930 trim_cache size 0 max 16384 2024-07-26T07:55:33.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.249+0000 7fbedf38e740 1 Processor -- start 2024-07-26T07:55:33.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.249+0000 7fbedf38e740 1 -- start start 2024-07-26T07:55:33.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.249+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 0x56062fa70720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.249+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.249+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.249+0000 7fbedf38e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbed01acb60 con 0x56062f6a04d0 2024-07-26T07:55:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.249+0000 7fbedf38e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbed01ae760 con 0x56062f6c11b0 2024-07-26T07:55:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.249+0000 7fbedf38e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbed01ad230 con 0x56062f6982f0 2024-07-26T07:55:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa624b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49802/0 (socket says 172.21.15.97:49802) 2024-07-26T07:55:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed4e00640 1 -- 172.21.15.97:0/1036485271 learned_addr learned my addr 172.21.15.97:0/1036485271 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed6200640 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed5800640 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 0x56062fa70720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed4e00640 1 -- 172.21.15.97:0/1036485271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 msgr2=0x56062fa70720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed4e00640 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 0x56062fa70720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed4e00640 1 -- 172.21.15.97:0/1036485271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 msgr2=0x56062fa629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed4e00640 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa629f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed4e00640 1 -- 172.21.15.97:0/1036485271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56062fa6b400 con 0x56062f6a04d0 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.250+0000 7fbed6200640 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa629f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.251+0000 7fbed4e00640 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa624b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009850 tx=0x7fbec4005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4e3b1560db7134 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.251+0000 7fbec2000640 1 -- 172.21.15.97:0/1036485271 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4035040 con 0x56062f6a04d0 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.251+0000 7fbec2000640 1 -- 172.21.15.97:0/1036485271 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbec40319c0 con 0x56062f6a04d0 2024-07-26T07:55:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.251+0000 7fbec2000640 1 -- 172.21.15.97:0/1036485271 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4003ae0 con 0x56062f6a04d0 2024-07-26T07:55:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 -- 172.21.15.97:0/1036485271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 msgr2=0x56062fa624b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa624b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009850 tx=0x7fbec4005e00 comp rx=0 tx=0).stop 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 -- 172.21.15.97:0/1036485271 shutdown_connections 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa629f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa624b0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 --2- 172.21.15.97:0/1036485271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 0x56062fa70720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 -- 172.21.15.97:0/1036485271 >> 172.21.15.97:0/1036485271 conn(0x56062f6b5300 msgr2=0x56062fa6b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 -- 172.21.15.97:0/1036485271 shutdown_connections 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 -- 172.21.15.97:0/1036485271 wait complete. 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.252+0000 7fbedf38e740 1 -- start start 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbedf38e740 1 Processor -- start 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 0x56062fa759e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa760a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbedf38e740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa76c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbedf38e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbed01adfd0 con 0x56062f6a04d0 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbedf38e740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbed01ad900 con 0x56062f6c11b0 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbedf38e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbed01aab40 con 0x56062f6982f0 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbedf38e740 10 client.0 _ll_register_callbacks cb 0x7fbec8008490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.253+0000 7fbed4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed4e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa760a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49818/0 (socket says 172.21.15.97:49818) 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed4e00640 1 -- 172.21.15.97:0/2837882764 learned_addr learned my addr 172.21.15.97:0/2837882764 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed6200640 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa76c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed5800640 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 0x56062fa759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed4e00640 1 -- 172.21.15.97:0/2837882764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 msgr2=0x56062fa759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed4e00640 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 0x56062fa759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed4e00640 1 -- 172.21.15.97:0/2837882764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 msgr2=0x56062fa76c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed4e00640 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa76c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed4e00640 1 -- 172.21.15.97:0/2837882764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56062fa77370 con 0x56062f6a04d0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbed4e00640 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa760a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbec4006010 tx=0x7fbec4031e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f823a041f882ad server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.254+0000 7fbeb7400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.255+0000 7fbeb7400640 1 -- 172.21.15.97:0/2837882764 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4050070 con 0x56062f6a04d0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.255+0000 7fbeb7400640 1 -- 172.21.15.97:0/2837882764 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fbec40042d0 con 0x56062f6a04d0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.255+0000 7fbedf38e740 1 -- 172.21.15.97:0/2837882764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec40319c0 con 0x56062f6a04d0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.255+0000 7fbeb7400640 1 -- 172.21.15.97:0/2837882764 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbec4035040 con 0x56062f6a04d0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.255+0000 7fbedf38e740 10 client.15936 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.255+0000 7fbedf38e740 1 -- 172.21.15.97:0/2837882764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56062fa78690 con 0x56062f6a04d0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.255+0000 7fbeb7400640 1 -- 172.21.15.97:0/2837882764 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbec40319c0 con 0x56062f6a04d0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.255+0000 7fbeb7400640 1 -- 172.21.15.97:0/2837882764 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fbec403c8d0 con 0x56062f6a04d0 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.256+0000 7fbedf38e740 10 client.15936 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.256+0000 7fbedf38e740 10 client.15936 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:55:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.256+0000 7fbedf38e740 20 client.15936 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.256+0000 7fbedf38e740 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56062f6551c0 0x56062fae87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.256+0000 7fbedf38e740 4 client.15936 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:55:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.256+0000 7fbedf38e740 1 -- 172.21.15.97:0/2837882764 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56062f9d6c60 con 0x56062f6551c0 2024-07-26T07:55:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.256+0000 7fbed5800640 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56062f6551c0 0x56062fae87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.257+0000 7fbed5800640 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56062f6551c0 0x56062fae87e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a89b98865c240e5 server_cookie=33f590fbae2d53cb in_seq=0 out_seq=0 2024-07-26T07:55:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.257+0000 7fbeb7400640 10 client.15936 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.391+0000 7fbeb7400640 1 -- 172.21.15.97:0/2837882764 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56062f9d6c60 con 0x56062f6551c0 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.391+0000 7fbeb7400640 10 client.15936 handle_command_reply: tid=0 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.391+0000 7fbedf38e740 1 client.15936 shutdown 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 -- 172.21.15.97:0/2837882764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 msgr2=0x56062fa760a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa760a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbec4006010 tx=0x7fbec4031e10 comp rx=0 tx=0).stop 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 -- 172.21.15.97:0/2837882764 shutdown_connections 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56062f6551c0 0x56062fae87e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56062f6c11b0 0x56062fa76c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56062f6a04d0 0x56062fa760a0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 --2- 172.21.15.97:0/2837882764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56062f6982f0 0x56062fa759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbeb7400640 0 client.15936 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 -- 172.21.15.97:0/2837882764 >> 172.21.15.97:0/2837882764 conn(0x56062f6b5300 msgr2=0x56062fa70690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 -- 172.21.15.97:0/2837882764 shutdown_connections 2024-07-26T07:55:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 1 -- 172.21.15.97:0/2837882764 wait complete. 2024-07-26T07:55:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.392+0000 7fbedf38e740 20 client.15936 trim_cache size 0 max 16384 2024-07-26T07:55:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:55:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:55:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:55:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:55:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 -- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbeb80b3bd0 msgr2=0x7fbeb80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbeb80b3bd0 0x7fbeb80b60e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbec8008e40 tx=0x7fbec80085f0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 -- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed0105120 msgr2=0x7fbed019b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed0105120 0x7fbed019b980 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00d770 tx=0x7fbebc00dc30 comp rx=0 tx=0).stop 2024-07-26T07:55:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 -- 172.21.15.97:0/3627622235 shutdown_connections 2024-07-26T07:55:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbeb80b3bd0 0x7fbeb80b60e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbed0106b40 0x7fbed019cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbed01061a0 0x7fbed019bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 --2- 172.21.15.97:0/3627622235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbed0105120 0x7fbed019b980 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.393+0000 7fbed6c00640 1 -- 172.21.15.97:0/3627622235 >> 172.21.15.97:0/3627622235 conn(0x7fbed0100a80 msgr2=0x7fbed01094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.394+0000 7fbed6c00640 1 -- 172.21.15.97:0/3627622235 shutdown_connections 2024-07-26T07:55:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:33.394+0000 7fbed6c00640 1 -- 172.21.15.97:0/3627622235 wait complete. 2024-07-26T07:55:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:34 smithi097 ceph-mon[32231]: pgmap v623: 97 pgs: 97 active+clean; 55 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 735 KiB/s wr, 243 op/s 2024-07-26T07:55:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:34 smithi115 ceph-mon[35364]: pgmap v623: 97 pgs: 97 active+clean; 55 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 735 KiB/s wr, 243 op/s 2024-07-26T07:55:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:34 smithi181 ceph-mon[35869]: pgmap v623: 97 pgs: 97 active+clean; 55 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 735 KiB/s wr, 243 op/s 2024-07-26T07:55:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:36 smithi115 ceph-mon[35364]: pgmap v624: 97 pgs: 97 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 697 KiB/s wr, 223 op/s 2024-07-26T07:55:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:36 smithi181 ceph-mon[35869]: pgmap v624: 97 pgs: 97 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 697 KiB/s wr, 223 op/s 2024-07-26T07:55:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:36 smithi097 ceph-mon[32231]: pgmap v624: 97 pgs: 97 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 697 KiB/s wr, 223 op/s 2024-07-26T07:55:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:38 smithi115 ceph-mon[35364]: pgmap v625: 97 pgs: 97 active+clean; 52 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 379 KiB/s wr, 50 op/s 2024-07-26T07:55:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:38 smithi181 ceph-mon[35869]: pgmap v625: 97 pgs: 97 active+clean; 52 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 379 KiB/s wr, 50 op/s 2024-07-26T07:55:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:38 smithi097 ceph-mon[32231]: pgmap v625: 97 pgs: 97 active+clean; 52 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 379 KiB/s wr, 50 op/s 2024-07-26T07:55:40.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:55:40.202 INFO:tasks.workunit.client.0.smithi097.stdout: 14 54 0 30 0 0 0 2024-07-26T07:55:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:40 smithi115 ceph-mon[35364]: pgmap v626: 97 pgs: 97 active+clean; 52 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 359 KiB/s wr, 34 op/s 2024-07-26T07:55:40.481 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:55:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:40 smithi181 ceph-mon[35869]: pgmap v626: 97 pgs: 97 active+clean; 52 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 359 KiB/s wr, 34 op/s 2024-07-26T07:55:40.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:40 smithi097 ceph-mon[32231]: pgmap v626: 97 pgs: 97 active+clean; 52 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 359 KiB/s wr, 34 op/s 2024-07-26T07:55:42.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:42 smithi097 ceph-mon[32231]: pgmap v627: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 358 KiB/s wr, 34 op/s 2024-07-26T07:55:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:42 smithi115 ceph-mon[35364]: pgmap v627: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 358 KiB/s wr, 34 op/s 2024-07-26T07:55:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:42 smithi181 ceph-mon[35869]: pgmap v627: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 358 KiB/s wr, 34 op/s 2024-07-26T07:55:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.718+0000 7fb6a6400640 1 -- 172.21.15.97:0/2763181030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a0102b20 msgr2=0x7fb6a0107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.718+0000 7fb6a6400640 1 --2- 172.21.15.97:0/2763181030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a0102b20 0x7fb6a0107050 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009930 tx=0x7fb68c02a420 comp rx=0 tx=0).stop 2024-07-26T07:55:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.719+0000 7fb6a6400640 1 -- 172.21.15.97:0/2763181030 shutdown_connections 2024-07-26T07:55:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.719+0000 7fb6a6400640 1 --2- 172.21.15.97:0/2763181030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6a0107610 0x7fb6a0107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.719+0000 7fb6a6400640 1 --2- 172.21.15.97:0/2763181030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a0102b20 0x7fb6a0107050 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.719+0000 7fb6a6400640 1 --2- 172.21.15.97:0/2763181030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a01021d0 0x7fb6a01025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.719+0000 7fb6a6400640 1 -- 172.21.15.97:0/2763181030 >> 172.21.15.97:0/2763181030 conn(0x7fb6a0079160 msgr2=0x7fb6a0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:42.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.719+0000 7fb6a6400640 1 -- 172.21.15.97:0/2763181030 shutdown_connections 2024-07-26T07:55:42.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.719+0000 7fb6a6400640 1 -- 172.21.15.97:0/2763181030 wait complete. 2024-07-26T07:55:42.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.720+0000 7fb6a6400640 1 Processor -- start 2024-07-26T07:55:42.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.720+0000 7fb6a6400640 1 -- start start 2024-07-26T07:55:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb6a6400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a01021d0 0x7fb6a01a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb6a6400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6a0102b20 0x7fb6a01a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb6a6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a0107610 0x7fb6a01a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb6a6400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a0119960 con 0x7fb6a01021d0 2024-07-26T07:55:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb6a6400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a01197e0 con 0x7fb6a0102b20 2024-07-26T07:55:42.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb6a6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a0119ae0 con 0x7fb6a0107610 2024-07-26T07:55:42.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb69f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a01021d0 0x7fb6a01a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:42.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb69f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a01021d0 0x7fb6a01a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47562/0 (socket says 172.21.15.97:47562) 2024-07-26T07:55:42.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb69f400640 1 -- 172.21.15.97:0/781836441 learned_addr learned my addr 172.21.15.97:0/781836441 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:42.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.721+0000 7fb69fe00640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a0107610 0x7fb6a01a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:42.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb69ea00640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6a0102b20 0x7fb6a01a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:42.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb69f400640 1 -- 172.21.15.97:0/781836441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a0107610 msgr2=0x7fb6a01a5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb69f400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a0107610 0x7fb6a01a5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb69f400640 1 -- 172.21.15.97:0/781836441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6a0102b20 msgr2=0x7fb6a01a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb69f400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6a0102b20 0x7fb6a01a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb69f400640 1 -- 172.21.15.97:0/781836441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a01a63a0 con 0x7fb6a01021d0 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb69f400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a01021d0 0x7fb6a01a0430 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb69000d950 tx=0x7fb69000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb697e00640 1 -- 172.21.15.97:0/781836441 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb690017070 con 0x7fb6a01021d0 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb697e00640 1 -- 172.21.15.97:0/781836441 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb690009c30 con 0x7fb6a01021d0 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb697e00640 1 -- 172.21.15.97:0/781836441 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69000f620 con 0x7fb6a01021d0 2024-07-26T07:55:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.722+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a01a6690 con 0x7fb6a01021d0 2024-07-26T07:55:42.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.723+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a01acdf0 con 0x7fb6a01021d0 2024-07-26T07:55:42.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.724+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a006af90 con 0x7fb6a01021d0 2024-07-26T07:55:42.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.727+0000 7fb697e00640 1 -- 172.21.15.97:0/781836441 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fb69001a070 con 0x7fb6a01021d0 2024-07-26T07:55:42.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.728+0000 7fb697e00640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6840b3c50 0x7fb6840b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:42.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.728+0000 7fb697e00640 1 -- 172.21.15.97:0/781836441 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fb6900e36e0 con 0x7fb6a01021d0 2024-07-26T07:55:42.729 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.728+0000 7fb69ea00640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6840b3c50 0x7fb6840b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:42.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.729+0000 7fb69ea00640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6840b3c50 0x7fb6840b6160 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009900 tx=0x7fb68c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:42.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.730+0000 7fb697e00640 1 -- 172.21.15.97:0/781836441 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb690009de0 con 0x7fb6a01021d0 2024-07-26T07:55:42.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.966+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6a01ad060 con 0x7fb6a01021d0 2024-07-26T07:55:42.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.967+0000 7fb697e00640 1 -- 172.21.15.97:0/781836441 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11998 (secure 0 0 0) 0x7fb6900aacf0 con 0x7fb6a01021d0 2024-07-26T07:55:42.969 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:55:42.969 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":31,"btime":"2024-07-26T07:54:39:931393+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":25369,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3957930666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3957930666},{"type":"v1","addr":"172.21.15.181:6837","nonce":3957930666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:54:38.906083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14691":{"gid":14691,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/2971688113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2971688113},{"type":"v1","addr":"172.21.15.97:6837","nonce":2971688113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:55:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 31 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6840b3c50 msgr2=0x7fb6840b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6840b3c50 0x7fb6840b6160 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009900 tx=0x7fb68c000bc0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a01021d0 msgr2=0x7fb6a01a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a01021d0 0x7fb6a01a0430 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb69000d950 tx=0x7fb69000de10 comp rx=0 tx=0).stop 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 shutdown_connections 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fb6840b3c50 0x7fb6840b6160 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a0107610 0x7fb6a01a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb6a0102b20 0x7fb6a01a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 --2- 172.21.15.97:0/781836441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6a01021d0 0x7fb6a01a0430 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 >> 172.21.15.97:0/781836441 conn(0x7fb6a0079160 msgr2=0x7fb6a0105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.970+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 shutdown_connections 2024-07-26T07:55:42.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:42.971+0000 7fb6a6400640 1 -- 172.21.15.97:0/781836441 wait complete. 2024-07-26T07:55:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:43 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/781836441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:55:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:43 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/781836441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:55:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:43 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/781836441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:55:43.773 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db --set-id 36ce223e --release --await' 2024-07-26T07:55:43.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db --set-id 36ce223e --release --await 2024-07-26T07:55:44.243 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:55:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:44 smithi115 ceph-mon[35364]: pgmap v628: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 359 KiB/s wr, 32 op/s 2024-07-26T07:55:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:44 smithi181 ceph-mon[35869]: pgmap v628: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 359 KiB/s wr, 32 op/s 2024-07-26T07:55:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:44 smithi097 ceph-mon[32231]: pgmap v628: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 359 KiB/s wr, 32 op/s 2024-07-26T07:55:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:46 smithi181 ceph-mon[35869]: pgmap v629: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 84 KiB/s wr, 32 op/s 2024-07-26T07:55:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:46 smithi097 ceph-mon[32231]: pgmap v629: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 84 KiB/s wr, 32 op/s 2024-07-26T07:55:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:46 smithi115 ceph-mon[35364]: pgmap v629: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 84 KiB/s wr, 32 op/s 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 -- 172.21.15.97:0/292438410 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8109830 msgr2=0x7f6ba8110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 --2- 172.21.15.97:0/292438410 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8109830 0x7f6ba8110060 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009ca0 tx=0x7f6ba002a8b0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 -- 172.21.15.97:0/292438410 shutdown_connections 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 --2- 172.21.15.97:0/292438410 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8109830 0x7f6ba8110060 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 --2- 172.21.15.97:0/292438410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ba8108ee0 0x7f6ba81092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 --2- 172.21.15.97:0/292438410 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba81035f0 0x7f6ba8103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 -- 172.21.15.97:0/292438410 >> 172.21.15.97:0/292438410 conn(0x7f6ba807b340 msgr2=0x7f6ba807b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 -- 172.21.15.97:0/292438410 shutdown_connections 2024-07-26T07:55:46.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.552+0000 7f6baca00640 1 -- 172.21.15.97:0/292438410 wait complete. 2024-07-26T07:55:46.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.554+0000 7f6baca00640 1 Processor -- start 2024-07-26T07:55:46.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.554+0000 7f6baca00640 1 -- start start 2024-07-26T07:55:46.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.554+0000 7f6baca00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba81035f0 0x7f6ba8199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6baca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8108ee0 0x7f6ba819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6baca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ba8109830 0x7f6ba819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6baca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba81133b0 con 0x7f6ba8108ee0 2024-07-26T07:55:46.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6baca00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba8113230 con 0x7f6ba81035f0 2024-07-26T07:55:46.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6baca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba8113530 con 0x7f6ba8109830 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6b9fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8108ee0 0x7f6ba819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6b9fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8108ee0 0x7f6ba819a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34524/0 (socket says 172.21.15.97:34524) 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6b9fe00640 1 -- 172.21.15.97:0/546852216 learned_addr learned my addr 172.21.15.97:0/546852216 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba81035f0 0x7f6ba8199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6ba6800640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ba8109830 0x7f6ba819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6b9fe00640 1 -- 172.21.15.97:0/546852216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ba8109830 msgr2=0x7f6ba819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ba8109830 0x7f6ba819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6b9fe00640 1 -- 172.21.15.97:0/546852216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba81035f0 msgr2=0x7f6ba8199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.555+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba81035f0 0x7f6ba8199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.556+0000 7f6b9fe00640 1 -- 172.21.15.97:0/546852216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba819fcb0 con 0x7f6ba8108ee0 2024-07-26T07:55:46.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.556+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8108ee0 0x7f6ba819a3a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000c970 tx=0x7f6b9000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.556+0000 7f6b9ea00640 1 -- 172.21.15.97:0/546852216 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90004220 con 0x7f6ba8108ee0 2024-07-26T07:55:46.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.556+0000 7f6b9ea00640 1 -- 172.21.15.97:0/546852216 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6b90007e40 con 0x7f6ba8108ee0 2024-07-26T07:55:46.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.556+0000 7f6baca00640 1 -- 172.21.15.97:0/546852216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba819ffa0 con 0x7f6ba8108ee0 2024-07-26T07:55:46.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.556+0000 7f6b9ea00640 1 -- 172.21.15.97:0/546852216 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b900047a0 con 0x7f6ba8108ee0 2024-07-26T07:55:46.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.556+0000 7f6baca00640 1 -- 172.21.15.97:0/546852216 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba81a67c0 con 0x7f6ba8108ee0 2024-07-26T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.560+0000 7f6bb37e4740 1 Processor -- start 2024-07-26T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.560+0000 7f6b9ea00640 1 -- 172.21.15.97:0/546852216 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f6b90004980 con 0x7f6ba8108ee0 2024-07-26T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.560+0000 7f6bb37e4740 1 -- start start 2024-07-26T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 0x5647221c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 0x5647221c8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56472213ef30 con 0x564721e02840 2024-07-26T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56472212e430 con 0x564721dfa660 2024-07-26T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564722132470 con 0x564721e23520 2024-07-26T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6b9fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6b9fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34536/0 (socket says 172.21.15.97:34536) 2024-07-26T07:55:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6b9fe00640 1 -- 172.21.15.97:0/4232907874 learned_addr learned my addr 172.21.15.97:0/4232907874 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6ba6800640 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 0x5647221c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.561+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 0x5647221c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.562+0000 7f6b9fe00640 1 -- 172.21.15.97:0/4232907874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 msgr2=0x5647221c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.562+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 0x5647221c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.562+0000 7f6b9fe00640 1 -- 172.21.15.97:0/4232907874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 msgr2=0x5647221c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.562+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 0x5647221c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.562+0000 7f6b9fe00640 1 -- 172.21.15.97:0/4232907874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564721e8eb00 con 0x564721e02840 2024-07-26T07:55:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.562+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221c8a40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6b900e9c20 tx=0x7f6b900eb670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465e8ad2385ee57b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.563+0000 7f6b97000640 1 -- 172.21.15.97:0/4232907874 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b900f2070 con 0x564721e02840 2024-07-26T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.563+0000 7f6b9ea00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6b8c0b3c80 0x7f6b8c0b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.563+0000 7f6b97000640 1 -- 172.21.15.97:0/4232907874 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6b900eeab0 con 0x564721e02840 2024-07-26T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.563+0000 7f6b97000640 1 -- 172.21.15.97:0/4232907874 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b900ed800 con 0x564721e02840 2024-07-26T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.563+0000 7f6b9ea00640 1 -- 172.21.15.97:0/546852216 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f6b900e3960 con 0x7f6ba8108ee0 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.563+0000 7f6ba6800640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6b8c0b3c80 0x7f6b8c0b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 -- 172.21.15.97:0/4232907874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 msgr2=0x5647221c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221c8a40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6b900e9c20 tx=0x7f6b900eb670 comp rx=0 tx=0).stop 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 -- 172.21.15.97:0/4232907874 shutdown_connections 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221c8a40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 0x5647221c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/4232907874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 0x5647221c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 -- 172.21.15.97:0/4232907874 >> 172.21.15.97:0/4232907874 conn(0x564721e17910 msgr2=0x5647221c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6ba6800640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6b8c0b3c80 0x7f6b8c0b6190 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0005cc0 tx=0x7f6ba0005c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 -- 172.21.15.97:0/4232907874 shutdown_connections 2024-07-26T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.564+0000 7f6bb37e4740 1 -- 172.21.15.97:0/4232907874 wait complete. 2024-07-26T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.565+0000 7f6bb37e4740 1 -- start start 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.565+0000 7f6bb37e4740 1 Processor -- start 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.565+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221d3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.565+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721e02840 0x5647221d3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221e3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x564721f91bb0 con 0x564721e23520 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5647220ca910 con 0x564721e02840 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5647221dabb0 con 0x564721dfa660 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6bb37e4740 10 client.0 _ll_register_callbacks cb 0x564721e3ad70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6ba6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221e3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6ba6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221e3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34544/0 (socket says 172.21.15.97:34544) 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6ba6800640 1 -- 172.21.15.97:0/1859673513 learned_addr learned my addr 172.21.15.97:0/1859673513 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221d3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.566+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721e02840 0x5647221d3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6ba6800640 1 -- 172.21.15.97:0/1859673513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 msgr2=0x5647221d3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6ba6800640 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221d3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6ba6800640 1 -- 172.21.15.97:0/1859673513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721e02840 msgr2=0x5647221d3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6ba6800640 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721e02840 0x5647221d3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6ba6800640 1 -- 172.21.15.97:0/1859673513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564721f05960 con 0x564721e23520 2024-07-26T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6ba6800640 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221e3ed0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0035000 tx=0x7f6ba002a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7816a2ee4c21bb8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6b8b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6b8b400640 1 -- 172.21.15.97:0/1859673513 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0003c40 con 0x564721e23520 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.567+0000 7f6bb37e4740 1 -- 172.21.15.97:0/1859673513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b900eeab0 con 0x564721e23520 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.568+0000 7f6b8b400640 1 -- 172.21.15.97:0/1859673513 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f6ba0003de0 con 0x564721e23520 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.568+0000 7f6b8b400640 1 -- 172.21.15.97:0/1859673513 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6ba005c050 con 0x564721e23520 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.568+0000 7f6bb37e4740 10 client.15954 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.568+0000 7f6bb37e4740 1 -- 172.21.15.97:0/1859673513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5647221e6910 con 0x564721e23520 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.568+0000 7f6b8b400640 1 -- 172.21.15.97:0/1859673513 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6b900eeab0 con 0x564721e23520 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.568+0000 7f6b8b400640 1 -- 172.21.15.97:0/1859673513 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f6ba0051020 con 0x564721e23520 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.568+0000 7f6bb37e4740 10 client.15954 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.569+0000 7f6bb37e4740 10 client.15954 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.569+0000 7f6bb37e4740 20 client.15954 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.569+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564721de1ea0 0x5647221e9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.569+0000 7f6bb37e4740 4 client.15954 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:55:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.569+0000 7f6bb37e4740 1 -- 172.21.15.97:0/1859673513 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564722140e80 con 0x564721de1ea0 2024-07-26T07:55:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.569+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564721de1ea0 0x5647221e9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.570+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564721de1ea0 0x5647221e9e50 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f0d61d9f30bf46 server_cookie=4453c8031f92a68d in_seq=0 out_seq=0 2024-07-26T07:55:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.570+0000 7f6b8b400640 10 client.15954 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6b8b400640 1 -- 172.21.15.97:0/1859673513 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564722140e80 con 0x564721de1ea0 2024-07-26T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6b8b400640 10 client.15954 handle_command_reply: tid=0 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 client.15954 shutdown 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 -- 172.21.15.97:0/1859673513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 msgr2=0x5647221e3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221e3ed0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0035000 tx=0x7f6ba002a8b0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 -- 172.21.15.97:0/1859673513 shutdown_connections 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564721de1ea0 0x5647221e9e50 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221e3ed0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721e02840 0x5647221d3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/1859673513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221d3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6bb37e4740 1 -- 172.21.15.97:0/1859673513 >> 172.21.15.97:0/1859673513 conn(0x564721e17910 msgr2=0x5647221cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.572+0000 7f6b8b400640 0 client.15954 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.573+0000 7f6bb37e4740 1 -- 172.21.15.97:0/1859673513 shutdown_connections 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.573+0000 7f6bb37e4740 1 -- 172.21.15.97:0/1859673513 wait complete. 2024-07-26T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.573+0000 7f6bb37e4740 20 client.15954 trim_cache size 0 max 16384 2024-07-26T07:55:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.591+0000 7f6bb37e4740 1 Processor -- start 2024-07-26T07:55:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.591+0000 7f6bb37e4740 1 -- start start 2024-07-26T07:55:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.592+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 0x5647221d5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.592+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221e3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.592+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 0x5647221e4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.592+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba81accc0 con 0x564721e02840 2024-07-26T07:55:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.592+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba81ae8c0 con 0x564721dfa660 2024-07-26T07:55:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.592+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba81ad390 con 0x564721e23520 2024-07-26T07:55:46.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.592+0000 7f6b9fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221e3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.592+0000 7f6b9fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221e3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34548/0 (socket says 172.21.15.97:34548) 2024-07-26T07:55:46.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.593+0000 7f6b9fe00640 1 -- 172.21.15.97:0/2983842205 learned_addr learned my addr 172.21.15.97:0/2983842205 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.593+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 0x5647221d5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.593+0000 7f6ba6800640 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 0x5647221e4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.593+0000 7f6b9fe00640 1 -- 172.21.15.97:0/2983842205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 msgr2=0x5647221e4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.593+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 0x5647221e4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.593+0000 7f6b9fe00640 1 -- 172.21.15.97:0/2983842205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 msgr2=0x5647221d5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.593+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 0x5647221d5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.593+0000 7f6b9fe00640 1 -- 172.21.15.97:0/2983842205 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647221cb520 con 0x564721e02840 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.594+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221e3cc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000c940 tx=0x7f6b90004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c4eb248341fcdc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.594+0000 7f6b96600640 1 -- 172.21.15.97:0/2983842205 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9001d070 con 0x564721e02840 2024-07-26T07:55:46.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.594+0000 7f6b96600640 1 -- 172.21.15.97:0/2983842205 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6b900e9300 con 0x564721e02840 2024-07-26T07:55:46.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.594+0000 7f6b96600640 1 -- 172.21.15.97:0/2983842205 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b900f2070 con 0x564721e02840 2024-07-26T07:55:46.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.595+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2983842205 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 msgr2=0x5647221e3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.595+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221e3cc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000c940 tx=0x7f6b90004570 comp rx=0 tx=0).stop 2024-07-26T07:55:46.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.596+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2983842205 shutdown_connections 2024-07-26T07:55:46.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.596+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721e23520 0x5647221e4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.596+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e02840 0x5647221e3cc0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.596+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2983842205 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721dfa660 0x5647221d5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.596+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2983842205 >> 172.21.15.97:0/2983842205 conn(0x564721e17910 msgr2=0x5647221d4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.596+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2983842205 shutdown_connections 2024-07-26T07:55:46.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.596+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2983842205 wait complete. 2024-07-26T07:55:46.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.596+0000 7f6bb37e4740 1 -- start start 2024-07-26T07:55:46.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.597+0000 7f6bb37e4740 1 Processor -- start 2024-07-26T07:55:46.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.597+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221e0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.597+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221dadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.597+0000 7f6bb37e4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721db7250 0x5647221db330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.597+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba81ae130 con 0x564721e23520 2024-07-26T07:55:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.597+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba81ada60 con 0x564721db7250 2024-07-26T07:55:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.597+0000 7f6bb37e4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba81aaca0 con 0x564721dfa660 2024-07-26T07:55:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.597+0000 7f6bb37e4740 10 client.0 _ll_register_callbacks cb 0x7f6ba0030f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.598+0000 7f6ba5e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221e0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.598+0000 7f6ba5e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221e0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:48910/0 (socket says 172.21.15.97:48910) 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.598+0000 7f6ba5e00640 1 -- 172.21.15.97:0/2527486774 learned_addr learned my addr 172.21.15.97:0/2527486774 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.598+0000 7f6ba6800640 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721db7250 0x5647221db330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.598+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221dadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.599+0000 7f6ba5e00640 1 -- 172.21.15.97:0/2527486774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721db7250 msgr2=0x5647221db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.599+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721db7250 0x5647221db330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.599+0000 7f6ba5e00640 1 -- 172.21.15.97:0/2527486774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 msgr2=0x5647221dadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.599+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221dadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.599+0000 7f6ba5e00640 1 -- 172.21.15.97:0/2527486774 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647221db870 con 0x564721dfa660 2024-07-26T07:55:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.599+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221dadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:55:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.599+0000 7f6ba5e00640 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221e0210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x5647221ea580 tx=0x7f6b9800cf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6983b35be4dba112 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.599+0000 7f6b8b400640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-07-26T07:55:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.600+0000 7f6b8b400640 1 -- 172.21.15.97:0/2527486774 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9801c610 con 0x564721dfa660 2024-07-26T07:55:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.600+0000 7f6b8b400640 1 -- 172.21.15.97:0/2527486774 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7f6b9801c7b0 con 0x564721dfa660 2024-07-26T07:55:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.600+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2527486774 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b900e9300 con 0x564721dfa660 2024-07-26T07:55:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.600+0000 7f6b8b400640 1 -- 172.21.15.97:0/2527486774 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6b9802c050 con 0x564721dfa660 2024-07-26T07:55:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.600+0000 7f6bb37e4740 10 client.25465 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.600+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2527486774 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5647222532f0 con 0x564721dfa660 2024-07-26T07:55:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.601+0000 7f6b8b400640 1 -- 172.21.15.97:0/2527486774 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6b900e9300 con 0x564721dfa660 2024-07-26T07:55:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.601+0000 7f6b8b400640 1 -- 172.21.15.97:0/2527486774 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f6b98010c50 con 0x564721dfa660 2024-07-26T07:55:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.601+0000 7f6bb37e4740 10 client.25465 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.601+0000 7f6bb37e4740 10 client.25465 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:55:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.601+0000 7f6bb37e4740 20 client.25465 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.601+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564722253b60 0x564722255f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.601+0000 7f6bb37e4740 4 client.25465 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "36ce223e", "release": true, "await": true}] 2024-07-26T07:55:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.601+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2527486774 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "set_id": "36ce223e", "release": true, "await": true}) -- 0x56472213c920 con 0x564722253b60 2024-07-26T07:55:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.602+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564722253b60 0x564722255f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.603+0000 7f6b9fe00640 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564722253b60 0x564722255f80 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82c21ff7227af411 server_cookie=849bca490d8e27ab in_seq=0 out_seq=0 2024-07-26T07:55:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.603+0000 7f6b8b400640 10 client.25465 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 -- 172.21.15.97:0/560944754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240103fa0 msgr2=0x7ff2401043b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 --2- 172.21.15.97:0/560944754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240103fa0 0x7ff2401043b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff228009900 tx=0x7ff22802a410 comp rx=0 tx=0).stop 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 -- 172.21.15.97:0/560944754 shutdown_connections 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 --2- 172.21.15.97:0/560944754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff240104940 0x7ff24010b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 --2- 172.21.15.97:0/560944754 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240103fa0 0x7ff2401043b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 --2- 172.21.15.97:0/560944754 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff240102f20 0x7ff240103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 -- 172.21.15.97:0/560944754 >> 172.21.15.97:0/560944754 conn(0x7ff2400fe860 msgr2=0x7ff240100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 -- 172.21.15.97:0/560944754 shutdown_connections 2024-07-26T07:55:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.625+0000 7ff23fe00640 1 -- 172.21.15.97:0/560944754 wait complete. 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.626+0000 7ff23fe00640 1 Processor -- start 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.626+0000 7ff23fe00640 1 -- start start 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.626+0000 7ff23fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff240102f20 0x7ff24019b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.626+0000 7ff23fe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff240103fa0 0x7ff24019bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.626+0000 7ff23fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240104940 0x7ff24019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.626+0000 7ff23fe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff24010eaa0 con 0x7ff240104940 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.626+0000 7ff23fe00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff24010e920 con 0x7ff240103fa0 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.626+0000 7ff23fe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff24010ec20 con 0x7ff240102f20 2024-07-26T07:55:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240104940 0x7ff24019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240104940 0x7ff24019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34568/0 (socket says 172.21.15.97:34568) 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 -- 172.21.15.97:0/2198444372 learned_addr learned my addr 172.21.15.97:0/2198444372 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23ea00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff240102f20 0x7ff24019b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff237e00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff240103fa0 0x7ff24019bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 -- 172.21.15.97:0/2198444372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff240102f20 msgr2=0x7ff24019b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff240102f20 0x7ff24019b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 -- 172.21.15.97:0/2198444372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff240103fa0 msgr2=0x7ff24019bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff240103fa0 0x7ff24019bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 -- 172.21.15.97:0/2198444372 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24019d3a0 con 0x7ff240104940 2024-07-26T07:55:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.627+0000 7ff23f400640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240104940 0x7ff24019cb80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff23800cad0 tx=0x7ff23800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.628+0000 7ff237400640 1 -- 172.21.15.97:0/2198444372 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23800ae20 con 0x7ff240104940 2024-07-26T07:55:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.628+0000 7ff237400640 1 -- 172.21.15.97:0/2198444372 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff238005020 con 0x7ff240104940 2024-07-26T07:55:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.628+0000 7ff237400640 1 -- 172.21.15.97:0/2198444372 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23800f6b0 con 0x7ff240104940 2024-07-26T07:55:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.628+0000 7ff23fe00640 1 -- 172.21.15.97:0/2198444372 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24019d690 con 0x7ff240104940 2024-07-26T07:55:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.628+0000 7ff23fe00640 1 -- 172.21.15.97:0/2198444372 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2401a3d80 con 0x7ff240104940 2024-07-26T07:55:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.632+0000 7ff24cc0f740 1 Processor -- start 2024-07-26T07:55:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.633+0000 7ff237400640 1 -- 172.21.15.97:0/2198444372 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7ff2380051d0 con 0x7ff240104940 2024-07-26T07:55:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.633+0000 7ff24cc0f740 1 -- start start 2024-07-26T07:55:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.633+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x5586635fc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.633+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55866322d2f0 0x5586635fc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.633+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632354d0 0x5586635fd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.633+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55866356e270 con 0x5586632561b0 2024-07-26T07:55:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.633+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55866355d770 con 0x5586632354d0 2024-07-26T07:55:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.633+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5586635617b0 con 0x55866322d2f0 2024-07-26T07:55:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x5586635fc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x5586635fc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34580/0 (socket says 172.21.15.97:34580) 2024-07-26T07:55:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 -- 172.21.15.97:0/4204765867 learned_addr learned my addr 172.21.15.97:0/4204765867 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff23ea00640 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55866322d2f0 0x5586635fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632354d0 0x5586635fd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 -- 172.21.15.97:0/4204765867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55866322d2f0 msgr2=0x5586635fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55866322d2f0 0x5586635fc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 -- 172.21.15.97:0/4204765867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632354d0 msgr2=0x5586635fd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632354d0 0x5586635fd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.634+0000 7ff237e00640 1 -- 172.21.15.97:0/4204765867 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586632c1790 con 0x5586632561b0 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.635+0000 7ff237e00640 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x5586635fc0e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff228031460 tx=0x7ff228005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256e47d1c0a1a0c0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.635+0000 7ff237400640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2240b3bd0 0x7ff2240b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.635+0000 7ff22fe00640 1 -- 172.21.15.97:0/4204765867 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228046070 con 0x5586632561b0 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.635+0000 7ff22fe00640 1 -- 172.21.15.97:0/4204765867 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff228031b90 con 0x5586632561b0 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.635+0000 7ff237400640 1 -- 172.21.15.97:0/2198444372 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff2380e4110 con 0x7ff240104940 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.635+0000 7ff22fe00640 1 -- 172.21.15.97:0/4204765867 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228035070 con 0x5586632561b0 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.635+0000 7ff23ea00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2240b3bd0 0x7ff2240b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 -- 172.21.15.97:0/4204765867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 msgr2=0x5586635fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff23ea00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2240b3bd0 0x7ff2240b60e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff23000bd20 tx=0x7ff23000b4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x5586635fc0e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff228031460 tx=0x7ff228005720 comp rx=0 tx=0).stop 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 -- 172.21.15.97:0/4204765867 shutdown_connections 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632354d0 0x5586635fd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55866322d2f0 0x5586635fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/4204765867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x5586635fc0e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 -- 172.21.15.97:0/4204765867 >> 172.21.15.97:0/4204765867 conn(0x55866324a300 msgr2=0x5586635f7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 -- 172.21.15.97:0/4204765867 shutdown_connections 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.636+0000 7ff24cc0f740 1 -- 172.21.15.97:0/4204765867 wait complete. 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.637+0000 7ff24cc0f740 1 -- start start 2024-07-26T07:55:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.637+0000 7ff24cc0f740 1 Processor -- start 2024-07-26T07:55:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.637+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x558663605b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663606240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x558663606db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558663497ab0 con 0x5586632561b0 2024-07-26T07:55:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5586634fad50 con 0x55866322d2f0 2024-07-26T07:55:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5586635fdd60 con 0x5586632354d0 2024-07-26T07:55:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff24cc0f740 10 client.0 _ll_register_callbacks cb 0x55866326da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x558663606db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x558663606db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:34586/0 (socket says 172.21.15.97:34586) 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23f400640 1 -- 172.21.15.97:0/3579408446 learned_addr learned my addr 172.21.15.97:0/3579408446 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23ea00640 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x558663605b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff237e00640 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663606240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23f400640 1 -- 172.21.15.97:0/3579408446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 msgr2=0x558663606240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23f400640 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663606240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23f400640 1 -- 172.21.15.97:0/3579408446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 msgr2=0x558663605b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23f400640 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x558663605b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.638+0000 7ff23f400640 1 -- 172.21.15.97:0/3579408446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55866333a620 con 0x5586632561b0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff23f400640 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x558663606db0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff23800caa0 tx=0x7ff2380aaab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65144f8a1c944bd1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff22cc00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff22cc00640 1 -- 172.21.15.97:0/3579408446 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23801d030 con 0x5586632561b0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff22cc00640 1 -- 172.21.15.97:0/3579408446 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff2380b3bc0 con 0x5586632561b0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff24cc0f740 1 -- 172.21.15.97:0/3579408446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff228031b90 con 0x5586632561b0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff22cc00640 1 -- 172.21.15.97:0/3579408446 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff2380af360 con 0x5586632561b0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff24cc0f740 10 client.15972 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff24cc0f740 1 -- 172.21.15.97:0/3579408446 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55866360fdd0 con 0x5586632561b0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff22cc00640 1 -- 172.21.15.97:0/3579408446 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff228031b90 con 0x5586632561b0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.639+0000 7ff22cc00640 1 -- 172.21.15.97:0/3579408446 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7ff2380701b0 con 0x5586632561b0 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.640+0000 7ff24cc0f740 10 client.15972 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.640+0000 7ff24cc0f740 10 client.15972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:55:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.640+0000 7ff24cc0f740 20 client.15972 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.640+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558663214d80 0x558663613610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.640+0000 7ff24cc0f740 4 client.15972 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:55:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.640+0000 7ff24cc0f740 1 -- 172.21.15.97:0/3579408446 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586635701c0 con 0x558663214d80 2024-07-26T07:55:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.640+0000 7ff23ea00640 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558663214d80 0x558663613610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.641+0000 7ff23ea00640 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558663214d80 0x558663613610 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=862a8cf78e71bb35 server_cookie=83708ba796016749 in_seq=0 out_seq=0 2024-07-26T07:55:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.641+0000 7ff22cc00640 10 client.15972 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.811+0000 7ff22cc00640 1 -- 172.21.15.97:0/3579408446 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5586635701c0 con 0x558663214d80 2024-07-26T07:55:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.811+0000 7ff22cc00640 10 client.15972 handle_command_reply: tid=0 2024-07-26T07:55:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.811+0000 7ff24cc0f740 1 client.15972 shutdown 2024-07-26T07:55:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff24cc0f740 1 -- 172.21.15.97:0/3579408446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 msgr2=0x558663606db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x558663606db0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff23800caa0 tx=0x7ff2380aaab0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff24cc0f740 1 -- 172.21.15.97:0/3579408446 shutdown_connections 2024-07-26T07:55:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558663214d80 0x558663613610 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x558663606db0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663606240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/3579408446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x558663605b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff24cc0f740 1 -- 172.21.15.97:0/3579408446 >> 172.21.15.97:0/3579408446 conn(0x55866324a300 msgr2=0x5586635f60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.812+0000 7ff22cc00640 0 client.15972 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.813+0000 7ff24cc0f740 1 -- 172.21.15.97:0/3579408446 shutdown_connections 2024-07-26T07:55:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.813+0000 7ff24cc0f740 1 -- 172.21.15.97:0/3579408446 wait complete. 2024-07-26T07:55:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.814+0000 7ff24cc0f740 20 client.15972 trim_cache size 0 max 16384 2024-07-26T07:55:46.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.815+0000 7f6b8b400640 1 -- 172.21.15.97:0/2527486774 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56472213c920 con 0x564722253b60 2024-07-26T07:55:46.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.815+0000 7f6b8b400640 10 client.25465 handle_command_reply: tid=0 2024-07-26T07:55:46.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.816+0000 7f6bb37e4740 1 client.25465 shutdown 2024-07-26T07:55:46.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2527486774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 msgr2=0x5647221e0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221e0210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x5647221ea580 tx=0x7f6b9800cf80 comp rx=0 tx=0).stop 2024-07-26T07:55:46.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2527486774 shutdown_connections 2024-07-26T07:55:46.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x564722253b60 0x564722255f80 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564721db7250 0x5647221db330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564721e23520 0x5647221dadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6bb37e4740 1 --2- 172.21.15.97:0/2527486774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564721dfa660 0x5647221e0210 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2527486774 >> 172.21.15.97:0/2527486774 conn(0x564721e17910 msgr2=0x5647221d5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.817+0000 7f6b8b400640 0 client.25465 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:46.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.818+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2527486774 shutdown_connections 2024-07-26T07:55:46.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.818+0000 7f6bb37e4740 1 -- 172.21.15.97:0/2527486774 wait complete. 2024-07-26T07:55:46.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.818+0000 7f6bb37e4740 20 client.25465 trim_cache size 0 max 16384 2024-07-26T07:55:46.820 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:55:46.820 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 30, 2024-07-26T07:55:46.820 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:55:46.820 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 12, 2024-07-26T07:55:46.820 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:55:46.821 INFO:teuthology.orchestra.run.smithi097.stdout: "36ce223e": { 2024-07-26T07:55:46.821 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 12, 2024-07-26T07:55:46.821 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:55:46.821 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:55:46.821 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:55:46.821 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:55:46.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:55:46.821 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 151.1, 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:55:46.822 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:55:46.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.820+0000 7f6baca00640 1 -- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6b8c0b3c80 msgr2=0x7f6b8c0b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.820+0000 7f6baca00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6b8c0b3c80 0x7f6b8c0b6190 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0005cc0 tx=0x7f6ba0005c30 comp rx=0 tx=0).stop 2024-07-26T07:55:46.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.820+0000 7f6baca00640 1 -- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8108ee0 msgr2=0x7f6ba819a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.820+0000 7f6baca00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8108ee0 0x7f6ba819a3a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000c970 tx=0x7f6b9000ce30 comp rx=0 tx=0).stop 2024-07-26T07:55:46.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.824+0000 7f6baca00640 1 -- 172.21.15.97:0/546852216 shutdown_connections 2024-07-26T07:55:46.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.824+0000 7f6baca00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f6b8c0b3c80 0x7f6b8c0b6190 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.824+0000 7f6baca00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ba8109830 0x7f6ba819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.824+0000 7f6baca00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ba8108ee0 0x7f6ba819a3a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.824+0000 7f6baca00640 1 --2- 172.21.15.97:0/546852216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba81035f0 0x7f6ba8199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.824+0000 7f6baca00640 1 -- 172.21.15.97:0/546852216 >> 172.21.15.97:0/546852216 conn(0x7f6ba807b340 msgr2=0x7f6ba8106f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.824+0000 7f6baca00640 1 -- 172.21.15.97:0/546852216 shutdown_connections 2024-07-26T07:55:46.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:55:46.824+0000 7f6baca00640 1 -- 172.21.15.97:0/546852216 wait complete. 2024-07-26T07:55:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.859+0000 7ff24cc0f740 1 Processor -- start 2024-07-26T07:55:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.859+0000 7ff24cc0f740 1 -- start start 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.859+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55866322d2f0 0x558663604d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.859+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663600b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.859+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632561b0 0x558663601090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.859+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401acb70 con 0x55866322d2f0 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.859+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401ae770 con 0x5586632561b0 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.859+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401ad240 con 0x5586632354d0 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23f400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632561b0 0x558663601090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23f400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632561b0 0x558663601090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:36386/0 (socket says 172.21.15.97:36386) 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23f400640 1 -- 172.21.15.97:0/134455273 learned_addr learned my addr 172.21.15.97:0/134455273 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23ea00640 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55866322d2f0 0x558663604d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff237e00640 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663600b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23f400640 1 -- 172.21.15.97:0/134455273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 msgr2=0x558663600b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23f400640 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663600b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23f400640 1 -- 172.21.15.97:0/134455273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55866322d2f0 msgr2=0x558663604d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23f400640 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55866322d2f0 0x558663604d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.860+0000 7ff23f400640 1 -- 172.21.15.97:0/134455273 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558663603d90 con 0x5586632561b0 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.861+0000 7ff23ea00640 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55866322d2f0 0x558663604d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.861+0000 7ff237e00640 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663600b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.861+0000 7ff23f400640 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632561b0 0x558663601090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff238004ff0 tx=0x7ff2380ab120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9508a5de8163323b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.861+0000 7ff22f400640 1 -- 172.21.15.97:0/134455273 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23801d070 con 0x5586632561b0 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.861+0000 7ff22f400640 1 -- 172.21.15.97:0/134455273 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2380b4050 con 0x5586632561b0 2024-07-26T07:55:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.861+0000 7ff22f400640 1 -- 172.21.15.97:0/134455273 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2380b3bc0 con 0x5586632561b0 2024-07-26T07:55:46.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.863+0000 7ff24cc0f740 1 -- 172.21.15.97:0/134455273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632561b0 msgr2=0x558663601090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.863+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632561b0 0x558663601090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff238004ff0 tx=0x7ff2380ab120 comp rx=0 tx=0).stop 2024-07-26T07:55:46.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.863+0000 7ff24cc0f740 1 -- 172.21.15.97:0/134455273 shutdown_connections 2024-07-26T07:55:46.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.863+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5586632561b0 0x558663601090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.863+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x558663600b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.863+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/134455273 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55866322d2f0 0x558663604d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.864+0000 7ff24cc0f740 1 -- 172.21.15.97:0/134455273 >> 172.21.15.97:0/134455273 conn(0x55866324a300 msgr2=0x5586635f7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.864+0000 7ff24cc0f740 1 -- 172.21.15.97:0/134455273 shutdown_connections 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.865+0000 7ff24cc0f740 1 -- 172.21.15.97:0/134455273 wait complete. 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.865+0000 7ff24cc0f740 1 -- start start 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.866+0000 7ff24cc0f740 1 Processor -- start 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.866+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x55866360ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.866+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x55866360b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.866+0000 7ff24cc0f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x55866360c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401adfe0 con 0x5586632561b0 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401ad910 con 0x55866322d2f0 2024-07-26T07:55:46.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff24cc0f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401aab50 con 0x5586632354d0 2024-07-26T07:55:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff24cc0f740 10 client.0 _ll_register_callbacks cb 0x7ff238007700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:55:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff23ea00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x55866360ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff23f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x55866360c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff23ea00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x55866360ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:36402/0 (socket says 172.21.15.97:36402) 2024-07-26T07:55:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff23ea00640 1 -- 172.21.15.97:0/2405999520 learned_addr learned my addr 172.21.15.97:0/2405999520 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:55:46.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.867+0000 7ff237e00640 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x55866360b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.868+0000 7ff23ea00640 1 -- 172.21.15.97:0/2405999520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 msgr2=0x55866360b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.868+0000 7ff23ea00640 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x55866360b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.868+0000 7ff23ea00640 1 -- 172.21.15.97:0/2405999520 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 msgr2=0x55866360c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.868+0000 7ff23ea00640 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x55866360c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.868+0000 7ff23ea00640 1 -- 172.21.15.97:0/2405999520 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55866360c7f0 con 0x55866322d2f0 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.868+0000 7ff23f400640 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x55866360c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.869+0000 7ff23ea00640 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x55866360ae60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x558663613d70 tx=0x7ff230014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31da8a5e3dfb1141 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.869+0000 7ff22cc00640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.869+0000 7ff22cc00640 1 -- 172.21.15.97:0/2405999520 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23001fd10 con 0x55866322d2f0 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.869+0000 7ff22cc00640 1 -- 172.21.15.97:0/2405999520 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff230030470 con 0x55866322d2f0 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.869+0000 7ff24cc0f740 1 -- 172.21.15.97:0/2405999520 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2380b4050 con 0x55866322d2f0 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.870+0000 7ff22cc00640 1 -- 172.21.15.97:0/2405999520 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff2300306b0 con 0x55866322d2f0 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.870+0000 7ff24cc0f740 10 client.35432 fetch_fsmap learned FSMap version 31 2024-07-26T07:55:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.870+0000 7ff24cc0f740 1 -- 172.21.15.97:0/2405999520 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55866360dbe0 con 0x55866322d2f0 2024-07-26T07:55:46.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.871+0000 7ff22cc00640 1 -- 172.21.15.97:0/2405999520 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2380b4050 con 0x55866322d2f0 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.871+0000 7ff22cc00640 1 -- 172.21.15.97:0/2405999520 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7ff23001e410 con 0x55866322d2f0 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.871+0000 7ff24cc0f740 10 client.35432 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.871+0000 7ff24cc0f740 10 client.35432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.871+0000 7ff24cc0f740 20 client.35432 populate_metadata read hostname 'smithi097' 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.871+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5586631ea1c0 0x558663686a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.871+0000 7ff24cc0f740 4 client.35432 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.871+0000 7ff24cc0f740 1 -- 172.21.15.97:0/2405999520 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55866356bc60 con 0x5586631ea1c0 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.872+0000 7ff237e00640 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5586631ea1c0 0x558663686a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.873+0000 7ff22cc00640 10 client.35432 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:46.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.873+0000 7ff237e00640 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5586631ea1c0 0x558663686a00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64727e4c9fbb56e4 server_cookie=44dc825dbe74d4c2 in_seq=0 out_seq=0 2024-07-26T07:55:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff22cc00640 1 -- 172.21.15.97:0/2405999520 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55866356bc60 con 0x5586631ea1c0 2024-07-26T07:55:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff22cc00640 10 client.35432 handle_command_reply: tid=0 2024-07-26T07:55:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 client.35432 shutdown 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 -- 172.21.15.97:0/2405999520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 msgr2=0x55866360ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x55866360ae60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x558663613d70 tx=0x7ff230014040 comp rx=0 tx=0).stop 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 -- 172.21.15.97:0/2405999520 shutdown_connections 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5586631ea1c0 0x558663686a00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5586632561b0 0x55866360c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5586632354d0 0x55866360b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 --2- 172.21.15.97:0/2405999520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55866322d2f0 0x55866360ae60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff24cc0f740 1 -- 172.21.15.97:0/2405999520 >> 172.21.15.97:0/2405999520 conn(0x55866324a300 msgr2=0x558663604d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.958+0000 7ff22cc00640 0 client.35432 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:55:46.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.959+0000 7ff24cc0f740 1 -- 172.21.15.97:0/2405999520 shutdown_connections 2024-07-26T07:55:46.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.959+0000 7ff24cc0f740 1 -- 172.21.15.97:0/2405999520 wait complete. 2024-07-26T07:55:46.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.960+0000 7ff24cc0f740 20 client.35432 trim_cache size 0 max 16384 2024-07-26T07:55:46.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:55:46.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:55:46.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:55:46.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:55:46.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.961+0000 7ff23fe00640 1 -- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2240b3bd0 msgr2=0x7ff2240b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.961+0000 7ff23fe00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2240b3bd0 0x7ff2240b60e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff23000bd20 tx=0x7ff23000b4d0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.961+0000 7ff23fe00640 1 -- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240104940 msgr2=0x7ff24019cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:55:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.961+0000 7ff23fe00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240104940 0x7ff24019cb80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff23800cad0 tx=0x7ff23800cf90 comp rx=0 tx=0).stop 2024-07-26T07:55:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.962+0000 7ff23fe00640 1 -- 172.21.15.97:0/2198444372 shutdown_connections 2024-07-26T07:55:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.962+0000 7ff23fe00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff2240b3bd0 0x7ff2240b60e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.962+0000 7ff23fe00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff240104940 0x7ff24019cb80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.962+0000 7ff23fe00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff240103fa0 0x7ff24019bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.962+0000 7ff23fe00640 1 --2- 172.21.15.97:0/2198444372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff240102f20 0x7ff24019b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.962+0000 7ff23fe00640 1 -- 172.21.15.97:0/2198444372 >> 172.21.15.97:0/2198444372 conn(0x7ff2400fe860 msgr2=0x7ff240075020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:55:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.962+0000 7ff23fe00640 1 -- 172.21.15.97:0/2198444372 shutdown_connections 2024-07-26T07:55:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:55:46.962+0000 7ff23fe00640 1 -- 172.21.15.97:0/2198444372 wait complete. 2024-07-26T07:55:47.701 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '36ce223e', total seconds elapsed: 38.8 2024-07-26T07:55:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:48 smithi115 ceph-mon[35364]: pgmap v630: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 73 KiB/s wr, 26 op/s 2024-07-26T07:55:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:48 smithi181 ceph-mon[35869]: pgmap v630: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 73 KiB/s wr, 26 op/s 2024-07-26T07:55:48.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:48 smithi097 ceph-mon[32231]: pgmap v630: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 73 KiB/s wr, 26 op/s 2024-07-26T07:55:50.202 INFO:tasks.workunit.client.0.smithi097.stdout: 15 2482 41 1912 25 1172 43 2024-07-26T07:55:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:50 smithi115 ceph-mon[35364]: pgmap v631: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 9.2 KiB/s wr, 12 op/s 2024-07-26T07:55:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:50 smithi181 ceph-mon[35869]: pgmap v631: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 9.2 KiB/s wr, 12 op/s 2024-07-26T07:55:50.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:50 smithi097 ceph-mon[32231]: pgmap v631: 97 pgs: 97 active+clean; 52 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 9.2 KiB/s wr, 12 op/s 2024-07-26T07:55:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:52 smithi115 ceph-mon[35364]: pgmap v632: 97 pgs: 97 active+clean; 54 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 182 KiB/s wr, 20 op/s 2024-07-26T07:55:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:52 smithi181 ceph-mon[35869]: pgmap v632: 97 pgs: 97 active+clean; 54 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 182 KiB/s wr, 20 op/s 2024-07-26T07:55:52.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:52 smithi097 ceph-mon[32231]: pgmap v632: 97 pgs: 97 active+clean; 54 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 182 KiB/s wr, 20 op/s 2024-07-26T07:55:53.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:53 smithi097 ceph-mon[32231]: pgmap v633: 97 pgs: 97 active+clean; 54 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 226 KiB/s wr, 25 op/s 2024-07-26T07:55:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:53 smithi115 ceph-mon[35364]: pgmap v633: 97 pgs: 97 active+clean; 54 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 226 KiB/s wr, 25 op/s 2024-07-26T07:55:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:53 smithi181 ceph-mon[35869]: pgmap v633: 97 pgs: 97 active+clean; 54 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 226 KiB/s wr, 25 op/s 2024-07-26T07:55:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:55 smithi115 ceph-mon[35364]: pgmap v634: 97 pgs: 97 active+clean; 61 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 826 KiB/s wr, 44 op/s 2024-07-26T07:55:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:55 smithi181 ceph-mon[35869]: pgmap v634: 97 pgs: 97 active+clean; 61 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 826 KiB/s wr, 44 op/s 2024-07-26T07:55:55.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:55 smithi097 ceph-mon[32231]: pgmap v634: 97 pgs: 97 active+clean; 61 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 826 KiB/s wr, 44 op/s 2024-07-26T07:55:57.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:57 smithi115 ceph-mon[35364]: pgmap v635: 97 pgs: 97 active+clean; 63 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 1021 KiB/s wr, 58 op/s 2024-07-26T07:55:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:57 smithi181 ceph-mon[35869]: pgmap v635: 97 pgs: 97 active+clean; 63 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 1021 KiB/s wr, 58 op/s 2024-07-26T07:55:57.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:57 smithi097 ceph-mon[32231]: pgmap v635: 97 pgs: 97 active+clean; 63 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 1021 KiB/s wr, 58 op/s 2024-07-26T07:55:59.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:55:59 smithi115 ceph-mon[35364]: pgmap v636: 97 pgs: 97 active+clean; 63 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 1.0 MiB/s wr, 59 op/s 2024-07-26T07:55:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:55:59 smithi181 ceph-mon[35869]: pgmap v636: 97 pgs: 97 active+clean; 63 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 1.0 MiB/s wr, 59 op/s 2024-07-26T07:55:59.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:55:59 smithi097 ceph-mon[32231]: pgmap v636: 97 pgs: 97 active+clean; 63 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 1.0 MiB/s wr, 59 op/s 2024-07-26T07:56:00.202 INFO:tasks.workunit.client.0.smithi097.stdout: 17 6661 93 4004 194 3567 309 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 -- 172.21.15.97:0/261267779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8104930 msgr2=0x7ff5f810b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 --2- 172.21.15.97:0/261267779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8104930 0x7ff5f810b2d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009ca0 tx=0x7ff5f002a8c0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 -- 172.21.15.97:0/261267779 shutdown_connections 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 --2- 172.21.15.97:0/261267779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8104930 0x7ff5f810b2d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 --2- 172.21.15.97:0/261267779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5f8103f90 0x7ff5f81043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 --2- 172.21.15.97:0/261267779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f8102f10 0x7ff5f8103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 -- 172.21.15.97:0/261267779 >> 172.21.15.97:0/261267779 conn(0x7ff5f80fe850 msgr2=0x7ff5f8100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 -- 172.21.15.97:0/261267779 shutdown_connections 2024-07-26T07:56:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.249+0000 7ff5fee00640 1 -- 172.21.15.97:0/261267779 wait complete. 2024-07-26T07:56:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.251+0000 7ff5fee00640 1 Processor -- start 2024-07-26T07:56:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.251+0000 7ff5fee00640 1 -- start start 2024-07-26T07:56:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.251+0000 7ff5fee00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8102f10 0x7ff5f819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.251+0000 7ff5fee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f8103f90 0x7ff5f819bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.251+0000 7ff5fee00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5f8104930 0x7ff5f819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.251+0000 7ff5fee00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f810ea90 con 0x7ff5f8102f10 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.251+0000 7ff5fee00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f810e910 con 0x7ff5f8104930 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.251+0000 7ff5fee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f810ec10 con 0x7ff5f8103f90 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.252+0000 7ff5fda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8102f10 0x7ff5f819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.252+0000 7ff5fda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8102f10 0x7ff5f819b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44126/0 (socket says 172.21.15.97:44126) 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.252+0000 7ff5fda00640 1 -- 172.21.15.97:0/3253374992 learned_addr learned my addr 172.21.15.97:0/3253374992 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.252+0000 7ff5fe400640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5f8104930 0x7ff5f819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.252+0000 7ff5fd000640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f8103f90 0x7ff5f819bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5fda00640 1 -- 172.21.15.97:0/3253374992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f8103f90 msgr2=0x7ff5f819bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5fda00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f8103f90 0x7ff5f819bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5fda00640 1 -- 172.21.15.97:0/3253374992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5f8104930 msgr2=0x7ff5f819cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5fda00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5f8104930 0x7ff5f819cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5fda00640 1 -- 172.21.15.97:0/3253374992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f819d2e0 con 0x7ff5f8102f10 2024-07-26T07:56:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5fda00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8102f10 0x7ff5f819b8a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00d950 tx=0x7ff5ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5f6000640 1 -- 172.21.15.97:0/3253374992 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec017070 con 0x7ff5f8102f10 2024-07-26T07:56:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5fee00640 1 -- 172.21.15.97:0/3253374992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f819d5d0 con 0x7ff5f8102f10 2024-07-26T07:56:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5f6000640 1 -- 172.21.15.97:0/3253374992 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5ec009c30 con 0x7ff5f8102f10 2024-07-26T07:56:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.253+0000 7ff5f6000640 1 -- 172.21.15.97:0/3253374992 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec00f620 con 0x7ff5f8102f10 2024-07-26T07:56:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.254+0000 7ff5fee00640 1 -- 172.21.15.97:0/3253374992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f81a3cc0 con 0x7ff5f8102f10 2024-07-26T07:56:00.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.257+0000 7ff5f6000640 1 -- 172.21.15.97:0/3253374992 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7ff5ec010460 con 0x7ff5f8102f10 2024-07-26T07:56:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.258+0000 7ff60737b740 1 Processor -- start 2024-07-26T07:56:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.258+0000 7ff60737b740 1 -- start start 2024-07-26T07:56:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.258+0000 7ff60737b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8f61b0 0x561f8aca48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff60737b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8aca4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff60737b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8d54d0 0x561f8aca5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff60737b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561f8ac0e270 con 0x561f8a8d54d0 2024-07-26T07:56:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff60737b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561f8abfd770 con 0x561f8a8f61b0 2024-07-26T07:56:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff60737b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561f8ac017b0 con 0x561f8a8cd2f0 2024-07-26T07:56:00.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff5fe400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8f61b0 0x561f8aca48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff5fd000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8d54d0 0x561f8aca5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff5fe400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8f61b0 0x561f8aca48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:33108/0 (socket says 172.21.15.97:33108) 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff5fe400640 1 -- 172.21.15.97:0/4103921671 learned_addr learned my addr 172.21.15.97:0/4103921671 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.259+0000 7ff5fda00640 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8aca4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5f6000640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5e00b3c50 0x7ff5e00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5fd000640 1 -- 172.21.15.97:0/4103921671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 msgr2=0x561f8aca4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5fd000640 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8aca4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5fd000640 1 -- 172.21.15.97:0/4103921671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8f61b0 msgr2=0x561f8aca48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5fd000640 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8f61b0 0x561f8aca48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5fd000640 1 -- 172.21.15.97:0/4103921671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f8a961790 con 0x561f8a8d54d0 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5fe400640 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8f61b0 0x561f8aca48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5f6000640 1 -- 172.21.15.97:0/3253374992 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff5ec0e3740 con 0x7ff5f8102f10 2024-07-26T07:56:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5fe400640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5e00b3c50 0x7ff5e00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.260+0000 7ff5fd000640 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8d54d0 0x561f8aca5ab0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4005fd0 tx=0x7ff5e40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66327a47df2ba03d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.261+0000 7ff5eaa00640 1 -- 172.21.15.97:0/4103921671 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e401c480 con 0x561f8a8d54d0 2024-07-26T07:56:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.261+0000 7ff5eaa00640 1 -- 172.21.15.97:0/4103921671 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5e401cab0 con 0x561f8a8d54d0 2024-07-26T07:56:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.261+0000 7ff5eaa00640 1 -- 172.21.15.97:0/4103921671 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4011a80 con 0x561f8a8d54d0 2024-07-26T07:56:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 -- 172.21.15.97:0/4103921671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8d54d0 msgr2=0x561f8aca5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8d54d0 0x561f8aca5ab0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4005fd0 tx=0x7ff5e40107a0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff5fe400640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5e00b3c50 0x7ff5e00b6160 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009a60 tx=0x7ff5f00095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 -- 172.21.15.97:0/4103921671 shutdown_connections 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8d54d0 0x561f8aca5ab0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8aca4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 --2- 172.21.15.97:0/4103921671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8f61b0 0x561f8aca48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 -- 172.21.15.97:0/4103921671 >> 172.21.15.97:0/4103921671 conn(0x561f8a8ea300 msgr2=0x561f8aca0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 -- 172.21.15.97:0/4103921671 shutdown_connections 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.262+0000 7ff60737b740 1 -- 172.21.15.97:0/4103921671 wait complete. 2024-07-26T07:56:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.263+0000 7ff60737b740 1 -- start start 2024-07-26T07:56:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.263+0000 7ff60737b740 1 Processor -- start 2024-07-26T07:56:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.264+0000 7ff60737b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.264+0000 7ff60737b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acaea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.264+0000 7ff60737b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8f61b0 0x561f8acaf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.264+0000 7ff60737b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561f8ab37ab0 con 0x561f8a8f61b0 2024-07-26T07:56:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.264+0000 7ff60737b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x561f8ab9ad50 con 0x561f8a8d54d0 2024-07-26T07:56:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.264+0000 7ff60737b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561f8aca65d0 con 0x561f8a8cd2f0 2024-07-26T07:56:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.264+0000 7ff60737b740 10 client.0 _ll_register_callbacks cb 0x561f8a90da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.265+0000 7ff5fda00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.265+0000 7ff5fd000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acaea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.265+0000 7ff5fe400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8f61b0 0x561f8acaf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.265+0000 7ff5fda00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acae3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:37686/0 (socket says 172.21.15.97:37686) 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.265+0000 7ff5fda00640 1 -- 172.21.15.97:0/407800480 learned_addr learned my addr 172.21.15.97:0/407800480 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.265+0000 7ff5fd000640 1 -- 172.21.15.97:0/407800480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 msgr2=0x561f8acae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.265+0000 7ff5fd000640 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.265+0000 7ff5fd000640 1 -- 172.21.15.97:0/407800480 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8f61b0 msgr2=0x561f8acaf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.266+0000 7ff5fd000640 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8f61b0 0x561f8acaf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.266+0000 7ff5fd000640 1 -- 172.21.15.97:0/407800480 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f8a9da620 con 0x561f8a8d54d0 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.266+0000 7ff5fda00640 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:00.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.266+0000 7ff5fe400640 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8f61b0 0x561f8acaf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.266+0000 7ff5fd000640 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acaea60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4000c00 tx=0x7ff5e400bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1faf119abe9dc9f5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.266+0000 7ff5df400640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.267+0000 7ff5df400640 1 -- 172.21.15.97:0/407800480 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e401c680 con 0x561f8a8d54d0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.267+0000 7ff60737b740 1 -- 172.21.15.97:0/407800480 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5e401cab0 con 0x561f8a8d54d0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.267+0000 7ff5df400640 1 -- 172.21.15.97:0/407800480 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff5e401c820 con 0x561f8a8d54d0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.267+0000 7ff5df400640 1 -- 172.21.15.97:0/407800480 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff5e4011780 con 0x561f8a8d54d0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.267+0000 7ff60737b740 10 client.35444 fetch_fsmap learned FSMap version 31 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.267+0000 7ff60737b740 1 -- 172.21.15.97:0/407800480 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561f8acb8570 con 0x561f8a8d54d0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff5df400640 1 -- 172.21.15.97:0/407800480 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5e401cab0 con 0x561f8a8d54d0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff5df400640 1 -- 172.21.15.97:0/407800480 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7ff5e4027850 con 0x561f8a8d54d0 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff60737b740 10 client.35444 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:56:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff60737b740 10 client.35444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff60737b740 20 client.35444 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff60737b740 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561f8a8b4d80 0x561f8acbb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff60737b740 4 client.35444 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:56:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff60737b740 1 -- 172.21.15.97:0/407800480 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f8ac101c0 con 0x561f8a8b4d80 2024-07-26T07:56:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.268+0000 7ff5fda00640 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561f8a8b4d80 0x561f8acbb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.269+0000 7ff5fda00640 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561f8a8b4d80 0x561f8acbb990 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a618591878a6d45 server_cookie=3ccf5ef9318be24c in_seq=0 out_seq=0 2024-07-26T07:56:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.269+0000 7ff5df400640 10 client.35444 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.273+0000 7ff5df400640 1 -- 172.21.15.97:0/407800480 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561f8ac101c0 con 0x561f8a8b4d80 2024-07-26T07:56:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.273+0000 7ff5df400640 10 client.35444 handle_command_reply: tid=0 2024-07-26T07:56:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.273+0000 7ff60737b740 1 client.35444 shutdown 2024-07-26T07:56:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 -- 172.21.15.97:0/407800480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 msgr2=0x561f8acaea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acaea60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4000c00 tx=0x7ff5e400bf50 comp rx=0 tx=0).stop 2024-07-26T07:56:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 -- 172.21.15.97:0/407800480 shutdown_connections 2024-07-26T07:56:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561f8a8b4d80 0x561f8acbb990 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff5df400640 0 client.35444 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8f61b0 0x561f8acaf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acaea60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 --2- 172.21.15.97:0/407800480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 -- 172.21.15.97:0/407800480 >> 172.21.15.97:0/407800480 conn(0x561f8a8ea300 msgr2=0x561f8ac960d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 -- 172.21.15.97:0/407800480 shutdown_connections 2024-07-26T07:56:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 1 -- 172.21.15.97:0/407800480 wait complete. 2024-07-26T07:56:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.274+0000 7ff60737b740 20 client.35444 trim_cache size 0 max 16384 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.296+0000 7ff60737b740 1 Processor -- start 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.296+0000 7ff60737b740 1 -- start start 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff60737b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff60737b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff60737b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acace30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff60737b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f81a8780 con 0x561f8a8b4d80 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff60737b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f81aa380 con 0x561f8a8d54d0 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff60737b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f81a8e50 con 0x561f8a8cd2f0 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff5fda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff5fda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44144/0 (socket says 172.21.15.97:44144) 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff5fda00640 1 -- 172.21.15.97:0/652110466 learned_addr learned my addr 172.21.15.97:0/652110466 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.297+0000 7ff5fe400640 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5fd000640 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5fda00640 1 -- 172.21.15.97:0/652110466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 msgr2=0x561f8acac8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5fda00640 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acac8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5fda00640 1 -- 172.21.15.97:0/652110466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 msgr2=0x561f8acace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5fda00640 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5fda00640 1 -- 172.21.15.97:0/652110466 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f8ac91d70 con 0x561f8a8b4d80 2024-07-26T07:56:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5fd000640 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:56:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5fda00640 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acad5c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0e1230 tx=0x7ff5ec0b3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b7541a8f0ad3f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5ea000640 1 -- 172.21.15.97:0/652110466 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec020040 con 0x561f8a8b4d80 2024-07-26T07:56:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5ea000640 1 -- 172.21.15.97:0/652110466 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5ec0aa1f0 con 0x561f8a8b4d80 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.298+0000 7ff5ea000640 1 -- 172.21.15.97:0/652110466 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0b36e0 con 0x561f8a8b4d80 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.299+0000 7ff60737b740 1 -- 172.21.15.97:0/652110466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 msgr2=0x561f8acad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.299+0000 7ff60737b740 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acad5c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0e1230 tx=0x7ff5ec0b3040 comp rx=0 tx=0).stop 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.300+0000 7ff60737b740 1 -- 172.21.15.97:0/652110466 shutdown_connections 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.300+0000 7ff60737b740 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8d54d0 0x561f8acace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.300+0000 7ff60737b740 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8cd2f0 0x561f8acac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.300+0000 7ff60737b740 1 --2- 172.21.15.97:0/652110466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acad5c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.300+0000 7ff60737b740 1 -- 172.21.15.97:0/652110466 >> 172.21.15.97:0/652110466 conn(0x561f8a8ea300 msgr2=0x561f8aca9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.300+0000 7ff60737b740 1 -- 172.21.15.97:0/652110466 shutdown_connections 2024-07-26T07:56:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.300+0000 7ff60737b740 1 -- 172.21.15.97:0/652110466 wait complete. 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.300+0000 7ff60737b740 1 -- start start 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff60737b740 1 Processor -- start 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff60737b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acb2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff60737b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8cd2f0 0x561f8acb3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff60737b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8d54d0 0x561f8acb41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff60737b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f81a9bf0 con 0x561f8a8b4d80 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff60737b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f81a9520 con 0x561f8a8cd2f0 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff60737b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f81a6780 con 0x561f8a8d54d0 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff60737b740 10 client.0 _ll_register_callbacks cb 0x7ff5f0004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff5fda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acb2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff5fda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acb2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44148/0 (socket says 172.21.15.97:44148) 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff5fda00640 1 -- 172.21.15.97:0/2555953838 learned_addr learned my addr 172.21.15.97:0/2555953838 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.301+0000 7ff5fd000640 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8cd2f0 0x561f8acb3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5fda00640 1 -- 172.21.15.97:0/2555953838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8d54d0 msgr2=0x561f8acb41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:00.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5fda00640 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8d54d0 0x561f8acb41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5fda00640 1 -- 172.21.15.97:0/2555953838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8cd2f0 msgr2=0x561f8acb3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:00.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5fda00640 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8cd2f0 0x561f8acb3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5fda00640 1 -- 172.21.15.97:0/2555953838 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f8acb4950 con 0x561f8a8b4d80 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5fda00640 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acb2fc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0aa3e0 tx=0x7ff5ec0a9f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4fee1233f220df server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5df400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5df400640 1 -- 172.21.15.97:0/2555953838 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec020070 con 0x561f8a8b4d80 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff5df400640 1 -- 172.21.15.97:0/2555953838 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7ff5ec0a6cd0 con 0x561f8a8b4d80 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.302+0000 7ff60737b740 1 -- 172.21.15.97:0/2555953838 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5ec0aa1f0 con 0x561f8a8b4d80 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.303+0000 7ff5df400640 1 -- 172.21.15.97:0/2555953838 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff5ec0c6050 con 0x561f8a8b4d80 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.303+0000 7ff60737b740 10 client.15996 fetch_fsmap learned FSMap version 31 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.303+0000 7ff60737b740 1 -- 172.21.15.97:0/2555953838 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561f8acb5be0 con 0x561f8a8b4d80 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.303+0000 7ff5df400640 1 -- 172.21.15.97:0/2555953838 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5ec0aa1f0 con 0x561f8a8b4d80 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.304+0000 7ff5df400640 1 -- 172.21.15.97:0/2555953838 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7ff5ec0e3c50 con 0x561f8a8b4d80 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.304+0000 7ff60737b740 10 client.15996 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.304+0000 7ff60737b740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.304+0000 7ff60737b740 20 client.15996 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.304+0000 7ff60737b740 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561f8a8f61b0 0x561f8ad26150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.304+0000 7ff60737b740 4 client.15996 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:56:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.304+0000 7ff60737b740 1 -- 172.21.15.97:0/2555953838 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f8ac0bc60 con 0x561f8a8f61b0 2024-07-26T07:56:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.304+0000 7ff5fd000640 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561f8a8f61b0 0x561f8ad26150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.305+0000 7ff5df400640 10 client.15996 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.305+0000 7ff5fd000640 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561f8a8f61b0 0x561f8ad26150 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69edc14948a3ef00 server_cookie=629c1a27628331aa in_seq=0 out_seq=0 2024-07-26T07:56:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.653+0000 7ff5df400640 1 -- 172.21.15.97:0/2555953838 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f8ac0bc60 con 0x561f8a8f61b0 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.653+0000 7ff5df400640 10 client.15996 handle_command_reply: tid=0 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.653+0000 7ff60737b740 1 client.15996 shutdown 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 -- 172.21.15.97:0/2555953838 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 msgr2=0x561f8acb2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acb2fc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0aa3e0 tx=0x7ff5ec0a9f20 comp rx=0 tx=0).stop 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 -- 172.21.15.97:0/2555953838 shutdown_connections 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x561f8a8f61b0 0x561f8ad26150 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561f8a8d54d0 0x561f8acb41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f8a8cd2f0 0x561f8acb3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff5df400640 0 client.15996 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 --2- 172.21.15.97:0/2555953838 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561f8a8b4d80 0x561f8acb2fc0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 -- 172.21.15.97:0/2555953838 >> 172.21.15.97:0/2555953838 conn(0x561f8a8ea300 msgr2=0x561f8acac820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.654+0000 7ff60737b740 1 -- 172.21.15.97:0/2555953838 shutdown_connections 2024-07-26T07:56:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.655+0000 7ff60737b740 1 -- 172.21.15.97:0/2555953838 wait complete. 2024-07-26T07:56:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.655+0000 7ff60737b740 20 client.15996 trim_cache size 0 max 16384 2024-07-26T07:56:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:56:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:56:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:56:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 -- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5e00b3c50 msgr2=0x7ff5e00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5e00b3c50 0x7ff5e00b6160 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009a60 tx=0x7ff5f00095c0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 -- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8102f10 msgr2=0x7ff5f819b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8102f10 0x7ff5f819b8a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00d950 tx=0x7ff5ec00de10 comp rx=0 tx=0).stop 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 -- 172.21.15.97:0/3253374992 shutdown_connections 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff5e00b3c50 0x7ff5e00b6160 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5f8104930 0x7ff5f819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5f8103f90 0x7ff5f819bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 --2- 172.21.15.97:0/3253374992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5f8102f10 0x7ff5f819b8a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 -- 172.21.15.97:0/3253374992 >> 172.21.15.97:0/3253374992 conn(0x7ff5f80fe850 msgr2=0x7ff5f8075020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 -- 172.21.15.97:0/3253374992 shutdown_connections 2024-07-26T07:56:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:00.657+0000 7ff5fee00640 1 -- 172.21.15.97:0/3253374992 wait complete. 2024-07-26T07:56:01.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:01 smithi115 ceph-mon[35364]: pgmap v637: 97 pgs: 97 active+clean; 70 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-07-26T07:56:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:01 smithi181 ceph-mon[35869]: pgmap v637: 97 pgs: 97 active+clean; 70 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-07-26T07:56:01.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:01 smithi097 ceph-mon[32231]: pgmap v637: 97 pgs: 97 active+clean; 70 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-07-26T07:56:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:03 smithi115 ceph-mon[35364]: pgmap v638: 97 pgs: 97 active+clean; 67 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 114 op/s 2024-07-26T07:56:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:03 smithi181 ceph-mon[35869]: pgmap v638: 97 pgs: 97 active+clean; 67 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 114 op/s 2024-07-26T07:56:03.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:03 smithi097 ceph-mon[32231]: pgmap v638: 97 pgs: 97 active+clean; 67 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 114 op/s 2024-07-26T07:56:05.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:05 smithi115 ceph-mon[35364]: pgmap v639: 97 pgs: 97 active+clean; 65 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-07-26T07:56:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:05 smithi181 ceph-mon[35869]: pgmap v639: 97 pgs: 97 active+clean; 65 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-07-26T07:56:05.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:05 smithi097 ceph-mon[32231]: pgmap v639: 97 pgs: 97 active+clean; 65 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-07-26T07:56:07.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:07 smithi115 ceph-mon[35364]: pgmap v640: 97 pgs: 97 active+clean; 68 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 287 op/s 2024-07-26T07:56:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:56:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:07 smithi181 ceph-mon[35869]: pgmap v640: 97 pgs: 97 active+clean; 68 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 287 op/s 2024-07-26T07:56:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:56:07.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:07 smithi097 ceph-mon[32231]: pgmap v640: 97 pgs: 97 active+clean; 68 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 287 op/s 2024-07-26T07:56:07.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T07:56:08.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T07:56:08.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:56:08.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:56:08.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T07:56:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T07:56:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:56:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:56:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:56:08.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:56:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T07:56:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T07:56:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:56:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:56:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:56:10.202 INFO:tasks.workunit.client.0.smithi097.stdout: 22 6683 261 3577 204 2328 256 2024-07-26T07:56:10.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:10 smithi115 ceph-mon[35364]: pgmap v641: 97 pgs: 97 active+clean; 69 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-07-26T07:56:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:10 smithi181 ceph-mon[35869]: pgmap v641: 97 pgs: 97 active+clean; 69 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-07-26T07:56:10.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:10 smithi097 ceph-mon[32231]: pgmap v641: 97 pgs: 97 active+clean; 69 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-07-26T07:56:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:12 smithi115 ceph-mon[35364]: pgmap v642: 97 pgs: 97 active+clean; 77 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 313 op/s 2024-07-26T07:56:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:12 smithi181 ceph-mon[35869]: pgmap v642: 97 pgs: 97 active+clean; 77 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 313 op/s 2024-07-26T07:56:12.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:12 smithi097 ceph-mon[32231]: pgmap v642: 97 pgs: 97 active+clean; 77 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 313 op/s 2024-07-26T07:56:13.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.954+0000 7fd456600640 1 -- 172.21.15.97:0/2596569923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45006b140 msgr2=0x7fd450105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:13.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.954+0000 7fd456600640 1 --2- 172.21.15.97:0/2596569923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45006b140 0x7fd450105bb0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd448009ca0 tx=0x7fd44802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.954+0000 7fd456600640 1 -- 172.21.15.97:0/2596569923 shutdown_connections 2024-07-26T07:56:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.954+0000 7fd456600640 1 --2- 172.21.15.97:0/2596569923 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd450108cb0 0x7fd45010b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.954+0000 7fd456600640 1 --2- 172.21.15.97:0/2596569923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4501060f0 0x7fd450108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.954+0000 7fd456600640 1 --2- 172.21.15.97:0/2596569923 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45006b140 0x7fd450105bb0 secure :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd448009ca0 tx=0x7fd44802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.954+0000 7fd456600640 1 -- 172.21.15.97:0/2596569923 >> 172.21.15.97:0/2596569923 conn(0x7fd4500fd7b0 msgr2=0x7fd4500ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.955+0000 7fd456600640 1 -- 172.21.15.97:0/2596569923 shutdown_connections 2024-07-26T07:56:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.955+0000 7fd456600640 1 -- 172.21.15.97:0/2596569923 wait complete. 2024-07-26T07:56:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.957+0000 7fd456600640 1 Processor -- start 2024-07-26T07:56:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.957+0000 7fd456600640 1 -- start start 2024-07-26T07:56:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.958+0000 7fd456600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4501060f0 0x7fd45019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.958+0000 7fd456600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd450108cb0 0x7fd45019c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.958+0000 7fd456600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45019cb70 0x7fd45019d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.958+0000 7fd456600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd450110c60 con 0x7fd45019cb70 2024-07-26T07:56:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.958+0000 7fd456600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd450110ae0 con 0x7fd4501060f0 2024-07-26T07:56:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.958+0000 7fd456600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd450110de0 con 0x7fd450108cb0 2024-07-26T07:56:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.958+0000 7fd455c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45019cb70 0x7fd45019d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd455c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45019cb70 0x7fd45019d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41860/0 (socket says 172.21.15.97:41860) 2024-07-26T07:56:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd455c00640 1 -- 172.21.15.97:0/2155067812 learned_addr learned my addr 172.21.15.97:0/2155067812 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd44fe00640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd450108cb0 0x7fd45019c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd455200640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4501060f0 0x7fd45019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd455c00640 1 -- 172.21.15.97:0/2155067812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd450108cb0 msgr2=0x7fd45019c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd455c00640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd450108cb0 0x7fd45019c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd455c00640 1 -- 172.21.15.97:0/2155067812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4501060f0 msgr2=0x7fd45019ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd455c00640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4501060f0 0x7fd45019ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.959+0000 7fd455c00640 1 -- 172.21.15.97:0/2155067812 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45019d890 con 0x7fd45019cb70 2024-07-26T07:56:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.960+0000 7fd455c00640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45019cb70 0x7fd45019d070 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00e970 tx=0x7fd43c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.960+0000 7fd44d600640 1 -- 172.21.15.97:0/2155067812 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c00ccc0 con 0x7fd45019cb70 2024-07-26T07:56:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.960+0000 7fd44d600640 1 -- 172.21.15.97:0/2155067812 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd43c00ce60 con 0x7fd45019cb70 2024-07-26T07:56:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.960+0000 7fd44d600640 1 -- 172.21.15.97:0/2155067812 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c010430 con 0x7fd45019cb70 2024-07-26T07:56:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.961+0000 7fd456600640 1 -- 172.21.15.97:0/2155067812 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4501a3c20 con 0x7fd45019cb70 2024-07-26T07:56:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.961+0000 7fd456600640 1 -- 172.21.15.97:0/2155067812 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4501a4120 con 0x7fd45019cb70 2024-07-26T07:56:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.969+0000 7fd45eb9a740 1 Processor -- start 2024-07-26T07:56:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.969+0000 7fd44d600640 1 -- 172.21.15.97:0/2155067812 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fd43c004ef0 con 0x7fd45019cb70 2024-07-26T07:56:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.969+0000 7fd45eb9a740 1 -- start start 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.969+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a71a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.969+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a71af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a71bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55e90a684270 con 0x55e90a36c1b0 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e90a673770 con 0x55e90a3432f0 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e90a6777b0 con 0x55e90a34b4d0 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd455200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a71af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd455200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a71af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:60290/0 (socket says 172.21.15.97:60290) 2024-07-26T07:56:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd455200640 1 -- 172.21.15.97:0/3989469043 learned_addr learned my addr 172.21.15.97:0/3989469043 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd44fe00640 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a71bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd455200640 1 -- 172.21.15.97:0/3989469043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 msgr2=0x55e90a71bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd455200640 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a71bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.970+0000 7fd455200640 1 -- 172.21.15.97:0/3989469043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 msgr2=0x55e90a71a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.971+0000 7fd44fe00640 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a71a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.971+0000 7fd455200640 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a71a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.971+0000 7fd455200640 1 -- 172.21.15.97:0/3989469043 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e90a3d7790 con 0x55e90a3432f0 2024-07-26T07:56:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.971+0000 7fd455200640 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a71af60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd45006a7e0 tx=0x7fd448009910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4fd9c52f8516604 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.972+0000 7fd44fe00640 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a71a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:56:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.972+0000 7fd446000640 1 -- 172.21.15.97:0/3989469043 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448031450 con 0x55e90a3432f0 2024-07-26T07:56:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.972+0000 7fd446000640 1 -- 172.21.15.97:0/3989469043 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd448036040 con 0x55e90a3432f0 2024-07-26T07:56:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.972+0000 7fd446000640 1 -- 172.21.15.97:0/3989469043 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448003970 con 0x55e90a3432f0 2024-07-26T07:56:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.973+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3989469043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 msgr2=0x55e90a71af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.973+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a71af60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd45006a7e0 tx=0x7fd448009910 comp rx=0 tx=0).stop 2024-07-26T07:56:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.973+0000 7fd44d600640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4340b3d00 0x7fd4340b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.974+0000 7fd44d600640 1 -- 172.21.15.97:0/2155067812 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd43c0e41b0 con 0x7fd45019cb70 2024-07-26T07:56:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.974+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3989469043 shutdown_connections 2024-07-26T07:56:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.974+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a71bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.974+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a71af60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.974+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3989469043 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a71a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.974+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3989469043 >> 172.21.15.97:0/3989469043 conn(0x55e90a360300 msgr2=0x55e90a716380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.974+0000 7fd44fe00640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4340b3d00 0x7fd4340b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.975+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3989469043 shutdown_connections 2024-07-26T07:56:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.975+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3989469043 wait complete. 2024-07-26T07:56:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.975+0000 7fd44fe00640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4340b3d00 0x7fd4340b6210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd4400058c0 tx=0x7fd440005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.976+0000 7fd45eb9a740 1 -- start start 2024-07-26T07:56:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.976+0000 7fd45eb9a740 1 Processor -- start 2024-07-26T07:56:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.976+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a7242a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.976+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a724960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.976+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a7254d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55e90a5adab0 con 0x55e90a36c1b0 2024-07-26T07:56:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd455c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a7254d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd455200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a7242a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55e90a610d50 con 0x55e90a3432f0 2024-07-26T07:56:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e90a71c570 con 0x55e90a34b4d0 2024-07-26T07:56:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd45eb9a740 10 client.0 _ll_register_callbacks cb 0x55e90a383a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd455200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a7242a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:60294/0 (socket says 172.21.15.97:60294) 2024-07-26T07:56:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd455c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a7254d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41876/0 (socket says 172.21.15.97:41876) 2024-07-26T07:56:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd455c00640 1 -- 172.21.15.97:0/2979147562 learned_addr learned my addr 172.21.15.97:0/2979147562 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.977+0000 7fd44fe00640 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a724960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.978+0000 7fd455200640 1 -- 172.21.15.97:0/2979147562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 msgr2=0x55e90a724960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.978+0000 7fd455200640 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a724960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.978+0000 7fd455200640 1 -- 172.21.15.97:0/2979147562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 msgr2=0x55e90a7254d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.978+0000 7fd455200640 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a7254d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.978+0000 7fd455200640 1 -- 172.21.15.97:0/2979147562 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e90a450620 con 0x55e90a3432f0 2024-07-26T07:56:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.978+0000 7fd455c00640 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a7254d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.978+0000 7fd455200640 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a7242a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd44802a900 tx=0x7fd448030820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fbee0f33a4ab7a3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.978+0000 7fd43aa00640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:56:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.979+0000 7fd43aa00640 1 -- 172.21.15.97:0/2979147562 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44804a040 con 0x55e90a3432f0 2024-07-26T07:56:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.979+0000 7fd45eb9a740 1 -- 172.21.15.97:0/2979147562 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd448036040 con 0x55e90a3432f0 2024-07-26T07:56:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.980+0000 7fd43aa00640 1 -- 172.21.15.97:0/2979147562 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd448031450 con 0x55e90a3432f0 2024-07-26T07:56:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.980+0000 7fd43aa00640 1 -- 172.21.15.97:0/2979147562 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd448031650 con 0x55e90a3432f0 2024-07-26T07:56:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.980+0000 7fd45eb9a740 10 client.35462 fetch_fsmap learned FSMap version 31 2024-07-26T07:56:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.980+0000 7fd45eb9a740 1 -- 172.21.15.97:0/2979147562 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e90a72e390 con 0x55e90a3432f0 2024-07-26T07:56:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.981+0000 7fd43aa00640 1 -- 172.21.15.97:0/2979147562 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd448036040 con 0x55e90a3432f0 2024-07-26T07:56:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.982+0000 7fd43aa00640 1 -- 172.21.15.97:0/2979147562 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fd44804d070 con 0x55e90a3432f0 2024-07-26T07:56:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.982+0000 7fd45eb9a740 10 client.35462 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:56:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.982+0000 7fd45eb9a740 10 client.35462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.982+0000 7fd45eb9a740 20 client.35462 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.982+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55e90a32ad80 0x55e90a731860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.982+0000 7fd45eb9a740 4 client.35462 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:56:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.982+0000 7fd45eb9a740 1 -- 172.21.15.97:0/2979147562 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e90a6861c0 con 0x55e90a32ad80 2024-07-26T07:56:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.983+0000 7fd44fe00640 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55e90a32ad80 0x55e90a731860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.984+0000 7fd44fe00640 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55e90a32ad80 0x55e90a731860 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=438408fb14c45568 server_cookie=6673ae97cfc9afa0 in_seq=0 out_seq=0 2024-07-26T07:56:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.984+0000 7fd43aa00640 10 client.35462 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.985+0000 7fd43aa00640 1 -- 172.21.15.97:0/2979147562 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e90a6861c0 con 0x55e90a32ad80 2024-07-26T07:56:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.985+0000 7fd43aa00640 10 client.35462 handle_command_reply: tid=0 2024-07-26T07:56:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.985+0000 7fd45eb9a740 1 client.35462 shutdown 2024-07-26T07:56:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 -- 172.21.15.97:0/2979147562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 msgr2=0x55e90a7242a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a7242a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd44802a900 tx=0x7fd448030820 comp rx=0 tx=0).stop 2024-07-26T07:56:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 -- 172.21.15.97:0/2979147562 shutdown_connections 2024-07-26T07:56:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55e90a32ad80 0x55e90a731860 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a7254d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a724960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/2979147562 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3432f0 0x55e90a7242a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 -- 172.21.15.97:0/2979147562 >> 172.21.15.97:0/2979147562 conn(0x55e90a360300 msgr2=0x55e90a70c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd43aa00640 0 client.35462 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 -- 172.21.15.97:0/2979147562 shutdown_connections 2024-07-26T07:56:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 1 -- 172.21.15.97:0/2979147562 wait complete. 2024-07-26T07:56:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:13.986+0000 7fd45eb9a740 20 client.35462 trim_cache size 0 max 16384 2024-07-26T07:56:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.012+0000 7fd45eb9a740 1 Processor -- start 2024-07-26T07:56:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.012+0000 7fd45eb9a740 1 -- start start 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.013+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a3432f0 0x55e90a71b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.013+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.013+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a36c1b0 0x55e90a716f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.013+0000 7fd455200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a3432f0 0x55e90a71b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.013+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501a8c70 con 0x55e90a3432f0 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501aa870 con 0x55e90a36c1b0 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501a9340 con 0x55e90a34b4d0 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd455200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a3432f0 0x55e90a71b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41882/0 (socket says 172.21.15.97:41882) 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd455200640 1 -- 172.21.15.97:0/396729924 learned_addr learned my addr 172.21.15.97:0/396729924 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd44fe00640 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:14.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd455c00640 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a36c1b0 0x55e90a716f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:14.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd455200640 1 -- 172.21.15.97:0/396729924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 msgr2=0x55e90a72c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:14.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd455200640 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd455200640 1 -- 172.21.15.97:0/396729924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a36c1b0 msgr2=0x55e90a716f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:14.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd455200640 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a36c1b0 0x55e90a716f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.014+0000 7fd455200640 1 -- 172.21.15.97:0/396729924 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e90a725840 con 0x55e90a3432f0 2024-07-26T07:56:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.015+0000 7fd455200640 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a3432f0 0x55e90a71b780 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd44802ade0 tx=0x7fd448030ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131efd7e62addab server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.015+0000 7fd445600640 1 -- 172.21.15.97:0/396729924 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448065070 con 0x55e90a3432f0 2024-07-26T07:56:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.015+0000 7fd445600640 1 -- 172.21.15.97:0/396729924 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd448036040 con 0x55e90a3432f0 2024-07-26T07:56:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.015+0000 7fd445600640 1 -- 172.21.15.97:0/396729924 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448004830 con 0x55e90a3432f0 2024-07-26T07:56:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 -- 172.21.15.97:0/396729924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a3432f0 msgr2=0x55e90a71b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a3432f0 0x55e90a71b780 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd44802ade0 tx=0x7fd448030ae0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 -- 172.21.15.97:0/396729924 shutdown_connections 2024-07-26T07:56:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a36c1b0 0x55e90a716f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/396729924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a3432f0 0x55e90a71b780 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 -- 172.21.15.97:0/396729924 >> 172.21.15.97:0/396729924 conn(0x55e90a360300 msgr2=0x55e90a723e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 -- 172.21.15.97:0/396729924 shutdown_connections 2024-07-26T07:56:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.017+0000 7fd45eb9a740 1 -- 172.21.15.97:0/396729924 wait complete. 2024-07-26T07:56:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.018+0000 7fd45eb9a740 1 -- start start 2024-07-26T07:56:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.018+0000 7fd45eb9a740 1 Processor -- start 2024-07-26T07:56:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.019+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.019+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a72c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.019+0000 7fd45eb9a740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3001c0 0x55e90a72abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.019+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501aa0e0 con 0x55e90a36c1b0 2024-07-26T07:56:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.019+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501a9a10 con 0x55e90a3001c0 2024-07-26T07:56:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.019+0000 7fd45eb9a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501a6c50 con 0x55e90a34b4d0 2024-07-26T07:56:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.019+0000 7fd45eb9a740 10 client.0 _ll_register_callbacks cb 0x7fd448009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.020+0000 7fd44fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a72c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.020+0000 7fd455200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.020+0000 7fd455200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:49024/0 (socket says 172.21.15.97:49024) 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.020+0000 7fd455200640 1 -- 172.21.15.97:0/3156707627 learned_addr learned my addr 172.21.15.97:0/3156707627 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.020+0000 7fd455c00640 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3001c0 0x55e90a72abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.021+0000 7fd455200640 1 -- 172.21.15.97:0/3156707627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3001c0 msgr2=0x55e90a72abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.021+0000 7fd455200640 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3001c0 0x55e90a72abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.021+0000 7fd455200640 1 -- 172.21.15.97:0/3156707627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 msgr2=0x55e90a72c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.021+0000 7fd455200640 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a72c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.021+0000 7fd455200640 1 -- 172.21.15.97:0/3156707627 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e90a72b380 con 0x55e90a34b4d0 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.021+0000 7fd44fe00640 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a72c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.021+0000 7fd455200640 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72bd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd4480094e0 tx=0x7fd448000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74944d961693ef2f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.021+0000 7fd43aa00640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-07-26T07:56:14.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.022+0000 7fd43aa00640 1 -- 172.21.15.97:0/3156707627 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44806e060 con 0x55e90a34b4d0 2024-07-26T07:56:14.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.022+0000 7fd43aa00640 1 -- 172.21.15.97:0/3156707627 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8327+0+0 (secure 0 0 0) 0x7fd448031a80 con 0x55e90a34b4d0 2024-07-26T07:56:14.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.022+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3156707627 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd448036040 con 0x55e90a34b4d0 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.023+0000 7fd43aa00640 1 -- 172.21.15.97:0/3156707627 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd448065040 con 0x55e90a34b4d0 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.023+0000 7fd45eb9a740 10 client.25504 fetch_fsmap learned FSMap version 31 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.024+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3156707627 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e90a72a210 con 0x55e90a34b4d0 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.024+0000 7fd43aa00640 1 -- 172.21.15.97:0/3156707627 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd448036040 con 0x55e90a34b4d0 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.024+0000 7fd43aa00640 1 -- 172.21.15.97:0/3156707627 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fd448072030 con 0x55e90a34b4d0 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.024+0000 7fd45eb9a740 10 client.25504 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.024+0000 7fd45eb9a740 10 client.25504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.024+0000 7fd45eb9a740 20 client.25504 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.025+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55e90a7ab5f0 0x55e90a7ada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.025+0000 7fd45eb9a740 4 client.25504 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:56:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.025+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3156707627 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e90a681c60 con 0x55e90a7ab5f0 2024-07-26T07:56:14.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.025+0000 7fd44fe00640 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55e90a7ab5f0 0x55e90a7ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:14.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.027+0000 7fd43aa00640 10 client.25504 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:14.028 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.027+0000 7fd44fe00640 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55e90a7ab5f0 0x55e90a7ada10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5778558adc903b80 server_cookie=2ede3f394699667f in_seq=0 out_seq=0 2024-07-26T07:56:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:14 smithi115 ceph-mon[35364]: pgmap v643: 97 pgs: 97 active+clean; 81 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 291 op/s 2024-07-26T07:56:14.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:14 smithi181 ceph-mon[35869]: pgmap v643: 97 pgs: 97 active+clean; 81 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 291 op/s 2024-07-26T07:56:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:14 smithi097 ceph-mon[32231]: pgmap v643: 97 pgs: 97 active+clean; 81 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 291 op/s 2024-07-26T07:56:14.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd43aa00640 1 -- 172.21.15.97:0/3156707627 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e90a681c60 con 0x55e90a7ab5f0 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd43aa00640 10 client.25504 handle_command_reply: tid=0 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 client.25504 shutdown 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3156707627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 msgr2=0x55e90a72bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72bd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd4480094e0 tx=0x7fd448000f40 comp rx=0 tx=0).stop 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3156707627 shutdown_connections 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55e90a7ab5f0 0x55e90a7ada10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd43aa00640 0 client.25504 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55e90a3001c0 0x55e90a72abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e90a36c1b0 0x55e90a72c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 --2- 172.21.15.97:0/3156707627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e90a34b4d0 0x55e90a72bd60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.688+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3156707627 >> 172.21.15.97:0/3156707627 conn(0x55e90a360300 msgr2=0x55e90a7178e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.689+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3156707627 shutdown_connections 2024-07-26T07:56:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.689+0000 7fd45eb9a740 1 -- 172.21.15.97:0/3156707627 wait complete. 2024-07-26T07:56:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.689+0000 7fd45eb9a740 20 client.25504 trim_cache size 0 max 16384 2024-07-26T07:56:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:56:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:56:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:56:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:56:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.690+0000 7fd456600640 1 -- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4340b3d00 msgr2=0x7fd4340b6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.690+0000 7fd456600640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4340b3d00 0x7fd4340b6210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd4400058c0 tx=0x7fd440005850 comp rx=0 tx=0).stop 2024-07-26T07:56:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.690+0000 7fd456600640 1 -- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45019cb70 msgr2=0x7fd45019d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.690+0000 7fd456600640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45019cb70 0x7fd45019d070 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00e970 tx=0x7fd43c00ed70 comp rx=0 tx=0).stop 2024-07-26T07:56:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.690+0000 7fd456600640 1 -- 172.21.15.97:0/2155067812 shutdown_connections 2024-07-26T07:56:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.690+0000 7fd456600640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fd4340b3d00 0x7fd4340b6210 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.690+0000 7fd456600640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd45019cb70 0x7fd45019d070 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.690+0000 7fd456600640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd450108cb0 0x7fd45019c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.691+0000 7fd456600640 1 --2- 172.21.15.97:0/2155067812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4501060f0 0x7fd45019ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.691+0000 7fd456600640 1 -- 172.21.15.97:0/2155067812 >> 172.21.15.97:0/2155067812 conn(0x7fd4500fd7b0 msgr2=0x7fd450104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.691+0000 7fd456600640 1 -- 172.21.15.97:0/2155067812 shutdown_connections 2024-07-26T07:56:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:14.691+0000 7fd456600640 1 -- 172.21.15.97:0/2155067812 wait complete. 2024-07-26T07:56:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:15 smithi115 ceph-mon[35364]: daemon mds.d restarted 2024-07-26T07:56:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:15 smithi181 ceph-mon[35869]: daemon mds.d restarted 2024-07-26T07:56:15.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:56:15 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: ignoring --setuser ceph since I am not root 2024-07-26T07:56:15.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:56:15 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: ignoring --setgroup ceph since I am not root 2024-07-26T07:56:15.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:56:15 smithi097 ceph-mds[75584]: -- 172.21.15.97:0/1529519313 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56401ac37860 con 0x56401edd9180 2024-07-26T07:56:15.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 07:56:15 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: starting mds.d at 2024-07-26T07:56:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:15 smithi097 ceph-mon[32231]: daemon mds.d restarted 2024-07-26T07:56:16.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:16 smithi097 ceph-mon[32231]: pgmap v644: 97 pgs: 97 active+clean; 88 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-07-26T07:56:16.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:16 smithi097 ceph-mon[32231]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T07:56:16.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:16 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:boot 2024-07-26T07:56:16.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:16 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:56:16.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T07:56:16.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:16 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:56:16.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:16 smithi115 ceph-mon[35364]: pgmap v644: 97 pgs: 97 active+clean; 88 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-07-26T07:56:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:16 smithi115 ceph-mon[35364]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T07:56:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:16 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:boot 2024-07-26T07:56:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:16 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:56:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T07:56:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:16 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:56:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:16 smithi181 ceph-mon[35869]: pgmap v644: 97 pgs: 97 active+clean; 88 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-07-26T07:56:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:16 smithi181 ceph-mon[35869]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T07:56:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:16 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:boot 2024-07-26T07:56:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:16 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:56:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T07:56:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:16 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:56:18.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:18 smithi115 ceph-mon[35364]: pgmap v646: 97 pgs: 97 active+clean; 83 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 180 op/s 2024-07-26T07:56:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:18 smithi181 ceph-mon[35869]: pgmap v646: 97 pgs: 97 active+clean; 83 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 180 op/s 2024-07-26T07:56:18.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:18 smithi097 ceph-mon[32231]: pgmap v646: 97 pgs: 97 active+clean; 83 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 180 op/s 2024-07-26T07:56:20.202 INFO:tasks.workunit.client.0.smithi097.stdout: 25 8386 152 5403 208 3096 181 2024-07-26T07:56:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:20 smithi115 ceph-mon[35364]: pgmap v647: 97 pgs: 97 active+clean; 83 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 196 op/s 2024-07-26T07:56:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:20 smithi181 ceph-mon[35869]: pgmap v647: 97 pgs: 97 active+clean; 83 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 196 op/s 2024-07-26T07:56:20.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:20 smithi097 ceph-mon[32231]: pgmap v647: 97 pgs: 97 active+clean; 83 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 196 op/s 2024-07-26T07:56:21.473 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:21 smithi115 ceph-mon[35364]: pgmap v648: 97 pgs: 97 active+clean; 82 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 2.4 MiB/s wr, 325 op/s 2024-07-26T07:56:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:21 smithi181 ceph-mon[35869]: pgmap v648: 97 pgs: 97 active+clean; 82 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 2.4 MiB/s wr, 325 op/s 2024-07-26T07:56:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:21 smithi097 ceph-mon[32231]: pgmap v648: 97 pgs: 97 active+clean; 82 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 2.4 MiB/s wr, 325 op/s 2024-07-26T07:56:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:23 smithi115 ceph-mon[35364]: pgmap v649: 97 pgs: 97 active+clean; 85 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-07-26T07:56:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:23 smithi181 ceph-mon[35869]: pgmap v649: 97 pgs: 97 active+clean; 85 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-07-26T07:56:23.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:23 smithi097 ceph-mon[32231]: pgmap v649: 97 pgs: 97 active+clean; 85 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-07-26T07:56:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:25 smithi115 ceph-mon[35364]: pgmap v650: 97 pgs: 97 active+clean; 90 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.0 MiB/s wr, 308 op/s 2024-07-26T07:56:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:25 smithi181 ceph-mon[35869]: pgmap v650: 97 pgs: 97 active+clean; 90 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.0 MiB/s wr, 308 op/s 2024-07-26T07:56:25.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:25 smithi097 ceph-mon[32231]: pgmap v650: 97 pgs: 97 active+clean; 90 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.0 MiB/s wr, 308 op/s 2024-07-26T07:56:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:27 smithi115 ceph-mon[35364]: pgmap v651: 97 pgs: 97 active+clean; 97 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 295 op/s 2024-07-26T07:56:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:27 smithi181 ceph-mon[35869]: pgmap v651: 97 pgs: 97 active+clean; 97 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 295 op/s 2024-07-26T07:56:27.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:27 smithi097 ceph-mon[32231]: pgmap v651: 97 pgs: 97 active+clean; 97 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 295 op/s 2024-07-26T07:56:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 -- 172.21.15.97:0/2319363296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 msgr2=0x7f0b00078ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 --2- 172.21.15.97:0/2319363296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 0x7f0b00078ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009860 tx=0x7f0aec02a340 comp rx=0 tx=0).stop 2024-07-26T07:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 -- 172.21.15.97:0/2319363296 shutdown_connections 2024-07-26T07:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 --2- 172.21.15.97:0/2319363296 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b00079070 0x7f0b000799e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 --2- 172.21.15.97:0/2319363296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 0x7f0b00078ae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 --2- 172.21.15.97:0/2319363296 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b00077650 0x7f0b00077ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 -- 172.21.15.97:0/2319363296 >> 172.21.15.97:0/2319363296 conn(0x7f0b00100c20 msgr2=0x7f0b001030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 -- 172.21.15.97:0/2319363296 shutdown_connections 2024-07-26T07:56:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.980+0000 7f0b07000640 1 -- 172.21.15.97:0/2319363296 wait complete. 2024-07-26T07:56:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.981+0000 7f0b07000640 1 Processor -- start 2024-07-26T07:56:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.981+0000 7f0b07000640 1 -- start start 2024-07-26T07:56:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.982+0000 7f0b07000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b00077650 0x7f0b0019fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.982+0000 7f0b07000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 0x7f0b001a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.982+0000 7f0b07000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b00079070 0x7f0b001a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.982+0000 7f0b07000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0b00115240 con 0x7f0b00077650 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.982+0000 7f0b07000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001150c0 con 0x7f0b00079070 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.982+0000 7f0b07000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001153c0 con 0x7f0b000786d0 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0affe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b00077650 0x7f0b0019fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0affe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b00077650 0x7f0b0019fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46910/0 (socket says 172.21.15.97:46910) 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0affe00640 1 -- 172.21.15.97:0/1157478514 learned_addr learned my addr 172.21.15.97:0/1157478514 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0aff400640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 0x7f0b001a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0b05000640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b00079070 0x7f0b001a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0affe00640 1 -- 172.21.15.97:0/1157478514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 msgr2=0x7f0b001a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0affe00640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 0x7f0b001a0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0affe00640 1 -- 172.21.15.97:0/1157478514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b00079070 msgr2=0x7f0b001a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0affe00640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b00079070 0x7f0b001a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.983+0000 7f0affe00640 1 -- 172.21.15.97:0/1157478514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b001a1910 con 0x7f0b00077650 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.984+0000 7f0aff400640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 0x7f0b001a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.984+0000 7f0affe00640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b00077650 0x7f0b0019fed0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0af000e9f0 tx=0x7f0af000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.984+0000 7f0afcc00640 1 -- 172.21.15.97:0/1157478514 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af000cd80 con 0x7f0b00077650 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.985+0000 7f0b07000640 1 -- 172.21.15.97:0/1157478514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b001a1c00 con 0x7f0b00077650 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.985+0000 7f0b07000640 1 -- 172.21.15.97:0/1157478514 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b001a82f0 con 0x7f0b00077650 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.985+0000 7f0afcc00640 1 -- 172.21.15.97:0/1157478514 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0af00108c0 con 0x7f0b00077650 2024-07-26T07:56:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.985+0000 7f0afcc00640 1 -- 172.21.15.97:0/1157478514 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af0026060 con 0x7f0b00077650 2024-07-26T07:56:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.989+0000 7f0b0df7c740 1 Processor -- start 2024-07-26T07:56:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.989+0000 7f0b0df7c740 1 -- start start 2024-07-26T07:56:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c32ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c33550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0afcc00640 1 -- 172.21.15.97:0/1157478514 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f0af0018840 con 0x7f0b00077650 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 0x562528c340a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562528ba5270 con 0x5625288642f0 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562528b94770 con 0x56252888d1b0 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562528b987b0 con 0x56252886c4d0 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0affe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c33550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b05000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c32ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0affe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c33550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46922/0 (socket says 172.21.15.97:46922) 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b05000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c32ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:58388/0 (socket says 172.21.15.97:58388) 2024-07-26T07:56:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.990+0000 7f0b05000640 1 -- 172.21.15.97:0/4207919204 learned_addr learned my addr 172.21.15.97:0/4207919204 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.991+0000 7f0aff400640 1 --2- 172.21.15.97:0/4207919204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 0x562528c340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.991+0000 7f0b05000640 1 -- 172.21.15.97:0/4207919204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 msgr2=0x562528c340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.991+0000 7f0b05000640 1 --2- 172.21.15.97:0/4207919204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 0x562528c340a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.991+0000 7f0b05000640 1 -- 172.21.15.97:0/4207919204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 msgr2=0x562528c33550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.991+0000 7f0b05000640 1 --2- 172.21.15.97:0/4207919204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c33550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.991+0000 7f0b05000640 1 -- 172.21.15.97:0/4207919204 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625288f8790 con 0x56252888d1b0 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.992+0000 7f0b05000640 1 --2- 172.21.15.97:0/4207919204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c32ee0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009f60 tx=0x7f0af8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a9d87e2ee152ce server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.992+0000 7f0afcc00640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ae40b3c50 0x7f0ae40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.992+0000 7f0afcc00640 1 -- 172.21.15.97:0/1157478514 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f0af00e3ff0 con 0x7f0b00077650 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.992+0000 7f0affe00640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ae40b3c50 0x7f0ae40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.992+0000 7f0af5600640 1 -- 172.21.15.97:0/4207919204 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af801d070 con 0x56252888d1b0 2024-07-26T07:56:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.993+0000 7f0af5600640 1 -- 172.21.15.97:0/4207919204 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0af800da80 con 0x56252888d1b0 2024-07-26T07:56:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.993+0000 7f0af5600640 1 -- 172.21.15.97:0/4207919204 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8018ba0 con 0x56252888d1b0 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0affe00640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ae40b3c50 0x7f0ae40b6160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0af00aab80 tx=0x7f0af00e7a50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0b0df7c740 1 -- 172.21.15.97:0/4207919204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 msgr2=0x562528c32ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/4207919204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c32ee0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009f60 tx=0x7f0af8009f90 comp rx=0 tx=0).stop 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0b0df7c740 1 -- 172.21.15.97:0/4207919204 shutdown_connections 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/4207919204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 0x562528c340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/4207919204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c33550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/4207919204 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c32ee0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0b0df7c740 1 -- 172.21.15.97:0/4207919204 >> 172.21.15.97:0/4207919204 conn(0x562528881300 msgr2=0x562528c2e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.994+0000 7f0b0df7c740 1 -- 172.21.15.97:0/4207919204 shutdown_connections 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.995+0000 7f0b0df7c740 1 -- 172.21.15.97:0/4207919204 wait complete. 2024-07-26T07:56:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.995+0000 7f0b0df7c740 1 -- start start 2024-07-26T07:56:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.995+0000 7f0b0df7c740 1 Processor -- start 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c4da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 0x562528c4e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c4ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562528aceab0 con 0x5625288642f0 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x562528b31d50 con 0x56252888d1b0 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562528c34bc0 con 0x56252886c4d0 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b0df7c740 10 client.0 _ll_register_callbacks cb 0x5625288a4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b05000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c4da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b05000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c4da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46930/0 (socket says 172.21.15.97:46930) 2024-07-26T07:56:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.996+0000 7f0b05000640 1 -- 172.21.15.97:0/3955286674 learned_addr learned my addr 172.21.15.97:0/3955286674 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.997+0000 7f0aff400640 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 0x562528c4e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.997+0000 7f0b05000640 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c4ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.997+0000 7f0b05000640 1 -- 172.21.15.97:0/3955286674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 msgr2=0x562528c4e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.997+0000 7f0b05000640 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 0x562528c4e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.997+0000 7f0b05000640 1 -- 172.21.15.97:0/3955286674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 msgr2=0x562528c4ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.997+0000 7f0b05000640 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c4ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.997+0000 7f0b05000640 1 -- 172.21.15.97:0/3955286674 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562528971620 con 0x5625288642f0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0b05000640 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c4da60 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009930 tx=0x7f0af8005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609927b04b963434 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0aea000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0aea000640 1 -- 172.21.15.97:0/3955286674 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af801d070 con 0x5625288642f0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0b0df7c740 1 -- 172.21.15.97:0/3955286674 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0af800da80 con 0x5625288642f0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0aea000640 1 -- 172.21.15.97:0/3955286674 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f0af8018ab0 con 0x5625288642f0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0aea000640 1 -- 172.21.15.97:0/3955286674 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0af8018cb0 con 0x5625288642f0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0b0df7c740 10 client.16029 fetch_fsmap learned FSMap version 33 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0b0df7c740 1 -- 172.21.15.97:0/3955286674 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562528c57c80 con 0x5625288642f0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.998+0000 7f0aea000640 1 -- 172.21.15.97:0/3955286674 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0af800da80 con 0x5625288642f0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.999+0000 7f0aea000640 1 -- 172.21.15.97:0/3955286674 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f0af802f020 con 0x5625288642f0 2024-07-26T07:56:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.999+0000 7f0b0df7c740 10 client.16029 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T07:56:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.999+0000 7f0b0df7c740 10 client.16029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.999+0000 7f0b0df7c740 20 client.16029 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.999+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56252884bd80 0x562528c5b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.999+0000 7f0b0df7c740 4 client.16029 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:56:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:27.999+0000 7f0b0df7c740 1 -- 172.21.15.97:0/3955286674 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562528ba71c0 con 0x56252884bd80 2024-07-26T07:56:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.000+0000 7f0aff400640 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56252884bd80 0x562528c5b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.001+0000 7f0aff400640 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56252884bd80 0x562528c5b260 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c015cb98ad87af8 server_cookie=131a570a8027ddc8 in_seq=0 out_seq=0 2024-07-26T07:56:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.001+0000 7f0aea000640 10 client.16029 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0aea000640 1 -- 172.21.15.97:0/3955286674 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562528ba71c0 con 0x56252884bd80 2024-07-26T07:56:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0aea000640 10 client.16029 handle_command_reply: tid=0 2024-07-26T07:56:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 client.16029 shutdown 2024-07-26T07:56:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 -- 172.21.15.97:0/3955286674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 msgr2=0x562528c4da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c4da60 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009930 tx=0x7f0af8005a70 comp rx=0 tx=0).stop 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 -- 172.21.15.97:0/3955286674 shutdown_connections 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56252884bd80 0x562528c5b260 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c4ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252886c4d0 0x562528c4e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/3955286674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5625288642f0 0x562528c4da60 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 -- 172.21.15.97:0/3955286674 >> 172.21.15.97:0/3955286674 conn(0x562528881300 msgr2=0x562528c33f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0aea000640 0 client.16029 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 -- 172.21.15.97:0/3955286674 shutdown_connections 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.003+0000 7f0b0df7c740 1 -- 172.21.15.97:0/3955286674 wait complete. 2024-07-26T07:56:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.004+0000 7f0b0df7c740 20 client.16029 trim_cache size 0 max 16384 2024-07-26T07:56:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.033+0000 7f0b0df7c740 1 Processor -- start 2024-07-26T07:56:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.033+0000 7f0b0df7c740 1 -- start start 2024-07-26T07:56:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5625288642f0 0x562528c37e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c4c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c4cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae000e620 con 0x56252886c4d0 2024-07-26T07:56:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae0010220 con 0x56252888d1b0 2024-07-26T07:56:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae000ecf0 con 0x5625288642f0 2024-07-26T07:56:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0aff400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c4c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0aff400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c4c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46936/0 (socket says 172.21.15.97:46936) 2024-07-26T07:56:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.034+0000 7f0aff400640 1 -- 172.21.15.97:0/1893748689 learned_addr learned my addr 172.21.15.97:0/1893748689 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.035+0000 7f0b05000640 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c4cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.035+0000 7f0b05000640 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5625288642f0 0x562528c37e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.035+0000 7f0aff400640 1 -- 172.21.15.97:0/1893748689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5625288642f0 msgr2=0x562528c37e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.035+0000 7f0aff400640 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5625288642f0 0x562528c37e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.035+0000 7f0aff400640 1 -- 172.21.15.97:0/1893748689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 msgr2=0x562528c4cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.035+0000 7f0aff400640 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c4cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.035+0000 7f0aff400640 1 -- 172.21.15.97:0/1893748689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562528c2a350 con 0x56252886c4d0 2024-07-26T07:56:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.036+0000 7f0aff400640 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c4c800 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0aec02a860 tx=0x7f0aec02a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a989c59801f2a108 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.036+0000 7f0af4c00640 1 -- 172.21.15.97:0/1893748689 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec037040 con 0x56252886c4d0 2024-07-26T07:56:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.036+0000 7f0af4c00640 1 -- 172.21.15.97:0/1893748689 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0aec0044d0 con 0x56252886c4d0 2024-07-26T07:56:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.036+0000 7f0af4c00640 1 -- 172.21.15.97:0/1893748689 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec003e40 con 0x56252886c4d0 2024-07-26T07:56:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.038+0000 7f0b0df7c740 1 -- 172.21.15.97:0/1893748689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 msgr2=0x562528c4c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.038+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c4c800 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0aec02a860 tx=0x7f0aec02a8e0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.038+0000 7f0b0df7c740 1 -- 172.21.15.97:0/1893748689 shutdown_connections 2024-07-26T07:56:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.038+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56252888d1b0 0x562528c4cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.038+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c4c800 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.038+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/1893748689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5625288642f0 0x562528c37e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.038+0000 7f0b0df7c740 1 -- 172.21.15.97:0/1893748689 >> 172.21.15.97:0/1893748689 conn(0x562528881300 msgr2=0x562528c4c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.039+0000 7f0b0df7c740 1 -- 172.21.15.97:0/1893748689 shutdown_connections 2024-07-26T07:56:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.039+0000 7f0b0df7c740 1 -- 172.21.15.97:0/1893748689 wait complete. 2024-07-26T07:56:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.040+0000 7f0b0df7c740 1 -- start start 2024-07-26T07:56:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.040+0000 7f0b0df7c740 1 Processor -- start 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.042+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625288642f0 0x562528c52060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.042+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c52720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.042+0000 7f0b0df7c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252888d1b0 0x562528c53240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.042+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae000fa90 con 0x56252886c4d0 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.042+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae000f3c0 con 0x5625288642f0 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.042+0000 7f0b0df7c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae000c600 con 0x56252888d1b0 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.042+0000 7f0b0df7c740 10 client.0 _ll_register_callbacks cb 0x7f0af801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.043+0000 7f0b05000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252888d1b0 0x562528c53240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.043+0000 7f0aff400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c52720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.043+0000 7f0b05000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625288642f0 0x562528c52060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.043+0000 7f0b05000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625288642f0 0x562528c52060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:58422/0 (socket says 172.21.15.97:58422) 2024-07-26T07:56:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.043+0000 7f0b05000640 1 -- 172.21.15.97:0/2236498593 learned_addr learned my addr 172.21.15.97:0/2236498593 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.044+0000 7f0b05000640 1 -- 172.21.15.97:0/2236498593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625288642f0 msgr2=0x562528c52060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.044+0000 7f0b05000640 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625288642f0 0x562528c52060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.044+0000 7f0b05000640 1 -- 172.21.15.97:0/2236498593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 msgr2=0x562528c52720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.044+0000 7f0b05000640 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c52720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.044+0000 7f0b05000640 1 -- 172.21.15.97:0/2236498593 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562528c539a0 con 0x56252888d1b0 2024-07-26T07:56:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.044+0000 7f0aff400640 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c52720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:28.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.045+0000 7f0b05000640 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252888d1b0 0x562528c53240 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009d40 tx=0x7f0af8018380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e69ce5c2f2f0f20e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:28.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.045+0000 7f0aea000640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-07-26T07:56:28.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.046+0000 7f0aea000640 1 -- 172.21.15.97:0/2236498593 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af801d070 con 0x56252888d1b0 2024-07-26T07:56:28.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.046+0000 7f0aea000640 1 -- 172.21.15.97:0/2236498593 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f0af8015040 con 0x56252888d1b0 2024-07-26T07:56:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.046+0000 7f0b0df7c740 1 -- 172.21.15.97:0/2236498593 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0aec0044d0 con 0x56252888d1b0 2024-07-26T07:56:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.046+0000 7f0aea000640 1 -- 172.21.15.97:0/2236498593 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0af8037070 con 0x56252888d1b0 2024-07-26T07:56:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.046+0000 7f0b0df7c740 10 client.25528 fetch_fsmap learned FSMap version 33 2024-07-26T07:56:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.046+0000 7f0b0df7c740 1 -- 172.21.15.97:0/2236498593 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562528c54c90 con 0x56252888d1b0 2024-07-26T07:56:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.047+0000 7f0aea000640 1 -- 172.21.15.97:0/2236498593 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0aec0044d0 con 0x56252888d1b0 2024-07-26T07:56:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.047+0000 7f0aea000640 1 -- 172.21.15.97:0/2236498593 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f0af803d040 con 0x56252888d1b0 2024-07-26T07:56:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.048+0000 7f0b0df7c740 10 client.25528 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T07:56:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.048+0000 7f0b0df7c740 10 client.25528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.048+0000 7f0b0df7c740 20 client.25528 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.048+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5625288211c0 0x562528c556d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.048+0000 7f0b0df7c740 4 client.25528 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:56:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.048+0000 7f0b0df7c740 1 -- 172.21.15.97:0/2236498593 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562528ba2c60 con 0x5625288211c0 2024-07-26T07:56:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.048+0000 7f0aff400640 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5625288211c0 0x562528c556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.050+0000 7f0aff400640 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5625288211c0 0x562528c556d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcb07150150065a2 server_cookie=cb1b57093374246d in_seq=0 out_seq=0 2024-07-26T07:56:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.050+0000 7f0aea000640 10 client.25528 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.481+0000 7f0aea000640 1 -- 172.21.15.97:0/2236498593 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562528ba2c60 con 0x5625288211c0 2024-07-26T07:56:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.481+0000 7f0aea000640 10 client.25528 handle_command_reply: tid=0 2024-07-26T07:56:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.481+0000 7f0b0df7c740 1 client.25528 shutdown 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0b0df7c740 1 -- 172.21.15.97:0/2236498593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252888d1b0 msgr2=0x562528c53240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252888d1b0 0x562528c53240 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009d40 tx=0x7f0af8018380 comp rx=0 tx=0).stop 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0b0df7c740 1 -- 172.21.15.97:0/2236498593 shutdown_connections 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5625288211c0 0x562528c556d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56252888d1b0 0x562528c53240 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0aea000640 0 client.25528 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56252886c4d0 0x562528c52720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0b0df7c740 1 --2- 172.21.15.97:0/2236498593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5625288642f0 0x562528c52060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.482+0000 7f0b0df7c740 1 -- 172.21.15.97:0/2236498593 >> 172.21.15.97:0/2236498593 conn(0x562528881300 msgr2=0x562528c2f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.483+0000 7f0b0df7c740 1 -- 172.21.15.97:0/2236498593 shutdown_connections 2024-07-26T07:56:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.483+0000 7f0b0df7c740 1 -- 172.21.15.97:0/2236498593 wait complete. 2024-07-26T07:56:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.484+0000 7f0b0df7c740 20 client.25528 trim_cache size 0 max 16384 2024-07-26T07:56:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:56:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:56:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:56:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:56:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 -- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ae40b3c50 msgr2=0x7f0ae40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ae40b3c50 0x7f0ae40b6160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0af00aab80 tx=0x7f0af00e7a50 comp rx=0 tx=0).stop 2024-07-26T07:56:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 -- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b00077650 msgr2=0x7f0b0019fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b00077650 0x7f0b0019fed0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0af000e9f0 tx=0x7f0af000eeb0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 -- 172.21.15.97:0/1157478514 shutdown_connections 2024-07-26T07:56:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f0ae40b3c50 0x7f0ae40b6160 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b00079070 0x7f0b001a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b000786d0 0x7f0b001a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 --2- 172.21.15.97:0/1157478514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b00077650 0x7f0b0019fed0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.485+0000 7f0b07000640 1 -- 172.21.15.97:0/1157478514 >> 172.21.15.97:0/1157478514 conn(0x7f0b00100c20 msgr2=0x7f0b0010f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.486+0000 7f0b07000640 1 -- 172.21.15.97:0/1157478514 shutdown_connections 2024-07-26T07:56:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:28.486+0000 7f0b07000640 1 -- 172.21.15.97:0/1157478514 wait complete. 2024-07-26T07:56:29.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:29 smithi115 ceph-mon[35364]: pgmap v652: 97 pgs: 97 active+clean; 97 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-07-26T07:56:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:29 smithi181 ceph-mon[35869]: pgmap v652: 97 pgs: 97 active+clean; 97 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-07-26T07:56:29.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:29 smithi097 ceph-mon[32231]: pgmap v652: 97 pgs: 97 active+clean; 97 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-07-26T07:56:30.202 INFO:tasks.workunit.client.0.smithi097.stdout: 28 8613 178 5969 147 4173 100 2024-07-26T07:56:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:31 smithi115 ceph-mon[35364]: pgmap v653: 97 pgs: 97 active+clean; 97 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 359 op/s 2024-07-26T07:56:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:31 smithi181 ceph-mon[35869]: pgmap v653: 97 pgs: 97 active+clean; 97 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 359 op/s 2024-07-26T07:56:31.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:31 smithi097 ceph-mon[32231]: pgmap v653: 97 pgs: 97 active+clean; 97 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 359 op/s 2024-07-26T07:56:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:33 smithi115 ceph-mon[35364]: pgmap v654: 97 pgs: 97 active+clean; 94 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 239 op/s 2024-07-26T07:56:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:33 smithi181 ceph-mon[35869]: pgmap v654: 97 pgs: 97 active+clean; 94 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 239 op/s 2024-07-26T07:56:33.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:33 smithi097 ceph-mon[32231]: pgmap v654: 97 pgs: 97 active+clean; 94 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 239 op/s 2024-07-26T07:56:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:35 smithi115 ceph-mon[35364]: pgmap v655: 97 pgs: 97 active+clean; 96 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 248 op/s 2024-07-26T07:56:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:35 smithi181 ceph-mon[35869]: pgmap v655: 97 pgs: 97 active+clean; 96 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 248 op/s 2024-07-26T07:56:35.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:35 smithi097 ceph-mon[32231]: pgmap v655: 97 pgs: 97 active+clean; 96 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 248 op/s 2024-07-26T07:56:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:37 smithi115 ceph-mon[35364]: pgmap v656: 97 pgs: 97 active+clean; 102 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-07-26T07:56:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:37 smithi181 ceph-mon[35869]: pgmap v656: 97 pgs: 97 active+clean; 102 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-07-26T07:56:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:37 smithi097 ceph-mon[32231]: pgmap v656: 97 pgs: 97 active+clean; 102 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-07-26T07:56:39.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:39 smithi115 ceph-mon[35364]: pgmap v657: 97 pgs: 97 active+clean; 102 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-07-26T07:56:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:39 smithi181 ceph-mon[35869]: pgmap v657: 97 pgs: 97 active+clean; 102 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-07-26T07:56:39.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:39 smithi097 ceph-mon[32231]: pgmap v657: 97 pgs: 97 active+clean; 102 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-07-26T07:56:40.202 INFO:tasks.workunit.client.0.smithi097.stdout: 30 8783 65 6577 72 4348 83 2024-07-26T07:56:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:41 smithi115 ceph-mon[35364]: pgmap v658: 97 pgs: 97 active+clean; 103 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-07-26T07:56:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:41 smithi181 ceph-mon[35869]: pgmap v658: 97 pgs: 97 active+clean; 103 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-07-26T07:56:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:41 smithi097 ceph-mon[32231]: pgmap v658: 97 pgs: 97 active+clean; 103 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-07-26T07:56:41.763 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.763+0000 7f4af3400640 1 -- 172.21.15.97:0/2288957976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 msgr2=0x7f4aec105d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.763+0000 7f4af3400640 1 --2- 172.21.15.97:0/2288957976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 0x7f4aec105d10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009900 tx=0x7f4ae002a3f0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.763+0000 7f4af3400640 1 -- 172.21.15.97:0/2288957976 shutdown_connections 2024-07-26T07:56:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.763+0000 7f4af3400640 1 --2- 172.21.15.97:0/2288957976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4aec108e10 0x7f4aec10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.763+0000 7f4af3400640 1 --2- 172.21.15.97:0/2288957976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4aec106250 0x7f4aec1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.763+0000 7f4af3400640 1 --2- 172.21.15.97:0/2288957976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 0x7f4aec105d10 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.763+0000 7f4af3400640 1 -- 172.21.15.97:0/2288957976 >> 172.21.15.97:0/2288957976 conn(0x7f4aec0fd910 msgr2=0x7f4aec0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.764+0000 7f4af3400640 1 -- 172.21.15.97:0/2288957976 shutdown_connections 2024-07-26T07:56:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.764+0000 7f4af3400640 1 -- 172.21.15.97:0/2288957976 wait complete. 2024-07-26T07:56:41.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.765+0000 7f4af3400640 1 Processor -- start 2024-07-26T07:56:41.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.766+0000 7f4af3400640 1 -- start start 2024-07-26T07:56:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.766+0000 7f4af3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 0x7f4aec19bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.766+0000 7f4af3400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4aec106250 0x7f4aec19c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.766+0000 7f4af3400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4aec108e10 0x7f4aec19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.766+0000 7f4af3400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec110e90 con 0x7f4aec06b140 2024-07-26T07:56:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.766+0000 7f4af3400640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec110d10 con 0x7f4aec106250 2024-07-26T07:56:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.766+0000 7f4af3400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec111010 con 0x7f4aec108e10 2024-07-26T07:56:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.767+0000 7f4aebe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4aec106250 0x7f4aec19c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.767+0000 7f4aebe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4aec106250 0x7f4aec19c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:36290/0 (socket says 172.21.15.97:36290) 2024-07-26T07:56:41.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.767+0000 7f4aebe00640 1 -- 172.21.15.97:0/3925103240 learned_addr learned my addr 172.21.15.97:0/3925103240 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:41.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.768+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 0x7f4aec19bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.768+0000 7f4af1400640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4aec108e10 0x7f4aec19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.768+0000 7f4aebe00640 1 -- 172.21.15.97:0/3925103240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4aec108e10 msgr2=0x7f4aec19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.768+0000 7f4aebe00640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4aec108e10 0x7f4aec19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.768+0000 7f4aebe00640 1 -- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 msgr2=0x7f4aec19bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.768+0000 7f4aebe00640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 0x7f4aec19bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.768+0000 7f4aebe00640 1 -- 172.21.15.97:0/3925103240 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec19d580 con 0x7f4aec106250 2024-07-26T07:56:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.768+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 0x7f4aec19bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:56:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.769+0000 7f4aebe00640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4aec106250 0x7f4aec19c1b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ad800b450 tx=0x7f4ad800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.770+0000 7f4ae9600640 1 -- 172.21.15.97:0/3925103240 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8015020 con 0x7f4aec106250 2024-07-26T07:56:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.770+0000 7f4ae9600640 1 -- 172.21.15.97:0/3925103240 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ad8011d20 con 0x7f4aec106250 2024-07-26T07:56:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.770+0000 7f4af3400640 1 -- 172.21.15.97:0/3925103240 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aec19d870 con 0x7f4aec106250 2024-07-26T07:56:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.770+0000 7f4ae9600640 1 -- 172.21.15.97:0/3925103240 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8010e10 con 0x7f4aec106250 2024-07-26T07:56:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.770+0000 7f4af3400640 1 -- 172.21.15.97:0/3925103240 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aec1a3f60 con 0x7f4aec106250 2024-07-26T07:56:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.773+0000 7f4afa375740 1 Processor -- start 2024-07-26T07:56:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.773+0000 7f4afa375740 1 -- start start 2024-07-26T07:56:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.774+0000 7f4afa375740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972ecdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.774+0000 7f4afa375740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ece570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.774+0000 7f4afa375740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b074d0 0x55c972ecf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.774+0000 7f4afa375740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55c972e40270 con 0x55c972b074d0 2024-07-26T07:56:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.774+0000 7f4afa375740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c972e2f770 con 0x55c972aff2f0 2024-07-26T07:56:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.774+0000 7f4afa375740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c972e337b0 con 0x55c972b281b0 2024-07-26T07:56:41.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.774+0000 7f4af1400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972ecdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.775+0000 7f4af1400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972ecdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:46500/0 (socket says 172.21.15.97:46500) 2024-07-26T07:56:41.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.775+0000 7f4af1400640 1 -- 172.21.15.97:0/3430786353 learned_addr learned my addr 172.21.15.97:0/3430786353 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:41.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.775+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ece570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.776+0000 7f4af1400640 1 -- 172.21.15.97:0/3430786353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 msgr2=0x55c972ece570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:41.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.776+0000 7f4af1400640 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ece570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.776+0000 7f4af1400640 1 -- 172.21.15.97:0/3430786353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b074d0 msgr2=0x55c972ecf0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:56:41.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.776+0000 7f4af1400640 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b074d0 0x55c972ecf0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.776+0000 7f4af1400640 1 -- 172.21.15.97:0/3430786353 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c972b93790 con 0x55c972b281b0 2024-07-26T07:56:41.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.776+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ece570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:41.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.777+0000 7f4ae9600640 1 -- 172.21.15.97:0/3925103240 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f4ad8018020 con 0x7f4aec106250 2024-07-26T07:56:41.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.777+0000 7f4af1400640 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972ecdf00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4008cf0 tx=0x7f4ae400d710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9694b10f9d9882fb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:41.781 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.779+0000 7f4ae9600640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4ad40b3c50 0x7f4ad40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.781 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.779+0000 7f4ae9600640 1 -- 172.21.15.97:0/3925103240 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f4ad80e37f0 con 0x7f4aec106250 2024-07-26T07:56:41.781 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.780+0000 7f4addc00640 1 -- 172.21.15.97:0/3430786353 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4004d70 con 0x55c972b281b0 2024-07-26T07:56:41.782 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.780+0000 7f4addc00640 1 -- 172.21.15.97:0/3430786353 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae40054b0 con 0x55c972b281b0 2024-07-26T07:56:41.782 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.780+0000 7f4addc00640 1 -- 172.21.15.97:0/3430786353 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae40178a0 con 0x55c972b281b0 2024-07-26T07:56:41.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.780+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4ad40b3c50 0x7f4ad40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.781+0000 7f4afa375740 1 -- 172.21.15.97:0/3430786353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 msgr2=0x55c972ecdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:41.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.781+0000 7f4afa375740 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972ecdf00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4008cf0 tx=0x7f4ae400d710 comp rx=0 tx=0).stop 2024-07-26T07:56:41.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.781+0000 7f4afa375740 1 -- 172.21.15.97:0/3430786353 shutdown_connections 2024-07-26T07:56:41.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.781+0000 7f4afa375740 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b074d0 0x55c972ecf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.781+0000 7f4afa375740 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ece570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.781+0000 7f4afa375740 1 --2- 172.21.15.97:0/3430786353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972ecdf00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.781+0000 7f4afa375740 1 -- 172.21.15.97:0/3430786353 >> 172.21.15.97:0/3430786353 conn(0x55c972b1c300 msgr2=0x55c972ec9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:41.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.781+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4ad40b3c50 0x7f4ad40b6160 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0006010 tx=0x7f4ae0009640 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:41.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.782+0000 7f4afa375740 1 -- 172.21.15.97:0/3430786353 shutdown_connections 2024-07-26T07:56:41.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.782+0000 7f4afa375740 1 -- 172.21.15.97:0/3430786353 wait complete. 2024-07-26T07:56:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.782+0000 7f4afa375740 1 -- start start 2024-07-26T07:56:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.783+0000 7f4afa375740 1 Processor -- start 2024-07-26T07:56:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.783+0000 7f4afa375740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.783+0000 7f4afa375740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 0x55c972ed81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.783+0000 7f4afa375740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ed8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.783+0000 7f4afa375740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55c972d69ab0 con 0x55c972b281b0 2024-07-26T07:56:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.783+0000 7f4afa375740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55c972dccd50 con 0x55c972aff2f0 2024-07-26T07:56:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.783+0000 7f4afa375740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c972ecfbe0 con 0x55c972b074d0 2024-07-26T07:56:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.784+0000 7f4afa375740 10 client.0 _ll_register_callbacks cb 0x55c972b3fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.784+0000 7f4af0a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.784+0000 7f4af0a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:36312/0 (socket says 172.21.15.97:36312) 2024-07-26T07:56:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.784+0000 7f4af0a00640 1 -- 172.21.15.97:0/3379332106 learned_addr learned my addr 172.21.15.97:0/3379332106 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.784+0000 7f4aebe00640 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 0x55c972ed81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.784+0000 7f4af1400640 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ed8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.785+0000 7f4af0a00640 1 -- 172.21.15.97:0/3379332106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 msgr2=0x55c972ed81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.785+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 0x55c972ed81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.785+0000 7f4af0a00640 1 -- 172.21.15.97:0/3379332106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 msgr2=0x55c972ed8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.785+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ed8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.785+0000 7f4af0a00640 1 -- 172.21.15.97:0/3379332106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c972c0c620 con 0x55c972aff2f0 2024-07-26T07:56:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.785+0000 7f4af1400640 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ed8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:56:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.785+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7af0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0033000 tx=0x7f4ae0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27df46f247d90426 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.786+0000 7f4ad2000640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:56:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.786+0000 7f4ad2000640 1 -- 172.21.15.97:0/3379332106 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0035070 con 0x55c972aff2f0 2024-07-26T07:56:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.786+0000 7f4afa375740 1 -- 172.21.15.97:0/3379332106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ae40054b0 con 0x55c972aff2f0 2024-07-26T07:56:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.786+0000 7f4ad2000640 1 -- 172.21.15.97:0/3379332106 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f4ae00319d0 con 0x55c972aff2f0 2024-07-26T07:56:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.787+0000 7f4ad2000640 1 -- 172.21.15.97:0/3379332106 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4ae0031bd0 con 0x55c972aff2f0 2024-07-26T07:56:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.787+0000 7f4afa375740 10 client.35501 fetch_fsmap learned FSMap version 33 2024-07-26T07:56:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.787+0000 7f4afa375740 1 -- 172.21.15.97:0/3379332106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c972ee1d10 con 0x55c972aff2f0 2024-07-26T07:56:41.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.787+0000 7f4ad2000640 1 -- 172.21.15.97:0/3379332106 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae40054b0 con 0x55c972aff2f0 2024-07-26T07:56:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.788+0000 7f4ad2000640 1 -- 172.21.15.97:0/3379332106 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f4ae0044030 con 0x55c972aff2f0 2024-07-26T07:56:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.789+0000 7f4afa375740 10 client.35501 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T07:56:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.789+0000 7f4afa375740 10 client.35501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.789+0000 7f4afa375740 20 client.35501 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.789+0000 7f4afa375740 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55c972ae6d80 0x55c972ee54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.789+0000 7f4afa375740 4 client.35501 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:56:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.789+0000 7f4afa375740 1 -- 172.21.15.97:0/3379332106 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c972e421c0 con 0x55c972ae6d80 2024-07-26T07:56:41.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.789+0000 7f4aebe00640 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55c972ae6d80 0x55c972ee54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.791+0000 7f4aebe00640 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55c972ae6d80 0x55c972ee54f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c0c836ca2e039f0 server_cookie=dd24277e6cac9964 in_seq=0 out_seq=0 2024-07-26T07:56:41.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.791+0000 7f4ad2000640 10 client.35501 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:41.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.794+0000 7f4ad2000640 1 -- 172.21.15.97:0/3379332106 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c972e421c0 con 0x55c972ae6d80 2024-07-26T07:56:41.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.794+0000 7f4ad2000640 10 client.35501 handle_command_reply: tid=0 2024-07-26T07:56:41.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.794+0000 7f4afa375740 1 client.35501 shutdown 2024-07-26T07:56:41.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.794+0000 7f4afa375740 1 -- 172.21.15.97:0/3379332106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 msgr2=0x55c972ed7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:41.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.794+0000 7f4afa375740 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7af0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0033000 tx=0x7f4ae0009260 comp rx=0 tx=0).stop 2024-07-26T07:56:41.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4afa375740 1 -- 172.21.15.97:0/3379332106 shutdown_connections 2024-07-26T07:56:41.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4afa375740 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55c972ae6d80 0x55c972ee54f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4afa375740 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ed8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4afa375740 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 0x55c972ed81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4afa375740 1 --2- 172.21.15.97:0/3379332106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7af0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4ad2000640 0 client.35501 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:41.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4afa375740 1 -- 172.21.15.97:0/3379332106 >> 172.21.15.97:0/3379332106 conn(0x55c972b1c300 msgr2=0x55c972ec80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:41.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4afa375740 1 -- 172.21.15.97:0/3379332106 shutdown_connections 2024-07-26T07:56:41.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.795+0000 7f4afa375740 1 -- 172.21.15.97:0/3379332106 wait complete. 2024-07-26T07:56:41.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.796+0000 7f4afa375740 20 client.35501 trim_cache size 0 max 16384 2024-07-26T07:56:41.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.821+0000 7f4afa375740 1 Processor -- start 2024-07-26T07:56:41.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.822+0000 7f4afa375740 1 -- start start 2024-07-26T07:56:41.824 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.822+0000 7f4afa375740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.824 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.822+0000 7f4afa375740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 0x55c972ec87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.822+0000 7f4afa375740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ec8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.822+0000 7f4afa375740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec1acdc0 con 0x55c972b281b0 2024-07-26T07:56:41.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.822+0000 7f4afa375740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec1ae9c0 con 0x55c972aff2f0 2024-07-26T07:56:41.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.822+0000 7f4afa375740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec1ad490 con 0x55c972b074d0 2024-07-26T07:56:41.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ec8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ec8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52702/0 (socket says 172.21.15.97:52702) 2024-07-26T07:56:41.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af1400640 1 -- 172.21.15.97:0/3631150707 learned_addr learned my addr 172.21.15.97:0/3631150707 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:41.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4aebe00640 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 0x55c972ec87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af1400640 1 -- 172.21.15.97:0/3631150707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 msgr2=0x55c972ec87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:41.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af1400640 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 0x55c972ec87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af1400640 1 -- 172.21.15.97:0/3631150707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 msgr2=0x55c972ed7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:41.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af1400640 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af1400640 1 -- 172.21.15.97:0/3631150707 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c972ec5370 con 0x55c972b281b0 2024-07-26T07:56:41.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.823+0000 7f4af0a00640 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:56:41.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.824+0000 7f4af1400640 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ec8d20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4016000 tx=0x7f4ae40089c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f68bee64df8dca7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:41.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.827+0000 7f4add200640 1 -- 172.21.15.97:0/3631150707 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae40174d0 con 0x55c972b281b0 2024-07-26T07:56:41.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.828+0000 7f4add200640 1 -- 172.21.15.97:0/3631150707 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae4027070 con 0x55c972b281b0 2024-07-26T07:56:41.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.828+0000 7f4add200640 1 -- 172.21.15.97:0/3631150707 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae40227a0 con 0x55c972b281b0 2024-07-26T07:56:41.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 -- 172.21.15.97:0/3631150707 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 msgr2=0x55c972ec8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:41.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ec8d20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4016000 tx=0x7f4ae40089c0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 -- 172.21.15.97:0/3631150707 shutdown_connections 2024-07-26T07:56:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972b281b0 0x55c972ec8d20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b074d0 0x55c972ec87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 --2- 172.21.15.97:0/3631150707 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972aff2f0 0x55c972ed7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 -- 172.21.15.97:0/3631150707 >> 172.21.15.97:0/3631150707 conn(0x55c972b1c300 msgr2=0x55c972ec80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 -- 172.21.15.97:0/3631150707 shutdown_connections 2024-07-26T07:56:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.829+0000 7f4afa375740 1 -- 172.21.15.97:0/3631150707 wait complete. 2024-07-26T07:56:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.830+0000 7f4afa375740 1 -- start start 2024-07-26T07:56:41.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.830+0000 7f4afa375740 1 Processor -- start 2024-07-26T07:56:41.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4afa375740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972aff2f0 0x55c972edc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4afa375740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972b074d0 0x55c972edce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4afa375740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972edd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4afa375740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec1ae230 con 0x55c972aff2f0 2024-07-26T07:56:41.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4afa375740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec1adb60 con 0x55c972b074d0 2024-07-26T07:56:41.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4afa375740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec1aada0 con 0x55c972b281b0 2024-07-26T07:56:41.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4afa375740 10 client.0 _ll_register_callbacks cb 0x7f4ae0005e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:41.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4af0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972aff2f0 0x55c972edc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4af0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972aff2f0 0x55c972edc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52704/0 (socket says 172.21.15.97:52704) 2024-07-26T07:56:41.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4af0a00640 1 -- 172.21.15.97:0/423770264 learned_addr learned my addr 172.21.15.97:0/423770264 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:41.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4aebe00640 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972b074d0 0x55c972edce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.831+0000 7f4af1400640 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972edd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.832+0000 7f4af0a00640 1 -- 172.21.15.97:0/423770264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 msgr2=0x55c972edd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.832+0000 7f4af0a00640 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972edd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.832+0000 7f4af0a00640 1 -- 172.21.15.97:0/423770264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972b074d0 msgr2=0x55c972edce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.832+0000 7f4af0a00640 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972b074d0 0x55c972edce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.832+0000 7f4af0a00640 1 -- 172.21.15.97:0/423770264 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c972ede0d0 con 0x55c972aff2f0 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.832+0000 7f4af0a00640 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972aff2f0 0x55c972edc740 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4ae002a910 tx=0x7f4ae0004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4755f86b9e5b40 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.832+0000 7f4ad2000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:56:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.833+0000 7f4ad2000640 1 -- 172.21.15.97:0/423770264 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0051740 con 0x55c972aff2f0 2024-07-26T07:56:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.833+0000 7f4ad2000640 1 -- 172.21.15.97:0/423770264 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f4ae003d450 con 0x55c972aff2f0 2024-07-26T07:56:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.833+0000 7f4afa375740 1 -- 172.21.15.97:0/423770264 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ae4027070 con 0x55c972aff2f0 2024-07-26T07:56:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.833+0000 7f4ad2000640 1 -- 172.21.15.97:0/423770264 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4ae0051b40 con 0x55c972aff2f0 2024-07-26T07:56:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.833+0000 7f4afa375740 10 client.16047 fetch_fsmap learned FSMap version 33 2024-07-26T07:56:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.833+0000 7f4afa375740 1 -- 172.21.15.97:0/423770264 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c972edf3f0 con 0x55c972aff2f0 2024-07-26T07:56:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.834+0000 7f4ad2000640 1 -- 172.21.15.97:0/423770264 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ae4027070 con 0x55c972aff2f0 2024-07-26T07:56:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.834+0000 7f4ad2000640 1 -- 172.21.15.97:0/423770264 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f4ae0044040 con 0x55c972aff2f0 2024-07-26T07:56:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.834+0000 7f4afa375740 10 client.16047 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T07:56:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.834+0000 7f4afa375740 10 client.16047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.834+0000 7f4afa375740 20 client.16047 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.834+0000 7f4afa375740 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55c972abc1c0 0x55c972f584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.834+0000 7f4afa375740 4 client.16047 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:56:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.834+0000 7f4afa375740 1 -- 172.21.15.97:0/423770264 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c972e3dc60 con 0x55c972abc1c0 2024-07-26T07:56:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.835+0000 7f4aebe00640 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55c972abc1c0 0x55c972f584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:41.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.836+0000 7f4aebe00640 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55c972abc1c0 0x55c972f584a0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1f5bab58f9692e1 server_cookie=9897ab22386c878a in_seq=0 out_seq=0 2024-07-26T07:56:41.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:41.836+0000 7f4ad2000640 10 client.16047 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4ad2000640 1 -- 172.21.15.97:0/423770264 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c972e3dc60 con 0x55c972abc1c0 2024-07-26T07:56:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4ad2000640 10 client.16047 handle_command_reply: tid=0 2024-07-26T07:56:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 client.16047 shutdown 2024-07-26T07:56:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 -- 172.21.15.97:0/423770264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972aff2f0 msgr2=0x55c972edc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972aff2f0 0x55c972edc740 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4ae002a910 tx=0x7f4ae0004590 comp rx=0 tx=0).stop 2024-07-26T07:56:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 -- 172.21.15.97:0/423770264 shutdown_connections 2024-07-26T07:56:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55c972abc1c0 0x55c972f584a0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c972b281b0 0x55c972edd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55c972b074d0 0x55c972edce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 --2- 172.21.15.97:0/423770264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c972aff2f0 0x55c972edc740 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4afa375740 1 -- 172.21.15.97:0/423770264 >> 172.21.15.97:0/423770264 conn(0x55c972b1c300 msgr2=0x55c972ec8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.403+0000 7f4ad2000640 0 client.16047 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.404+0000 7f4afa375740 1 -- 172.21.15.97:0/423770264 shutdown_connections 2024-07-26T07:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.404+0000 7f4afa375740 1 -- 172.21.15.97:0/423770264 wait complete. 2024-07-26T07:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.404+0000 7f4afa375740 20 client.16047 trim_cache size 0 max 16384 2024-07-26T07:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:56:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 -- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4ad40b3c50 msgr2=0x7f4ad40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4ad40b3c50 0x7f4ad40b6160 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0006010 tx=0x7f4ae0009640 comp rx=0 tx=0).stop 2024-07-26T07:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 -- 172.21.15.97:0/3925103240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4aec106250 msgr2=0x7f4aec19c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4aec106250 0x7f4aec19c1b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ad800b450 tx=0x7f4ad800b910 comp rx=0 tx=0).stop 2024-07-26T07:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 -- 172.21.15.97:0/3925103240 shutdown_connections 2024-07-26T07:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f4ad40b3c50 0x7f4ad40b6160 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4aec108e10 0x7f4aec19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4aec106250 0x7f4aec19c1b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 --2- 172.21.15.97:0/3925103240 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4aec06b140 0x7f4aec19bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.405+0000 7f4af3400640 1 -- 172.21.15.97:0/3925103240 >> 172.21.15.97:0/3925103240 conn(0x7f4aec0fd910 msgr2=0x7f4aec104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.406+0000 7f4af3400640 1 -- 172.21.15.97:0/3925103240 shutdown_connections 2024-07-26T07:56:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:42.406+0000 7f4af3400640 1 -- 172.21.15.97:0/3925103240 wait complete. 2024-07-26T07:56:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:43 smithi115 ceph-mon[35364]: pgmap v659: 97 pgs: 97 active+clean; 105 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-07-26T07:56:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:43 smithi181 ceph-mon[35869]: pgmap v659: 97 pgs: 97 active+clean; 105 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-07-26T07:56:43.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:43 smithi097 ceph-mon[32231]: pgmap v659: 97 pgs: 97 active+clean; 105 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-07-26T07:56:45.459 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-07-26T07:56:45.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:56:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:45 smithi115 ceph-mon[35364]: pgmap v660: 97 pgs: 97 active+clean; 110 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-07-26T07:56:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:45 smithi181 ceph-mon[35869]: pgmap v660: 97 pgs: 97 active+clean; 110 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-07-26T07:56:45.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:45 smithi097 ceph-mon[32231]: pgmap v660: 97 pgs: 97 active+clean; 110 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-07-26T07:56:46.044 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:56:47.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:47 smithi115 ceph-mon[35364]: pgmap v661: 97 pgs: 97 active+clean; 102 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-07-26T07:56:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:47 smithi181 ceph-mon[35869]: pgmap v661: 97 pgs: 97 active+clean; 102 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-07-26T07:56:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:47 smithi097 ceph-mon[32231]: pgmap v661: 97 pgs: 97 active+clean; 102 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.345+0000 7fe4b3e00640 1 -- 172.21.15.97:0/2489201212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b4109670 msgr2=0x7fe4b410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.345+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/2489201212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b4109670 0x7fe4b410fea0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009ca0 tx=0x7fe4ac02a8c0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.345+0000 7fe4b3e00640 1 -- 172.21.15.97:0/2489201212 shutdown_connections 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.345+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/2489201212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b4109670 0x7fe4b410fea0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.345+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/2489201212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4b4108d20 0x7fe4b4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.345+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/2489201212 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b41033e0 0x7fe4b41037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.345+0000 7fe4b3e00640 1 -- 172.21.15.97:0/2489201212 >> 172.21.15.97:0/2489201212 conn(0x7fe4b40fee70 msgr2=0x7fe4b41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.346+0000 7fe4b3e00640 1 -- 172.21.15.97:0/2489201212 shutdown_connections 2024-07-26T07:56:49.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.346+0000 7fe4b3e00640 1 -- 172.21.15.97:0/2489201212 wait complete. 2024-07-26T07:56:49.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.348+0000 7fe4b3e00640 1 Processor -- start 2024-07-26T07:56:49.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.348+0000 7fe4b3e00640 1 -- start start 2024-07-26T07:56:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b41033e0 0x7fe4b410db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4b4108d20 0x7fe4b4109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b4109670 0x7fe4b410a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b4113220 con 0x7fe4b41033e0 2024-07-26T07:56:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b41130a0 con 0x7fe4b4109670 2024-07-26T07:56:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b41133a0 con 0x7fe4b4108d20 2024-07-26T07:56:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b41033e0 0x7fe4b410db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b41033e0 0x7fe4b410db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47696/0 (socket says 172.21.15.97:47696) 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b2a00640 1 -- 172.21.15.97:0/3724302184 learned_addr learned my addr 172.21.15.97:0/3724302184 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.349+0000 7fe4b2000640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4b4108d20 0x7fe4b4109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.350+0000 7fe4b2a00640 1 -- 172.21.15.97:0/3724302184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4b4108d20 msgr2=0x7fe4b4109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.350+0000 7fe4b2a00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4b4108d20 0x7fe4b4109e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.350+0000 7fe4b2a00640 1 -- 172.21.15.97:0/3724302184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b4109670 msgr2=0x7fe4b410a590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.350+0000 7fe4b2a00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b4109670 0x7fe4b410a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.350+0000 7fe4b2a00640 1 -- 172.21.15.97:0/3724302184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b419bb00 con 0x7fe4b41033e0 2024-07-26T07:56:49.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.351+0000 7fe4b2a00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b41033e0 0x7fe4b410db90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000d950 tx=0x7fe4a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:49.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.351+0000 7fe4ab400640 1 -- 172.21.15.97:0/3724302184 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0017070 con 0x7fe4b41033e0 2024-07-26T07:56:49.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.352+0000 7fe4ab400640 1 -- 172.21.15.97:0/3724302184 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4a0009c30 con 0x7fe4b41033e0 2024-07-26T07:56:49.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.352+0000 7fe4ab400640 1 -- 172.21.15.97:0/3724302184 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000f620 con 0x7fe4b41033e0 2024-07-26T07:56:49.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.352+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b419bdf0 con 0x7fe4b41033e0 2024-07-26T07:56:49.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.352+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b419c2e0 con 0x7fe4b41033e0 2024-07-26T07:56:49.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.354+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4940052c0 con 0x7fe4b41033e0 2024-07-26T07:56:49.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.357+0000 7fe4ab400640 1 -- 172.21.15.97:0/3724302184 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fe4a0010460 con 0x7fe4b41033e0 2024-07-26T07:56:49.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.360+0000 7fe4ab400640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4980b3cd0 0x7fe4980b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:49.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.360+0000 7fe4ab400640 1 -- 172.21.15.97:0/3724302184 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fe4a00e36f0 con 0x7fe4b41033e0 2024-07-26T07:56:49.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.361+0000 7fe4b2000640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4980b3cd0 0x7fe4980b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:49.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.362+0000 7fe4b2000640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4980b3cd0 0x7fe4980b61e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe4b41056b0 tx=0x7fe4a400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:49.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.365+0000 7fe4ab400640 1 -- 172.21.15.97:0/3724302184 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe4a00aac80 con 0x7fe4b41033e0 2024-07-26T07:56:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:49 smithi115 ceph-mon[35364]: pgmap v662: 97 pgs: 97 active+clean; 100 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-07-26T07:56:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:49 smithi181 ceph-mon[35869]: pgmap v662: 97 pgs: 97 active+clean; 100 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-07-26T07:56:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:49 smithi097 ceph-mon[32231]: pgmap v662: 97 pgs: 97 active+clean; 100 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-07-26T07:56:49.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.755+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4940055b0 con 0x7fe4b41033e0 2024-07-26T07:56:49.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.756+0000 7fe4ab400640 1 -- 172.21.15.97:0/3724302184 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11998 (secure 0 0 0) 0x7fe4a0009de0 con 0x7fe4b41033e0 2024-07-26T07:56:49.758 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:56:49.758 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 33 2024-07-26T07:56:49.759 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":33,"btime":"2024-07-26T07:56:15:216944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/4227989530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4227989530},{"type":"v1","addr":"172.21.15.97:6837","nonce":4227989530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:56:15.216901+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3957930666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3957930666},{"type":"v1","addr":"172.21.15.181:6837","nonce":3957930666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:56:49.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4980b3cd0 msgr2=0x7fe4980b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:49.761 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4980b3cd0 0x7fe4980b61e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe4b41056b0 tx=0x7fe4a400b410 comp rx=0 tx=0).stop 2024-07-26T07:56:49.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b41033e0 msgr2=0x7fe4b410db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:49.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b41033e0 0x7fe4b410db90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000d950 tx=0x7fe4a000de10 comp rx=0 tx=0).stop 2024-07-26T07:56:49.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 shutdown_connections 2024-07-26T07:56:49.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe4980b3cd0 0x7fe4980b61e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b4109670 0x7fe4b410a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4b4108d20 0x7fe4b4109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 --2- 172.21.15.97:0/3724302184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4b41033e0 0x7fe4b410db90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:49.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.759+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 >> 172.21.15.97:0/3724302184 conn(0x7fe4b40fee70 msgr2=0x7fe4b410e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:49.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.761+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 shutdown_connections 2024-07-26T07:56:49.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:49.761+0000 7fe4b3e00640 1 -- 172.21.15.97:0/3724302184 wait complete. 2024-07-26T07:56:50.202 INFO:tasks.workunit.client.0.smithi097.stdout: 31 8209 105 5868 35 3900 143 2024-07-26T07:56:50.634 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-07-26T07:56:50.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-07-26T07:56:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:50 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3724302184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:56:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:50 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3724302184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:56:50.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:50 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3724302184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:56:51.211 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:56:51.705 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:51 smithi097 ceph-mon[32231]: pgmap v663: 97 pgs: 97 active+clean; 102 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-07-26T07:56:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:51 smithi181 ceph-mon[35869]: pgmap v663: 97 pgs: 97 active+clean; 102 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-07-26T07:56:51.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:51 smithi115 ceph-mon[35364]: pgmap v663: 97 pgs: 97 active+clean; 102 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-07-26T07:56:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:53 smithi181 ceph-mon[35869]: pgmap v664: 97 pgs: 97 active+clean; 103 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-07-26T07:56:53.736 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:53 smithi097 ceph-mon[32231]: pgmap v664: 97 pgs: 97 active+clean; 103 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-07-26T07:56:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:53 smithi115 ceph-mon[35364]: pgmap v664: 97 pgs: 97 active+clean; 103 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-07-26T07:56:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.677+0000 7fdfca600640 1 -- 172.21.15.97:0/818897570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc41027f0 msgr2=0x7fdfc410f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.677+0000 7fdfca600640 1 --2- 172.21.15.97:0/818897570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc41027f0 0x7fdfc410f510 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0009900 tx=0x7fdfb002a480 comp rx=0 tx=0).stop 2024-07-26T07:56:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.678+0000 7fdfca600640 1 -- 172.21.15.97:0/818897570 shutdown_connections 2024-07-26T07:56:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.678+0000 7fdfca600640 1 --2- 172.21.15.97:0/818897570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfc410fa50 0x7fdfc4111e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.678+0000 7fdfca600640 1 --2- 172.21.15.97:0/818897570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc41027f0 0x7fdfc410f510 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.678+0000 7fdfca600640 1 --2- 172.21.15.97:0/818897570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc4101ea0 0x7fdfc41022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.678+0000 7fdfca600640 1 -- 172.21.15.97:0/818897570 >> 172.21.15.97:0/818897570 conn(0x7fdfc40fdd60 msgr2=0x7fdfc41001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.678+0000 7fdfca600640 1 -- 172.21.15.97:0/818897570 shutdown_connections 2024-07-26T07:56:54.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.678+0000 7fdfca600640 1 -- 172.21.15.97:0/818897570 wait complete. 2024-07-26T07:56:54.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.679+0000 7fdfca600640 1 Processor -- start 2024-07-26T07:56:54.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.680+0000 7fdfca600640 1 -- start start 2024-07-26T07:56:54.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.680+0000 7fdfca600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc4101ea0 0x7fdfc4107ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.680+0000 7fdfca600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41027f0 0x7fdfc41041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.680+0000 7fdfca600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfc410fa50 0x7fdfc41048f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.680+0000 7fdfca600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41153a0 con 0x7fdfc4101ea0 2024-07-26T07:56:54.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.680+0000 7fdfca600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc4115220 con 0x7fdfc410fa50 2024-07-26T07:56:54.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.680+0000 7fdfca600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc4115520 con 0x7fdfc41027f0 2024-07-26T07:56:54.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.681+0000 7fdfc9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc4101ea0 0x7fdfc4107ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.681+0000 7fdfc9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc4101ea0 0x7fdfc4107ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47712/0 (socket says 172.21.15.97:47712) 2024-07-26T07:56:54.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.681+0000 7fdfc9200640 1 -- 172.21.15.97:0/3679432198 learned_addr learned my addr 172.21.15.97:0/3679432198 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.681+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41027f0 0x7fdfc41041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.681+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfc410fa50 0x7fdfc41048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.681+0000 7fdfc9200640 1 -- 172.21.15.97:0/3679432198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41027f0 msgr2=0x7fdfc41041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.682+0000 7fdfc9200640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41027f0 0x7fdfc41041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.682+0000 7fdfc9200640 1 -- 172.21.15.97:0/3679432198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfc410fa50 msgr2=0x7fdfc41048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.682+0000 7fdfc9200640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfc410fa50 0x7fdfc41048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.682+0000 7fdfc9200640 1 -- 172.21.15.97:0/3679432198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc41ac360 con 0x7fdfc4101ea0 2024-07-26T07:56:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.682+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfc410fa50 0x7fdfc41048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:56:54.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.682+0000 7fdfc9200640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc4101ea0 0x7fdfc4107ef0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdfb800c920 tx=0x7fdfb800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:54.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.682+0000 7fdfc1600640 1 -- 172.21.15.97:0/3679432198 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8007c70 con 0x7fdfc4101ea0 2024-07-26T07:56:54.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.683+0000 7fdfc1600640 1 -- 172.21.15.97:0/3679432198 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfb8007e10 con 0x7fdfc4101ea0 2024-07-26T07:56:54.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.683+0000 7fdfc1600640 1 -- 172.21.15.97:0/3679432198 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8018710 con 0x7fdfc4101ea0 2024-07-26T07:56:54.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.683+0000 7fdfca600640 1 -- 172.21.15.97:0/3679432198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc41ac530 con 0x7fdfc4101ea0 2024-07-26T07:56:54.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.684+0000 7fdfca600640 1 -- 172.21.15.97:0/3679432198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc41ac9f0 con 0x7fdfc4101ea0 2024-07-26T07:56:54.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.688+0000 7fdfd27de740 1 Processor -- start 2024-07-26T07:56:54.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.688+0000 7fdfd27de740 1 -- start start 2024-07-26T07:56:54.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d130e9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130e98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130e9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfd27de740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558d130b3cf0 con 0x558d12d43840 2024-07-26T07:56:54.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfd27de740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558d1305eb40 con 0x558d12d22b60 2024-07-26T07:56:54.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfd27de740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558d1304e040 con 0x558d12d1a980 2024-07-26T07:56:54.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfc9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d130e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfc9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d130e9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47726/0 (socket says 172.21.15.97:47726) 2024-07-26T07:56:54.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfc9c00640 1 -- 172.21.15.97:0/3624123826 learned_addr learned my addr 172.21.15.97:0/3624123826 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:54.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.689+0000 7fdfc1600640 1 -- 172.21.15.97:0/3679432198 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fdfb800f450 con 0x7fdfc4101ea0 2024-07-26T07:56:54.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.690+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130e9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.690+0000 7fdfc9200640 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130e98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.691+0000 7fdfc3e00640 1 -- 172.21.15.97:0/3624123826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 msgr2=0x558d130e98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:54.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.691+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130e98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.691+0000 7fdfc3e00640 1 -- 172.21.15.97:0/3624123826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 msgr2=0x558d130e9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:54.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.691+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d130e9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.691+0000 7fdfc3e00640 1 -- 172.21.15.97:0/3624123826 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d12fd5af0 con 0x558d12d22b60 2024-07-26T07:56:54.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.691+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d130e9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:54.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.692+0000 7fdfc1600640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfac0b3d10 0x7fdfac0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.692+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130e9e10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0009850 tx=0x7fdfb00042d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc841ebaba0ec86 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:54.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.692+0000 7fdfc1600640 1 -- 172.21.15.97:0/3679432198 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fdfb80e3eb0 con 0x7fdfc4101ea0 2024-07-26T07:56:54.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.693+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfac0b3d10 0x7fdfac0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.693+0000 7fdfb6000640 1 -- 172.21.15.97:0/3624123826 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0035030 con 0x558d12d22b60 2024-07-26T07:56:54.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.693+0000 7fdfb6000640 1 -- 172.21.15.97:0/3624123826 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfb00054d0 con 0x558d12d22b60 2024-07-26T07:56:54.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.693+0000 7fdfb6000640 1 -- 172.21.15.97:0/3624123826 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb00035f0 con 0x558d12d22b60 2024-07-26T07:56:54.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.694+0000 7fdfd27de740 1 -- 172.21.15.97:0/3624123826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 msgr2=0x558d130e9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:54.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.694+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130e9e10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0009850 tx=0x7fdfb00042d0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.694+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfac0b3d10 0x7fdfac0b6220 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009fd0 tx=0x7fdfbc009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.694+0000 7fdfd27de740 1 -- 172.21.15.97:0/3624123826 shutdown_connections 2024-07-26T07:56:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.694+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130e9e10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.694+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130e98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.694+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3624123826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d130e9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.694+0000 7fdfd27de740 1 -- 172.21.15.97:0/3624123826 >> 172.21.15.97:0/3624123826 conn(0x558d12d37990 msgr2=0x558d130e7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.695+0000 7fdfd27de740 1 -- 172.21.15.97:0/3624123826 shutdown_connections 2024-07-26T07:56:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.695+0000 7fdfd27de740 1 -- 172.21.15.97:0/3624123826 wait complete. 2024-07-26T07:56:54.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.695+0000 7fdfd27de740 1 -- start start 2024-07-26T07:56:54.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.695+0000 7fdfd27de740 1 Processor -- start 2024-07-26T07:56:54.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.696+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.696+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130f5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.696+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d13100cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.696+0000 7fdfd27de740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558d12fea500 con 0x558d12d43840 2024-07-26T07:56:54.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.696+0000 7fdfd27de740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x558d13052080 con 0x558d12d22b60 2024-07-26T07:56:54.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.696+0000 7fdfd27de740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558d12eb17a0 con 0x558d12d1a980 2024-07-26T07:56:54.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.696+0000 7fdfd27de740 10 client.0 _ll_register_callbacks cb 0x558d13046bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:54.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.697+0000 7fdfc9200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.697+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.697+0000 7fdfc9c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d13100cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.697+0000 7fdfc9200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130f4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:40324/0 (socket says 172.21.15.97:40324) 2024-07-26T07:56:54.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.697+0000 7fdfc9200640 1 -- 172.21.15.97:0/3572066998 learned_addr learned my addr 172.21.15.97:0/3572066998 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:54.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.698+0000 7fdfc9200640 1 -- 172.21.15.97:0/3572066998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 msgr2=0x558d130f5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:54.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.698+0000 7fdfc9200640 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130f5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.698+0000 7fdfc9200640 1 -- 172.21.15.97:0/3572066998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 msgr2=0x558d13100cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:54.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.698+0000 7fdfc9200640 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d13100cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.698+0000 7fdfc9200640 1 -- 172.21.15.97:0/3572066998 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d12dbdb90 con 0x558d12d1a980 2024-07-26T07:56:54.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.698+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d13100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:56:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.698+0000 7fdfc9200640 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130f4c60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80a6dc0 tx=0x7fdfb800fca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c2b8c85ed819c78 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.698+0000 7fdfaaa00640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-07-26T07:56:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.699+0000 7fdfaaa00640 1 -- 172.21.15.97:0/3572066998 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb80aa850 con 0x558d12d1a980 2024-07-26T07:56:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.699+0000 7fdfaaa00640 1 -- 172.21.15.97:0/3572066998 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fdfb80aafa0 con 0x558d12d1a980 2024-07-26T07:56:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.699+0000 7fdfd27de740 1 -- 172.21.15.97:0/3572066998 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfb00054d0 con 0x558d12d1a980 2024-07-26T07:56:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.700+0000 7fdfaaa00640 1 -- 172.21.15.97:0/3572066998 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdfb80ab440 con 0x558d12d1a980 2024-07-26T07:56:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.700+0000 7fdfd27de740 10 client.25555 fetch_fsmap learned FSMap version 33 2024-07-26T07:56:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.700+0000 7fdfd27de740 1 -- 172.21.15.97:0/3572066998 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558d131038e0 con 0x558d12d1a980 2024-07-26T07:56:54.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.701+0000 7fdfaaa00640 1 -- 172.21.15.97:0/3572066998 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfb00054d0 con 0x558d12d1a980 2024-07-26T07:56:54.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.701+0000 7fdfaaa00640 1 -- 172.21.15.97:0/3572066998 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7fdfb80b3960 con 0x558d12d1a980 2024-07-26T07:56:54.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.701+0000 7fdfd27de740 10 client.25555 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T07:56:54.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.701+0000 7fdfd27de740 10 client.25555 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:56:54.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.701+0000 7fdfd27de740 20 client.25555 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:54.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.702+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558d12d021c0 0x558d13106f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.702+0000 7fdfd27de740 4 client.25555 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:56:54.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.702+0000 7fdfd27de740 1 -- 172.21.15.97:0/3572066998 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d13060a90 con 0x558d12d021c0 2024-07-26T07:56:54.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.702+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558d12d021c0 0x558d13106f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.703+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558d12d021c0 0x558d13106f50 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13066127c38fe2bb server_cookie=b045277e5738ba4 in_seq=0 out_seq=0 2024-07-26T07:56:54.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.703+0000 7fdfaaa00640 10 client.25555 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:54.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.704+0000 7fdfaaa00640 1 -- 172.21.15.97:0/3572066998 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558d13060a90 con 0x558d12d021c0 2024-07-26T07:56:54.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.704+0000 7fdfaaa00640 10 client.25555 handle_command_reply: tid=0 2024-07-26T07:56:54.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.704+0000 7fdfd27de740 1 client.25555 shutdown 2024-07-26T07:56:54.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfd27de740 1 -- 172.21.15.97:0/3572066998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 msgr2=0x558d130f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:54.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130f4c60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80a6dc0 tx=0x7fdfb800fca0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfd27de740 1 -- 172.21.15.97:0/3572066998 shutdown_connections 2024-07-26T07:56:54.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558d12d021c0 0x558d13106f50 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d43840 0x558d13100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d22b60 0x558d130f5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfd27de740 1 --2- 172.21.15.97:0/3572066998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d1a980 0x558d130f4c60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfd27de740 1 -- 172.21.15.97:0/3572066998 >> 172.21.15.97:0/3572066998 conn(0x558d12d37990 msgr2=0x558d130ec570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:54.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.705+0000 7fdfaaa00640 0 client.25555 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:54.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.706+0000 7fdfd27de740 1 -- 172.21.15.97:0/3572066998 shutdown_connections 2024-07-26T07:56:54.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.706+0000 7fdfd27de740 1 -- 172.21.15.97:0/3572066998 wait complete. 2024-07-26T07:56:54.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.707+0000 7fdfd27de740 20 client.25555 trim_cache size 0 max 16384 2024-07-26T07:56:54.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.737+0000 7fdfd27de740 1 Processor -- start 2024-07-26T07:56:54.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.737+0000 7fdfd27de740 1 -- start start 2024-07-26T07:56:54.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.737+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.737+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 0x558d130e5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.737+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d130f6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.737+0000 7fdfd27de740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b5120 con 0x558d12d1a980 2024-07-26T07:56:54.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.737+0000 7fdfd27de740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b6d20 con 0x558d12d43840 2024-07-26T07:56:54.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.737+0000 7fdfd27de740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b57f0 con 0x558d12d22b60 2024-07-26T07:56:54.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.738+0000 7fdfc9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.738+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 0x558d130e5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.738+0000 7fdfc9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130ef4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47750/0 (socket says 172.21.15.97:47750) 2024-07-26T07:56:54.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.738+0000 7fdfc9200640 1 -- 172.21.15.97:0/1940495915 learned_addr learned my addr 172.21.15.97:0/1940495915 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:54.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.738+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/1940495915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d130f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.739+0000 7fdfc9200640 1 -- 172.21.15.97:0/1940495915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 msgr2=0x558d130e5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:54.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.739+0000 7fdfc9200640 1 --2- 172.21.15.97:0/1940495915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 0x558d130e5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.739+0000 7fdfc9200640 1 -- 172.21.15.97:0/1940495915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 msgr2=0x558d130f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:54.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.739+0000 7fdfc9200640 1 --2- 172.21.15.97:0/1940495915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d130f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.739+0000 7fdfc9200640 1 -- 172.21.15.97:0/1940495915 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d130f5f20 con 0x558d12d1a980 2024-07-26T07:56:54.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.739+0000 7fdfc9200640 1 --2- 172.21.15.97:0/1940495915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130ef4b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80a7fb0 tx=0x7fdfb8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1dc61c9aff3ead server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:54.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.740+0000 7fdfb5600640 1 -- 172.21.15.97:0/1940495915 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb80c9070 con 0x558d12d1a980 2024-07-26T07:56:54.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.740+0000 7fdfb5600640 1 -- 172.21.15.97:0/1940495915 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfb80aa850 con 0x558d12d1a980 2024-07-26T07:56:54.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.740+0000 7fdfb5600640 1 -- 172.21.15.97:0/1940495915 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb80b3df0 con 0x558d12d1a980 2024-07-26T07:56:54.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.741+0000 7fdfd27de740 1 -- 172.21.15.97:0/1940495915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 msgr2=0x558d130ef4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:54.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.741+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1940495915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130ef4b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80a7fb0 tx=0x7fdfb8000bc0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.741+0000 7fdfd27de740 1 -- 172.21.15.97:0/1940495915 shutdown_connections 2024-07-26T07:56:54.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.741+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1940495915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d130f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.741+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1940495915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 0x558d130e5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.741+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1940495915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130ef4b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.741+0000 7fdfd27de740 1 -- 172.21.15.97:0/1940495915 >> 172.21.15.97:0/1940495915 conn(0x558d12d37990 msgr2=0x558d130ead80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.742+0000 7fdfd27de740 1 -- 172.21.15.97:0/1940495915 shutdown_connections 2024-07-26T07:56:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.742+0000 7fdfd27de740 1 -- 172.21.15.97:0/1940495915 wait complete. 2024-07-26T07:56:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.742+0000 7fdfd27de740 1 -- start start 2024-07-26T07:56:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.742+0000 7fdfd27de740 1 Processor -- start 2024-07-26T07:56:54.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130f8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 0x558d130f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfd27de740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d131677c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfd27de740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b6590 con 0x558d12d1a980 2024-07-26T07:56:54.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfd27de740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b5ec0 con 0x558d12d43840 2024-07-26T07:56:54.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfd27de740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b3100 con 0x558d12d22b60 2024-07-26T07:56:54.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfd27de740 10 client.0 _ll_register_callbacks cb 0x7fdfb80a6c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:54.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfc9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfc9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130f8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47754/0 (socket says 172.21.15.97:47754) 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.743+0000 7fdfc9200640 1 -- 172.21.15.97:0/1107387716 learned_addr learned my addr 172.21.15.97:0/1107387716 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d131677c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 0x558d130f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc9200640 1 -- 172.21.15.97:0/1107387716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 msgr2=0x558d130f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc9200640 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 0x558d130f8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc9200640 1 -- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 msgr2=0x558d131677c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc9200640 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d131677c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc9200640 1 -- 172.21.15.97:0/1107387716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d13167e30 con 0x558d12d1a980 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc9c00640 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d131677c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.744+0000 7fdfc9200640 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130f8570 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80a6dc0 tx=0x7fdfb80af270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a31633bd835fd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:54.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.745+0000 7fdfaaa00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:56:54.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.745+0000 7fdfaaa00640 1 -- 172.21.15.97:0/1107387716 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb80c9070 con 0x558d12d1a980 2024-07-26T07:56:54.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.745+0000 7fdfaaa00640 1 -- 172.21.15.97:0/1107387716 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fdfb80a67b0 con 0x558d12d1a980 2024-07-26T07:56:54.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.745+0000 7fdfd27de740 1 -- 172.21.15.97:0/1107387716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfb80aa850 con 0x558d12d1a980 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.746+0000 7fdfaaa00640 1 -- 172.21.15.97:0/1107387716 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdfb80b3df0 con 0x558d12d1a980 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.746+0000 7fdfd27de740 10 client.16071 fetch_fsmap learned FSMap version 33 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.746+0000 7fdfd27de740 1 -- 172.21.15.97:0/1107387716 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558d13167e30 con 0x558d12d1a980 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.746+0000 7fdfaaa00640 1 -- 172.21.15.97:0/1107387716 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfb80aa850 con 0x558d12d1a980 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.747+0000 7fdfaaa00640 1 -- 172.21.15.97:0/1107387716 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7fdfb80a9a10 con 0x558d12d1a980 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.747+0000 7fdfd27de740 10 client.16071 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.747+0000 7fdfd27de740 10 client.16071 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.747+0000 7fdfd27de740 20 client.16071 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:54.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.747+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558d12cd7570 0x558d131825f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:54.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.747+0000 7fdfd27de740 4 client.16071 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-07-26T07:56:54.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.747+0000 7fdfd27de740 1 -- 172.21.15.97:0/1107387716 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x558d1305c530 con 0x558d12cd7570 2024-07-26T07:56:54.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.748+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558d12cd7570 0x558d131825f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:54.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.749+0000 7fdfaaa00640 10 client.16071 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:54.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:54.749+0000 7fdfc3e00640 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558d12cd7570 0x558d131825f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84d5ceb1ce94e79b server_cookie=6a896b0a1d80c46a in_seq=0 out_seq=0 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.731+0000 7f590ac00640 1 -- 172.21.15.97:0/3604424426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f590406b140 msgr2=0x7f5904105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.731+0000 7f590ac00640 1 --2- 172.21.15.97:0/3604424426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f590406b140 0x7f5904105bb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009900 tx=0x7f58f802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.731+0000 7f590ac00640 1 -- 172.21.15.97:0/3604424426 shutdown_connections 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.731+0000 7f590ac00640 1 --2- 172.21.15.97:0/3604424426 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904108cb0 0x7f590410b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.731+0000 7f590ac00640 1 --2- 172.21.15.97:0/3604424426 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59041060f0 0x7f5904108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.731+0000 7f590ac00640 1 --2- 172.21.15.97:0/3604424426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f590406b140 0x7f5904105bb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.731+0000 7f590ac00640 1 -- 172.21.15.97:0/3604424426 >> 172.21.15.97:0/3604424426 conn(0x7f59040fd7b0 msgr2=0x7f59040ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.732+0000 7f590ac00640 1 -- 172.21.15.97:0/3604424426 shutdown_connections 2024-07-26T07:56:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.732+0000 7f590ac00640 1 -- 172.21.15.97:0/3604424426 wait complete. 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.733+0000 7f590ac00640 1 Processor -- start 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.733+0000 7f590ac00640 1 -- start start 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.733+0000 7f590ac00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590406b140 0x7f590419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.733+0000 7f590ac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59041060f0 0x7f590419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.733+0000 7f590ac00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904108cb0 0x7f590419cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.733+0000 7f590ac00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f590410ea10 con 0x7f5904108cb0 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.733+0000 7f590ac00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f590410e890 con 0x7f590406b140 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.733+0000 7f590ac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f590410eb90 con 0x7f59041060f0 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f590a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904108cb0 0x7f590419cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f590a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904108cb0 0x7f590419cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47774/0 (socket says 172.21.15.97:47774) 2024-07-26T07:56:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f590a200640 1 -- 172.21.15.97:0/1434763546 learned_addr learned my addr 172.21.15.97:0/1434763546 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f5909800640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590406b140 0x7f590419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f5908e00640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59041060f0 0x7f590419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f590a200640 1 -- 172.21.15.97:0/1434763546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59041060f0 msgr2=0x7f590419bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f590a200640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59041060f0 0x7f590419bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f590a200640 1 -- 172.21.15.97:0/1434763546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590406b140 msgr2=0x7f590419b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f590a200640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590406b140 0x7f590419b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.734+0000 7f590a200640 1 -- 172.21.15.97:0/1434763546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590419d340 con 0x7f5904108cb0 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.735+0000 7f590a200640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904108cb0 0x7f590419cb20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00eb10 tx=0x7f58fc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.735+0000 7f5902000640 1 -- 172.21.15.97:0/1434763546 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc004b20 con 0x7f5904108cb0 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.735+0000 7f5902000640 1 -- 172.21.15.97:0/1434763546 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58fc004cc0 con 0x7f5904108cb0 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.735+0000 7f590ac00640 1 -- 172.21.15.97:0/1434763546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590419d630 con 0x7f5904108cb0 2024-07-26T07:56:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.735+0000 7f5902000640 1 -- 172.21.15.97:0/1434763546 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc010690 con 0x7f5904108cb0 2024-07-26T07:56:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.735+0000 7f590ac00640 1 -- 172.21.15.97:0/1434763546 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59041a3d20 con 0x7f5904108cb0 2024-07-26T07:56:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.739+0000 7f5913214740 1 Processor -- start 2024-07-26T07:56:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.740+0000 7f5902000640 1 -- 172.21.15.97:0/1434763546 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f58fc005020 con 0x7f5904108cb0 2024-07-26T07:56:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.740+0000 7f5913214740 1 -- start start 2024-07-26T07:56:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.740+0000 7f5913214740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9fec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.740+0000 7f5913214740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 0x556ed9fec750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.740+0000 7f5913214740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 0x556ed9fed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.740+0000 7f5913214740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x556ed9f5e270 con 0x556ed9c461b0 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.740+0000 7f5913214740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556ed9f4d770 con 0x556ed9c254d0 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.740+0000 7f5913214740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556ed9f517b0 con 0x556ed9c1d2f0 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9fec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9fec0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47778/0 (socket says 172.21.15.97:47778) 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 -- 172.21.15.97:0/551345096 learned_addr learned my addr 172.21.15.97:0/551345096 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5909800640 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 0x556ed9fec750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 0x556ed9fed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 -- 172.21.15.97:0/551345096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 msgr2=0x556ed9fec750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 0x556ed9fec750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 -- 172.21.15.97:0/551345096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 msgr2=0x556ed9fed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 0x556ed9fed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.741+0000 7f5908e00640 1 -- 172.21.15.97:0/551345096 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed9cb1790 con 0x556ed9c461b0 2024-07-26T07:56:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.742+0000 7f5908e00640 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9fec0e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009fa0 tx=0x7f58f00097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4219bcd248f163 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.742+0000 7f5902000640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58ec0b3c50 0x7f58ec0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.742+0000 7f58f6a00640 1 -- 172.21.15.97:0/551345096 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0004230 con 0x556ed9c461b0 2024-07-26T07:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.742+0000 7f58f6a00640 1 -- 172.21.15.97:0/551345096 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58f00043d0 con 0x556ed9c461b0 2024-07-26T07:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.742+0000 7f58f6a00640 1 -- 172.21.15.97:0/551345096 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f00108b0 con 0x556ed9c461b0 2024-07-26T07:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.742+0000 7f5902000640 1 -- 172.21.15.97:0/1434763546 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f58fc0e4370 con 0x7f5904108cb0 2024-07-26T07:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.742+0000 7f5909800640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58ec0b3c50 0x7f58ec0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 -- 172.21.15.97:0/551345096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 msgr2=0x556ed9fec0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9fec0e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009fa0 tx=0x7f58f00097a0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 -- 172.21.15.97:0/551345096 shutdown_connections 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 0x556ed9fed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 0x556ed9fec750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 --2- 172.21.15.97:0/551345096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9fec0e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 -- 172.21.15.97:0/551345096 >> 172.21.15.97:0/551345096 conn(0x556ed9c3a300 msgr2=0x556ed9fe7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5909800640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58ec0b3c50 0x7f58ec0b6160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f58f8000c00 tx=0x7f58f8005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 -- 172.21.15.97:0/551345096 shutdown_connections 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.743+0000 7f5913214740 1 -- 172.21.15.97:0/551345096 wait complete. 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 1 -- start start 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 1 Processor -- start 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 0x556ed9ff5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 0x556ed9ff6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9ff6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x556ed9e87ab0 con 0x556ed9c461b0 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x556ed9eead50 con 0x556ed9c254d0 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556ed9feddc0 con 0x556ed9c1d2f0 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.744+0000 7f5913214740 10 client.0 _ll_register_callbacks cb 0x556ed9c5da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f590a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9ff6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f5908e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 0x556ed9ff6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f590a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9ff6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47792/0 (socket says 172.21.15.97:47792) 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f590a200640 1 -- 172.21.15.97:0/842932175 learned_addr learned my addr 172.21.15.97:0/842932175 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f5909800640 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 0x556ed9ff5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f590a200640 1 -- 172.21.15.97:0/842932175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 msgr2=0x556ed9ff5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f590a200640 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 0x556ed9ff5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f590a200640 1 -- 172.21.15.97:0/842932175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 msgr2=0x556ed9ff6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f590a200640 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 0x556ed9ff6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.745+0000 7f590a200640 1 -- 172.21.15.97:0/842932175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed9d2a620 con 0x556ed9c461b0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.746+0000 7f590a200640 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9ff6d90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0aaf00 tx=0x7f58fc0aa970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a70462a585cc707 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.746+0000 7f58eb400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.746+0000 7f58eb400640 1 -- 172.21.15.97:0/842932175 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc0c4020 con 0x556ed9c461b0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.746+0000 7f58eb400640 1 -- 172.21.15.97:0/842932175 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f58fc0ab670 con 0x556ed9c461b0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.746+0000 7f5913214740 1 -- 172.21.15.97:0/842932175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58f00043d0 con 0x556ed9c461b0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.746+0000 7f58eb400640 1 -- 172.21.15.97:0/842932175 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f58fc0ab200 con 0x556ed9c461b0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.746+0000 7f5913214740 10 client.16083 fetch_fsmap learned FSMap version 33 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.746+0000 7f5913214740 1 -- 172.21.15.97:0/842932175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556ed9fffdb0 con 0x556ed9c461b0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.747+0000 7f58eb400640 1 -- 172.21.15.97:0/842932175 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58f00043d0 con 0x556ed9c461b0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.747+0000 7f58eb400640 1 -- 172.21.15.97:0/842932175 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f58fc092070 con 0x556ed9c461b0 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.747+0000 7f5913214740 10 client.16083 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.747+0000 7f5913214740 10 client.16083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.747+0000 7f5913214740 20 client.16083 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.747+0000 7f5913214740 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556ed9c04d80 0x556eda003650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.747+0000 7f5913214740 4 client.16083 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:56:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.747+0000 7f5913214740 1 -- 172.21.15.97:0/842932175 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ed9f601c0 con 0x556ed9c04d80 2024-07-26T07:56:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.748+0000 7f5909800640 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556ed9c04d80 0x556eda003650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.749+0000 7f58eb400640 10 client.16083 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.749+0000 7f5909800640 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556ed9c04d80 0x556eda003650 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a930842cbd41a7 server_cookie=60c0018c384cc1f4 in_seq=0 out_seq=0 2024-07-26T07:56:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:55 smithi097 ceph-mon[32231]: pgmap v665: 97 pgs: 97 active+clean; 107 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-07-26T07:56:55.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f58eb400640 1 -- 172.21.15.97:0/842932175 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556ed9f601c0 con 0x556ed9c04d80 2024-07-26T07:56:55.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f58eb400640 10 client.16083 handle_command_reply: tid=0 2024-07-26T07:56:55.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 client.16083 shutdown 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 -- 172.21.15.97:0/842932175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 msgr2=0x556ed9ff6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9ff6d90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0aaf00 tx=0x7f58fc0aa970 comp rx=0 tx=0).stop 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 -- 172.21.15.97:0/842932175 shutdown_connections 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556ed9c04d80 0x556eda003650 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c461b0 0x556ed9ff6d90 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c254d0 0x556ed9ff6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 --2- 172.21.15.97:0/842932175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c1d2f0 0x556ed9ff5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f5913214740 1 -- 172.21.15.97:0/842932175 >> 172.21.15.97:0/842932175 conn(0x556ed9c3a300 msgr2=0x556ed9fe60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.765+0000 7f58eb400640 0 client.16083 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.766+0000 7f5913214740 1 -- 172.21.15.97:0/842932175 shutdown_connections 2024-07-26T07:56:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.766+0000 7f5913214740 1 -- 172.21.15.97:0/842932175 wait complete. 2024-07-26T07:56:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.766+0000 7f5913214740 20 client.16083 trim_cache size 0 max 16384 2024-07-26T07:56:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.788+0000 7f5913214740 1 Processor -- start 2024-07-26T07:56:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.788+0000 7f5913214740 1 -- start start 2024-07-26T07:56:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.788+0000 7f5913214740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c1d2f0 0x556eda0035c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.788+0000 7f5913214740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c254d0 0x556ed9fe83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.788+0000 7f5913214740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c461b0 0x556ed9fe8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.788+0000 7f5913214740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f59041a85a0 con 0x556ed9c1d2f0 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.788+0000 7f5913214740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f59041aa1a0 con 0x556ed9c461b0 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.788+0000 7f5913214740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f59041a8c70 con 0x556ed9c254d0 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5909800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c1d2f0 0x556eda0035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5909800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c1d2f0 0x556eda0035c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47804/0 (socket says 172.21.15.97:47804) 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5909800640 1 -- 172.21.15.97:0/2437494237 learned_addr learned my addr 172.21.15.97:0/2437494237 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f590a200640 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c461b0 0x556ed9fe8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5908e00640 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c254d0 0x556ed9fe83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5909800640 1 -- 172.21.15.97:0/2437494237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c254d0 msgr2=0x556ed9fe83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5909800640 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c254d0 0x556ed9fe83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5909800640 1 -- 172.21.15.97:0/2437494237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c461b0 msgr2=0x556ed9fe8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5909800640 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c461b0 0x556ed9fe8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f5909800640 1 -- 172.21.15.97:0/2437494237 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed9fffba0 con 0x556ed9c1d2f0 2024-07-26T07:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.789+0000 7f590a200640 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c461b0 0x556ed9fe8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:56:55.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.790+0000 7f5909800640 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c1d2f0 0x556eda0035c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009900 tx=0x7f58f803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b1d8ac314950b5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.790+0000 7f58f6000640 1 -- 172.21.15.97:0/2437494237 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8035050 con 0x556ed9c1d2f0 2024-07-26T07:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.790+0000 7f58f6000640 1 -- 172.21.15.97:0/2437494237 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58f8056070 con 0x556ed9c1d2f0 2024-07-26T07:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.790+0000 7f58f6000640 1 -- 172.21.15.97:0/2437494237 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8051310 con 0x556ed9c1d2f0 2024-07-26T07:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.791+0000 7f5913214740 1 -- 172.21.15.97:0/2437494237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c1d2f0 msgr2=0x556eda0035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.791+0000 7f5913214740 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c1d2f0 0x556eda0035c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009900 tx=0x7f58f803e000 comp rx=0 tx=0).stop 2024-07-26T07:56:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 -- 172.21.15.97:0/2437494237 shutdown_connections 2024-07-26T07:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c461b0 0x556ed9fe8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c254d0 0x556ed9fe83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 --2- 172.21.15.97:0/2437494237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c1d2f0 0x556eda0035c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 -- 172.21.15.97:0/2437494237 >> 172.21.15.97:0/2437494237 conn(0x556ed9c3a300 msgr2=0x556ed9fec080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 -- 172.21.15.97:0/2437494237 shutdown_connections 2024-07-26T07:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 -- 172.21.15.97:0/2437494237 wait complete. 2024-07-26T07:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 -- start start 2024-07-26T07:56:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.792+0000 7f5913214740 1 Processor -- start 2024-07-26T07:56:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5913214740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c1d2f0 0x556ed9ffae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5913214740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c254d0 0x556ed9ffb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5913214740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c461b0 0x556ed9ffc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5913214740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f59041a9a10 con 0x556ed9c254d0 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5913214740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f59041a9340 con 0x556ed9c1d2f0 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5913214740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f59041a6580 con 0x556ed9c461b0 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5913214740 10 client.0 _ll_register_callbacks cb 0x7f58fc005710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5908e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c254d0 0x556ed9ffb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5908e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c254d0 0x556ed9ffb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60884/0 (socket says 172.21.15.97:60884) 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5908e00640 1 -- 172.21.15.97:0/4215170778 learned_addr learned my addr 172.21.15.97:0/4215170778 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f5909800640 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c1d2f0 0x556ed9ffae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.793+0000 7f590a200640 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c461b0 0x556ed9ffc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.794+0000 7f5908e00640 1 -- 172.21.15.97:0/4215170778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c461b0 msgr2=0x556ed9ffc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.794+0000 7f5908e00640 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c461b0 0x556ed9ffc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.794+0000 7f5908e00640 1 -- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c1d2f0 msgr2=0x556ed9ffae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.794+0000 7f5908e00640 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c1d2f0 0x556ed9ffae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.794+0000 7f5908e00640 1 -- 172.21.15.97:0/4215170778 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed9ffc7a0 con 0x556ed9c254d0 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.794+0000 7f5909800640 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c1d2f0 0x556ed9ffae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.794+0000 7f5908e00640 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c254d0 0x556ed9ffb4d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009fd0 tx=0x7f58f0005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991af06d0ad36d8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.794+0000 7f58eb400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.795+0000 7f58eb400640 1 -- 172.21.15.97:0/4215170778 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f002f060 con 0x556ed9c254d0 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.795+0000 7f58eb400640 1 -- 172.21.15.97:0/4215170778 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f58f00035f0 con 0x556ed9c254d0 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.795+0000 7f5913214740 1 -- 172.21.15.97:0/4215170778 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58f8056070 con 0x556ed9c254d0 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.795+0000 7f58eb400640 1 -- 172.21.15.97:0/4215170778 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f58f00037f0 con 0x556ed9c254d0 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.795+0000 7f5913214740 10 client.16089 fetch_fsmap learned FSMap version 33 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.795+0000 7f5913214740 1 -- 172.21.15.97:0/4215170778 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556ed9ffdac0 con 0x556ed9c254d0 2024-07-26T07:56:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.795+0000 7f58eb400640 1 -- 172.21.15.97:0/4215170778 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58f8056070 con 0x556ed9c254d0 2024-07-26T07:56:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.795+0000 7f58eb400640 1 -- 172.21.15.97:0/4215170778 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7f58f0033030 con 0x556ed9c254d0 2024-07-26T07:56:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.796+0000 7f5913214740 10 client.16089 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T07:56:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.796+0000 7f5913214740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:56:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.796+0000 7f5913214740 20 client.16089 populate_metadata read hostname 'smithi097' 2024-07-26T07:56:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.796+0000 7f5913214740 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556ed9bda1c0 0x556eda06df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:56:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.796+0000 7f5913214740 4 client.16089 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:56:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.796+0000 7f5913214740 1 -- 172.21.15.97:0/4215170778 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ed9f5bc60 con 0x556ed9bda1c0 2024-07-26T07:56:55.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.796+0000 7f5909800640 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556ed9bda1c0 0x556eda06df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:56:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.797+0000 7f5909800640 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556ed9bda1c0 0x556eda06df70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb3ccfa41dd8b4d6 server_cookie=59c5661b7650f921 in_seq=0 out_seq=0 2024-07-26T07:56:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:55.797+0000 7f58eb400640 10 client.16089 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:55.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:55 smithi115 ceph-mon[35364]: pgmap v665: 97 pgs: 97 active+clean; 107 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-07-26T07:56:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:55 smithi181 ceph-mon[35869]: pgmap v665: 97 pgs: 97 active+clean; 107 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-07-26T07:56:56.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f58eb400640 1 -- 172.21.15.97:0/4215170778 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556ed9f5bc60 con 0x556ed9bda1c0 2024-07-26T07:56:56.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f58eb400640 10 client.16089 handle_command_reply: tid=0 2024-07-26T07:56:56.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f5913214740 1 client.16089 shutdown 2024-07-26T07:56:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f5913214740 1 -- 172.21.15.97:0/4215170778 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c254d0 msgr2=0x556ed9ffb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f5913214740 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c254d0 0x556ed9ffb4d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009fd0 tx=0x7f58f0005270 comp rx=0 tx=0).stop 2024-07-26T07:56:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f5913214740 1 -- 172.21.15.97:0/4215170778 shutdown_connections 2024-07-26T07:56:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f5913214740 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x556ed9bda1c0 0x556eda06df70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f5913214740 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556ed9c461b0 0x556ed9ffc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f5913214740 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556ed9c254d0 0x556ed9ffb4d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.075+0000 7f5913214740 1 --2- 172.21.15.97:0/4215170778 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x556ed9c1d2f0 0x556ed9ffae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.076+0000 7f5913214740 1 -- 172.21.15.97:0/4215170778 >> 172.21.15.97:0/4215170778 conn(0x556ed9c3a300 msgr2=0x556ed9fe60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.076+0000 7f58eb400640 0 client.16089 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.076+0000 7f5913214740 1 -- 172.21.15.97:0/4215170778 shutdown_connections 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.076+0000 7f5913214740 1 -- 172.21.15.97:0/4215170778 wait complete. 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.076+0000 7f5913214740 20 client.16089 trim_cache size 0 max 16384 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:56:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:56:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:56:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:56:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.077+0000 7f590ac00640 1 -- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58ec0b3c50 msgr2=0x7f58ec0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.077+0000 7f590ac00640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58ec0b3c50 0x7f58ec0b6160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f58f8000c00 tx=0x7f58f8005d50 comp rx=0 tx=0).stop 2024-07-26T07:56:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.077+0000 7f590ac00640 1 -- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904108cb0 msgr2=0x7f590419cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.077+0000 7f590ac00640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904108cb0 0x7f590419cb20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00eb10 tx=0x7f58fc00c2d0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.077+0000 7f590ac00640 1 -- 172.21.15.97:0/1434763546 shutdown_connections 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.077+0000 7f590ac00640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f58ec0b3c50 0x7f58ec0b6160 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.077+0000 7f590ac00640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904108cb0 0x7f590419cb20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.077+0000 7f590ac00640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59041060f0 0x7f590419bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.078+0000 7f590ac00640 1 --2- 172.21.15.97:0/1434763546 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f590406b140 0x7f590419b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.078+0000 7f590ac00640 1 -- 172.21.15.97:0/1434763546 >> 172.21.15.97:0/1434763546 conn(0x7f59040fd7b0 msgr2=0x7f5904104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.078+0000 7f590ac00640 1 -- 172.21.15.97:0/1434763546 shutdown_connections 2024-07-26T07:56:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:56:56.078+0000 7f590ac00640 1 -- 172.21.15.97:0/1434763546 wait complete. 2024-07-26T07:56:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:57 smithi097 ceph-mon[32231]: pgmap v666: 97 pgs: 97 active+clean; 111 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-07-26T07:56:57.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:57 smithi115 ceph-mon[35364]: pgmap v666: 97 pgs: 97 active+clean; 111 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-07-26T07:56:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:57 smithi181 ceph-mon[35869]: pgmap v666: 97 pgs: 97 active+clean; 111 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-07-26T07:56:59.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.207+0000 7fdfaaa00640 1 -- 172.21.15.97:0/1107387716 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x558d1305c530 con 0x558d12cd7570 2024-07-26T07:56:59.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.207+0000 7fdfaaa00640 10 client.16071 handle_command_reply: tid=0 2024-07-26T07:56:59.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.207+0000 7fdfd27de740 1 client.16071 shutdown 2024-07-26T07:56:59.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 -- 172.21.15.97:0/1107387716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 msgr2=0x558d130f8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:59.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130f8570 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80a6dc0 tx=0x7fdfb80af270 comp rx=0 tx=0).stop 2024-07-26T07:56:59.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 -- 172.21.15.97:0/1107387716 shutdown_connections 2024-07-26T07:56:59.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x558d12cd7570 0x558d131825f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x558d12d43840 0x558d131677c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d12d22b60 0x558d130f8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 --2- 172.21.15.97:0/1107387716 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558d12d1a980 0x558d130f8570 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 -- 172.21.15.97:0/1107387716 >> 172.21.15.97:0/1107387716 conn(0x558d12d37990 msgr2=0x558d130ead80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfaaa00640 0 client.16071 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:56:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.208+0000 7fdfd27de740 1 -- 172.21.15.97:0/1107387716 shutdown_connections 2024-07-26T07:56:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.209+0000 7fdfd27de740 1 -- 172.21.15.97:0/1107387716 wait complete. 2024-07-26T07:56:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.209+0000 7fdfd27de740 20 client.16071 trim_cache size 0 max 16384 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 33, 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 14, 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "631d9159": { 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 14, 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:56:59.212 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 130.0, 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:56:59.213 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:56:59.214 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:56:59.214 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:56:59.214 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:56:59.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 -- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfac0b3d10 msgr2=0x7fdfac0b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:59.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfac0b3d10 0x7fdfac0b6220 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009fd0 tx=0x7fdfbc009500 comp rx=0 tx=0).stop 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 -- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc4101ea0 msgr2=0x7fdfc4107ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc4101ea0 0x7fdfc4107ef0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdfb800c920 tx=0x7fdfb800cde0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 -- 172.21.15.97:0/3679432198 shutdown_connections 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fdfac0b3d10 0x7fdfac0b6220 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfc410fa50 0x7fdfc41048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41027f0 0x7fdfc41041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 --2- 172.21.15.97:0/3679432198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdfc4101ea0 0x7fdfc4107ef0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.210+0000 7fdfca600640 1 -- 172.21.15.97:0/3679432198 >> 172.21.15.97:0/3679432198 conn(0x7fdfc40fdd60 msgr2=0x7fdfc4100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.211+0000 7fdfca600640 1 -- 172.21.15.97:0/3679432198 shutdown_connections 2024-07-26T07:56:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:56:59.211+0000 7fdfca600640 1 -- 172.21.15.97:0/3679432198 wait complete. 2024-07-26T07:56:59.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:56:59 smithi097 ceph-mon[32231]: pgmap v667: 97 pgs: 97 active+clean; 107 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.2 MiB/s wr, 74 op/s 2024-07-26T07:56:59.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:56:59 smithi115 ceph-mon[35364]: pgmap v667: 97 pgs: 97 active+clean; 107 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.2 MiB/s wr, 74 op/s 2024-07-26T07:56:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:56:59 smithi181 ceph-mon[35869]: pgmap v667: 97 pgs: 97 active+clean; 107 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.2 MiB/s wr, 74 op/s 2024-07-26T07:57:00.202 INFO:tasks.workunit.client.0.smithi097.stdout: 31 4619 70 3340 14 2090 56 2024-07-26T07:57:00.513 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '631d9159', quiesce took 15.1 seconds. Will release after: -5.1 2024-07-26T07:57:00.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:57:01.117 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:57:01.609 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:01 smithi097 ceph-mon[32231]: pgmap v668: 97 pgs: 97 active+clean; 104 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-07-26T07:57:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:01 smithi115 ceph-mon[35364]: pgmap v668: 97 pgs: 97 active+clean; 104 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-07-26T07:57:02.011 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:01 smithi181 ceph-mon[35869]: pgmap v668: 97 pgs: 97 active+clean; 104 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-07-26T07:57:03.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.400+0000 7fa1f3200640 1 -- 172.21.15.97:0/4012148027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f4075b30 msgr2=0x7fa1f410f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:03.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.400+0000 7fa1f3200640 1 --2- 172.21.15.97:0/4012148027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f4075b30 0x7fa1f410f400 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009900 tx=0x7fa1e002a410 comp rx=0 tx=0).stop 2024-07-26T07:57:03.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.400+0000 7fa1f3200640 1 -- 172.21.15.97:0/4012148027 shutdown_connections 2024-07-26T07:57:03.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.400+0000 7fa1f3200640 1 --2- 172.21.15.97:0/4012148027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1f410f940 0x7fa1f4111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.400+0000 7fa1f3200640 1 --2- 172.21.15.97:0/4012148027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f4075b30 0x7fa1f410f400 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.400+0000 7fa1f3200640 1 --2- 172.21.15.97:0/4012148027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40751e0 0x7fa1f40755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.400+0000 7fa1f3200640 1 -- 172.21.15.97:0/4012148027 >> 172.21.15.97:0/4012148027 conn(0x7fa1f40fee70 msgr2=0x7fa1f41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:03.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.401+0000 7fa1f3200640 1 -- 172.21.15.97:0/4012148027 shutdown_connections 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.401+0000 7fa1f3200640 1 -- 172.21.15.97:0/4012148027 wait complete. 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.402+0000 7fa1f3200640 1 Processor -- start 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.402+0000 7fa1f3200640 1 -- start start 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.402+0000 7fa1f3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40751e0 0x7fa1f419be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.402+0000 7fa1f3200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1f4075b30 0x7fa1f419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.402+0000 7fa1f3200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f410f940 0x7fa1f41a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.402+0000 7fa1f3200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f41153e0 con 0x7fa1f410f940 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.402+0000 7fa1f3200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f4115260 con 0x7fa1f4075b30 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.402+0000 7fa1f3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f4115560 con 0x7fa1f40751e0 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.403+0000 7fa1f2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f410f940 0x7fa1f41a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:03.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.403+0000 7fa1f2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f410f940 0x7fa1f41a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60894/0 (socket says 172.21.15.97:60894) 2024-07-26T07:57:03.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.403+0000 7fa1f2800640 1 -- 172.21.15.97:0/2891081263 learned_addr learned my addr 172.21.15.97:0/2891081263 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:03.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.403+0000 7fa1f1400640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1f4075b30 0x7fa1f419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:03.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.403+0000 7fa1f1e00640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40751e0 0x7fa1f419be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:03.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.404+0000 7fa1f2800640 1 -- 172.21.15.97:0/2891081263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40751e0 msgr2=0x7fa1f419be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:03.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.404+0000 7fa1f2800640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40751e0 0x7fa1f419be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.404+0000 7fa1f2800640 1 -- 172.21.15.97:0/2891081263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1f4075b30 msgr2=0x7fa1f419c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:03.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.404+0000 7fa1f2800640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1f4075b30 0x7fa1f419c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.404+0000 7fa1f2800640 1 -- 172.21.15.97:0/2891081263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f41a1de0 con 0x7fa1f410f940 2024-07-26T07:57:03.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.405+0000 7fa1f2800640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f410f940 0x7fa1f41a16b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00eb10 tx=0x7fa1ec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:03.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.405+0000 7fa1eaa00640 1 -- 172.21.15.97:0/2891081263 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec004b20 con 0x7fa1f410f940 2024-07-26T07:57:03.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.406+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f41a20d0 con 0x7fa1f410f940 2024-07-26T07:57:03.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.406+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f4102f90 con 0x7fa1f410f940 2024-07-26T07:57:03.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.407+0000 7fa1eaa00640 1 -- 172.21.15.97:0/2891081263 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa1ec004cc0 con 0x7fa1f410f940 2024-07-26T07:57:03.409 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.407+0000 7fa1eaa00640 1 -- 172.21.15.97:0/2891081263 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec021440 con 0x7fa1f410f940 2024-07-26T07:57:03.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.409+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d4005290 con 0x7fa1f410f940 2024-07-26T07:57:03.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.410+0000 7fa1eaa00640 1 -- 172.21.15.97:0/2891081263 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fa1ec021620 con 0x7fa1f410f940 2024-07-26T07:57:03.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.413+0000 7fa1eaa00640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa1d80b3c50 0x7fa1d80b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:03.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.413+0000 7fa1eaa00640 1 -- 172.21.15.97:0/2891081263 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fa1ec0e4220 con 0x7fa1f410f940 2024-07-26T07:57:03.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.413+0000 7fa1f1e00640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa1d80b3c50 0x7fa1d80b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:03.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.414+0000 7fa1f1e00640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa1d80b3c50 0x7fa1d80b6160 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009bb0 tx=0x7fa1e4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:03.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.418+0000 7fa1eaa00640 1 -- 172.21.15.97:0/2891081263 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa1ec0ab7b0 con 0x7fa1f410f940 2024-07-26T07:57:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:03 smithi097 ceph-mon[32231]: pgmap v669: 97 pgs: 97 active+clean; 104 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 233 op/s 2024-07-26T07:57:03.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.824+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1d4005580 con 0x7fa1f410f940 2024-07-26T07:57:03.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.825+0000 7fa1eaa00640 1 -- 172.21.15.97:0/2891081263 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11998 (secure 0 0 0) 0x7fa1ec0ab0b0 con 0x7fa1f410f940 2024-07-26T07:57:03.827 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:57:03.827 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":33,"btime":"2024-07-26T07:56:15:216944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/4227989530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4227989530},{"type":"v1","addr":"172.21.15.97:6837","nonce":4227989530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:56:15.216901+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3957930666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3957930666},{"type":"v1","addr":"172.21.15.181:6837","nonce":3957930666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:57:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 33 2024-07-26T07:57:03.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa1d80b3c50 msgr2=0x7fa1d80b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:03.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa1d80b3c50 0x7fa1d80b6160 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009bb0 tx=0x7fa1e4005f90 comp rx=0 tx=0).stop 2024-07-26T07:57:03.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f410f940 msgr2=0x7fa1f41a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:03.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f410f940 0x7fa1f41a16b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00eb10 tx=0x7fa1ec00c2d0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 shutdown_connections 2024-07-26T07:57:03.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fa1d80b3c50 0x7fa1d80b6160 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009bb0 tx=0x7fa1e4005f90 comp rx=0 tx=0).stop 2024-07-26T07:57:03.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1f410f940 0x7fa1f41a16b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1f4075b30 0x7fa1f419c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 --2- 172.21.15.97:0/2891081263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40751e0 0x7fa1f419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:03.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.829+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 >> 172.21.15.97:0/2891081263 conn(0x7fa1f40fee70 msgr2=0x7fa1f4109520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:03.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.830+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 shutdown_connections 2024-07-26T07:57:03.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:03.830+0000 7fa1f3200640 1 -- 172.21.15.97:0/2891081263 wait complete. 2024-07-26T07:57:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:03 smithi115 ceph-mon[35364]: pgmap v669: 97 pgs: 97 active+clean; 104 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 233 op/s 2024-07-26T07:57:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:03 smithi181 ceph-mon[35869]: pgmap v669: 97 pgs: 97 active+clean; 104 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 233 op/s 2024-07-26T07:57:04.519 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:04 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2891081263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:04 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2891081263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:04 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2891081263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:05 smithi097 ceph-mon[32231]: pgmap v670: 97 pgs: 97 active+clean; 104 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 227 op/s 2024-07-26T07:57:05.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:05 smithi181 ceph-mon[35869]: pgmap v670: 97 pgs: 97 active+clean; 104 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 227 op/s 2024-07-26T07:57:05.937 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db --set-id 631d9159 --release --await' 2024-07-26T07:57:05.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db --set-id 631d9159 --release --await 2024-07-26T07:57:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:05 smithi115 ceph-mon[35364]: pgmap v670: 97 pgs: 97 active+clean; 104 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 227 op/s 2024-07-26T07:57:06.530 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:57:07.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:07 smithi097 ceph-mon[32231]: pgmap v671: 97 pgs: 97 active+clean; 107 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 213 op/s 2024-07-26T07:57:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:07 smithi115 ceph-mon[35364]: pgmap v671: 97 pgs: 97 active+clean; 107 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 213 op/s 2024-07-26T07:57:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:07 smithi181 ceph-mon[35869]: pgmap v671: 97 pgs: 97 active+clean; 107 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 213 op/s 2024-07-26T07:57:08.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.831+0000 7fbac3200640 1 -- 172.21.15.97:0/4181151202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbac410fa20 msgr2=0x7fbac4111e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.831+0000 7fbac3200640 1 --2- 172.21.15.97:0/4181151202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbac410fa20 0x7fbac4111e40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009be0 tx=0x7fbabc02a750 comp rx=0 tx=0).stop 2024-07-26T07:57:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.832+0000 7fbac3200640 1 -- 172.21.15.97:0/4181151202 shutdown_connections 2024-07-26T07:57:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.832+0000 7fbac3200640 1 --2- 172.21.15.97:0/4181151202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbac410fa20 0x7fbac4111e40 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.832+0000 7fbac3200640 1 --2- 172.21.15.97:0/4181151202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac41027c0 0x7fbac410f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.832+0000 7fbac3200640 1 --2- 172.21.15.97:0/4181151202 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbac4101e70 0x7fbac4102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.832+0000 7fbac3200640 1 -- 172.21.15.97:0/4181151202 >> 172.21.15.97:0/4181151202 conn(0x7fbac40fdcf0 msgr2=0x7fbac4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.832+0000 7fbac3200640 1 -- 172.21.15.97:0/4181151202 shutdown_connections 2024-07-26T07:57:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.832+0000 7fbac3200640 1 -- 172.21.15.97:0/4181151202 wait complete. 2024-07-26T07:57:08.836 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:57:08.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.834+0000 7fbac3200640 1 Processor -- start 2024-07-26T07:57:08.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.834+0000 7fbac3200640 1 -- start start 2024-07-26T07:57:08.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.834+0000 7fbac3200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbac4101e70 0x7fbac419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.834+0000 7fbac3200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbac41027c0 0x7fbac419c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.834+0000 7fbac3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac410fa20 0x7fbac41a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.834+0000 7fbac3200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4115320 con 0x7fbac41027c0 2024-07-26T07:57:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.834+0000 7fbac3200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbac41151a0 con 0x7fbac4101e70 2024-07-26T07:57:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.834+0000 7fbac3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbac41154a0 con 0x7fbac410fa20 2024-07-26T07:57:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.835+0000 7fbac1e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbac4101e70 0x7fbac419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.835+0000 7fbac2800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac410fa20 0x7fbac41a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.835+0000 7fbac1e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbac4101e70 0x7fbac419bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:38682/0 (socket says 172.21.15.97:38682) 2024-07-26T07:57:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.835+0000 7fbac1e00640 1 -- 172.21.15.97:0/3366899828 learned_addr learned my addr 172.21.15.97:0/3366899828 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.835+0000 7fbac1400640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbac41027c0 0x7fbac419c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.836+0000 7fbac2800640 1 -- 172.21.15.97:0/3366899828 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbac4101e70 msgr2=0x7fbac419bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:08.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.836+0000 7fbac2800640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbac4101e70 0x7fbac419bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.836+0000 7fbac2800640 1 -- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbac41027c0 msgr2=0x7fbac419c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:08.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.836+0000 7fbac2800640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbac41027c0 0x7fbac419c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.836+0000 7fbac2800640 1 -- 172.21.15.97:0/3366899828 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac41a1d00 con 0x7fbac410fa20 2024-07-26T07:57:08.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.836+0000 7fbac2800640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac410fa20 0x7fbac41a15d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009b30 tx=0x7fbabc003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:08.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.837+0000 7fbabaa00640 1 -- 172.21.15.97:0/3366899828 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc041020 con 0x7fbac410fa20 2024-07-26T07:57:08.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.837+0000 7fbac3200640 1 -- 172.21.15.97:0/3366899828 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac41a1f90 con 0x7fbac410fa20 2024-07-26T07:57:08.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.837+0000 7fbac3200640 1 -- 172.21.15.97:0/3366899828 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac41a8750 con 0x7fbac410fa20 2024-07-26T07:57:08.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.837+0000 7fbabaa00640 1 -- 172.21.15.97:0/3366899828 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbabc037040 con 0x7fbac410fa20 2024-07-26T07:57:08.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.837+0000 7fbabaa00640 1 -- 172.21.15.97:0/3366899828 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc036070 con 0x7fbac410fa20 2024-07-26T07:57:08.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.841+0000 7fbacfbc4740 1 Processor -- start 2024-07-26T07:57:08.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.842+0000 7fbacfbc4740 1 -- start start 2024-07-26T07:57:08.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.842+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa088ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.842+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08519660 0x55aa088efef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.842+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08521840 0x55aa088f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.842+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55aa0885df50 con 0x55aa08519660 2024-07-26T07:57:08.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.842+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55aa0884d450 con 0x55aa08521840 2024-07-26T07:57:08.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.842+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55aa08851490 con 0x55aa08542520 2024-07-26T07:57:08.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.843+0000 7fbac1400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa088ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.843+0000 7fbac1400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa088ef9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:60984/0 (socket says 172.21.15.97:60984) 2024-07-26T07:57:08.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.843+0000 7fbac1400640 1 -- 172.21.15.97:0/2531741756 learned_addr learned my addr 172.21.15.97:0/2531741756 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:08.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.843+0000 7fbac1400640 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08521840 0x55aa088f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.844+0000 7fbac1e00640 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08519660 0x55aa088efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.845+0000 7fbac1400640 1 -- 172.21.15.97:0/2531741756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08521840 msgr2=0x55aa088f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:08.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.845+0000 7fbac1400640 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08521840 0x55aa088f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.845+0000 7fbac1400640 1 -- 172.21.15.97:0/2531741756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08519660 msgr2=0x55aa088efef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:08.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.845+0000 7fbac1400640 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08519660 0x55aa088efef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.845+0000 7fbac1400640 1 -- 172.21.15.97:0/2531741756 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa085adb00 con 0x55aa08542520 2024-07-26T07:57:08.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.845+0000 7fbac1e00640 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08519660 0x55aa088efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:57:08.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.846+0000 7fbac1400640 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa088ef9b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbab00127e0 tx=0x7fbab000b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47ccd44111d3ee09 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:08.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.846+0000 7fbabaa00640 1 -- 172.21.15.97:0/3366899828 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fbabc044020 con 0x7fbac410fa20 2024-07-26T07:57:08.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.846+0000 7fbaaf400640 1 -- 172.21.15.97:0/2531741756 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0012e20 con 0x55aa08542520 2024-07-26T07:57:08.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.847+0000 7fbaaf400640 1 -- 172.21.15.97:0/2531741756 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbab0010c00 con 0x55aa08542520 2024-07-26T07:57:08.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.847+0000 7fbaaf400640 1 -- 172.21.15.97:0/2531741756 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab001e950 con 0x55aa08542520 2024-07-26T07:57:08.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.848+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2531741756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 msgr2=0x55aa088ef9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:08.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.848+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa088ef9b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbab00127e0 tx=0x7fbab000b790 comp rx=0 tx=0).stop 2024-07-26T07:57:08.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.849+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2531741756 shutdown_connections 2024-07-26T07:57:08.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.849+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08521840 0x55aa088f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.849+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08519660 0x55aa088efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.849+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/2531741756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa088ef9b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.849+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2531741756 >> 172.21.15.97:0/2531741756 conn(0x55aa08536910 msgr2=0x55aa088ed8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:08.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.849+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2531741756 shutdown_connections 2024-07-26T07:57:08.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.849+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2531741756 wait complete. 2024-07-26T07:57:08.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.850+0000 7fbacfbc4740 1 -- start start 2024-07-26T07:57:08.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.850+0000 7fbacfbc4740 1 Processor -- start 2024-07-26T07:57:08.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088fb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 0x55aa088fb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbabaa00640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbaa80b3c00 0x7fbaa80b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55aa086b0bb0 con 0x55aa08542520 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55aa087e9930 con 0x55aa08519660 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55aa0890f1a0 con 0x55aa08521840 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbac1e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088fb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbac1e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088fb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:38714/0 (socket says 172.21.15.97:38714) 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbac1e00640 1 -- 172.21.15.97:0/4225736601 learned_addr learned my addr 172.21.15.97:0/4225736601 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.851+0000 7fbacfbc4740 10 client.0 _ll_register_callbacks cb 0x55aa08559d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.852+0000 7fbabaa00640 1 -- 172.21.15.97:0/3366899828 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fbabc1043b0 con 0x7fbac410fa20 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.852+0000 7fbac2800640 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.852+0000 7fbac1400640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbaa80b3c00 0x7fbaa80b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.852+0000 7fbac1400640 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 0x55aa088fb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.853+0000 7fbac1e00640 1 -- 172.21.15.97:0/4225736601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 msgr2=0x55aa088fb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.853+0000 7fbac1e00640 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 0x55aa088fb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.853+0000 7fbac1e00640 1 -- 172.21.15.97:0/4225736601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 msgr2=0x55aa08918520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.853+0000 7fbac1e00640 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.853+0000 7fbac1e00640 1 -- 172.21.15.97:0/4225736601 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa08624960 con 0x55aa08519660 2024-07-26T07:57:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.853+0000 7fbac2800640 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:57:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.853+0000 7fbac1e00640 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088fb170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbab4013790 tx=0x7fbab4013c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87652dc2e9ac59cb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.853+0000 7fbaa3e00640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:57:08.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.854+0000 7fbac1400640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbaa80b3c00 0x7fbaa80b6110 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbab00059f0 tx=0x7fbab001e480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:08.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.854+0000 7fbaa3e00640 1 -- 172.21.15.97:0/4225736601 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab4020060 con 0x55aa08519660 2024-07-26T07:57:08.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.854+0000 7fbacfbc4740 1 -- 172.21.15.97:0/4225736601 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbab0010c00 con 0x55aa08519660 2024-07-26T07:57:08.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.854+0000 7fbaa3e00640 1 -- 172.21.15.97:0/4225736601 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fbab4011dd0 con 0x55aa08519660 2024-07-26T07:57:08.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.855+0000 7fbaa3e00640 1 -- 172.21.15.97:0/4225736601 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbab400f450 con 0x55aa08519660 2024-07-26T07:57:08.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.855+0000 7fbacfbc4740 10 client.35558 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:08.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.855+0000 7fbacfbc4740 1 -- 172.21.15.97:0/4225736601 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55aa0891b0a0 con 0x55aa08519660 2024-07-26T07:57:08.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.856+0000 7fbaa3e00640 1 -- 172.21.15.97:0/4225736601 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbab0010c00 con 0x55aa08519660 2024-07-26T07:57:08.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.857+0000 7fbaa3e00640 1 -- 172.21.15.97:0/4225736601 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7fbab4024030 con 0x55aa08519660 2024-07-26T07:57:08.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.857+0000 7fbacfbc4740 10 client.35558 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:08.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.857+0000 7fbacfbc4740 10 client.35558 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:57:08.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.857+0000 7fbacfbc4740 20 client.35558 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:08.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.857+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55aa08500ea0 0x55aa0891e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.857+0000 7fbacfbc4740 4 client.35558 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:57:08.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.857+0000 7fbacfbc4740 1 -- 172.21.15.97:0/4225736601 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa0885fea0 con 0x55aa08500ea0 2024-07-26T07:57:08.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.857+0000 7fbac1400640 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55aa08500ea0 0x55aa0891e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.858+0000 7fbac1400640 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55aa08500ea0 0x55aa0891e5f0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd6297c3522c4c2d server_cookie=28d794359f28e528 in_seq=0 out_seq=0 2024-07-26T07:57:08.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.858+0000 7fbaa3e00640 10 client.35558 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:08.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.859+0000 7fbaa3e00640 1 -- 172.21.15.97:0/4225736601 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aa0885fea0 con 0x55aa08500ea0 2024-07-26T07:57:08.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.859+0000 7fbaa3e00640 10 client.35558 handle_command_reply: tid=0 2024-07-26T07:57:08.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 client.35558 shutdown 2024-07-26T07:57:08.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 -- 172.21.15.97:0/4225736601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 msgr2=0x55aa088fb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088fb170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbab4013790 tx=0x7fbab4013c50 comp rx=0 tx=0).stop 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 -- 172.21.15.97:0/4225736601 shutdown_connections 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55aa08500ea0 0x55aa0891e5f0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 0x55aa088fb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/4225736601 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088fb170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbacfbc4740 1 -- 172.21.15.97:0/4225736601 >> 172.21.15.97:0/4225736601 conn(0x55aa08536910 msgr2=0x55aa088f2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.860+0000 7fbaa3e00640 0 client.35558 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:08.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.861+0000 7fbacfbc4740 1 -- 172.21.15.97:0/4225736601 shutdown_connections 2024-07-26T07:57:08.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.861+0000 7fbacfbc4740 1 -- 172.21.15.97:0/4225736601 wait complete. 2024-07-26T07:57:08.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.861+0000 7fbacfbc4740 20 client.35558 trim_cache size 0 max 16384 2024-07-26T07:57:08.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.897+0000 7fbacfbc4740 1 Processor -- start 2024-07-26T07:57:08.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.898+0000 7fbacfbc4740 1 -- start start 2024-07-26T07:57:08.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.898+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088f68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.898+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 0x55aa088e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.898+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.898+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa400e610 con 0x55aa08542520 2024-07-26T07:57:08.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.898+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa4010180 con 0x55aa08519660 2024-07-26T07:57:08.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.898+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa400ece0 con 0x55aa08521840 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.899+0000 7fbac2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.899+0000 7fbac2800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39214/0 (socket says 172.21.15.97:39214) 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.900+0000 7fbac2800640 1 -- 172.21.15.97:0/2907470267 learned_addr learned my addr 172.21.15.97:0/2907470267 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.900+0000 7fbac1e00640 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088f68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.900+0000 7fbac1400640 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 0x55aa088e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.900+0000 7fbac2800640 1 -- 172.21.15.97:0/2907470267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 msgr2=0x55aa088e3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.900+0000 7fbac2800640 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 0x55aa088e3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.900+0000 7fbac2800640 1 -- 172.21.15.97:0/2907470267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 msgr2=0x55aa088f68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:08.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.901+0000 7fbac2800640 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088f68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.901+0000 7fbac2800640 1 -- 172.21.15.97:0/2907470267 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa088f9670 con 0x55aa08542520 2024-07-26T07:57:08.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.901+0000 7fbac1e00640 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088f68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:57:08.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.902+0000 7fbac2800640 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918310 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fbabc101df0 tx=0x7fbabc0093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1128bfe3cff97729 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:08.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.902+0000 7fbaaea00640 1 -- 172.21.15.97:0/2907470267 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc0cad50 con 0x55aa08542520 2024-07-26T07:57:08.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.903+0000 7fbaaea00640 1 -- 172.21.15.97:0/2907470267 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbabc0cb940 con 0x55aa08542520 2024-07-26T07:57:08.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.903+0000 7fbaaea00640 1 -- 172.21.15.97:0/2907470267 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc0c74d0 con 0x55aa08542520 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.904+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2907470267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 msgr2=0x55aa08918310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.904+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918310 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fbabc101df0 tx=0x7fbabc0093a0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.904+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2907470267 shutdown_connections 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.904+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08542520 0x55aa08918310 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.904+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08521840 0x55aa088e3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.904+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/2907470267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa088f68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.904+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2907470267 >> 172.21.15.97:0/2907470267 conn(0x55aa08536910 msgr2=0x55aa088f1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.905+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2907470267 shutdown_connections 2024-07-26T07:57:08.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.905+0000 7fbacfbc4740 1 -- 172.21.15.97:0/2907470267 wait complete. 2024-07-26T07:57:08.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.906+0000 7fbacfbc4740 1 -- start start 2024-07-26T07:57:08.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.906+0000 7fbacfbc4740 1 Processor -- start 2024-07-26T07:57:08.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.907+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa089102f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.907+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08521840 0x55aa08910830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.908+0000 7fbac1e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa089102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.908+0000 7fbacfbc4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa08910d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.908+0000 7fbac1400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08521840 0x55aa08910830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.908+0000 7fbac1e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa089102f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:38728/0 (socket says 172.21.15.97:38728) 2024-07-26T07:57:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.908+0000 7fbacfbc4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa400fa80 con 0x55aa08521840 2024-07-26T07:57:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.908+0000 7fbac1e00640 1 -- 172.21.15.97:0/1293211047 learned_addr learned my addr 172.21.15.97:0/1293211047 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.908+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa400f3b0 con 0x55aa08519660 2024-07-26T07:57:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.908+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa400c5f0 con 0x55aa08542520 2024-07-26T07:57:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.909+0000 7fbacfbc4740 10 client.0 _ll_register_callbacks cb 0x7fbab40096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.909+0000 7fbac2800640 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa08910d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.909+0000 7fbac1400640 1 -- 172.21.15.97:0/1293211047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 msgr2=0x55aa08910d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:08.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.909+0000 7fbac1400640 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa08910d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.909+0000 7fbac1400640 1 -- 172.21.15.97:0/1293211047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 msgr2=0x55aa089102f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:08.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.909+0000 7fbac1400640 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa089102f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:08.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.909+0000 7fbac1400640 1 -- 172.21.15.97:0/1293211047 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa08985dc0 con 0x55aa08521840 2024-07-26T07:57:08.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.909+0000 7fbac2800640 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa08910d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:57:08.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.910+0000 7fbac1400640 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08521840 0x55aa08910830 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbab001e630 tx=0x7fbab000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37282af0ff96788a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:08.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.910+0000 7fbaa3e00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:08.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.911+0000 7fbaa3e00640 1 -- 172.21.15.97:0/1293211047 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0042060 con 0x55aa08521840 2024-07-26T07:57:08.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.911+0000 7fbaa3e00640 1 -- 172.21.15.97:0/1293211047 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fbab0010c90 con 0x55aa08521840 2024-07-26T07:57:08.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.911+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbabc0cb940 con 0x55aa08521840 2024-07-26T07:57:08.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.911+0000 7fbaa3e00640 1 -- 172.21.15.97:0/1293211047 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbab0039040 con 0x55aa08521840 2024-07-26T07:57:08.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.912+0000 7fbacfbc4740 10 client.16113 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:08.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.912+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55aa08986f90 con 0x55aa08521840 2024-07-26T07:57:08.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.912+0000 7fbaa3e00640 1 -- 172.21.15.97:0/1293211047 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbabc0cb940 con 0x55aa08521840 2024-07-26T07:57:08.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.913+0000 7fbaa3e00640 1 -- 172.21.15.97:0/1293211047 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7fbab0046030 con 0x55aa08521840 2024-07-26T07:57:08.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.914+0000 7fbacfbc4740 10 client.16113 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:08.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.914+0000 7fbacfbc4740 10 client.16113 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:57:08.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.914+0000 7fbacfbc4740 20 client.16113 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:08.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.914+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55aa084d6250 0x55aa08989590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:08.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.914+0000 7fbacfbc4740 4 client.16113 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "631d9159", "release": true, "await": true}] 2024-07-26T07:57:08.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.914+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "set_id": "631d9159", "release": true, "await": true}) -- 0x55aa0885b940 con 0x55aa084d6250 2024-07-26T07:57:08.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.914+0000 7fbac1e00640 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55aa084d6250 0x55aa08989590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:08.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.915+0000 7fbaa3e00640 10 client.16113 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:08.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:08.915+0000 7fbac1e00640 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55aa084d6250 0x55aa08989590 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96532e3d04b6fa5f server_cookie=b8230ab2a89d7ad0 in_seq=0 out_seq=0 2024-07-26T07:57:08.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:57:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:57:09.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.205+0000 7fbaa3e00640 1 -- 172.21.15.97:0/1293211047 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55aa0885b940 con 0x55aa084d6250 2024-07-26T07:57:09.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.205+0000 7fbaa3e00640 10 client.16113 handle_command_reply: tid=0 2024-07-26T07:57:09.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.205+0000 7fbacfbc4740 1 client.16113 shutdown 2024-07-26T07:57:09.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.206+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08521840 msgr2=0x55aa08910830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.206+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08521840 0x55aa08910830 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbab001e630 tx=0x7fbab000b390 comp rx=0 tx=0).stop 2024-07-26T07:57:09.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.206+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 shutdown_connections 2024-07-26T07:57:09.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.206+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55aa084d6250 0x55aa08989590 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.206+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55aa08542520 0x55aa08910d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.206+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa08521840 0x55aa08910830 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.207+0000 7fbacfbc4740 1 --2- 172.21.15.97:0/1293211047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55aa08519660 0x55aa089102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.207+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 >> 172.21.15.97:0/1293211047 conn(0x55aa08536910 msgr2=0x55aa088f1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:09.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.207+0000 7fbaa3e00640 0 client.16113 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:09.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.207+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 shutdown_connections 2024-07-26T07:57:09.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.207+0000 7fbacfbc4740 1 -- 172.21.15.97:0/1293211047 wait complete. 2024-07-26T07:57:09.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.208+0000 7fbacfbc4740 20 client.16113 trim_cache size 0 max 16384 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 34, 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 16, 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "631d9159": { 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 16, 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:57:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:57:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:57:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:57:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:57:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 130.0, 2024-07-26T07:57:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:57:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:57:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:57:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:57:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:57:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:57:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:09.212 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:57:09.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 -- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbaa80b3c00 msgr2=0x7fbaa80b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbaa80b3c00 0x7fbaa80b6110 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbab00059f0 tx=0x7fbab001e480 comp rx=0 tx=0).stop 2024-07-26T07:57:09.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 -- 172.21.15.97:0/3366899828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac410fa20 msgr2=0x7fbac41a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac410fa20 0x7fbac41a15d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009b30 tx=0x7fbabc003f10 comp rx=0 tx=0).stop 2024-07-26T07:57:09.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 -- 172.21.15.97:0/3366899828 shutdown_connections 2024-07-26T07:57:09.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbaa80b3c00 0x7fbaa80b6110 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac410fa20 0x7fbac41a15d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbac41027c0 0x7fbac419c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 --2- 172.21.15.97:0/3366899828 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbac4101e70 0x7fbac419bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.209+0000 7fbac3200640 1 -- 172.21.15.97:0/3366899828 >> 172.21.15.97:0/3366899828 conn(0x7fbac40fdcf0 msgr2=0x7fbac4100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:09.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.210+0000 7fbac3200640 1 -- 172.21.15.97:0/3366899828 shutdown_connections 2024-07-26T07:57:09.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:09.211+0000 7fbac3200640 1 -- 172.21.15.97:0/3366899828 wait complete. 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.293+0000 7f2027800640 1 -- 172.21.15.97:0/2179951423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f202010a770 msgr2=0x7f202010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.293+0000 7f2027800640 1 --2- 172.21.15.97:0/2179951423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f202010a770 0x7f202010ab80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2008009900 tx=0x7f200802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.293+0000 7f2027800640 1 -- 172.21.15.97:0/2179951423 shutdown_connections 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.293+0000 7f2027800640 1 --2- 172.21.15.97:0/2179951423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202010b110 0x7f2020111ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.293+0000 7f2027800640 1 --2- 172.21.15.97:0/2179951423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f202010a770 0x7f202010ab80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.293+0000 7f2027800640 1 --2- 172.21.15.97:0/2179951423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20201096f0 0x7f2020109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.293+0000 7f2027800640 1 -- 172.21.15.97:0/2179951423 >> 172.21.15.97:0/2179951423 conn(0x7f2020078a80 msgr2=0x7f202007af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.294+0000 7f2027800640 1 -- 172.21.15.97:0/2179951423 shutdown_connections 2024-07-26T07:57:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.294+0000 7f2027800640 1 -- 172.21.15.97:0/2179951423 wait complete. 2024-07-26T07:57:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.295+0000 7f2027800640 1 Processor -- start 2024-07-26T07:57:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.295+0000 7f2027800640 1 -- start start 2024-07-26T07:57:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.296+0000 7f2027800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20201096f0 0x7f2020080be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.296+0000 7f2027800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f202010a770 0x7f2020081250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.296+0000 7f2027800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202010b110 0x7f2020081e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.296+0000 7f2027800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2020115220 con 0x7f20201096f0 2024-07-26T07:57:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.296+0000 7f2027800640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f20201150a0 con 0x7f202010a770 2024-07-26T07:57:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.296+0000 7f2027800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f20201153a0 con 0x7f202010b110 2024-07-26T07:57:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.296+0000 7f2017e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f202010a770 0x7f2020081250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2024e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20201096f0 0x7f2020080be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2017e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f202010a770 0x7f2020081250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:38736/0 (socket says 172.21.15.97:38736) 2024-07-26T07:57:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2017e00640 1 -- 172.21.15.97:0/1743336328 learned_addr learned my addr 172.21.15.97:0/1743336328 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2025800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202010b110 0x7f2020081e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2017e00640 1 -- 172.21.15.97:0/1743336328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202010b110 msgr2=0x7f2020081e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2017e00640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202010b110 0x7f2020081e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2017e00640 1 -- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20201096f0 msgr2=0x7f2020080be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2017e00640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20201096f0 0x7f2020080be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2017e00640 1 -- 172.21.15.97:0/1743336328 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2020082620 con 0x7f202010a770 2024-07-26T07:57:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.297+0000 7f2024e00640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20201096f0 0x7f2020080be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:57:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.298+0000 7f2017e00640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f202010a770 0x7f2020081250 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2008031460 tx=0x7f2008003bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.299+0000 7f201e000640 1 -- 172.21.15.97:0/1743336328 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200803f070 con 0x7f202010a770 2024-07-26T07:57:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.299+0000 7f201e000640 1 -- 172.21.15.97:0/1743336328 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2008031b90 con 0x7f202010a770 2024-07-26T07:57:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.299+0000 7f2027800640 1 -- 172.21.15.97:0/1743336328 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20200828b0 con 0x7f202010a770 2024-07-26T07:57:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.299+0000 7f201e000640 1 -- 172.21.15.97:0/1743336328 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008035030 con 0x7f202010a770 2024-07-26T07:57:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.299+0000 7f2027800640 1 -- 172.21.15.97:0/1743336328 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2020082bc0 con 0x7f202010a770 2024-07-26T07:57:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.304+0000 7f202e876740 1 Processor -- start 2024-07-26T07:57:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f202e876740 1 -- start start 2024-07-26T07:57:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f202e876740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a40f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f202e876740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 0x5640e3a41570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f202e876740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e367a4d0 0x5640e3a420c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f202e876740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5640e39b3270 con 0x5640e367a4d0 2024-07-26T07:57:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f202e876740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5640e39a2770 con 0x5640e369b1b0 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f202e876740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5640e39a67b0 con 0x5640e36722f0 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f201e000640 1 -- 172.21.15.97:0/1743336328 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f2008042070 con 0x7f202010a770 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f2025800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a40f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f2025800640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a40f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:38750/0 (socket says 172.21.15.97:38750) 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f2025800640 1 -- 172.21.15.97:0/1256753922 learned_addr learned my addr 172.21.15.97:0/1256753922 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.305+0000 7f2024e00640 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 0x5640e3a41570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.306+0000 7f2017e00640 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e367a4d0 0x5640e3a420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.306+0000 7f2025800640 1 -- 172.21.15.97:0/1256753922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 msgr2=0x5640e3a41570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.306+0000 7f2025800640 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 0x5640e3a41570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.306+0000 7f2025800640 1 -- 172.21.15.97:0/1256753922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e367a4d0 msgr2=0x5640e3a420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.306+0000 7f2025800640 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e367a4d0 0x5640e3a420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.306+0000 7f2025800640 1 -- 172.21.15.97:0/1256753922 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e3706790 con 0x5640e369b1b0 2024-07-26T07:57:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.306+0000 7f2017e00640 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e367a4d0 0x5640e3a420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:57:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.306+0000 7f2025800640 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a40f00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2018009ee0 tx=0x7f2018010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f07bf556f287de2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.307+0000 7f2016000640 1 -- 172.21.15.97:0/1256753922 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018016070 con 0x5640e369b1b0 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.307+0000 7f2016000640 1 -- 172.21.15.97:0/1256753922 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2018004d20 con 0x5640e369b1b0 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.307+0000 7f2016000640 1 -- 172.21.15.97:0/1256753922 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20180126b0 con 0x5640e369b1b0 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 -- 172.21.15.97:0/1256753922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 msgr2=0x5640e3a40f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a40f00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2018009ee0 tx=0x7f2018010740 comp rx=0 tx=0).stop 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 -- 172.21.15.97:0/1256753922 shutdown_connections 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e367a4d0 0x5640e3a420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 0x5640e3a41570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 --2- 172.21.15.97:0/1256753922 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a40f00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 -- 172.21.15.97:0/1256753922 >> 172.21.15.97:0/1256753922 conn(0x5640e368f300 msgr2=0x5640e3a3c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 -- 172.21.15.97:0/1256753922 shutdown_connections 2024-07-26T07:57:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.308+0000 7f202e876740 1 -- 172.21.15.97:0/1256753922 wait complete. 2024-07-26T07:57:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.309+0000 7f202e876740 1 -- start start 2024-07-26T07:57:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.309+0000 7f201e000640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20000b3c70 0x7f20000b6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f202e876740 1 Processor -- start 2024-07-26T07:57:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f201e000640 1 -- 172.21.15.97:0/1743336328 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f20081044f0 con 0x7f202010a770 2024-07-26T07:57:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f2025800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20000b3c70 0x7f20000b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f202e876740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 0x5640e3a4aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f202e876740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e367a4d0 0x5640e3a4b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f202e876740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 0x5640e3a4bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f202e876740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5640e38dcab0 con 0x5640e369b1b0 2024-07-26T07:57:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f202e876740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5640e393fd50 con 0x5640e367a4d0 2024-07-26T07:57:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f202e876740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5640e3a42be0 con 0x5640e36722f0 2024-07-26T07:57:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.310+0000 7f202e876740 10 client.0 _ll_register_callbacks cb 0x5640e36b2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.311+0000 7f2017e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e367a4d0 0x5640e3a4b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.311+0000 7f2017e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e367a4d0 0x5640e3a4b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:38754/0 (socket says 172.21.15.97:38754) 2024-07-26T07:57:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.311+0000 7f2017e00640 1 -- 172.21.15.97:0/140261443 learned_addr learned my addr 172.21.15.97:0/140261443 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.311+0000 7f2017e00640 1 -- 172.21.15.97:0/140261443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 msgr2=0x5640e3a4aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.311+0000 7f2017e00640 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 0x5640e3a4aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.311+0000 7f2017e00640 1 -- 172.21.15.97:0/140261443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 msgr2=0x5640e3a4bc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.311+0000 7f2017e00640 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 0x5640e3a4bc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.311+0000 7f2017e00640 1 -- 172.21.15.97:0/140261443 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e377f620 con 0x5640e367a4d0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f2017e00640 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e367a4d0 0x5640e3a4b0e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2008004650 tx=0x7f20080cabb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6adf5da8317fc06e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f200ea00640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f2025800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20000b3c70 0x7f20000b6180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2018005b40 tx=0x7f2018000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f200ea00640 1 -- 172.21.15.97:0/140261443 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008049070 con 0x5640e367a4d0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f200ea00640 1 -- 172.21.15.97:0/140261443 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f200803b460 con 0x5640e367a4d0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f202e876740 1 -- 172.21.15.97:0/140261443 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2018004d20 con 0x5640e367a4d0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f200ea00640 1 -- 172.21.15.97:0/140261443 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f20080cf980 con 0x5640e367a4d0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f202e876740 10 client.35573 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.312+0000 7f202e876740 1 -- 172.21.15.97:0/140261443 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5640e3a54af0 con 0x5640e367a4d0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.313+0000 7f200ea00640 1 -- 172.21.15.97:0/140261443 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2018004d20 con 0x5640e367a4d0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.313+0000 7f200ea00640 1 -- 172.21.15.97:0/140261443 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7f20080cb5e0 con 0x5640e367a4d0 2024-07-26T07:57:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.313+0000 7f202e876740 10 client.35573 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.313+0000 7f202e876740 10 client.35573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:57:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.313+0000 7f202e876740 20 client.35573 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.313+0000 7f202e876740 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5640e3659d80 0x5640e3a58370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.313+0000 7f202e876740 4 client.35573 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:57:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.313+0000 7f202e876740 1 -- 172.21.15.97:0/140261443 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640e39b51c0 con 0x5640e3659d80 2024-07-26T07:57:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.314+0000 7f2024e00640 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5640e3659d80 0x5640e3a58370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.315+0000 7f2024e00640 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5640e3659d80 0x5640e3a58370 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=429e715a55cf1cf2 server_cookie=34da8be89cc52438 in_seq=0 out_seq=0 2024-07-26T07:57:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.315+0000 7f200ea00640 10 client.35573 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.316+0000 7f200ea00640 1 -- 172.21.15.97:0/140261443 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5640e39b51c0 con 0x5640e3659d80 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.316+0000 7f200ea00640 10 client.35573 handle_command_reply: tid=0 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 client.35573 shutdown 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 -- 172.21.15.97:0/140261443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e367a4d0 msgr2=0x5640e3a4b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e367a4d0 0x5640e3a4b0e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2008004650 tx=0x7f20080cabb0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 -- 172.21.15.97:0/140261443 shutdown_connections 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5640e3659d80 0x5640e3a58370 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 0x5640e3a4bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e367a4d0 0x5640e3a4b0e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 --2- 172.21.15.97:0/140261443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e36722f0 0x5640e3a4aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 -- 172.21.15.97:0/140261443 >> 172.21.15.97:0/140261443 conn(0x5640e368f300 msgr2=0x5640e3a3b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f200ea00640 0 client.35573 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.317+0000 7f202e876740 1 -- 172.21.15.97:0/140261443 shutdown_connections 2024-07-26T07:57:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.318+0000 7f202e876740 1 -- 172.21.15.97:0/140261443 wait complete. 2024-07-26T07:57:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.318+0000 7f202e876740 20 client.35573 trim_cache size 0 max 16384 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.340+0000 7f202e876740 1 Processor -- start 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f202e876740 1 -- start start 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f202e876740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e36722f0 0x5640e3a49640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f202e876740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e367a4d0 0x5640e3a52b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f202e876740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 0x5640e3a3c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f202e876740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2020074730 con 0x5640e369b1b0 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f202e876740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f20201c3bd0 con 0x5640e36722f0 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f202e876740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2020074e00 con 0x5640e367a4d0 2024-07-26T07:57:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f2025800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 0x5640e3a3c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f2024e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e36722f0 0x5640e3a49640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f2025800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 0x5640e3a3c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39246/0 (socket says 172.21.15.97:39246) 2024-07-26T07:57:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.341+0000 7f2025800640 1 -- 172.21.15.97:0/158806352 learned_addr learned my addr 172.21.15.97:0/158806352 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.342+0000 7f2017e00640 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e367a4d0 0x5640e3a52b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.342+0000 7f2024e00640 1 -- 172.21.15.97:0/158806352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e367a4d0 msgr2=0x5640e3a52b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.342+0000 7f2024e00640 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e367a4d0 0x5640e3a52b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.342+0000 7f2024e00640 1 -- 172.21.15.97:0/158806352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 msgr2=0x5640e3a3c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.342+0000 7f2024e00640 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 0x5640e3a3c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.342+0000 7f2024e00640 1 -- 172.21.15.97:0/158806352 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e3a3c0e0 con 0x5640e36722f0 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.342+0000 7f2024e00640 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e36722f0 0x5640e3a49640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x5640e3a58ab0 tx=0x7f20100139b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf0f21f1f8c1771f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.343+0000 7f2015600640 1 -- 172.21.15.97:0/158806352 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010004220 con 0x5640e36722f0 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.343+0000 7f2015600640 1 -- 172.21.15.97:0/158806352 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f20100043c0 con 0x5640e36722f0 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.344+0000 7f2015600640 1 -- 172.21.15.97:0/158806352 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010004de0 con 0x5640e36722f0 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.344+0000 7f202e876740 1 -- 172.21.15.97:0/158806352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e36722f0 msgr2=0x5640e3a49640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.344+0000 7f202e876740 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e36722f0 0x5640e3a49640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x5640e3a58ab0 tx=0x7f20100139b0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.344+0000 7f202e876740 1 -- 172.21.15.97:0/158806352 shutdown_connections 2024-07-26T07:57:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.344+0000 7f202e876740 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e369b1b0 0x5640e3a3c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.344+0000 7f202e876740 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e367a4d0 0x5640e3a52b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.344+0000 7f202e876740 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e36722f0 0x5640e3a49640 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.344+0000 7f202e876740 1 -- 172.21.15.97:0/158806352 >> 172.21.15.97:0/158806352 conn(0x5640e368f300 msgr2=0x5640e3a48f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.345+0000 7f2017e00640 1 --2- 172.21.15.97:0/158806352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e367a4d0 0x5640e3a52b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:57:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.345+0000 7f202e876740 1 -- 172.21.15.97:0/158806352 shutdown_connections 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.345+0000 7f202e876740 1 -- 172.21.15.97:0/158806352 wait complete. 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.345+0000 7f202e876740 1 -- start start 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.346+0000 7f202e876740 1 Processor -- start 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.346+0000 7f202e876740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e36722f0 0x5640e3a4f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.346+0000 7f202e876740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a4fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.346+0000 7f202e876740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e362f1c0 0x5640e3a506e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.346+0000 7f202e876740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f20201c34d0 con 0x5640e36722f0 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.346+0000 7f202e876740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f20201c2e00 con 0x5640e369b1b0 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.346+0000 7f202e876740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2020072710 con 0x5640e362f1c0 2024-07-26T07:57:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.346+0000 7f202e876740 10 client.0 _ll_register_callbacks cb 0x7f2008009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.347+0000 7f2024e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e36722f0 0x5640e3a4f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.347+0000 7f2024e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e36722f0 0x5640e3a4f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39252/0 (socket says 172.21.15.97:39252) 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.347+0000 7f2024e00640 1 -- 172.21.15.97:0/2606024328 learned_addr learned my addr 172.21.15.97:0/2606024328 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.347+0000 7f2017e00640 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a4fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.347+0000 7f2025800640 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e362f1c0 0x5640e3a506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.348+0000 7f2024e00640 1 -- 172.21.15.97:0/2606024328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e362f1c0 msgr2=0x5640e3a506e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.348+0000 7f2024e00640 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e362f1c0 0x5640e3a506e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.348+0000 7f2024e00640 1 -- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 msgr2=0x5640e3a4fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.348+0000 7f2024e00640 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a4fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.348+0000 7f2024e00640 1 -- 172.21.15.97:0/2606024328 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e3a50e40 con 0x5640e36722f0 2024-07-26T07:57:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.348+0000 7f2017e00640 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a4fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:57:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.348+0000 7f2024e00640 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e36722f0 0x5640e3a4f500 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f20100097b0 tx=0x7f20100046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44578ee18f6dfe62 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.348+0000 7f200ea00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.349+0000 7f200ea00640 1 -- 172.21.15.97:0/2606024328 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201001f890 con 0x5640e36722f0 2024-07-26T07:57:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.349+0000 7f200ea00640 1 -- 172.21.15.97:0/2606024328 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f201001fa30 con 0x5640e36722f0 2024-07-26T07:57:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.350+0000 7f202e876740 1 -- 172.21.15.97:0/2606024328 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20100043c0 con 0x5640e36722f0 2024-07-26T07:57:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f200ea00640 1 -- 172.21.15.97:0/2606024328 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f201001fc70 con 0x5640e36722f0 2024-07-26T07:57:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f202e876740 10 client.16125 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f202e876740 1 -- 172.21.15.97:0/2606024328 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5640e3a521d0 con 0x5640e36722f0 2024-07-26T07:57:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f200ea00640 1 -- 172.21.15.97:0/2606024328 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f20100043c0 con 0x5640e36722f0 2024-07-26T07:57:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f200ea00640 1 -- 172.21.15.97:0/2606024328 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7f201002f020 con 0x5640e36722f0 2024-07-26T07:57:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f202e876740 10 client.16125 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f202e876740 10 client.16125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:57:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f202e876740 20 client.16125 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f202e876740 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5640e3a52a90 0x5640e3ac2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:09.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f202e876740 4 client.16125 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:57:09.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.351+0000 7f202e876740 1 -- 172.21.15.97:0/2606024328 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640e39b0c60 con 0x5640e3a52a90 2024-07-26T07:57:09.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.352+0000 7f2017e00640 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5640e3a52a90 0x5640e3ac2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:09.354 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.353+0000 7f2017e00640 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5640e3a52a90 0x5640e3ac2b50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcfbbc87f241ecf2 server_cookie=5f1bbbad2f0f70cf in_seq=0 out_seq=0 2024-07-26T07:57:09.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.353+0000 7f200ea00640 10 client.16125 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:09.528 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.527+0000 7f200ea00640 1 -- 172.21.15.97:0/2606024328 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640e39b0c60 con 0x5640e3a52a90 2024-07-26T07:57:09.528 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.527+0000 7f200ea00640 10 client.16125 handle_command_reply: tid=0 2024-07-26T07:57:09.528 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.527+0000 7f202e876740 1 client.16125 shutdown 2024-07-26T07:57:09.529 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.528+0000 7f202e876740 1 -- 172.21.15.97:0/2606024328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e36722f0 msgr2=0x5640e3a4f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.529 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.528+0000 7f202e876740 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e36722f0 0x5640e3a4f500 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f20100097b0 tx=0x7f20100046e0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.530 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.529+0000 7f202e876740 1 -- 172.21.15.97:0/2606024328 shutdown_connections 2024-07-26T07:57:09.530 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.529+0000 7f202e876740 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5640e3a52a90 0x5640e3ac2b50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.530 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.529+0000 7f202e876740 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5640e362f1c0 0x5640e3a506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.530 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.529+0000 7f202e876740 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5640e369b1b0 0x5640e3a4fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.529+0000 7f202e876740 1 --2- 172.21.15.97:0/2606024328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640e36722f0 0x5640e3a4f500 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.529+0000 7f202e876740 1 -- 172.21.15.97:0/2606024328 >> 172.21.15.97:0/2606024328 conn(0x5640e368f300 msgr2=0x5640e3a48b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.532+0000 7f200ea00640 0 client.16125 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.532+0000 7f202e876740 1 -- 172.21.15.97:0/2606024328 shutdown_connections 2024-07-26T07:57:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.532+0000 7f202e876740 1 -- 172.21.15.97:0/2606024328 wait complete. 2024-07-26T07:57:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.532+0000 7f202e876740 20 client.16125 trim_cache size 0 max 16384 2024-07-26T07:57:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:57:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:57:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:57:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:57:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 -- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20000b3c70 msgr2=0x7f20000b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20000b3c70 0x7f20000b6180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2018005b40 tx=0x7f2018000f40 comp rx=0 tx=0).stop 2024-07-26T07:57:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 -- 172.21.15.97:0/1743336328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f202010a770 msgr2=0x7f2020081250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f202010a770 0x7f2020081250 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2008031460 tx=0x7f2008003bb0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 -- 172.21.15.97:0/1743336328 shutdown_connections 2024-07-26T07:57:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f20000b3c70 0x7f20000b6180 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202010b110 0x7f2020081e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f202010a770 0x7f2020081250 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 --2- 172.21.15.97:0/1743336328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20201096f0 0x7f2020080be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.533+0000 7f2027800640 1 -- 172.21.15.97:0/1743336328 >> 172.21.15.97:0/1743336328 conn(0x7f2020078a80 msgr2=0x7f202010da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.534+0000 7f2027800640 1 -- 172.21.15.97:0/1743336328 shutdown_connections 2024-07-26T07:57:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:09.534+0000 7f2027800640 1 -- 172.21.15.97:0/1743336328 wait complete. 2024-07-26T07:57:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:09 smithi097 ceph-mon[32231]: pgmap v672: 97 pgs: 97 active+clean; 111 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 201 op/s 2024-07-26T07:57:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:09 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:57:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:09 smithi115 ceph-mon[35364]: pgmap v672: 97 pgs: 97 active+clean; 111 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 201 op/s 2024-07-26T07:57:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:09 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:57:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:09 smithi181 ceph-mon[35869]: pgmap v672: 97 pgs: 97 active+clean; 111 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 201 op/s 2024-07-26T07:57:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:09 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:57:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:10.100 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '631d9159', total seconds elapsed: 24.6 2024-07-26T07:57:10.202 INFO:tasks.workunit.client.0.smithi097.stdout: 31 7860 5 5657 2 3905 9 2024-07-26T07:57:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:10 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T07:57:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:10 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T07:57:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:57:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:57:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:57:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:10 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T07:57:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:10 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T07:57:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:57:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:57:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:57:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:10 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T07:57:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:10 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T07:57:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:57:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:57:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:57:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:11 smithi115 ceph-mon[35364]: pgmap v673: 97 pgs: 97 active+clean; 111 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-07-26T07:57:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:11 smithi181 ceph-mon[35869]: pgmap v673: 97 pgs: 97 active+clean; 111 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-07-26T07:57:12.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:11 smithi097 ceph-mon[32231]: pgmap v673: 97 pgs: 97 active+clean; 111 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-07-26T07:57:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:13 smithi115 ceph-mon[35364]: pgmap v674: 97 pgs: 97 active+clean; 111 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 657 KiB/s wr, 48 op/s 2024-07-26T07:57:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:13 smithi181 ceph-mon[35869]: pgmap v674: 97 pgs: 97 active+clean; 111 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 657 KiB/s wr, 48 op/s 2024-07-26T07:57:14.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:13 smithi097 ceph-mon[32231]: pgmap v674: 97 pgs: 97 active+clean; 111 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 657 KiB/s wr, 48 op/s 2024-07-26T07:57:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:15 smithi115 ceph-mon[35364]: pgmap v675: 97 pgs: 97 active+clean; 113 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 69 op/s 2024-07-26T07:57:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:15 smithi181 ceph-mon[35869]: pgmap v675: 97 pgs: 97 active+clean; 113 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 69 op/s 2024-07-26T07:57:16.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:15 smithi097 ceph-mon[32231]: pgmap v675: 97 pgs: 97 active+clean; 113 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 69 op/s 2024-07-26T07:57:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:17 smithi115 ceph-mon[35364]: pgmap v676: 97 pgs: 97 active+clean; 113 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-07-26T07:57:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:17 smithi181 ceph-mon[35869]: pgmap v676: 97 pgs: 97 active+clean; 113 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-07-26T07:57:18.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:17 smithi097 ceph-mon[32231]: pgmap v676: 97 pgs: 97 active+clean; 113 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-07-26T07:57:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:19 smithi115 ceph-mon[35364]: pgmap v677: 97 pgs: 97 active+clean; 113 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 87 op/s 2024-07-26T07:57:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:19 smithi181 ceph-mon[35869]: pgmap v677: 97 pgs: 97 active+clean; 113 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 87 op/s 2024-07-26T07:57:20.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:19 smithi097 ceph-mon[32231]: pgmap v677: 97 pgs: 97 active+clean; 113 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 87 op/s 2024-07-26T07:57:20.202 INFO:tasks.workunit.client.0.smithi097.stdout: 31 8167 42 5386 140 3312 130 2024-07-26T07:57:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:21 smithi115 ceph-mon[35364]: pgmap v678: 97 pgs: 97 active+clean; 115 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-07-26T07:57:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:21 smithi181 ceph-mon[35869]: pgmap v678: 97 pgs: 97 active+clean; 115 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-07-26T07:57:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:21 smithi097 ceph-mon[32231]: pgmap v678: 97 pgs: 97 active+clean; 115 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-07-26T07:57:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.830+0000 7f073ce00640 1 -- 172.21.15.97:0/833466938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0738108cb0 msgr2=0x7f073810b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.830+0000 7f073ce00640 1 --2- 172.21.15.97:0/833466938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0738108cb0 0x7f073810b1c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0730009ca0 tx=0x7f073002a8c0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.830+0000 7f073ce00640 1 -- 172.21.15.97:0/833466938 shutdown_connections 2024-07-26T07:57:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.830+0000 7f073ce00640 1 --2- 172.21.15.97:0/833466938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0738108cb0 0x7f073810b1c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.830+0000 7f073ce00640 1 --2- 172.21.15.97:0/833466938 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f07381060f0 0x7f0738108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.830+0000 7f073ce00640 1 --2- 172.21.15.97:0/833466938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f073806b140 0x7f0738105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.830+0000 7f073ce00640 1 -- 172.21.15.97:0/833466938 >> 172.21.15.97:0/833466938 conn(0x7f07380fd7b0 msgr2=0x7f07380ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.831+0000 7f073ce00640 1 -- 172.21.15.97:0/833466938 shutdown_connections 2024-07-26T07:57:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.831+0000 7f073ce00640 1 -- 172.21.15.97:0/833466938 wait complete. 2024-07-26T07:57:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.832+0000 7f073ce00640 1 Processor -- start 2024-07-26T07:57:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.832+0000 7f073ce00640 1 -- start start 2024-07-26T07:57:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.833+0000 7f073ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f073806b140 0x7f0738199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.833+0000 7f073ce00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07381060f0 0x7f0738199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.833+0000 7f073ce00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0738108cb0 0x7f073819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.833+0000 7f073ce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f073810ea60 con 0x7f073806b140 2024-07-26T07:57:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.833+0000 7f073ce00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f073810e8e0 con 0x7f0738108cb0 2024-07-26T07:57:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.833+0000 7f073ce00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f073810ebe0 con 0x7f07381060f0 2024-07-26T07:57:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.834+0000 7f0737e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0738108cb0 0x7f073819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.834+0000 7f0737e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0738108cb0 0x7f073819a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:51186/0 (socket says 172.21.15.97:51186) 2024-07-26T07:57:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.834+0000 7f0737e00640 1 -- 172.21.15.97:0/3014054711 learned_addr learned my addr 172.21.15.97:0/3014054711 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.834+0000 7f0737400640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f073806b140 0x7f0738199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.834+0000 7f0736a00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07381060f0 0x7f0738199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.835+0000 7f0737e00640 1 -- 172.21.15.97:0/3014054711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07381060f0 msgr2=0x7f0738199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.835+0000 7f0737e00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07381060f0 0x7f0738199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.835+0000 7f0737e00640 1 -- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f073806b140 msgr2=0x7f0738199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.835+0000 7f0737e00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f073806b140 0x7f0738199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.835+0000 7f0737e00640 1 -- 172.21.15.97:0/3014054711 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073819b0d0 con 0x7f0738108cb0 2024-07-26T07:57:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.835+0000 7f0737400640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f073806b140 0x7f0738199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:57:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.835+0000 7f0737e00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0738108cb0 0x7f073819a940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0730031040 tx=0x7f0730003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.836+0000 7f072fe00640 1 -- 172.21.15.97:0/3014054711 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0730004040 con 0x7f0738108cb0 2024-07-26T07:57:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.836+0000 7f072fe00640 1 -- 172.21.15.97:0/3014054711 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0730004670 con 0x7f0738108cb0 2024-07-26T07:57:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.836+0000 7f073ce00640 1 -- 172.21.15.97:0/3014054711 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073819b360 con 0x7f0738108cb0 2024-07-26T07:57:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.836+0000 7f072fe00640 1 -- 172.21.15.97:0/3014054711 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0730030ce0 con 0x7f0738108cb0 2024-07-26T07:57:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.836+0000 7f073ce00640 1 -- 172.21.15.97:0/3014054711 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0738101990 con 0x7f0738108cb0 2024-07-26T07:57:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.841+0000 7f0745434740 1 Processor -- start 2024-07-26T07:57:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.842+0000 7f072fe00640 1 -- 172.21.15.97:0/3014054711 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f07300041e0 con 0x7f0738108cb0 2024-07-26T07:57:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.842+0000 7f0745434740 1 -- start start 2024-07-26T07:57:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0745434740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563965a081b0 0x563965dae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0745434740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659df2f0 0x563965dae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0745434740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659e74d0 0x563965daf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0745434740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x563965d20270 con 0x5639659df2f0 2024-07-26T07:57:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0745434740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563965d0f770 con 0x563965a081b0 2024-07-26T07:57:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0745434740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563965d137b0 con 0x5639659e74d0 2024-07-26T07:57:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0736a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563965a081b0 0x563965dae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0736a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563965a081b0 0x563965dae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:51190/0 (socket says 172.21.15.97:51190) 2024-07-26T07:57:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0736a00640 1 -- 172.21.15.97:0/2006482088 learned_addr learned my addr 172.21.15.97:0/2006482088 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.843+0000 7f0737400640 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659df2f0 0x563965dae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f0736a00640 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659e74d0 0x563965daf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f0736a00640 1 -- 172.21.15.97:0/2006482088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659e74d0 msgr2=0x563965daf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f0736a00640 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659e74d0 0x563965daf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f0736a00640 1 -- 172.21.15.97:0/2006482088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659df2f0 msgr2=0x563965dae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f072fe00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f071c0b3bd0 0x7f071c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f072fe00640 1 -- 172.21.15.97:0/3014054711 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f07301053b0 con 0x7f0738108cb0 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f0736a00640 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659df2f0 0x563965dae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f0736a00640 1 -- 172.21.15.97:0/2006482088 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563965a73790 con 0x563965a081b0 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.844+0000 7f0737400640 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659df2f0 0x563965dae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.845+0000 7f0736a00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f071c0b3bd0 0x7f071c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.845+0000 7f0736a00640 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563965a081b0 0x563965dae0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f072400af90 tx=0x7f072400a790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8e74e1c8fd898b7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.846+0000 7f072cc00640 1 -- 172.21.15.97:0/2006482088 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724023070 con 0x563965a081b0 2024-07-26T07:57:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.846+0000 7f072cc00640 1 -- 172.21.15.97:0/2006482088 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f072401c3d0 con 0x563965a081b0 2024-07-26T07:57:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.846+0000 7f072cc00640 1 -- 172.21.15.97:0/2006482088 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072401b370 con 0x563965a081b0 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 -- 172.21.15.97:0/2006482088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563965a081b0 msgr2=0x563965dae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563965a081b0 0x563965dae0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f072400af90 tx=0x7f072400a790 comp rx=0 tx=0).stop 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0736a00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f071c0b3bd0 0x7f071c0b60e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0724020000 tx=0x7f072401d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 -- 172.21.15.97:0/2006482088 shutdown_connections 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659e74d0 0x563965daf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659df2f0 0x563965dae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 --2- 172.21.15.97:0/2006482088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x563965a081b0 0x563965dae0c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 -- 172.21.15.97:0/2006482088 >> 172.21.15.97:0/2006482088 conn(0x5639659fc300 msgr2=0x563965da9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 -- 172.21.15.97:0/2006482088 shutdown_connections 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.847+0000 7f0745434740 1 -- 172.21.15.97:0/2006482088 wait complete. 2024-07-26T07:57:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.848+0000 7f0745434740 1 -- start start 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.848+0000 7f0745434740 1 Processor -- start 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0745434740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 0x563965db7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0745434740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965db8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0745434740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563965a081b0 0x563965db8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0745434740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x563965c49ab0 con 0x5639659e74d0 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0745434740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x563965cacd50 con 0x5639659df2f0 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0745434740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563965dafda0 con 0x563965a081b0 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0745434740 10 client.0 _ll_register_callbacks cb 0x563965a1fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0736a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965db8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0736a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965db8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46250/0 (socket says 172.21.15.97:46250) 2024-07-26T07:57:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.849+0000 7f0736a00640 1 -- 172.21.15.97:0/2250930260 learned_addr learned my addr 172.21.15.97:0/2250930260 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.850+0000 7f0736a00640 1 -- 172.21.15.97:0/2250930260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563965a081b0 msgr2=0x563965db8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.850+0000 7f0736a00640 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563965a081b0 0x563965db8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.850+0000 7f0736a00640 1 -- 172.21.15.97:0/2250930260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 msgr2=0x563965db7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.850+0000 7f0736a00640 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 0x563965db7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.850+0000 7f0736a00640 1 -- 172.21.15.97:0/2250930260 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563965aec620 con 0x5639659e74d0 2024-07-26T07:57:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.851+0000 7f0736a00640 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965db8290 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f072401ca10 tx=0x7f072401ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b84c7d766a568ac server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.851+0000 7f0721600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.851+0000 7f0721600640 1 -- 172.21.15.97:0/2250930260 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724035060 con 0x5639659e74d0 2024-07-26T07:57:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.851+0000 7f0745434740 1 -- 172.21.15.97:0/2250930260 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f072401c3d0 con 0x5639659e74d0 2024-07-26T07:57:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.852+0000 7f0721600640 1 -- 172.21.15.97:0/2250930260 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f072401dcd0 con 0x5639659e74d0 2024-07-26T07:57:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.852+0000 7f0721600640 1 -- 172.21.15.97:0/2250930260 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0724023050 con 0x5639659e74d0 2024-07-26T07:57:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.852+0000 7f0745434740 10 client.16137 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.852+0000 7f0745434740 1 -- 172.21.15.97:0/2250930260 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563965dc1ce0 con 0x5639659e74d0 2024-07-26T07:57:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.853+0000 7f0721600640 1 -- 172.21.15.97:0/2250930260 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f072401c3d0 con 0x5639659e74d0 2024-07-26T07:57:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.853+0000 7f0721600640 1 -- 172.21.15.97:0/2250930260 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7f0724039030 con 0x5639659e74d0 2024-07-26T07:57:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.853+0000 7f0745434740 10 client.16137 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.853+0000 7f0745434740 10 client.16137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:57:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.853+0000 7f0745434740 20 client.16137 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.853+0000 7f0745434740 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5639659c6d80 0x563965dc53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.853+0000 7f0745434740 4 client.16137 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:57:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.853+0000 7f0745434740 1 -- 172.21.15.97:0/2250930260 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563965d221c0 con 0x5639659c6d80 2024-07-26T07:57:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.854+0000 7f0737400640 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5639659c6d80 0x563965dc53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.855+0000 7f0721600640 10 client.16137 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.855+0000 7f0737400640 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5639659c6d80 0x563965dc53a0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bed4ede81f7134d server_cookie=1c82f04bbed0b8de in_seq=0 out_seq=0 2024-07-26T07:57:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.858+0000 7f0721600640 1 -- 172.21.15.97:0/2250930260 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563965d221c0 con 0x5639659c6d80 2024-07-26T07:57:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.858+0000 7f0721600640 10 client.16137 handle_command_reply: tid=0 2024-07-26T07:57:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.858+0000 7f0745434740 1 client.16137 shutdown 2024-07-26T07:57:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.858+0000 7f0745434740 1 -- 172.21.15.97:0/2250930260 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 msgr2=0x563965db8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965db8290 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f072401ca10 tx=0x7f072401ca40 comp rx=0 tx=0).stop 2024-07-26T07:57:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 -- 172.21.15.97:0/2250930260 shutdown_connections 2024-07-26T07:57:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5639659c6d80 0x563965dc53a0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563965a081b0 0x563965db8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965db8290 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 --2- 172.21.15.97:0/2250930260 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 0x563965db7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0721600640 0 client.16137 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 -- 172.21.15.97:0/2250930260 >> 172.21.15.97:0/2250930260 conn(0x5639659fc300 msgr2=0x563965da80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 -- 172.21.15.97:0/2250930260 shutdown_connections 2024-07-26T07:57:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 1 -- 172.21.15.97:0/2250930260 wait complete. 2024-07-26T07:57:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.859+0000 7f0745434740 20 client.16137 trim_cache size 0 max 16384 2024-07-26T07:57:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.885+0000 7f0745434740 1 Processor -- start 2024-07-26T07:57:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.885+0000 7f0745434740 1 -- start start 2024-07-26T07:57:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0745434740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659fc300 0x563965da88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0745434740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 0x563965dbfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0745434740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965da8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0745434740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f07381b0e30 con 0x5639659e74d0 2024-07-26T07:57:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0745434740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f07381b29a0 con 0x5639659df2f0 2024-07-26T07:57:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0745434740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f07381b1500 con 0x5639659fc300 2024-07-26T07:57:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0737e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965da8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0737e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965da8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46254/0 (socket says 172.21.15.97:46254) 2024-07-26T07:57:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0737e00640 1 -- 172.21.15.97:0/121574769 learned_addr learned my addr 172.21.15.97:0/121574769 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0737400640 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659fc300 0x563965da88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.886+0000 7f0736a00640 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 0x563965dbfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.887+0000 7f0737e00640 1 -- 172.21.15.97:0/121574769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659fc300 msgr2=0x563965da88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.887+0000 7f0737e00640 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659fc300 0x563965da88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.887+0000 7f0737e00640 1 -- 172.21.15.97:0/121574769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 msgr2=0x563965dbfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.887+0000 7f0737e00640 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 0x563965dbfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.887+0000 7f0737e00640 1 -- 172.21.15.97:0/121574769 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563965db7f40 con 0x5639659e74d0 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.887+0000 7f0737e00640 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965da8e00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f07300cc1d0 tx=0x7f07300c81f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a53241807968db server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.888+0000 7f0723e00640 1 -- 172.21.15.97:0/121574769 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07300e6070 con 0x5639659e74d0 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.888+0000 7f0723e00640 1 -- 172.21.15.97:0/121574769 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0730036030 con 0x5639659e74d0 2024-07-26T07:57:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.888+0000 7f0723e00640 1 -- 172.21.15.97:0/121574769 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07300cc240 con 0x5639659e74d0 2024-07-26T07:57:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.889+0000 7f0745434740 1 -- 172.21.15.97:0/121574769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 msgr2=0x563965da8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.889+0000 7f0745434740 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965da8e00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f07300cc1d0 tx=0x7f07300c81f0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.889+0000 7f0745434740 1 -- 172.21.15.97:0/121574769 shutdown_connections 2024-07-26T07:57:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.889+0000 7f0745434740 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965da8e00 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.889+0000 7f0745434740 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659df2f0 0x563965dbfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.889+0000 7f0745434740 1 --2- 172.21.15.97:0/121574769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659fc300 0x563965da88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.889+0000 7f0745434740 1 -- 172.21.15.97:0/121574769 >> 172.21.15.97:0/121574769 conn(0x563965dae990 msgr2=0x563965daf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.890+0000 7f0745434740 1 -- 172.21.15.97:0/121574769 shutdown_connections 2024-07-26T07:57:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.890+0000 7f0745434740 1 -- 172.21.15.97:0/121574769 wait complete. 2024-07-26T07:57:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.890+0000 7f0745434740 1 -- start start 2024-07-26T07:57:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.890+0000 7f0745434740 1 Processor -- start 2024-07-26T07:57:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.891+0000 7f0745434740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659df2f0 0x563965dbcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0745434740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965dbd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0745434740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659fc300 0x563965dbdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0745434740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f07381b22a0 con 0x5639659e74d0 2024-07-26T07:57:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0745434740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f07381b1bd0 con 0x5639659fc300 2024-07-26T07:57:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0745434740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f07381a53e0 con 0x5639659df2f0 2024-07-26T07:57:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0745434740 10 client.0 _ll_register_callbacks cb 0x7f0724021070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0736a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965dbd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0736a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965dbd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46258/0 (socket says 172.21.15.97:46258) 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0736a00640 1 -- 172.21.15.97:0/1456342999 learned_addr learned my addr 172.21.15.97:0/1456342999 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0737400640 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659df2f0 0x563965dbcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0737e00640 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659fc300 0x563965dbdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0736a00640 1 -- 172.21.15.97:0/1456342999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659df2f0 msgr2=0x563965dbcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.892+0000 7f0736a00640 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659df2f0 0x563965dbcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0736a00640 1 -- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659fc300 msgr2=0x563965dbdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0736a00640 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659fc300 0x563965dbdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0736a00640 1 -- 172.21.15.97:0/1456342999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563965dbe740 con 0x5639659e74d0 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0737e00640 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659fc300 0x563965dbdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0736a00640 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965dbd470 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f072400a5c0 tx=0x7f072400a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92a74049fa69183 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0721600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0721600640 1 -- 172.21.15.97:0/1456342999 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724035040 con 0x5639659e74d0 2024-07-26T07:57:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0721600640 1 -- 172.21.15.97:0/1456342999 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f072401dad0 con 0x5639659e74d0 2024-07-26T07:57:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.893+0000 7f0745434740 1 -- 172.21.15.97:0/1456342999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0730036030 con 0x5639659e74d0 2024-07-26T07:57:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.894+0000 7f0721600640 1 -- 172.21.15.97:0/1456342999 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f072401c3e0 con 0x5639659e74d0 2024-07-26T07:57:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.894+0000 7f0745434740 10 client.16143 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.894+0000 7f0745434740 1 -- 172.21.15.97:0/1456342999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563965dbfa60 con 0x5639659e74d0 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0721600640 1 -- 172.21.15.97:0/1456342999 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0730036030 con 0x5639659e74d0 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0721600640 1 -- 172.21.15.97:0/1456342999 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7f0724039050 con 0x5639659e74d0 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0745434740 10 client.16143 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0745434740 10 client.16143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0745434740 20 client.16143 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0745434740 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x563965a081b0 0x563965e2fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0745434740 4 client.16143 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0745434740 1 -- 172.21.15.97:0/1456342999 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563965d1dc60 con 0x563965a081b0 2024-07-26T07:57:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.895+0000 7f0737400640 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x563965a081b0 0x563965e2fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.896+0000 7f0737400640 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x563965a081b0 0x563965e2fdf0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2feede831724873 server_cookie=f0abb4be6f238392 in_seq=0 out_seq=0 2024-07-26T07:57:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:22.896+0000 7f0721600640 10 client.16143 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.608+0000 7f0721600640 1 -- 172.21.15.97:0/1456342999 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563965d1dc60 con 0x563965a081b0 2024-07-26T07:57:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.608+0000 7f0721600640 10 client.16143 handle_command_reply: tid=0 2024-07-26T07:57:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.609+0000 7f0745434740 1 client.16143 shutdown 2024-07-26T07:57:23.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.609+0000 7f0745434740 1 -- 172.21.15.97:0/1456342999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 msgr2=0x563965dbd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.609+0000 7f0745434740 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965dbd470 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f072400a5c0 tx=0x7f072400a6e0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.609+0000 7f0745434740 1 -- 172.21.15.97:0/1456342999 shutdown_connections 2024-07-26T07:57:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.609+0000 7f0745434740 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x563965a081b0 0x563965e2fdf0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.610+0000 7f0745434740 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5639659fc300 0x563965dbdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.610+0000 7f0745434740 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5639659e74d0 0x563965dbd470 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.610+0000 7f0745434740 1 --2- 172.21.15.97:0/1456342999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5639659df2f0 0x563965dbcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.610+0000 7f0721600640 0 client.16143 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.610+0000 7f0745434740 1 -- 172.21.15.97:0/1456342999 >> 172.21.15.97:0/1456342999 conn(0x563965dae990 msgr2=0x563965da8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.610+0000 7f0745434740 1 -- 172.21.15.97:0/1456342999 shutdown_connections 2024-07-26T07:57:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.610+0000 7f0745434740 1 -- 172.21.15.97:0/1456342999 wait complete. 2024-07-26T07:57:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.610+0000 7f0745434740 20 client.16143 trim_cache size 0 max 16384 2024-07-26T07:57:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:57:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:57:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:57:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:57:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 -- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f071c0b3bd0 msgr2=0x7f071c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f071c0b3bd0 0x7f071c0b60e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0724020000 tx=0x7f072401d000 comp rx=0 tx=0).stop 2024-07-26T07:57:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 -- 172.21.15.97:0/3014054711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0738108cb0 msgr2=0x7f073819a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0738108cb0 0x7f073819a940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0730031040 tx=0x7f0730003ba0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 -- 172.21.15.97:0/3014054711 shutdown_connections 2024-07-26T07:57:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f071c0b3bd0 0x7f071c0b60e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0738108cb0 0x7f073819a940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07381060f0 0x7f0738199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 --2- 172.21.15.97:0/3014054711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f073806b140 0x7f0738199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 -- 172.21.15.97:0/3014054711 >> 172.21.15.97:0/3014054711 conn(0x7f07380fd7b0 msgr2=0x7f07381046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 -- 172.21.15.97:0/3014054711 shutdown_connections 2024-07-26T07:57:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:23.612+0000 7f073ce00640 1 -- 172.21.15.97:0/3014054711 wait complete. 2024-07-26T07:57:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:23 smithi115 ceph-mon[35364]: pgmap v679: 97 pgs: 97 active+clean; 118 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-07-26T07:57:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:23 smithi181 ceph-mon[35869]: pgmap v679: 97 pgs: 97 active+clean; 118 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-07-26T07:57:24.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:23 smithi097 ceph-mon[32231]: pgmap v679: 97 pgs: 97 active+clean; 118 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-07-26T07:57:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:25 smithi115 ceph-mon[35364]: pgmap v680: 97 pgs: 97 active+clean; 118 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 185 op/s 2024-07-26T07:57:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:25 smithi181 ceph-mon[35869]: pgmap v680: 97 pgs: 97 active+clean; 118 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 185 op/s 2024-07-26T07:57:26.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:25 smithi097 ceph-mon[32231]: pgmap v680: 97 pgs: 97 active+clean; 118 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 185 op/s 2024-07-26T07:57:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:27 smithi115 ceph-mon[35364]: pgmap v681: 97 pgs: 97 active+clean; 117 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 213 op/s 2024-07-26T07:57:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:27 smithi181 ceph-mon[35869]: pgmap v681: 97 pgs: 97 active+clean; 117 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 213 op/s 2024-07-26T07:57:28.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:27 smithi097 ceph-mon[32231]: pgmap v681: 97 pgs: 97 active+clean; 117 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 213 op/s 2024-07-26T07:57:28.672 INFO:tasks.quiescer.fs.[cephfs]:Total duration (43.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-07-26T07:57:28.672 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.95 2024-07-26T07:57:28.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:57:29.172 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:57:29.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:29 smithi097 ceph-mon[32231]: pgmap v682: 97 pgs: 97 active+clean; 113 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.5 MiB/s wr, 203 op/s 2024-07-26T07:57:29.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:29 smithi115 ceph-mon[35364]: pgmap v682: 97 pgs: 97 active+clean; 113 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.5 MiB/s wr, 203 op/s 2024-07-26T07:57:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:29 smithi181 ceph-mon[35869]: pgmap v682: 97 pgs: 97 active+clean; 113 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.5 MiB/s wr, 203 op/s 2024-07-26T07:57:30.203 INFO:tasks.workunit.client.0.smithi097.stdout: 34 7440 147 5144 112 3518 160 2024-07-26T07:57:31.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.626+0000 7f94f7200640 1 -- 172.21.15.97:0/1524909462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077cc0 msgr2=0x7f94f8113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:31.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.626+0000 7f94f7200640 1 --2- 172.21.15.97:0/1524909462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077cc0 0x7f94f8113a70 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f94e8009900 tx=0x7f94e802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:57:31.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.627+0000 7f94f7200640 1 -- 172.21.15.97:0/1524909462 shutdown_connections 2024-07-26T07:57:31.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.627+0000 7f94f7200640 1 --2- 172.21.15.97:0/1524909462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f8113fb0 0x7f94f81163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:31.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.627+0000 7f94f7200640 1 --2- 172.21.15.97:0/1524909462 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077cc0 0x7f94f8113a70 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:31.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.627+0000 7f94f7200640 1 --2- 172.21.15.97:0/1524909462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94f8077370 0x7f94f8077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:31.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.627+0000 7f94f7200640 1 -- 172.21.15.97:0/1524909462 >> 172.21.15.97:0/1524909462 conn(0x7f94f8101060 msgr2=0x7f94f81034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:31.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.629+0000 7f94f7200640 1 -- 172.21.15.97:0/1524909462 shutdown_connections 2024-07-26T07:57:31.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.629+0000 7f94f7200640 1 -- 172.21.15.97:0/1524909462 wait complete. 2024-07-26T07:57:31.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.631+0000 7f94f7200640 1 Processor -- start 2024-07-26T07:57:31.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.631+0000 7f94f7200640 1 -- start start 2024-07-26T07:57:31.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.632+0000 7f94f7200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077370 0x7f94f81a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:31.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.632+0000 7f94f7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f8077cc0 0x7f94f81a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:31.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.632+0000 7f94f7200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94f8113fb0 0x7f94f81a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:31.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.632+0000 7f94f7200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f94f8119810 con 0x7f94f8077370 2024-07-26T07:57:31.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.632+0000 7f94f5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077370 0x7f94f81a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:31.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.632+0000 7f94f5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077370 0x7f94f81a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60566/0 (socket says 172.21.15.97:60566) 2024-07-26T07:57:31.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.632+0000 7f94f5e00640 1 -- 172.21.15.97:0/2808656857 learned_addr learned my addr 172.21.15.97:0/2808656857 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:31.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.632+0000 7f94f5400640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f8077cc0 0x7f94f81a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.633+0000 7f94f6800640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94f8113fb0 0x7f94f81a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.633+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f94f8119690 con 0x7f94f8113fb0 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.633+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94f8119990 con 0x7f94f8077cc0 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.633+0000 7f94f5e00640 1 -- 172.21.15.97:0/2808656857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f8077cc0 msgr2=0x7f94f81a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.633+0000 7f94f5e00640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f8077cc0 0x7f94f81a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.633+0000 7f94f5e00640 1 -- 172.21.15.97:0/2808656857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94f8113fb0 msgr2=0x7f94f81a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.633+0000 7f94f5e00640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94f8113fb0 0x7f94f81a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.633+0000 7f94f5e00640 1 -- 172.21.15.97:0/2808656857 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f81a61d0 con 0x7f94f8077370 2024-07-26T07:57:31.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.634+0000 7f94f5e00640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077370 0x7f94f81a0260 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f94e000e9f0 tx=0x7f94e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:31.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.635+0000 7f94eea00640 1 -- 172.21.15.97:0/2808656857 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e000cd80 con 0x7f94f8077370 2024-07-26T07:57:31.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.635+0000 7f94eea00640 1 -- 172.21.15.97:0/2808656857 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94e00108c0 con 0x7f94f8077370 2024-07-26T07:57:31.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.635+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f81a64c0 con 0x7f94f8077370 2024-07-26T07:57:31.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.635+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f81acc20 con 0x7f94f8077370 2024-07-26T07:57:31.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.635+0000 7f94eea00640 1 -- 172.21.15.97:0/2808656857 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e00187c0 con 0x7f94f8077370 2024-07-26T07:57:31.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.636+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d40052c0 con 0x7f94f8077370 2024-07-26T07:57:31.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.641+0000 7f94eea00640 1 -- 172.21.15.97:0/2808656857 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f94e0010430 con 0x7f94f8077370 2024-07-26T07:57:31.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.643+0000 7f94eea00640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f94dc0b3da0 0x7f94dc0b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:31.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.644+0000 7f94eea00640 1 -- 172.21.15.97:0/2808656857 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f94e00e41b0 con 0x7f94f8077370 2024-07-26T07:57:31.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.644+0000 7f94f5400640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f94dc0b3da0 0x7f94dc0b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:31.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.645+0000 7f94eea00640 1 -- 172.21.15.97:0/2808656857 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f94e00ab170 con 0x7f94f8077370 2024-07-26T07:57:31.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.645+0000 7f94f5400640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f94dc0b3da0 0x7f94dc0b62b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f94e802a910 tx=0x7f94e8005a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:31 smithi115 ceph-mon[35364]: pgmap v683: 97 pgs: 97 active+clean; 119 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 226 op/s 2024-07-26T07:57:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:31 smithi181 ceph-mon[35869]: pgmap v683: 97 pgs: 97 active+clean; 119 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 226 op/s 2024-07-26T07:57:31.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.997+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94d40055b0 con 0x7f94f8077370 2024-07-26T07:57:31.998 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:31 smithi097 ceph-mon[32231]: pgmap v683: 97 pgs: 97 active+clean; 119 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 226 op/s 2024-07-26T07:57:31.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:31.998+0000 7f94eea00640 1 -- 172.21.15.97:0/2808656857 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11992 (secure 0 0 0) 0x7f94e00af6c0 con 0x7f94f8077370 2024-07-26T07:57:32.000 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:57:32.000 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":34,"btime":"2024-07-26T07:57:08:526790+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/4227989530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4227989530},{"type":"v1","addr":"172.21.15.97:6837","nonce":4227989530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:57:08.310841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3957930666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3957930666},{"type":"v1","addr":"172.21.15.181:6837","nonce":3957930666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:57:32.000 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 34 2024-07-26T07:57:32.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.001+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f94dc0b3da0 msgr2=0x7f94dc0b62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:32.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.001+0000 7f94f7200640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f94dc0b3da0 0x7f94dc0b62b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f94e802a910 tx=0x7f94e8005a40 comp rx=0 tx=0).stop 2024-07-26T07:57:32.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.001+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077370 msgr2=0x7f94f81a0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:32.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.001+0000 7f94f7200640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077370 0x7f94f81a0260 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f94e000e9f0 tx=0x7f94e000eeb0 comp rx=0 tx=0).stop 2024-07-26T07:57:32.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.002+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 shutdown_connections 2024-07-26T07:57:32.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.002+0000 7f94f7200640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f94dc0b3da0 0x7f94dc0b62b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:32.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.002+0000 7f94f7200640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94f8113fb0 0x7f94f81a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:32.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.002+0000 7f94f7200640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f8077cc0 0x7f94f81a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:32.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.002+0000 7f94f7200640 1 --2- 172.21.15.97:0/2808656857 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94f8077370 0x7f94f81a0260 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:32.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.002+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 >> 172.21.15.97:0/2808656857 conn(0x7f94f8101060 msgr2=0x7f94f81160f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:32.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.002+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 shutdown_connections 2024-07-26T07:57:32.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:32.002+0000 7f94f7200640 1 -- 172.21.15.97:0/2808656857 wait complete. 2024-07-26T07:57:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:32 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/2808656857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:32 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/2808656857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:33.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:32 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/2808656857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:33.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:33 smithi097 ceph-mon[32231]: pgmap v684: 97 pgs: 97 active+clean; 119 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-07-26T07:57:33.763 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db / --timeout 90 --expiration 141.95 --await-for 45.0' 2024-07-26T07:57:33.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db / --timeout 90 --expiration 141.95 --await-for 45.0 2024-07-26T07:57:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:33 smithi115 ceph-mon[35364]: pgmap v684: 97 pgs: 97 active+clean; 119 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-07-26T07:57:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:33 smithi181 ceph-mon[35869]: pgmap v684: 97 pgs: 97 active+clean; 119 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-07-26T07:57:34.312 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:57:35.676 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:35 smithi097 ceph-mon[32231]: pgmap v685: 97 pgs: 97 active+clean; 129 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 215 op/s 2024-07-26T07:57:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:35 smithi115 ceph-mon[35364]: pgmap v685: 97 pgs: 97 active+clean; 129 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 215 op/s 2024-07-26T07:57:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:35 smithi181 ceph-mon[35869]: pgmap v685: 97 pgs: 97 active+clean; 129 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 215 op/s 2024-07-26T07:57:36.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.560+0000 7fed36600640 1 -- 172.21.15.97:0/263256915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30108ee0 msgr2=0x7fed301092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.560+0000 7fed36600640 1 --2- 172.21.15.97:0/263256915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30108ee0 0x7fed301092f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fed18009900 tx=0x7fed1802a410 comp rx=0 tx=0).stop 2024-07-26T07:57:36.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.561+0000 7fed36600640 1 -- 172.21.15.97:0/263256915 shutdown_connections 2024-07-26T07:57:36.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.561+0000 7fed36600640 1 --2- 172.21.15.97:0/263256915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed30109830 0x7fed30110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.561+0000 7fed36600640 1 --2- 172.21.15.97:0/263256915 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30108ee0 0x7fed301092f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.561+0000 7fed36600640 1 --2- 172.21.15.97:0/263256915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed301035a0 0x7fed301039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.561+0000 7fed36600640 1 -- 172.21.15.97:0/263256915 >> 172.21.15.97:0/263256915 conn(0x7fed300fefd0 msgr2=0x7fed30101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:36.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.561+0000 7fed36600640 1 -- 172.21.15.97:0/263256915 shutdown_connections 2024-07-26T07:57:36.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.561+0000 7fed36600640 1 -- 172.21.15.97:0/263256915 wait complete. 2024-07-26T07:57:36.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.562+0000 7fed36600640 1 Processor -- start 2024-07-26T07:57:36.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.562+0000 7fed36600640 1 -- start start 2024-07-26T07:57:36.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.562+0000 7fed36600640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed301035a0 0x7fed30075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.562+0000 7fed36600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed30108ee0 0x7fed30075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed36600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30109830 0x7fed30075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed36600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fed301133a0 con 0x7fed30109830 2024-07-26T07:57:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed36600640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fed30113220 con 0x7fed301035a0 2024-07-26T07:57:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed36600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fed30113520 con 0x7fed30108ee0 2024-07-26T07:57:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed2f400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed301035a0 0x7fed30075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed2fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30109830 0x7fed30075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed2f400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed301035a0 0x7fed30075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:49726/0 (socket says 172.21.15.97:49726) 2024-07-26T07:57:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed2f400640 1 -- 172.21.15.97:0/3366040671 learned_addr learned my addr 172.21.15.97:0/3366040671 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.563+0000 7fed27e00640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed30108ee0 0x7fed30075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.564+0000 7fed2fe00640 1 -- 172.21.15.97:0/3366040671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed30108ee0 msgr2=0x7fed30075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.564+0000 7fed2fe00640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed30108ee0 0x7fed30075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.564+0000 7fed2fe00640 1 -- 172.21.15.97:0/3366040671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed301035a0 msgr2=0x7fed30075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.564+0000 7fed2fe00640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed301035a0 0x7fed30075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.564+0000 7fed2fe00640 1 -- 172.21.15.97:0/3366040671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed301a8150 con 0x7fed30109830 2024-07-26T07:57:36.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.564+0000 7fed2f400640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed301035a0 0x7fed30075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:57:36.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.564+0000 7fed2fe00640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30109830 0x7fed30075bc0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fed2800e940 tx=0x7fed2800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.565+0000 7fed2cc00640 1 -- 172.21.15.97:0/3366040671 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed28004d20 con 0x7fed30109830 2024-07-26T07:57:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.565+0000 7fed2cc00640 1 -- 172.21.15.97:0/3366040671 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed28005020 con 0x7fed30109830 2024-07-26T07:57:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.565+0000 7fed2cc00640 1 -- 172.21.15.97:0/3366040671 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280116d0 con 0x7fed30109830 2024-07-26T07:57:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.565+0000 7fed36600640 1 -- 172.21.15.97:0/3366040671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed301a8440 con 0x7fed30109830 2024-07-26T07:57:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.565+0000 7fed36600640 1 -- 172.21.15.97:0/3366040671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed301a8900 con 0x7fed30109830 2024-07-26T07:57:36.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.569+0000 7fed3d263740 1 Processor -- start 2024-07-26T07:57:36.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.569+0000 7fed2cc00640 1 -- 172.21.15.97:0/3366040671 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fed2800cb20 con 0x7fed30109830 2024-07-26T07:57:36.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.569+0000 7fed3d263740 1 -- start start 2024-07-26T07:57:36.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed3d263740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700fa850 0x560e704a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed3d263740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700d1990 0x560e704a1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed3d263740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 0x560e704a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed3d263740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560e70415970 con 0x560e700fa850 2024-07-26T07:57:36.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed3d263740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560e70404e70 con 0x560e700d9b70 2024-07-26T07:57:36.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed3d263740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560e70408eb0 con 0x560e700d1990 2024-07-26T07:57:36.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed27e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700fa850 0x560e704a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed27e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700fa850 0x560e704a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42814/0 (socket says 172.21.15.97:42814) 2024-07-26T07:57:36.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed27e00640 1 -- 172.21.15.97:0/511915304 learned_addr learned my addr 172.21.15.97:0/511915304 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed2f400640 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700d1990 0x560e704a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.570+0000 7fed27e00640 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 0x560e704a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed27e00640 1 -- 172.21.15.97:0/511915304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700d1990 msgr2=0x560e704a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed27e00640 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700d1990 0x560e704a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed27e00640 1 -- 172.21.15.97:0/511915304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 msgr2=0x560e704a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed27e00640 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 0x560e704a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed27e00640 1 -- 172.21.15.97:0/511915304 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e7038d8e0 con 0x560e700fa850 2024-07-26T07:57:36.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed2cc00640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed140b3c50 0x7fed140b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed27e00640 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700fa850 0x560e704a0ee0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fed18005ab0 tx=0x7fed18005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba32fcbbea50939 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed2cc00640 1 -- 172.21.15.97:0/3366040671 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fed280e3c40 con 0x7fed30109830 2024-07-26T07:57:36.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed2f400640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed140b3c50 0x7fed140b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.571+0000 7fed24c00640 1 -- 172.21.15.97:0/511915304 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18035070 con 0x560e700fa850 2024-07-26T07:57:36.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.572+0000 7fed24c00640 1 -- 172.21.15.97:0/511915304 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed18003ed0 con 0x560e700fa850 2024-07-26T07:57:36.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.572+0000 7fed24c00640 1 -- 172.21.15.97:0/511915304 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1803be20 con 0x560e700fa850 2024-07-26T07:57:36.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 -- 172.21.15.97:0/511915304 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700fa850 msgr2=0x560e704a0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700fa850 0x560e704a0ee0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fed18005ab0 tx=0x7fed18005f00 comp rx=0 tx=0).stop 2024-07-26T07:57:36.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed2f400640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed140b3c50 0x7fed140b6160 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x560e7049fb60 tx=0x7fed2000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 -- 172.21.15.97:0/511915304 shutdown_connections 2024-07-26T07:57:36.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 0x560e704a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700d1990 0x560e704a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 --2- 172.21.15.97:0/511915304 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700fa850 0x560e704a0ee0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 -- 172.21.15.97:0/511915304 >> 172.21.15.97:0/511915304 conn(0x560e700eec40 msgr2=0x560e7049edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:36.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 -- 172.21.15.97:0/511915304 shutdown_connections 2024-07-26T07:57:36.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.573+0000 7fed3d263740 1 -- 172.21.15.97:0/511915304 wait complete. 2024-07-26T07:57:36.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.574+0000 7fed3d263740 1 -- start start 2024-07-26T07:57:36.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.574+0000 7fed3d263740 1 Processor -- start 2024-07-26T07:57:36.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed3d263740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed3d263740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704acf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed3d263740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e704b8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed3d263740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560e702687b0 con 0x560e700d9b70 2024-07-26T07:57:36.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed3d263740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x560e703a1420 con 0x560e700d1990 2024-07-26T07:57:36.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed3d263740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560e704af660 con 0x560e700fa850 2024-07-26T07:57:36.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed3d263740 10 client.0 _ll_register_callbacks cb 0x560e701120a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:36.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed2f400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed27e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed2f400640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704ac9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:49750/0 (socket says 172.21.15.97:49750) 2024-07-26T07:57:36.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.575+0000 7fed2f400640 1 -- 172.21.15.97:0/3659908578 learned_addr learned my addr 172.21.15.97:0/3659908578 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed2f400640 1 -- 172.21.15.97:0/3659908578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 msgr2=0x560e704b8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed2fe00640 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e704b8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed2f400640 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e704b8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed2f400640 1 -- 172.21.15.97:0/3659908578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 msgr2=0x560e704acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed2f400640 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704acf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed2f400640 1 -- 172.21.15.97:0/3659908578 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e70399c50 con 0x560e700d1990 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed27e00640 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed2f400640 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704ac9e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fed2000fdb0 tx=0x7fed20004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69c52acc77e6b95a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.576+0000 7fed1d600640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.577+0000 7fed1d600640 1 -- 172.21.15.97:0/3659908578 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2002c060 con 0x560e700d1990 2024-07-26T07:57:36.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.577+0000 7fed3d263740 1 -- 172.21.15.97:0/3659908578 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed18003ed0 con 0x560e700d1990 2024-07-26T07:57:36.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.577+0000 7fed1d600640 1 -- 172.21.15.97:0/3659908578 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fed20016030 con 0x560e700d1990 2024-07-26T07:57:36.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.578+0000 7fed1d600640 1 -- 172.21.15.97:0/3659908578 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fed2001c8a0 con 0x560e700d1990 2024-07-26T07:57:36.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.578+0000 7fed3d263740 10 client.35606 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:36.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.578+0000 7fed3d263740 1 -- 172.21.15.97:0/3659908578 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560e70174ba0 con 0x560e700d1990 2024-07-26T07:57:36.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.578+0000 7fed1d600640 1 -- 172.21.15.97:0/3659908578 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed18003ed0 con 0x560e700d1990 2024-07-26T07:57:36.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.579+0000 7fed1d600640 1 -- 172.21.15.97:0/3659908578 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7fed200047a0 con 0x560e700d1990 2024-07-26T07:57:36.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.579+0000 7fed3d263740 10 client.35606 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:36.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.579+0000 7fed3d263740 10 client.35606 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:57:36.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.579+0000 7fed3d263740 20 client.35606 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:36.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.579+0000 7fed3d263740 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560e700b91d0 0x560e704be870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.579+0000 7fed3d263740 4 client.35606 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:57:36.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.579+0000 7fed3d263740 1 -- 172.21.15.97:0/3659908578 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e704178c0 con 0x560e700b91d0 2024-07-26T07:57:36.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.580+0000 7fed27e00640 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560e700b91d0 0x560e704be870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.581+0000 7fed27e00640 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560e700b91d0 0x560e704be870 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e37eee33b6986da server_cookie=df269a4f4222bede in_seq=0 out_seq=0 2024-07-26T07:57:36.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.581+0000 7fed1d600640 10 client.35606 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:36.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.582+0000 7fed1d600640 1 -- 172.21.15.97:0/3659908578 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560e704178c0 con 0x560e700b91d0 2024-07-26T07:57:36.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.582+0000 7fed1d600640 10 client.35606 handle_command_reply: tid=0 2024-07-26T07:57:36.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.582+0000 7fed3d263740 1 client.35606 shutdown 2024-07-26T07:57:36.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 -- 172.21.15.97:0/3659908578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 msgr2=0x560e704ac9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704ac9e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fed2000fdb0 tx=0x7fed20004730 comp rx=0 tx=0).stop 2024-07-26T07:57:36.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 -- 172.21.15.97:0/3659908578 shutdown_connections 2024-07-26T07:57:36.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560e700b91d0 0x560e704be870 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e704b8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 --2- 172.21.15.97:0/3659908578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704ac9e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 -- 172.21.15.97:0/3659908578 >> 172.21.15.97:0/3659908578 conn(0x560e700eec40 msgr2=0x560e7049d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:36.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed1d600640 0 client.35606 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:36.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 -- 172.21.15.97:0/3659908578 shutdown_connections 2024-07-26T07:57:36.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.583+0000 7fed3d263740 1 -- 172.21.15.97:0/3659908578 wait complete. 2024-07-26T07:57:36.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.584+0000 7fed3d263740 20 client.35606 trim_cache size 0 max 16384 2024-07-26T07:57:36.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.610+0000 7fed3d263740 1 Processor -- start 2024-07-26T07:57:36.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.610+0000 7fed3d263740 1 -- start start 2024-07-26T07:57:36.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.610+0000 7fed3d263740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d1990 0x560e704a7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.610+0000 7fed3d263740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 0x560e704b8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.610+0000 7fed3d263740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e704b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.610+0000 7fed3d263740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fed301b1160 con 0x560e700d1990 2024-07-26T07:57:36.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.610+0000 7fed3d263740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fed301b2cd0 con 0x560e700d9b70 2024-07-26T07:57:36.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.610+0000 7fed3d263740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fed301b1830 con 0x560e700fa850 2024-07-26T07:57:36.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.611+0000 7fed2f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d1990 0x560e704a7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.611+0000 7fed2f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d1990 0x560e704a7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42840/0 (socket says 172.21.15.97:42840) 2024-07-26T07:57:36.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.611+0000 7fed27e00640 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 0x560e704b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.611+0000 7fed2f400640 1 -- 172.21.15.97:0/2136356772 learned_addr learned my addr 172.21.15.97:0/2136356772 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.611+0000 7fed2fe00640 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e704b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.612+0000 7fed2f400640 1 -- 172.21.15.97:0/2136356772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 msgr2=0x560e704b8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.612+0000 7fed2f400640 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e704b8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.612+0000 7fed2f400640 1 -- 172.21.15.97:0/2136356772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 msgr2=0x560e704b8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.612+0000 7fed2f400640 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 0x560e704b8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.612+0000 7fed2f400640 1 -- 172.21.15.97:0/2136356772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e7049dc70 con 0x560e700d1990 2024-07-26T07:57:36.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.612+0000 7fed2f400640 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d1990 0x560e704a7c30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fed20000c00 tx=0x7fed2000aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4480831a103d8061 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.613+0000 7fed1fe00640 1 -- 172.21.15.97:0/2136356772 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed20040070 con 0x560e700d1990 2024-07-26T07:57:36.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.613+0000 7fed1fe00640 1 -- 172.21.15.97:0/2136356772 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed20004d80 con 0x560e700d1990 2024-07-26T07:57:36.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.613+0000 7fed1fe00640 1 -- 172.21.15.97:0/2136356772 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2003b430 con 0x560e700d1990 2024-07-26T07:57:36.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.614+0000 7fed3d263740 1 -- 172.21.15.97:0/2136356772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d1990 msgr2=0x560e704a7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.614+0000 7fed3d263740 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d1990 0x560e704a7c30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fed20000c00 tx=0x7fed2000aea0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.614+0000 7fed3d263740 1 -- 172.21.15.97:0/2136356772 shutdown_connections 2024-07-26T07:57:36.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.614+0000 7fed3d263740 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e704b8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.614+0000 7fed3d263740 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d9b70 0x560e704b8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.614+0000 7fed3d263740 1 --2- 172.21.15.97:0/2136356772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d1990 0x560e704a7c30 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.614+0000 7fed3d263740 1 -- 172.21.15.97:0/2136356772 >> 172.21.15.97:0/2136356772 conn(0x560e700eec40 msgr2=0x560e704ab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:36.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.614+0000 7fed3d263740 1 -- 172.21.15.97:0/2136356772 shutdown_connections 2024-07-26T07:57:36.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.615+0000 7fed3d263740 1 -- 172.21.15.97:0/2136356772 wait complete. 2024-07-26T07:57:36.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.615+0000 7fed3d263740 1 -- start start 2024-07-26T07:57:36.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.615+0000 7fed3d263740 1 Processor -- start 2024-07-26T07:57:36.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.616+0000 7fed3d263740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704afac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.616+0000 7fed3d263740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.616+0000 7fed3d263740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e7052d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.616+0000 7fed3d263740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fed301b25d0 con 0x560e700d9b70 2024-07-26T07:57:36.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.616+0000 7fed3d263740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fed301b1f00 con 0x560e700d1990 2024-07-26T07:57:36.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.616+0000 7fed3d263740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fed301af0e0 con 0x560e700fa850 2024-07-26T07:57:36.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.616+0000 7fed3d263740 10 client.0 _ll_register_callbacks cb 0x7fed2000ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.617+0000 7fed2fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e7052d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.617+0000 7fed2fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e7052d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.97:60286/0 (socket says 172.21.15.97:60286) 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.617+0000 7fed2fe00640 1 -- 172.21.15.97:0/1592066820 learned_addr learned my addr 172.21.15.97:0/1592066820 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.617+0000 7fed2f400640 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.617+0000 7fed27e00640 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.618+0000 7fed2fe00640 1 -- 172.21.15.97:0/1592066820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 msgr2=0x560e704afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.618+0000 7fed2fe00640 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704afac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.618+0000 7fed2fe00640 1 -- 172.21.15.97:0/1592066820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 msgr2=0x560e704b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.618+0000 7fed2fe00640 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.618+0000 7fed2fe00640 1 -- 172.21.15.97:0/1592066820 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e704b08f0 con 0x560e700fa850 2024-07-26T07:57:36.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.618+0000 7fed2fe00640 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e7052d800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fed28005fd0 tx=0x7fed280aa5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72e06a5e0b4e9740 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.618+0000 7fed1d600640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-07-26T07:57:36.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.619+0000 7fed1d600640 1 -- 172.21.15.97:0/1592066820 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280c1030 con 0x560e700fa850 2024-07-26T07:57:36.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.619+0000 7fed3d263740 1 -- 172.21.15.97:0/1592066820 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed20004d80 con 0x560e700fa850 2024-07-26T07:57:36.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.619+0000 7fed1d600640 1 -- 172.21.15.97:0/1592066820 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fed280abe00 con 0x560e700fa850 2024-07-26T07:57:36.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.619+0000 7fed1d600640 1 -- 172.21.15.97:0/1592066820 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fed2801f040 con 0x560e700fa850 2024-07-26T07:57:36.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.619+0000 7fed3d263740 10 client.25642 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:36.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.620+0000 7fed3d263740 1 -- 172.21.15.97:0/1592066820 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560e7052ee80 con 0x560e700fa850 2024-07-26T07:57:36.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.620+0000 7fed1d600640 1 -- 172.21.15.97:0/1592066820 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed20004d80 con 0x560e700fa850 2024-07-26T07:57:36.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.620+0000 7fed1d600640 1 -- 172.21.15.97:0/1592066820 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7fed2800f040 con 0x560e700fa850 2024-07-26T07:57:36.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.620+0000 7fed3d263740 10 client.25642 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:36.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.621+0000 7fed3d263740 10 client.25642 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:57:36.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.621+0000 7fed3d263740 20 client.25642 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:36.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.621+0000 7fed3d263740 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560e7008e580 0x560e705316e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.621+0000 7fed3d263740 4 client.25642 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.95, "await_for": 45.0}] 2024-07-26T07:57:36.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.621+0000 7fed3d263740 1 -- 172.21.15.97:0/1592066820 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.95, "await_for": 45.0}) -- 0x560e70413360 con 0x560e7008e580 2024-07-26T07:57:36.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.622+0000 7fed2f400640 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560e7008e580 0x560e705316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.623+0000 7fed2f400640 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560e7008e580 0x560e705316e0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d6793ffc0e3648 server_cookie=5cfc751690e603a1 in_seq=0 out_seq=0 2024-07-26T07:57:36.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:36.623+0000 7fed1d600640 10 client.25642 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.859+0000 7fbdd3000640 1 -- 172.21.15.97:0/833266978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 msgr2=0x7fbdcc103380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.859+0000 7fbdd3000640 1 --2- 172.21.15.97:0/833266978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 0x7fbdcc103380 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009900 tx=0x7fbdc002a410 comp rx=0 tx=0).stop 2024-07-26T07:57:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.860+0000 7fbdd3000640 1 -- 172.21.15.97:0/833266978 shutdown_connections 2024-07-26T07:57:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.860+0000 7fbdd3000640 1 --2- 172.21.15.97:0/833266978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdcc104930 0x7fbdcc10b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.860+0000 7fbdd3000640 1 --2- 172.21.15.97:0/833266978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdcc103f90 0x7fbdcc1043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.860+0000 7fbdd3000640 1 --2- 172.21.15.97:0/833266978 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 0x7fbdcc103380 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.860+0000 7fbdd3000640 1 -- 172.21.15.97:0/833266978 >> 172.21.15.97:0/833266978 conn(0x7fbdcc0fe850 msgr2=0x7fbdcc100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.860+0000 7fbdd3000640 1 -- 172.21.15.97:0/833266978 shutdown_connections 2024-07-26T07:57:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.860+0000 7fbdd3000640 1 -- 172.21.15.97:0/833266978 wait complete. 2024-07-26T07:57:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.861+0000 7fbdd3000640 1 Processor -- start 2024-07-26T07:57:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.861+0000 7fbdd3000640 1 -- start start 2024-07-26T07:57:36.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd3000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 0x7fbdcc076810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd3000640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdcc103f90 0x7fbdcc075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd3000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdcc104930 0x7fbdcc075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd3000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc10eab0 con 0x7fbdcc102f10 2024-07-26T07:57:36.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd3000640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc10e930 con 0x7fbdcc103f90 2024-07-26T07:57:36.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd3000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc10ec30 con 0x7fbdcc104930 2024-07-26T07:57:36.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 0x7fbdcc076810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd1c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 0x7fbdcc076810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42866/0 (socket says 172.21.15.97:42866) 2024-07-26T07:57:36.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd1c00640 1 -- 172.21.15.97:0/2307499367 learned_addr learned my addr 172.21.15.97:0/2307499367 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd1200640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdcc103f90 0x7fbdcc075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.862+0000 7fbdd2600640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdcc104930 0x7fbdcc075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdd1c00640 1 -- 172.21.15.97:0/2307499367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdcc104930 msgr2=0x7fbdcc075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdcc104930 0x7fbdcc075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdd1c00640 1 -- 172.21.15.97:0/2307499367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdcc103f90 msgr2=0x7fbdcc075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdcc103f90 0x7fbdcc075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdd1c00640 1 -- 172.21.15.97:0/2307499367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdcc075d20 con 0x7fbdcc102f10 2024-07-26T07:57:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 0x7fbdcc076810 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0005760 tx=0x7fbdc0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdca000640 1 -- 172.21.15.97:0/2307499367 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc003f070 con 0x7fbdcc102f10 2024-07-26T07:57:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdd3000640 1 -- 172.21.15.97:0/2307499367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdcc1a3810 con 0x7fbdcc102f10 2024-07-26T07:57:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.863+0000 7fbdca000640 1 -- 172.21.15.97:0/2307499367 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdc0031ba0 con 0x7fbdcc102f10 2024-07-26T07:57:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.864+0000 7fbdd3000640 1 -- 172.21.15.97:0/2307499367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdcc1a3c40 con 0x7fbdcc102f10 2024-07-26T07:57:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.864+0000 7fbdca000640 1 -- 172.21.15.97:0/2307499367 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0049060 con 0x7fbdcc102f10 2024-07-26T07:57:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.868+0000 7fbddb796740 1 Processor -- start 2024-07-26T07:57:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.868+0000 7fbddb796740 1 -- start start 2024-07-26T07:57:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbddb796740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcaf51b0 0x557ebcea38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbddb796740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbddb796740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebcea4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbddb796740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x557ebce0d270 con 0x557ebcacc2f0 2024-07-26T07:57:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbddb796740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557ebcdfc770 con 0x557ebcad44d0 2024-07-26T07:57:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbddb796740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557ebce007b0 con 0x557ebcaf51b0 2024-07-26T07:57:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebcea4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebcea4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.97:49784/0 (socket says 172.21.15.97:49784) 2024-07-26T07:57:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.869+0000 7fbdd1200640 1 -- 172.21.15.97:0/4234599872 learned_addr learned my addr 172.21.15.97:0/4234599872 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.870+0000 7fbdca000640 1 -- 172.21.15.97:0/2307499367 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fbdc0042070 con 0x7fbdcc102f10 2024-07-26T07:57:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.870+0000 7fbdd1200640 1 -- 172.21.15.97:0/4234599872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcaf51b0 msgr2=0x557ebcea38d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T07:57:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.870+0000 7fbdd1200640 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcaf51b0 0x557ebcea38d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.870+0000 7fbdd1200640 1 -- 172.21.15.97:0/4234599872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcacc2f0 msgr2=0x557ebcea3f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T07:57:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.870+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.870+0000 7fbdd1200640 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.870+0000 7fbdd1200640 1 -- 172.21.15.97:0/4234599872 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebcb60790 con 0x557ebcad44d0 2024-07-26T07:57:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.870+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:57:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.871+0000 7fbdd1200640 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebcea4a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009cf0 tx=0x7fbdb8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b450f19d8b0c362d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.871+0000 7fbdbea00640 1 -- 172.21.15.97:0/4234599872 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8004230 con 0x557ebcad44d0 2024-07-26T07:57:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.871+0000 7fbdbea00640 1 -- 172.21.15.97:0/4234599872 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdb80043d0 con 0x557ebcad44d0 2024-07-26T07:57:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.871+0000 7fbdbea00640 1 -- 172.21.15.97:0/4234599872 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb80196b0 con 0x557ebcad44d0 2024-07-26T07:57:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbdca000640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbdb40b3c50 0x7fbdb40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbdca000640 1 -- 172.21.15.97:0/2307499367 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fbdc01049a0 con 0x7fbdcc102f10 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbdd2600640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbdb40b3c50 0x7fbdb40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbddb796740 1 -- 172.21.15.97:0/4234599872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 msgr2=0x557ebcea4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbddb796740 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebcea4a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009cf0 tx=0x7fbdb8011810 comp rx=0 tx=0).stop 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbddb796740 1 -- 172.21.15.97:0/4234599872 shutdown_connections 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbddb796740 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebcea4a90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbddb796740 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbddb796740 1 --2- 172.21.15.97:0/4234599872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcaf51b0 0x557ebcea38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbddb796740 1 -- 172.21.15.97:0/4234599872 >> 172.21.15.97:0/4234599872 conn(0x557ebcae9300 msgr2=0x557ebce9f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:36.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.872+0000 7fbddb796740 1 -- 172.21.15.97:0/4234599872 shutdown_connections 2024-07-26T07:57:36.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.873+0000 7fbddb796740 1 -- 172.21.15.97:0/4234599872 wait complete. 2024-07-26T07:57:36.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.873+0000 7fbdd2600640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbdb40b3c50 0x7fbdb40b6160 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009e20 tx=0x7fbdc4009410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.873+0000 7fbddb796740 1 -- start start 2024-07-26T07:57:36.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbddb796740 1 Processor -- start 2024-07-26T07:57:36.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbddb796740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebcead380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbddb796740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbddb796740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbddb796740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x557ebcd36ab0 con 0x557ebcad44d0 2024-07-26T07:57:36.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbddb796740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x557ebcd99d50 con 0x557ebcaf51b0 2024-07-26T07:57:36.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbddb796740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557ebcea55b0 con 0x557ebcacc2f0 2024-07-26T07:57:36.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbddb796740 10 client.0 _ll_register_callbacks cb 0x557ebcb0ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:36.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceada40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42892/0 (socket says 172.21.15.97:42892) 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.874+0000 7fbdd1200640 1 -- 172.21.15.97:0/3923645032 learned_addr learned my addr 172.21.15.97:0/3923645032 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.875+0000 7fbdd2600640 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.875+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebcead380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.875+0000 7fbdd2600640 1 -- 172.21.15.97:0/3923645032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 msgr2=0x557ebcead380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.875+0000 7fbdd2600640 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebcead380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.875+0000 7fbdd2600640 1 -- 172.21.15.97:0/3923645032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 msgr2=0x557ebceada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.875+0000 7fbdd2600640 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.875+0000 7fbdd2600640 1 -- 172.21.15.97:0/3923645032 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebcbd9620 con 0x557ebcaf51b0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.875+0000 7fbdd2600640 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceae5b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400cc80 tx=0x7fbdc4004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe91db04394ab60 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.876+0000 7fbdb3400640 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.876+0000 7fbdb3400640 1 -- 172.21.15.97:0/3923645032 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4027060 con 0x557ebcaf51b0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.876+0000 7fbddb796740 1 -- 172.21.15.97:0/3923645032 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdb80043d0 con 0x557ebcaf51b0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.876+0000 7fbdb3400640 1 -- 172.21.15.97:0/3923645032 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fbdc401d030 con 0x557ebcaf51b0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.876+0000 7fbdb3400640 1 -- 172.21.15.97:0/3923645032 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbdc4027060 con 0x557ebcaf51b0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.876+0000 7fbddb796740 10 client.35621 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.876+0000 7fbddb796740 1 -- 172.21.15.97:0/3923645032 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557ebce98360 con 0x557ebcaf51b0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbdb3400640 1 -- 172.21.15.97:0/3923645032 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdb80043d0 con 0x557ebcaf51b0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbdb3400640 1 -- 172.21.15.97:0/3923645032 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7fbdc402b030 con 0x557ebcaf51b0 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbddb796740 10 client.35621 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbddb796740 10 client.35621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:57:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbddb796740 20 client.35621 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbddb796740 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x557ebcab3d80 0x557ebce9bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbddb796740 4 client.35621 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:57:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbddb796740 1 -- 172.21.15.97:0/3923645032 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ebce0f1c0 con 0x557ebcab3d80 2024-07-26T07:57:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.877+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x557ebcab3d80 0x557ebce9bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.878+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x557ebcab3d80 0x557ebce9bb20 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c947506d2cc86575 server_cookie=3b62dff4c330e2b in_seq=0 out_seq=0 2024-07-26T07:57:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.878+0000 7fbdb3400640 10 client.35621 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.936+0000 7fbdb3400640 1 -- 172.21.15.97:0/3923645032 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557ebce0f1c0 con 0x557ebcab3d80 2024-07-26T07:57:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.936+0000 7fbdb3400640 10 client.35621 handle_command_reply: tid=0 2024-07-26T07:57:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.936+0000 7fbddb796740 1 client.35621 shutdown 2024-07-26T07:57:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 -- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 msgr2=0x557ebceae5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceae5b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400cc80 tx=0x7fbdc4004b50 comp rx=0 tx=0).stop 2024-07-26T07:57:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 -- 172.21.15.97:0/3923645032 shutdown_connections 2024-07-26T07:57:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x557ebcab3d80 0x557ebce9bb20 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceae5b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 --2- 172.21.15.97:0/3923645032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebcead380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 -- 172.21.15.97:0/3923645032 >> 172.21.15.97:0/3923645032 conn(0x557ebcae9300 msgr2=0x557ebce950b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbdb3400640 0 client.35621 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 -- 172.21.15.97:0/3923645032 shutdown_connections 2024-07-26T07:57:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 1 -- 172.21.15.97:0/3923645032 wait complete. 2024-07-26T07:57:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.937+0000 7fbddb796740 20 client.35621 trim_cache size 0 max 16384 2024-07-26T07:57:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.969+0000 7fbddb796740 1 Processor -- start 2024-07-26T07:57:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.969+0000 7fbddb796740 1 -- start start 2024-07-26T07:57:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.969+0000 7fbddb796740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.969+0000 7fbddb796740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebceaba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.969+0000 7fbddb796740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcaf51b0 0x557ebceabf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.969+0000 7fbddb796740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc1a8900 con 0x557ebcaf51b0 2024-07-26T07:57:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.969+0000 7fbddb796740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc1aa470 con 0x557ebcad44d0 2024-07-26T07:57:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.969+0000 7fbddb796740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc1a8fd0 con 0x557ebcacc2f0 2024-07-26T07:57:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcaf51b0 0x557ebceabf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcaf51b0 0x557ebceabf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42894/0 (socket says 172.21.15.97:42894) 2024-07-26T07:57:36.973 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 -- 172.21.15.97:0/631498065 learned_addr learned my addr 172.21.15.97:0/631498065 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd1200640 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebceaba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 -- 172.21.15.97:0/631498065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 msgr2=0x557ebcea3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 -- 172.21.15.97:0/631498065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 msgr2=0x557ebceaba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebceaba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 -- 172.21.15.97:0/631498065 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebceab500 con 0x557ebcaf51b0 2024-07-26T07:57:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.970+0000 7fbdd2600640 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcaf51b0 0x557ebceabf80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fbdc401e2f0 tx=0x7fbdc40099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8fc45289065bdd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.971+0000 7fbdbe000640 1 -- 172.21.15.97:0/631498065 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4027060 con 0x557ebcaf51b0 2024-07-26T07:57:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.971+0000 7fbdbe000640 1 -- 172.21.15.97:0/631498065 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdc40139f0 con 0x557ebcaf51b0 2024-07-26T07:57:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.971+0000 7fbdbe000640 1 -- 172.21.15.97:0/631498065 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc402a3e0 con 0x557ebcaf51b0 2024-07-26T07:57:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 -- 172.21.15.97:0/631498065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcaf51b0 msgr2=0x557ebceabf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcaf51b0 0x557ebceabf80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fbdc401e2f0 tx=0x7fbdc40099f0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 -- 172.21.15.97:0/631498065 shutdown_connections 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcaf51b0 0x557ebceabf80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcad44d0 0x557ebceaba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 --2- 172.21.15.97:0/631498065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebcea3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 -- 172.21.15.97:0/631498065 >> 172.21.15.97:0/631498065 conn(0x557ebcae9300 msgr2=0x557ebcea3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 -- 172.21.15.97:0/631498065 shutdown_connections 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.972+0000 7fbddb796740 1 -- 172.21.15.97:0/631498065 wait complete. 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.973+0000 7fbddb796740 1 -- start start 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.973+0000 7fbddb796740 1 Processor -- start 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbddb796740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebceb1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbddb796740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceb2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbddb796740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceb2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbddb796740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc1a9d70 con 0x557ebcad44d0 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbddb796740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc1a96a0 con 0x557ebcaf51b0 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbddb796740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc1a68c0 con 0x557ebcacc2f0 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbddb796740 10 client.0 _ll_register_callbacks cb 0x7fbdc4009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceb2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceb2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:42910/0 (socket says 172.21.15.97:42910) 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbdd1200640 1 -- 172.21.15.97:0/4270084912 learned_addr learned my addr 172.21.15.97:0/4270084912 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebceb1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.974+0000 7fbdd2600640 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceb2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdd1200640 1 -- 172.21.15.97:0/4270084912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 msgr2=0x557ebceb1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdd1200640 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebceb1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdd1200640 1 -- 172.21.15.97:0/4270084912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 msgr2=0x557ebceb2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdd1200640 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceb2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdd1200640 1 -- 172.21.15.97:0/4270084912 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebceb32a0 con 0x557ebcad44d0 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdd1200640 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceb2020 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fbdb801b040 tx=0x7fbdb8009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120f2d437c8cf430 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdb3400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdb3400640 1 -- 172.21.15.97:0/4270084912 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8004a70 con 0x557ebcad44d0 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbdb3400640 1 -- 172.21.15.97:0/4270084912 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fbdb8004c10 con 0x557ebcad44d0 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.975+0000 7fbddb796740 1 -- 172.21.15.97:0/4270084912 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557ebce0ac60 con 0x557ebcad44d0 2024-07-26T07:57:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbdb3400640 1 -- 172.21.15.97:0/4270084912 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbdb8009910 con 0x557ebcad44d0 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbddb796740 10 client.16182 fetch_fsmap learned FSMap version 34 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbddb796740 1 -- 172.21.15.97:0/4270084912 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557ebceb4630 con 0x557ebcad44d0 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbdb3400640 1 -- 172.21.15.97:0/4270084912 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x557ebce0ac60 con 0x557ebcad44d0 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbdb3400640 1 -- 172.21.15.97:0/4270084912 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 34) ==== 8366+0+0 (secure 0 0 0) 0x7fbdb8034020 con 0x557ebcad44d0 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbddb796740 10 client.16182 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbddb796740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbddb796740 20 client.16182 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbddb796740 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x557ebca891c0 0x557ebceb4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:36.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbddb796740 4 client.16182 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:57:36.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.976+0000 7fbddb796740 1 -- 172.21.15.97:0/4270084912 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ebce083a0 con 0x557ebca891c0 2024-07-26T07:57:36.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.977+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x557ebca891c0 0x557ebceb4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:36.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.978+0000 7fbdd1c00640 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x557ebca891c0 0x557ebceb4e80 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a11418b5d876b761 server_cookie=c0c71d859dad2f52 in_seq=0 out_seq=0 2024-07-26T07:57:36.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:36.978+0000 7fbdb3400640 10 client.16182 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:37 smithi115 ceph-mon[35364]: pgmap v686: 97 pgs: 97 active+clean; 130 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.8 MiB/s wr, 141 op/s 2024-07-26T07:57:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:37 smithi181 ceph-mon[35869]: pgmap v686: 97 pgs: 97 active+clean; 130 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.8 MiB/s wr, 141 op/s 2024-07-26T07:57:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.997+0000 7fbdb3400640 1 -- 172.21.15.97:0/4270084912 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ebce083a0 con 0x557ebca891c0 2024-07-26T07:57:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.997+0000 7fbdb3400640 10 client.16182 handle_command_reply: tid=0 2024-07-26T07:57:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.997+0000 7fbddb796740 1 client.16182 shutdown 2024-07-26T07:57:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbddb796740 1 -- 172.21.15.97:0/4270084912 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 msgr2=0x557ebceb2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbddb796740 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceb2020 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fbdb801b040 tx=0x7fbdb8009280 comp rx=0 tx=0).stop 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbddb796740 1 -- 172.21.15.97:0/4270084912 shutdown_connections 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbddb796740 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x557ebca891c0 0x557ebceb4e80 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbddb796740 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x557ebcaf51b0 0x557ebceb2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbddb796740 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557ebcad44d0 0x557ebceb2020 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbddb796740 1 --2- 172.21.15.97:0/4270084912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ebcacc2f0 0x557ebceb1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbddb796740 1 -- 172.21.15.97:0/4270084912 >> 172.21.15.97:0/4270084912 conn(0x557ebcae9300 msgr2=0x557ebceaba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.998+0000 7fbdb3400640 0 client.16182 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.999+0000 7fbddb796740 1 -- 172.21.15.97:0/4270084912 shutdown_connections 2024-07-26T07:57:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.999+0000 7fbddb796740 1 -- 172.21.15.97:0/4270084912 wait complete. 2024-07-26T07:57:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:37.999+0000 7fbddb796740 20 client.16182 trim_cache size 0 max 16384 2024-07-26T07:57:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:57:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:57:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:57:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:57:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:37 smithi097 ceph-mon[32231]: pgmap v686: 97 pgs: 97 active+clean; 130 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.8 MiB/s wr, 141 op/s 2024-07-26T07:57:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 -- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbdb40b3c50 msgr2=0x7fbdb40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbdb40b3c50 0x7fbdb40b6160 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009e20 tx=0x7fbdc4009410 comp rx=0 tx=0).stop 2024-07-26T07:57:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 -- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 msgr2=0x7fbdcc076810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 0x7fbdcc076810 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0005760 tx=0x7fbdc0003c10 comp rx=0 tx=0).stop 2024-07-26T07:57:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 -- 172.21.15.97:0/2307499367 shutdown_connections 2024-07-26T07:57:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fbdb40b3c50 0x7fbdb40b6160 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdcc104930 0x7fbdcc075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbdcc103f90 0x7fbdcc075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 --2- 172.21.15.97:0/2307499367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdcc102f10 0x7fbdcc076810 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.000+0000 7fbdd3000640 1 -- 172.21.15.97:0/2307499367 >> 172.21.15.97:0/2307499367 conn(0x7fbdcc0fe850 msgr2=0x7fbdcc1072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.001+0000 7fbdd3000640 1 -- 172.21.15.97:0/2307499367 shutdown_connections 2024-07-26T07:57:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:38.001+0000 7fbdd3000640 1 -- 172.21.15.97:0/2307499367 wait complete. 2024-07-26T07:57:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:39 smithi115 ceph-mon[35364]: pgmap v687: 97 pgs: 97 active+clean; 131 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-07-26T07:57:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:39 smithi181 ceph-mon[35869]: pgmap v687: 97 pgs: 97 active+clean; 131 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-07-26T07:57:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:39 smithi097 ceph-mon[32231]: pgmap v687: 97 pgs: 97 active+clean; 131 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-07-26T07:57:40.203 INFO:tasks.workunit.client.0.smithi097.stdout: 36 4592 102 3101 76 1854 112 2024-07-26T07:57:41.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.256+0000 7fed1d600640 1 -- 172.21.15.97:0/1592066820 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560e70413360 con 0x560e7008e580 2024-07-26T07:57:41.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.256+0000 7fed1d600640 10 client.25642 handle_command_reply: tid=0 2024-07-26T07:57:41.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.257+0000 7fed3d263740 1 client.25642 shutdown 2024-07-26T07:57:41.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.257+0000 7fed3d263740 1 -- 172.21.15.97:0/1592066820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 msgr2=0x560e7052d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.257+0000 7fed3d263740 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e7052d800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fed28005fd0 tx=0x7fed280aa5c0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.257+0000 7fed3d263740 1 -- 172.21.15.97:0/1592066820 shutdown_connections 2024-07-26T07:57:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.257+0000 7fed3d263740 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x560e7008e580 0x560e705316e0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.258+0000 7fed3d263740 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560e700fa850 0x560e7052d800 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.258+0000 7fed3d263740 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560e700d9b70 0x560e704b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.258+0000 7fed3d263740 1 --2- 172.21.15.97:0/1592066820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560e700d1990 0x560e704afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.258+0000 7fed3d263740 1 -- 172.21.15.97:0/1592066820 >> 172.21.15.97:0/1592066820 conn(0x560e700eec40 msgr2=0x560e704ab020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:41.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.258+0000 7fed1d600640 0 client.25642 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:41.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.258+0000 7fed3d263740 1 -- 172.21.15.97:0/1592066820 shutdown_connections 2024-07-26T07:57:41.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.258+0000 7fed3d263740 1 -- 172.21.15.97:0/1592066820 wait complete. 2024-07-26T07:57:41.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.259+0000 7fed3d263740 20 client.25642 trim_cache size 0 max 16384 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 34, 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 18, 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "4d2f40c2": { 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 18, 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:57:41.261 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 141.9, 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "QUIESCED", 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:41.262 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:57:41.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.260+0000 7fed36600640 1 -- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed140b3c50 msgr2=0x7fed140b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:41.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.260+0000 7fed36600640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed140b3c50 0x7fed140b6160 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x560e7049fb60 tx=0x7fed2000a380 comp rx=0 tx=0).stop 2024-07-26T07:57:41.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.260+0000 7fed36600640 1 -- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30109830 msgr2=0x7fed30075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.260+0000 7fed36600640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30109830 0x7fed30075bc0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fed2800e940 tx=0x7fed2800ee00 comp rx=0 tx=0).stop 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.261+0000 7fed36600640 1 -- 172.21.15.97:0/3366040671 shutdown_connections 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.261+0000 7fed36600640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fed140b3c50 0x7fed140b6160 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.261+0000 7fed36600640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed30109830 0x7fed30075bc0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.261+0000 7fed36600640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed30108ee0 0x7fed30075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.261+0000 7fed36600640 1 --2- 172.21.15.97:0/3366040671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed301035a0 0x7fed30075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.261+0000 7fed36600640 1 -- 172.21.15.97:0/3366040671 >> 172.21.15.97:0/3366040671 conn(0x7fed300fefd0 msgr2=0x7fed301013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.262+0000 7fed36600640 1 -- 172.21.15.97:0/3366040671 shutdown_connections 2024-07-26T07:57:41.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:41.262+0000 7fed36600640 1 -- 172.21.15.97:0/3366040671 wait complete. 2024-07-26T07:57:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:41 smithi115 ceph-mon[35364]: pgmap v688: 97 pgs: 97 active+clean; 121 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-07-26T07:57:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:41 smithi181 ceph-mon[35869]: pgmap v688: 97 pgs: 97 active+clean; 121 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-07-26T07:57:41.998 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:41 smithi097 ceph-mon[32231]: pgmap v688: 97 pgs: 97 active+clean; 121 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-07-26T07:57:42.059 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '4d2f40c2', quiesce took 13.4 seconds. Will release after: 8.549999999999999 2024-07-26T07:57:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:43 smithi115 ceph-mon[35364]: pgmap v689: 97 pgs: 97 active+clean; 121 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-07-26T07:57:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:43 smithi181 ceph-mon[35869]: pgmap v689: 97 pgs: 97 active+clean; 121 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-07-26T07:57:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:43 smithi097 ceph-mon[32231]: pgmap v689: 97 pgs: 97 active+clean; 121 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-07-26T07:57:44.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:44 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:57:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:44 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:57:45.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:44 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:57:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:45 smithi181 ceph-mon[35869]: pgmap v690: 97 pgs: 97 active+clean; 123 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 273 op/s 2024-07-26T07:57:46.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:45 smithi097 ceph-mon[32231]: pgmap v690: 97 pgs: 97 active+clean; 123 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 273 op/s 2024-07-26T07:57:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:45 smithi115 ceph-mon[35364]: pgmap v690: 97 pgs: 97 active+clean; 123 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 273 op/s 2024-07-26T07:57:48.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:47 smithi097 ceph-mon[32231]: pgmap v691: 97 pgs: 97 active+clean; 123 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 964 KiB/s wr, 254 op/s 2024-07-26T07:57:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:47 smithi115 ceph-mon[35364]: pgmap v691: 97 pgs: 97 active+clean; 123 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 964 KiB/s wr, 254 op/s 2024-07-26T07:57:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:47 smithi181 ceph-mon[35869]: pgmap v691: 97 pgs: 97 active+clean; 123 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 964 KiB/s wr, 254 op/s 2024-07-26T07:57:50.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:49 smithi097 ceph-mon[32231]: pgmap v692: 97 pgs: 97 active+clean; 123 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 869 KiB/s wr, 246 op/s 2024-07-26T07:57:50.203 INFO:tasks.workunit.client.0.smithi097.stdout: 36 7504 0 4579 0 2830 0 2024-07-26T07:57:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:49 smithi115 ceph-mon[35364]: pgmap v692: 97 pgs: 97 active+clean; 123 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 869 KiB/s wr, 246 op/s 2024-07-26T07:57:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:49 smithi181 ceph-mon[35869]: pgmap v692: 97 pgs: 97 active+clean; 123 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 869 KiB/s wr, 246 op/s 2024-07-26T07:57:50.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:57:51.156 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:57:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 -- 172.21.15.97:0/1660478163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784102f10 msgr2=0x7f2784103380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 --2- 172.21.15.97:0/1660478163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784102f10 0x7f2784103380 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2778009900 tx=0x7f277802a410 comp rx=0 tx=0).stop 2024-07-26T07:57:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 -- 172.21.15.97:0/1660478163 shutdown_connections 2024-07-26T07:57:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 --2- 172.21.15.97:0/1660478163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2784104930 0x7f278410b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 --2- 172.21.15.97:0/1660478163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2784103f90 0x7f27841043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 --2- 172.21.15.97:0/1660478163 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784102f10 0x7f2784103380 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 -- 172.21.15.97:0/1660478163 >> 172.21.15.97:0/1660478163 conn(0x7f27840fe850 msgr2=0x7f2784100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 -- 172.21.15.97:0/1660478163 shutdown_connections 2024-07-26T07:57:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.259+0000 7f278ae00640 1 -- 172.21.15.97:0/1660478163 wait complete. 2024-07-26T07:57:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.260+0000 7f278ae00640 1 Processor -- start 2024-07-26T07:57:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.260+0000 7f278ae00640 1 -- start start 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.260+0000 7f278ae00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2784102f10 0x7f278419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.260+0000 7f278ae00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2784103f90 0x7f278419bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278ae00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784104930 0x7f278419cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278ae00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f278410ea60 con 0x7f2784104930 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278ae00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f278410e8e0 con 0x7f2784103f90 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278ae00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f278410ebe0 con 0x7f2784102f10 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784104930 0x7f278419cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784104930 0x7f278419cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48702/0 (socket says 172.21.15.97:48702) 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278a400640 1 -- 172.21.15.97:0/3807397017 learned_addr learned my addr 172.21.15.97:0/3807397017 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f2789a00640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2784102f10 0x7f278419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f2789000640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2784103f90 0x7f278419bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278a400640 1 -- 172.21.15.97:0/3807397017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2784102f10 msgr2=0x7f278419b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278a400640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2784102f10 0x7f278419b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278a400640 1 -- 172.21.15.97:0/3807397017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2784103f90 msgr2=0x7f278419bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278a400640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2784103f90 0x7f278419bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.261+0000 7f278a400640 1 -- 172.21.15.97:0/3807397017 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278419d390 con 0x7f2784104930 2024-07-26T07:57:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.262+0000 7f278a400640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784104930 0x7f278419cb70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f277c00b970 tx=0x7f277c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.262+0000 7f2782000640 1 -- 172.21.15.97:0/3807397017 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c019070 con 0x7f2784104930 2024-07-26T07:57:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.262+0000 7f2782000640 1 -- 172.21.15.97:0/3807397017 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f277c014440 con 0x7f2784104930 2024-07-26T07:57:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.262+0000 7f278ae00640 1 -- 172.21.15.97:0/3807397017 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278419d680 con 0x7f2784104930 2024-07-26T07:57:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.262+0000 7f2782000640 1 -- 172.21.15.97:0/3807397017 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c0075d0 con 0x7f2784104930 2024-07-26T07:57:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.262+0000 7f278ae00640 1 -- 172.21.15.97:0/3807397017 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27841a3d70 con 0x7f2784104930 2024-07-26T07:57:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.266+0000 7f27933fc740 1 Processor -- start 2024-07-26T07:57:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.266+0000 7f2782000640 1 -- 172.21.15.97:0/3807397017 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f277c01c070 con 0x7f2784104930 2024-07-26T07:57:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f27933fc740 1 -- start start 2024-07-26T07:57:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f27933fc740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f27933fc740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f27933fc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f27933fc740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55bd9e842270 con 0x55bd9e5012f0 2024-07-26T07:57:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f27933fc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55bd9e831770 con 0x55bd9e5094d0 2024-07-26T07:57:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f27933fc740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55bd9e8357b0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f2789a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f2789a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48710/0 (socket says 172.21.15.97:48710) 2024-07-26T07:57:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.267+0000 7f2789a00640 1 -- 172.21.15.97:0/303489038 learned_addr learned my addr 172.21.15.97:0/303489038 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789000640 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789000640 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789a00640 1 -- 172.21.15.97:0/303489038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e52a1b0 msgr2=0x55bd9e8d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789a00640 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789a00640 1 -- 172.21.15.97:0/303489038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5094d0 msgr2=0x55bd9e8d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789a00640 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789a00640 1 -- 172.21.15.97:0/303489038 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd9e595790 con 0x55bd9e5012f0 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789000640 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.268+0000 7f2789a00640 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d0730 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f27780096c0 tx=0x7f2778031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52544f89f113f6f0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.269+0000 7f2776a00640 1 -- 172.21.15.97:0/303489038 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778035030 con 0x55bd9e5012f0 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.269+0000 7f2782000640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f276c0b3bd0 0x7f276c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.269+0000 7f2776a00640 1 -- 172.21.15.97:0/303489038 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27780044c0 con 0x55bd9e5012f0 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.269+0000 7f2776a00640 1 -- 172.21.15.97:0/303489038 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778003b80 con 0x55bd9e5012f0 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.269+0000 7f2782000640 1 -- 172.21.15.97:0/3807397017 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f277c0e3410 con 0x7f2784104930 2024-07-26T07:57:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.269+0000 7f2789000640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f276c0b3bd0 0x7f276c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 -- 172.21.15.97:0/303489038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 msgr2=0x55bd9e8d0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f2789000640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f276c0b3bd0 0x7f276c0b60e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f27700058c0 tx=0x7f27700046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d0730 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f27780096c0 tx=0x7f2778031650 comp rx=0 tx=0).stop 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 -- 172.21.15.97:0/303489038 shutdown_connections 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d0730 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 --2- 172.21.15.97:0/303489038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 -- 172.21.15.97:0/303489038 >> 172.21.15.97:0/303489038 conn(0x55bd9e51e300 msgr2=0x55bd9e8cbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 -- 172.21.15.97:0/303489038 shutdown_connections 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.270+0000 7f27933fc740 1 -- 172.21.15.97:0/303489038 wait complete. 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.271+0000 7f27933fc740 1 -- start start 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.271+0000 7f27933fc740 1 Processor -- start 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.271+0000 7f27933fc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.271+0000 7f27933fc740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8da1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.271+0000 7f27933fc740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.271+0000 7f27933fc740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55bd9e76bab0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.271+0000 7f27933fc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55bd9e7ced50 con 0x55bd9e5012f0 2024-07-26T07:57:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.271+0000 7f27933fc740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55bd9e8d1d40 con 0x55bd9e5094d0 2024-07-26T07:57:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f27933fc740 10 client.0 _ll_register_callbacks cb 0x55bd9e541a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f278a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f278a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48722/0 (socket says 172.21.15.97:48722) 2024-07-26T07:57:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f278a400640 1 -- 172.21.15.97:0/3259410953 learned_addr learned my addr 172.21.15.97:0/3259410953 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f2789a00640 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f2789000640 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8da1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f278a400640 1 -- 172.21.15.97:0/3259410953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 msgr2=0x55bd9e8da1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f278a400640 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8da1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f278a400640 1 -- 172.21.15.97:0/3259410953 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 msgr2=0x55bd9e8d9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f278a400640 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.272+0000 7f278a400640 1 -- 172.21.15.97:0/3259410953 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd9e60e620 con 0x55bd9e52a1b0 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f278a400640 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dad30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f277c0a6380 tx=0x7f277c0aaac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a4673fc25eb213 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f276b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f276b400640 1 -- 172.21.15.97:0/3259410953 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c0b33b0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f276b400640 1 -- 172.21.15.97:0/3259410953 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f277c0b2f20 con 0x55bd9e52a1b0 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f27933fc740 1 -- 172.21.15.97:0/3259410953 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd9e8441c0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f276b400640 1 -- 172.21.15.97:0/3259410953 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f277c0b3140 con 0x55bd9e52a1b0 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f27933fc740 10 client.16194 fetch_fsmap learned FSMap version 35 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f27933fc740 1 -- 172.21.15.97:0/3259410953 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bd9e8e3d10 con 0x55bd9e52a1b0 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.273+0000 7f276b400640 1 -- 172.21.15.97:0/3259410953 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55bd9e8441c0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.274+0000 7f276b400640 1 -- 172.21.15.97:0/3259410953 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f277c0a5e00 con 0x55bd9e52a1b0 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.274+0000 7f27933fc740 10 client.16194 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.274+0000 7f27933fc740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.274+0000 7f27933fc740 20 client.16194 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.274+0000 7f27933fc740 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55bd9e4e8d80 0x55bd9e8e7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.274+0000 7f27933fc740 4 client.16194 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:57:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.274+0000 7f27933fc740 1 -- 172.21.15.97:0/3259410953 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd9e83fc60 con 0x55bd9e4e8d80 2024-07-26T07:57:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.274+0000 7f2789a00640 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55bd9e4e8d80 0x55bd9e8e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.276+0000 7f276b400640 10 client.16194 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.276+0000 7f2789a00640 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55bd9e4e8d80 0x55bd9e8e7430 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a32dd0779d113f8 server_cookie=e1d38a2f86dffa02 in_seq=0 out_seq=0 2024-07-26T07:57:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.278+0000 7f276b400640 1 -- 172.21.15.97:0/3259410953 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bd9e83fc60 con 0x55bd9e4e8d80 2024-07-26T07:57:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.278+0000 7f276b400640 10 client.16194 handle_command_reply: tid=0 2024-07-26T07:57:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 client.16194 shutdown 2024-07-26T07:57:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 -- 172.21.15.97:0/3259410953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 msgr2=0x55bd9e8dad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dad30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f277c0a6380 tx=0x7f277c0aaac0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 -- 172.21.15.97:0/3259410953 shutdown_connections 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55bd9e4e8d80 0x55bd9e8e7430 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dad30 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8da1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 --2- 172.21.15.97:0/3259410953 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 -- 172.21.15.97:0/3259410953 >> 172.21.15.97:0/3259410953 conn(0x55bd9e51e300 msgr2=0x55bd9e8cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f276b400640 0 client.16194 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 -- 172.21.15.97:0/3259410953 shutdown_connections 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 1 -- 172.21.15.97:0/3259410953 wait complete. 2024-07-26T07:57:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.279+0000 7f27933fc740 20 client.16194 trim_cache size 0 max 16384 2024-07-26T07:57:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.302+0000 7f27933fc740 1 Processor -- start 2024-07-26T07:57:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.302+0000 7f27933fc740 1 -- start start 2024-07-26T07:57:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.302+0000 7f27933fc740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.302+0000 7f27933fc740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f27933fc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f27933fc740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f27841ac9e0 con 0x55bd9e5012f0 2024-07-26T07:57:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f27933fc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27841ae520 con 0x55bd9e52a1b0 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f27933fc740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f27841ad0b0 con 0x55bd9e5094d0 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48724/0 (socket says 172.21.15.97:48724) 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789a00640 1 -- 172.21.15.97:0/4238070736 learned_addr learned my addr 172.21.15.97:0/4238070736 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f278a400640 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789000640 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789a00640 1 -- 172.21.15.97:0/4238070736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 msgr2=0x55bd9e8d8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789a00640 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789a00640 1 -- 172.21.15.97:0/4238070736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e52a1b0 msgr2=0x55bd9e8cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789a00640 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789a00640 1 -- 172.21.15.97:0/4238070736 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd9e8d8ad0 con 0x55bd9e5012f0 2024-07-26T07:57:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.303+0000 7f2789000640 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.304+0000 7f2789a00640 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d8050 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f277802a930 tx=0x7f277802ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487c95893489a58d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.304+0000 7f2776000640 1 -- 172.21.15.97:0/4238070736 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778035040 con 0x55bd9e5012f0 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.304+0000 7f2776000640 1 -- 172.21.15.97:0/4238070736 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2778050ad0 con 0x55bd9e5012f0 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.304+0000 7f2776000640 1 -- 172.21.15.97:0/4238070736 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277803cd00 con 0x55bd9e5012f0 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 -- 172.21.15.97:0/4238070736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 msgr2=0x55bd9e8d8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d8050 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f277802a930 tx=0x7f277802ab10 comp rx=0 tx=0).stop 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 -- 172.21.15.97:0/4238070736 shutdown_connections 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8d8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 --2- 172.21.15.97:0/4238070736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e5012f0 0x55bd9e8d8050 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 -- 172.21.15.97:0/4238070736 >> 172.21.15.97:0/4238070736 conn(0x55bd9e51e300 msgr2=0x55bd9e8d48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 -- 172.21.15.97:0/4238070736 shutdown_connections 2024-07-26T07:57:51.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.305+0000 7f27933fc740 1 -- 172.21.15.97:0/4238070736 wait complete. 2024-07-26T07:57:51.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 1 -- start start 2024-07-26T07:57:51.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 1 Processor -- start 2024-07-26T07:57:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8de8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8def80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f27841ade50 con 0x55bd9e52a1b0 2024-07-26T07:57:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f27841ad780 con 0x55bd9e5012f0 2024-07-26T07:57:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f27841aa9c0 con 0x55bd9e5094d0 2024-07-26T07:57:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.306+0000 7f27933fc740 10 client.0 _ll_register_callbacks cb 0x7f277c0a61c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48736/0 (socket says 172.21.15.97:48736) 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 -- 172.21.15.97:0/3399191524 learned_addr learned my addr 172.21.15.97:0/3399191524 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f2789a00640 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8de8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f2789000640 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8def80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 -- 172.21.15.97:0/3399191524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 msgr2=0x55bd9e8def80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8def80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 -- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 msgr2=0x55bd9e8de8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8de8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 -- 172.21.15.97:0/3399191524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd9e8e0250 con 0x55bd9e52a1b0 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f2789a00640 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8de8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f278a400640 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dfaf0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f277c0a7570 tx=0x7f277c0a9b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d428e9f44399abe server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.307+0000 7f276b400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f276b400640 1 -- 172.21.15.97:0/3399191524 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c06e070 con 0x55bd9e52a1b0 2024-07-26T07:57:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f276b400640 1 -- 172.21.15.97:0/3399191524 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f277c0a5e80 con 0x55bd9e52a1b0 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f27933fc740 1 -- 172.21.15.97:0/3399191524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2778050ad0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f276b400640 1 -- 172.21.15.97:0/3399191524 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f277c0b33b0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f27933fc740 10 client.16200 fetch_fsmap learned FSMap version 35 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f27933fc740 1 -- 172.21.15.97:0/3399191524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bd9e8e1520 con 0x55bd9e52a1b0 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f276b400640 1 -- 172.21.15.97:0/3399191524 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2778050ad0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f276b400640 1 -- 172.21.15.97:0/3399191524 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f277c0cb2a0 con 0x55bd9e52a1b0 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f27933fc740 10 client.16200 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f27933fc740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f27933fc740 20 client.16200 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.308+0000 7f27933fc740 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55bd9e4be1c0 0x55bd9e951bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.309+0000 7f27933fc740 4 client.16200 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.309+0000 7f27933fc740 1 -- 172.21.15.97:0/3399191524 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd9e83d3a0 con 0x55bd9e4be1c0 2024-07-26T07:57:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.309+0000 7f2789a00640 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55bd9e4be1c0 0x55bd9e951bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.310+0000 7f2789a00640 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55bd9e4be1c0 0x55bd9e951bd0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3369af49f17a360 server_cookie=65ad2a1080964acf in_seq=0 out_seq=0 2024-07-26T07:57:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:51.310+0000 7f276b400640 10 client.16200 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:51 smithi097 ceph-mon[32231]: pgmap v693: 97 pgs: 97 active+clean; 128 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 248 op/s 2024-07-26T07:57:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:51 smithi115 ceph-mon[35364]: pgmap v693: 97 pgs: 97 active+clean; 128 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 248 op/s 2024-07-26T07:57:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:51 smithi181 ceph-mon[35869]: pgmap v693: 97 pgs: 97 active+clean; 128 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 248 op/s 2024-07-26T07:57:53.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 -- 172.21.15.97:0/471668622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 msgr2=0x7f1f800751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 --2- 172.21.15.97:0/471668622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 0x7f1f800751e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009900 tx=0x7f1f7002a410 comp rx=0 tx=0).stop 2024-07-26T07:57:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 -- 172.21.15.97:0/471668622 shutdown_connections 2024-07-26T07:57:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 --2- 172.21.15.97:0/471668622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80113cb0 0x7f1f80116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 --2- 172.21.15.97:0/471668622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f80075800 0x7f1f80075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 --2- 172.21.15.97:0/471668622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 0x7f1f800751e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 -- 172.21.15.97:0/471668622 >> 172.21.15.97:0/471668622 conn(0x7f1f800fee70 msgr2=0x7f1f801012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 -- 172.21.15.97:0/471668622 shutdown_connections 2024-07-26T07:57:53.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.372+0000 7f1f7f200640 1 -- 172.21.15.97:0/471668622 wait complete. 2024-07-26T07:57:53.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.374+0000 7f1f7f200640 1 Processor -- start 2024-07-26T07:57:53.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.374+0000 7f1f7f200640 1 -- start start 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.374+0000 7f1f7f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80075800 0x7f1f801a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.374+0000 7f1f7f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 0x7f1f801a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.374+0000 7f1f7f200640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f80113cb0 0x7f1f801a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.374+0000 7f1f7f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1f801197a0 con 0x7f1f80076db0 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.374+0000 7f1f7f200640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80119620 con 0x7f1f80113cb0 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.374+0000 7f1f7f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80119920 con 0x7f1f80075800 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 0x7f1f801a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 0x7f1f801a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:48766/0 (socket says 172.21.15.97:48766) 2024-07-26T07:57:53.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f77e00640 1 -- 172.21.15.97:0/3306145827 learned_addr learned my addr 172.21.15.97:0/3306145827 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f7de00640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80075800 0x7f1f801a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f7e800640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f80113cb0 0x7f1f801a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f77e00640 1 -- 172.21.15.97:0/3306145827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80075800 msgr2=0x7f1f801a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f77e00640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80075800 0x7f1f801a0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f77e00640 1 -- 172.21.15.97:0/3306145827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f80113cb0 msgr2=0x7f1f801a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f77e00640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f80113cb0 0x7f1f801a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f77e00640 1 -- 172.21.15.97:0/3306145827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f801a61a0 con 0x7f1f80076db0 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.375+0000 7f1f7e800640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f80113cb0 0x7f1f801a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.376+0000 7f1f77e00640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 0x7f1f801a07d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800d770 tx=0x7f1f6800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.376+0000 7f1f76a00640 1 -- 172.21.15.97:0/3306145827 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f68017070 con 0x7f1f80076db0 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.376+0000 7f1f76a00640 1 -- 172.21.15.97:0/3306145827 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f68009c30 con 0x7f1f80076db0 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.376+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f801a6490 con 0x7f1f80076db0 2024-07-26T07:57:53.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.376+0000 7f1f76a00640 1 -- 172.21.15.97:0/3306145827 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800f620 con 0x7f1f80076db0 2024-07-26T07:57:53.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.376+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f801acbf0 con 0x7f1f80076db0 2024-07-26T07:57:53.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.378+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f80075c10 con 0x7f1f80076db0 2024-07-26T07:57:53.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.380+0000 7f1f76a00640 1 -- 172.21.15.97:0/3306145827 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f1f6801a070 con 0x7f1f80076db0 2024-07-26T07:57:53.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.382+0000 7f1f76a00640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1f640b3c50 0x7f1f640b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.383+0000 7f1f7de00640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1f640b3c50 0x7f1f640b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.383+0000 7f1f76a00640 1 -- 172.21.15.97:0/3306145827 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f1f680e36f0 con 0x7f1f80076db0 2024-07-26T07:57:53.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.384+0000 7f1f7de00640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1f640b3c50 0x7f1f640b6160 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1f700057c0 tx=0x7f1f700058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:53.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.387+0000 7f1f76a00640 1 -- 172.21.15.97:0/3306145827 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1f68009de0 con 0x7f1f80076db0 2024-07-26T07:57:53.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.634+0000 7f276b400640 1 -- 172.21.15.97:0/3399191524 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd9e83d3a0 con 0x55bd9e4be1c0 2024-07-26T07:57:53.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.634+0000 7f276b400640 10 client.16200 handle_command_reply: tid=0 2024-07-26T07:57:53.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 client.16200 shutdown 2024-07-26T07:57:53.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 -- 172.21.15.97:0/3399191524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 msgr2=0x55bd9e8dfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dfaf0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f277c0a7570 tx=0x7f277c0a9b00 comp rx=0 tx=0).stop 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 -- 172.21.15.97:0/3399191524 shutdown_connections 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55bd9e4be1c0 0x55bd9e951bd0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bd9e52a1b0 0x55bd9e8dfaf0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bd9e5094d0 0x55bd9e8def80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 --2- 172.21.15.97:0/3399191524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bd9e5012f0 0x55bd9e8de8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.635+0000 7f27933fc740 1 -- 172.21.15.97:0/3399191524 >> 172.21.15.97:0/3399191524 conn(0x55bd9e51e300 msgr2=0x55bd9e8d7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.636+0000 7f276b400640 0 client.16200 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.636+0000 7f27933fc740 1 -- 172.21.15.97:0/3399191524 shutdown_connections 2024-07-26T07:57:53.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.636+0000 7f27933fc740 1 -- 172.21.15.97:0/3399191524 wait complete. 2024-07-26T07:57:53.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.636+0000 7f27933fc740 20 client.16200 trim_cache size 0 max 16384 2024-07-26T07:57:53.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-07-26T07:57:53.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-07-26T07:57:53.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-07-26T07:57:53.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-07-26T07:57:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.637+0000 7f278ae00640 1 -- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f276c0b3bd0 msgr2=0x7f276c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.637+0000 7f278ae00640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f276c0b3bd0 0x7f276c0b60e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f27700058c0 tx=0x7f27700046c0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.637+0000 7f278ae00640 1 -- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784104930 msgr2=0x7f278419cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.637+0000 7f278ae00640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784104930 0x7f278419cb70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f277c00b970 tx=0x7f277c00be30 comp rx=0 tx=0).stop 2024-07-26T07:57:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.638+0000 7f278ae00640 1 -- 172.21.15.97:0/3807397017 shutdown_connections 2024-07-26T07:57:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.638+0000 7f278ae00640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f276c0b3bd0 0x7f276c0b60e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.638+0000 7f278ae00640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2784104930 0x7f278419cb70 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.638+0000 7f278ae00640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2784103f90 0x7f278419bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.638+0000 7f278ae00640 1 --2- 172.21.15.97:0/3807397017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2784102f10 0x7f278419b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.638+0000 7f278ae00640 1 -- 172.21.15.97:0/3807397017 >> 172.21.15.97:0/3807397017 conn(0x7f27840fe850 msgr2=0x7f2784075020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:53.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.638+0000 7f278ae00640 1 -- 172.21.15.97:0/3807397017 shutdown_connections 2024-07-26T07:57:53.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:57:53.638+0000 7f278ae00640 1 -- 172.21.15.97:0/3807397017 wait complete. 2024-07-26T07:57:53.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.649+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f8002e0c0 con 0x7f1f80076db0 2024-07-26T07:57:53.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.650+0000 7f1f76a00640 1 -- 172.21.15.97:0/3306145827 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11991 (secure 0 0 0) 0x7f1f680aac80 con 0x7f1f80076db0 2024-07-26T07:57:53.651 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:57:53.652 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":35,"btime":"2024-07-26T07:57:43:695369+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/4227989530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4227989530},{"type":"v1","addr":"172.21.15.97:6837","nonce":4227989530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:57:43.511151+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3957930666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3957930666},{"type":"v1","addr":"172.21.15.181:6837","nonce":3957930666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[24562,34517,34556,34535,34583]},"id":1}]} 2024-07-26T07:57:53.652 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 35 2024-07-26T07:57:53.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1f640b3c50 msgr2=0x7f1f640b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:53.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1f640b3c50 0x7f1f640b6160 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1f700057c0 tx=0x7f1f700058d0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 msgr2=0x7f1f801a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:53.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 0x7f1f801a07d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800d770 tx=0x7f1f6800dc30 comp rx=0 tx=0).stop 2024-07-26T07:57:53.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 shutdown_connections 2024-07-26T07:57:53.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f1f640b3c50 0x7f1f640b6160 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f80113cb0 0x7f1f801a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f80076db0 0x7f1f801a07d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 --2- 172.21.15.97:0/3306145827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80075800 0x7f1f801a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:53.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.653+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 >> 172.21.15.97:0/3306145827 conn(0x7f1f800fee70 msgr2=0x7f1f8010fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:53.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.654+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 shutdown_connections 2024-07-26T07:57:53.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:53.654+0000 7f1f7f200640 1 -- 172.21.15.97:0/3306145827 wait complete. 2024-07-26T07:57:53.993 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:53 smithi097 ceph-mon[32231]: pgmap v694: 97 pgs: 97 active+clean; 128 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 721 KiB/s wr, 103 op/s 2024-07-26T07:57:53.993 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:53 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/3306145827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:53 smithi115 ceph-mon[35364]: pgmap v694: 97 pgs: 97 active+clean; 128 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 721 KiB/s wr, 103 op/s 2024-07-26T07:57:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:53 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/3306145827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:53 smithi181 ceph-mon[35869]: pgmap v694: 97 pgs: 97 active+clean; 128 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 721 KiB/s wr, 103 op/s 2024-07-26T07:57:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:53 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/3306145827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:57:54.399 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db --set-id 4d2f40c2 --release --await' 2024-07-26T07:57:54.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db --set-id 4d2f40c2 --release --await 2024-07-26T07:57:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:57:56.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:55 smithi097 ceph-mon[32231]: pgmap v695: 97 pgs: 97 active+clean; 124 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 710 KiB/s wr, 91 op/s 2024-07-26T07:57:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:55 smithi115 ceph-mon[35364]: pgmap v695: 97 pgs: 97 active+clean; 124 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 710 KiB/s wr, 91 op/s 2024-07-26T07:57:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:55 smithi181 ceph-mon[35869]: pgmap v695: 97 pgs: 97 active+clean; 124 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 710 KiB/s wr, 91 op/s 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 -- 172.21.15.97:0/341769459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 msgr2=0x7fe0b8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/341769459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 0x7fe0b8109e00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009900 tx=0x7fe0a802a410 comp rx=0 tx=0).stop 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 -- 172.21.15.97:0/341769459 shutdown_connections 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/341769459 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0b810fc80 0x7fe0b81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/341769459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b810f330 0x7fe0b810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/341769459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 0x7fe0b8109e00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 -- 172.21.15.97:0/341769459 >> 172.21.15.97:0/341769459 conn(0x7fe0b8078ff0 msgr2=0x7fe0b8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 -- 172.21.15.97:0/341769459 shutdown_connections 2024-07-26T07:57:57.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.627+0000 7fe0b7e00640 1 -- 172.21.15.97:0/341769459 wait complete. 2024-07-26T07:57:57.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.629+0000 7fe0b7e00640 1 Processor -- start 2024-07-26T07:57:57.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.629+0000 7fe0b7e00640 1 -- start start 2024-07-26T07:57:57.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 0x7fe0b81a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0b810f330 0x7fe0b81a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b810fc80 0x7fe0b81a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b7e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8119840 con 0x7fe0b81099f0 2024-07-26T07:57:57.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b7e00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81196c0 con 0x7fe0b810fc80 2024-07-26T07:57:57.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81199c0 con 0x7fe0b810f330 2024-07-26T07:57:57.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 0x7fe0b81a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 0x7fe0b81a0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47596/0 (socket says 172.21.15.97:47596) 2024-07-26T07:57:57.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.630+0000 7fe0b6a00640 1 -- 172.21.15.97:0/3972924993 learned_addr learned my addr 172.21.15.97:0/3972924993 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:57.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.631+0000 7fe0b6000640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0b810f330 0x7fe0b81a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.631+0000 7fe0b7400640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b810fc80 0x7fe0b81a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.631+0000 7fe0b6a00640 1 -- 172.21.15.97:0/3972924993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0b810f330 msgr2=0x7fe0b81a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:57.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.631+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0b810f330 0x7fe0b81a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.631+0000 7fe0b6a00640 1 -- 172.21.15.97:0/3972924993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b810fc80 msgr2=0x7fe0b81a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:57.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.631+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b810fc80 0x7fe0b81a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.631+0000 7fe0b6a00640 1 -- 172.21.15.97:0/3972924993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b81a61c0 con 0x7fe0b81099f0 2024-07-26T07:57:57.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.632+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 0x7fe0b81a0250 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802a930 tx=0x7fe0a8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:57.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.632+0000 7fe0bca00640 1 -- 172.21.15.97:0/3972924993 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a803f070 con 0x7fe0b81099f0 2024-07-26T07:57:57.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.632+0000 7fe0bca00640 1 -- 172.21.15.97:0/3972924993 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0a803baf0 con 0x7fe0b81099f0 2024-07-26T07:57:57.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.632+0000 7fe0bca00640 1 -- 172.21.15.97:0/3972924993 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8035030 con 0x7fe0b81099f0 2024-07-26T07:57:57.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.632+0000 7fe0b7e00640 1 -- 172.21.15.97:0/3972924993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b81a6450 con 0x7fe0b81099f0 2024-07-26T07:57:57.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.632+0000 7fe0b7e00640 1 -- 172.21.15.97:0/3972924993 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b81acc10 con 0x7fe0b81099f0 2024-07-26T07:57:57.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0c4a5b740 1 Processor -- start 2024-07-26T07:57:57.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0c4a5b740 1 -- start start 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0bca00640 1 -- 172.21.15.97:0/3972924993 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7fe0a8042070 con 0x7fe0b81099f0 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc54c520 0x5598cc8f1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 0x5598cc8f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5598cc867f30 con 0x5598cc523660 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5598cc857430 con 0x5598cc52b840 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.636+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5598cc85b470 con 0x5598cc54c520 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47606/0 (socket says 172.21.15.97:47606) 2024-07-26T07:57:57.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6a00640 1 -- 172.21.15.97:0/824830758 learned_addr learned my addr 172.21.15.97:0/824830758 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:57.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b7400640 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc54c520 0x5598cc8f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6000640 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 0x5598cc8f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6a00640 1 -- 172.21.15.97:0/824830758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc54c520 msgr2=0x5598cc8f1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:57.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc54c520 0x5598cc8f1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6a00640 1 -- 172.21.15.97:0/824830758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 msgr2=0x5598cc8f1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:57.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 0x5598cc8f1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.637+0000 7fe0b6a00640 1 -- 172.21.15.97:0/824830758 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598cc5b7b00 con 0x5598cc523660 2024-07-26T07:57:57.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.638+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f16c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe0a810c4b0 tx=0x7fe0a810c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9af18f8e69f667 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:57.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.638+0000 7fe0a3e00640 1 -- 172.21.15.97:0/824830758 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a810c8e0 con 0x5598cc523660 2024-07-26T07:57:57.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.638+0000 7fe0a3e00640 1 -- 172.21.15.97:0/824830758 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0a8111030 con 0x5598cc523660 2024-07-26T07:57:57.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.638+0000 7fe0a3e00640 1 -- 172.21.15.97:0/824830758 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8104df0 con 0x5598cc523660 2024-07-26T07:57:57.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0bca00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe09c0b3c50 0x7fe09c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0bca00640 1 -- 172.21.15.97:0/3972924993 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fe0a81041e0 con 0x7fe0b81099f0 2024-07-26T07:57:57.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0b7400640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe09c0b3c50 0x7fe09c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/824830758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 msgr2=0x5598cc8f16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:57.641 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f16c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe0a810c4b0 tx=0x7fe0a810c5a0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/824830758 shutdown_connections 2024-07-26T07:57:57.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 0x5598cc8f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f16c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/824830758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc54c520 0x5598cc8f1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.639+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/824830758 >> 172.21.15.97:0/824830758 conn(0x5598cc540910 msgr2=0x5598cc8ef070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.640+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/824830758 shutdown_connections 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.640+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/824830758 wait complete. 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.640+0000 7fe0b7400640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe09c0b3c50 0x7fe09c0b6160 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0005fd0 tx=0x7fe0b0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.640+0000 7fe0c4a5b740 1 -- start start 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.641+0000 7fe0c4a5b740 1 Processor -- start 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.641+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc523660 0x5598cc8fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.641+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8fcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.641+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc8fd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.641+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5598cc6babb0 con 0x5598cc54c520 2024-07-26T07:57:57.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.641+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x5598cc7f3910 con 0x5598cc523660 2024-07-26T07:57:57.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.641+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5598cc8ff720 con 0x5598cc52b840 2024-07-26T07:57:57.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0c4a5b740 10 client.0 _ll_register_callbacks cb 0x5598cc563d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc8fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc8fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47622/0 (socket says 172.21.15.97:47622) 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b7400640 1 -- 172.21.15.97:0/602877979 learned_addr learned my addr 172.21.15.97:0/602877979 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc523660 0x5598cc8fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b6000640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8fcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b7400640 1 -- 172.21.15.97:0/602877979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 msgr2=0x5598cc8fcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b7400640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8fcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b7400640 1 -- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc523660 msgr2=0x5598cc8fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:57.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b7400640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc523660 0x5598cc8fc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b7400640 1 -- 172.21.15.97:0/602877979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598cc62e960 con 0x5598cc54c520 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b6000640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8fcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.642+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc523660 0x5598cc8fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.643+0000 7fe0b7400640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc8fd270 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe0b001ea30 tx=0x7fe0b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f169f16b01fd27 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.643+0000 7fe0a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.643+0000 7fe0a0c00640 1 -- 172.21.15.97:0/602877979 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b001e020 con 0x5598cc54c520 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.643+0000 7fe0a0c00640 1 -- 172.21.15.97:0/602877979 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fe0b0018030 con 0x5598cc54c520 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.643+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/602877979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0a8111030 con 0x5598cc54c520 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0a0c00640 1 -- 172.21.15.97:0/602877979 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe0b001e1c0 con 0x5598cc54c520 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0c4a5b740 10 client.16218 fetch_fsmap learned FSMap version 35 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/602877979 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5598cc90b370 con 0x5598cc54c520 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0a0c00640 1 -- 172.21.15.97:0/602877979 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0a8111030 con 0x5598cc54c520 2024-07-26T07:57:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0a0c00640 1 -- 172.21.15.97:0/602877979 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe0b00294b0 con 0x5598cc54c520 2024-07-26T07:57:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0c4a5b740 10 client.16218 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T07:57:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0c4a5b740 10 client.16218 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:57:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0c4a5b740 20 client.16218 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5598cc50aea0 0x5598cc90e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0c4a5b740 4 client.16218 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:57:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.644+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/602877979 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598cc869e80 con 0x5598cc50aea0 2024-07-26T07:57:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.645+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5598cc50aea0 0x5598cc90e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.646+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5598cc50aea0 0x5598cc90e900 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1daa7e30d21185d server_cookie=887634846f0d5352 in_seq=0 out_seq=0 2024-07-26T07:57:57.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.646+0000 7fe0a0c00640 10 client.16218 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:57.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0a0c00640 1 -- 172.21.15.97:0/602877979 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5598cc869e80 con 0x5598cc50aea0 2024-07-26T07:57:57.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0a0c00640 10 client.16218 handle_command_reply: tid=0 2024-07-26T07:57:57.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0c4a5b740 1 client.16218 shutdown 2024-07-26T07:57:57.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/602877979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 msgr2=0x5598cc8fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:57.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc8fd270 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe0b001ea30 tx=0x7fe0b0004a40 comp rx=0 tx=0).stop 2024-07-26T07:57:57.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/602877979 shutdown_connections 2024-07-26T07:57:57.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5598cc50aea0 0x5598cc90e900 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc8fd270 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0a0c00640 0 client.16218 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:57.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8fcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.648+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/602877979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc523660 0x5598cc8fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.649+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/602877979 >> 172.21.15.97:0/602877979 conn(0x5598cc540910 msgr2=0x5598cc8f4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:57.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.649+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/602877979 shutdown_connections 2024-07-26T07:57:57.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.649+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/602877979 wait complete. 2024-07-26T07:57:57.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.649+0000 7fe0c4a5b740 20 client.16218 trim_cache size 0 max 16384 2024-07-26T07:57:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.668+0000 7fe0c4a5b740 1 Processor -- start 2024-07-26T07:57:57.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.668+0000 7fe0c4a5b740 1 -- start start 2024-07-26T07:57:57.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc54c520 0x5598cc8f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81b5390 con 0x5598cc523660 2024-07-26T07:57:57.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81b6ed0 con 0x5598cc54c520 2024-07-26T07:57:57.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81b5a60 con 0x5598cc52b840 2024-07-26T07:57:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0b6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0b6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47638/0 (socket says 172.21.15.97:47638) 2024-07-26T07:57:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0b6a00640 1 -- 172.21.15.97:0/1741566624 learned_addr learned my addr 172.21.15.97:0/1741566624 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.669+0000 7fe0b6000640 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.670+0000 7fe0b7400640 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc54c520 0x5598cc8f7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.670+0000 7fe0b6a00640 1 -- 172.21.15.97:0/1741566624 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 msgr2=0x5598cc8f7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.670+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8f7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.670+0000 7fe0b6a00640 1 -- 172.21.15.97:0/1741566624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc54c520 msgr2=0x5598cc8f7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.670+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc54c520 0x5598cc8f7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.670+0000 7fe0b6a00640 1 -- 172.21.15.97:0/1741566624 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598cc8feba0 con 0x5598cc523660 2024-07-26T07:57:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.670+0000 7fe0b7400640 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc54c520 0x5598cc8f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:57:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.670+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f29d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe0a810c6d0 tx=0x7fe0a8104950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6331cee82948350e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:57.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.671+0000 7fe0a3400640 1 -- 172.21.15.97:0/1741566624 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8049070 con 0x5598cc523660 2024-07-26T07:57:57.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.671+0000 7fe0a3400640 1 -- 172.21.15.97:0/1741566624 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0a8111070 con 0x5598cc523660 2024-07-26T07:57:57.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.671+0000 7fe0a3400640 1 -- 172.21.15.97:0/1741566624 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80cb120 con 0x5598cc523660 2024-07-26T07:57:57.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/1741566624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 msgr2=0x5598cc8f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:57.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f29d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe0a810c6d0 tx=0x7fe0a8104950 comp rx=0 tx=0).stop 2024-07-26T07:57:57.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/1741566624 shutdown_connections 2024-07-26T07:57:57.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc54c520 0x5598cc8f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc52b840 0x5598cc8f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/1741566624 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc523660 0x5598cc8f29d0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/1741566624 >> 172.21.15.97:0/1741566624 conn(0x5598cc540910 msgr2=0x5598cc8f7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:57.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/1741566624 shutdown_connections 2024-07-26T07:57:57.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.672+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/1741566624 wait complete. 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.673+0000 7fe0c4a5b740 1 -- start start 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.673+0000 7fe0c4a5b740 1 Processor -- start 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.673+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc523660 0x5598cc901310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.673+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 0x5598cc9018b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.673+0000 7fe0c4a5b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc9701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.673+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81b6800 con 0x5598cc54c520 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.673+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81b6130 con 0x5598cc52b840 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.673+0000 7fe0c4a5b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81b3370 con 0x5598cc523660 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.674+0000 7fe0c4a5b740 10 client.0 _ll_register_callbacks cb 0x7fe0b0009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.674+0000 7fe0b7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc9701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.674+0000 7fe0b7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc9701b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47654/0 (socket says 172.21.15.97:47654) 2024-07-26T07:57:57.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.674+0000 7fe0b7400640 1 -- 172.21.15.97:0/3259889540 learned_addr learned my addr 172.21.15.97:0/3259889540 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:57:57.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.674+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc523660 0x5598cc901310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.674+0000 7fe0b6000640 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 0x5598cc9018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.675+0000 7fe0b7400640 1 -- 172.21.15.97:0/3259889540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc523660 msgr2=0x5598cc901310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:57.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.675+0000 7fe0b7400640 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc523660 0x5598cc901310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.675+0000 7fe0b7400640 1 -- 172.21.15.97:0/3259889540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 msgr2=0x5598cc9018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:57:57.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.675+0000 7fe0b7400640 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 0x5598cc9018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:57.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.675+0000 7fe0b7400640 1 -- 172.21.15.97:0/3259889540 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598cc970820 con 0x5598cc54c520 2024-07-26T07:57:57.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.675+0000 7fe0b7400640 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc9701b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009fd0 tx=0x7fe0b001e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea571f84526e57c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:57:57.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.675+0000 7fe0a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:57:57.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.676+0000 7fe0a0c00640 1 -- 172.21.15.97:0/3259889540 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0004ab0 con 0x5598cc54c520 2024-07-26T07:57:57.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.676+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/3259889540 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0a8111070 con 0x5598cc54c520 2024-07-26T07:57:57.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.676+0000 7fe0a0c00640 1 -- 172.21.15.97:0/3259889540 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7fe0b0004c50 con 0x5598cc54c520 2024-07-26T07:57:57.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.676+0000 7fe0a0c00640 1 -- 172.21.15.97:0/3259889540 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe0b000f5c0 con 0x5598cc54c520 2024-07-26T07:57:57.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.676+0000 7fe0c4a5b740 10 client.16224 fetch_fsmap learned FSMap version 35 2024-07-26T07:57:57.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.676+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/3259889540 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5598cc977c50 con 0x5598cc54c520 2024-07-26T07:57:57.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.676+0000 7fe0a0c00640 1 -- 172.21.15.97:0/3259889540 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0a8111070 con 0x5598cc54c520 2024-07-26T07:57:57.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.677+0000 7fe0a0c00640 1 -- 172.21.15.97:0/3259889540 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe0b0005310 con 0x5598cc54c520 2024-07-26T07:57:57.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.677+0000 7fe0c4a5b740 10 client.16224 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T07:57:57.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.677+0000 7fe0c4a5b740 10 client.16224 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:57:57.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.677+0000 7fe0c4a5b740 20 client.16224 populate_metadata read hostname 'smithi097' 2024-07-26T07:57:57.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.677+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5598cc4e0250 0x5598cc97a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:57:57.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.677+0000 7fe0c4a5b740 4 client.16224 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "4d2f40c2", "release": true, "await": true}] 2024-07-26T07:57:57.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.677+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/3259889540 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "quiesce db", "set_id": "4d2f40c2", "release": true, "await": true}) -- 0x5598cc865920 con 0x5598cc4e0250 2024-07-26T07:57:57.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.677+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5598cc4e0250 0x5598cc97a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:57:57.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.678+0000 7fe0b6a00640 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5598cc4e0250 0x5598cc97a3b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e548d5548c5650b server_cookie=824fb1d8730d41a5 in_seq=0 out_seq=0 2024-07-26T07:57:57.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:57.678+0000 7fe0a0c00640 10 client.16224 ms_handle_connect on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:57 smithi097 ceph-mon[32231]: pgmap v696: 97 pgs: 97 active+clean; 120 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 784 KiB/s wr, 50 op/s 2024-07-26T07:57:58.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.024+0000 7fe0a0c00640 1 -- 172.21.15.97:0/3259889540 <== mds.0 v2:172.21.15.115:6838/1903638498 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5598cc865920 con 0x5598cc4e0250 2024-07-26T07:57:58.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.024+0000 7fe0a0c00640 10 client.16224 handle_command_reply: tid=0 2024-07-26T07:57:58.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.024+0000 7fe0c4a5b740 1 client.16224 shutdown 2024-07-26T07:57:58.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/3259889540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 msgr2=0x5598cc9701b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:58.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc9701b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009fd0 tx=0x7fe0b001e3c0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/3259889540 shutdown_connections 2024-07-26T07:57:58.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x5598cc4e0250 0x5598cc97a3b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5598cc54c520 0x5598cc9701b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5598cc52b840 0x5598cc9018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0c4a5b740 1 --2- 172.21.15.97:0/3259889540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598cc523660 0x5598cc901310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/3259889540 >> 172.21.15.97:0/3259889540 conn(0x5598cc540910 msgr2=0x5598cc8f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:58.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.025+0000 7fe0a0c00640 0 client.16224 ms_handle_reset on v2:172.21.15.115:6838/1903638498 2024-07-26T07:57:58.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.026+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/3259889540 shutdown_connections 2024-07-26T07:57:58.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.026+0000 7fe0c4a5b740 1 -- 172.21.15.97:0/3259889540 wait complete. 2024-07-26T07:57:58.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.026+0000 7fe0c4a5b740 20 client.16224 trim_cache size 0 max 16384 2024-07-26T07:57:58.029 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-07-26T07:57:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 35, 2024-07-26T07:57:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "leader": 34583, 2024-07-26T07:57:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "set_version": 20, 2024-07-26T07:57:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "sets": { 2024-07-26T07:57:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "4d2f40c2": { 2024-07-26T07:57:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "version": 20, 2024-07-26T07:57:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "age_ref": 0.0, 2024-07-26T07:57:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:57:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:57:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:57:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-07-26T07:57:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "timeout": 90.0, 2024-07-26T07:57:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "expiration": 141.9, 2024-07-26T07:57:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "members": { 2024-07-26T07:57:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "file:/": { 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "excluded": false, 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "state": { 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RELEASED", 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-07-26T07:57:58.032 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-07-26T07:57:58.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 -- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe09c0b3c50 msgr2=0x7fe09c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:58.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe09c0b3c50 0x7fe09c0b6160 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0005fd0 tx=0x7fe0b0009500 comp rx=0 tx=0).stop 2024-07-26T07:57:58.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 -- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 msgr2=0x7fe0b81a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:57:58.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 0x7fe0b81a0250 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802a930 tx=0x7fe0a8003b90 comp rx=0 tx=0).stop 2024-07-26T07:57:58.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 -- 172.21.15.97:0/3972924993 shutdown_connections 2024-07-26T07:57:58.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7fe09c0b3c50 0x7fe09c0b6160 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b810fc80 0x7fe0b81a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0b810f330 0x7fe0b81a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 --2- 172.21.15.97:0/3972924993 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0b81099f0 0x7fe0b81a0250 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:57:58.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.028+0000 7fe0b7e00640 1 -- 172.21.15.97:0/3972924993 >> 172.21.15.97:0/3972924993 conn(0x7fe0b8078ff0 msgr2=0x7fe0b80797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:57:58.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.029+0000 7fe0b7e00640 1 -- 172.21.15.97:0/3972924993 shutdown_connections 2024-07-26T07:57:58.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:57:58.029+0000 7fe0b7e00640 1 -- 172.21.15.97:0/3972924993 wait complete. 2024-07-26T07:57:58.223 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:57 smithi115 ceph-mon[35364]: pgmap v696: 97 pgs: 97 active+clean; 120 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 784 KiB/s wr, 50 op/s 2024-07-26T07:57:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:57 smithi181 ceph-mon[35869]: pgmap v696: 97 pgs: 97 active+clean; 120 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 784 KiB/s wr, 50 op/s 2024-07-26T07:57:58.825 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '4d2f40c2', total seconds elapsed: 30.2 2024-07-26T07:57:58.825 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.95 2024-07-26T07:57:58.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T07:57:59.346 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:57:59.477 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T07:57:59.478 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: *** Caught signal (Aborted) ** 2024-07-26T07:57:59.478 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: in thread 7f6984e00640 thread_name:ms_dispatch 2024-07-26T07:57:59.478 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:57:59.479 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f698b03e6f0] 2024-07-26T07:57:59.479 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f698b08b94c] 2024-07-26T07:57:59.479 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 3: raise() 2024-07-26T07:57:59.479 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 4: abort() 2024-07-26T07:57:59.479 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 5: /lib64/libc.so.6(+0x2871b) [0x7f698b02871b] 2024-07-26T07:57:59.480 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 6: /lib64/libc.so.6(+0x37386) [0x7f698b037386] 2024-07-26T07:57:59.480 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555ad2c966b9] 2024-07-26T07:57:59.480 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555ad2e5480c] 2024-07-26T07:57:59.480 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x555ad2da2cde] 2024-07-26T07:57:59.481 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x555ad2d64763] 2024-07-26T07:57:59.481 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555ad2d65a1b] 2024-07-26T07:57:59.481 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555ad2d49752] 2024-07-26T07:57:59.482 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555ad2c97a56] 2024-07-26T07:57:59.482 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555ad2c9b637] 2024-07-26T07:57:59.482 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555ad2c9bf5e] 2024-07-26T07:57:59.483 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555ad2c7f03a] 2024-07-26T07:57:59.483 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f698bb9b407] 2024-07-26T07:57:59.483 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 18: (DispatchQueue::entry()+0x60f) [0x7f698bb9bcf9] 2024-07-26T07:57:59.483 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f698bc785a1] 2024-07-26T07:57:59.484 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 20: (Thread::entry_wrapper()+0x43) [0x7f698ba25cdd] 2024-07-26T07:57:59.484 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 21: (Thread::_entry_func(void*)+0xd) [0x7f698ba25cf9] 2024-07-26T07:57:59.484 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 22: /lib64/libc.so.6(+0x89c02) [0x7f698b089c02] 2024-07-26T07:57:59.484 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f698b10ec40] 2024-07-26T07:57:59.485 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:57:59.006+0000 7f6984e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:57:59.485 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: in thread 7f6984e00640 thread_name:ms_dispatch 2024-07-26T07:57:59.485 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:57:59.485 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:57:59.485 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f698b03e6f0] 2024-07-26T07:57:59.486 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f698b08b94c] 2024-07-26T07:57:59.486 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 3: raise() 2024-07-26T07:57:59.486 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 4: abort() 2024-07-26T07:57:59.486 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 5: /lib64/libc.so.6(+0x2871b) [0x7f698b02871b] 2024-07-26T07:57:59.486 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 6: /lib64/libc.so.6(+0x37386) [0x7f698b037386] 2024-07-26T07:57:59.486 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555ad2c966b9] 2024-07-26T07:57:59.486 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555ad2e5480c] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x555ad2da2cde] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x555ad2d64763] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555ad2d65a1b] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555ad2d49752] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555ad2c97a56] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555ad2c9b637] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555ad2c9bf5e] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555ad2c7f03a] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f698bb9b407] 2024-07-26T07:57:59.487 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 18: (DispatchQueue::entry()+0x60f) [0x7f698bb9bcf9] 2024-07-26T07:57:59.488 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f698bc785a1] 2024-07-26T07:57:59.488 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 20: (Thread::entry_wrapper()+0x43) [0x7f698ba25cdd] 2024-07-26T07:57:59.488 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 21: (Thread::_entry_func(void*)+0xd) [0x7f698ba25cf9] 2024-07-26T07:57:59.488 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 22: /lib64/libc.so.6(+0x89c02) [0x7f698b089c02] 2024-07-26T07:57:59.488 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f698b10ec40] 2024-07-26T07:57:59.489 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:57:59.489 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:57:59.489 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 0> 2024-07-26T07:57:59.006+0000 7f6984e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:57:59.489 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: in thread 7f6984e00640 thread_name:ms_dispatch 2024-07-26T07:57:59.489 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:57:59.489 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:57:59.490 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f698b03e6f0] 2024-07-26T07:57:59.490 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f698b08b94c] 2024-07-26T07:57:59.490 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 3: raise() 2024-07-26T07:57:59.490 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 4: abort() 2024-07-26T07:57:59.490 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 5: /lib64/libc.so.6(+0x2871b) [0x7f698b02871b] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 6: /lib64/libc.so.6(+0x37386) [0x7f698b037386] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555ad2c966b9] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555ad2e5480c] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x555ad2da2cde] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x555ad2d64763] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555ad2d65a1b] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555ad2d49752] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555ad2c97a56] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555ad2c9b637] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555ad2c9bf5e] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555ad2c7f03a] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f698bb9b407] 2024-07-26T07:57:59.491 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 18: (DispatchQueue::entry()+0x60f) [0x7f698bb9bcf9] 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f698bc785a1] 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 20: (Thread::entry_wrapper()+0x43) [0x7f698ba25cdd] 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 21: (Thread::_entry_func(void*)+0xd) [0x7f698ba25cf9] 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 22: /lib64/libc.so.6(+0x89c02) [0x7f698b089c02] 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f698b10ec40] 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 0> 2024-07-26T07:57:59.006+0000 7f6984e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: in thread 7f6984e00640 thread_name:ms_dispatch 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f698b03e6f0] 2024-07-26T07:57:59.492 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f698b08b94c] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 3: raise() 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 4: abort() 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 5: /lib64/libc.so.6(+0x2871b) [0x7f698b02871b] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 6: /lib64/libc.so.6(+0x37386) [0x7f698b037386] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555ad2c966b9] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555ad2e5480c] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x555ad2da2cde] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x555ad2d64763] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555ad2d65a1b] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555ad2d49752] 2024-07-26T07:57:59.493 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555ad2c97a56] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555ad2c9b637] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555ad2c9bf5e] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555ad2c7f03a] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f698bb9b407] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 18: (DispatchQueue::entry()+0x60f) [0x7f698bb9bcf9] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f698bc785a1] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 20: (Thread::entry_wrapper()+0x43) [0x7f698ba25cdd] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 21: (Thread::_entry_func(void*)+0xd) [0x7f698ba25cf9] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 22: /lib64/libc.so.6(+0x89c02) [0x7f698b089c02] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f698b10ec40] 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[66126]: 2024-07-26T07:57:59.494 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 podman[69001]: 2024-07-26 07:57:59.157158639 +0000 UTC m=+0.036919478 container died ec7f3da47c46fb79b7a26167dd751bf927c455d2904ae81720c431825b807cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:58:00.031 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-mon[35364]: pgmap v697: 97 pgs: 97 active+clean; 120 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 785 KiB/s wr, 49 op/s 2024-07-26T07:58:00.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:57:59 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:00.032 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:57:59 smithi115 podman[69001]: 2024-07-26 07:57:59.994356763 +0000 UTC m=+0.874117637 container remove ec7f3da47c46fb79b7a26167dd751bf927c455d2904ae81720c431825b807cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, ceph=True) 2024-07-26T07:58:00.032 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:00 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T07:58:00.203 INFO:tasks.workunit.client.0.smithi097.stdout: 37 1575 7 1135 4 1042 10 2024-07-26T07:58:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:59 smithi181 ceph-mon[35869]: pgmap v697: 97 pgs: 97 active+clean; 120 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 785 KiB/s wr, 49 op/s 2024-07-26T07:58:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:57:59 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:59 smithi097 ceph-mon[32231]: pgmap v697: 97 pgs: 97 active+clean; 120 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 785 KiB/s wr, 49 op/s 2024-07-26T07:58:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:57:59 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:00.974 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:00 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T07:58:00.974 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:00 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.b.service: Consumed 6min 26.226s CPU time. 2024-07-26T07:58:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.577+0000 7f69fbe00640 1 -- 172.21.15.97:0/2798136859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc10f940 msgr2=0x7f69fc111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.577+0000 7f69fbe00640 1 --2- 172.21.15.97:0/2798136859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc10f940 0x7f69fc111d80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009900 tx=0x7f69f402a410 comp rx=0 tx=0).stop 2024-07-26T07:58:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.577+0000 7f69fbe00640 1 -- 172.21.15.97:0/2798136859 shutdown_connections 2024-07-26T07:58:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.577+0000 7f69fbe00640 1 --2- 172.21.15.97:0/2798136859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc10f940 0x7f69fc111d80 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.577+0000 7f69fbe00640 1 --2- 172.21.15.97:0/2798136859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc075b30 0x7f69fc10f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.577+0000 7f69fbe00640 1 --2- 172.21.15.97:0/2798136859 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69fc0751e0 0x7f69fc0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.577+0000 7f69fbe00640 1 -- 172.21.15.97:0/2798136859 >> 172.21.15.97:0/2798136859 conn(0x7f69fc0fee70 msgr2=0x7f69fc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:58:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.577+0000 7f69fbe00640 1 -- 172.21.15.97:0/2798136859 shutdown_connections 2024-07-26T07:58:01.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.578+0000 7f69fbe00640 1 -- 172.21.15.97:0/2798136859 wait complete. 2024-07-26T07:58:01.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.580+0000 7f69fbe00640 1 Processor -- start 2024-07-26T07:58:01.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.580+0000 7f69fbe00640 1 -- start start 2024-07-26T07:58:01.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.580+0000 7f69fbe00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69fc0751e0 0x7f69fc19be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:01.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.580+0000 7f69fbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc075b30 0x7f69fc19c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:01.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.580+0000 7f69fbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc10f940 0x7f69fc1a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:01.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.580+0000 7f69fbe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc1153f0 con 0x7f69fc075b30 2024-07-26T07:58:01.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.580+0000 7f69fbe00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc115270 con 0x7f69fc0751e0 2024-07-26T07:58:01.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.580+0000 7f69fbe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc115570 con 0x7f69fc10f940 2024-07-26T07:58:01.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc075b30 0x7f69fc19c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:01.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc075b30 0x7f69fc19c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47660/0 (socket says 172.21.15.97:47660) 2024-07-26T07:58:01.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fa000640 1 -- 172.21.15.97:0/1767871671 learned_addr learned my addr 172.21.15.97:0/1767871671 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:58:01.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fb400640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc10f940 0x7f69fc1a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:01.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69faa00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69fc0751e0 0x7f69fc19be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fa000640 1 -- 172.21.15.97:0/1767871671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc10f940 msgr2=0x7f69fc1a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fa000640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc10f940 0x7f69fc1a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fa000640 1 -- 172.21.15.97:0/1767871671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69fc0751e0 msgr2=0x7f69fc19be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fa000640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69fc0751e0 0x7f69fc19be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fa000640 1 -- 172.21.15.97:0/1767871671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69fc1a1d70 con 0x7f69fc075b30 2024-07-26T07:58:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69faa00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69fc0751e0 0x7f69fc19be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:58:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.581+0000 7f69fb400640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc10f940 0x7f69fc1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:58:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.582+0000 7f69fa000640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc075b30 0x7f69fc19c3a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f69e400eaa0 tx=0x7f69e400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:01.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.582+0000 7f69f3400640 1 -- 172.21.15.97:0/1767871671 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4016070 con 0x7f69fc075b30 2024-07-26T07:58:01.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.582+0000 7f69f3400640 1 -- 172.21.15.97:0/1767871671 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69e400ce10 con 0x7f69fc075b30 2024-07-26T07:58:01.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.582+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69fc1a2060 con 0x7f69fc075b30 2024-07-26T07:58:01.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.582+0000 7f69f3400640 1 -- 172.21.15.97:0/1767871671 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4011470 con 0x7f69fc075b30 2024-07-26T07:58:01.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.582+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69fc1a87c0 con 0x7f69fc075b30 2024-07-26T07:58:01.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.584+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d80052c0 con 0x7f69fc075b30 2024-07-26T07:58:01.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.588+0000 7f69f3400640 1 -- 172.21.15.97:0/1767871671 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f69e4019070 con 0x7f69fc075b30 2024-07-26T07:58:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.589+0000 7f69f3400640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69dc0b3bd0 0x7f69dc0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.589+0000 7f69f3400640 1 -- 172.21.15.97:0/1767871671 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f69e40e3cb0 con 0x7f69fc075b30 2024-07-26T07:58:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.590+0000 7f69faa00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69dc0b3bd0 0x7f69dc0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:01.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.590+0000 7f69faa00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69dc0b3bd0 0x7f69dc0b60e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0058c0 tx=0x7f69ec005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:01.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.594+0000 7f69f3400640 1 -- 172.21.15.97:0/1767871671 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f69e40ab240 con 0x7f69fc075b30 2024-07-26T07:58:01.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.835+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69d80055b0 con 0x7f69fc075b30 2024-07-26T07:58:01.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.836+0000 7f69f3400640 1 -- 172.21.15.97:0/1767871671 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11994 (secure 0 0 0) 0x7f69e40aab40 con 0x7f69fc075b30 2024-07-26T07:58:01.838 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-07-26T07:58:01.838 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":36,"btime":"2024-07-26T07:57:58:758974+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14685,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1716973566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1716973566},{"type":"v1","addr":"172.21.15.97:6835","nonce":1716973566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":16017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/4227989530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":4227989530},{"type":"v1","addr":"172.21.15.97:6837","nonce":4227989530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T07:47:48.482950+0000","modified":"2024-07-26T07:57:58.501655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34583,"mds_1":34556,"mds_2":34535,"mds_3":34517,"mds_4":24562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14748":{"gid":14748,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/4174464953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":4174464953},{"type":"v1","addr":"172.21.15.97:6839","nonce":4174464953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14760":{"gid":14760,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6841/4022202837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":4022202837},{"type":"v1","addr":"172.21.15.97:6841","nonce":4022202837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1511551515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1511551515},{"type":"v1","addr":"172.21.15.181:6833","nonce":1511551515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/4001900111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4001900111},{"type":"v1","addr":"172.21.15.181:6835","nonce":4001900111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"i","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.181:6839/4258683328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":4258683328},{"type":"v1","addr":"172.21.15.181:6839","nonce":4258683328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/3957930666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3957930666},{"type":"v1","addr":"172.21.15.181:6837","nonce":3957930666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34517":{"gid":34517,"name":"k","rank":3,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.115:6833/1028321962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1028321962},{"type":"v1","addr":"172.21.15.115:6833","nonce":1028321962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34535":{"gid":34535,"name":"e","rank":2,"incarnation":20,"state":"up:active","state_seq":18,"addr":"172.21.15.115:6835/4018170367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":4018170367},{"type":"v1","addr":"172.21.15.115:6835","nonce":4018170367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34556":{"gid":34556,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.115:6837/3478625369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":3478625369},{"type":"v1","addr":"172.21.15.115:6837","nonce":3478625369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34583":{"gid":34583,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.115:6839/1903638498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":1903638498},{"type":"v1","addr":"172.21.15.115:6839","nonce":1903638498}]},"join_fscid":-1,"export_targets":[3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34583,"qdb_cluster":[34583,34535,34556,34517,24562]},"id":1}]} 2024-07-26T07:58:01.838 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 36 2024-07-26T07:58:01.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.838+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69dc0b3bd0 msgr2=0x7f69dc0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:01.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69dc0b3bd0 0x7f69dc0b60e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0058c0 tx=0x7f69ec005850 comp rx=0 tx=0).stop 2024-07-26T07:58:01.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc075b30 msgr2=0x7f69fc19c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:01.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc075b30 0x7f69fc19c3a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f69e400eaa0 tx=0x7f69e400ef60 comp rx=0 tx=0).stop 2024-07-26T07:58:01.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 shutdown_connections 2024-07-26T07:58:01.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f69dc0b3bd0 0x7f69dc0b60e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc10f940 0x7f69fc1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69fc075b30 0x7f69fc19c3a0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 --2- 172.21.15.97:0/1767871671 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69fc0751e0 0x7f69fc19be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:01.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 >> 172.21.15.97:0/1767871671 conn(0x7f69fc0fee70 msgr2=0x7f69fc1094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:58:01.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 shutdown_connections 2024-07-26T07:58:01.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:01.839+0000 7f69fbe00640 1 -- 172.21.15.97:0/1767871671 wait complete. 2024-07-26T07:58:01.928 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:01 smithi097 ceph-mon[32231]: pgmap v698: 97 pgs: 97 active+clean; 120 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 801 KiB/s wr, 51 op/s 2024-07-26T07:58:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:01 smithi115 ceph-mon[35364]: pgmap v698: 97 pgs: 97 active+clean; 120 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 801 KiB/s wr, 51 op/s 2024-07-26T07:58:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:01 smithi181 ceph-mon[35869]: pgmap v698: 97 pgs: 97 active+clean; 120 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 801 KiB/s wr, 51 op/s 2024-07-26T07:58:02.556 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.34583 quiesce db / --timeout 90 --expiration 141.95 --await-for 45.0' 2024-07-26T07:58:02.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 9621a072-4b21-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.34583 quiesce db / --timeout 90 --expiration 141.95 --await-for 45.0 2024-07-26T07:58:03.001 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/9621a072-4b21-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T07:58:03.151 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:02 smithi097 ceph-mon[32231]: from='client.? 172.21.15.97:0/1767871671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:58:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:02 smithi115 ceph-mon[35364]: from='client.? 172.21.15.97:0/1767871671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:58:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:02 smithi181 ceph-mon[35869]: from='client.? 172.21.15.97:0/1767871671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T07:58:04.184 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:03 smithi097 ceph-mon[32231]: pgmap v699: 97 pgs: 97 active+clean; 121 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 453 KiB/s wr, 52 op/s 2024-07-26T07:58:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:03 smithi115 ceph-mon[35364]: pgmap v699: 97 pgs: 97 active+clean; 121 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 453 KiB/s wr, 52 op/s 2024-07-26T07:58:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:03 smithi181 ceph-mon[35869]: pgmap v699: 97 pgs: 97 active+clean; 121 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 453 KiB/s wr, 52 op/s 2024-07-26T07:58:05.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 -- 172.21.15.97:0/3924421685 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f195010f980 msgr2=0x7f1950111da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:05.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 --2- 172.21.15.97:0/3924421685 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f195010f980 0x7f1950111da0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1948009ca0 tx=0x7f194802a8c0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 -- 172.21.15.97:0/3924421685 shutdown_connections 2024-07-26T07:58:05.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 --2- 172.21.15.97:0/3924421685 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f195010f980 0x7f1950111da0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 --2- 172.21.15.97:0/3924421685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1950075ad0 0x7f195010f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 --2- 172.21.15.97:0/3924421685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1950075180 0x7f1950075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 -- 172.21.15.97:0/3924421685 >> 172.21.15.97:0/3924421685 conn(0x7f19500fee10 msgr2=0x7f1950101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:58:05.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 -- 172.21.15.97:0/3924421685 shutdown_connections 2024-07-26T07:58:05.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.202+0000 7f1956c00640 1 -- 172.21.15.97:0/3924421685 wait complete. 2024-07-26T07:58:05.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.203+0000 7f1956c00640 1 Processor -- start 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.203+0000 7f1956c00640 1 -- start start 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1956c00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1950075180 0x7f195019be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1956c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1950075ad0 0x7f195019c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1956c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f195010f980 0x7f19501a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1956c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f19501153f0 con 0x7f1950075ad0 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1956c00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7f1950115270 con 0x7f1950075180 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1956c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1950115570 con 0x7f195010f980 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1954e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1950075ad0 0x7f195019c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1954e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1950075ad0 0x7f195019c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47676/0 (socket says 172.21.15.97:47676) 2024-07-26T07:58:05.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1954e00640 1 -- 172.21.15.97:0/1409471545 learned_addr learned my addr 172.21.15.97:0/1409471545 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:58:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1955800640 1 --2- 172.21.15.97:0/1409471545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1950075180 0x7f195019be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.204+0000 7f1956200640 1 --2- 172.21.15.97:0/1409471545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f195010f980 0x7f19501a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f1954e00640 1 -- 172.21.15.97:0/1409471545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f195010f980 msgr2=0x7f19501a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f1954e00640 1 --2- 172.21.15.97:0/1409471545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f195010f980 0x7f19501a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f1954e00640 1 -- 172.21.15.97:0/1409471545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1950075180 msgr2=0x7f195019be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f1954e00640 1 --2- 172.21.15.97:0/1409471545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1950075180 0x7f195019be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f1954e00640 1 -- 172.21.15.97:0/1409471545 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19501a1d90 con 0x7f1950075ad0 2024-07-26T07:58:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f1954e00640 1 --2- 172.21.15.97:0/1409471545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1950075ad0 0x7f195019c3c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f193c00d740 tx=0x7f193c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:05.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f194e000640 1 -- 172.21.15.97:0/1409471545 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c017070 con 0x7f1950075ad0 2024-07-26T07:58:05.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f194e000640 1 -- 172.21.15.97:0/1409471545 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f193c00de50 con 0x7f1950075ad0 2024-07-26T07:58:05.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f194e000640 1 -- 172.21.15.97:0/1409471545 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c00f620 con 0x7f1950075ad0 2024-07-26T07:58:05.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f1956c00640 1 -- 172.21.15.97:0/1409471545 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19501a2080 con 0x7f1950075ad0 2024-07-26T07:58:05.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.205+0000 7f1956c00640 1 -- 172.21.15.97:0/1409471545 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19501a87e0 con 0x7f1950075ad0 2024-07-26T07:58:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.209+0000 7f195eeb6740 1 Processor -- start 2024-07-26T07:58:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.209+0000 7f195eeb6740 1 -- start start 2024-07-26T07:58:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.209+0000 7f194e000640 1 -- 172.21.15.97:0/1409471545 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7f193c01a070 con 0x7f1950075ad0 2024-07-26T07:58:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f195eeb6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d08c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f195eeb6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 0x56373d08c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f195eeb6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d08cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f195eeb6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56373d0008a0 con 0x56373ccc4b70 2024-07-26T07:58:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f195eeb6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56373cfefda0 con 0x56373cce5850 2024-07-26T07:58:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f195eeb6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56373cff3de0 con 0x56373ccbc990 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f1954e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d08cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f1954e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d08cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47684/0 (socket says 172.21.15.97:47684) 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f1954e00640 1 -- 172.21.15.97:0/777992793 learned_addr learned my addr 172.21.15.97:0/777992793 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f1956200640 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d08c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.210+0000 7f1955800640 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 0x56373d08c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f1954e00640 1 -- 172.21.15.97:0/777992793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 msgr2=0x56373d08c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f1954e00640 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 0x56373d08c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f1954e00640 1 -- 172.21.15.97:0/777992793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 msgr2=0x56373d08c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f1954e00640 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d08c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f1954e00640 1 -- 172.21.15.97:0/777992793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56373cf84b70 con 0x56373ccc4b70 2024-07-26T07:58:05.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f1956200640 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d08c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T07:58:05.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f1954e00640 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d08cf00 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f193c0e9770 tx=0x7f193c0e9c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea083177d7fa87 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f194e000640 1 --2- 172.21.15.97:0/1409471545 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19380b3c40 0x7f19380b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.211+0000 7f1942a00640 1 -- 172.21.15.97:0/777992793 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c0f6070 con 0x56373ccc4b70 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.212+0000 7f1942a00640 1 -- 172.21.15.97:0/777992793 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f193c0b7330 con 0x56373ccc4b70 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.212+0000 7f1942a00640 1 -- 172.21.15.97:0/777992793 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c0a9c70 con 0x56373ccc4b70 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.212+0000 7f194e000640 1 -- 172.21.15.97:0/1409471545 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f193c0e3670 con 0x7f1950075ad0 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.212+0000 7f1956200640 1 --2- 172.21.15.97:0/1409471545 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19380b3c40 0x7f19380b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 -- 172.21.15.97:0/777992793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 msgr2=0x56373d08cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d08cf00 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f193c0e9770 tx=0x7f193c0e9c30 comp rx=0 tx=0).stop 2024-07-26T07:58:05.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f1956200640 1 --2- 172.21.15.97:0/1409471545 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7f19380b3c40 0x7f19380b6150 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1948005cc0 tx=0x7f1948005c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:05.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 -- 172.21.15.97:0/777992793 shutdown_connections 2024-07-26T07:58:05.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d08cf00 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 0x56373d08c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 --2- 172.21.15.97:0/777992793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d08c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 -- 172.21.15.97:0/777992793 >> 172.21.15.97:0/777992793 conn(0x56373ccd9c40 msgr2=0x56373d08a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:58:05.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 -- 172.21.15.97:0/777992793 shutdown_connections 2024-07-26T07:58:05.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.213+0000 7f195eeb6740 1 -- 172.21.15.97:0/777992793 wait complete. 2024-07-26T07:58:05.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.214+0000 7f195eeb6740 1 -- start start 2024-07-26T07:58:05.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.214+0000 7f195eeb6740 1 Processor -- start 2024-07-26T07:58:05.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.214+0000 7f195eeb6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 0x56373d097f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.214+0000 7f195eeb6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d098440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.214+0000 7f195eeb6740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f195eeb6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56373ce537b0 con 0x56373ccc4b70 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f195eeb6740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x56373cf8c380 con 0x56373cce5850 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f195eeb6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56373d09ac60 con 0x56373ccbc990 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f195eeb6740 10 client.0 _ll_register_callbacks cb 0x56373cf9b830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1954e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d098440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1954e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d098440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47700/0 (socket says 172.21.15.97:47700) 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1954e00640 1 -- 172.21.15.97:0/3705515786 learned_addr learned my addr 172.21.15.97:0/3705515786 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 0x56373d097f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1956200640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1954e00640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 msgr2=0x56373d097f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1954e00640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56373ccbc990 0x56373d097f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1954e00640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 msgr2=0x56373d0a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1954e00640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d0a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1954e00640 1 -- 172.21.15.97:0/3705515786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56373cd5fba0 con 0x56373ccc4b70 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.215+0000 7f1956200640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56373cce5850 0x56373d0a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.216+0000 7f1954e00640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56373ccc4b70 0x56373d098440 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f193c0aa200 tx=0x7f193c009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4487de87565c8f2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.216+0000 7f1937400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.216+0000 7f1937400640 1 -- 172.21.15.97:0/3705515786 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c0ced80 con 0x56373ccc4b70 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.216+0000 7f1937400640 1 -- 172.21.15.97:0/3705515786 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7f193c009c30 con 0x56373ccc4b70 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.216+0000 7f195eeb6740 1 -- 172.21.15.97:0/3705515786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f193c0b7330 con 0x56373ccc4b70 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.216+0000 7f1937400640 1 -- 172.21.15.97:0/3705515786 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f193c021030 con 0x56373ccc4b70 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.216+0000 7f195eeb6740 10 client.16242 fetch_fsmap learned FSMap version 36 2024-07-26T07:58:05.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.216+0000 7f195eeb6740 1 -- 172.21.15.97:0/3705515786 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56373d0a6b60 con 0x56373ccc4b70 2024-07-26T07:58:05.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f1937400640 1 -- 172.21.15.97:0/3705515786 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f193c0b7330 con 0x56373ccc4b70 2024-07-26T07:58:05.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f1937400640 1 -- 172.21.15.97:0/3705515786 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f193c0a6790 con 0x56373ccc4b70 2024-07-26T07:58:05.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f195eeb6740 10 client.16242 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T07:58:05.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f195eeb6740 10 client.16242 resolve_mds: validated gid 34583 aka daemon mds.b 2024-07-26T07:58:05.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f195eeb6740 20 client.16242 populate_metadata read hostname 'smithi097' 2024-07-26T07:58:05.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f195eeb6740 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:05.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f195eeb6740 4 client.16242 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:58:05.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f195eeb6740 1 -- 172.21.15.97:0/3705515786 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56373d0027f0 con 0x56373cca41d0 2024-07-26T07:58:05.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:05.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f1937400640 1 client.16242 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:05.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.217+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T07:58:05.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.418+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:05.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.418+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T07:58:05.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.418+0000 7f1937400640 1 client.16242 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:05.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.819+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:05.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.819+0000 7f1937400640 1 client.16242 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:05.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:05.819+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T07:58:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:05 smithi115 ceph-mon[35364]: pgmap v700: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 450 KiB/s wr, 53 op/s 2024-07-26T07:58:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:05 smithi181 ceph-mon[35869]: pgmap v700: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 450 KiB/s wr, 53 op/s 2024-07-26T07:58:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:05 smithi097 ceph-mon[32231]: pgmap v700: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 450 KiB/s wr, 53 op/s 2024-07-26T07:58:06.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:06.621+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:06.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:06.621+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T07:58:06.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:06.621+0000 7f1937400640 1 client.16242 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.867+0000 7ff3f5a00640 1 -- 172.21.15.97:0/3020521023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3f01061a0 msgr2=0x7ff3f01065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.867+0000 7ff3f5a00640 1 --2- 172.21.15.97:0/3020521023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3f01061a0 0x7ff3f01065b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009900 tx=0x7ff3d802a3f0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.867+0000 7ff3f5a00640 1 -- 172.21.15.97:0/3020521023 shutdown_connections 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.867+0000 7ff3f5a00640 1 --2- 172.21.15.97:0/3020521023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f0106b40 0x7ff3f010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.867+0000 7ff3f5a00640 1 --2- 172.21.15.97:0/3020521023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3f01061a0 0x7ff3f01065b0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.867+0000 7ff3f5a00640 1 --2- 172.21.15.97:0/3020521023 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3f0105120 0x7ff3f0105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.867+0000 7ff3f5a00640 1 -- 172.21.15.97:0/3020521023 >> 172.21.15.97:0/3020521023 conn(0x7ff3f0100a80 msgr2=0x7ff3f0102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.867+0000 7ff3f5a00640 1 -- 172.21.15.97:0/3020521023 shutdown_connections 2024-07-26T07:58:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.868+0000 7ff3f5a00640 1 -- 172.21.15.97:0/3020521023 wait complete. 2024-07-26T07:58:06.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3f5a00640 1 Processor -- start 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3f5a00640 1 -- start start 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3f5a00640 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f0105120 0x7ff3f019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3f5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3f01061a0 0x7ff3f019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3f5a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3f0106b40 0x7ff3f019caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3f5a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f0110c50 con 0x7ff3f01061a0 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3f5a00640 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f0110ad0 con 0x7ff3f0105120 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3f5a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f0110dd0 con 0x7ff3f0106b40 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3e7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3f01061a0 0x7ff3f019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3e7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3f01061a0 0x7ff3f019bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36048/0 (socket says 172.21.15.97:36048) 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.869+0000 7ff3e7400640 1 -- 172.21.15.97:0/530760114 learned_addr learned my addr 172.21.15.97:0/530760114 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f0105120 0x7ff3f019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3f5000640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3f0106b40 0x7ff3f019caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3e7400640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3f0106b40 msgr2=0x7ff3f019caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3e7400640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3f0106b40 0x7ff3f019caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3e7400640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f0105120 msgr2=0x7ff3f019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3e7400640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3f0105120 0x7ff3f019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3e7400640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f019d310 con 0x7ff3f01061a0 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3f5000640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3f0106b40 0x7ff3f019caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3e7400640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3f01061a0 0x7ff3f019bf40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009230 tx=0x7ff3d8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.870+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d803f030 con 0x7ff3f01061a0 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.871+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3d8003a80 con 0x7ff3f01061a0 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.871+0000 7ff3f5a00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f019d5a0 con 0x7ff3f01061a0 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.871+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8035070 con 0x7ff3f01061a0 2024-07-26T07:58:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.871+0000 7ff3f5a00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f01a3cf0 con 0x7ff3f01061a0 2024-07-26T07:58:06.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3fdf8d740 1 Processor -- start 2024-07-26T07:58:06.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3fdf8d740 1 -- start start 2024-07-26T07:58:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3fdf8d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2507341b0 0x55d250ada0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3fdf8d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d25070b2f0 0x55d250ada750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3fdf8d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507134d0 0x55d250adb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3fdf8d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55d250a4c270 con 0x55d2507341b0 2024-07-26T07:58:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3fdf8d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d250a3b770 con 0x55d2507134d0 2024-07-26T07:58:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3fdf8d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d250a3f7b0 con 0x55d25070b2f0 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.874+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 20) ==== 151059+0+0 (secure 0 0 0) 0x7ff3d8042030 con 0x7ff3f01061a0 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3f5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2507341b0 0x55d250ada0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3f5000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2507341b0 0x55d250ada0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36064/0 (socket says 172.21.15.97:36064) 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3f5000640 1 -- 172.21.15.97:0/1385465623 learned_addr learned my addr 172.21.15.97:0/1385465623 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3efe00640 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d25070b2f0 0x55d250ada750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3f5000640 1 -- 172.21.15.97:0/1385465623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d25070b2f0 msgr2=0x55d250ada750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3f5000640 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d25070b2f0 0x55d250ada750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3f5000640 1 -- 172.21.15.97:0/1385465623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507134d0 msgr2=0x55d250adb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3f5000640 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507134d0 0x55d250adb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3f5000640 1 -- 172.21.15.97:0/1385465623 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d25079f790 con 0x55d2507341b0 2024-07-26T07:58:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.875+0000 7ff3efe00640 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d25070b2f0 0x55d250ada750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.876+0000 7ff3f5000640 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2507341b0 0x55d250ada0e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8009ea0 tx=0x7ff3e8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b145c3c3f15fc6b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.876+0000 7ff3e5600640 1 -- 172.21.15.97:0/1385465623 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8004d20 con 0x55d2507341b0 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.876+0000 7ff3e5600640 1 -- 172.21.15.97:0/1385465623 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3e80054b0 con 0x55d2507341b0 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.876+0000 7ff3e5600640 1 -- 172.21.15.97:0/1385465623 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e80116f0 con 0x55d2507341b0 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.876+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7ff3d8104560 con 0x7ff3f01061a0 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 -- 172.21.15.97:0/1385465623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2507341b0 msgr2=0x55d250ada0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2507341b0 0x55d250ada0e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8009ea0 tx=0x7ff3e8009f90 comp rx=0 tx=0).stop 2024-07-26T07:58:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 -- 172.21.15.97:0/1385465623 shutdown_connections 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507134d0 0x55d250adb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d25070b2f0 0x55d250ada750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 --2- 172.21.15.97:0/1385465623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2507341b0 0x55d250ada0e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 -- 172.21.15.97:0/1385465623 >> 172.21.15.97:0/1385465623 conn(0x55d250728300 msgr2=0x55d250ad5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 -- 172.21.15.97:0/1385465623 shutdown_connections 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3fdf8d740 1 -- 172.21.15.97:0/1385465623 wait complete. 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.877+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b6160 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000bc80 tx=0x7ff3e0005f30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 1 -- start start 2024-07-26T07:58:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 1 Processor -- start 2024-07-26T07:58:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d25070b2f0 0x55d250ae3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2507134d0 0x55d250ae4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507341b0 0x55d250ae4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55d250975ab0 con 0x55d25070b2f0 2024-07-26T07:58:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 -- 0x55d2509d8d50 con 0x55d2507341b0 2024-07-26T07:58:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d250adbdc0 con 0x55d2507134d0 2024-07-26T07:58:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.878+0000 7ff3fdf8d740 10 client.0 _ll_register_callbacks cb 0x55d25074ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T07:58:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d25070b2f0 0x55d250ae3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d25070b2f0 0x55d250ae3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36080/0 (socket says 172.21.15.97:36080) 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 -- 172.21.15.97:0/1934891825 learned_addr learned my addr 172.21.15.97:0/1934891825 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2507134d0 0x55d250ae4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3f5000640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507341b0 0x55d250ae4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2507134d0 msgr2=0x55d250ae4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2507134d0 0x55d250ae4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507341b0 msgr2=0x55d250ae4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507341b0 0x55d250ae4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d250818620 con 0x55d25070b2f0 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3f5000640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55d2507341b0 0x55d250ae4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3efe00640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d25070b2f0 0x55d250ae3c50 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0011010 tx=0x7ff3e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5c2a89e66ab2b9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.879+0000 7ff3de000640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-07-26T07:58:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.880+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0023070 con 0x55d25070b2f0 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.880+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8327+0+0 (secure 0 0 0) 0x7ff3e00118b0 con 0x55d25070b2f0 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.880+0000 7ff3fdf8d740 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3e80054b0 con 0x55d25070b2f0 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.880+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff3e002f050 con 0x55d25070b2f0 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.880+0000 7ff3fdf8d740 10 client.16254 fetch_fsmap learned FSMap version 36 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.880+0000 7ff3fdf8d740 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d250aede10 con 0x55d25070b2f0 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.880+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3e80054b0 con 0x55d25070b2f0 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.880+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7ff3e000f040 con 0x55d25070b2f0 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3fdf8d740 10 client.16254 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3fdf8d740 10 client.16254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3fdf8d740 20 client.16254 populate_metadata read hostname 'smithi097' 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3fdf8d740 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3fdf8d740 4 client.16254 mds_command: new command op to 34583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3fdf8d740 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d250a4e1c0 con 0x55d2506f2d80 2024-07-26T07:58:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3de000640 1 client.16254 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:06.881+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T07:58:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:07.083+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:07.083+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T07:58:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:07.083+0000 7ff3de000640 1 client.16254 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:07.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:07.484+0000 7ff3de000640 1 client.16254 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:07.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T07:58:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:07 smithi115 ceph-mon[35364]: pgmap v701: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 436 KiB/s wr, 42 op/s 2024-07-26T07:58:08.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:08.223+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:08.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:08.223+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T07:58:08.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:08.223+0000 7f1937400640 1 client.16242 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:08.225 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:07 smithi097 ceph-mon[32231]: pgmap v701: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 436 KiB/s wr, 42 op/s 2024-07-26T07:58:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:07 smithi181 ceph-mon[35869]: pgmap v701: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 436 KiB/s wr, 42 op/s 2024-07-26T07:58:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:08.286+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:08.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:08.286+0000 7ff3de000640 1 client.16254 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:08.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:08.286+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T07:58:09.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:09.887+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:09.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:09.887+0000 7ff3de000640 1 client.16254 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:09.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:09.888+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T07:58:09.973 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:09 smithi115 ceph-mon[35364]: pgmap v702: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 117 KiB/s wr, 19 op/s 2024-07-26T07:58:10.203 INFO:tasks.workunit.client.0.smithi097.stdout: 37 0 0 0 0 0 0 2024-07-26T07:58:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:09 smithi181 ceph-mon[35869]: pgmap v702: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 117 KiB/s wr, 19 op/s 2024-07-26T07:58:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:09 smithi097 ceph-mon[32231]: pgmap v702: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 117 KiB/s wr, 19 op/s 2024-07-26T07:58:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:58:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:10.974 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:10 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-07-26T07:58:10.974 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:10 smithi115 systemd[1]: Stopped Ceph mds.b for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:58:10.974 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:10 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.b.service: Consumed 6min 26.226s CPU time. 2024-07-26T07:58:10.974 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:10 smithi115 systemd[1]: Starting Ceph mds.b for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:58:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:58:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:58:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:11.399 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:11 smithi115 podman[69201]: 2024-07-26 07:58:11.013037823 +0000 UTC m=+0.039113061 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:58:11.478 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:11 smithi115 podman[69201]: 2024-07-26 07:58:11.119401867 +0000 UTC m=+0.145477095 container create df5b59d6f9933b2fc5a4a8dd9c4bdd0ac989124e71f2d15e96007440a1ae893b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-07-26T07:58:11.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:11.426+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:11.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:11.426+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T07:58:11.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:11.426+0000 7f1937400640 1 client.16242 ms_handle_refused on v2:172.21.15.115:6838/1903638498 2024-07-26T07:58:11.724 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:11 smithi115 podman[69201]: 2024-07-26 07:58:11.398490959 +0000 UTC m=+0.424566190 container init df5b59d6f9933b2fc5a4a8dd9c4bdd0ac989124e71f2d15e96007440a1ae893b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T07:58:11.725 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:11 smithi115 podman[69201]: 2024-07-26 07:58:11.409683586 +0000 UTC m=+0.435758787 container start df5b59d6f9933b2fc5a4a8dd9c4bdd0ac989124e71f2d15e96007440a1ae893b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-07-26T07:58:11.725 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:11 smithi115 bash[69201]: df5b59d6f9933b2fc5a4a8dd9c4bdd0ac989124e71f2d15e96007440a1ae893b 2024-07-26T07:58:11.725 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:11 smithi115 systemd[1]: Started Ceph mds.b for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:58:11.725 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mds[69219]: -- 172.21.15.115:0/1416757074 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560830e41860 con 0x560834fe3180 2024-07-26T07:58:11.725 INFO:journalctl@ceph.mds.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-b[69215]: starting mds.b at 2024-07-26T07:58:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: pgmap v703: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 115 KiB/s wr, 17 op/s 2024-07-26T07:58:12.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:58:12.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:58:12.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:12.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: daemon mds.b restarted 2024-07-26T07:58:12.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:58:12.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T07:58:12.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6838/3470885939,v1:172.21.15.115:6839/3470885939] up:boot 2024-07-26T07:58:12.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-07-26T07:58:12.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {1=h=up:active,2=e=up:active,3=k=up:active,4=i=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T07:58:12.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T07:58:12.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:11 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T07:58:12.322 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: pgmap v703: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 115 KiB/s wr, 17 op/s 2024-07-26T07:58:12.322 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:58:12.322 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:58:12.322 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: daemon mds.b restarted 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6838/3470885939,v1:172.21.15.115:6839/3470885939] up:boot 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {1=h=up:active,2=e=up:active,3=k=up:active,4=i=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:11 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: pgmap v703: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 115 KiB/s wr, 17 op/s 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:58:12.323 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: daemon mds.b restarted 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6838/3470885939,v1:172.21.15.115:6839/3470885939] up:boot 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {1=h=up:active,2=e=up:active,3=k=up:active,4=i=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T07:58:12.324 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:11 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T07:58:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:13.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:13.093+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T07:58:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:13.093+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:58:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:13.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:58:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:13.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T07:58:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:13 smithi115 ceph-mon[35364]: pgmap v705: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.8 KiB/s wr, 13 op/s 2024-07-26T07:58:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:13 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:resolve 2024-07-26T07:58:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:13 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:58:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:13 smithi181 ceph-mon[35869]: pgmap v705: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.8 KiB/s wr, 13 op/s 2024-07-26T07:58:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:13 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:resolve 2024-07-26T07:58:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:13 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:58:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:13 smithi097 ceph-mon[32231]: pgmap v705: 97 pgs: 97 active+clean; 121 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.8 KiB/s wr, 13 op/s 2024-07-26T07:58:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:13 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:resolve 2024-07-26T07:58:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:13 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:58:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:14 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:reconnect 2024-07-26T07:58:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:14 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T07:58:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:14 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0469991 2024-07-26T07:58:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:14 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:reconnect 2024-07-26T07:58:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:14 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T07:58:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:14 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0469991 2024-07-26T07:58:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:14 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:reconnect 2024-07-26T07:58:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:14 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T07:58:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:14 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0469991 2024-07-26T07:58:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:15 smithi115 ceph-mon[35364]: pgmap v706: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-07-26T07:58:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:15 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:rejoin 2024-07-26T07:58:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:15 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T07:58:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:15 smithi181 ceph-mon[35869]: pgmap v706: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-07-26T07:58:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:15 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:rejoin 2024-07-26T07:58:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:15 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T07:58:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:15 smithi097 ceph-mon[32231]: pgmap v706: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-07-26T07:58:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:15 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:rejoin 2024-07-26T07:58:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:15 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T07:58:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:17 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:clientreplay 2024-07-26T07:58:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:17 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-07-26T07:58:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:17 smithi115 ceph-mon[35364]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-07-26T07:58:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:17 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:clientreplay 2024-07-26T07:58:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:17 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-07-26T07:58:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:17 smithi181 ceph-mon[35869]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-07-26T07:58:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:17 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:clientreplay 2024-07-26T07:58:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:17 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-07-26T07:58:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:17 smithi097 ceph-mon[32231]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-07-26T07:58:17.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:17.833+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:17.835+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T07:58:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:17.835+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:58:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:17.835+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:58:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:17.835+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T07:58:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:18 smithi097 ceph-mon[32231]: pgmap v707: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 307 B/s wr, 28 op/s 2024-07-26T07:58:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:18 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:58:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:18 smithi097 ceph-mon[32231]: Cluster is now healthy 2024-07-26T07:58:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:18 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:active 2024-07-26T07:58:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:18 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:58:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:18 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:18 smithi115 ceph-mon[35364]: pgmap v707: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 307 B/s wr, 28 op/s 2024-07-26T07:58:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:18 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:58:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:18 smithi115 ceph-mon[35364]: Cluster is now healthy 2024-07-26T07:58:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:18 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:active 2024-07-26T07:58:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:18 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:58:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:18 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:18 smithi181 ceph-mon[35869]: pgmap v707: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 307 B/s wr, 28 op/s 2024-07-26T07:58:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:18 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:58:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:18 smithi181 ceph-mon[35869]: Cluster is now healthy 2024-07-26T07:58:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:18 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:active 2024-07-26T07:58:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:18 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:58:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:18 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:19.500+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:19.501+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T07:58:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:19.501+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:58:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:19.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:58:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:19.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T07:58:20.203 INFO:tasks.workunit.client.0.smithi097.stdout: 38 1772 16 1138 46 463 77 2024-07-26T07:58:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:20 smithi115 ceph-mon[35364]: pgmap v708: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 409 B/s wr, 32 op/s 2024-07-26T07:58:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:20 smithi181 ceph-mon[35869]: pgmap v708: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 409 B/s wr, 32 op/s 2024-07-26T07:58:20.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:58:19 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T07:58:19.983+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T07:58:20.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:58:19 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T07:58:19.983+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T07:58:20.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:58:19 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T07:58:19.983+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T07:58:20.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:58:19 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T07:58:19.983+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T07:58:20.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 07:58:19 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T07:58:19.983+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T07:58:20.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:20 smithi097 ceph-mon[32231]: pgmap v708: 97 pgs: 97 active+clean; 121 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 409 B/s wr, 32 op/s 2024-07-26T07:58:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:22 smithi115 ceph-mon[35364]: pgmap v709: 97 pgs: 97 active+clean; 122 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 113 KiB/s wr, 157 op/s 2024-07-26T07:58:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:22 smithi181 ceph-mon[35869]: pgmap v709: 97 pgs: 97 active+clean; 122 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 113 KiB/s wr, 157 op/s 2024-07-26T07:58:22.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:22 smithi097 ceph-mon[32231]: pgmap v709: 97 pgs: 97 active+clean; 122 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 113 KiB/s wr, 157 op/s 2024-07-26T07:58:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:24 smithi115 ceph-mon[35364]: pgmap v710: 97 pgs: 97 active+clean; 124 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 277 KiB/s wr, 179 op/s 2024-07-26T07:58:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:24 smithi181 ceph-mon[35869]: pgmap v710: 97 pgs: 97 active+clean; 124 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 277 KiB/s wr, 179 op/s 2024-07-26T07:58:24.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:24 smithi097 ceph-mon[32231]: pgmap v710: 97 pgs: 97 active+clean; 124 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 277 KiB/s wr, 179 op/s 2024-07-26T07:58:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:26 smithi115 ceph-mon[35364]: pgmap v711: 97 pgs: 97 active+clean; 127 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 539 KiB/s wr, 347 op/s 2024-07-26T07:58:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:26 smithi181 ceph-mon[35869]: pgmap v711: 97 pgs: 97 active+clean; 127 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 539 KiB/s wr, 347 op/s 2024-07-26T07:58:26.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:26 smithi097 ceph-mon[32231]: pgmap v711: 97 pgs: 97 active+clean; 127 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 539 KiB/s wr, 347 op/s 2024-07-26T07:58:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:28 smithi115 ceph-mon[35364]: pgmap v712: 97 pgs: 97 active+clean; 130 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 853 KiB/s wr, 429 op/s 2024-07-26T07:58:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:28 smithi181 ceph-mon[35869]: pgmap v712: 97 pgs: 97 active+clean; 130 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 853 KiB/s wr, 429 op/s 2024-07-26T07:58:28.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:28 smithi097 ceph-mon[32231]: pgmap v712: 97 pgs: 97 active+clean; 130 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 853 KiB/s wr, 429 op/s 2024-07-26T07:58:30.203 INFO:tasks.workunit.client.0.smithi097.stdout: 42 8094 216 5353 133 3275 185 2024-07-26T07:58:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:30 smithi115 ceph-mon[35364]: pgmap v713: 97 pgs: 97 active+clean; 130 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 871 KiB/s wr, 426 op/s 2024-07-26T07:58:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:30 smithi181 ceph-mon[35869]: pgmap v713: 97 pgs: 97 active+clean; 130 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 871 KiB/s wr, 426 op/s 2024-07-26T07:58:30.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:30 smithi097 ceph-mon[32231]: pgmap v713: 97 pgs: 97 active+clean; 130 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 871 KiB/s wr, 426 op/s 2024-07-26T07:58:30.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:30.648+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:30.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:30.650+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T07:58:30.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:30.650+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:58:30.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:30.650+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:58:30.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:30.650+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:58:32.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:32.303+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:32.304+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T07:58:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:32.304+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:58:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:32.304+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:58:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:32.304+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:58:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:32 smithi115 ceph-mon[35364]: pgmap v714: 97 pgs: 97 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.4 MiB/s wr, 522 op/s 2024-07-26T07:58:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:32 smithi181 ceph-mon[35869]: pgmap v714: 97 pgs: 97 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.4 MiB/s wr, 522 op/s 2024-07-26T07:58:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:32 smithi097 ceph-mon[32231]: pgmap v714: 97 pgs: 97 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.4 MiB/s wr, 522 op/s 2024-07-26T07:58:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:34 smithi115 ceph-mon[35364]: pgmap v715: 97 pgs: 97 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.3 MiB/s wr, 431 op/s 2024-07-26T07:58:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:34 smithi181 ceph-mon[35869]: pgmap v715: 97 pgs: 97 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.3 MiB/s wr, 431 op/s 2024-07-26T07:58:34.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:34 smithi097 ceph-mon[32231]: pgmap v715: 97 pgs: 97 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.3 MiB/s wr, 431 op/s 2024-07-26T07:58:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:36 smithi115 ceph-mon[35364]: pgmap v716: 97 pgs: 97 active+clean; 140 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.5 MiB/s wr, 424 op/s 2024-07-26T07:58:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:36 smithi181 ceph-mon[35869]: pgmap v716: 97 pgs: 97 active+clean; 140 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.5 MiB/s wr, 424 op/s 2024-07-26T07:58:36.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:36 smithi097 ceph-mon[32231]: pgmap v716: 97 pgs: 97 active+clean; 140 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.5 MiB/s wr, 424 op/s 2024-07-26T07:58:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:38 smithi115 ceph-mon[35364]: pgmap v717: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 257 op/s 2024-07-26T07:58:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:38 smithi181 ceph-mon[35869]: pgmap v717: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 257 op/s 2024-07-26T07:58:38.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:38 smithi097 ceph-mon[32231]: pgmap v717: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 257 op/s 2024-07-26T07:58:40.203 INFO:tasks.workunit.client.0.smithi097.stdout: 44 9241 196 6561 213 4025 150 2024-07-26T07:58:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:40 smithi115 ceph-mon[35364]: pgmap v718: 97 pgs: 97 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 175 op/s 2024-07-26T07:58:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:40 smithi181 ceph-mon[35869]: pgmap v718: 97 pgs: 97 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 175 op/s 2024-07-26T07:58:40.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:40 smithi097 ceph-mon[32231]: pgmap v718: 97 pgs: 97 active+clean; 144 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 175 op/s 2024-07-26T07:58:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:42 smithi115 ceph-mon[35364]: pgmap v719: 97 pgs: 97 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-07-26T07:58:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:42 smithi181 ceph-mon[35869]: pgmap v719: 97 pgs: 97 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-07-26T07:58:42.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:42 smithi097 ceph-mon[32231]: pgmap v719: 97 pgs: 97 active+clean; 148 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-07-26T07:58:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:44 smithi115 ceph-mon[35364]: pgmap v720: 97 pgs: 97 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.5 MiB/s wr, 109 op/s 2024-07-26T07:58:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:44 smithi181 ceph-mon[35869]: pgmap v720: 97 pgs: 97 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.5 MiB/s wr, 109 op/s 2024-07-26T07:58:44.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:44 smithi097 ceph-mon[32231]: pgmap v720: 97 pgs: 97 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.5 MiB/s wr, 109 op/s 2024-07-26T07:58:45.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:45.665+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:45.667+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T07:58:45.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:45.667+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:58:45.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:45.667+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:58:45.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:58:45.667+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:58:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:46 smithi115 ceph-mon[35364]: pgmap v721: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-07-26T07:58:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:46 smithi181 ceph-mon[35869]: pgmap v721: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-07-26T07:58:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:46 smithi097 ceph-mon[32231]: pgmap v721: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-07-26T07:58:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:47.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:58:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:47.313+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T07:58:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:47.313+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:58:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:47.313+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:58:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:58:47.313+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:58:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:48 smithi115 ceph-mon[35364]: pgmap v722: 97 pgs: 97 active+clean; 157 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-07-26T07:58:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:48 smithi181 ceph-mon[35869]: pgmap v722: 97 pgs: 97 active+clean; 157 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-07-26T07:58:48.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:48 smithi097 ceph-mon[32231]: pgmap v722: 97 pgs: 97 active+clean; 157 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-07-26T07:58:50.203 INFO:tasks.workunit.client.0.smithi097.stdout: 49 9621 229 6821 150 3958 159 2024-07-26T07:58:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:50 smithi115 ceph-mon[35364]: pgmap v723: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 88 op/s 2024-07-26T07:58:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:50 smithi181 ceph-mon[35869]: pgmap v723: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 88 op/s 2024-07-26T07:58:50.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:50 smithi097 ceph-mon[32231]: pgmap v723: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 88 op/s 2024-07-26T07:58:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:52 smithi115 ceph-mon[35364]: pgmap v724: 97 pgs: 97 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 1.8 MiB/s wr, 116 op/s 2024-07-26T07:58:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:52 smithi181 ceph-mon[35869]: pgmap v724: 97 pgs: 97 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 1.8 MiB/s wr, 116 op/s 2024-07-26T07:58:52.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:52 smithi097 ceph-mon[32231]: pgmap v724: 97 pgs: 97 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 1.8 MiB/s wr, 116 op/s 2024-07-26T07:58:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:54 smithi115 ceph-mon[35364]: pgmap v725: 97 pgs: 97 active+clean; 164 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-07-26T07:58:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:54 smithi181 ceph-mon[35869]: pgmap v725: 97 pgs: 97 active+clean; 164 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-07-26T07:58:54.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:54 smithi097 ceph-mon[32231]: pgmap v725: 97 pgs: 97 active+clean; 164 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-07-26T07:58:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:56 smithi115 ceph-mon[35364]: pgmap v726: 97 pgs: 97 active+clean; 169 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.5 MiB/s wr, 94 op/s 2024-07-26T07:58:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:56 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:56 smithi181 ceph-mon[35869]: pgmap v726: 97 pgs: 97 active+clean; 169 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.5 MiB/s wr, 94 op/s 2024-07-26T07:58:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:56 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:56 smithi097 ceph-mon[32231]: pgmap v726: 97 pgs: 97 active+clean; 169 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.5 MiB/s wr, 94 op/s 2024-07-26T07:58:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:56 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:58:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:58:58 smithi115 ceph-mon[35364]: pgmap v727: 97 pgs: 97 active+clean; 170 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-07-26T07:58:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:58:58 smithi181 ceph-mon[35869]: pgmap v727: 97 pgs: 97 active+clean; 170 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-07-26T07:58:58.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:58:58 smithi097 ceph-mon[32231]: pgmap v727: 97 pgs: 97 active+clean; 170 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-07-26T07:59:00.204 INFO:tasks.workunit.client.0.smithi097.stdout: 53 9460 192 6438 155 4553 196 2024-07-26T07:59:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:00 smithi115 ceph-mon[35364]: pgmap v728: 97 pgs: 97 active+clean; 174 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-07-26T07:59:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:00 smithi181 ceph-mon[35869]: pgmap v728: 97 pgs: 97 active+clean; 174 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-07-26T07:59:00.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:00 smithi097 ceph-mon[32231]: pgmap v728: 97 pgs: 97 active+clean; 174 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-07-26T07:59:00.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:00.682+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:59:00.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:00.683+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T07:59:00.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:00.683+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:59:00.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:00.683+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:59:00.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:00.683+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:02.315+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:59:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:02.317+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T07:59:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:02.317+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:59:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:02.317+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:59:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:02.317+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:02 smithi115 ceph-mon[35364]: pgmap v729: 97 pgs: 97 active+clean; 179 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-07-26T07:59:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:02 smithi181 ceph-mon[35869]: pgmap v729: 97 pgs: 97 active+clean; 179 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-07-26T07:59:02.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:02 smithi097 ceph-mon[32231]: pgmap v729: 97 pgs: 97 active+clean; 179 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-07-26T07:59:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:04 smithi115 ceph-mon[35364]: pgmap v730: 97 pgs: 97 active+clean; 180 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-07-26T07:59:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:04 smithi181 ceph-mon[35869]: pgmap v730: 97 pgs: 97 active+clean; 180 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-07-26T07:59:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:04 smithi097 ceph-mon[32231]: pgmap v730: 97 pgs: 97 active+clean; 180 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-07-26T07:59:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:06 smithi115 ceph-mon[35364]: pgmap v731: 97 pgs: 97 active+clean; 186 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.0 MiB/s wr, 113 op/s 2024-07-26T07:59:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:06 smithi181 ceph-mon[35869]: pgmap v731: 97 pgs: 97 active+clean; 186 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.0 MiB/s wr, 113 op/s 2024-07-26T07:59:06.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:06 smithi097 ceph-mon[32231]: pgmap v731: 97 pgs: 97 active+clean; 186 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.0 MiB/s wr, 113 op/s 2024-07-26T07:59:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:08 smithi115 ceph-mon[35364]: pgmap v732: 97 pgs: 97 active+clean; 189 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2024-07-26T07:59:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:08 smithi181 ceph-mon[35869]: pgmap v732: 97 pgs: 97 active+clean; 189 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2024-07-26T07:59:08.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:08 smithi097 ceph-mon[32231]: pgmap v732: 97 pgs: 97 active+clean; 189 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2024-07-26T07:59:10.204 INFO:tasks.workunit.client.0.smithi097.stdout: 55 9265 200 6268 176 3937 225 2024-07-26T07:59:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:10 smithi115 ceph-mon[35364]: pgmap v733: 97 pgs: 97 active+clean; 189 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-07-26T07:59:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:10 smithi181 ceph-mon[35869]: pgmap v733: 97 pgs: 97 active+clean; 189 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-07-26T07:59:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:10 smithi097 ceph-mon[32231]: pgmap v733: 97 pgs: 97 active+clean; 189 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-07-26T07:59:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:12 smithi115 ceph-mon[35364]: pgmap v734: 97 pgs: 97 active+clean; 196 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-07-26T07:59:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:59:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:12 smithi181 ceph-mon[35869]: pgmap v734: 97 pgs: 97 active+clean; 196 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-07-26T07:59:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:59:12.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:12 smithi097 ceph-mon[32231]: pgmap v734: 97 pgs: 97 active+clean; 196 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-07-26T07:59:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:13 smithi097 ceph-mon[32231]: pgmap v735: 97 pgs: 97 active+clean; 198 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-07-26T07:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:13 smithi115 ceph-mon[35364]: pgmap v735: 97 pgs: 97 active+clean; 198 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-07-26T07:59:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:59:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:59:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:13 smithi181 ceph-mon[35869]: pgmap v735: 97 pgs: 97 active+clean; 198 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-07-26T07:59:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:59:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:59:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T07:59:15.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:15.699+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:59:15.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:15.700+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T07:59:15.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:15.700+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:59:15.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:15.700+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:59:15.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:15.700+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:15 smithi097 ceph-mon[32231]: pgmap v736: 97 pgs: 97 active+clean; 203 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.1 MiB/s wr, 151 op/s 2024-07-26T07:59:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:15 smithi115 ceph-mon[35364]: pgmap v736: 97 pgs: 97 active+clean; 203 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.1 MiB/s wr, 151 op/s 2024-07-26T07:59:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:15 smithi181 ceph-mon[35869]: pgmap v736: 97 pgs: 97 active+clean; 203 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.1 MiB/s wr, 151 op/s 2024-07-26T07:59:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:17.324+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:17.326+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T07:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:17.326+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:17.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:17.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:17 smithi097 ceph-mon[32231]: pgmap v737: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-07-26T07:59:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:17 smithi115 ceph-mon[35364]: pgmap v737: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-07-26T07:59:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:17 smithi181 ceph-mon[35869]: pgmap v737: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-07-26T07:59:18.630 INFO:tasks.workunit.client.0.smithi097.stdout: 2024-07-26T07:59:18.631 INFO:tasks.workunit.client.0.smithi097.stdout:Final score for writes: 55 2024-07-26T07:59:18.631 INFO:tasks.workunit.client.0.smithi097.stdout:Final score for reads : 4744 2024-07-26T07:59:18.631 INFO:tasks.workunit.client.0.smithi097.stdout: 2024-07-26T07:59:18.633 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-07-26T07:59:18.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-07-26T07:59:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:19 smithi115 ceph-mon[35364]: pgmap v738: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-07-26T07:59:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:19 smithi181 ceph-mon[35869]: pgmap v738: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-07-26T07:59:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:19 smithi097 ceph-mon[32231]: pgmap v738: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-07-26T07:59:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:21 smithi115 ceph-mon[35364]: pgmap v739: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.8 MiB/s wr, 137 op/s 2024-07-26T07:59:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:21 smithi181 ceph-mon[35869]: pgmap v739: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.8 MiB/s wr, 137 op/s 2024-07-26T07:59:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:21 smithi097 ceph-mon[32231]: pgmap v739: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.8 MiB/s wr, 137 op/s 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: *** Caught signal (Aborted) ** 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: in thread 7fea05e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fea0c03e6f0] 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fea0c08b94c] 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 3: raise() 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 4: abort() 2024-07-26T07:59:22.475 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 5: /lib64/libc.so.6(+0x2871b) [0x7fea0c02871b] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 6: /lib64/libc.so.6(+0x37386) [0x7fea0c037386] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5585334006b9] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5585335be80c] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5585334fe2ca] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5585334ce753] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5585334cfa1b] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5585334b3752] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558533401a56] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558533405637] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558533405f5e] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5585333e903a] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fea0cb9b407] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 18: (DispatchQueue::entry()+0x60f) [0x7fea0cb9bcf9] 2024-07-26T07:59:22.476 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fea0cc785a1] 2024-07-26T07:59:22.477 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 20: (Thread::entry_wrapper()+0x43) [0x7fea0ca25cdd] 2024-07-26T07:59:22.477 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 21: (Thread::_entry_func(void*)+0xd) [0x7fea0ca25cf9] 2024-07-26T07:59:22.477 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 22: /lib64/libc.so.6(+0x89c02) [0x7fea0c089c02] 2024-07-26T07:59:22.477 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fea0c10ec40] 2024-07-26T07:59:22.477 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2024-07-26T07:59:22.021+0000 7fea05e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.477 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: in thread 7fea05e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.477 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2024-07-26T07:59:22.477 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.478 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fea0c03e6f0] 2024-07-26T07:59:22.478 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fea0c08b94c] 2024-07-26T07:59:22.478 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 3: raise() 2024-07-26T07:59:22.478 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 4: abort() 2024-07-26T07:59:22.478 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 5: /lib64/libc.so.6(+0x2871b) [0x7fea0c02871b] 2024-07-26T07:59:22.478 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 6: /lib64/libc.so.6(+0x37386) [0x7fea0c037386] 2024-07-26T07:59:22.478 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5585334006b9] 2024-07-26T07:59:22.478 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5585335be80c] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5585334fe2ca] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5585334ce753] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5585334cfa1b] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5585334b3752] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558533401a56] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558533405637] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558533405f5e] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5585333e903a] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fea0cb9b407] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 18: (DispatchQueue::entry()+0x60f) [0x7fea0cb9bcf9] 2024-07-26T07:59:22.479 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fea0cc785a1] 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: *** Caught signal (Aborted) ** 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: in thread 7f1650e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f165703e6f0] 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f165708b94c] 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 3: raise() 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 4: abort() 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 5: /lib64/libc.so.6(+0x2871b) [0x7f165702871b] 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 6: /lib64/libc.so.6(+0x37386) [0x7f1657037386] 2024-07-26T07:59:22.480 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c4475216b9] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c4476df80c] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c44761f2ca] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c4475ef753] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c4475f0a1b] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c4475d4752] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c447522a56] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c447526637] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c447526f5e] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c44750a03a] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1657b9b407] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 18: (DispatchQueue::entry()+0x60f) [0x7f1657b9bcf9] 2024-07-26T07:59:22.481 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1657c785a1] 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: *** Caught signal (Aborted) ** 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: in thread 7f3f75400640 thread_name:ms_dispatch 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3f7b63e6f0] 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3f7b68b94c] 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 3: raise() 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 4: abort() 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3f7b62871b] 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 6: /lib64/libc.so.6(+0x37386) [0x7f3f7b637386] 2024-07-26T07:59:22.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5559fe1896b9] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5559fe34780c] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5559fe2872ca] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5559fe257753] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5559fe258a1b] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5559fe23c752] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5559fe18aa56] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5559fe18e637] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5559fe18ef5e] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5559fe17203a] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3f7c19b407] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 18: (DispatchQueue::entry()+0x60f) [0x7f3f7c19bcf9] 2024-07-26T07:59:22.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3f7c2785a1] 2024-07-26T07:59:22.484 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 20: (Thread::entry_wrapper()+0x43) [0x7fea0ca25cdd] 2024-07-26T07:59:22.484 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 21: (Thread::_entry_func(void*)+0xd) [0x7fea0ca25cf9] 2024-07-26T07:59:22.484 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 22: /lib64/libc.so.6(+0x89c02) [0x7fea0c089c02] 2024-07-26T07:59:22.484 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fea0c10ec40] 2024-07-26T07:59:22.484 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 0> 2024-07-26T07:59:22.021+0000 7fea05e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: in thread 7fea05e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fea0c03e6f0] 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fea0c08b94c] 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 3: raise() 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 4: abort() 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 5: /lib64/libc.so.6(+0x2871b) [0x7fea0c02871b] 2024-07-26T07:59:22.485 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 6: /lib64/libc.so.6(+0x37386) [0x7fea0c037386] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5585334006b9] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5585335be80c] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5585334fe2ca] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5585334ce753] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5585334cfa1b] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5585334b3752] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558533401a56] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558533405637] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558533405f5e] 2024-07-26T07:59:22.486 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5585333e903a] 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fea0cb9b407] 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 18: (DispatchQueue::entry()+0x60f) [0x7fea0cb9bcf9] 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fea0cc785a1] 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 20: (Thread::entry_wrapper()+0x43) [0x7fea0ca25cdd] 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 21: (Thread::_entry_func(void*)+0xd) [0x7fea0ca25cf9] 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 22: /lib64/libc.so.6(+0x89c02) [0x7fea0c089c02] 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fea0c10ec40] 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: -2> 2024-07-26T07:59:22.021+0000 7fea05e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: in thread 7fea05e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.487 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fea0c03e6f0] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fea0c08b94c] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 3: raise() 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 4: abort() 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 5: /lib64/libc.so.6(+0x2871b) [0x7fea0c02871b] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 6: /lib64/libc.so.6(+0x37386) [0x7fea0c037386] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5585334006b9] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5585335be80c] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5585334fe2ca] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5585334ce753] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5585334cfa1b] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5585334b3752] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558533401a56] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558533405637] 2024-07-26T07:59:22.488 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558533405f5e] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5585333e903a] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fea0cb9b407] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 18: (DispatchQueue::entry()+0x60f) [0x7fea0cb9bcf9] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fea0cc785a1] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 20: (Thread::entry_wrapper()+0x43) [0x7fea0ca25cdd] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 21: (Thread::_entry_func(void*)+0xd) [0x7fea0ca25cf9] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 22: /lib64/libc.so.6(+0x89c02) [0x7fea0c089c02] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fea0c10ec40] 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[65821]: 2024-07-26T07:59:22.489 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:22 smithi115 podman[69550]: 2024-07-26 07:59:22.182474333 +0000 UTC m=+0.032691801 container died fa4447fff829e0e16149fb4953e54f689c5cd329108b86761aa4248de8c8e652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 20: (Thread::entry_wrapper()+0x43) [0x7f1657a25cdd] 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1657a25cf9] 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1657089c02] 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f165710ec40] 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2024-07-26T07:59:22.021+0000 7f1650e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: in thread 7f1650e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f165703e6f0] 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f165708b94c] 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 3: raise() 2024-07-26T07:59:22.490 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 4: abort() 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 5: /lib64/libc.so.6(+0x2871b) [0x7f165702871b] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 6: /lib64/libc.so.6(+0x37386) [0x7f1657037386] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c4475216b9] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c4476df80c] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c44761f2ca] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c4475ef753] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c4475f0a1b] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c4475d4752] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c447522a56] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c447526637] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c447526f5e] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c44750a03a] 2024-07-26T07:59:22.491 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1657b9b407] 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 18: (DispatchQueue::entry()+0x60f) [0x7f1657b9bcf9] 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1657c785a1] 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 20: (Thread::entry_wrapper()+0x43) [0x7f3f7c025cdd] 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3f7c025cf9] 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3f7b689c02] 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3f7b70ec40] 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2024-07-26T07:59:22.022+0000 7f3f75400640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: in thread 7f3f75400640 thread_name:ms_dispatch 2024-07-26T07:59:22.492 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3f7b63e6f0] 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3f7b68b94c] 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 3: raise() 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 4: abort() 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3f7b62871b] 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 6: /lib64/libc.so.6(+0x37386) [0x7f3f7b637386] 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5559fe1896b9] 2024-07-26T07:59:22.493 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5559fe34780c] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5559fe2872ca] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5559fe257753] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5559fe258a1b] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5559fe23c752] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5559fe18aa56] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5559fe18e637] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5559fe18ef5e] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5559fe17203a] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3f7c19b407] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 18: (DispatchQueue::entry()+0x60f) [0x7f3f7c19bcf9] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3f7c2785a1] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 20: (Thread::entry_wrapper()+0x43) [0x7f3f7c025cdd] 2024-07-26T07:59:22.494 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3f7c025cf9] 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3f7b689c02] 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3f7b70ec40] 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 0> 2024-07-26T07:59:22.022+0000 7f3f75400640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: in thread 7f3f75400640 thread_name:ms_dispatch 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3f7b63e6f0] 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3f7b68b94c] 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 3: raise() 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 4: abort() 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3f7b62871b] 2024-07-26T07:59:22.495 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 6: /lib64/libc.so.6(+0x37386) [0x7f3f7b637386] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5559fe1896b9] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5559fe34780c] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5559fe2872ca] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5559fe257753] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5559fe258a1b] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5559fe23c752] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5559fe18aa56] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5559fe18e637] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5559fe18ef5e] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5559fe17203a] 2024-07-26T07:59:22.496 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3f7c19b407] 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 20: (Thread::entry_wrapper()+0x43) [0x7f1657a25cdd] 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1657a25cf9] 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1657089c02] 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f165710ec40] 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 0> 2024-07-26T07:59:22.021+0000 7f1650e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: in thread 7f1650e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f165703e6f0] 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f165708b94c] 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 3: raise() 2024-07-26T07:59:22.497 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 4: abort() 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 5: /lib64/libc.so.6(+0x2871b) [0x7f165702871b] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 6: /lib64/libc.so.6(+0x37386) [0x7f1657037386] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c4475216b9] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c4476df80c] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c44761f2ca] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c4475ef753] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c4475f0a1b] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c4475d4752] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c447522a56] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c447526637] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c447526f5e] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c44750a03a] 2024-07-26T07:59:22.498 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1657b9b407] 2024-07-26T07:59:22.499 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 18: (DispatchQueue::entry()+0x60f) [0x7f3f7c19bcf9] 2024-07-26T07:59:22.499 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3f7c2785a1] 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 20: (Thread::entry_wrapper()+0x43) [0x7f3f7c025cdd] 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3f7c025cf9] 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3f7b689c02] 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3f7b70ec40] 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 0> 2024-07-26T07:59:22.022+0000 7f3f75400640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: in thread 7f3f75400640 thread_name:ms_dispatch 2024-07-26T07:59:22.500 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3f7b63e6f0] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3f7b68b94c] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 3: raise() 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 4: abort() 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3f7b62871b] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 6: /lib64/libc.so.6(+0x37386) [0x7f3f7b637386] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5559fe1896b9] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5559fe34780c] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5559fe2872ca] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5559fe257753] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5559fe258a1b] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5559fe23c752] 2024-07-26T07:59:22.501 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5559fe18aa56] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5559fe18e637] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5559fe18ef5e] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5559fe17203a] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3f7c19b407] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 18: (DispatchQueue::entry()+0x60f) [0x7f3f7c19bcf9] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3f7c2785a1] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 20: (Thread::entry_wrapper()+0x43) [0x7f3f7c025cdd] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3f7c025cf9] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3f7b689c02] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3f7b70ec40] 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.502 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[65507]: 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 18: (DispatchQueue::entry()+0x60f) [0x7f1657b9bcf9] 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1657c785a1] 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 20: (Thread::entry_wrapper()+0x43) [0x7f1657a25cdd] 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1657a25cf9] 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1657089c02] 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f165710ec40] 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: -2> 2024-07-26T07:59:22.021+0000 7f1650e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: in thread 7f1650e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f165703e6f0] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f165708b94c] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 3: raise() 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 4: abort() 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 5: /lib64/libc.so.6(+0x2871b) [0x7f165702871b] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 6: /lib64/libc.so.6(+0x37386) [0x7f1657037386] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c4475216b9] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c4476df80c] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c44761f2ca] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c4475ef753] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c4475f0a1b] 2024-07-26T07:59:22.504 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c4475d4752] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c447522a56] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c447526637] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c447526f5e] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c44750a03a] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1657b9b407] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 18: (DispatchQueue::entry()+0x60f) [0x7f1657b9bcf9] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1657c785a1] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 20: (Thread::entry_wrapper()+0x43) [0x7f1657a25cdd] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1657a25cf9] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1657089c02] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f165710ec40] 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[65209]: 2024-07-26T07:59:22.505 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:22 smithi115 podman[69549]: 2024-07-26 07:59:22.18234194 +0000 UTC m=+0.034904157 container died a7747372ca68603d4423886415a8244c89f9509a3364d39dd5a254168924c1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-07-26T07:59:22.506 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: *** Caught signal (Aborted) ** 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: in thread 7f5ee0e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5ee703e6f0] 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5ee708b94c] 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 3: raise() 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 4: abort() 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5ee702871b] 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 6: /lib64/libc.so.6(+0x37386) [0x7f5ee7037386] 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55aaf0a886b9] 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55aaf0c4680c] 2024-07-26T07:59:22.507 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55aaf0b862ca] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55aaf0b56753] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55aaf0b57a1b] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55aaf0b3b752] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55aaf0a89a56] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55aaf0a8d637] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55aaf0a8df5e] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55aaf0a7103a] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5ee7b9b407] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 18: (DispatchQueue::entry()+0x60f) [0x7f5ee7b9bcf9] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5ee7c785a1] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 20: (Thread::entry_wrapper()+0x43) [0x7f5ee7a25cdd] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5ee7a25cf9] 2024-07-26T07:59:22.508 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5ee7089c02] 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5ee710ec40] 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2024-07-26T07:59:22.020+0000 7f5ee0e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: in thread 7f5ee0e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5ee703e6f0] 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5ee708b94c] 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 3: raise() 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 4: abort() 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5ee702871b] 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 6: /lib64/libc.so.6(+0x37386) [0x7f5ee7037386] 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55aaf0a886b9] 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55aaf0c4680c] 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55aaf0b862ca] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55aaf0b56753] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55aaf0b57a1b] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55aaf0b3b752] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55aaf0a89a56] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55aaf0a8d637] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55aaf0a8df5e] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55aaf0a7103a] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5ee7b9b407] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 18: (DispatchQueue::entry()+0x60f) [0x7f5ee7b9bcf9] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5ee7c785a1] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 20: (Thread::entry_wrapper()+0x43) [0x7f5ee7a25cdd] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5ee7a25cf9] 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5ee7089c02] 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5ee710ec40] 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 0> 2024-07-26T07:59:22.020+0000 7f5ee0e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: in thread 7f5ee0e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5ee703e6f0] 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5ee708b94c] 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 3: raise() 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 4: abort() 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5ee702871b] 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 6: /lib64/libc.so.6(+0x37386) [0x7f5ee7037386] 2024-07-26T07:59:22.511 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55aaf0a886b9] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55aaf0c4680c] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55aaf0b862ca] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55aaf0b56753] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55aaf0b57a1b] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55aaf0b3b752] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55aaf0a89a56] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55aaf0a8d637] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55aaf0a8df5e] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55aaf0a7103a] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5ee7b9b407] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 18: (DispatchQueue::entry()+0x60f) [0x7f5ee7b9bcf9] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5ee7c785a1] 2024-07-26T07:59:22.512 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 20: (Thread::entry_wrapper()+0x43) [0x7f5ee7a25cdd] 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5ee7a25cf9] 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5ee7089c02] 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5ee710ec40] 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: -2> 2024-07-26T07:59:22.020+0000 7f5ee0e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: in thread 7f5ee0e00640 thread_name:ms_dispatch 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5ee703e6f0] 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5ee708b94c] 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 3: raise() 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 4: abort() 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5ee702871b] 2024-07-26T07:59:22.513 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 6: /lib64/libc.so.6(+0x37386) [0x7f5ee7037386] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55aaf0a886b9] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55aaf0c4680c] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55aaf0b862ca] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55aaf0b56753] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55aaf0b57a1b] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55aaf0b3b752] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55aaf0a89a56] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55aaf0a8d637] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55aaf0a8df5e] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55aaf0a7103a] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5ee7b9b407] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 18: (DispatchQueue::entry()+0x60f) [0x7f5ee7b9bcf9] 2024-07-26T07:59:22.514 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5ee7c785a1] 2024-07-26T07:59:22.515 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 20: (Thread::entry_wrapper()+0x43) [0x7f5ee7a25cdd] 2024-07-26T07:59:22.515 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5ee7a25cf9] 2024-07-26T07:59:22.515 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5ee7089c02] 2024-07-26T07:59:22.515 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5ee710ec40] 2024-07-26T07:59:22.515 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:22.515 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[66201]: 2024-07-26T07:59:22.515 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 podman[69915]: 2024-07-26 07:59:22.180096639 +0000 UTC m=+0.035257728 container died 51179b6e8ec8559a701dfd6fbebec0fc06982a130f25e8729c8fa995e79a7d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-07-26T07:59:23.232 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 podman[69915]: 2024-07-26 07:59:22.889387994 +0000 UTC m=+0.744549138 container remove 51179b6e8ec8559a701dfd6fbebec0fc06982a130f25e8729c8fa995e79a7d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.36.0) 2024-07-26T07:59:23.232 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:22 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-26T07:59:23.721 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:23 smithi115 ceph-mon[35364]: pgmap v740: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-07-26T07:59:23.732 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:23 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-26T07:59:23.732 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:23 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.i.service: Consumed 13.764s CPU time. 2024-07-26T07:59:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:23 smithi181 ceph-mon[35869]: pgmap v740: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-07-26T07:59:23.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:23 smithi115 podman[69551]: 2024-07-26 07:59:23.734510949 +0000 UTC m=+1.576760916 container died 0940ab2ad1c41a58e461c71ee7a35b17ce5ab826838f35aa44c0ec47ed9cfd0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T07:59:23.974 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:23 smithi115 podman[69550]: 2024-07-26 07:59:23.720574297 +0000 UTC m=+1.570791805 container remove fa4447fff829e0e16149fb4953e54f689c5cd329108b86761aa4248de8c8e652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-07-26T07:59:23.974 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:23 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T07:59:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:23 smithi097 ceph-mon[32231]: pgmap v740: 97 pgs: 97 active+clean; 208 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-07-26T07:59:24.724 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:24 smithi115 podman[69549]: 2024-07-26 07:59:24.281977221 +0000 UTC m=+2.134539464 container remove a7747372ca68603d4423886415a8244c89f9509a3364d39dd5a254168924c1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:59:24.724 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:24 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-07-26T07:59:25.172 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:24 smithi115 podman[69551]: 2024-07-26 07:59:24.751092442 +0000 UTC m=+2.593342422 container remove 0940ab2ad1c41a58e461c71ee7a35b17ce5ab826838f35aa44c0ec47ed9cfd0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-07-26T07:59:25.173 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:24 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T07:59:25.173 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:24 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T07:59:25.173 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:24 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.h.service: Consumed 1min 14.976s CPU time. 2024-07-26T07:59:25.174 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:24 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-07-26T07:59:25.174 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:24 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.k.service: Consumed 17.813s CPU time. 2024-07-26T07:59:25.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:25 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T07:59:25.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:25 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 11.370s CPU time. 2024-07-26T07:59:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:25 smithi115 ceph-mon[35364]: pgmap v741: 97 pgs: 97 active+clean; 209 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.0 MiB/s wr, 90 op/s 2024-07-26T07:59:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:25 smithi181 ceph-mon[35869]: pgmap v741: 97 pgs: 97 active+clean; 209 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.0 MiB/s wr, 90 op/s 2024-07-26T07:59:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:25 smithi097 ceph-mon[32231]: pgmap v741: 97 pgs: 97 active+clean; 209 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.0 MiB/s wr, 90 op/s 2024-07-26T07:59:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:27 smithi115 ceph-mon[35364]: pgmap v742: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 674 KiB/s wr, 64 op/s 2024-07-26T07:59:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:27 smithi181 ceph-mon[35869]: pgmap v742: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 674 KiB/s wr, 64 op/s 2024-07-26T07:59:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:27 smithi097 ceph-mon[32231]: pgmap v742: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 674 KiB/s wr, 64 op/s 2024-07-26T07:59:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:29 smithi115 ceph-mon[35364]: pgmap v743: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 214 KiB/s wr, 28 op/s 2024-07-26T07:59:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:29 smithi181 ceph-mon[35869]: pgmap v743: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 214 KiB/s wr, 28 op/s 2024-07-26T07:59:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:29 smithi097 ceph-mon[32231]: pgmap v743: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 214 KiB/s wr, 28 op/s 2024-07-26T07:59:30.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:30.711+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:59:30.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:30.713+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T07:59:30.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:30.713+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:59:30.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:30.713+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:59:30.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:30.713+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:31 smithi115 ceph-mon[35364]: pgmap v744: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 210 KiB/s wr, 28 op/s 2024-07-26T07:59:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:31 smithi181 ceph-mon[35869]: pgmap v744: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 210 KiB/s wr, 28 op/s 2024-07-26T07:59:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:31 smithi097 ceph-mon[32231]: pgmap v744: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 210 KiB/s wr, 28 op/s 2024-07-26T07:59:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:32.330+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:59:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:32.331+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T07:59:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:32.331+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:59:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:32.331+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:59:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:32.331+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:33.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:33 smithi181 ceph-mon[35869]: pgmap v745: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 142 KiB/s wr, 18 op/s 2024-07-26T07:59:33.884 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:33 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-07-26T07:59:33.884 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:33 smithi181 systemd[1]: Stopped Ceph mds.i for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:33.884 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:33 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.i.service: Consumed 13.764s CPU time. 2024-07-26T07:59:33.884 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:33 smithi181 systemd[1]: Starting Ceph mds.i for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:59:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:33 smithi115 ceph-mon[35364]: pgmap v745: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 142 KiB/s wr, 18 op/s 2024-07-26T07:59:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:33 smithi097 ceph-mon[32231]: pgmap v745: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 142 KiB/s wr, 18 op/s 2024-07-26T07:59:34.195 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:33 smithi181 podman[70007]: 2024-07-26 07:59:33.786674499 +0000 UTC m=+0.031782528 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:59:34.195 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:33 smithi181 podman[70007]: 2024-07-26 07:59:33.956890869 +0000 UTC m=+0.201998932 container create cf78de60d5820bd9176749000a9465441c53f81ff21773eaeb37465dc93250e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T07:59:34.483 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:34 smithi181 podman[70007]: 2024-07-26 07:59:34.193828238 +0000 UTC m=+0.438936305 container init cf78de60d5820bd9176749000a9465441c53f81ff21773eaeb37465dc93250e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-07-26T07:59:34.483 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:34 smithi181 podman[70007]: 2024-07-26 07:59:34.203608476 +0000 UTC m=+0.448716509 container start cf78de60d5820bd9176749000a9465441c53f81ff21773eaeb37465dc93250e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:59:34.483 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:34 smithi181 bash[70007]: cf78de60d5820bd9176749000a9465441c53f81ff21773eaeb37465dc93250e8 2024-07-26T07:59:34.483 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:34 smithi181 ceph-mds[70026]: -- 172.21.15.181:0/1242107500 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af64f15860 con 0x55af690b7a80 2024-07-26T07:59:34.484 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:34 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-i[70022]: starting mds.i at 2024-07-26T07:59:34.484 INFO:journalctl@ceph.mds.i.smithi181.stdout:Jul 26 07:59:34 smithi181 systemd[1]: Started Ceph mds.i for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:34.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:34 smithi097 ceph-mon[32231]: Replacing daemon mds.i as rank 4 with standby daemon mds.g 2024-07-26T07:59:34.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:34 smithi097 ceph-mon[32231]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:34.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:34 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:59:34.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:34 smithi097 ceph-mon[32231]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T07:59:34.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:34 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T07:59:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:34 smithi115 ceph-mon[35364]: Replacing daemon mds.i as rank 4 with standby daemon mds.g 2024-07-26T07:59:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:34 smithi115 ceph-mon[35364]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:34 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:59:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:34 smithi115 ceph-mon[35364]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T07:59:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:34 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T07:59:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:34 smithi181 ceph-mon[35869]: Replacing daemon mds.i as rank 4 with standby daemon mds.g 2024-07-26T07:59:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:34 smithi181 ceph-mon[35869]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:34 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:59:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:34 smithi181 ceph-mon[35869]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T07:59:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:34 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T07:59:35.246 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:34 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-07-26T07:59:35.248 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:34 smithi115 systemd[1]: Stopped Ceph mds.k for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:35.248 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:34 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.k.service: Consumed 17.813s CPU time. 2024-07-26T07:59:35.248 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:35 smithi115 systemd[1]: Starting Ceph mds.k for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:59:35.248 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:35 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-07-26T07:59:35.248 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:35 smithi115 systemd[1]: Stopped Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:35.249 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:35 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 11.370s CPU time. 2024-07-26T07:59:35.249 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:34 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-07-26T07:59:35.249 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:34 smithi115 systemd[1]: Stopped Ceph mds.h for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:35.249 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:34 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.h.service: Consumed 1min 14.976s CPU time. 2024-07-26T07:59:35.249 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:34 smithi115 systemd[1]: Starting Ceph mds.h for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:59:35.503 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:35 smithi115 systemd[1]: Starting Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:59:35.503 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:35 smithi115 podman[69776]: 2024-07-26 07:59:35.284632817 +0000 UTC m=+0.037029316 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:59:35.503 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:35 smithi115 podman[69775]: 2024-07-26 07:59:35.280516834 +0000 UTC m=+0.035722798 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:59:35.503 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:35 smithi115 podman[69775]: 2024-07-26 07:59:35.401380245 +0000 UTC m=+0.156586229 container create 38f584e59fbb6b458708af936088e9f7ce70b735cc6ab9564f4a0afcc96521d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T07:59:35.824 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:35 smithi115 podman[69776]: 2024-07-26 07:59:35.502253988 +0000 UTC m=+0.254650521 container create 3006186e2a0787311cb7d5a52b29ba6086c81e79f4d460633602da4ebee55f0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-07-26T07:59:36.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: pgmap v747: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 50 KiB/s wr, 11 op/s 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:resolve 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6838/3015684238,v1:172.21.15.181:6839/3015684238] up:boot 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: Replacing daemon mds.h as rank 1 with standby daemon mds.f 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T07:59:36.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mon[35364]: daemon mds.k restarted 2024-07-26T07:59:36.135 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:35 smithi115 podman[69776]: 2024-07-26 07:59:35.882235745 +0000 UTC m=+0.634632286 container init 3006186e2a0787311cb7d5a52b29ba6086c81e79f4d460633602da4ebee55f0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-26T07:59:36.135 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:35 smithi115 podman[69776]: 2024-07-26 07:59:35.891642381 +0000 UTC m=+0.644038925 container start 3006186e2a0787311cb7d5a52b29ba6086c81e79f4d460633602da4ebee55f0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:59:36.135 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mds[69822]: -- 172.21.15.115:0/3657622549 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558b57327c20 con 0x558b5b4c9180 2024-07-26T07:59:36.135 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-k[69814]: starting mds.k at 2024-07-26T07:59:36.135 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:36 smithi115 podman[69775]: 2024-07-26 07:59:36.023466583 +0000 UTC m=+0.778672560 container init 38f584e59fbb6b458708af936088e9f7ce70b735cc6ab9564f4a0afcc96521d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T07:59:36.135 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:36 smithi115 podman[69775]: 2024-07-26 07:59:36.031913566 +0000 UTC m=+0.787119515 container start 38f584e59fbb6b458708af936088e9f7ce70b735cc6ab9564f4a0afcc96521d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-26T07:59:36.135 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:36 smithi115 bash[69775]: 38f584e59fbb6b458708af936088e9f7ce70b735cc6ab9564f4a0afcc96521d4 2024-07-26T07:59:36.474 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:36 smithi115 bash[69776]: 3006186e2a0787311cb7d5a52b29ba6086c81e79f4d460633602da4ebee55f0a 2024-07-26T07:59:36.474 INFO:journalctl@ceph.mds.k.smithi115.stdout:Jul 26 07:59:36 smithi115 systemd[1]: Started Ceph mds.k for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:36.474 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:36 smithi115 systemd[1]: Started Ceph mds.h for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:36.474 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-mds[69843]: -- 172.21.15.115:0/3718929964 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55733d58f860 con 0x557341731a80 2024-07-26T07:59:36.474 INFO:journalctl@ceph.mds.h.smithi115.stdout:Jul 26 07:59:36 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-h[69818]: starting mds.h at 2024-07-26T07:59:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: pgmap v747: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 50 KiB/s wr, 11 op/s 2024-07-26T07:59:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:resolve 2024-07-26T07:59:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6838/3015684238,v1:172.21.15.181:6839/3015684238] up:boot 2024-07-26T07:59:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: Replacing daemon mds.h as rank 1 with standby daemon mds.f 2024-07-26T07:59:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:59:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T07:59:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T07:59:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T07:59:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:36 smithi181 ceph-mon[35869]: daemon mds.k restarted 2024-07-26T07:59:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: pgmap v747: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 50 KiB/s wr, 11 op/s 2024-07-26T07:59:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:resolve 2024-07-26T07:59:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6838/3015684238,v1:172.21.15.181:6839/3015684238] up:boot 2024-07-26T07:59:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: Replacing daemon mds.h as rank 1 with standby daemon mds.f 2024-07-26T07:59:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:59:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T07:59:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T07:59:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T07:59:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:36 smithi097 ceph-mon[32231]: daemon mds.k restarted 2024-07-26T07:59:36.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:36 smithi115 podman[69897]: 2024-07-26 07:59:36.544919845 +0000 UTC m=+0.036078995 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:59:36.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:36 smithi115 podman[69897]: 2024-07-26 07:59:36.700262471 +0000 UTC m=+0.191421643 container create 3e30665f37099433207720f18e41d648973edab0570fee2fecc2e67e96dea802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:59:37.182 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T07:59:37.182 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:resolve 2024-07-26T07:59:37.182 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6832/3398483973,v1:172.21.15.115:6833/3398483973] up:boot 2024-07-26T07:59:37.182 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: Replacing daemon mds.e as rank 2 with standby daemon mds.l 2024-07-26T07:59:37.182 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:37.182 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T07:59:37.183 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,1=f=up:resolve,2=e=up:active,4=g=up:resolve} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T07:59:37.183 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T07:59:37.183 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T07:59:37.183 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:37 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 1 up:standby-replay 4 up:standby 2 up:replay 2 up:resolve 1 up:active 2024-07-26T07:59:37.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:37 smithi115 podman[69897]: 2024-07-26 07:59:37.008518188 +0000 UTC m=+0.499677393 container init 3e30665f37099433207720f18e41d648973edab0570fee2fecc2e67e96dea802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T07:59:37.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:37 smithi115 podman[69897]: 2024-07-26 07:59:37.01833321 +0000 UTC m=+0.509492382 container start 3e30665f37099433207720f18e41d648973edab0570fee2fecc2e67e96dea802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-26T07:59:37.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:37 smithi115 bash[69897]: 3e30665f37099433207720f18e41d648973edab0570fee2fecc2e67e96dea802 2024-07-26T07:59:37.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:37 smithi115 systemd[1]: Started Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:37.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mds[69915]: -- 172.21.15.115:0/3584296261 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565113c51860 con 0x565117df3180 2024-07-26T07:59:37.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: starting mds.e at 2024-07-26T07:59:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:resolve 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6832/3398483973,v1:172.21.15.115:6833/3398483973] up:boot 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: Replacing daemon mds.e as rank 2 with standby daemon mds.l 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,1=f=up:resolve,2=e=up:active,4=g=up:resolve} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T07:59:37.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:37 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 1 up:standby-replay 4 up:standby 2 up:replay 2 up:resolve 1 up:active 2024-07-26T07:59:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T07:59:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:resolve 2024-07-26T07:59:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6832/3398483973,v1:172.21.15.115:6833/3398483973] up:boot 2024-07-26T07:59:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: Replacing daemon mds.e as rank 2 with standby daemon mds.l 2024-07-26T07:59:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T07:59:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T07:59:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,1=f=up:resolve,2=e=up:active,4=g=up:resolve} 3 up:standby-replay 4 up:standby, 1 failed 2024-07-26T07:59:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T07:59:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T07:59:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:37 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 1 up:standby-replay 4 up:standby 2 up:replay 2 up:resolve 1 up:active 2024-07-26T07:59:38.379 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:38 smithi115 ceph-mon[35364]: pgmap v751: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 14 op/s 2024-07-26T07:59:38.379 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:38 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:resolve 2024-07-26T07:59:38.379 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:38 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6834/1128163649,v1:172.21.15.115:6835/1128163649] up:boot 2024-07-26T07:59:38.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:38 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:resolve 2024-07-26T07:59:38.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:38 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 4 up:resolve 1 up:active 2024-07-26T07:59:38.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T07:59:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:38 smithi181 ceph-mon[35869]: pgmap v751: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 14 op/s 2024-07-26T07:59:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:38 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:resolve 2024-07-26T07:59:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:38 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6834/1128163649,v1:172.21.15.115:6835/1128163649] up:boot 2024-07-26T07:59:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:38 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:resolve 2024-07-26T07:59:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:38 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 4 up:resolve 1 up:active 2024-07-26T07:59:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T07:59:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:38 smithi097 ceph-mon[32231]: pgmap v751: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 14 op/s 2024-07-26T07:59:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:38 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:resolve 2024-07-26T07:59:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:38 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6834/1128163649,v1:172.21.15.115:6835/1128163649] up:boot 2024-07-26T07:59:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:38 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:resolve 2024-07-26T07:59:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:38 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 4 up:resolve 1 up:active 2024-07-26T07:59:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T07:59:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:reconnect 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:reconnect 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:reconnect 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:reconnect 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:boot 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899982 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0149997 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0189997 2024-07-26T07:59:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:39 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0259995 2024-07-26T07:59:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:reconnect 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:reconnect 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:reconnect 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:reconnect 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:boot 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899982 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0149997 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0189997 2024-07-26T07:59:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:39 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0259995 2024-07-26T07:59:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:reconnect 2024-07-26T07:59:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:reconnect 2024-07-26T07:59:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:reconnect 2024-07-26T07:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:reconnect 2024-07-26T07:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:boot 2024-07-26T07:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899982 2024-07-26T07:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T07:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T07:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0149997 2024-07-26T07:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0189997 2024-07-26T07:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:39 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0259995 2024-07-26T07:59:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: pgmap v752: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 9 op/s 2024-07-26T07:59:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:rejoin 2024-07-26T07:59:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:rejoin 2024-07-26T07:59:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:rejoin 2024-07-26T07:59:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:rejoin 2024-07-26T07:59:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T07:59:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T07:59:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T07:59:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T07:59:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:40 smithi115 ceph-mon[35364]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T07:59:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: pgmap v752: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 9 op/s 2024-07-26T07:59:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:rejoin 2024-07-26T07:59:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:rejoin 2024-07-26T07:59:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:rejoin 2024-07-26T07:59:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:rejoin 2024-07-26T07:59:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T07:59:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T07:59:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T07:59:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T07:59:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-mon[35869]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: *** Caught signal (Aborted) ** 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: in thread 7fde12200640 thread_name:ms_dispatch 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fde1843e6f0] 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fde1848b94c] 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 3: raise() 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 4: abort() 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 5: /lib64/libc.so.6(+0x2871b) [0x7fde1842871b] 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 6: /lib64/libc.so.6(+0x37386) [0x7fde18437386] 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563cf2a146b9] 2024-07-26T07:59:40.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563cf2bd280c] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563cf2b122ca] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563cf2ae2753] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563cf2ae3a1b] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563cf2ac7752] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563cf2a15a56] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563cf2a19637] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563cf2a19f5e] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563cf29fd03a] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fde18f9b407] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 18: (DispatchQueue::entry()+0x60f) [0x7fde18f9bcf9] 2024-07-26T07:59:40.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fde190785a1] 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 20: (Thread::entry_wrapper()+0x43) [0x7fde18e25cdd] 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 21: (Thread::_entry_func(void*)+0xd) [0x7fde18e25cf9] 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 22: /lib64/libc.so.6(+0x89c02) [0x7fde18489c02] 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fde1850ec40] 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2024-07-26T07:59:40.108+0000 7fde12200640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: in thread 7fde12200640 thread_name:ms_dispatch 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:40.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fde1843e6f0] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fde1848b94c] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 3: raise() 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 4: abort() 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 5: /lib64/libc.so.6(+0x2871b) [0x7fde1842871b] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 6: /lib64/libc.so.6(+0x37386) [0x7fde18437386] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563cf2a146b9] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563cf2bd280c] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563cf2b122ca] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563cf2ae2753] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563cf2ae3a1b] 2024-07-26T07:59:40.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563cf2ac7752] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563cf2a15a56] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563cf2a19637] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563cf2a19f5e] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563cf29fd03a] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fde18f9b407] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 18: (DispatchQueue::entry()+0x60f) [0x7fde18f9bcf9] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fde190785a1] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 20: (Thread::entry_wrapper()+0x43) [0x7fde18e25cdd] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 21: (Thread::_entry_func(void*)+0xd) [0x7fde18e25cf9] 2024-07-26T07:59:40.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 22: /lib64/libc.so.6(+0x89c02) [0x7fde18489c02] 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fde1850ec40] 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 0> 2024-07-26T07:59:40.108+0000 7fde12200640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: in thread 7fde12200640 thread_name:ms_dispatch 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fde1843e6f0] 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fde1848b94c] 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 3: raise() 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 4: abort() 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 5: /lib64/libc.so.6(+0x2871b) [0x7fde1842871b] 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 6: /lib64/libc.so.6(+0x37386) [0x7fde18437386] 2024-07-26T07:59:40.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563cf2a146b9] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563cf2bd280c] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563cf2b122ca] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563cf2ae2753] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563cf2ae3a1b] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563cf2ac7752] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563cf2a15a56] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563cf2a19637] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563cf2a19f5e] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563cf29fd03a] 2024-07-26T07:59:40.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fde18f9b407] 2024-07-26T07:59:40.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 18: (DispatchQueue::entry()+0x60f) [0x7fde18f9bcf9] 2024-07-26T07:59:40.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fde190785a1] 2024-07-26T07:59:40.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 20: (Thread::entry_wrapper()+0x43) [0x7fde18e25cdd] 2024-07-26T07:59:40.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 21: (Thread::_entry_func(void*)+0xd) [0x7fde18e25cf9] 2024-07-26T07:59:40.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 22: /lib64/libc.so.6(+0x89c02) [0x7fde18489c02] 2024-07-26T07:59:40.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fde1850ec40] 2024-07-26T07:59:40.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:40.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 0> 2024-07-26T07:59:40.108+0000 7fde12200640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: in thread 7fde12200640 thread_name:ms_dispatch 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fde1843e6f0] 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fde1848b94c] 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 3: raise() 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 4: abort() 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 5: /lib64/libc.so.6(+0x2871b) [0x7fde1842871b] 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 6: /lib64/libc.so.6(+0x37386) [0x7fde18437386] 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563cf2a146b9] 2024-07-26T07:59:40.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563cf2bd280c] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563cf2b122ca] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563cf2ae2753] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563cf2ae3a1b] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563cf2ac7752] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563cf2a15a56] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563cf2a19637] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563cf2a19f5e] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563cf29fd03a] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fde18f9b407] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 18: (DispatchQueue::entry()+0x60f) [0x7fde18f9bcf9] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fde190785a1] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 20: (Thread::entry_wrapper()+0x43) [0x7fde18e25cdd] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 21: (Thread::_entry_func(void*)+0xd) [0x7fde18e25cf9] 2024-07-26T07:59:40.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 22: /lib64/libc.so.6(+0x89c02) [0x7fde18489c02] 2024-07-26T07:59:40.494 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fde1850ec40] 2024-07-26T07:59:40.494 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:40.494 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[65600]: 2024-07-26T07:59:40.494 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:40 smithi181 podman[70075]: 2024-07-26 07:59:40.23222738 +0000 UTC m=+0.031581127 container died 15af4714afedd9e96861fee038e9989a13e2ca71decf3512280ec039b0e0ae57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-26T07:59:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: pgmap v752: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 9 op/s 2024-07-26T07:59:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:rejoin 2024-07-26T07:59:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:rejoin 2024-07-26T07:59:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:rejoin 2024-07-26T07:59:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:rejoin 2024-07-26T07:59:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T07:59:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T07:59:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T07:59:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T07:59:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:40 smithi097 ceph-mon[32231]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T07:59:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:41 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:59:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:41 smithi115 ceph-mon[35364]: Cluster is now healthy 2024-07-26T07:59:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:41 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:active 2024-07-26T07:59:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:41 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:active 2024-07-26T07:59:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:41 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:active 2024-07-26T07:59:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:41 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:active 2024-07-26T07:59:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:41 smithi115 ceph-mon[35364]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T07:59:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:41 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:59:41.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:41 smithi181 podman[70075]: 2024-07-26 07:59:41.015781899 +0000 UTC m=+0.815135684 container remove 15af4714afedd9e96861fee038e9989a13e2ca71decf3512280ec039b0e0ae57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:41 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:41 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:41 smithi181 ceph-mon[35869]: Cluster is now healthy 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:41 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:active 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:41 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:active 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:41 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:active 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:41 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:active 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:41 smithi181 ceph-mon[35869]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T07:59:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:41 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:59:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:41 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:59:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:41 smithi097 ceph-mon[32231]: Cluster is now healthy 2024-07-26T07:59:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:41 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6840/4022202837,v1:172.21.15.97:6841/4022202837] up:active 2024-07-26T07:59:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:41 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6838/4174464953,v1:172.21.15.97:6839/4174464953] up:active 2024-07-26T07:59:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:41 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6832/1511551515,v1:172.21.15.181:6833/1511551515] up:active 2024-07-26T07:59:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:41 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4001900111,v1:172.21.15.181:6835/4001900111] up:active 2024-07-26T07:59:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:41 smithi097 ceph-mon[32231]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T07:59:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:41 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:59:41.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:41 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T07:59:41.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:41 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 5.404s CPU time. 2024-07-26T07:59:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:42 smithi115 ceph-mon[35364]: pgmap v753: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 341 B/s wr, 16 op/s 2024-07-26T07:59:42.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:42 smithi181 ceph-mon[35869]: pgmap v753: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 341 B/s wr, 16 op/s 2024-07-26T07:59:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:42 smithi097 ceph-mon[32231]: pgmap v753: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 341 B/s wr, 16 op/s 2024-07-26T07:59:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:44 smithi115 ceph-mon[35364]: pgmap v754: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 556 B/s wr, 18 op/s 2024-07-26T07:59:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:44 smithi181 ceph-mon[35869]: pgmap v754: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 556 B/s wr, 18 op/s 2024-07-26T07:59:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:44 smithi097 ceph-mon[32231]: pgmap v754: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 556 B/s wr, 18 op/s 2024-07-26T07:59:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:45.719+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:59:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:45.721+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T07:59:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:45.721+0000 7f1955800640 1 -- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 msgr2=0x56373d0a9ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:59:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:45.721+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:59:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-07-26T07:59:45.721+0000 7f1955800640 1 --2- 172.21.15.97:0/3705515786 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x56373cca41d0 0x56373d0a9ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:46 smithi097 ceph-mon[32231]: pgmap v755: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.3 KiB/s wr, 30 op/s 2024-07-26T07:59:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:46 smithi115 ceph-mon[35364]: pgmap v755: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.3 KiB/s wr, 30 op/s 2024-07-26T07:59:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:46 smithi181 ceph-mon[35869]: pgmap v755: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 3.3 KiB/s wr, 30 op/s 2024-07-26T07:59:47.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:47.340+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:59:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:47.341+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T07:59:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:47.341+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T07:59:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:47.341+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:59:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T07:59:47.341+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:48 smithi115 ceph-mon[35364]: pgmap v756: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 5.7 KiB/s wr, 33 op/s 2024-07-26T07:59:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:48 smithi181 ceph-mon[35869]: pgmap v756: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 5.7 KiB/s wr, 33 op/s 2024-07-26T07:59:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:48 smithi097 ceph-mon[32231]: pgmap v756: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 5.7 KiB/s wr, 33 op/s 2024-07-26T07:59:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:50 smithi115 ceph-mon[35364]: pgmap v757: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 4.9 KiB/s wr, 30 op/s 2024-07-26T07:59:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:50 smithi181 ceph-mon[35869]: pgmap v757: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 4.9 KiB/s wr, 30 op/s 2024-07-26T07:59:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:50 smithi097 ceph-mon[32231]: pgmap v757: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 4.9 KiB/s wr, 30 op/s 2024-07-26T07:59:52.122 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:51 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-07-26T07:59:52.123 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:51 smithi181 systemd[1]: Stopped Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:52.123 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:51 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 5.404s CPU time. 2024-07-26T07:59:52.123 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:51 smithi181 systemd[1]: Starting Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T07:59:52.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:52 smithi181 ceph-mon[35869]: pgmap v758: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 7.8 KiB/s wr, 31 op/s 2024-07-26T07:59:52.385 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:52 smithi181 podman[70171]: 2024-07-26 07:59:52.027716116 +0000 UTC m=+0.036122666 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T07:59:52.385 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:52 smithi181 podman[70171]: 2024-07-26 07:59:52.131213362 +0000 UTC m=+0.139619923 container create 9f3a24bb12184f471183768ce933cc94216847792906bcf89fa24803cddb732f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T07:59:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:52 smithi115 ceph-mon[35364]: pgmap v758: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 7.8 KiB/s wr, 31 op/s 2024-07-26T07:59:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:52 smithi097 ceph-mon[32231]: pgmap v758: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 7.8 KiB/s wr, 31 op/s 2024-07-26T07:59:52.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:52 smithi181 podman[70171]: 2024-07-26 07:59:52.384458346 +0000 UTC m=+0.392864926 container init 9f3a24bb12184f471183768ce933cc94216847792906bcf89fa24803cddb732f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T07:59:52.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:52 smithi181 podman[70171]: 2024-07-26 07:59:52.393960817 +0000 UTC m=+0.402367361 container start 9f3a24bb12184f471183768ce933cc94216847792906bcf89fa24803cddb732f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T07:59:52.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:52 smithi181 bash[70171]: 9f3a24bb12184f471183768ce933cc94216847792906bcf89fa24803cddb732f 2024-07-26T07:59:52.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:52 smithi181 systemd[1]: Started Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T07:59:52.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:52 smithi181 ceph-mds[70190]: -- 172.21.15.181:0/835671906 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56076a263c20 con 0x56076e405a80 2024-07-26T07:59:52.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 07:59:52 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: starting mds.f at 2024-07-26T07:59:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:53 smithi115 ceph-mon[35364]: daemon mds.f restarted 2024-07-26T07:59:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:53 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:59:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:53 smithi115 ceph-mon[35364]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T07:59:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:53 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:boot 2024-07-26T07:59:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:53 smithi115 ceph-mon[35364]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T07:59:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:53 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T07:59:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T07:59:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:53 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T07:59:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:53 smithi181 ceph-mon[35869]: daemon mds.f restarted 2024-07-26T07:59:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:53 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:59:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:53 smithi181 ceph-mon[35869]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T07:59:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:53 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:boot 2024-07-26T07:59:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:53 smithi181 ceph-mon[35869]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T07:59:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:53 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T07:59:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T07:59:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:53 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T07:59:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:53 smithi097 ceph-mon[32231]: daemon mds.f restarted 2024-07-26T07:59:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:53 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T07:59:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:53 smithi097 ceph-mon[32231]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T07:59:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:53 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:boot 2024-07-26T07:59:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:53 smithi097 ceph-mon[32231]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T07:59:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:53 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T07:59:53.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T07:59:53.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:53 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T07:59:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:54 smithi115 ceph-mon[35364]: pgmap v759: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-07-26T07:59:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:54 smithi181 ceph-mon[35869]: pgmap v759: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-07-26T07:59:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:54 smithi097 ceph-mon[32231]: pgmap v759: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-07-26T07:59:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:55 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:resolve 2024-07-26T07:59:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:55 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:59:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:55 smithi115 ceph-mon[35364]: pgmap v761: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 KiB/s wr, 18 op/s 2024-07-26T07:59:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:55 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:resolve 2024-07-26T07:59:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:55 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:59:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:55 smithi181 ceph-mon[35869]: pgmap v761: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 KiB/s wr, 18 op/s 2024-07-26T07:59:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:55 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:resolve 2024-07-26T07:59:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:55 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T07:59:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:55 smithi097 ceph-mon[32231]: pgmap v761: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 KiB/s wr, 18 op/s 2024-07-26T07:59:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:56 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:reconnect 2024-07-26T07:59:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:56 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T07:59:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:56 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799985 2024-07-26T07:59:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:56 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:reconnect 2024-07-26T07:59:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:56 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T07:59:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:56 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799985 2024-07-26T07:59:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:56 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:reconnect 2024-07-26T07:59:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:56 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T07:59:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:56 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799985 2024-07-26T07:59:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-mon[35364]: pgmap v762: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 11 op/s 2024-07-26T07:59:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:rejoin 2024-07-26T07:59:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T07:59:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-mon[35364]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T07:59:57.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T07:59:57.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: *** Caught signal (Aborted) ** 2024-07-26T07:59:57.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: in thread 7fbb54c00640 thread_name:ms_dispatch 2024-07-26T07:59:57.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:57.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbb5ae3e6f0] 2024-07-26T07:59:57.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fbb5ae8b94c] 2024-07-26T07:59:57.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 3: raise() 2024-07-26T07:59:57.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 4: abort() 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 5: /lib64/libc.so.6(+0x2871b) [0x7fbb5ae2871b] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 6: /lib64/libc.so.6(+0x37386) [0x7fbb5ae37386] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5650eeaac6b9] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5650eec6a80c] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5650eebaa2ca] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5650eeb7a753] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5650eeb7ba1b] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5650eeb5f752] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5650eeaada56] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5650eeab1637] 2024-07-26T07:59:57.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5650eeab1f5e] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5650eea9503a] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fbb5b99b407] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 18: (DispatchQueue::entry()+0x60f) [0x7fbb5b99bcf9] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fbb5ba785a1] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 20: (Thread::entry_wrapper()+0x43) [0x7fbb5b825cdd] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 21: (Thread::_entry_func(void*)+0xd) [0x7fbb5b825cf9] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 22: /lib64/libc.so.6(+0x89c02) [0x7fbb5ae89c02] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fbb5af0ec40] 2024-07-26T07:59:57.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2024-07-26T07:59:57.541+0000 7fbb54c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: in thread 7fbb54c00640 thread_name:ms_dispatch 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbb5ae3e6f0] 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fbb5ae8b94c] 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 3: raise() 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 4: abort() 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 5: /lib64/libc.so.6(+0x2871b) [0x7fbb5ae2871b] 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 6: /lib64/libc.so.6(+0x37386) [0x7fbb5ae37386] 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5650eeaac6b9] 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5650eec6a80c] 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5650eebaa2ca] 2024-07-26T07:59:57.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5650eeb7a753] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5650eeb7ba1b] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5650eeb5f752] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5650eeaada56] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5650eeab1637] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5650eeab1f5e] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5650eea9503a] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fbb5b99b407] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 18: (DispatchQueue::entry()+0x60f) [0x7fbb5b99bcf9] 2024-07-26T07:59:57.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fbb5ba785a1] 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 20: (Thread::entry_wrapper()+0x43) [0x7fbb5b825cdd] 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 21: (Thread::_entry_func(void*)+0xd) [0x7fbb5b825cf9] 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 22: /lib64/libc.so.6(+0x89c02) [0x7fbb5ae89c02] 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fbb5af0ec40] 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 0> 2024-07-26T07:59:57.541+0000 7fbb54c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: in thread 7fbb54c00640 thread_name:ms_dispatch 2024-07-26T07:59:57.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbb5ae3e6f0] 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fbb5ae8b94c] 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 3: raise() 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 4: abort() 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 5: /lib64/libc.so.6(+0x2871b) [0x7fbb5ae2871b] 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 6: /lib64/libc.so.6(+0x37386) [0x7fbb5ae37386] 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5650eeaac6b9] 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5650eec6a80c] 2024-07-26T07:59:57.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5650eebaa2ca] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5650eeb7a753] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5650eeb7ba1b] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5650eeb5f752] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5650eeaada56] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5650eeab1637] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5650eeab1f5e] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5650eea9503a] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fbb5b99b407] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 18: (DispatchQueue::entry()+0x60f) [0x7fbb5b99bcf9] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fbb5ba785a1] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 20: (Thread::entry_wrapper()+0x43) [0x7fbb5b825cdd] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 21: (Thread::_entry_func(void*)+0xd) [0x7fbb5b825cf9] 2024-07-26T07:59:57.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 22: /lib64/libc.so.6(+0x89c02) [0x7fbb5ae89c02] 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fbb5af0ec40] 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 0> 2024-07-26T07:59:57.541+0000 7fbb54c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: in thread 7fbb54c00640 thread_name:ms_dispatch 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbb5ae3e6f0] 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fbb5ae8b94c] 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 3: raise() 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 4: abort() 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 5: /lib64/libc.so.6(+0x2871b) [0x7fbb5ae2871b] 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 6: /lib64/libc.so.6(+0x37386) [0x7fbb5ae37386] 2024-07-26T07:59:57.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5650eeaac6b9] 2024-07-26T07:59:57.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5650eec6a80c] 2024-07-26T07:59:57.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5650eebaa2ca] 2024-07-26T07:59:57.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5650eeb7a753] 2024-07-26T07:59:57.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5650eeb7ba1b] 2024-07-26T07:59:57.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5650eeb5f752] 2024-07-26T07:59:57.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5650eeaada56] 2024-07-26T07:59:57.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5650eeab1637] 2024-07-26T07:59:57.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5650eeab1f5e] 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:57 smithi181 ceph-mon[35869]: pgmap v762: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 11 op/s 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:57 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:rejoin 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:57 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:57 smithi181 ceph-mon[35869]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5650eea9503a] 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fbb5b99b407] 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 18: (DispatchQueue::entry()+0x60f) [0x7fbb5b99bcf9] 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fbb5ba785a1] 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 20: (Thread::entry_wrapper()+0x43) [0x7fbb5b825cdd] 2024-07-26T07:59:57.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 21: (Thread::_entry_func(void*)+0xd) [0x7fbb5b825cf9] 2024-07-26T07:59:57.987 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 22: /lib64/libc.so.6(+0x89c02) [0x7fbb5ae89c02] 2024-07-26T07:59:57.987 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fbb5af0ec40] 2024-07-26T07:59:57.987 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T07:59:57.987 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[69911]: 2024-07-26T07:59:57.987 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:57 smithi115 podman[69980]: 2024-07-26 07:59:57.71944501 +0000 UTC m=+0.037477702 container died 3e30665f37099433207720f18e41d648973edab0570fee2fecc2e67e96dea802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T07:59:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:57 smithi097 ceph-mon[32231]: pgmap v762: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 11 op/s 2024-07-26T07:59:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:57 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:rejoin 2024-07-26T07:59:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:57 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T07:59:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:57 smithi097 ceph-mon[32231]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T07:59:58.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:58 smithi115 podman[69980]: 2024-07-26 07:59:58.622035949 +0000 UTC m=+0.940068644 container remove 3e30665f37099433207720f18e41d648973edab0570fee2fecc2e67e96dea802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T07:59:58.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:58 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T07:59:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:58 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:59:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:58 smithi115 ceph-mon[35364]: Cluster is now healthy 2024-07-26T07:59:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:58 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:active 2024-07-26T07:59:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:58 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:59:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:58 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:59:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:58 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:59:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:58 smithi181 ceph-mon[35869]: Cluster is now healthy 2024-07-26T07:59:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:58 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:active 2024-07-26T07:59:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:58 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:59:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:58 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:59:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:58 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T07:59:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:58 smithi097 ceph-mon[32231]: Cluster is now healthy 2024-07-26T07:59:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:58 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3916364577,v1:172.21.15.115:6837/3916364577] up:active 2024-07-26T07:59:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:58 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T07:59:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:58 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T07:59:59.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:59 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T07:59:59.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 07:59:59 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.138s CPU time. 2024-07-26T07:59:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 07:59:59 smithi115 ceph-mon[35364]: pgmap v763: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 KiB/s wr, 11 op/s 2024-07-26T07:59:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 07:59:59 smithi181 ceph-mon[35869]: pgmap v763: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 KiB/s wr, 11 op/s 2024-07-26T08:00:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 07:59:59 smithi097 ceph-mon[32231]: pgmap v763: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 KiB/s wr, 11 op/s 2024-07-26T08:00:00.482 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:00:00 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:00:00.034+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:00:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:00 smithi115 ceph-mon[35364]: overall HEALTH_OK 2024-07-26T08:00:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:00 smithi181 ceph-mon[35869]: overall HEALTH_OK 2024-07-26T08:00:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:00 smithi097 ceph-mon[32231]: overall HEALTH_OK 2024-07-26T08:00:01.832 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:01 smithi097 ceph-mon[32231]: pgmap v764: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-07-26T08:00:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-07-26T08:00:01.892 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't parse response with error Expecting value: line 1 column 1 (char 0); rc: 124 (EMEDIUMTYPE); stdout: 2024-07-26T08:00:01.893 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 294, in do_quiesce response = json.loads(stdout) File "/usr/lib/python3.10/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 357, in _run self.do_quiesce(d) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 298, in do_quiesce raise RuntimeError(f"Error parsing quiesce response: {e}") RuntimeError: Error parsing quiesce response: Expecting value: line 1 column 1 (char 0) 2024-07-26T08:00:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:01 smithi115 ceph-mon[35364]: pgmap v764: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-07-26T08:00:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:01 smithi181 ceph-mon[35869]: pgmap v764: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-07-26T08:00:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:02.345+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:00:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:02.346+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:00:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:02.346+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:00:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:02.346+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:00:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:02.347+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:00:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:03 smithi115 ceph-mon[35364]: pgmap v765: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-07-26T08:00:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:03 smithi181 ceph-mon[35869]: pgmap v765: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-07-26T08:00:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:03 smithi097 ceph-mon[32231]: pgmap v765: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-07-26T08:00:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:05 smithi115 ceph-mon[35364]: pgmap v766: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-07-26T08:00:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:05 smithi181 ceph-mon[35869]: pgmap v766: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-07-26T08:00:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:05 smithi097 ceph-mon[32231]: pgmap v766: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-07-26T08:00:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:07 smithi115 ceph-mon[35364]: pgmap v767: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-07-26T08:00:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:07 smithi181 ceph-mon[35869]: pgmap v767: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-07-26T08:00:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:07 smithi097 ceph-mon[32231]: pgmap v767: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-07-26T08:00:09.603 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:09 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-07-26T08:00:09.603 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:09 smithi115 systemd[1]: Stopped Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:00:09.604 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:09 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.138s CPU time. 2024-07-26T08:00:09.604 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:09 smithi115 systemd[1]: Starting Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:00:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:09 smithi115 ceph-mon[35364]: pgmap v768: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-07-26T08:00:09.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:09 smithi115 podman[70073]: 2024-07-26 08:00:09.54090118 +0000 UTC m=+0.036955805 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:00:09.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:09 smithi115 podman[70073]: 2024-07-26 08:00:09.655523186 +0000 UTC m=+0.151577830 container create 46ab31efba43388fbf1f3d6838716efaef908bf10faae1663381338ade4755c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:00:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:09 smithi181 ceph-mon[35869]: pgmap v768: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-07-26T08:00:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:09 smithi097 ceph-mon[32231]: pgmap v768: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-07-26T08:00:10.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:09 smithi115 podman[70073]: 2024-07-26 08:00:09.984080093 +0000 UTC m=+0.480134743 container init 46ab31efba43388fbf1f3d6838716efaef908bf10faae1663381338ade4755c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-07-26T08:00:10.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:09 smithi115 podman[70073]: 2024-07-26 08:00:09.994478189 +0000 UTC m=+0.490532842 container start 46ab31efba43388fbf1f3d6838716efaef908bf10faae1663381338ade4755c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0) 2024-07-26T08:00:10.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:10 smithi115 bash[70073]: 46ab31efba43388fbf1f3d6838716efaef908bf10faae1663381338ade4755c6 2024-07-26T08:00:10.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:10 smithi115 systemd[1]: Started Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:00:10.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mds[70091]: -- 172.21.15.115:0/1237219661 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559daa13f860 con 0x559dae2e1180 2024-07-26T08:00:10.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: starting mds.e at 2024-07-26T08:00:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mon[35364]: daemon mds.e restarted 2024-07-26T08:00:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mon[35364]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T08:00:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:boot 2024-07-26T08:00:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mon[35364]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:00:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:00:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:10 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:10 smithi181 ceph-mon[35869]: daemon mds.e restarted 2024-07-26T08:00:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:10 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:10 smithi181 ceph-mon[35869]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T08:00:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:10 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:boot 2024-07-26T08:00:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:10 smithi181 ceph-mon[35869]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:00:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:10 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:00:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:10 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:10 smithi097 ceph-mon[32231]: daemon mds.e restarted 2024-07-26T08:00:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:10 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:10 smithi097 ceph-mon[32231]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T08:00:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:10 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:boot 2024-07-26T08:00:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:10 smithi097 ceph-mon[32231]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:00:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:10 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:00:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:10 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:12 smithi115 ceph-mon[35364]: pgmap v770: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 614 B/s wr, 10 op/s 2024-07-26T08:00:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:12 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:resolve 2024-07-26T08:00:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:12 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:12 smithi181 ceph-mon[35869]: pgmap v770: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 614 B/s wr, 10 op/s 2024-07-26T08:00:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:12 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:resolve 2024-07-26T08:00:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:12 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:12 smithi097 ceph-mon[32231]: pgmap v770: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 614 B/s wr, 10 op/s 2024-07-26T08:00:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:12 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:resolve 2024-07-26T08:00:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:12 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:13 smithi115 ceph-mon[35364]: pgmap v771: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:00:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:13 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0049999 2024-07-26T08:00:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:13 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:reconnect 2024-07-26T08:00:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:13 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:13 smithi181 ceph-mon[35869]: pgmap v771: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:00:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:13 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0049999 2024-07-26T08:00:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:13 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:reconnect 2024-07-26T08:00:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:13 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:13 smithi097 ceph-mon[32231]: pgmap v771: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:00:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:13 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.0049999 2024-07-26T08:00:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:13 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:reconnect 2024-07-26T08:00:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:13 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: *** Caught signal (Aborted) ** 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: in thread 7fb8c6c00640 thread_name:ms_dispatch 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb8cce3e6f0] 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb8cce8b94c] 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 3: raise() 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 4: abort() 2024-07-26T08:00:14.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb8cce2871b] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 6: /lib64/libc.so.6(+0x37386) [0x7fb8cce37386] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560748b986b9] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560748d5680c] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560748c962ca] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560748c66753] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560748c67a1b] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560748c4b752] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560748b99a56] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560748b9d637] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560748b9df5e] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560748b8103a] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb8cd99b407] 2024-07-26T08:00:14.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 18: (DispatchQueue::entry()+0x60f) [0x7fb8cd99bcf9] 2024-07-26T08:00:14.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb8cda785a1] 2024-07-26T08:00:14.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 20: (Thread::entry_wrapper()+0x43) [0x7fb8cd825cdd] 2024-07-26T08:00:14.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb8cd825cf9] 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb8cce89c02] 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb8ccf0ec40] 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2024-07-26T08:00:14.057+0000 7fb8c6c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: in thread 7fb8c6c00640 thread_name:ms_dispatch 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb8cce3e6f0] 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb8cce8b94c] 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 3: raise() 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 4: abort() 2024-07-26T08:00:14.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb8cce2871b] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 6: /lib64/libc.so.6(+0x37386) [0x7fb8cce37386] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560748b986b9] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560748d5680c] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560748c962ca] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560748c66753] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560748c67a1b] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560748c4b752] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560748b99a56] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560748b9d637] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560748b9df5e] 2024-07-26T08:00:14.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560748b8103a] 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb8cd99b407] 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 18: (DispatchQueue::entry()+0x60f) [0x7fb8cd99bcf9] 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb8cda785a1] 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 20: (Thread::entry_wrapper()+0x43) [0x7fb8cd825cdd] 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb8cd825cf9] 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb8cce89c02] 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb8ccf0ec40] 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 0> 2024-07-26T08:00:14.057+0000 7fb8c6c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: in thread 7fb8c6c00640 thread_name:ms_dispatch 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:14.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb8cce3e6f0] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb8cce8b94c] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 3: raise() 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 4: abort() 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb8cce2871b] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 6: /lib64/libc.so.6(+0x37386) [0x7fb8cce37386] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560748b986b9] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560748d5680c] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560748c962ca] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560748c66753] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560748c67a1b] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560748c4b752] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560748b99a56] 2024-07-26T08:00:14.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560748b9d637] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560748b9df5e] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560748b8103a] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb8cd99b407] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 18: (DispatchQueue::entry()+0x60f) [0x7fb8cd99bcf9] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb8cda785a1] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 20: (Thread::entry_wrapper()+0x43) [0x7fb8cd825cdd] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb8cd825cf9] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb8cce89c02] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb8ccf0ec40] 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2024-07-26T08:00:14.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 0> 2024-07-26T08:00:14.057+0000 7fb8c6c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: in thread 7fb8c6c00640 thread_name:ms_dispatch 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb8cce3e6f0] 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb8cce8b94c] 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 3: raise() 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 4: abort() 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb8cce2871b] 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 6: /lib64/libc.so.6(+0x37386) [0x7fb8cce37386] 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560748b986b9] 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560748d5680c] 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560748c962ca] 2024-07-26T08:00:14.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560748c66753] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560748c67a1b] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560748c4b752] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560748b99a56] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560748b9d637] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560748b9df5e] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560748b8103a] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb8cd99b407] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 18: (DispatchQueue::entry()+0x60f) [0x7fb8cd99bcf9] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb8cda785a1] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 20: (Thread::entry_wrapper()+0x43) [0x7fb8cd825cdd] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb8cd825cf9] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb8cce89c02] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb8ccf0ec40] 2024-07-26T08:00:14.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:14.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70186]: 2024-07-26T08:00:14.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 podman[70291]: 2024-07-26 08:00:14.206429191 +0000 UTC m=+0.033253773 container died 9f3a24bb12184f471183768ce933cc94216847792906bcf89fa24803cddb732f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:00:14.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:14 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:rejoin 2024-07-26T08:00:14.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:14 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:14.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:14 smithi097 ceph-mon[32231]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:00:14.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:00:14.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:14 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:14.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:14 smithi097 ceph-mon[32231]: Cluster is now healthy 2024-07-26T08:00:14.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:14 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:active 2024-07-26T08:00:14.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:14 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:14.956 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:rejoin 2024-07-26T08:00:14.956 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:14.956 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-mon[35869]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:00:14.956 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:00:14.956 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:14.956 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-mon[35869]: Cluster is now healthy 2024-07-26T08:00:14.957 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:active 2024-07-26T08:00:14.957 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:14 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:14.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:14 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:rejoin 2024-07-26T08:00:14.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:14 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:14.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:14 smithi115 ceph-mon[35364]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:00:14.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:00:14.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:14 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:14.964 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:14 smithi115 ceph-mon[35364]: Cluster is now healthy 2024-07-26T08:00:14.964 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:14 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/891011565,v1:172.21.15.181:6835/891011565] up:active 2024-07-26T08:00:14.964 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:14 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:15.232 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:00:15 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:00:15.035+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:00:15.232 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 podman[70291]: 2024-07-26 08:00:14.954181612 +0000 UTC m=+0.781006221 container remove 9f3a24bb12184f471183768ce933cc94216847792906bcf89fa24803cddb732f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-07-26T08:00:15.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:14 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:00:15.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:15 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:00:15.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:15 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 1.928s CPU time. 2024-07-26T08:00:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:16 smithi115 ceph-mon[35364]: pgmap v772: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 10 op/s 2024-07-26T08:00:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:16 smithi181 ceph-mon[35869]: pgmap v772: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 10 op/s 2024-07-26T08:00:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:16 smithi097 ceph-mon[32231]: pgmap v772: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 10 op/s 2024-07-26T08:00:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:17.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:00:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:17.355+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:00:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:17.355+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:00:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:17.355+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:00:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:17.355+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:00:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:17 smithi115 ceph-mon[35364]: pgmap v773: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:17 smithi181 ceph-mon[35869]: pgmap v773: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:17 smithi097 ceph-mon[32231]: pgmap v773: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:00:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:00:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:00:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:19 smithi115 ceph-mon[35364]: pgmap v774: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T08:00:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:00:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:19 smithi115 ceph-mon[35364]: pgmap v775: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T08:00:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:19 smithi181 ceph-mon[35869]: pgmap v774: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T08:00:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:00:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:19 smithi181 ceph-mon[35869]: pgmap v775: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T08:00:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:19 smithi097 ceph-mon[32231]: pgmap v774: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T08:00:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:00:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:19 smithi097 ceph-mon[32231]: pgmap v775: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T08:00:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:00:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:20 smithi115 ceph-mon[35364]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:00:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:20 smithi181 ceph-mon[35869]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:00:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:20 smithi097 ceph-mon[32231]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:00:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:21 smithi115 ceph-mon[35364]: pgmap v776: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T08:00:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:21 smithi181 ceph-mon[35869]: pgmap v776: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T08:00:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:21 smithi097 ceph-mon[32231]: pgmap v776: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T08:00:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:23 smithi115 ceph-mon[35364]: pgmap v777: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-26T08:00:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:23 smithi181 ceph-mon[35869]: pgmap v777: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-26T08:00:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:23 smithi097 ceph-mon[32231]: pgmap v777: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-26T08:00:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:25 smithi115 ceph-mon[35364]: pgmap v778: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 303 B/s wr, 11 op/s 2024-07-26T08:00:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:25 smithi181 ceph-mon[35869]: pgmap v778: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 303 B/s wr, 11 op/s 2024-07-26T08:00:25.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:25 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-07-26T08:00:25.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:25 smithi181 systemd[1]: Stopped Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:00:25.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:25 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 1.928s CPU time. 2024-07-26T08:00:25.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:25 smithi181 systemd[1]: Starting Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:00:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:25 smithi097 ceph-mon[32231]: pgmap v778: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 303 B/s wr, 11 op/s 2024-07-26T08:00:26.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:26 smithi181 podman[70795]: 2024-07-26 08:00:26.042202385 +0000 UTC m=+0.033959796 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:00:26.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:26 smithi181 podman[70795]: 2024-07-26 08:00:26.158230293 +0000 UTC m=+0.149987695 container create 8f92644ba1388747ddaf4adb4c0977fc7c0a862b104895a416aef18c81d55fa0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:00:26.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:26 smithi181 podman[70795]: 2024-07-26 08:00:26.603579426 +0000 UTC m=+0.595336904 container init 8f92644ba1388747ddaf4adb4c0977fc7c0a862b104895a416aef18c81d55fa0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-26T08:00:26.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:26 smithi181 podman[70795]: 2024-07-26 08:00:26.614913155 +0000 UTC m=+0.606670552 container start 8f92644ba1388747ddaf4adb4c0977fc7c0a862b104895a416aef18c81d55fa0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T08:00:26.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:26 smithi181 bash[70795]: 8f92644ba1388747ddaf4adb4c0977fc7c0a862b104895a416aef18c81d55fa0 2024-07-26T08:00:26.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:26 smithi181 systemd[1]: Started Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:00:26.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:26 smithi181 ceph-mds[70813]: -- 172.21.15.181:0/2743823658 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566fc0d7860 con 0x556700279a80 2024-07-26T08:00:26.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:26 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: starting mds.f at 2024-07-26T08:00:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: pgmap v779: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 202 B/s wr, 10 op/s 2024-07-26T08:00:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: daemon mds.f restarted 2024-07-26T08:00:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T08:00:27.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:boot 2024-07-26T08:00:27.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:00:27.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:27.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:00:27.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:27 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: pgmap v779: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 202 B/s wr, 10 op/s 2024-07-26T08:00:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: daemon mds.f restarted 2024-07-26T08:00:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T08:00:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:boot 2024-07-26T08:00:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:00:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:00:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:27 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: pgmap v779: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 202 B/s wr, 10 op/s 2024-07-26T08:00:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: daemon mds.f restarted 2024-07-26T08:00:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T08:00:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:boot 2024-07-26T08:00:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:00:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:00:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:27 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:28 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:resolve 2024-07-26T08:00:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:28 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:28 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:resolve 2024-07-26T08:00:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:28 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:28 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:resolve 2024-07-26T08:00:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:28 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:29 smithi115 ceph-mon[35364]: pgmap v781: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T08:00:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:29 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:00:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:29 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:reconnect 2024-07-26T08:00:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:29 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:29 smithi181 ceph-mon[35869]: pgmap v781: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T08:00:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:29 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:00:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:29 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:reconnect 2024-07-26T08:00:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:29 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:29 smithi097 ceph-mon[32231]: pgmap v781: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T08:00:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:29 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:00:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:29 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:reconnect 2024-07-26T08:00:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:29 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:rejoin 2024-07-26T08:00:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-mon[35364]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: *** Caught signal (Aborted) ** 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: in thread 7fd7af400640 thread_name:ms_dispatch 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd7b563e6f0] 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd7b568b94c] 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 3: raise() 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 4: abort() 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd7b562871b] 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 6: /lib64/libc.so.6(+0x37386) [0x7fd7b5637386] 2024-07-26T08:00:31.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559d7bee56b9] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559d7c0a380c] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559d7bfe32ca] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559d7bfb3753] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559d7bfb4a1b] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559d7bf98752] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559d7bee6a56] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559d7beea637] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559d7beeaf5e] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559d7bece03a] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd7b619b407] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 18: (DispatchQueue::entry()+0x60f) [0x7fd7b619bcf9] 2024-07-26T08:00:31.227 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd7b62785a1] 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 20: (Thread::entry_wrapper()+0x43) [0x7fd7b6025cdd] 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd7b6025cf9] 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd7b5689c02] 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd7b570ec40] 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2024-07-26T08:00:30.830+0000 7fd7af400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: in thread 7fd7af400640 thread_name:ms_dispatch 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd7b563e6f0] 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd7b568b94c] 2024-07-26T08:00:31.228 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 3: raise() 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 4: abort() 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd7b562871b] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 6: /lib64/libc.so.6(+0x37386) [0x7fd7b5637386] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559d7bee56b9] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559d7c0a380c] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559d7bfe32ca] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559d7bfb3753] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559d7bfb4a1b] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559d7bf98752] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559d7bee6a56] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559d7beea637] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559d7beeaf5e] 2024-07-26T08:00:31.229 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559d7bece03a] 2024-07-26T08:00:31.230 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd7b619b407] 2024-07-26T08:00:31.230 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 18: (DispatchQueue::entry()+0x60f) [0x7fd7b619bcf9] 2024-07-26T08:00:31.230 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd7b62785a1] 2024-07-26T08:00:31.230 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 20: (Thread::entry_wrapper()+0x43) [0x7fd7b6025cdd] 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd7b6025cf9] 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd7b5689c02] 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd7b570ec40] 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 0> 2024-07-26T08:00:30.830+0000 7fd7af400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: in thread 7fd7af400640 thread_name:ms_dispatch 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd7b563e6f0] 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd7b568b94c] 2024-07-26T08:00:31.231 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 3: raise() 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 4: abort() 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd7b562871b] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 6: /lib64/libc.so.6(+0x37386) [0x7fd7b5637386] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559d7bee56b9] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559d7c0a380c] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559d7bfe32ca] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559d7bfb3753] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559d7bfb4a1b] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559d7bf98752] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559d7bee6a56] 2024-07-26T08:00:31.232 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559d7beea637] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559d7beeaf5e] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559d7bece03a] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd7b619b407] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 18: (DispatchQueue::entry()+0x60f) [0x7fd7b619bcf9] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd7b62785a1] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 20: (Thread::entry_wrapper()+0x43) [0x7fd7b6025cdd] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd7b6025cf9] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd7b5689c02] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd7b570ec40] 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: -2> 2024-07-26T08:00:30.830+0000 7fd7af400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:31.233 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: in thread 7fd7af400640 thread_name:ms_dispatch 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd7b563e6f0] 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd7b568b94c] 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 3: raise() 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 4: abort() 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd7b562871b] 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 6: /lib64/libc.so.6(+0x37386) [0x7fd7b5637386] 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559d7bee56b9] 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559d7c0a380c] 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559d7bfe32ca] 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559d7bfb3753] 2024-07-26T08:00:31.234 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559d7bfb4a1b] 2024-07-26T08:00:31.235 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559d7bf98752] 2024-07-26T08:00:31.235 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559d7bee6a56] 2024-07-26T08:00:31.235 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559d7beea637] 2024-07-26T08:00:31.235 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559d7beeaf5e] 2024-07-26T08:00:31.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:30 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:rejoin 2024-07-26T08:00:31.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:30 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:31.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:30 smithi181 ceph-mon[35869]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:00:31.236 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559d7bece03a] 2024-07-26T08:00:31.236 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd7b619b407] 2024-07-26T08:00:31.236 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 18: (DispatchQueue::entry()+0x60f) [0x7fd7b619bcf9] 2024-07-26T08:00:31.236 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd7b62785a1] 2024-07-26T08:00:31.236 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 20: (Thread::entry_wrapper()+0x43) [0x7fd7b6025cdd] 2024-07-26T08:00:31.236 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd7b6025cf9] 2024-07-26T08:00:31.237 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd7b5689c02] 2024-07-26T08:00:31.237 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd7b570ec40] 2024-07-26T08:00:31.237 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:31.237 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70087]: 2024-07-26T08:00:31.237 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:30 smithi115 podman[70588]: 2024-07-26 08:00:30.997870502 +0000 UTC m=+0.035698168 container died 46ab31efba43388fbf1f3d6838716efaef908bf10faae1663381338ade4755c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-26T08:00:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:30 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:rejoin 2024-07-26T08:00:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:30 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:30 smithi097 ceph-mon[32231]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:00:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:31 smithi115 ceph-mon[35364]: pgmap v782: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:00:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:31 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:31 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:active 2024-07-26T08:00:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:31 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:00:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:31 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:32.225 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:31 smithi115 podman[70588]: 2024-07-26 08:00:31.839211566 +0000 UTC m=+0.877039260 container remove 46ab31efba43388fbf1f3d6838716efaef908bf10faae1663381338ade4755c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T08:00:32.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:31 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:00:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:31 smithi181 ceph-mon[35869]: pgmap v782: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:00:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:31 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:31 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:active 2024-07-26T08:00:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:31 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:00:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:31 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:31 smithi097 ceph-mon[32231]: pgmap v782: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:00:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:31 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:31 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/3764679716,v1:172.21.15.115:6837/3764679716] up:active 2024-07-26T08:00:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:31 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:00:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:31 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:32.360 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:32.358+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:00:32.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:32.360+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:00:32.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:32.360+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:00:32.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:32.360+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:00:32.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:32.360+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:00:32.724 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:32 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T08:00:32.724 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:32 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.124s CPU time. 2024-07-26T08:00:34.069 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:33 smithi115 ceph-mon[35364]: pgmap v783: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-07-26T08:00:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:33 smithi181 ceph-mon[35869]: pgmap v783: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-07-26T08:00:34.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:33 smithi097 ceph-mon[32231]: pgmap v783: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-07-26T08:00:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:35 smithi115 ceph-mon[35364]: pgmap v784: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:35 smithi181 ceph-mon[35869]: pgmap v784: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:35 smithi097 ceph-mon[32231]: pgmap v784: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:37 smithi115 ceph-mon[35364]: pgmap v785: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:37 smithi181 ceph-mon[35869]: pgmap v785: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:37 smithi097 ceph-mon[32231]: pgmap v785: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T08:00:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:39 smithi181 ceph-mon[35869]: pgmap v786: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 866 B/s wr, 11 op/s 2024-07-26T08:00:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:39 smithi115 ceph-mon[35364]: pgmap v786: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 866 B/s wr, 11 op/s 2024-07-26T08:00:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:39 smithi097 ceph-mon[32231]: pgmap v786: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 866 B/s wr, 11 op/s 2024-07-26T08:00:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:41 smithi115 ceph-mon[35364]: pgmap v787: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 852 B/s wr, 12 op/s 2024-07-26T08:00:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:41 smithi181 ceph-mon[35869]: pgmap v787: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 852 B/s wr, 12 op/s 2024-07-26T08:00:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:41 smithi097 ceph-mon[32231]: pgmap v787: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 852 B/s wr, 12 op/s 2024-07-26T08:00:42.873 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:42 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-07-26T08:00:42.873 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:42 smithi115 systemd[1]: Stopped Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:00:42.873 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:42 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.124s CPU time. 2024-07-26T08:00:42.873 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:42 smithi115 systemd[1]: Starting Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:00:43.211 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:42 smithi115 podman[70691]: 2024-07-26 08:00:42.77652289 +0000 UTC m=+0.033154676 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:00:43.212 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:42 smithi115 podman[70691]: 2024-07-26 08:00:42.924012986 +0000 UTC m=+0.180644793 container create c6545edf59da074182af87e949b5b731061bfc954211a8ebdaf042fc4003d9e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0) 2024-07-26T08:00:43.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:43 smithi115 podman[70691]: 2024-07-26 08:00:43.210680417 +0000 UTC m=+0.467312243 container init c6545edf59da074182af87e949b5b731061bfc954211a8ebdaf042fc4003d9e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-07-26T08:00:43.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:43 smithi115 podman[70691]: 2024-07-26 08:00:43.22205216 +0000 UTC m=+0.478683949 container start c6545edf59da074182af87e949b5b731061bfc954211a8ebdaf042fc4003d9e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:00:43.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:43 smithi115 bash[70691]: c6545edf59da074182af87e949b5b731061bfc954211a8ebdaf042fc4003d9e0 2024-07-26T08:00:43.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:43 smithi115 systemd[1]: Started Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:00:43.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mds[70709]: -- 172.21.15.115:0/2939740067 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f3409b860 con 0x563f3823da80 2024-07-26T08:00:43.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: starting mds.e at 2024-07-26T08:00:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: pgmap v788: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T08:00:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: daemon mds.e restarted 2024-07-26T08:00:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T08:00:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:boot 2024-07-26T08:00:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:00:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:00:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:43 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: pgmap v788: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T08:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: daemon mds.e restarted 2024-07-26T08:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T08:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:boot 2024-07-26T08:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:43 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: pgmap v788: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T08:00:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: daemon mds.e restarted 2024-07-26T08:00:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:00:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T08:00:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:boot 2024-07-26T08:00:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:00:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:00:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:00:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:43 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:00:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:45 smithi115 ceph-mon[35364]: pgmap v790: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 11 op/s 2024-07-26T08:00:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:45 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:resolve 2024-07-26T08:00:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:45 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:45 smithi181 ceph-mon[35869]: pgmap v790: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 11 op/s 2024-07-26T08:00:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:45 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:resolve 2024-07-26T08:00:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:45 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:45 smithi097 ceph-mon[32231]: pgmap v790: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 11 op/s 2024-07-26T08:00:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:45 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:resolve 2024-07-26T08:00:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:45 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:00:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:46 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:00:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:46 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:reconnect 2024-07-26T08:00:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:46 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:46 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:00:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:46 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:reconnect 2024-07-26T08:00:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:46 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:46 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:00:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:46 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:reconnect 2024-07-26T08:00:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:46 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:00:47.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:47.368+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:00:47.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:47.369+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:00:47.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:47.369+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:00:47.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:47.369+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:00:47.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:00:47.369+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:00:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:47 smithi115 ceph-mon[35364]: pgmap v791: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:00:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:47 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:rejoin 2024-07-26T08:00:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:47 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:47 smithi115 ceph-mon[35364]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:00:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-mon[35869]: pgmap v791: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:00:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:rejoin 2024-07-26T08:00:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-mon[35869]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: *** Caught signal (Aborted) ** 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: in thread 7f47e0400640 thread_name:ms_dispatch 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47e663e6f0] 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f47e668b94c] 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 3: raise() 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 4: abort() 2024-07-26T08:00:48.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 5: /lib64/libc.so.6(+0x2871b) [0x7f47e662871b] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 6: /lib64/libc.so.6(+0x37386) [0x7f47e6637386] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5566ea9086b9] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5566eaac680c] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5566eaa062ca] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5566ea9d6753] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5566ea9d7a1b] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5566ea9bb752] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5566ea909a56] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5566ea90d637] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5566ea90df5e] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5566ea8f103a] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f47e719b407] 2024-07-26T08:00:48.235 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 18: (DispatchQueue::entry()+0x60f) [0x7f47e719bcf9] 2024-07-26T08:00:48.236 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f47e72785a1] 2024-07-26T08:00:48.236 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 20: (Thread::entry_wrapper()+0x43) [0x7f47e7025cdd] 2024-07-26T08:00:48.236 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 21: (Thread::_entry_func(void*)+0xd) [0x7f47e7025cf9] 2024-07-26T08:00:48.236 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 22: /lib64/libc.so.6(+0x89c02) [0x7f47e6689c02] 2024-07-26T08:00:48.236 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f47e670ec40] 2024-07-26T08:00:48.236 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2024-07-26T08:00:47.923+0000 7f47e0400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:48.236 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: in thread 7f47e0400640 thread_name:ms_dispatch 2024-07-26T08:00:48.236 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47e663e6f0] 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f47e668b94c] 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 3: raise() 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 4: abort() 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 5: /lib64/libc.so.6(+0x2871b) [0x7f47e662871b] 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 6: /lib64/libc.so.6(+0x37386) [0x7f47e6637386] 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5566ea9086b9] 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5566eaac680c] 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5566eaa062ca] 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5566ea9d6753] 2024-07-26T08:00:48.237 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5566ea9d7a1b] 2024-07-26T08:00:48.238 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5566ea9bb752] 2024-07-26T08:00:48.238 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5566ea909a56] 2024-07-26T08:00:48.238 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5566ea90d637] 2024-07-26T08:00:48.238 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5566ea90df5e] 2024-07-26T08:00:48.238 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5566ea8f103a] 2024-07-26T08:00:48.238 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f47e719b407] 2024-07-26T08:00:48.238 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 18: (DispatchQueue::entry()+0x60f) [0x7f47e719bcf9] 2024-07-26T08:00:48.238 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f47e72785a1] 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 20: (Thread::entry_wrapper()+0x43) [0x7f47e7025cdd] 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 21: (Thread::_entry_func(void*)+0xd) [0x7f47e7025cf9] 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 22: /lib64/libc.so.6(+0x89c02) [0x7f47e6689c02] 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f47e670ec40] 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 0> 2024-07-26T08:00:47.923+0000 7f47e0400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: in thread 7f47e0400640 thread_name:ms_dispatch 2024-07-26T08:00:48.239 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47e663e6f0] 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f47e668b94c] 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 3: raise() 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 4: abort() 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 5: /lib64/libc.so.6(+0x2871b) [0x7f47e662871b] 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 6: /lib64/libc.so.6(+0x37386) [0x7f47e6637386] 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5566ea9086b9] 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5566eaac680c] 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5566eaa062ca] 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5566ea9d6753] 2024-07-26T08:00:48.240 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5566ea9d7a1b] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5566ea9bb752] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5566ea909a56] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5566ea90d637] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5566ea90df5e] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5566ea8f103a] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f47e719b407] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 18: (DispatchQueue::entry()+0x60f) [0x7f47e719bcf9] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f47e72785a1] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 20: (Thread::entry_wrapper()+0x43) [0x7f47e7025cdd] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 21: (Thread::_entry_func(void*)+0xd) [0x7f47e7025cf9] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 22: /lib64/libc.so.6(+0x89c02) [0x7f47e6689c02] 2024-07-26T08:00:48.241 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f47e670ec40] 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 0> 2024-07-26T08:00:47.923+0000 7f47e0400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: in thread 7f47e0400640 thread_name:ms_dispatch 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47e663e6f0] 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f47e668b94c] 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 3: raise() 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 4: abort() 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 5: /lib64/libc.so.6(+0x2871b) [0x7f47e662871b] 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 6: /lib64/libc.so.6(+0x37386) [0x7f47e6637386] 2024-07-26T08:00:48.242 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5566ea9086b9] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5566eaac680c] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5566eaa062ca] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5566ea9d6753] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5566ea9d7a1b] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5566ea9bb752] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5566ea909a56] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5566ea90d637] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5566ea90df5e] 2024-07-26T08:00:48.243 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5566ea8f103a] 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f47e719b407] 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 18: (DispatchQueue::entry()+0x60f) [0x7f47e719bcf9] 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f47e72785a1] 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 20: (Thread::entry_wrapper()+0x43) [0x7f47e7025cdd] 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 21: (Thread::_entry_func(void*)+0xd) [0x7f47e7025cf9] 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 22: /lib64/libc.so.6(+0x89c02) [0x7f47e6689c02] 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f47e670ec40] 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:47 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70809]: 2024-07-26T08:00:48.244 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:48 smithi181 podman[70859]: 2024-07-26 08:00:48.069605785 +0000 UTC m=+0.039026933 container died 8f92644ba1388747ddaf4adb4c0977fc7c0a862b104895a416aef18c81d55fa0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-07-26T08:00:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:47 smithi097 ceph-mon[32231]: pgmap v791: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:00:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:47 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:rejoin 2024-07-26T08:00:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:47 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:00:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:47 smithi097 ceph-mon[32231]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:00:48.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:48 smithi181 podman[70859]: 2024-07-26 08:00:48.697800405 +0000 UTC m=+0.667221597 container remove 8f92644ba1388747ddaf4adb4c0977fc7c0a862b104895a416aef18c81d55fa0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True) 2024-07-26T08:00:48.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:48 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:00:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:48 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:48 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:active 2024-07-26T08:00:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:48 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:00:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:48 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:48 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:48 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:active 2024-07-26T08:00:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:48 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:00:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:48 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:48 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:00:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:48 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/1370597110,v1:172.21.15.181:6835/1370597110] up:active 2024-07-26T08:00:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:48 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:00:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:48 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:00:49.482 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:49 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:00:49.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:49 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 2.120s CPU time. 2024-07-26T08:00:50.041 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:49 smithi181 ceph-mon[35869]: pgmap v792: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:00:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:49 smithi115 ceph-mon[35364]: pgmap v792: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:00:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:49 smithi097 ceph-mon[32231]: pgmap v792: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:00:50.482 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:00:50 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:00:50.040+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:00:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:51 smithi115 ceph-mon[35364]: pgmap v793: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 716 B/s wr, 11 op/s 2024-07-26T08:00:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:51 smithi181 ceph-mon[35869]: pgmap v793: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 716 B/s wr, 11 op/s 2024-07-26T08:00:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:51 smithi097 ceph-mon[32231]: pgmap v793: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 716 B/s wr, 11 op/s 2024-07-26T08:00:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:53 smithi115 ceph-mon[35364]: pgmap v794: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 716 B/s wr, 11 op/s 2024-07-26T08:00:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:53 smithi181 ceph-mon[35869]: pgmap v794: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 716 B/s wr, 11 op/s 2024-07-26T08:00:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:53 smithi097 ceph-mon[32231]: pgmap v794: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 716 B/s wr, 11 op/s 2024-07-26T08:00:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:55 smithi115 ceph-mon[35364]: pgmap v795: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 639 B/s wr, 12 op/s 2024-07-26T08:00:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:55 smithi181 ceph-mon[35869]: pgmap v795: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 639 B/s wr, 12 op/s 2024-07-26T08:00:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:55 smithi097 ceph-mon[32231]: pgmap v795: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 639 B/s wr, 12 op/s 2024-07-26T08:00:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:57 smithi115 ceph-mon[35364]: pgmap v796: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 596 B/s wr, 10 op/s 2024-07-26T08:00:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:57 smithi181 ceph-mon[35869]: pgmap v796: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 596 B/s wr, 10 op/s 2024-07-26T08:00:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:57 smithi097 ceph-mon[32231]: pgmap v796: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 596 B/s wr, 10 op/s 2024-07-26T08:00:59.627 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:59 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-07-26T08:00:59.627 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:59 smithi181 systemd[1]: Stopped Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:00:59.627 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:59 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 2.120s CPU time. 2024-07-26T08:00:59.627 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:59 smithi181 systemd[1]: Starting Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:00:59.960 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:59 smithi181 podman[70965]: 2024-07-26 08:00:59.530134866 +0000 UTC m=+0.031461537 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:00:59.960 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:59 smithi181 podman[70965]: 2024-07-26 08:00:59.664878008 +0000 UTC m=+0.166204723 container create 67d8e32fe8bc6641c526fef75e4ade8f241d33957e43ff0e4061c07bf4f49243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-07-26T08:01:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:00:59 smithi115 ceph-mon[35364]: pgmap v797: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 596 B/s wr, 10 op/s 2024-07-26T08:01:00.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:59 smithi181 podman[70965]: 2024-07-26 08:00:59.959773735 +0000 UTC m=+0.461100457 container init 67d8e32fe8bc6641c526fef75e4ade8f241d33957e43ff0e4061c07bf4f49243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T08:01:00.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:00:59 smithi181 podman[70965]: 2024-07-26 08:00:59.971498121 +0000 UTC m=+0.472824795 container start 67d8e32fe8bc6641c526fef75e4ade8f241d33957e43ff0e4061c07bf4f49243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-07-26T08:01:00.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:00 smithi181 bash[70965]: 67d8e32fe8bc6641c526fef75e4ade8f241d33957e43ff0e4061c07bf4f49243 2024-07-26T08:01:00.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:00 smithi181 systemd[1]: Started Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:01:00.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mds[70984]: -- 172.21.15.181:0/3671680792 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f16511860 con 0x561f1a6b3600 2024-07-26T08:01:00.234 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: starting mds.f at 2024-07-26T08:01:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:00:59 smithi181 ceph-mon[35869]: pgmap v797: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 596 B/s wr, 10 op/s 2024-07-26T08:01:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:00:59 smithi097 ceph-mon[32231]: pgmap v797: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 596 B/s wr, 10 op/s 2024-07-26T08:01:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:00 smithi115 ceph-mon[35364]: daemon mds.f restarted 2024-07-26T08:01:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:00 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:00 smithi115 ceph-mon[35364]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T08:01:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:00 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:boot 2024-07-26T08:01:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:00 smithi115 ceph-mon[35364]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:01:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:00 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:01:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:00 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mon[35869]: daemon mds.f restarted 2024-07-26T08:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mon[35869]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T08:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:boot 2024-07-26T08:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mon[35869]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:00 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:00 smithi097 ceph-mon[32231]: daemon mds.f restarted 2024-07-26T08:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:00 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:00 smithi097 ceph-mon[32231]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T08:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:00 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:boot 2024-07-26T08:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:00 smithi097 ceph-mon[32231]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:00 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:00 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:02.373+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:01:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:02.374+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:01:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:02.374+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:01:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:02.374+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:01:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:02.374+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:01:02.469 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:02 smithi181 ceph-mon[35869]: pgmap v799: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:01:02.469 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:02 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:resolve 2024-07-26T08:01:02.469 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:02 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:02 smithi115 ceph-mon[35364]: pgmap v799: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:01:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:02 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:resolve 2024-07-26T08:01:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:02 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:02 smithi097 ceph-mon[32231]: pgmap v799: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:01:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:02 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:resolve 2024-07-26T08:01:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:02 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:03 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:reconnect 2024-07-26T08:01:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:03 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:03 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:01:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:03 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:reconnect 2024-07-26T08:01:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:03 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:03 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:01:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:03 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:reconnect 2024-07-26T08:01:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:03 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:03 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:01:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-mon[35364]: pgmap v800: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:01:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:rejoin 2024-07-26T08:01:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-mon[35364]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:01:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:active 2024-07-26T08:01:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:04.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: *** Caught signal (Aborted) ** 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: in thread 7f4e1bc00640 thread_name:ms_dispatch 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4e21e3e6f0] 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4e21e8b94c] 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 3: raise() 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 4: abort() 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4e21e2871b] 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 6: /lib64/libc.so.6(+0x37386) [0x7f4e21e37386] 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563effe3c6b9] 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563effffa80c] 2024-07-26T08:01:04.476 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563efff3a2ca] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563efff0a753] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563efff0ba1b] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563effeef752] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563effe3da56] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563effe41637] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563effe41f5e] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563effe2503a] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4e2299b407] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 18: (DispatchQueue::entry()+0x60f) [0x7f4e2299bcf9] 2024-07-26T08:01:04.477 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4e22a785a1] 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 20: (Thread::entry_wrapper()+0x43) [0x7f4e22825cdd] 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4e22825cf9] 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4e21e89c02] 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4e21f0ec40] 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2024-07-26T08:01:04.076+0000 7f4e1bc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: in thread 7f4e1bc00640 thread_name:ms_dispatch 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4e21e3e6f0] 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4e21e8b94c] 2024-07-26T08:01:04.478 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 3: raise() 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 4: abort() 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4e21e2871b] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 6: /lib64/libc.so.6(+0x37386) [0x7f4e21e37386] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563effe3c6b9] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563effffa80c] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563efff3a2ca] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563efff0a753] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563efff0ba1b] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563effeef752] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563effe3da56] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563effe41637] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563effe41f5e] 2024-07-26T08:01:04.479 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563effe2503a] 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4e2299b407] 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 18: (DispatchQueue::entry()+0x60f) [0x7f4e2299bcf9] 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4e22a785a1] 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 20: (Thread::entry_wrapper()+0x43) [0x7f4e22825cdd] 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4e22825cf9] 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4e21e89c02] 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4e21f0ec40] 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:04.480 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 0> 2024-07-26T08:01:04.076+0000 7f4e1bc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: in thread 7f4e1bc00640 thread_name:ms_dispatch 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4e21e3e6f0] 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4e21e8b94c] 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 3: raise() 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 4: abort() 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4e21e2871b] 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 6: /lib64/libc.so.6(+0x37386) [0x7f4e21e37386] 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563effe3c6b9] 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563effffa80c] 2024-07-26T08:01:04.481 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563efff3a2ca] 2024-07-26T08:01:04.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563efff0a753] 2024-07-26T08:01:04.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563efff0ba1b] 2024-07-26T08:01:04.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563effeef752] 2024-07-26T08:01:04.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563effe3da56] 2024-07-26T08:01:04.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563effe41637] 2024-07-26T08:01:04.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563effe41f5e] 2024-07-26T08:01:04.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563effe2503a] 2024-07-26T08:01:04.482 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4e2299b407] 2024-07-26T08:01:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:04 smithi181 ceph-mon[35869]: pgmap v800: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:01:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:04 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:rejoin 2024-07-26T08:01:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:04 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:04 smithi181 ceph-mon[35869]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:01:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:04 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:04 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:active 2024-07-26T08:01:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:04 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:04.483 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 18: (DispatchQueue::entry()+0x60f) [0x7f4e2299bcf9] 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4e22a785a1] 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 20: (Thread::entry_wrapper()+0x43) [0x7f4e22825cdd] 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4e22825cf9] 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4e21e89c02] 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4e21f0ec40] 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 0> 2024-07-26T08:01:04.076+0000 7f4e1bc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: in thread 7f4e1bc00640 thread_name:ms_dispatch 2024-07-26T08:01:04.484 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4e21e3e6f0] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4e21e8b94c] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 3: raise() 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 4: abort() 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4e21e2871b] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 6: /lib64/libc.so.6(+0x37386) [0x7f4e21e37386] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563effe3c6b9] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563effffa80c] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563efff3a2ca] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563efff0a753] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563efff0ba1b] 2024-07-26T08:01:04.485 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563effeef752] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563effe3da56] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563effe41637] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563effe41f5e] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563effe2503a] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4e2299b407] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 18: (DispatchQueue::entry()+0x60f) [0x7f4e2299bcf9] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4e22a785a1] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 20: (Thread::entry_wrapper()+0x43) [0x7f4e22825cdd] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4e22825cf9] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4e21e89c02] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4e21f0ec40] 2024-07-26T08:01:04.486 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:04.487 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70705]: 2024-07-26T08:01:04.487 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 podman[70759]: 2024-07-26 08:01:04.252689635 +0000 UTC m=+0.039330288 container died c6545edf59da074182af87e949b5b731061bfc954211a8ebdaf042fc4003d9e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-07-26T08:01:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:04 smithi097 ceph-mon[32231]: pgmap v800: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:01:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:04 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:rejoin 2024-07-26T08:01:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:04 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:04 smithi097 ceph-mon[32231]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:01:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:04 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:04 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/2637899339,v1:172.21.15.115:6837/2637899339] up:active 2024-07-26T08:01:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:04 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:05.224 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 podman[70759]: 2024-07-26 08:01:04.966119788 +0000 UTC m=+0.752760482 container remove c6545edf59da074182af87e949b5b731061bfc954211a8ebdaf042fc4003d9e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:01:05.224 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:04 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:01:05.482 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:01:05 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:01:05.041+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:01:05.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:05 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T08:01:05.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:05 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.228s CPU time. 2024-07-26T08:01:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:06 smithi115 ceph-mon[35364]: pgmap v801: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:01:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:06 smithi181 ceph-mon[35869]: pgmap v801: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:01:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:06 smithi097 ceph-mon[32231]: pgmap v801: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:01:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:06.896+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034c0 con 0x7ff3f01061a0 2024-07-26T08:01:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:06.897+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d80b7070 con 0x7ff3f01061a0 2024-07-26T08:01:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:06.902+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc0034c0 con 0x55d25070b2f0 2024-07-26T08:01:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:06.903+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc0034c0 con 0x55d25070b2f0 2024-07-26T08:01:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:08 smithi115 ceph-mon[35364]: pgmap v802: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:01:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:08 smithi181 ceph-mon[35869]: pgmap v802: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:01:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:08 smithi097 ceph-mon[32231]: pgmap v802: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:01:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:10 smithi115 ceph-mon[35364]: pgmap v803: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:01:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:10 smithi181 ceph-mon[35869]: pgmap v803: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:01:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:10 smithi097 ceph-mon[32231]: pgmap v803: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:01:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:12 smithi115 ceph-mon[35364]: pgmap v804: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 195 B/s wr, 11 op/s 2024-07-26T08:01:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:12 smithi181 ceph-mon[35869]: pgmap v804: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 195 B/s wr, 11 op/s 2024-07-26T08:01:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:12 smithi097 ceph-mon[32231]: pgmap v804: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 195 B/s wr, 11 op/s 2024-07-26T08:01:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:14 smithi115 ceph-mon[35364]: pgmap v805: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:01:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:14 smithi181 ceph-mon[35869]: pgmap v805: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:01:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:14 smithi097 ceph-mon[32231]: pgmap v805: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:01:16.130 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:15 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-07-26T08:01:16.130 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:15 smithi115 systemd[1]: Stopped Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:01:16.130 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:15 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.228s CPU time. 2024-07-26T08:01:16.130 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:15 smithi115 systemd[1]: Starting Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:01:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:16 smithi115 ceph-mon[35364]: pgmap v806: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:16 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:16.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:16 smithi115 podman[70854]: 2024-07-26 08:01:16.033301811 +0000 UTC m=+0.032636819 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:01:16.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:16 smithi115 podman[70854]: 2024-07-26 08:01:16.334027218 +0000 UTC m=+0.333362242 container create 5f95414cb76f2b3c31c248e285d50118af89d93a5c3007e7a33303399fe348c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:01:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:16 smithi181 ceph-mon[35869]: pgmap v806: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:16 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:16 smithi097 ceph-mon[32231]: pgmap v806: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:16 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:16.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:16 smithi115 podman[70854]: 2024-07-26 08:01:16.654350524 +0000 UTC m=+0.653685622 container init 5f95414cb76f2b3c31c248e285d50118af89d93a5c3007e7a33303399fe348c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-07-26T08:01:16.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:16 smithi115 podman[70854]: 2024-07-26 08:01:16.663998285 +0000 UTC m=+0.663333290 container start 5f95414cb76f2b3c31c248e285d50118af89d93a5c3007e7a33303399fe348c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T08:01:16.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:16 smithi115 ceph-mds[70874]: -- 172.21.15.115:0/3979219586 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558894ee5860 con 0x558899087180 2024-07-26T08:01:16.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:16 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: starting mds.e at 2024-07-26T08:01:16.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:16 smithi115 bash[70854]: 5f95414cb76f2b3c31c248e285d50118af89d93a5c3007e7a33303399fe348c2 2024-07-26T08:01:16.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:16 smithi115 systemd[1]: Started Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:01:17.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:17.382+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:01:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:17.383+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:01:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:17.383+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:01:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:17.383+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:01:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:17.383+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:01:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:17 smithi097 ceph-mon[32231]: daemon mds.e restarted 2024-07-26T08:01:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:17 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:17 smithi097 ceph-mon[32231]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T08:01:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:17 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:boot 2024-07-26T08:01:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:17 smithi097 ceph-mon[32231]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:01:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:17 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:01:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:17 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:17 smithi115 ceph-mon[35364]: daemon mds.e restarted 2024-07-26T08:01:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:17 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:17 smithi115 ceph-mon[35364]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T08:01:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:17 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:boot 2024-07-26T08:01:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:17 smithi115 ceph-mon[35364]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:01:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:17 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:01:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:17 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:17 smithi181 ceph-mon[35869]: daemon mds.e restarted 2024-07-26T08:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:17 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:17 smithi181 ceph-mon[35869]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T08:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:17 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:boot 2024-07-26T08:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:17 smithi181 ceph-mon[35869]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:17 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:17 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:18 smithi097 ceph-mon[32231]: pgmap v807: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:18 smithi115 ceph-mon[35364]: pgmap v807: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:18 smithi181 ceph-mon[35869]: pgmap v807: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:19 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:resolve 2024-07-26T08:01:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:19 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:01:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:19 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:resolve 2024-07-26T08:01:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:19 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:01:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:19 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:resolve 2024-07-26T08:01:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:19 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:01:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:20 smithi115 ceph-mon[35364]: pgmap v809: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:01:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:20 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:reconnect 2024-07-26T08:01:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:20 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:20 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:01:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:01:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:01:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:20 smithi115 ceph-mon[35364]: pgmap v810: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-26T08:01:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:01:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:20 smithi181 ceph-mon[35869]: pgmap v809: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:01:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:20 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:reconnect 2024-07-26T08:01:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:20 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:20 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:01:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:01:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:01:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:20 smithi181 ceph-mon[35869]: pgmap v810: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-26T08:01:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:01:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:20 smithi097 ceph-mon[32231]: pgmap v809: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:01:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:20 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:reconnect 2024-07-26T08:01:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:20 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:20 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:01:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:01:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:01:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:20 smithi097 ceph-mon[32231]: pgmap v810: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-26T08:01:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:01:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:21 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:rejoin 2024-07-26T08:01:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:21 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:21 smithi115 ceph-mon[35364]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:01:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:rejoin 2024-07-26T08:01:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-mon[35869]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:01:21.734 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:01:21.734 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: *** Caught signal (Aborted) ** 2024-07-26T08:01:21.734 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: in thread 7f9b31800640 thread_name:ms_dispatch 2024-07-26T08:01:21.734 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:21.734 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b37a3e6f0] 2024-07-26T08:01:21.734 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b37a8b94c] 2024-07-26T08:01:21.734 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 3: raise() 2024-07-26T08:01:21.734 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 4: abort() 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b37a2871b] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b37a37386] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561edb6b76b9] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561edb87580c] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561edb7b52ca] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561edb785753] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561edb786a1b] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561edb76a752] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561edb6b8a56] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561edb6bc637] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561edb6bcf5e] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561edb6a003a] 2024-07-26T08:01:21.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b3859b407] 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b3859bcf9] 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b386785a1] 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b38425cdd] 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b38425cf9] 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b37a89c02] 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b37b0ec40] 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2024-07-26T08:01:21.296+0000 7f9b31800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: in thread 7f9b31800640 thread_name:ms_dispatch 2024-07-26T08:01:21.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b37a3e6f0] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b37a8b94c] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 3: raise() 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 4: abort() 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b37a2871b] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b37a37386] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561edb6b76b9] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561edb87580c] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561edb7b52ca] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561edb785753] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561edb786a1b] 2024-07-26T08:01:21.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561edb76a752] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561edb6b8a56] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561edb6bc637] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561edb6bcf5e] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561edb6a003a] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b3859b407] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b3859bcf9] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b386785a1] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b38425cdd] 2024-07-26T08:01:21.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b38425cf9] 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b37a89c02] 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b37b0ec40] 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: -1> 2024-07-26T08:01:21.296+0000 7f9b31800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: in thread 7f9b31800640 thread_name:ms_dispatch 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b37a3e6f0] 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b37a8b94c] 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 3: raise() 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 4: abort() 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b37a2871b] 2024-07-26T08:01:21.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b37a37386] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561edb6b76b9] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561edb87580c] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561edb7b52ca] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561edb785753] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561edb786a1b] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561edb76a752] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561edb6b8a56] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561edb6bc637] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561edb6bcf5e] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561edb6a003a] 2024-07-26T08:01:21.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b3859b407] 2024-07-26T08:01:21.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b3859bcf9] 2024-07-26T08:01:21.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b386785a1] 2024-07-26T08:01:21.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b38425cdd] 2024-07-26T08:01:21.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b38425cf9] 2024-07-26T08:01:21.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b37a89c02] 2024-07-26T08:01:21.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b37b0ec40] 2024-07-26T08:01:21.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: -3> 2024-07-26T08:01:21.296+0000 7f9b31800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: in thread 7f9b31800640 thread_name:ms_dispatch 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b37a3e6f0] 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b37a8b94c] 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 3: raise() 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 4: abort() 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b37a2871b] 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b37a37386] 2024-07-26T08:01:21.742 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561edb6b76b9] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561edb87580c] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561edb7b52ca] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561edb785753] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561edb786a1b] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561edb76a752] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561edb6b8a56] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561edb6bc637] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561edb6bcf5e] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561edb6a003a] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b3859b407] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b3859bcf9] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b386785a1] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b38425cdd] 2024-07-26T08:01:21.743 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b38425cf9] 2024-07-26T08:01:21.744 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b37a89c02] 2024-07-26T08:01:21.744 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b37b0ec40] 2024-07-26T08:01:21.744 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:21.744 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[70980]: 2024-07-26T08:01:21.744 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:21 smithi181 podman[71163]: 2024-07-26 08:01:21.425575266 +0000 UTC m=+0.035380907 container died 67d8e32fe8bc6641c526fef75e4ade8f241d33957e43ff0e4061c07bf4f49243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T08:01:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:21 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:rejoin 2024-07-26T08:01:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:21 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:21 smithi097 ceph-mon[32231]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:01:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:22 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:22 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:active 2024-07-26T08:01:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:22 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:01:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:22 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:22 smithi181 ceph-mon[35869]: pgmap v811: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 108 B/s wr, 9 op/s 2024-07-26T08:01:22.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:22 smithi181 podman[71163]: 2024-07-26 08:01:22.130544955 +0000 UTC m=+0.740350627 container remove 67d8e32fe8bc6641c526fef75e4ade8f241d33957e43ff0e4061c07bf4f49243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-07-26T08:01:22.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:22 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:01:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:22 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:22 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:active 2024-07-26T08:01:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:22 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:01:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:22 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:22 smithi115 ceph-mon[35364]: pgmap v811: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 108 B/s wr, 9 op/s 2024-07-26T08:01:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:22 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:22 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/3847500855,v1:172.21.15.181:6835/3847500855] up:active 2024-07-26T08:01:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:22 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:01:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:22 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:22 smithi097 ceph-mon[32231]: pgmap v811: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 108 B/s wr, 9 op/s 2024-07-26T08:01:22.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:22 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:01:22.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:22 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 2.102s CPU time. 2024-07-26T08:01:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:25 smithi115 ceph-mon[35364]: pgmap v812: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 216 B/s wr, 10 op/s 2024-07-26T08:01:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:25 smithi181 ceph-mon[35869]: pgmap v812: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 216 B/s wr, 10 op/s 2024-07-26T08:01:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:25 smithi097 ceph-mon[32231]: pgmap v812: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 216 B/s wr, 10 op/s 2024-07-26T08:01:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:27 smithi115 ceph-mon[35364]: pgmap v813: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 216 B/s wr, 12 op/s 2024-07-26T08:01:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:27 smithi181 ceph-mon[35869]: pgmap v813: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 216 B/s wr, 12 op/s 2024-07-26T08:01:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:27 smithi097 ceph-mon[32231]: pgmap v813: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 216 B/s wr, 12 op/s 2024-07-26T08:01:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:29 smithi115 ceph-mon[35364]: pgmap v814: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 182 B/s wr, 11 op/s 2024-07-26T08:01:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:29 smithi181 ceph-mon[35869]: pgmap v814: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 182 B/s wr, 11 op/s 2024-07-26T08:01:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:29 smithi097 ceph-mon[32231]: pgmap v814: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 182 B/s wr, 11 op/s 2024-07-26T08:01:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:31 smithi115 ceph-mon[35364]: pgmap v815: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:01:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:31 smithi181 ceph-mon[35869]: pgmap v815: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:01:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:31 smithi097 ceph-mon[32231]: pgmap v815: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:01:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:32.387+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:01:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:32.388+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:01:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:32.388+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:01:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:32.388+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:01:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:32.388+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:01:33.113 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:32 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-07-26T08:01:33.113 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:32 smithi181 systemd[1]: Stopped Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:01:33.113 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:32 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 2.102s CPU time. 2024-07-26T08:01:33.113 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:32 smithi181 systemd[1]: Starting Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:01:33.427 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:33 smithi181 ceph-mon[35869]: pgmap v816: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:33.427 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:33 smithi181 podman[71254]: 2024-07-26 08:01:33.016519812 +0000 UTC m=+0.030611551 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:01:33.427 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:33 smithi181 podman[71254]: 2024-07-26 08:01:33.181251262 +0000 UTC m=+0.195343037 container create bba3062b52e314f09055981429f70f39b64403ad0af35d04796fe9466f6cf1b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.build-date=20240724) 2024-07-26T08:01:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:33 smithi115 ceph-mon[35364]: pgmap v816: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:33 smithi097 ceph-mon[32231]: pgmap v816: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:33.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:33 smithi181 podman[71254]: 2024-07-26 08:01:33.42656437 +0000 UTC m=+0.440656149 container init bba3062b52e314f09055981429f70f39b64403ad0af35d04796fe9466f6cf1b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-07-26T08:01:33.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:33 smithi181 podman[71254]: 2024-07-26 08:01:33.4365978 +0000 UTC m=+0.450689549 container start bba3062b52e314f09055981429f70f39b64403ad0af35d04796fe9466f6cf1b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:01:33.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:33 smithi181 bash[71254]: bba3062b52e314f09055981429f70f39b64403ad0af35d04796fe9466f6cf1b6 2024-07-26T08:01:33.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:33 smithi181 ceph-mds[71273]: -- 172.21.15.181:0/2153501205 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562f38833860 con 0x562f3c9d5600 2024-07-26T08:01:33.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:33 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: starting mds.f at 2024-07-26T08:01:33.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:33 smithi181 systemd[1]: Started Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:01:34.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:34 smithi097 ceph-mon[32231]: daemon mds.f restarted 2024-07-26T08:01:34.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:34 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:34.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:34 smithi097 ceph-mon[32231]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T08:01:34.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:34 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:boot 2024-07-26T08:01:34.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:34 smithi097 ceph-mon[32231]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:01:34.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:34 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:34.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:01:34.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:34 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:34 smithi115 ceph-mon[35364]: daemon mds.f restarted 2024-07-26T08:01:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:34 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:34 smithi115 ceph-mon[35364]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T08:01:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:34 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:boot 2024-07-26T08:01:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:34 smithi115 ceph-mon[35364]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:01:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:34 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:01:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:34 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:34 smithi181 ceph-mon[35869]: daemon mds.f restarted 2024-07-26T08:01:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:34 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:34 smithi181 ceph-mon[35869]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T08:01:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:34 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:boot 2024-07-26T08:01:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:34 smithi181 ceph-mon[35869]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:01:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:34 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:01:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:34 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:35 smithi115 ceph-mon[35364]: pgmap v818: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 10 op/s 2024-07-26T08:01:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:35 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:resolve 2024-07-26T08:01:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:35 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:35 smithi181 ceph-mon[35869]: pgmap v818: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 10 op/s 2024-07-26T08:01:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:35 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:resolve 2024-07-26T08:01:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:35 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:35 smithi097 ceph-mon[32231]: pgmap v818: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 10 op/s 2024-07-26T08:01:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:35 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:resolve 2024-07-26T08:01:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:35 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:36 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:reconnect 2024-07-26T08:01:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:36 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:36 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699987 2024-07-26T08:01:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:36 smithi115 ceph-mon[35364]: pgmap v819: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:01:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:36 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:reconnect 2024-07-26T08:01:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:36 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:36 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699987 2024-07-26T08:01:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:36 smithi181 ceph-mon[35869]: pgmap v819: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:01:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:36 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:reconnect 2024-07-26T08:01:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:36 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:36 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699987 2024-07-26T08:01:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:36 smithi097 ceph-mon[32231]: pgmap v819: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:01:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:rejoin 2024-07-26T08:01:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-mon[35364]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:01:37.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:01:37.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: *** Caught signal (Aborted) ** 2024-07-26T08:01:37.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: in thread 7fd9bd200640 thread_name:ms_dispatch 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd9c343e6f0] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd9c348b94c] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 3: raise() 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 4: abort() 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd9c342871b] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 6: /lib64/libc.so.6(+0x37386) [0x7fd9c3437386] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5588594f86b9] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5588596b680c] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5588595f62ca] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5588595c6753] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5588595c7a1b] 2024-07-26T08:01:37.976 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5588595ab752] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5588594f9a56] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5588594fd637] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5588594fdf5e] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5588594e103a] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd9c3f9b407] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 18: (DispatchQueue::entry()+0x60f) [0x7fd9c3f9bcf9] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd9c40785a1] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 20: (Thread::entry_wrapper()+0x43) [0x7fd9c3e25cdd] 2024-07-26T08:01:37.977 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd9c3e25cf9] 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd9c3489c02] 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd9c350ec40] 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2024-07-26T08:01:37.651+0000 7fd9bd200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: in thread 7fd9bd200640 thread_name:ms_dispatch 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd9c343e6f0] 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd9c348b94c] 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 3: raise() 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 4: abort() 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd9c342871b] 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 6: /lib64/libc.so.6(+0x37386) [0x7fd9c3437386] 2024-07-26T08:01:37.978 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5588594f86b9] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5588596b680c] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5588595f62ca] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5588595c6753] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5588595c7a1b] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5588595ab752] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5588594f9a56] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5588594fd637] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5588594fdf5e] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5588594e103a] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd9c3f9b407] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 18: (DispatchQueue::entry()+0x60f) [0x7fd9c3f9bcf9] 2024-07-26T08:01:37.979 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd9c40785a1] 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 20: (Thread::entry_wrapper()+0x43) [0x7fd9c3e25cdd] 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd9c3e25cf9] 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd9c3489c02] 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd9c350ec40] 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 0> 2024-07-26T08:01:37.651+0000 7fd9bd200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: in thread 7fd9bd200640 thread_name:ms_dispatch 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2024-07-26T08:01:37.980 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd9c343e6f0] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd9c348b94c] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 3: raise() 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 4: abort() 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd9c342871b] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 6: /lib64/libc.so.6(+0x37386) [0x7fd9c3437386] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5588594f86b9] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5588596b680c] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5588595f62ca] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5588595c6753] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5588595c7a1b] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5588595ab752] 2024-07-26T08:01:37.981 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5588594f9a56] 2024-07-26T08:01:37.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5588594fd637] 2024-07-26T08:01:37.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5588594fdf5e] 2024-07-26T08:01:37.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5588594e103a] 2024-07-26T08:01:37.982 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd9c3f9b407] 2024-07-26T08:01:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:37 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:rejoin 2024-07-26T08:01:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:37 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:37 smithi181 ceph-mon[35869]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:01:37.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 18: (DispatchQueue::entry()+0x60f) [0x7fd9c3f9bcf9] 2024-07-26T08:01:37.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd9c40785a1] 2024-07-26T08:01:37.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 20: (Thread::entry_wrapper()+0x43) [0x7fd9c3e25cdd] 2024-07-26T08:01:37.983 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd9c3e25cf9] 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd9c3489c02] 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd9c350ec40] 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: -2> 2024-07-26T08:01:37.651+0000 7fd9bd200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: in thread 7fd9bd200640 thread_name:ms_dispatch 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd9c343e6f0] 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd9c348b94c] 2024-07-26T08:01:37.984 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 3: raise() 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 4: abort() 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd9c342871b] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 6: /lib64/libc.so.6(+0x37386) [0x7fd9c3437386] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5588594f86b9] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5588596b680c] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5588595f62ca] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5588595c6753] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5588595c7a1b] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5588595ab752] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5588594f9a56] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5588594fd637] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5588594fdf5e] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5588594e103a] 2024-07-26T08:01:37.985 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd9c3f9b407] 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 18: (DispatchQueue::entry()+0x60f) [0x7fd9c3f9bcf9] 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd9c40785a1] 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 20: (Thread::entry_wrapper()+0x43) [0x7fd9c3e25cdd] 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd9c3e25cf9] 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd9c3489c02] 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd9c350ec40] 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[70870]: 2024-07-26T08:01:37.986 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:37 smithi115 podman[71038]: 2024-07-26 08:01:37.778305679 +0000 UTC m=+0.037139409 container died 5f95414cb76f2b3c31c248e285d50118af89d93a5c3007e7a33303399fe348c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724) 2024-07-26T08:01:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:37 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:rejoin 2024-07-26T08:01:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:37 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:37 smithi097 ceph-mon[32231]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:01:38.918 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:38 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:38.918 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:38 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:active 2024-07-26T08:01:38.918 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:38 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:01:38.918 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:38 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:38.919 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:38 smithi181 ceph-mon[35869]: pgmap v820: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:38.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:38 smithi115 podman[71038]: 2024-07-26 08:01:38.548069045 +0000 UTC m=+0.806902811 container remove 5f95414cb76f2b3c31c248e285d50118af89d93a5c3007e7a33303399fe348c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T08:01:38.975 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:38 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:01:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:38 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:38 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:active 2024-07-26T08:01:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:38 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:01:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:38 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:38 smithi115 ceph-mon[35364]: pgmap v820: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:38 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:38 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/1628966781,v1:172.21.15.115:6837/1628966781] up:active 2024-07-26T08:01:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:38 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:01:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:38 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:38 smithi097 ceph-mon[32231]: pgmap v820: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:39.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:39 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T08:01:39.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:39 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.126s CPU time. 2024-07-26T08:01:40.482 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:01:40 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:01:40.046+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:01:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:41 smithi115 ceph-mon[35364]: pgmap v821: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:41 smithi181 ceph-mon[35869]: pgmap v821: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:41 smithi097 ceph-mon[32231]: pgmap v821: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:43 smithi115 ceph-mon[35364]: pgmap v822: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:01:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:43 smithi181 ceph-mon[35869]: pgmap v822: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:01:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:43 smithi097 ceph-mon[32231]: pgmap v822: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:01:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:45 smithi115 ceph-mon[35364]: pgmap v823: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 195 B/s wr, 12 op/s 2024-07-26T08:01:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:45 smithi181 ceph-mon[35869]: pgmap v823: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 195 B/s wr, 12 op/s 2024-07-26T08:01:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:45 smithi097 ceph-mon[32231]: pgmap v823: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 195 B/s wr, 12 op/s 2024-07-26T08:01:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:47.397+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:01:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:47.398+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:01:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:47.398+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:01:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:47.399+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:01:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:01:47.399+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:01:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:47 smithi115 ceph-mon[35364]: pgmap v824: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:01:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:47 smithi181 ceph-mon[35869]: pgmap v824: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:01:47.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:47 smithi097 ceph-mon[32231]: pgmap v824: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:01:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:49 smithi115 ceph-mon[35364]: pgmap v825: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:49.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 5. 2024-07-26T08:01:49.474 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 systemd[1]: Stopped Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:01:49.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.126s CPU time. 2024-07-26T08:01:49.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 systemd[1]: Starting Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:01:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:49 smithi181 ceph-mon[35869]: pgmap v825: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:49 smithi097 ceph-mon[32231]: pgmap v825: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:01:49.903 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 podman[71142]: 2024-07-26 08:01:49.51212037 +0000 UTC m=+0.032022283 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:01:49.903 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 podman[71142]: 2024-07-26 08:01:49.631490279 +0000 UTC m=+0.151392228 container create e82bde1afb1b52971015ed0565d25481b792a4ea17995293dc9971ba8370d383 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T08:01:49.903 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 podman[71142]: 2024-07-26 08:01:49.901937831 +0000 UTC m=+0.421839830 container init e82bde1afb1b52971015ed0565d25481b792a4ea17995293dc9971ba8370d383 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-07-26T08:01:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mon[35364]: daemon mds.e restarted 2024-07-26T08:01:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mon[35364]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T08:01:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:boot 2024-07-26T08:01:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mon[35364]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:01:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:01:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:50.225 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 podman[71142]: 2024-07-26 08:01:49.911893346 +0000 UTC m=+0.431795301 container start e82bde1afb1b52971015ed0565d25481b792a4ea17995293dc9971ba8370d383 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:01:50.225 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:49 smithi115 bash[71142]: e82bde1afb1b52971015ed0565d25481b792a4ea17995293dc9971ba8370d383 2024-07-26T08:01:50.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:50 smithi115 systemd[1]: Started Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:01:50.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-mds[71160]: -- 172.21.15.115:0/2250121031 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f739d23860 con 0x55f73dec5600 2024-07-26T08:01:50.226 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:01:50 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: starting mds.e at 2024-07-26T08:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:50 smithi181 ceph-mon[35869]: daemon mds.e restarted 2024-07-26T08:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:50 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:50 smithi181 ceph-mon[35869]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T08:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:50 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:boot 2024-07-26T08:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:50 smithi181 ceph-mon[35869]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:50 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:50 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:50 smithi097 ceph-mon[32231]: daemon mds.e restarted 2024-07-26T08:01:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:50 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:01:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:50 smithi097 ceph-mon[32231]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T08:01:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:50 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:boot 2024-07-26T08:01:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:50 smithi097 ceph-mon[32231]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-07-26T08:01:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:50 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:01:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:01:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:50 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:01:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:51 smithi115 ceph-mon[35364]: pgmap v827: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:51 smithi181 ceph-mon[35869]: pgmap v827: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:51 smithi097 ceph-mon[32231]: pgmap v827: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:01:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:52 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:resolve 2024-07-26T08:01:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:52 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:52 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:resolve 2024-07-26T08:01:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:52 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:52 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:resolve 2024-07-26T08:01:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:52 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:01:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:53 smithi115 ceph-mon[35364]: pgmap v828: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:01:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:53 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699986 2024-07-26T08:01:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:53 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:reconnect 2024-07-26T08:01:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:53 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:53 smithi181 ceph-mon[35869]: pgmap v828: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:01:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:53 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699986 2024-07-26T08:01:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:53 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:reconnect 2024-07-26T08:01:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:53 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:53 smithi097 ceph-mon[32231]: pgmap v828: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:01:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:53 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699986 2024-07-26T08:01:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:53 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:reconnect 2024-07-26T08:01:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:53 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:01:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:54 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:rejoin 2024-07-26T08:01:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:54 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:54 smithi115 ceph-mon[35364]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:01:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:54 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:54 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:active 2024-07-26T08:01:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:54 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:rejoin 2024-07-26T08:01:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-mon[35869]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:01:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:active 2024-07-26T08:01:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:54.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:01:54.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: *** Caught signal (Aborted) ** 2024-07-26T08:01:54.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: in thread 7f0027000640 thread_name:ms_dispatch 2024-07-26T08:01:54.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:54.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f002d23e6f0] 2024-07-26T08:01:54.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f002d28b94c] 2024-07-26T08:01:54.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 3: raise() 2024-07-26T08:01:54.484 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 4: abort() 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 5: /lib64/libc.so.6(+0x2871b) [0x7f002d22871b] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 6: /lib64/libc.so.6(+0x37386) [0x7f002d237386] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562efba416b9] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562efbbff80c] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x562efbb3f2ca] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x562efbb0f753] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562efbb10a1b] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562efbaf4752] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562efba42a56] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562efba46637] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562efba46f5e] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562efba2a03a] 2024-07-26T08:01:54.485 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f002dd9b407] 2024-07-26T08:01:54.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 18: (DispatchQueue::entry()+0x60f) [0x7f002dd9bcf9] 2024-07-26T08:01:54.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f002de785a1] 2024-07-26T08:01:54.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 20: (Thread::entry_wrapper()+0x43) [0x7f002dc25cdd] 2024-07-26T08:01:54.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 21: (Thread::_entry_func(void*)+0xd) [0x7f002dc25cf9] 2024-07-26T08:01:54.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 22: /lib64/libc.so.6(+0x89c02) [0x7f002d289c02] 2024-07-26T08:01:54.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f002d30ec40] 2024-07-26T08:01:54.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2024-07-26T08:01:54.101+0000 7f0027000640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:54.486 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: in thread 7f0027000640 thread_name:ms_dispatch 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f002d23e6f0] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f002d28b94c] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 3: raise() 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 4: abort() 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 5: /lib64/libc.so.6(+0x2871b) [0x7f002d22871b] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 6: /lib64/libc.so.6(+0x37386) [0x7f002d237386] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562efba416b9] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562efbbff80c] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x562efbb3f2ca] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x562efbb0f753] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562efbb10a1b] 2024-07-26T08:01:54.487 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562efbaf4752] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562efba42a56] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562efba46637] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562efba46f5e] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562efba2a03a] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f002dd9b407] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 18: (DispatchQueue::entry()+0x60f) [0x7f002dd9bcf9] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f002de785a1] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 20: (Thread::entry_wrapper()+0x43) [0x7f002dc25cdd] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 21: (Thread::_entry_func(void*)+0xd) [0x7f002dc25cf9] 2024-07-26T08:01:54.488 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 22: /lib64/libc.so.6(+0x89c02) [0x7f002d289c02] 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f002d30ec40] 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 0> 2024-07-26T08:01:54.101+0000 7f0027000640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: in thread 7f0027000640 thread_name:ms_dispatch 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f002d23e6f0] 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f002d28b94c] 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 3: raise() 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 4: abort() 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 5: /lib64/libc.so.6(+0x2871b) [0x7f002d22871b] 2024-07-26T08:01:54.489 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 6: /lib64/libc.so.6(+0x37386) [0x7f002d237386] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562efba416b9] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562efbbff80c] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x562efbb3f2ca] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x562efbb0f753] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562efbb10a1b] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562efbaf4752] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562efba42a56] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562efba46637] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562efba46f5e] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562efba2a03a] 2024-07-26T08:01:54.490 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f002dd9b407] 2024-07-26T08:01:54.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 18: (DispatchQueue::entry()+0x60f) [0x7f002dd9bcf9] 2024-07-26T08:01:54.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f002de785a1] 2024-07-26T08:01:54.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 20: (Thread::entry_wrapper()+0x43) [0x7f002dc25cdd] 2024-07-26T08:01:54.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 21: (Thread::_entry_func(void*)+0xd) [0x7f002dc25cf9] 2024-07-26T08:01:54.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 22: /lib64/libc.so.6(+0x89c02) [0x7f002d289c02] 2024-07-26T08:01:54.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f002d30ec40] 2024-07-26T08:01:54.491 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 0> 2024-07-26T08:01:54.101+0000 7f0027000640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: in thread 7f0027000640 thread_name:ms_dispatch 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f002d23e6f0] 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f002d28b94c] 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 3: raise() 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 4: abort() 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 5: /lib64/libc.so.6(+0x2871b) [0x7f002d22871b] 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 6: /lib64/libc.so.6(+0x37386) [0x7f002d237386] 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562efba416b9] 2024-07-26T08:01:54.492 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562efbbff80c] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x562efbb3f2ca] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x562efbb0f753] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562efbb10a1b] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562efbaf4752] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562efba42a56] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562efba46637] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562efba46f5e] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562efba2a03a] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f002dd9b407] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 18: (DispatchQueue::entry()+0x60f) [0x7f002dd9bcf9] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f002de785a1] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 20: (Thread::entry_wrapper()+0x43) [0x7f002dc25cdd] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 21: (Thread::_entry_func(void*)+0xd) [0x7f002dc25cf9] 2024-07-26T08:01:54.493 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 22: /lib64/libc.so.6(+0x89c02) [0x7f002d289c02] 2024-07-26T08:01:54.494 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f002d30ec40] 2024-07-26T08:01:54.494 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:01:54.494 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71269]: 2024-07-26T08:01:54.494 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 podman[71320]: 2024-07-26 08:01:54.263106839 +0000 UTC m=+0.040422019 container died bba3062b52e314f09055981429f70f39b64403ad0af35d04796fe9466f6cf1b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0) 2024-07-26T08:01:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:54 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:rejoin 2024-07-26T08:01:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:54 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:01:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:54 smithi097 ceph-mon[32231]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:01:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:54 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:01:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:54 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/730588235,v1:172.21.15.181:6835/730588235] up:active 2024-07-26T08:01:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:54 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:01:55.200 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 podman[71320]: 2024-07-26 08:01:54.929033423 +0000 UTC m=+0.706348584 container remove bba3062b52e314f09055981429f70f39b64403ad0af35d04796fe9466f6cf1b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-07-26T08:01:55.200 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:54 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:01:55.201 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:01:55 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:01:55.046+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:01:55.467 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:55 smithi181 ceph-mon[35869]: pgmap v829: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:01:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:55 smithi115 ceph-mon[35364]: pgmap v829: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:01:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:55 smithi097 ceph-mon[32231]: pgmap v829: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:01:55.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:55 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:01:55.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:01:55 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 2.185s CPU time. 2024-07-26T08:01:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:57 smithi115 ceph-mon[35364]: pgmap v830: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:01:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:57 smithi181 ceph-mon[35869]: pgmap v830: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:01:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:57 smithi097 ceph-mon[32231]: pgmap v830: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:01:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:01:59 smithi115 ceph-mon[35364]: pgmap v831: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:01:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:01:59 smithi181 ceph-mon[35869]: pgmap v831: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:01:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:01:59 smithi097 ceph-mon[32231]: pgmap v831: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:01 smithi115 ceph-mon[35364]: pgmap v832: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:01 smithi181 ceph-mon[35869]: pgmap v832: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:01 smithi097 ceph-mon[32231]: pgmap v832: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:02.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:02.402+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:02:02.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:02.403+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:02:02.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:02.403+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:02:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:02.403+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:02:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:02.403+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:02:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:03 smithi181 ceph-mon[35869]: pgmap v833: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:03 smithi097 ceph-mon[32231]: pgmap v833: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:03 smithi115 ceph-mon[35364]: pgmap v833: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:05 smithi097 ceph-mon[32231]: pgmap v834: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:02:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:05 smithi115 ceph-mon[35364]: pgmap v834: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:02:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:05 smithi181 ceph-mon[35869]: pgmap v834: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:02:06.132 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:05 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-07-26T08:02:06.132 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:05 smithi181 systemd[1]: Stopped Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:02:06.132 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:05 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 2.185s CPU time. 2024-07-26T08:02:06.132 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:05 smithi181 systemd[1]: Starting Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:02:06.433 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:06 smithi181 podman[71413]: 2024-07-26 08:02:06.035865772 +0000 UTC m=+0.034897896 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:02:06.434 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:06 smithi181 podman[71413]: 2024-07-26 08:02:06.146067857 +0000 UTC m=+0.145099997 container create 27d29ebbfc17af4abed2d7726ede6a7f8fa57d0dacf2b1d25240f5d497f48a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-07-26T08:02:06.733 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:06 smithi181 podman[71413]: 2024-07-26 08:02:06.432872478 +0000 UTC m=+0.431904625 container init 27d29ebbfc17af4abed2d7726ede6a7f8fa57d0dacf2b1d25240f5d497f48a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0) 2024-07-26T08:02:06.810 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:06 smithi181 podman[71413]: 2024-07-26 08:02:06.443935695 +0000 UTC m=+0.442967868 container start 27d29ebbfc17af4abed2d7726ede6a7f8fa57d0dacf2b1d25240f5d497f48a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0) 2024-07-26T08:02:06.810 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:06 smithi181 bash[71413]: 27d29ebbfc17af4abed2d7726ede6a7f8fa57d0dacf2b1d25240f5d497f48a78 2024-07-26T08:02:06.810 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:06 smithi181 systemd[1]: Started Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:02:06.811 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:06 smithi181 ceph-mds[71433]: -- 172.21.15.181:0/3231902756 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7aba6f860 con 0x55b7afc11a80 2024-07-26T08:02:06.811 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:06 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: starting mds.f at 2024-07-26T08:02:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: pgmap v835: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: daemon mds.f restarted 2024-07-26T08:02:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T08:02:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:boot 2024-07-26T08:02:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:02:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:02:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:02:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:07 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: pgmap v835: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: daemon mds.f restarted 2024-07-26T08:02:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T08:02:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:boot 2024-07-26T08:02:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:02:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:02:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:02:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:07 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: pgmap v835: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: daemon mds.f restarted 2024-07-26T08:02:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T08:02:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:boot 2024-07-26T08:02:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T08:02:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:02:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:02:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:07 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:09.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:09 smithi181 ceph-mon[35869]: pgmap v837: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:02:09.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:09 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:resolve 2024-07-26T08:02:09.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:09 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:09 smithi115 ceph-mon[35364]: pgmap v837: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:02:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:09 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:resolve 2024-07-26T08:02:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:09 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:09 smithi097 ceph-mon[32231]: pgmap v837: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:02:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:09 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:resolve 2024-07-26T08:02:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:09 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:10.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:10 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:02:10.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:10 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:reconnect 2024-07-26T08:02:10.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:10 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:10 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:02:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:10 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:reconnect 2024-07-26T08:02:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:10 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:10 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599989 2024-07-26T08:02:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:10 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:reconnect 2024-07-26T08:02:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:10 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-mon[35364]: pgmap v838: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:02:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:rejoin 2024-07-26T08:02:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-mon[35364]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: *** Caught signal (Aborted) ** 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: in thread 7fe258800640 thread_name:ms_dispatch 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe25ea3e6f0] 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe25ea8b94c] 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 3: raise() 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 4: abort() 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe25ea2871b] 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 6: /lib64/libc.so.6(+0x37386) [0x7fe25ea37386] 2024-07-26T08:02:11.726 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f70a06a6b9] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f70a22880c] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f70a1682ca] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f70a138753] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f70a139a1b] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f70a11d752] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f70a06ba56] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f70a06f637] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f70a06ff5e] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f70a05303a] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe25f59b407] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 18: (DispatchQueue::entry()+0x60f) [0x7fe25f59bcf9] 2024-07-26T08:02:11.727 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe25f6785a1] 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 20: (Thread::entry_wrapper()+0x43) [0x7fe25f425cdd] 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe25f425cf9] 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe25ea89c02] 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe25eb0ec40] 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2024-07-26T08:02:11.316+0000 7fe258800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: in thread 7fe258800640 thread_name:ms_dispatch 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe25ea3e6f0] 2024-07-26T08:02:11.728 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe25ea8b94c] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 3: raise() 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 4: abort() 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe25ea2871b] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 6: /lib64/libc.so.6(+0x37386) [0x7fe25ea37386] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f70a06a6b9] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f70a22880c] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f70a1682ca] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f70a138753] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f70a139a1b] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f70a11d752] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f70a06ba56] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f70a06f637] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f70a06ff5e] 2024-07-26T08:02:11.729 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f70a05303a] 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe25f59b407] 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 18: (DispatchQueue::entry()+0x60f) [0x7fe25f59bcf9] 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe25f6785a1] 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 20: (Thread::entry_wrapper()+0x43) [0x7fe25f425cdd] 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe25f425cf9] 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe25ea89c02] 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe25eb0ec40] 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:11.730 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 0> 2024-07-26T08:02:11.316+0000 7fe258800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: in thread 7fe258800640 thread_name:ms_dispatch 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe25ea3e6f0] 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe25ea8b94c] 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 3: raise() 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 4: abort() 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe25ea2871b] 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 6: /lib64/libc.so.6(+0x37386) [0x7fe25ea37386] 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f70a06a6b9] 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f70a22880c] 2024-07-26T08:02:11.731 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f70a1682ca] 2024-07-26T08:02:11.732 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f70a138753] 2024-07-26T08:02:11.732 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f70a139a1b] 2024-07-26T08:02:11.732 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f70a11d752] 2024-07-26T08:02:11.732 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f70a06ba56] 2024-07-26T08:02:11.732 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f70a06f637] 2024-07-26T08:02:11.732 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f70a06ff5e] 2024-07-26T08:02:11.732 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f70a05303a] 2024-07-26T08:02:11.732 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe25f59b407] 2024-07-26T08:02:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:11 smithi181 ceph-mon[35869]: pgmap v838: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:02:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:11 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:rejoin 2024-07-26T08:02:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:11 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:11 smithi181 ceph-mon[35869]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:02:11.733 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 18: (DispatchQueue::entry()+0x60f) [0x7fe25f59bcf9] 2024-07-26T08:02:11.733 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe25f6785a1] 2024-07-26T08:02:11.733 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 20: (Thread::entry_wrapper()+0x43) [0x7fe25f425cdd] 2024-07-26T08:02:11.733 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe25f425cf9] 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe25ea89c02] 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe25eb0ec40] 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: -2> 2024-07-26T08:02:11.316+0000 7fe258800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: in thread 7fe258800640 thread_name:ms_dispatch 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe25ea3e6f0] 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe25ea8b94c] 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 3: raise() 2024-07-26T08:02:11.734 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 4: abort() 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe25ea2871b] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 6: /lib64/libc.so.6(+0x37386) [0x7fe25ea37386] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f70a06a6b9] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f70a22880c] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f70a1682ca] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f70a138753] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f70a139a1b] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f70a11d752] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f70a06ba56] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f70a06f637] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f70a06ff5e] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f70a05303a] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe25f59b407] 2024-07-26T08:02:11.735 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 18: (DispatchQueue::entry()+0x60f) [0x7fe25f59bcf9] 2024-07-26T08:02:11.736 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe25f6785a1] 2024-07-26T08:02:11.736 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 20: (Thread::entry_wrapper()+0x43) [0x7fe25f425cdd] 2024-07-26T08:02:11.736 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe25f425cf9] 2024-07-26T08:02:11.736 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe25ea89c02] 2024-07-26T08:02:11.736 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe25eb0ec40] 2024-07-26T08:02:11.736 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:11.736 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e[71156]: 2024-07-26T08:02:11.736 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:11 smithi115 podman[71194]: 2024-07-26 08:02:11.468769035 +0000 UTC m=+0.036212404 container died e82bde1afb1b52971015ed0565d25481b792a4ea17995293dc9971ba8370d383 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724) 2024-07-26T08:02:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:11 smithi097 ceph-mon[32231]: pgmap v838: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:02:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:11 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:rejoin 2024-07-26T08:02:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:11 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:11 smithi097 ceph-mon[32231]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T08:02:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:12 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:12 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:active 2024-07-26T08:02:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:12 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:02:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:12 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:12 smithi115 ceph-mon[35364]: pgmap v839: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:02:12.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:12 smithi115 podman[71194]: 2024-07-26 08:02:12.177988538 +0000 UTC m=+0.745431948 container remove e82bde1afb1b52971015ed0565d25481b792a4ea17995293dc9971ba8370d383 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T08:02:12.475 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:12 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:02:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:12 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:12 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:active 2024-07-26T08:02:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:12 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:02:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:12 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:12 smithi181 ceph-mon[35869]: pgmap v839: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:02:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:12 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:12 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.115:6836/204794148,v1:172.21.15.115:6837/204794148] up:active 2024-07-26T08:02:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:12 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:02:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:12 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:12 smithi097 ceph-mon[32231]: pgmap v839: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:02:12.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:12 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T08:02:12.974 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:12 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.150s CPU time. 2024-07-26T08:02:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:15 smithi115 ceph-mon[35364]: pgmap v840: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:02:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:15 smithi181 ceph-mon[35869]: pgmap v840: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:02:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:15 smithi097 ceph-mon[32231]: pgmap v840: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:02:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:17.407+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:02:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:17.408+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:02:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:17.408+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:02:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:17.408+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:02:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:17.408+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:02:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:17 smithi115 ceph-mon[35364]: pgmap v841: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:17 smithi181 ceph-mon[35869]: pgmap v841: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:17 smithi097 ceph-mon[32231]: pgmap v841: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:19 smithi115 ceph-mon[35364]: pgmap v842: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:02:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:19 smithi181 ceph-mon[35869]: pgmap v842: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:02:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:19 smithi097 ceph-mon[32231]: pgmap v842: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:02:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:21 smithi115 ceph-mon[35364]: pgmap v843: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:02:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:21 smithi181 ceph-mon[35869]: pgmap v843: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:02:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:21 smithi097 ceph-mon[32231]: pgmap v843: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:02:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:02:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:02:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:02:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:02:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:02:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:02:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:02:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:02:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:02:23.157 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:22 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 6. 2024-07-26T08:02:23.157 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:22 smithi115 systemd[1]: Stopped Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:02:23.157 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:22 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Consumed 2.150s CPU time. 2024-07-26T08:02:23.157 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:22 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Start request repeated too quickly. 2024-07-26T08:02:23.157 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:22 smithi115 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T08:02:23.157 INFO:journalctl@ceph.mds.e.smithi115.stdout:Jul 26 08:02:22 smithi115 systemd[1]: Failed to start Ceph mds.e for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:02:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:23 smithi115 ceph-mon[35364]: pgmap v844: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 179 B/s wr, 11 op/s 2024-07-26T08:02:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:23 smithi115 ceph-mon[35364]: pgmap v845: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 108 B/s wr, 11 op/s 2024-07-26T08:02:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:23 smithi181 ceph-mon[35869]: pgmap v844: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 179 B/s wr, 11 op/s 2024-07-26T08:02:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:23 smithi181 ceph-mon[35869]: pgmap v845: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 108 B/s wr, 11 op/s 2024-07-26T08:02:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:23 smithi097 ceph-mon[32231]: pgmap v844: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 179 B/s wr, 11 op/s 2024-07-26T08:02:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:23 smithi097 ceph-mon[32231]: pgmap v845: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 108 B/s wr, 11 op/s 2024-07-26T08:02:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:25 smithi115 ceph-mon[35364]: pgmap v846: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 9 op/s 2024-07-26T08:02:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:25 smithi115 ceph-mon[35364]: 1 slow requests, 1 included below; oldest blocked for > 182.599209 secs 2024-07-26T08:02:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:25 smithi115 ceph-mon[35364]: slow request 182.599208 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:02:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:25 smithi181 ceph-mon[35869]: pgmap v846: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 9 op/s 2024-07-26T08:02:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:25 smithi181 ceph-mon[35869]: 1 slow requests, 1 included below; oldest blocked for > 182.599209 secs 2024-07-26T08:02:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:25 smithi181 ceph-mon[35869]: slow request 182.599208 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:02:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:25 smithi097 ceph-mon[32231]: pgmap v846: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 9 op/s 2024-07-26T08:02:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:25 smithi097 ceph-mon[32231]: 1 slow requests, 1 included below; oldest blocked for > 182.599209 secs 2024-07-26T08:02:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:25 smithi097 ceph-mon[32231]: slow request 182.599208 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:02:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:27 smithi115 ceph-mon[35364]: pgmap v847: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T08:02:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:27 smithi181 ceph-mon[35869]: pgmap v847: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T08:02:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:27 smithi097 ceph-mon[32231]: pgmap v847: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T08:02:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:28 smithi181 ceph-mon[35869]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T08:02:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:28 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:active 2024-07-26T08:02:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:28 smithi181 ceph-mon[35869]: Replacing daemon mds.e as rank 1 with standby daemon mds.f 2024-07-26T08:02:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:28 smithi181 ceph-mon[35869]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T08:02:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:28 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:28 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:28 smithi181 ceph-mon[35869]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T08:02:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:28 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:28 smithi097 ceph-mon[32231]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T08:02:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:28 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:active 2024-07-26T08:02:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:28 smithi097 ceph-mon[32231]: Replacing daemon mds.e as rank 1 with standby daemon mds.f 2024-07-26T08:02:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:28 smithi097 ceph-mon[32231]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T08:02:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:28 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:28 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:28 smithi097 ceph-mon[32231]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T08:02:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:28 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:28 smithi115 ceph-mon[35364]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T08:02:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:28 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6836/3957930666,v1:172.21.15.181:6837/3957930666] up:active 2024-07-26T08:02:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:28 smithi115 ceph-mon[35364]: Replacing daemon mds.e as rank 1 with standby daemon mds.f 2024-07-26T08:02:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:28 smithi115 ceph-mon[35364]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T08:02:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:28 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:28 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:28 smithi115 ceph-mon[35364]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T08:02:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:28 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:29 smithi097 ceph-mon[32231]: pgmap v848: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:02:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:29 smithi115 ceph-mon[35364]: pgmap v848: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:02:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:29 smithi181 ceph-mon[35869]: pgmap v848: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:02:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:30 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:resolve 2024-07-26T08:02:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:30 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 187.599353 secs 2024-07-26T08:02:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:30 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:resolve 2024-07-26T08:02:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:30 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 187.599353 secs 2024-07-26T08:02:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:30 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:resolve 2024-07-26T08:02:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:30 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 187.599353 secs 2024-07-26T08:02:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:31 smithi115 ceph-mon[35364]: pgmap v850: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9 op/s 2024-07-26T08:02:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:31 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:02:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:31 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:reconnect 2024-07-26T08:02:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:31 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:31 smithi181 ceph-mon[35869]: pgmap v850: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9 op/s 2024-07-26T08:02:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:31 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:02:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:31 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:reconnect 2024-07-26T08:02:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:31 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:31 smithi097 ceph-mon[32231]: pgmap v850: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9 op/s 2024-07-26T08:02:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:31 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00599988 2024-07-26T08:02:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:31 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:reconnect 2024-07-26T08:02:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:31 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:32.325 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:rejoin 2024-07-26T08:02:32.325 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-mon[35869]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-mon[35869]: pgmap v851: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: *** Caught signal (Aborted) ** 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: in thread 7ff644800640 thread_name:ms_dispatch 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff64aa3e6f0] 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff64aa8b94c] 2024-07-26T08:02:32.326 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 3: raise() 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 4: abort() 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff64aa2871b] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 6: /lib64/libc.so.6(+0x37386) [0x7ff64aa37386] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b772fdb6b9] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b77319980c] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b7730d92ca] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b7730a9753] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b7730aaa1b] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b77308e752] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b772fdca56] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b772fe0637] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b772fe0f5e] 2024-07-26T08:02:32.327 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b772fc403a] 2024-07-26T08:02:32.328 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff64b59b407] 2024-07-26T08:02:32.328 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 18: (DispatchQueue::entry()+0x60f) [0x7ff64b59bcf9] 2024-07-26T08:02:32.328 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff64b6785a1] 2024-07-26T08:02:32.328 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 20: (Thread::entry_wrapper()+0x43) [0x7ff64b425cdd] 2024-07-26T08:02:32.328 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff64b425cf9] 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff64aa89c02] 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff64ab0ec40] 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2024-07-26T08:02:32.313+0000 7ff644800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: in thread 7ff644800640 thread_name:ms_dispatch 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff64aa3e6f0] 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff64aa8b94c] 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 3: raise() 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 4: abort() 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff64aa2871b] 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 6: /lib64/libc.so.6(+0x37386) [0x7ff64aa37386] 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b772fdb6b9] 2024-07-26T08:02:32.329 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b77319980c] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b7730d92ca] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b7730a9753] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b7730aaa1b] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b77308e752] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b772fdca56] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b772fe0637] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b772fe0f5e] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b772fc403a] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff64b59b407] 2024-07-26T08:02:32.330 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 18: (DispatchQueue::entry()+0x60f) [0x7ff64b59bcf9] 2024-07-26T08:02:32.331 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff64b6785a1] 2024-07-26T08:02:32.331 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 20: (Thread::entry_wrapper()+0x43) [0x7ff64b425cdd] 2024-07-26T08:02:32.331 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff64b425cf9] 2024-07-26T08:02:32.331 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff64aa89c02] 2024-07-26T08:02:32.331 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff64ab0ec40] 2024-07-26T08:02:32.331 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:32.331 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2024-07-26T08:02:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:32.412+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:02:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:32.413+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:02:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:32.413+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:02:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:32.413+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:02:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:32.414+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:02:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:32 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:rejoin 2024-07-26T08:02:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:32 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:32 smithi115 ceph-mon[35364]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:02:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:32 smithi115 ceph-mon[35364]: pgmap v851: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:02:32.735 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 0> 2024-07-26T08:02:32.313+0000 7ff644800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: in thread 7ff644800640 thread_name:ms_dispatch 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff64aa3e6f0] 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff64aa8b94c] 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 3: raise() 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 4: abort() 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff64aa2871b] 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 6: /lib64/libc.so.6(+0x37386) [0x7ff64aa37386] 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b772fdb6b9] 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b77319980c] 2024-07-26T08:02:32.736 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b7730d92ca] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b7730a9753] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b7730aaa1b] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b77308e752] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b772fdca56] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b772fe0637] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b772fe0f5e] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b772fc403a] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff64b59b407] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 18: (DispatchQueue::entry()+0x60f) [0x7ff64b59bcf9] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff64b6785a1] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 20: (Thread::entry_wrapper()+0x43) [0x7ff64b425cdd] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff64b425cf9] 2024-07-26T08:02:32.737 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff64aa89c02] 2024-07-26T08:02:32.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff64ab0ec40] 2024-07-26T08:02:32.738 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 0> 2024-07-26T08:02:32.313+0000 7ff644800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: in thread 7ff644800640 thread_name:ms_dispatch 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff64aa3e6f0] 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff64aa8b94c] 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 3: raise() 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 4: abort() 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff64aa2871b] 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 6: /lib64/libc.so.6(+0x37386) [0x7ff64aa37386] 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b772fdb6b9] 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b77319980c] 2024-07-26T08:02:32.739 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b7730d92ca] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b7730a9753] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b7730aaa1b] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b77308e752] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b772fdca56] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b772fe0637] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b772fe0f5e] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b772fc403a] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff64b59b407] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 18: (DispatchQueue::entry()+0x60f) [0x7ff64b59bcf9] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff64b6785a1] 2024-07-26T08:02:32.740 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 20: (Thread::entry_wrapper()+0x43) [0x7ff64b425cdd] 2024-07-26T08:02:32.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff64b425cf9] 2024-07-26T08:02:32.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff64aa89c02] 2024-07-26T08:02:32.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff64ab0ec40] 2024-07-26T08:02:32.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:32.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f[71429]: 2024-07-26T08:02:32.741 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:32 smithi181 podman[71597]: 2024-07-26 08:02:32.495010463 +0000 UTC m=+0.037275959 container died 27d29ebbfc17af4abed2d7726ede6a7f8fa57d0dacf2b1d25240f5d497f48a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T08:02:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:32 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:rejoin 2024-07-26T08:02:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:32 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:32 smithi097 ceph-mon[32231]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-07-26T08:02:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:32 smithi097 ceph-mon[32231]: pgmap v851: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:02:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:33 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:33 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:active 2024-07-26T08:02:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:33 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:33 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:02:33.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:33 smithi181 podman[71597]: 2024-07-26 08:02:33.21092008 +0000 UTC m=+0.753185610 container remove 27d29ebbfc17af4abed2d7726ede6a7f8fa57d0dacf2b1d25240f5d497f48a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T08:02:33.483 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:33 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:02:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:33 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:33 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:active 2024-07-26T08:02:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:33 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:33 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:02:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:33 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:33 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.181:6834/4115257333,v1:172.21.15.181:6835/4115257333] up:active 2024-07-26T08:02:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:33 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:02:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:33 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:02:33.982 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:33 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:02:33.983 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:33 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 2.120s CPU time. 2024-07-26T08:02:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:34 smithi115 ceph-mon[35364]: pgmap v852: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:02:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:34 smithi181 ceph-mon[35869]: pgmap v852: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:02:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:34 smithi097 ceph-mon[32231]: pgmap v852: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:02:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 192.599497 secs 2024-07-26T08:02:35.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:02:35 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:02:35.052+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:02:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 192.599497 secs 2024-07-26T08:02:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 192.599497 secs 2024-07-26T08:02:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:36 smithi115 ceph-mon[35364]: pgmap v853: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:36 smithi181 ceph-mon[35869]: pgmap v853: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:36 smithi097 ceph-mon[32231]: pgmap v853: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:38 smithi115 ceph-mon[35364]: pgmap v854: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:38 smithi181 ceph-mon[35869]: pgmap v854: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:38 smithi097 ceph-mon[32231]: pgmap v854: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:40.809 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:40 smithi115 ceph-mon[35364]: pgmap v855: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 181 B/s wr, 11 op/s 2024-07-26T08:02:40.810 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 197.599623 secs 2024-07-26T08:02:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:40 smithi181 ceph-mon[35869]: pgmap v855: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 181 B/s wr, 11 op/s 2024-07-26T08:02:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 197.599623 secs 2024-07-26T08:02:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:40 smithi097 ceph-mon[32231]: pgmap v855: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 181 B/s wr, 11 op/s 2024-07-26T08:02:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 197.599623 secs 2024-07-26T08:02:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:42 smithi115 ceph-mon[35364]: pgmap v856: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:42 smithi181 ceph-mon[35869]: pgmap v856: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:42 smithi097 ceph-mon[32231]: pgmap v856: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:02:44.232 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:43 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-07-26T08:02:44.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:43 smithi181 systemd[1]: Stopped Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:02:44.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:43 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Consumed 2.120s CPU time. 2024-07-26T08:02:44.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:43 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-07-26T08:02:44.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:43 smithi181 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:02:44.233 INFO:journalctl@ceph.mds.f.smithi181.stdout:Jul 26 08:02:43 smithi181 systemd[1]: Failed to start Ceph mds.f for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:02:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:44 smithi115 ceph-mon[35364]: pgmap v857: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:02:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:44 smithi181 ceph-mon[35869]: pgmap v857: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:02:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:44 smithi097 ceph-mon[32231]: pgmap v857: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:02:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 202.599786 secs 2024-07-26T08:02:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 202.599786 secs 2024-07-26T08:02:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 202.599786 secs 2024-07-26T08:02:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:46 smithi115 ceph-mon[35364]: pgmap v858: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-26T08:02:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:46 smithi181 ceph-mon[35869]: pgmap v858: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-26T08:02:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:47.423+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:02:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:47.424+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:02:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:47.424+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:02:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:47.424+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:02:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:02:47.424+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:02:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:48 smithi115 ceph-mon[35364]: pgmap v859: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T08:02:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:48 smithi181 ceph-mon[35869]: pgmap v859: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T08:02:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:46 smithi097 ceph-mon[32231]: pgmap v858: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-26T08:02:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:48 smithi097 ceph-mon[32231]: pgmap v859: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T08:02:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:49 smithi115 ceph-mon[35364]: Replacing daemon mds.f as rank 1 with standby daemon mds.d 2024-07-26T08:02:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:49 smithi115 ceph-mon[35364]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T08:02:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:49 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:49 smithi115 ceph-mon[35364]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T08:02:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:49 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:49 smithi181 ceph-mon[35869]: Replacing daemon mds.f as rank 1 with standby daemon mds.d 2024-07-26T08:02:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:49 smithi181 ceph-mon[35869]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T08:02:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:49 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:49 smithi181 ceph-mon[35869]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T08:02:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:49 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:49 smithi097 ceph-mon[32231]: Replacing daemon mds.f as rank 1 with standby daemon mds.d 2024-07-26T08:02:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:49 smithi097 ceph-mon[32231]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T08:02:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:49 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:02:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:49 smithi097 ceph-mon[32231]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T08:02:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:49 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:02:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:50 smithi115 ceph-mon[35364]: pgmap v861: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:02:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 207.599873 secs 2024-07-26T08:02:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:50 smithi181 ceph-mon[35869]: pgmap v861: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:02:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 207.599873 secs 2024-07-26T08:02:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:50 smithi097 ceph-mon[32231]: pgmap v861: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:02:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 207.599873 secs 2024-07-26T08:02:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:51 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:resolve 2024-07-26T08:02:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:51 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:51 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:resolve 2024-07-26T08:02:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:51 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:51 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:resolve 2024-07-26T08:02:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:51 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:02:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:52 smithi115 ceph-mon[35364]: pgmap v862: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:02:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:52 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00999983 2024-07-26T08:02:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:52 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:reconnect 2024-07-26T08:02:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:52 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:52 smithi181 ceph-mon[35869]: pgmap v862: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:02:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:52 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00999983 2024-07-26T08:02:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:52 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:reconnect 2024-07-26T08:02:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:52 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:52 smithi097 ceph-mon[32231]: pgmap v862: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:02:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:52 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00999983 2024-07-26T08:02:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:52 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:reconnect 2024-07-26T08:02:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:52 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:02:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:53 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:rejoin 2024-07-26T08:02:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:53 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:53 smithi115 ceph-mon[35364]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:02:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:53 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:rejoin 2024-07-26T08:02:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:53 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:53 smithi181 ceph-mon[35869]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:02:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:rejoin 2024-07-26T08:02:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:02:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-mon[32231]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: *** Caught signal (Aborted) ** 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: in thread 7fc410800640 thread_name:ms_dispatch 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc416a3e6f0] 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc416a8b94c] 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 3: raise() 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 4: abort() 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc416a2871b] 2024-07-26T08:02:54.002 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 6: /lib64/libc.so.6(+0x37386) [0x7fc416a37386] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563fe4c536b9] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563fe4e1180c] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563fe4d512ca] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563fe4d21753] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563fe4d22a1b] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563fe4d06752] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563fe4c54a56] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563fe4c58637] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563fe4c58f5e] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563fe4c3c03a] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc41759b407] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 18: (DispatchQueue::entry()+0x60f) [0x7fc41759bcf9] 2024-07-26T08:02:54.003 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc4176785a1] 2024-07-26T08:02:54.004 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 20: (Thread::entry_wrapper()+0x43) [0x7fc417425cdd] 2024-07-26T08:02:54.004 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc417425cf9] 2024-07-26T08:02:54.004 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc416a89c02] 2024-07-26T08:02:54.004 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc416b0ec40] 2024-07-26T08:02:54.004 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2024-07-26T08:02:53.663+0000 7fc410800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:54.004 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: in thread 7fc410800640 thread_name:ms_dispatch 2024-07-26T08:02:54.004 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2024-07-26T08:02:54.004 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc416a3e6f0] 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc416a8b94c] 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 3: raise() 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 4: abort() 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc416a2871b] 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 6: /lib64/libc.so.6(+0x37386) [0x7fc416a37386] 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563fe4c536b9] 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563fe4e1180c] 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563fe4d512ca] 2024-07-26T08:02:54.005 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563fe4d21753] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563fe4d22a1b] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563fe4d06752] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563fe4c54a56] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563fe4c58637] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563fe4c58f5e] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563fe4c3c03a] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc41759b407] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 18: (DispatchQueue::entry()+0x60f) [0x7fc41759bcf9] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc4176785a1] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 20: (Thread::entry_wrapper()+0x43) [0x7fc417425cdd] 2024-07-26T08:02:54.006 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc417425cf9] 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc416a89c02] 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc416b0ec40] 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 0> 2024-07-26T08:02:53.663+0000 7fc410800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: in thread 7fc410800640 thread_name:ms_dispatch 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc416a3e6f0] 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc416a8b94c] 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 3: raise() 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 4: abort() 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc416a2871b] 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 6: /lib64/libc.so.6(+0x37386) [0x7fc416a37386] 2024-07-26T08:02:54.007 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563fe4c536b9] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563fe4e1180c] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563fe4d512ca] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563fe4d21753] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563fe4d22a1b] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563fe4d06752] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563fe4c54a56] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563fe4c58637] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563fe4c58f5e] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563fe4c3c03a] 2024-07-26T08:02:54.008 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc41759b407] 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 18: (DispatchQueue::entry()+0x60f) [0x7fc41759bcf9] 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc4176785a1] 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 20: (Thread::entry_wrapper()+0x43) [0x7fc417425cdd] 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc417425cf9] 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc416a89c02] 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc416b0ec40] 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 0> 2024-07-26T08:02:53.663+0000 7fc410800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: in thread 7fc410800640 thread_name:ms_dispatch 2024-07-26T08:02:54.009 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc416a3e6f0] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc416a8b94c] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 3: raise() 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 4: abort() 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc416a2871b] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 6: /lib64/libc.so.6(+0x37386) [0x7fc416a37386] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563fe4c536b9] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563fe4e1180c] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563fe4d512ca] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563fe4d21753] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563fe4d22a1b] 2024-07-26T08:02:54.010 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563fe4d06752] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563fe4c54a56] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563fe4c58637] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563fe4c58f5e] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563fe4c3c03a] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc41759b407] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 18: (DispatchQueue::entry()+0x60f) [0x7fc41759bcf9] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc4176785a1] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 20: (Thread::entry_wrapper()+0x43) [0x7fc417425cdd] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc417425cf9] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc416a89c02] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc416b0ec40] 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[75579]: 2024-07-26T08:02:54.011 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:53 smithi097 podman[194183]: 2024-07-26 08:02:53.834486249 +0000 UTC m=+0.037098230 container died c053db36be3d8ab03a75401de14c75e977d1448db2687db5924d888ec9b2a3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-07-26T08:02:54.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:54 smithi097 podman[194183]: 2024-07-26 08:02:54.748063723 +0000 UTC m=+0.950675733 container remove c053db36be3d8ab03a75401de14c75e977d1448db2687db5924d888ec9b2a3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-07-26T08:02:54.761 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:54 smithi097 ceph-mon[32231]: pgmap v863: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:02:54.761 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:54 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:54.761 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:54 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:active 2024-07-26T08:02:54.761 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:54 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:02:54.761 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:54 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:02:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:54 smithi115 ceph-mon[35364]: pgmap v863: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:02:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:54 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:54 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:active 2024-07-26T08:02:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:54 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:02:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:54 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:02:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:54 smithi181 ceph-mon[35869]: pgmap v863: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:02:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:54 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:02:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:54 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4227989530,v1:172.21.15.97:6837/4227989530] up:active 2024-07-26T08:02:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:54 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:02:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:54 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:02:55.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:54 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:02:55.482 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:02:55 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:02:55.054+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:02:55.668 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:55 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T08:02:55.668 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:02:55 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 7.778s CPU time. 2024-07-26T08:02:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 212.599971 secs 2024-07-26T08:02:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 212.599971 secs 2024-07-26T08:02:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 212.599971 secs 2024-07-26T08:02:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:56 smithi115 ceph-mon[35364]: pgmap v864: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:02:56.979 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:56 smithi097 ceph-mon[32231]: pgmap v864: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:02:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:56 smithi181 ceph-mon[35869]: pgmap v864: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:02:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:58 smithi115 ceph-mon[35364]: pgmap v865: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:58 smithi181 ceph-mon[35869]: pgmap v865: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:58 smithi097 ceph-mon[32231]: pgmap v865: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:02:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:02:59 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 217.600109 secs 2024-07-26T08:02:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:02:59 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 217.600109 secs 2024-07-26T08:03:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:02:59 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 217.600109 secs 2024-07-26T08:03:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:00 smithi115 ceph-mon[35364]: pgmap v866: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 197 B/s wr, 11 op/s 2024-07-26T08:03:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:00 smithi181 ceph-mon[35869]: pgmap v866: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 197 B/s wr, 11 op/s 2024-07-26T08:03:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:00 smithi097 ceph-mon[32231]: pgmap v866: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 197 B/s wr, 11 op/s 2024-07-26T08:03:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:02.427+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:03:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:02.428+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:03:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:02.428+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:03:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:02.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:03:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:02.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:03:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:02 smithi115 ceph-mon[35364]: pgmap v867: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:02 smithi181 ceph-mon[35869]: pgmap v867: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:02 smithi097 ceph-mon[32231]: pgmap v867: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:04 smithi115 ceph-mon[35364]: pgmap v868: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:04 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 222.600284 secs 2024-07-26T08:03:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:04 smithi181 ceph-mon[35869]: pgmap v868: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:04 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 222.600284 secs 2024-07-26T08:03:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:04 smithi097 ceph-mon[32231]: pgmap v868: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:04 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 222.600284 secs 2024-07-26T08:03:05.905 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:05 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-07-26T08:03:05.905 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:05 smithi097 systemd[1]: Stopped Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:03:05.905 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:05 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 7.778s CPU time. 2024-07-26T08:03:05.905 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:05 smithi097 systemd[1]: Starting Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:03:06.222 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:05 smithi097 podman[194287]: 2024-07-26 08:03:05.830105601 +0000 UTC m=+0.041632443 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:03:06.223 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:05 smithi097 podman[194287]: 2024-07-26 08:03:05.974783438 +0000 UTC m=+0.186310286 container create 18dc838ebf34177f33e59906dafe675b9e0971920e213091d27c737fb5542c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-07-26T08:03:06.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:06 smithi097 podman[194287]: 2024-07-26 08:03:06.221188163 +0000 UTC m=+0.432715059 container init 18dc838ebf34177f33e59906dafe675b9e0971920e213091d27c737fb5542c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:03:06.502 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:06 smithi097 podman[194287]: 2024-07-26 08:03:06.231320916 +0000 UTC m=+0.442847737 container start 18dc838ebf34177f33e59906dafe675b9e0971920e213091d27c737fb5542c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-26T08:03:06.502 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:06 smithi097 bash[194287]: 18dc838ebf34177f33e59906dafe675b9e0971920e213091d27c737fb5542c85 2024-07-26T08:03:06.502 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:06 smithi097 systemd[1]: Started Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:03:06.502 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mds[194307]: -- 172.21.15.97:0/2919837278 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c966e93860 con 0x55c96b035180 2024-07-26T08:03:06.502 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: starting mds.d at 2024-07-26T08:03:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: pgmap v869: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: daemon mds.d restarted 2024-07-26T08:03:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T08:03:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:boot 2024-07-26T08:03:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:03:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:03:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:06 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: pgmap v869: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: daemon mds.d restarted 2024-07-26T08:03:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T08:03:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:boot 2024-07-26T08:03:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:03:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:03:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:06 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: pgmap v869: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: daemon mds.d restarted 2024-07-26T08:03:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T08:03:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:boot 2024-07-26T08:03:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:03:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:03:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:06 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:08 smithi097 ceph-mon[32231]: pgmap v871: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:03:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:08 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:resolve 2024-07-26T08:03:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:08 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:08 smithi115 ceph-mon[35364]: pgmap v871: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:03:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:08 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:resolve 2024-07-26T08:03:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:08 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:08 smithi181 ceph-mon[35869]: pgmap v871: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:03:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:08 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:resolve 2024-07-26T08:03:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:08 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:09 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:reconnect 2024-07-26T08:03:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:09 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:09 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899984 2024-07-26T08:03:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:09 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 227.600418 secs 2024-07-26T08:03:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:09 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:reconnect 2024-07-26T08:03:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:09 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:09 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899984 2024-07-26T08:03:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:09 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 227.600418 secs 2024-07-26T08:03:10.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:09 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:reconnect 2024-07-26T08:03:10.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:09 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:10.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:09 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899984 2024-07-26T08:03:10.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:09 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 227.600418 secs 2024-07-26T08:03:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:10 smithi115 ceph-mon[35364]: pgmap v872: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:03:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:10 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:rejoin 2024-07-26T08:03:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:10 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:10 smithi115 ceph-mon[35364]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:03:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:10 smithi181 ceph-mon[35869]: pgmap v872: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:03:11.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:10 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:rejoin 2024-07-26T08:03:11.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:10 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:11.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:10 smithi181 ceph-mon[35869]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:03:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-mon[32231]: pgmap v872: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:03:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:rejoin 2024-07-26T08:03:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-mon[32231]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:03:11.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:03:11.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: *** Caught signal (Aborted) ** 2024-07-26T08:03:11.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: in thread 7f6997e00640 thread_name:ms_dispatch 2024-07-26T08:03:11.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:11.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f699e03e6f0] 2024-07-26T08:03:11.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f699e08b94c] 2024-07-26T08:03:11.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 3: raise() 2024-07-26T08:03:11.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 4: abort() 2024-07-26T08:03:11.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 5: /lib64/libc.so.6(+0x2871b) [0x7f699e02871b] 2024-07-26T08:03:11.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 6: /lib64/libc.so.6(+0x37386) [0x7f699e037386] 2024-07-26T08:03:11.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cd33f416b9] 2024-07-26T08:03:11.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cd340ff80c] 2024-07-26T08:03:11.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cd3403f2ca] 2024-07-26T08:03:11.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cd3400f753] 2024-07-26T08:03:11.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cd34010a1b] 2024-07-26T08:03:11.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cd33ff4752] 2024-07-26T08:03:11.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cd33f42a56] 2024-07-26T08:03:11.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cd33f46637] 2024-07-26T08:03:11.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cd33f46f5e] 2024-07-26T08:03:11.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cd33f2a03a] 2024-07-26T08:03:11.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f699eb9b407] 2024-07-26T08:03:11.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 18: (DispatchQueue::entry()+0x60f) [0x7f699eb9bcf9] 2024-07-26T08:03:11.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f699ec785a1] 2024-07-26T08:03:11.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 20: (Thread::entry_wrapper()+0x43) [0x7f699ea25cdd] 2024-07-26T08:03:11.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 21: (Thread::_entry_func(void*)+0xd) [0x7f699ea25cf9] 2024-07-26T08:03:11.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 22: /lib64/libc.so.6(+0x89c02) [0x7f699e089c02] 2024-07-26T08:03:11.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f699e10ec40] 2024-07-26T08:03:11.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2024-07-26T08:03:10.780+0000 7f6997e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:11.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: in thread 7f6997e00640 thread_name:ms_dispatch 2024-07-26T08:03:11.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2024-07-26T08:03:11.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:11.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f699e03e6f0] 2024-07-26T08:03:11.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f699e08b94c] 2024-07-26T08:03:11.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 3: raise() 2024-07-26T08:03:11.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 4: abort() 2024-07-26T08:03:11.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 5: /lib64/libc.so.6(+0x2871b) [0x7f699e02871b] 2024-07-26T08:03:11.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 6: /lib64/libc.so.6(+0x37386) [0x7f699e037386] 2024-07-26T08:03:11.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cd33f416b9] 2024-07-26T08:03:11.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cd340ff80c] 2024-07-26T08:03:11.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cd3403f2ca] 2024-07-26T08:03:11.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cd3400f753] 2024-07-26T08:03:11.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cd34010a1b] 2024-07-26T08:03:11.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cd33ff4752] 2024-07-26T08:03:11.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cd33f42a56] 2024-07-26T08:03:11.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cd33f46637] 2024-07-26T08:03:11.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cd33f46f5e] 2024-07-26T08:03:11.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cd33f2a03a] 2024-07-26T08:03:11.263 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f699eb9b407] 2024-07-26T08:03:11.263 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 18: (DispatchQueue::entry()+0x60f) [0x7f699eb9bcf9] 2024-07-26T08:03:11.264 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f699ec785a1] 2024-07-26T08:03:11.264 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 20: (Thread::entry_wrapper()+0x43) [0x7f699ea25cdd] 2024-07-26T08:03:11.264 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 21: (Thread::_entry_func(void*)+0xd) [0x7f699ea25cf9] 2024-07-26T08:03:11.264 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 22: /lib64/libc.so.6(+0x89c02) [0x7f699e089c02] 2024-07-26T08:03:11.264 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f699e10ec40] 2024-07-26T08:03:11.264 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:11.264 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2024-07-26T08:03:11.265 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 0> 2024-07-26T08:03:10.780+0000 7f6997e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:11.265 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: in thread 7f6997e00640 thread_name:ms_dispatch 2024-07-26T08:03:11.265 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2024-07-26T08:03:11.265 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:11.265 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f699e03e6f0] 2024-07-26T08:03:11.265 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f699e08b94c] 2024-07-26T08:03:11.265 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 3: raise() 2024-07-26T08:03:11.266 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 4: abort() 2024-07-26T08:03:11.266 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 5: /lib64/libc.so.6(+0x2871b) [0x7f699e02871b] 2024-07-26T08:03:11.266 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 6: /lib64/libc.so.6(+0x37386) [0x7f699e037386] 2024-07-26T08:03:11.266 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cd33f416b9] 2024-07-26T08:03:11.266 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cd340ff80c] 2024-07-26T08:03:11.267 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cd3403f2ca] 2024-07-26T08:03:11.267 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cd3400f753] 2024-07-26T08:03:11.267 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cd34010a1b] 2024-07-26T08:03:11.267 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cd33ff4752] 2024-07-26T08:03:11.267 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cd33f42a56] 2024-07-26T08:03:11.267 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cd33f46637] 2024-07-26T08:03:11.267 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cd33f46f5e] 2024-07-26T08:03:11.268 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cd33f2a03a] 2024-07-26T08:03:11.268 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f699eb9b407] 2024-07-26T08:03:11.269 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 18: (DispatchQueue::entry()+0x60f) [0x7f699eb9bcf9] 2024-07-26T08:03:11.269 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f699ec785a1] 2024-07-26T08:03:11.269 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 20: (Thread::entry_wrapper()+0x43) [0x7f699ea25cdd] 2024-07-26T08:03:11.269 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 21: (Thread::_entry_func(void*)+0xd) [0x7f699ea25cf9] 2024-07-26T08:03:11.269 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 22: /lib64/libc.so.6(+0x89c02) [0x7f699e089c02] 2024-07-26T08:03:11.269 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f699e10ec40] 2024-07-26T08:03:11.269 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:11.269 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2024-07-26T08:03:11.270 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 0> 2024-07-26T08:03:10.780+0000 7f6997e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:11.270 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: in thread 7f6997e00640 thread_name:ms_dispatch 2024-07-26T08:03:11.270 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2024-07-26T08:03:11.270 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:11.270 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f699e03e6f0] 2024-07-26T08:03:11.270 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f699e08b94c] 2024-07-26T08:03:11.271 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 3: raise() 2024-07-26T08:03:11.271 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 4: abort() 2024-07-26T08:03:11.271 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 5: /lib64/libc.so.6(+0x2871b) [0x7f699e02871b] 2024-07-26T08:03:11.271 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 6: /lib64/libc.so.6(+0x37386) [0x7f699e037386] 2024-07-26T08:03:11.271 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cd33f416b9] 2024-07-26T08:03:11.271 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cd340ff80c] 2024-07-26T08:03:11.271 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cd3403f2ca] 2024-07-26T08:03:11.271 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cd3400f753] 2024-07-26T08:03:11.272 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cd34010a1b] 2024-07-26T08:03:11.272 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cd33ff4752] 2024-07-26T08:03:11.272 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cd33f42a56] 2024-07-26T08:03:11.272 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cd33f46637] 2024-07-26T08:03:11.272 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cd33f46f5e] 2024-07-26T08:03:11.273 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cd33f2a03a] 2024-07-26T08:03:11.273 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f699eb9b407] 2024-07-26T08:03:11.273 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 18: (DispatchQueue::entry()+0x60f) [0x7f699eb9bcf9] 2024-07-26T08:03:11.273 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f699ec785a1] 2024-07-26T08:03:11.273 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 20: (Thread::entry_wrapper()+0x43) [0x7f699ea25cdd] 2024-07-26T08:03:11.273 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 21: (Thread::_entry_func(void*)+0xd) [0x7f699ea25cf9] 2024-07-26T08:03:11.274 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 22: /lib64/libc.so.6(+0x89c02) [0x7f699e089c02] 2024-07-26T08:03:11.274 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f699e10ec40] 2024-07-26T08:03:11.274 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:11.274 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[75165]: 2024-07-26T08:03:11.274 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:10 smithi097 podman[194341]: 2024-07-26 08:03:10.931274859 +0000 UTC m=+0.033113949 container died e1efe33a3a8a0bd1a0f8b95ef23912bc23661ea65d8626d9ae2ab2be75708d82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:03:12.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:11 smithi097 podman[194341]: 2024-07-26 08:03:11.698902192 +0000 UTC m=+0.800741321 container remove e1efe33a3a8a0bd1a0f8b95ef23912bc23661ea65d8626d9ae2ab2be75708d82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T08:03:12.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:11 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:03:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:11 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:11 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:active 2024-07-26T08:03:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:11 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:03:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:11 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:11 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:11 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:active 2024-07-26T08:03:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:11 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:03:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:11 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:11 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:11 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1716973566,v1:172.21.15.97:6835/1716973566] up:active 2024-07-26T08:03:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:11 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:03:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:11 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:12.751 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:12 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:03:12.751 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:12 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.693s CPU time. 2024-07-26T08:03:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:12 smithi115 ceph-mon[35364]: pgmap v873: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:03:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:12 smithi181 ceph-mon[35869]: pgmap v873: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:03:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:12 smithi097 ceph-mon[32231]: pgmap v873: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:03:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:14 smithi115 ceph-mon[35364]: pgmap v874: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:03:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:14 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 232.600559 secs 2024-07-26T08:03:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:14 smithi181 ceph-mon[35869]: pgmap v874: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:03:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:14 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 232.600559 secs 2024-07-26T08:03:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:14 smithi097 ceph-mon[32231]: pgmap v874: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:03:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:14 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 232.600559 secs 2024-07-26T08:03:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:16 smithi115 ceph-mon[35364]: pgmap v875: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:16 smithi181 ceph-mon[35869]: pgmap v875: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:16 smithi097 ceph-mon[32231]: pgmap v875: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:17.434+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:03:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:17.435+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:03:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:17.435+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:03:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:17.435+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:03:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:17.436+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:03:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:18 smithi115 ceph-mon[35364]: pgmap v876: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 183 B/s wr, 11 op/s 2024-07-26T08:03:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:18 smithi181 ceph-mon[35869]: pgmap v876: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 183 B/s wr, 11 op/s 2024-07-26T08:03:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:18 smithi097 ceph-mon[32231]: pgmap v876: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 183 B/s wr, 11 op/s 2024-07-26T08:03:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:19 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 237.600714 secs 2024-07-26T08:03:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:19 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 237.600714 secs 2024-07-26T08:03:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:19 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 237.600714 secs 2024-07-26T08:03:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:20 smithi115 ceph-mon[35364]: pgmap v877: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:20 smithi181 ceph-mon[35869]: pgmap v877: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:20 smithi097 ceph-mon[32231]: pgmap v877: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:22.826 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:22 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-07-26T08:03:22.826 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:22 smithi097 systemd[1]: Stopped Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:03:22.826 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:22 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.693s CPU time. 2024-07-26T08:03:22.826 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:22 smithi097 systemd[1]: Starting Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:03:23.147 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:22 smithi097 ceph-mon[32231]: pgmap v878: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:23.147 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:03:23.147 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:22 smithi097 podman[194528]: 2024-07-26 08:03:22.816880043 +0000 UTC m=+0.039270608 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:03:23.147 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:22 smithi097 podman[194528]: 2024-07-26 08:03:22.966940865 +0000 UTC m=+0.189331411 container create 99740268e075369b648d9938df4f503e94169b54ec571751e7b7c442bbe67761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD) 2024-07-26T08:03:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:22 smithi115 ceph-mon[35364]: pgmap v878: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:03:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:22 smithi181 ceph-mon[35869]: pgmap v878: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:03:23.751 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:23 smithi097 podman[194528]: 2024-07-26 08:03:23.263561401 +0000 UTC m=+0.485952025 container init 99740268e075369b648d9938df4f503e94169b54ec571751e7b7c442bbe67761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-07-26T08:03:23.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:23 smithi097 podman[194528]: 2024-07-26 08:03:23.27537115 +0000 UTC m=+0.497761672 container start 99740268e075369b648d9938df4f503e94169b54ec571751e7b7c442bbe67761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-07-26T08:03:23.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:23 smithi097 bash[194528]: 99740268e075369b648d9938df4f503e94169b54ec571751e7b7c442bbe67761 2024-07-26T08:03:23.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:23 smithi097 systemd[1]: Started Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:03:23.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mds[194579]: -- 172.21.15.97:0/838213542 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b040ed1860 con 0x55b045073600 2024-07-26T08:03:23.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: starting mds.j at 2024-07-26T08:03:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: pgmap v879: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 89 B/s wr, 11 op/s 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: pgmap v880: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 12 op/s 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: daemon mds.j restarted 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:boot 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:03:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:23 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: pgmap v879: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 89 B/s wr, 11 op/s 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: pgmap v880: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 12 op/s 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: daemon mds.j restarted 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:boot 2024-07-26T08:03:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:03:24.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:24.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:03:24.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:23 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: pgmap v879: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 89 B/s wr, 11 op/s 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: pgmap v880: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 12 op/s 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: daemon mds.j restarted 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:boot 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:03:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:23 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:24 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 242.600854 secs 2024-07-26T08:03:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:24 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 242.600854 secs 2024-07-26T08:03:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:24 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 242.600854 secs 2024-07-26T08:03:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:25 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:resolve 2024-07-26T08:03:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:25 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:25 smithi115 ceph-mon[35364]: pgmap v882: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:03:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:25 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:resolve 2024-07-26T08:03:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:25 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:25 smithi181 ceph-mon[35869]: pgmap v882: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:03:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:25 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:resolve 2024-07-26T08:03:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:25 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:25 smithi097 ceph-mon[32231]: pgmap v882: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:03:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:26 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:03:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:26 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:reconnect 2024-07-26T08:03:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:26 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:26 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:03:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:26 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:reconnect 2024-07-26T08:03:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:26 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:26 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:03:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:26 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:reconnect 2024-07-26T08:03:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:26 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:27 smithi115 ceph-mon[35364]: pgmap v883: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-07-26T08:03:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:27 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:rejoin 2024-07-26T08:03:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:27 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:27 smithi115 ceph-mon[35364]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:03:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:27 smithi181 ceph-mon[35869]: pgmap v883: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-07-26T08:03:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:27 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:rejoin 2024-07-26T08:03:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:27 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:27 smithi181 ceph-mon[35869]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:03:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-mon[32231]: pgmap v883: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-07-26T08:03:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:rejoin 2024-07-26T08:03:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-mon[32231]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:03:28.253 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:03:28.253 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: *** Caught signal (Aborted) ** 2024-07-26T08:03:28.253 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: in thread 7f47a7800640 thread_name:ms_dispatch 2024-07-26T08:03:28.253 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:28.253 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47ada3e6f0] 2024-07-26T08:03:28.253 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f47ada8b94c] 2024-07-26T08:03:28.253 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 3: raise() 2024-07-26T08:03:28.253 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 4: abort() 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 5: /lib64/libc.so.6(+0x2871b) [0x7f47ada2871b] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 6: /lib64/libc.so.6(+0x37386) [0x7f47ada37386] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c9438656b9] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c943a2380c] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c9439632ca] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c943933753] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c943934a1b] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c943918752] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c943866a56] 2024-07-26T08:03:28.254 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c94386a637] 2024-07-26T08:03:28.255 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c94386af5e] 2024-07-26T08:03:28.255 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c94384e03a] 2024-07-26T08:03:28.255 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f47ae59b407] 2024-07-26T08:03:28.255 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 18: (DispatchQueue::entry()+0x60f) [0x7f47ae59bcf9] 2024-07-26T08:03:28.255 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f47ae6785a1] 2024-07-26T08:03:28.256 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 20: (Thread::entry_wrapper()+0x43) [0x7f47ae425cdd] 2024-07-26T08:03:28.256 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 21: (Thread::_entry_func(void*)+0xd) [0x7f47ae425cf9] 2024-07-26T08:03:28.256 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 22: /lib64/libc.so.6(+0x89c02) [0x7f47ada89c02] 2024-07-26T08:03:28.256 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f47adb0ec40] 2024-07-26T08:03:28.256 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2024-07-26T08:03:27.915+0000 7f47a7800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:28.256 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: in thread 7f47a7800640 thread_name:ms_dispatch 2024-07-26T08:03:28.256 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2024-07-26T08:03:28.256 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47ada3e6f0] 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f47ada8b94c] 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 3: raise() 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 4: abort() 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 5: /lib64/libc.so.6(+0x2871b) [0x7f47ada2871b] 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 6: /lib64/libc.so.6(+0x37386) [0x7f47ada37386] 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c9438656b9] 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c943a2380c] 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c9439632ca] 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c943933753] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c943934a1b] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c943918752] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c943866a56] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c94386a637] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c94386af5e] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c94384e03a] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f47ae59b407] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 18: (DispatchQueue::entry()+0x60f) [0x7f47ae59bcf9] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f47ae6785a1] 2024-07-26T08:03:28.258 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 20: (Thread::entry_wrapper()+0x43) [0x7f47ae425cdd] 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 21: (Thread::_entry_func(void*)+0xd) [0x7f47ae425cf9] 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 22: /lib64/libc.so.6(+0x89c02) [0x7f47ada89c02] 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f47adb0ec40] 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 0> 2024-07-26T08:03:27.915+0000 7f47a7800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: in thread 7f47a7800640 thread_name:ms_dispatch 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47ada3e6f0] 2024-07-26T08:03:28.259 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f47ada8b94c] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 3: raise() 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 4: abort() 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 5: /lib64/libc.so.6(+0x2871b) [0x7f47ada2871b] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 6: /lib64/libc.so.6(+0x37386) [0x7f47ada37386] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c9438656b9] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c943a2380c] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c9439632ca] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c943933753] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c943934a1b] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c943918752] 2024-07-26T08:03:28.260 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c943866a56] 2024-07-26T08:03:28.261 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c94386a637] 2024-07-26T08:03:28.261 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c94386af5e] 2024-07-26T08:03:28.261 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c94384e03a] 2024-07-26T08:03:28.262 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f47ae59b407] 2024-07-26T08:03:28.262 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 18: (DispatchQueue::entry()+0x60f) [0x7f47ae59bcf9] 2024-07-26T08:03:28.262 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f47ae6785a1] 2024-07-26T08:03:28.262 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 20: (Thread::entry_wrapper()+0x43) [0x7f47ae425cdd] 2024-07-26T08:03:28.262 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 21: (Thread::_entry_func(void*)+0xd) [0x7f47ae425cf9] 2024-07-26T08:03:28.262 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 22: /lib64/libc.so.6(+0x89c02) [0x7f47ada89c02] 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f47adb0ec40] 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 0> 2024-07-26T08:03:27.915+0000 7f47a7800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: in thread 7f47a7800640 thread_name:ms_dispatch 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47ada3e6f0] 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f47ada8b94c] 2024-07-26T08:03:28.263 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 3: raise() 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 4: abort() 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 5: /lib64/libc.so.6(+0x2871b) [0x7f47ada2871b] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 6: /lib64/libc.so.6(+0x37386) [0x7f47ada37386] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c9438656b9] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c943a2380c] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c9439632ca] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c943933753] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c943934a1b] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c943918752] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c943866a56] 2024-07-26T08:03:28.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c94386a637] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c94386af5e] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c94384e03a] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f47ae59b407] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 18: (DispatchQueue::entry()+0x60f) [0x7f47ae59bcf9] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f47ae6785a1] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 20: (Thread::entry_wrapper()+0x43) [0x7f47ae425cdd] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 21: (Thread::_entry_func(void*)+0xd) [0x7f47ae425cf9] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 22: /lib64/libc.so.6(+0x89c02) [0x7f47ada89c02] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f47adb0ec40] 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:28.265 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:27 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194303]: 2024-07-26T08:03:28.266 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:28 smithi097 podman[194613]: 2024-07-26 08:03:28.056228066 +0000 UTC m=+0.038301490 container died 18dc838ebf34177f33e59906dafe675b9e0971920e213091d27c737fb5542c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-07-26T08:03:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:28 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:28 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:active 2024-07-26T08:03:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:28 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:03:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:28 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:28 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:28 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:active 2024-07-26T08:03:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:28 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:03:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:28 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:28 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:28 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2004553229,v1:172.21.15.97:6837/2004553229] up:active 2024-07-26T08:03:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:28 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:03:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:28 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:29.251 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:28 smithi097 podman[194613]: 2024-07-26 08:03:28.784240898 +0000 UTC m=+0.766314364 container remove 18dc838ebf34177f33e59906dafe675b9e0971920e213091d27c737fb5542c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:03:29.251 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:28 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:03:29.751 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:29 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T08:03:29.751 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:29 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.253s CPU time. 2024-07-26T08:03:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:29 smithi115 ceph-mon[35364]: pgmap v884: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 138 B/s wr, 10 op/s 2024-07-26T08:03:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:29 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 247.601008 secs 2024-07-26T08:03:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:29 smithi181 ceph-mon[35869]: pgmap v884: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 138 B/s wr, 10 op/s 2024-07-26T08:03:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:29 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 247.601008 secs 2024-07-26T08:03:30.233 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:03:30 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:03:30.059+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:03:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:29 smithi097 ceph-mon[32231]: pgmap v884: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 138 B/s wr, 10 op/s 2024-07-26T08:03:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:29 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 247.601008 secs 2024-07-26T08:03:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:31 smithi115 ceph-mon[35364]: pgmap v885: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T08:03:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:31 smithi181 ceph-mon[35869]: pgmap v885: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T08:03:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:31 smithi097 ceph-mon[32231]: pgmap v885: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T08:03:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:32.440+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:03:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:32.441+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:03:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:32.441+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:03:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:32.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:03:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:32.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:03:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:33 smithi115 ceph-mon[35364]: pgmap v886: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:33 smithi181 ceph-mon[35869]: pgmap v886: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:33 smithi097 ceph-mon[32231]: pgmap v886: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:34 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 252.601098 secs 2024-07-26T08:03:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:34 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 252.601098 secs 2024-07-26T08:03:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:34 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 252.601098 secs 2024-07-26T08:03:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:35 smithi115 ceph-mon[35364]: pgmap v887: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:03:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:35 smithi181 ceph-mon[35869]: pgmap v887: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:03:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:35 smithi097 ceph-mon[32231]: pgmap v887: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:03:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:37 smithi115 ceph-mon[35364]: pgmap v888: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:37 smithi181 ceph-mon[35869]: pgmap v888: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:37 smithi097 ceph-mon[32231]: pgmap v888: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:39.910 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:39 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-07-26T08:03:39.910 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:39 smithi097 systemd[1]: Stopped Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:03:39.910 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:39 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.253s CPU time. 2024-07-26T08:03:39.910 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:39 smithi097 systemd[1]: Starting Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:03:40.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:39 smithi097 ceph-mon[32231]: pgmap v889: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:40.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:39 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 257.601275 secs 2024-07-26T08:03:40.163 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:39 smithi097 podman[194715]: 2024-07-26 08:03:39.813717094 +0000 UTC m=+0.037626480 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:03:40.163 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:39 smithi097 podman[194715]: 2024-07-26 08:03:39.985555344 +0000 UTC m=+0.209464758 container create 134da85c87e7d587a4c19d7e9e211629baafc47af6c366f43eee7587a676fecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0) 2024-07-26T08:03:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:39 smithi115 ceph-mon[35364]: pgmap v889: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:39 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 257.601275 secs 2024-07-26T08:03:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:39 smithi181 ceph-mon[35869]: pgmap v889: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:03:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:39 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 257.601275 secs 2024-07-26T08:03:40.751 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:40 smithi097 podman[194715]: 2024-07-26 08:03:40.273364888 +0000 UTC m=+0.497274315 container init 134da85c87e7d587a4c19d7e9e211629baafc47af6c366f43eee7587a676fecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:03:40.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:40 smithi097 podman[194715]: 2024-07-26 08:03:40.282391721 +0000 UTC m=+0.506301096 container start 134da85c87e7d587a4c19d7e9e211629baafc47af6c366f43eee7587a676fecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:03:40.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:40 smithi097 bash[194715]: 134da85c87e7d587a4c19d7e9e211629baafc47af6c366f43eee7587a676fecb 2024-07-26T08:03:40.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:40 smithi097 systemd[1]: Started Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:03:40.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mds[194734]: -- 172.21.15.97:0/3721558596 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce0663d860 con 0x55ce0a7df600 2024-07-26T08:03:40.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: starting mds.d at 2024-07-26T08:03:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:40 smithi115 ceph-mon[35364]: daemon mds.d restarted 2024-07-26T08:03:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:40 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:40 smithi115 ceph-mon[35364]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T08:03:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:40 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:boot 2024-07-26T08:03:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:40 smithi115 ceph-mon[35364]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:03:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:40 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:03:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:40 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:40 smithi181 ceph-mon[35869]: daemon mds.d restarted 2024-07-26T08:03:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:40 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:40 smithi181 ceph-mon[35869]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T08:03:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:40 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:boot 2024-07-26T08:03:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:40 smithi181 ceph-mon[35869]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:03:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:40 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:03:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:40 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mon[32231]: daemon mds.d restarted 2024-07-26T08:03:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mon[32231]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T08:03:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:boot 2024-07-26T08:03:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mon[32231]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:03:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:03:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:40 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:41 smithi115 ceph-mon[35364]: pgmap v891: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:03:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:41 smithi181 ceph-mon[35869]: pgmap v891: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:03:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:41 smithi097 ceph-mon[32231]: pgmap v891: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:03:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:42 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:resolve 2024-07-26T08:03:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:42 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:42 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:resolve 2024-07-26T08:03:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:42 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:42 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:resolve 2024-07-26T08:03:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:42 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:43 smithi097 ceph-mon[32231]: pgmap v892: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:03:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:43 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:03:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:43 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:reconnect 2024-07-26T08:03:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:43 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: *** Caught signal (Aborted) ** 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: in thread 7f3eb3800640 thread_name:ms_dispatch 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3eb9a3e6f0] 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3eb9a8b94c] 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 3: raise() 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 4: abort() 2024-07-26T08:03:44.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3eb9a2871b] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 6: /lib64/libc.so.6(+0x37386) [0x7f3eb9a37386] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b02315e6b9] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b02331c80c] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b02325c2ca] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b02322c753] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b02322da1b] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b023211752] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b02315fa56] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b023163637] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b023163f5e] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b02314703a] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3eba59b407] 2024-07-26T08:03:44.253 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 18: (DispatchQueue::entry()+0x60f) [0x7f3eba59bcf9] 2024-07-26T08:03:44.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3eba6785a1] 2024-07-26T08:03:44.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 20: (Thread::entry_wrapper()+0x43) [0x7f3eba425cdd] 2024-07-26T08:03:44.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3eba425cf9] 2024-07-26T08:03:44.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3eb9a89c02] 2024-07-26T08:03:44.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3eb9b0ec40] 2024-07-26T08:03:44.254 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2024-07-26T08:03:44.153+0000 7f3eb3800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: in thread 7f3eb3800640 thread_name:ms_dispatch 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3eb9a3e6f0] 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3eb9a8b94c] 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 3: raise() 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 4: abort() 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3eb9a2871b] 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 6: /lib64/libc.so.6(+0x37386) [0x7f3eb9a37386] 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b02315e6b9] 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b02331c80c] 2024-07-26T08:03:44.255 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b02325c2ca] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b02322c753] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b02322da1b] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b023211752] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b02315fa56] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b023163637] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b023163f5e] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b02314703a] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3eba59b407] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 18: (DispatchQueue::entry()+0x60f) [0x7f3eba59bcf9] 2024-07-26T08:03:44.256 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3eba6785a1] 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 20: (Thread::entry_wrapper()+0x43) [0x7f3eba425cdd] 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3eba425cf9] 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3eb9a89c02] 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3eb9b0ec40] 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 0> 2024-07-26T08:03:44.153+0000 7f3eb3800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: in thread 7f3eb3800640 thread_name:ms_dispatch 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2024-07-26T08:03:44.257 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3eb9a3e6f0] 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3eb9a8b94c] 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 3: raise() 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 4: abort() 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3eb9a2871b] 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 6: /lib64/libc.so.6(+0x37386) [0x7f3eb9a37386] 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b02315e6b9] 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b02331c80c] 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b02325c2ca] 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b02322c753] 2024-07-26T08:03:44.258 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b02322da1b] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b023211752] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b02315fa56] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b023163637] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b023163f5e] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b02314703a] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3eba59b407] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 18: (DispatchQueue::entry()+0x60f) [0x7f3eba59bcf9] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3eba6785a1] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 20: (Thread::entry_wrapper()+0x43) [0x7f3eba425cdd] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3eba425cf9] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3eb9a89c02] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3eb9b0ec40] 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:44.259 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: -2> 2024-07-26T08:03:44.153+0000 7f3eb3800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: in thread 7f3eb3800640 thread_name:ms_dispatch 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3eb9a3e6f0] 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3eb9a8b94c] 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 3: raise() 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 4: abort() 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3eb9a2871b] 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 6: /lib64/libc.so.6(+0x37386) [0x7f3eb9a37386] 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b02315e6b9] 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b02331c80c] 2024-07-26T08:03:44.260 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b02325c2ca] 2024-07-26T08:03:44.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b02322c753] 2024-07-26T08:03:44.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b02322da1b] 2024-07-26T08:03:44.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b023211752] 2024-07-26T08:03:44.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b02315fa56] 2024-07-26T08:03:44.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b023163637] 2024-07-26T08:03:44.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b023163f5e] 2024-07-26T08:03:44.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b02314703a] 2024-07-26T08:03:44.261 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3eba59b407] 2024-07-26T08:03:44.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 18: (DispatchQueue::entry()+0x60f) [0x7f3eba59bcf9] 2024-07-26T08:03:44.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3eba6785a1] 2024-07-26T08:03:44.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 20: (Thread::entry_wrapper()+0x43) [0x7f3eba425cdd] 2024-07-26T08:03:44.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3eba425cf9] 2024-07-26T08:03:44.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3eb9a89c02] 2024-07-26T08:03:44.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3eb9b0ec40] 2024-07-26T08:03:44.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:03:44.262 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194574]: 2024-07-26T08:03:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:43 smithi115 ceph-mon[35364]: pgmap v892: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:03:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:43 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:03:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:43 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:reconnect 2024-07-26T08:03:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:43 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:43 smithi181 ceph-mon[35869]: pgmap v892: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:03:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:43 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:03:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:43 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:reconnect 2024-07-26T08:03:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:43 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:03:44.631 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 podman[194769]: 2024-07-26 08:03:44.316918437 +0000 UTC m=+0.052126028 container died 99740268e075369b648d9938df4f503e94169b54ec571751e7b7c442bbe67761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T08:03:45.251 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 podman[194769]: 2024-07-26 08:03:44.975867958 +0000 UTC m=+0.711075537 container remove 99740268e075369b648d9938df4f503e94169b54ec571751e7b7c442bbe67761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux ) 2024-07-26T08:03:45.251 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:44 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:03:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:45 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:rejoin 2024-07-26T08:03:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:45 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:45 smithi097 ceph-mon[32231]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:03:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:45 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:45 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:active 2024-07-26T08:03:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:45 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 262.601416 secs 2024-07-26T08:03:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:45 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:rejoin 2024-07-26T08:03:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:45 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:45 smithi115 ceph-mon[35364]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:03:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:45 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:45 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:active 2024-07-26T08:03:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:45 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 262.601416 secs 2024-07-26T08:03:45.486 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:03:45 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:03:45.060+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:03:45.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:45 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:rejoin 2024-07-26T08:03:45.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:45 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:03:45.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:45 smithi181 ceph-mon[35869]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:03:45.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:45 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:03:45.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:45 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4064489700,v1:172.21.15.97:6835/4064489700] up:active 2024-07-26T08:03:45.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:45 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:03:45.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 262.601416 secs 2024-07-26T08:03:46.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:45 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:03:46.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:45 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.155s CPU time. 2024-07-26T08:03:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:46 smithi115 ceph-mon[35364]: pgmap v893: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:03:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:46 smithi181 ceph-mon[35869]: pgmap v893: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:03:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:46 smithi097 ceph-mon[32231]: pgmap v893: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:03:47.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:47.449+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:03:47.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:47.451+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:03:47.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:47.451+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:03:47.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:47.451+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:03:47.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:03:47.451+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:03:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:48 smithi115 ceph-mon[35364]: pgmap v894: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:03:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:48 smithi181 ceph-mon[35869]: pgmap v894: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:03:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:48 smithi097 ceph-mon[32231]: pgmap v894: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:03:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:50 smithi115 ceph-mon[35364]: pgmap v895: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 267.601564 secs 2024-07-26T08:03:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:50 smithi181 ceph-mon[35869]: pgmap v895: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 267.601564 secs 2024-07-26T08:03:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:50 smithi097 ceph-mon[32231]: pgmap v895: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:03:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 267.601564 secs 2024-07-26T08:03:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:52 smithi115 ceph-mon[35364]: pgmap v896: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 195 B/s wr, 11 op/s 2024-07-26T08:03:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:52 smithi181 ceph-mon[35869]: pgmap v896: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 195 B/s wr, 11 op/s 2024-07-26T08:03:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:52 smithi097 ceph-mon[32231]: pgmap v896: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 195 B/s wr, 11 op/s 2024-07-26T08:03:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:54 smithi115 ceph-mon[35364]: pgmap v897: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:54 smithi181 ceph-mon[35869]: pgmap v897: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:54 smithi097 ceph-mon[32231]: pgmap v897: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 272.601681 secs 2024-07-26T08:03:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 272.601681 secs 2024-07-26T08:03:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 272.601681 secs 2024-07-26T08:03:56.160 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:55 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-07-26T08:03:56.160 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:55 smithi097 systemd[1]: Stopped Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:03:56.160 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:55 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.155s CPU time. 2024-07-26T08:03:56.160 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:55 smithi097 systemd[1]: Starting Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:03:56.448 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:56 smithi097 ceph-mon[32231]: pgmap v898: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:56.449 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:56 smithi097 podman[194859]: 2024-07-26 08:03:56.063755589 +0000 UTC m=+0.036332185 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:03:56.449 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:56 smithi097 podman[194859]: 2024-07-26 08:03:56.18136951 +0000 UTC m=+0.153946121 container create 14fc6ffafc694b357d9907cd4a5099abc8afb7093fba2437fa98bdbe24e2e2c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-07-26T08:03:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:56 smithi115 ceph-mon[35364]: pgmap v898: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:56 smithi181 ceph-mon[35869]: pgmap v898: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:03:56.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:56 smithi097 podman[194859]: 2024-07-26 08:03:56.447983722 +0000 UTC m=+0.420560339 container init 14fc6ffafc694b357d9907cd4a5099abc8afb7093fba2437fa98bdbe24e2e2c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:03:56.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:56 smithi097 podman[194859]: 2024-07-26 08:03:56.458318424 +0000 UTC m=+0.430895008 container start 14fc6ffafc694b357d9907cd4a5099abc8afb7093fba2437fa98bdbe24e2e2c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724) 2024-07-26T08:03:56.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:56 smithi097 bash[194859]: 14fc6ffafc694b357d9907cd4a5099abc8afb7093fba2437fa98bdbe24e2e2c2 2024-07-26T08:03:56.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:56 smithi097 systemd[1]: Started Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:03:56.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:56 smithi097 ceph-mds[194877]: -- 172.21.15.97:0/292182020 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed2d91d860 con 0x55ed31abfa80 2024-07-26T08:03:56.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:03:56 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: starting mds.j at 2024-07-26T08:03:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:57 smithi115 ceph-mon[35364]: daemon mds.j restarted 2024-07-26T08:03:57.522 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:57 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:57.522 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:57 smithi115 ceph-mon[35364]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T08:03:57.523 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:57 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:boot 2024-07-26T08:03:57.523 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:57 smithi115 ceph-mon[35364]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:03:57.523 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:57 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:57.523 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:03:57.523 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:57 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:57.524 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:57 smithi097 ceph-mon[32231]: daemon mds.j restarted 2024-07-26T08:03:57.524 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:57 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:57.524 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:57 smithi097 ceph-mon[32231]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T08:03:57.524 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:57 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:boot 2024-07-26T08:03:57.524 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:57 smithi097 ceph-mon[32231]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:03:57.524 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:57 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:57.524 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:03:57.524 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:57 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:57.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:57 smithi181 ceph-mon[35869]: daemon mds.j restarted 2024-07-26T08:03:57.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:57 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:03:57.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:57 smithi181 ceph-mon[35869]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T08:03:57.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:57 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:boot 2024-07-26T08:03:57.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:57 smithi181 ceph-mon[35869]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:03:57.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:57 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:03:57.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:03:57.525 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:57 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:03:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:58 smithi115 ceph-mon[35364]: pgmap v900: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:03:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:58 smithi181 ceph-mon[35869]: pgmap v900: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:03:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:58 smithi097 ceph-mon[32231]: pgmap v900: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:03:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:59 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:resolve 2024-07-26T08:03:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:03:59 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:59 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:resolve 2024-07-26T08:03:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:03:59 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:03:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:59 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:resolve 2024-07-26T08:03:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:03:59 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:00 smithi097 ceph-mon[32231]: pgmap v901: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:04:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:00 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:00 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:reconnect 2024-07-26T08:04:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:00 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 277.601820 secs 2024-07-26T08:04:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:00 smithi115 ceph-mon[35364]: pgmap v901: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:04:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:00 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:00 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:reconnect 2024-07-26T08:04:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:00 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 277.601820 secs 2024-07-26T08:04:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:00 smithi181 ceph-mon[35869]: pgmap v901: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:04:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:00 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:00 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:reconnect 2024-07-26T08:04:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:00 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 277.601820 secs 2024-07-26T08:04:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:01 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:rejoin 2024-07-26T08:04:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:01 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:01 smithi115 ceph-mon[35364]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:04:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:01 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:rejoin 2024-07-26T08:04:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:01 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:01 smithi181 ceph-mon[35869]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:04:01.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:rejoin 2024-07-26T08:04:01.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:01.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-mon[32231]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:04:01.751 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: *** Caught signal (Aborted) ** 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: in thread 7f8eb9c00640 thread_name:ms_dispatch 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8ebfe3e6f0] 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8ebfe8b94c] 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 3: raise() 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 4: abort() 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8ebfe2871b] 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 6: /lib64/libc.so.6(+0x37386) [0x7f8ebfe37386] 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cdf9c576b9] 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cdf9e1580c] 2024-07-26T08:04:01.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cdf9d552ca] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cdf9d25753] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cdf9d26a1b] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cdf9d0a752] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cdf9c58a56] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cdf9c5c637] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cdf9c5cf5e] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cdf9c4003a] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8ec099b407] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 18: (DispatchQueue::entry()+0x60f) [0x7f8ec099bcf9] 2024-07-26T08:04:01.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8ec0a785a1] 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 20: (Thread::entry_wrapper()+0x43) [0x7f8ec0825cdd] 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8ec0825cf9] 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8ebfe89c02] 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8ebff0ec40] 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2024-07-26T08:04:01.273+0000 7f8eb9c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: in thread 7f8eb9c00640 thread_name:ms_dispatch 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8ebfe3e6f0] 2024-07-26T08:04:01.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8ebfe8b94c] 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 3: raise() 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 4: abort() 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8ebfe2871b] 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 6: /lib64/libc.so.6(+0x37386) [0x7f8ebfe37386] 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cdf9c576b9] 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cdf9e1580c] 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cdf9d552ca] 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cdf9d25753] 2024-07-26T08:04:01.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cdf9d26a1b] 2024-07-26T08:04:01.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cdf9d0a752] 2024-07-26T08:04:01.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cdf9c58a56] 2024-07-26T08:04:01.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cdf9c5c637] 2024-07-26T08:04:01.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cdf9c5cf5e] 2024-07-26T08:04:01.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cdf9c4003a] 2024-07-26T08:04:01.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8ec099b407] 2024-07-26T08:04:01.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 18: (DispatchQueue::entry()+0x60f) [0x7f8ec099bcf9] 2024-07-26T08:04:01.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8ec0a785a1] 2024-07-26T08:04:01.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 20: (Thread::entry_wrapper()+0x43) [0x7f8ec0825cdd] 2024-07-26T08:04:01.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8ec0825cf9] 2024-07-26T08:04:01.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8ebfe89c02] 2024-07-26T08:04:01.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8ebff0ec40] 2024-07-26T08:04:01.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:01.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2024-07-26T08:04:01.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 0> 2024-07-26T08:04:01.273+0000 7f8eb9c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:01.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: in thread 7f8eb9c00640 thread_name:ms_dispatch 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8ebfe3e6f0] 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8ebfe8b94c] 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 3: raise() 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 4: abort() 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8ebfe2871b] 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 6: /lib64/libc.so.6(+0x37386) [0x7f8ebfe37386] 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cdf9c576b9] 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cdf9e1580c] 2024-07-26T08:04:01.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cdf9d552ca] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cdf9d25753] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cdf9d26a1b] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cdf9d0a752] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cdf9c58a56] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cdf9c5c637] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cdf9c5cf5e] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cdf9c4003a] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8ec099b407] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 18: (DispatchQueue::entry()+0x60f) [0x7f8ec099bcf9] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8ec0a785a1] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 20: (Thread::entry_wrapper()+0x43) [0x7f8ec0825cdd] 2024-07-26T08:04:01.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8ec0825cf9] 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8ebfe89c02] 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8ebff0ec40] 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: -2> 2024-07-26T08:04:01.273+0000 7f8eb9c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: in thread 7f8eb9c00640 thread_name:ms_dispatch 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8ebfe3e6f0] 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8ebfe8b94c] 2024-07-26T08:04:01.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 3: raise() 2024-07-26T08:04:01.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 4: abort() 2024-07-26T08:04:01.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8ebfe2871b] 2024-07-26T08:04:01.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 6: /lib64/libc.so.6(+0x37386) [0x7f8ebfe37386] 2024-07-26T08:04:01.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cdf9c576b9] 2024-07-26T08:04:01.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cdf9e1580c] 2024-07-26T08:04:01.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cdf9d552ca] 2024-07-26T08:04:01.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cdf9d25753] 2024-07-26T08:04:01.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cdf9d26a1b] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cdf9d0a752] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cdf9c58a56] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cdf9c5c637] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cdf9c5cf5e] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cdf9c4003a] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8ec099b407] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 18: (DispatchQueue::entry()+0x60f) [0x7f8ec099bcf9] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8ec0a785a1] 2024-07-26T08:04:01.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 20: (Thread::entry_wrapper()+0x43) [0x7f8ec0825cdd] 2024-07-26T08:04:01.763 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8ec0825cf9] 2024-07-26T08:04:01.763 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8ebfe89c02] 2024-07-26T08:04:01.763 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8ebff0ec40] 2024-07-26T08:04:01.763 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:01.763 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[194730]: 2024-07-26T08:04:01.763 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:01 smithi097 podman[194911]: 2024-07-26 08:04:01.410052552 +0000 UTC m=+0.032409177 container died 134da85c87e7d587a4c19d7e9e211629baafc47af6c366f43eee7587a676fecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-07-26T08:04:02.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:02 smithi097 podman[194911]: 2024-07-26 08:04:02.093749735 +0000 UTC m=+0.716106399 container remove 134da85c87e7d587a4c19d7e9e211629baafc47af6c366f43eee7587a676fecb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:04:02.264 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:02 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:04:02.455 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:02.454+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:04:02.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:02.455+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:04:02.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:02.455+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:04:02.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:02.455+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:04:02.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:02.455+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:04:02.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:02 smithi097 ceph-mon[32231]: pgmap v902: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:02.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:02 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:02.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:02 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:active 2024-07-26T08:04:02.614 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:02 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:02.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:02 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:02 smithi115 ceph-mon[35364]: pgmap v902: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:02 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:02 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:active 2024-07-26T08:04:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:02 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:02 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:02 smithi181 ceph-mon[35869]: pgmap v902: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:02 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:02 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/1951182899,v1:172.21.15.97:6837/1951182899] up:active 2024-07-26T08:04:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:02 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:02 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:03.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:02 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T08:04:03.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:02 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.170s CPU time. 2024-07-26T08:04:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:04 smithi115 ceph-mon[35364]: pgmap v903: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:04 smithi181 ceph-mon[35869]: pgmap v903: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:04 smithi097 ceph-mon[32231]: pgmap v903: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:05.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 282.601975 secs 2024-07-26T08:04:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 282.601975 secs 2024-07-26T08:04:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 282.601975 secs 2024-07-26T08:04:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:06 smithi115 ceph-mon[35364]: pgmap v904: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:06 smithi181 ceph-mon[35869]: pgmap v904: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:06.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:06 smithi097 ceph-mon[32231]: pgmap v904: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:06.926+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0005850 con 0x7ff3f01061a0 2024-07-26T08:04:06.928 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:06.927+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0005850 con 0x7ff3f01061a0 2024-07-26T08:04:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:06.929+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc005850 con 0x55d25070b2f0 2024-07-26T08:04:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:06.930+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc005850 con 0x55d25070b2f0 2024-07-26T08:04:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:08 smithi115 ceph-mon[35364]: pgmap v905: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 198 B/s wr, 11 op/s 2024-07-26T08:04:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:08 smithi181 ceph-mon[35869]: pgmap v905: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 198 B/s wr, 11 op/s 2024-07-26T08:04:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:08 smithi097 ceph-mon[32231]: pgmap v905: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 198 B/s wr, 11 op/s 2024-07-26T08:04:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:09 smithi115 ceph-mon[35364]: pgmap v906: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:09 smithi181 ceph-mon[35869]: pgmap v906: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:09 smithi097 ceph-mon[32231]: pgmap v906: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:10.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 287.602071 secs 2024-07-26T08:04:10.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 287.602071 secs 2024-07-26T08:04:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 287.602071 secs 2024-07-26T08:04:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:11 smithi115 ceph-mon[35364]: pgmap v907: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:11 smithi181 ceph-mon[35869]: pgmap v907: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:11 smithi097 ceph-mon[32231]: pgmap v907: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:13.152 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:12 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-07-26T08:04:13.152 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:12 smithi097 systemd[1]: Stopped Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:04:13.153 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:12 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.170s CPU time. 2024-07-26T08:04:13.153 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:12 smithi097 systemd[1]: Starting Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:04:13.433 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:13 smithi097 podman[195003]: 2024-07-26 08:04:13.056664567 +0000 UTC m=+0.032350903 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:04:13.434 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:13 smithi097 podman[195003]: 2024-07-26 08:04:13.16172803 +0000 UTC m=+0.137414387 container create 3761a40295bfb8e579305691e603c05464ed9010349ebbf97fbf717a661b3ac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-26T08:04:13.434 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:13 smithi097 podman[195003]: 2024-07-26 08:04:13.432563024 +0000 UTC m=+0.408249393 container init 3761a40295bfb8e579305691e603c05464ed9010349ebbf97fbf717a661b3ac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-07-26T08:04:13.751 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:13 smithi097 podman[195003]: 2024-07-26 08:04:13.442329544 +0000 UTC m=+0.418015877 container start 3761a40295bfb8e579305691e603c05464ed9010349ebbf97fbf717a661b3ac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-07-26T08:04:13.751 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:13 smithi097 bash[195003]: 3761a40295bfb8e579305691e603c05464ed9010349ebbf97fbf717a661b3ac0 2024-07-26T08:04:13.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:13 smithi097 systemd[1]: Started Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:04:13.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mds[195021]: -- 172.21.15.97:0/3967904949 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56188a9f1860 con 0x56188eb93180 2024-07-26T08:04:13.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: starting mds.d at 2024-07-26T08:04:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: pgmap v908: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: daemon mds.d restarted 2024-07-26T08:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T08:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:boot 2024-07-26T08:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:13 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: pgmap v908: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: daemon mds.d restarted 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:boot 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:13 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: pgmap v908: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:04:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: daemon mds.d restarted 2024-07-26T08:04:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T08:04:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:boot 2024-07-26T08:04:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:04:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:04:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:13 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 292.602217 secs 2024-07-26T08:04:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 292.602217 secs 2024-07-26T08:04:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 292.602217 secs 2024-07-26T08:04:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:16 smithi115 ceph-mon[35364]: pgmap v910: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 10 op/s 2024-07-26T08:04:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:16 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:resolve 2024-07-26T08:04:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:16 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:16 smithi181 ceph-mon[35869]: pgmap v910: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 10 op/s 2024-07-26T08:04:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:16 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:resolve 2024-07-26T08:04:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:16 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:16 smithi097 ceph-mon[32231]: pgmap v910: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 10 op/s 2024-07-26T08:04:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:16 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:resolve 2024-07-26T08:04:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:16 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:17.464+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:04:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:17.465+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:04:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:17.465+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:04:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:17.465+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:04:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:17.465+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:04:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:17 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:reconnect 2024-07-26T08:04:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:17 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:04:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:17 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:17 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:reconnect 2024-07-26T08:04:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:17 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:04:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:17 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:17 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:reconnect 2024-07-26T08:04:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:17 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:04:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:17 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:18 smithi115 ceph-mon[35364]: pgmap v911: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:04:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:18 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:rejoin 2024-07-26T08:04:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:18 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:18 smithi115 ceph-mon[35364]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:18 smithi181 ceph-mon[35869]: pgmap v911: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:18 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:rejoin 2024-07-26T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:18 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:18 smithi181 ceph-mon[35869]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:04:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-mon[32231]: pgmap v911: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:04:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:rejoin 2024-07-26T08:04:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-mon[32231]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:04:18.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:04:18.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: *** Caught signal (Aborted) ** 2024-07-26T08:04:18.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: in thread 7f76ac000640 thread_name:ms_dispatch 2024-07-26T08:04:18.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:18.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f76b223e6f0] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f76b228b94c] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 3: raise() 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 4: abort() 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 5: /lib64/libc.so.6(+0x2871b) [0x7f76b222871b] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 6: /lib64/libc.so.6(+0x37386) [0x7f76b2237386] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ecf64956b9] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ecf665380c] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ecf65932ca] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ecf6563753] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ecf6564a1b] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ecf6548752] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ecf6496a56] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ecf649a637] 2024-07-26T08:04:18.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ecf649af5e] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ecf647e03a] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f76b2d9b407] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 18: (DispatchQueue::entry()+0x60f) [0x7f76b2d9bcf9] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f76b2e785a1] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 20: (Thread::entry_wrapper()+0x43) [0x7f76b2c25cdd] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 21: (Thread::_entry_func(void*)+0xd) [0x7f76b2c25cf9] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 22: /lib64/libc.so.6(+0x89c02) [0x7f76b2289c02] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f76b230ec40] 2024-07-26T08:04:18.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2024-07-26T08:04:18.245+0000 7f76ac000640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: in thread 7f76ac000640 thread_name:ms_dispatch 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f76b223e6f0] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f76b228b94c] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 3: raise() 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 4: abort() 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 5: /lib64/libc.so.6(+0x2871b) [0x7f76b222871b] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 6: /lib64/libc.so.6(+0x37386) [0x7f76b2237386] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ecf64956b9] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ecf665380c] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ecf65932ca] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ecf6563753] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ecf6564a1b] 2024-07-26T08:04:18.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ecf6548752] 2024-07-26T08:04:18.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ecf6496a56] 2024-07-26T08:04:18.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ecf649a637] 2024-07-26T08:04:18.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ecf649af5e] 2024-07-26T08:04:18.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ecf647e03a] 2024-07-26T08:04:18.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f76b2d9b407] 2024-07-26T08:04:18.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 18: (DispatchQueue::entry()+0x60f) [0x7f76b2d9bcf9] 2024-07-26T08:04:18.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f76b2e785a1] 2024-07-26T08:04:18.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 20: (Thread::entry_wrapper()+0x43) [0x7f76b2c25cdd] 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 21: (Thread::_entry_func(void*)+0xd) [0x7f76b2c25cf9] 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 22: /lib64/libc.so.6(+0x89c02) [0x7f76b2289c02] 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f76b230ec40] 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 0> 2024-07-26T08:04:18.245+0000 7f76ac000640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: in thread 7f76ac000640 thread_name:ms_dispatch 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f76b223e6f0] 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f76b228b94c] 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 3: raise() 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 4: abort() 2024-07-26T08:04:18.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 5: /lib64/libc.so.6(+0x2871b) [0x7f76b222871b] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 6: /lib64/libc.so.6(+0x37386) [0x7f76b2237386] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ecf64956b9] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ecf665380c] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ecf65932ca] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ecf6563753] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ecf6564a1b] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ecf6548752] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ecf6496a56] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ecf649a637] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ecf649af5e] 2024-07-26T08:04:18.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ecf647e03a] 2024-07-26T08:04:18.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f76b2d9b407] 2024-07-26T08:04:18.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 18: (DispatchQueue::entry()+0x60f) [0x7f76b2d9bcf9] 2024-07-26T08:04:18.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f76b2e785a1] 2024-07-26T08:04:18.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 20: (Thread::entry_wrapper()+0x43) [0x7f76b2c25cdd] 2024-07-26T08:04:18.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 21: (Thread::_entry_func(void*)+0xd) [0x7f76b2c25cf9] 2024-07-26T08:04:18.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 22: /lib64/libc.so.6(+0x89c02) [0x7f76b2289c02] 2024-07-26T08:04:18.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f76b230ec40] 2024-07-26T08:04:18.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 0> 2024-07-26T08:04:18.245+0000 7f76ac000640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: in thread 7f76ac000640 thread_name:ms_dispatch 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f76b223e6f0] 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f76b228b94c] 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 3: raise() 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 4: abort() 2024-07-26T08:04:18.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 5: /lib64/libc.so.6(+0x2871b) [0x7f76b222871b] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 6: /lib64/libc.so.6(+0x37386) [0x7f76b2237386] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ecf64956b9] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ecf665380c] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ecf65932ca] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ecf6563753] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ecf6564a1b] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ecf6548752] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ecf6496a56] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ecf649a637] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ecf649af5e] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ecf647e03a] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f76b2d9b407] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 18: (DispatchQueue::entry()+0x60f) [0x7f76b2d9bcf9] 2024-07-26T08:04:18.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f76b2e785a1] 2024-07-26T08:04:18.512 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 20: (Thread::entry_wrapper()+0x43) [0x7f76b2c25cdd] 2024-07-26T08:04:18.512 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 21: (Thread::_entry_func(void*)+0xd) [0x7f76b2c25cf9] 2024-07-26T08:04:18.512 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 22: /lib64/libc.so.6(+0x89c02) [0x7f76b2289c02] 2024-07-26T08:04:18.512 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f76b230ec40] 2024-07-26T08:04:18.512 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:18.512 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[194873]: 2024-07-26T08:04:18.512 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:18 smithi097 podman[195055]: 2024-07-26 08:04:18.386957117 +0000 UTC m=+0.037371480 container died 14fc6ffafc694b357d9907cd4a5099abc8afb7093fba2437fa98bdbe24e2e2c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:04:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:19 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:19 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:active 2024-07-26T08:04:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:19 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:19 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:19 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:19 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:active 2024-07-26T08:04:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:19 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:19 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:19.501 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:19 smithi097 podman[195055]: 2024-07-26 08:04:19.044421426 +0000 UTC m=+0.694835808 container remove 14fc6ffafc694b357d9907cd4a5099abc8afb7093fba2437fa98bdbe24e2e2c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-26T08:04:19.501 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:19 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:04:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:19 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:19 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/3304604647,v1:172.21.15.97:6835/3304604647] up:active 2024-07-26T08:04:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:19 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:19 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:20.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:19 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:04:20.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:19 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.070s CPU time. 2024-07-26T08:04:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:20 smithi181 ceph-mon[35869]: pgmap v912: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 297.602352 secs 2024-07-26T08:04:20.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:04:20 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:04:20.065+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:04:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:20 smithi115 ceph-mon[35364]: pgmap v912: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 297.602352 secs 2024-07-26T08:04:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:20 smithi097 ceph-mon[32231]: pgmap v912: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 297.602352 secs 2024-07-26T08:04:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:22 smithi115 ceph-mon[35364]: pgmap v913: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:22 smithi181 ceph-mon[35869]: pgmap v913: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:22 smithi097 ceph-mon[32231]: pgmap v913: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:04:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:04:23.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:04:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:24 smithi115 ceph-mon[35364]: pgmap v914: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:24 smithi181 ceph-mon[35869]: pgmap v914: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:24 smithi097 ceph-mon[32231]: pgmap v914: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:04:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:04:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:04:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 302.602433 secs 2024-07-26T08:04:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:04:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:04:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:04:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 302.602433 secs 2024-07-26T08:04:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:04:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:04:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:04:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 302.602433 secs 2024-07-26T08:04:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:26 smithi115 ceph-mon[35364]: pgmap v915: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 190 B/s wr, 12 op/s 2024-07-26T08:04:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:26 smithi115 ceph-mon[35364]: pgmap v916: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 11 op/s 2024-07-26T08:04:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:26 smithi181 ceph-mon[35869]: pgmap v915: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 190 B/s wr, 12 op/s 2024-07-26T08:04:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:26 smithi181 ceph-mon[35869]: pgmap v916: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 11 op/s 2024-07-26T08:04:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:26 smithi097 ceph-mon[32231]: pgmap v915: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 190 B/s wr, 12 op/s 2024-07-26T08:04:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:26 smithi097 ceph-mon[32231]: pgmap v916: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 11 op/s 2024-07-26T08:04:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:28 smithi115 ceph-mon[35364]: pgmap v917: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 11 op/s 2024-07-26T08:04:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:28 smithi181 ceph-mon[35869]: pgmap v917: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 11 op/s 2024-07-26T08:04:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:28 smithi097 ceph-mon[32231]: pgmap v917: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 11 op/s 2024-07-26T08:04:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:29 smithi115 ceph-mon[35364]: pgmap v918: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 108 B/s wr, 10 op/s 2024-07-26T08:04:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:29 smithi181 ceph-mon[35869]: pgmap v918: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 108 B/s wr, 10 op/s 2024-07-26T08:04:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:29 smithi097 ceph-mon[32231]: pgmap v918: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 108 B/s wr, 10 op/s 2024-07-26T08:04:30.157 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:29 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-07-26T08:04:30.157 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:29 smithi097 systemd[1]: Stopped Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:04:30.157 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:29 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.070s CPU time. 2024-07-26T08:04:30.157 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:29 smithi097 systemd[1]: Starting Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:04:30.459 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 307.602536 secs 2024-07-26T08:04:30.459 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:30 smithi097 podman[195278]: 2024-07-26 08:04:30.060594986 +0000 UTC m=+0.034657642 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:04:30.459 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:30 smithi097 podman[195278]: 2024-07-26 08:04:30.204684122 +0000 UTC m=+0.178746804 container create 11a973c277f366f21fe422f27c7989e4189fb7a9b74784aa3677c3005a193c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-07-26T08:04:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 307.602536 secs 2024-07-26T08:04:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 307.602536 secs 2024-07-26T08:04:30.751 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:30 smithi097 podman[195278]: 2024-07-26 08:04:30.458451035 +0000 UTC m=+0.432513720 container init 11a973c277f366f21fe422f27c7989e4189fb7a9b74784aa3677c3005a193c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-07-26T08:04:30.751 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:30 smithi097 podman[195278]: 2024-07-26 08:04:30.468388593 +0000 UTC m=+0.442451281 container start 11a973c277f366f21fe422f27c7989e4189fb7a9b74784aa3677c3005a193c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-26T08:04:30.751 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:30 smithi097 bash[195278]: 11a973c277f366f21fe422f27c7989e4189fb7a9b74784aa3677c3005a193c97 2024-07-26T08:04:30.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:30 smithi097 systemd[1]: Started Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:04:30.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:30 smithi097 ceph-mds[195297]: -- 172.21.15.97:0/2817129125 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a29615f860 con 0x55a29a301180 2024-07-26T08:04:30.752 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:30 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: starting mds.j at 2024-07-26T08:04:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: pgmap v919: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T08:04:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: daemon mds.j restarted 2024-07-26T08:04:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T08:04:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:boot 2024-07-26T08:04:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:04:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:04:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:31 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: pgmap v919: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: daemon mds.j restarted 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:boot 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:31 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: pgmap v919: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T08:04:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: daemon mds.j restarted 2024-07-26T08:04:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T08:04:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:boot 2024-07-26T08:04:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:04:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:04:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:31 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:32.469+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:04:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:32.470+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:04:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:32.470+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:04:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:32.470+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:04:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:32.470+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:04:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:33 smithi115 ceph-mon[35364]: pgmap v921: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:04:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:33 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:resolve 2024-07-26T08:04:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:33 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:33 smithi181 ceph-mon[35869]: pgmap v921: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:04:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:33 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:resolve 2024-07-26T08:04:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:33 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:33 smithi097 ceph-mon[32231]: pgmap v921: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:04:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:33 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:resolve 2024-07-26T08:04:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:33 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:34 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:reconnect 2024-07-26T08:04:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:34 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:34 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:34 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:reconnect 2024-07-26T08:04:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:34 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:34 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:34 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:reconnect 2024-07-26T08:04:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:34 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:34 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-mon[32231]: pgmap v922: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:04:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:rejoin 2024-07-26T08:04:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-mon[32231]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:04:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 312.602681 secs 2024-07-26T08:04:35.654 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:04:35.654 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: *** Caught signal (Aborted) ** 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: in thread 7f53d6a00640 thread_name:ms_dispatch 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f53dcc3e6f0] 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f53dcc8b94c] 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 3: raise() 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 4: abort() 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 5: /lib64/libc.so.6(+0x2871b) [0x7f53dcc2871b] 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 6: /lib64/libc.so.6(+0x37386) [0x7f53dcc37386] 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56185aa846b9] 2024-07-26T08:04:35.655 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56185ac4280c] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56185ab822ca] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56185ab52753] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56185ab53a1b] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56185ab37752] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56185aa85a56] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56185aa89637] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56185aa89f5e] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56185aa6d03a] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f53dd79b407] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 18: (DispatchQueue::entry()+0x60f) [0x7f53dd79bcf9] 2024-07-26T08:04:35.656 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f53dd8785a1] 2024-07-26T08:04:35.657 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 20: (Thread::entry_wrapper()+0x43) [0x7f53dd625cdd] 2024-07-26T08:04:35.657 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 21: (Thread::_entry_func(void*)+0xd) [0x7f53dd625cf9] 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 22: /lib64/libc.so.6(+0x89c02) [0x7f53dcc89c02] 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f53dcd0ec40] 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2024-07-26T08:04:35.407+0000 7f53d6a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: in thread 7f53d6a00640 thread_name:ms_dispatch 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f53dcc3e6f0] 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f53dcc8b94c] 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 3: raise() 2024-07-26T08:04:35.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 4: abort() 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 5: /lib64/libc.so.6(+0x2871b) [0x7f53dcc2871b] 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 6: /lib64/libc.so.6(+0x37386) [0x7f53dcc37386] 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56185aa846b9] 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56185ac4280c] 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56185ab822ca] 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56185ab52753] 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56185ab53a1b] 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56185ab37752] 2024-07-26T08:04:35.659 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56185aa85a56] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56185aa89637] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56185aa89f5e] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56185aa6d03a] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f53dd79b407] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 18: (DispatchQueue::entry()+0x60f) [0x7f53dd79bcf9] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f53dd8785a1] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 20: (Thread::entry_wrapper()+0x43) [0x7f53dd625cdd] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 21: (Thread::_entry_func(void*)+0xd) [0x7f53dd625cf9] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 22: /lib64/libc.so.6(+0x89c02) [0x7f53dcc89c02] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f53dcd0ec40] 2024-07-26T08:04:35.660 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 0> 2024-07-26T08:04:35.407+0000 7f53d6a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: in thread 7f53d6a00640 thread_name:ms_dispatch 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f53dcc3e6f0] 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f53dcc8b94c] 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 3: raise() 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 4: abort() 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 5: /lib64/libc.so.6(+0x2871b) [0x7f53dcc2871b] 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 6: /lib64/libc.so.6(+0x37386) [0x7f53dcc37386] 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56185aa846b9] 2024-07-26T08:04:35.661 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56185ac4280c] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56185ab822ca] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56185ab52753] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56185ab53a1b] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56185ab37752] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56185aa85a56] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56185aa89637] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56185aa89f5e] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56185aa6d03a] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f53dd79b407] 2024-07-26T08:04:35.662 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 18: (DispatchQueue::entry()+0x60f) [0x7f53dd79bcf9] 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f53dd8785a1] 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 20: (Thread::entry_wrapper()+0x43) [0x7f53dd625cdd] 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 21: (Thread::_entry_func(void*)+0xd) [0x7f53dd625cf9] 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 22: /lib64/libc.so.6(+0x89c02) [0x7f53dcc89c02] 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f53dcd0ec40] 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 0> 2024-07-26T08:04:35.407+0000 7f53d6a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: in thread 7f53d6a00640 thread_name:ms_dispatch 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:35.663 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f53dcc3e6f0] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f53dcc8b94c] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 3: raise() 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 4: abort() 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 5: /lib64/libc.so.6(+0x2871b) [0x7f53dcc2871b] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 6: /lib64/libc.so.6(+0x37386) [0x7f53dcc37386] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56185aa846b9] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56185ac4280c] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56185ab822ca] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56185ab52753] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56185ab53a1b] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56185ab37752] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56185aa85a56] 2024-07-26T08:04:35.664 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56185aa89637] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56185aa89f5e] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56185aa6d03a] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f53dd79b407] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 18: (DispatchQueue::entry()+0x60f) [0x7f53dd79bcf9] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f53dd8785a1] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 20: (Thread::entry_wrapper()+0x43) [0x7f53dd625cdd] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 21: (Thread::_entry_func(void*)+0xd) [0x7f53dd625cf9] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 22: /lib64/libc.so.6(+0x89c02) [0x7f53dcc89c02] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f53dcd0ec40] 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195017]: 2024-07-26T08:04:35.665 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:35 smithi097 podman[195331]: 2024-07-26 08:04:35.54793617 +0000 UTC m=+0.033701990 container died 3761a40295bfb8e579305691e603c05464ed9010349ebbf97fbf717a661b3ac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-07-26T08:04:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:35 smithi115 ceph-mon[35364]: pgmap v922: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:04:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:35 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:rejoin 2024-07-26T08:04:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:35 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:35 smithi115 ceph-mon[35364]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:04:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 312.602681 secs 2024-07-26T08:04:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:35 smithi181 ceph-mon[35869]: pgmap v922: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:04:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:35 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:rejoin 2024-07-26T08:04:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:35 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:35 smithi181 ceph-mon[35869]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:04:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 312.602681 secs 2024-07-26T08:04:36.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:36 smithi097 podman[195331]: 2024-07-26 08:04:36.245750487 +0000 UTC m=+0.731516342 container remove 3761a40295bfb8e579305691e603c05464ed9010349ebbf97fbf717a661b3ac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-07-26T08:04:36.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:36 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:04:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:36 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:36 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:active 2024-07-26T08:04:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:36 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:36 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:36 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:36 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:active 2024-07-26T08:04:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:36 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:36 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:36 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:36 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/823708821,v1:172.21.15.97:6837/823708821] up:active 2024-07-26T08:04:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:36 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:36 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:37.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:36 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T08:04:37.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:36 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.088s CPU time. 2024-07-26T08:04:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:37 smithi115 ceph-mon[35364]: pgmap v923: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:37 smithi181 ceph-mon[35869]: pgmap v923: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:37 smithi097 ceph-mon[32231]: pgmap v923: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:39 smithi115 ceph-mon[35364]: pgmap v924: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:39 smithi181 ceph-mon[35869]: pgmap v924: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:39 smithi097 ceph-mon[32231]: pgmap v924: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:04:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 317.602836 secs 2024-07-26T08:04:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 317.602836 secs 2024-07-26T08:04:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 317.602836 secs 2024-07-26T08:04:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:41 smithi115 ceph-mon[35364]: pgmap v925: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:41 smithi181 ceph-mon[35869]: pgmap v925: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:41 smithi097 ceph-mon[32231]: pgmap v925: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:43 smithi115 ceph-mon[35364]: pgmap v926: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 174 B/s wr, 10 op/s 2024-07-26T08:04:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:43 smithi181 ceph-mon[35869]: pgmap v926: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 174 B/s wr, 10 op/s 2024-07-26T08:04:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:43 smithi097 ceph-mon[32231]: pgmap v926: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 174 B/s wr, 10 op/s 2024-07-26T08:04:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:45 smithi115 ceph-mon[35364]: pgmap v927: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:04:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 322.602982 secs 2024-07-26T08:04:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:45 smithi181 ceph-mon[35869]: pgmap v927: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:04:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 322.602982 secs 2024-07-26T08:04:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:45 smithi097 ceph-mon[32231]: pgmap v927: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:04:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 322.602982 secs 2024-07-26T08:04:47.144 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:46 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-07-26T08:04:47.144 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:46 smithi097 systemd[1]: Stopped Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:04:47.144 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:46 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.088s CPU time. 2024-07-26T08:04:47.144 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:46 smithi097 systemd[1]: Starting Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:04:47.410 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:47 smithi097 podman[195434]: 2024-07-26 08:04:47.047766228 +0000 UTC m=+0.033425683 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:04:47.410 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:47 smithi097 podman[195434]: 2024-07-26 08:04:47.197430333 +0000 UTC m=+0.183089809 container create ef5969c23ae599c242089984d66b7b86aa684bb2f78e6ae5e28e95ab34490d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-26T08:04:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:47.474+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:04:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:47.475+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:04:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:47.475+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:04:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:47.475+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:04:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:04:47.475+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:04:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:47 smithi115 ceph-mon[35364]: pgmap v928: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:47 smithi181 ceph-mon[35869]: pgmap v928: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:47 smithi097 ceph-mon[32231]: pgmap v928: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:04:47.751 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:47 smithi097 podman[195434]: 2024-07-26 08:04:47.409615906 +0000 UTC m=+0.395275388 container init ef5969c23ae599c242089984d66b7b86aa684bb2f78e6ae5e28e95ab34490d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:04:47.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:47 smithi097 podman[195434]: 2024-07-26 08:04:47.420466254 +0000 UTC m=+0.406125786 container start ef5969c23ae599c242089984d66b7b86aa684bb2f78e6ae5e28e95ab34490d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-07-26T08:04:47.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:47 smithi097 bash[195434]: ef5969c23ae599c242089984d66b7b86aa684bb2f78e6ae5e28e95ab34490d2e 2024-07-26T08:04:47.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:47 smithi097 systemd[1]: Started Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:04:47.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:47 smithi097 ceph-mds[195453]: -- 172.21.15.97:0/2382771882 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563331b85860 con 0x563335d27600 2024-07-26T08:04:47.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:04:47 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: starting mds.d at 2024-07-26T08:04:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:48 smithi115 ceph-mon[35364]: daemon mds.d restarted 2024-07-26T08:04:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:48 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:48 smithi115 ceph-mon[35364]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T08:04:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:48 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:boot 2024-07-26T08:04:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:48 smithi115 ceph-mon[35364]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:04:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:48 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:48.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:04:48.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:48 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:48 smithi181 ceph-mon[35869]: daemon mds.d restarted 2024-07-26T08:04:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:48 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:48 smithi181 ceph-mon[35869]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T08:04:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:48 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:boot 2024-07-26T08:04:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:48 smithi181 ceph-mon[35869]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:04:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:48 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:04:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:48 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:48 smithi097 ceph-mon[32231]: daemon mds.d restarted 2024-07-26T08:04:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:48 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:04:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:48 smithi097 ceph-mon[32231]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T08:04:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:48 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:boot 2024-07-26T08:04:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:48 smithi097 ceph-mon[32231]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:04:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:48 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:04:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:04:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:48 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:04:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:49 smithi097 ceph-mon[32231]: pgmap v930: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T08:04:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:49 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:resolve 2024-07-26T08:04:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:49 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:49 smithi115 ceph-mon[35364]: pgmap v930: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T08:04:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:49 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:resolve 2024-07-26T08:04:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:49 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:49 smithi181 ceph-mon[35869]: pgmap v930: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T08:04:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:49 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:resolve 2024-07-26T08:04:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:49 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:04:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:50 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:reconnect 2024-07-26T08:04:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:50 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:50 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 327.603138 secs 2024-07-26T08:04:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:50 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:reconnect 2024-07-26T08:04:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:50 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:50 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 327.603138 secs 2024-07-26T08:04:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:50 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:reconnect 2024-07-26T08:04:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:50 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:04:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:50 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:04:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 327.603138 secs 2024-07-26T08:04:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:51 smithi115 ceph-mon[35364]: pgmap v931: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:04:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:51 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:rejoin 2024-07-26T08:04:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:51 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:51 smithi115 ceph-mon[35364]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:04:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:51 smithi181 ceph-mon[35869]: pgmap v931: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:04:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:51 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:rejoin 2024-07-26T08:04:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:51 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:51 smithi181 ceph-mon[35869]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:04:52.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-mon[32231]: pgmap v931: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:04:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:rejoin 2024-07-26T08:04:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:04:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-mon[32231]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:04:52.002 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:04:52.002 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: *** Caught signal (Aborted) ** 2024-07-26T08:04:52.002 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: in thread 7f7d03200640 thread_name:ms_dispatch 2024-07-26T08:04:52.002 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:52.002 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7d0943e6f0] 2024-07-26T08:04:52.002 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7d0948b94c] 2024-07-26T08:04:52.002 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 3: raise() 2024-07-26T08:04:52.002 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 4: abort() 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7d0942871b] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 6: /lib64/libc.so.6(+0x37386) [0x7f7d09437386] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a26462d6b9] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a2647eb80c] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a26472b2ca] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a2646fb753] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a2646fca1b] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a2646e0752] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a26462ea56] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a264632637] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a264632f5e] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a26461603a] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7d09f9b407] 2024-07-26T08:04:52.003 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 18: (DispatchQueue::entry()+0x60f) [0x7f7d09f9bcf9] 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7d0a0785a1] 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 20: (Thread::entry_wrapper()+0x43) [0x7f7d09e25cdd] 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7d09e25cf9] 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7d09489c02] 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7d0950ec40] 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2024-07-26T08:04:51.636+0000 7f7d03200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: in thread 7f7d03200640 thread_name:ms_dispatch 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7d0943e6f0] 2024-07-26T08:04:52.004 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7d0948b94c] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 3: raise() 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 4: abort() 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7d0942871b] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 6: /lib64/libc.so.6(+0x37386) [0x7f7d09437386] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a26462d6b9] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a2647eb80c] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a26472b2ca] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a2646fb753] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a2646fca1b] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a2646e0752] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a26462ea56] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a264632637] 2024-07-26T08:04:52.005 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a264632f5e] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a26461603a] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7d09f9b407] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 18: (DispatchQueue::entry()+0x60f) [0x7f7d09f9bcf9] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7d0a0785a1] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 20: (Thread::entry_wrapper()+0x43) [0x7f7d09e25cdd] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7d09e25cf9] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7d09489c02] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7d0950ec40] 2024-07-26T08:04:52.006 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 0> 2024-07-26T08:04:51.636+0000 7f7d03200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: in thread 7f7d03200640 thread_name:ms_dispatch 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7d0943e6f0] 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7d0948b94c] 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 3: raise() 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 4: abort() 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7d0942871b] 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 6: /lib64/libc.so.6(+0x37386) [0x7f7d09437386] 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a26462d6b9] 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a2647eb80c] 2024-07-26T08:04:52.007 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a26472b2ca] 2024-07-26T08:04:52.008 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a2646fb753] 2024-07-26T08:04:52.008 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a2646fca1b] 2024-07-26T08:04:52.008 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a2646e0752] 2024-07-26T08:04:52.008 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a26462ea56] 2024-07-26T08:04:52.008 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a264632637] 2024-07-26T08:04:52.008 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a264632f5e] 2024-07-26T08:04:52.008 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a26461603a] 2024-07-26T08:04:52.008 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7d09f9b407] 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 18: (DispatchQueue::entry()+0x60f) [0x7f7d09f9bcf9] 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7d0a0785a1] 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 20: (Thread::entry_wrapper()+0x43) [0x7f7d09e25cdd] 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7d09e25cf9] 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7d09489c02] 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7d0950ec40] 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2024-07-26T08:04:52.009 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: -2> 2024-07-26T08:04:51.636+0000 7f7d03200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: in thread 7f7d03200640 thread_name:ms_dispatch 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7d0943e6f0] 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7d0948b94c] 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 3: raise() 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 4: abort() 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7d0942871b] 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 6: /lib64/libc.so.6(+0x37386) [0x7f7d09437386] 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a26462d6b9] 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a2647eb80c] 2024-07-26T08:04:52.010 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55a26472b2ca] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55a2646fb753] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a2646fca1b] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a2646e0752] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a26462ea56] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a264632637] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a264632f5e] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a26461603a] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7d09f9b407] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 18: (DispatchQueue::entry()+0x60f) [0x7f7d09f9bcf9] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7d0a0785a1] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 20: (Thread::entry_wrapper()+0x43) [0x7f7d09e25cdd] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7d09e25cf9] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7d09489c02] 2024-07-26T08:04:52.011 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7d0950ec40] 2024-07-26T08:04:52.012 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:04:52.012 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195293]: 2024-07-26T08:04:52.012 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:51 smithi097 podman[195487]: 2024-07-26 08:04:51.760249558 +0000 UTC m=+0.037116471 container died 11a973c277f366f21fe422f27c7989e4189fb7a9b74784aa3677c3005a193c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T08:04:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:52 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:52 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:active 2024-07-26T08:04:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:52 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:52 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:52 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:52 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:active 2024-07-26T08:04:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:52 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:52 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:53.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:52 smithi097 podman[195487]: 2024-07-26 08:04:52.524507482 +0000 UTC m=+0.801374421 container remove 11a973c277f366f21fe422f27c7989e4189fb7a9b74784aa3677c3005a193c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T08:04:53.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:52 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:04:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:52 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:04:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:52 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1919466133,v1:172.21.15.97:6835/1919466133] up:active 2024-07-26T08:04:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:52 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:04:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:52 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:04:53.501 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:53 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:04:53.501 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:04:53 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.101s CPU time. 2024-07-26T08:04:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:53 smithi115 ceph-mon[35364]: pgmap v932: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:04:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:53 smithi181 ceph-mon[35869]: pgmap v932: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:04:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:53 smithi097 ceph-mon[32231]: pgmap v932: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:04:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:55 smithi115 ceph-mon[35364]: pgmap v933: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 332.603300 secs 2024-07-26T08:04:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:55 smithi181 ceph-mon[35869]: pgmap v933: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 332.603300 secs 2024-07-26T08:04:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:55 smithi097 ceph-mon[32231]: pgmap v933: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:04:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 332.603300 secs 2024-07-26T08:04:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:57 smithi115 ceph-mon[35364]: pgmap v934: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:57 smithi181 ceph-mon[35869]: pgmap v934: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:57 smithi097 ceph-mon[32231]: pgmap v934: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:04:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:04:59 smithi115 ceph-mon[35364]: pgmap v935: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 189 B/s wr, 10 op/s 2024-07-26T08:04:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:04:59 smithi181 ceph-mon[35869]: pgmap v935: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 189 B/s wr, 10 op/s 2024-07-26T08:05:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:04:59 smithi097 ceph-mon[32231]: pgmap v935: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 189 B/s wr, 10 op/s 2024-07-26T08:05:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 337.603434 secs 2024-07-26T08:05:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 337.603434 secs 2024-07-26T08:05:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 337.603434 secs 2024-07-26T08:05:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:01 smithi115 ceph-mon[35364]: pgmap v936: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:01 smithi181 ceph-mon[35869]: pgmap v936: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:01 smithi097 ceph-mon[32231]: pgmap v936: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:02.477+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:05:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:02.479+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:05:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:02.479+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:05:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:02.479+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:05:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:02.479+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:05:03.664 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:03 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-07-26T08:05:03.664 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:03 smithi097 systemd[1]: Stopped Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:05:03.664 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:03 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.101s CPU time. 2024-07-26T08:05:03.664 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:03 smithi097 systemd[1]: Starting Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:05:03.970 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:03 smithi097 ceph-mon[32231]: pgmap v937: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:03.970 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:03 smithi097 podman[195576]: 2024-07-26 08:05:03.569519813 +0000 UTC m=+0.035540571 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:05:03.970 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:03 smithi097 podman[195576]: 2024-07-26 08:05:03.689823508 +0000 UTC m=+0.155844284 container create d4fb6ee685d00536838653a7bb889acb4dbd84f8c6585c908a3c9a157cbe316b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:05:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:03 smithi115 ceph-mon[35364]: pgmap v937: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:03 smithi181 ceph-mon[35869]: pgmap v937: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:04.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:03 smithi097 podman[195576]: 2024-07-26 08:05:03.969101238 +0000 UTC m=+0.435122050 container init d4fb6ee685d00536838653a7bb889acb4dbd84f8c6585c908a3c9a157cbe316b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-07-26T08:05:04.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:03 smithi097 podman[195576]: 2024-07-26 08:05:03.979115443 +0000 UTC m=+0.445136222 container start d4fb6ee685d00536838653a7bb889acb4dbd84f8c6585c908a3c9a157cbe316b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T08:05:04.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:04 smithi097 bash[195576]: d4fb6ee685d00536838653a7bb889acb4dbd84f8c6585c908a3c9a157cbe316b 2024-07-26T08:05:04.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:04 smithi097 systemd[1]: Started Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:05:04.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mds[195595]: -- 172.21.15.97:0/633004041 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f42f2ff860 con 0x55f4334a1600 2024-07-26T08:05:04.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: starting mds.j at 2024-07-26T08:05:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:04 smithi115 ceph-mon[35364]: daemon mds.j restarted 2024-07-26T08:05:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:04 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:04 smithi115 ceph-mon[35364]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T08:05:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:04 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:boot 2024-07-26T08:05:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:04 smithi115 ceph-mon[35364]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:05:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:04 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:05:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:04 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:04 smithi181 ceph-mon[35869]: daemon mds.j restarted 2024-07-26T08:05:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:04 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:04 smithi181 ceph-mon[35869]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T08:05:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:04 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:boot 2024-07-26T08:05:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:04 smithi181 ceph-mon[35869]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:05:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:04 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:05:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:04 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mon[32231]: daemon mds.j restarted 2024-07-26T08:05:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mon[32231]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T08:05:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:boot 2024-07-26T08:05:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mon[32231]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:05:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:05:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:04 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:05 smithi115 ceph-mon[35364]: pgmap v939: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 342.603567 secs 2024-07-26T08:05:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:05 smithi181 ceph-mon[35869]: pgmap v939: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 342.603567 secs 2024-07-26T08:05:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:05 smithi097 ceph-mon[32231]: pgmap v939: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 342.603567 secs 2024-07-26T08:05:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:06 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:resolve 2024-07-26T08:05:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:06 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:06 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:resolve 2024-07-26T08:05:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:06 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:06 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:resolve 2024-07-26T08:05:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:06 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:07 smithi181 ceph-mon[35869]: pgmap v940: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:05:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:07 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:05:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:07 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:reconnect 2024-07-26T08:05:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:07 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:07 smithi097 ceph-mon[32231]: pgmap v940: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:05:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:07 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:05:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:07 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:reconnect 2024-07-26T08:05:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:07 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:07 smithi115 ceph-mon[35364]: pgmap v940: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:05:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:07 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00799986 2024-07-26T08:05:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:07 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:reconnect 2024-07-26T08:05:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:07 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:08 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:rejoin 2024-07-26T08:05:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:08 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:08 smithi115 ceph-mon[35364]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:05:09.228 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:rejoin 2024-07-26T08:05:09.228 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:09.228 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-mon[32231]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: *** Caught signal (Aborted) ** 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: in thread 7f73cfe00640 thread_name:ms_dispatch 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f73d603e6f0] 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f73d608b94c] 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 3: raise() 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 4: abort() 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 5: /lib64/libc.so.6(+0x2871b) [0x7f73d602871b] 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 6: /lib64/libc.so.6(+0x37386) [0x7f73d6037386] 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56331eda36b9] 2024-07-26T08:05:09.230 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56331ef6180c] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56331eea12ca] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56331ee71753] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56331ee72a1b] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56331ee56752] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56331eda4a56] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56331eda8637] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56331eda8f5e] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56331ed8c03a] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f73d6b9b407] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 18: (DispatchQueue::entry()+0x60f) [0x7f73d6b9bcf9] 2024-07-26T08:05:09.231 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f73d6c785a1] 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 20: (Thread::entry_wrapper()+0x43) [0x7f73d6a25cdd] 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 21: (Thread::_entry_func(void*)+0xd) [0x7f73d6a25cf9] 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 22: /lib64/libc.so.6(+0x89c02) [0x7f73d6089c02] 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f73d610ec40] 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2024-07-26T08:05:08.771+0000 7f73cfe00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: in thread 7f73cfe00640 thread_name:ms_dispatch 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f73d603e6f0] 2024-07-26T08:05:09.232 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f73d608b94c] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 3: raise() 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 4: abort() 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 5: /lib64/libc.so.6(+0x2871b) [0x7f73d602871b] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 6: /lib64/libc.so.6(+0x37386) [0x7f73d6037386] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56331eda36b9] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56331ef6180c] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56331eea12ca] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56331ee71753] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56331ee72a1b] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56331ee56752] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56331eda4a56] 2024-07-26T08:05:09.233 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56331eda8637] 2024-07-26T08:05:09.234 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56331eda8f5e] 2024-07-26T08:05:09.234 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56331ed8c03a] 2024-07-26T08:05:09.234 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f73d6b9b407] 2024-07-26T08:05:09.234 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 18: (DispatchQueue::entry()+0x60f) [0x7f73d6b9bcf9] 2024-07-26T08:05:09.234 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f73d6c785a1] 2024-07-26T08:05:09.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:08 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:rejoin 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:08 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:08 smithi181 ceph-mon[35869]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 20: (Thread::entry_wrapper()+0x43) [0x7f73d6a25cdd] 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 21: (Thread::_entry_func(void*)+0xd) [0x7f73d6a25cf9] 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 22: /lib64/libc.so.6(+0x89c02) [0x7f73d6089c02] 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f73d610ec40] 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 0> 2024-07-26T08:05:08.771+0000 7f73cfe00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: in thread 7f73cfe00640 thread_name:ms_dispatch 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:09.235 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f73d603e6f0] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f73d608b94c] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 3: raise() 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 4: abort() 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 5: /lib64/libc.so.6(+0x2871b) [0x7f73d602871b] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 6: /lib64/libc.so.6(+0x37386) [0x7f73d6037386] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56331eda36b9] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56331ef6180c] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56331eea12ca] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56331ee71753] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56331ee72a1b] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56331ee56752] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56331eda4a56] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56331eda8637] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56331eda8f5e] 2024-07-26T08:05:09.236 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56331ed8c03a] 2024-07-26T08:05:09.237 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f73d6b9b407] 2024-07-26T08:05:09.237 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 18: (DispatchQueue::entry()+0x60f) [0x7f73d6b9bcf9] 2024-07-26T08:05:09.237 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f73d6c785a1] 2024-07-26T08:05:09.237 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 20: (Thread::entry_wrapper()+0x43) [0x7f73d6a25cdd] 2024-07-26T08:05:09.237 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 21: (Thread::_entry_func(void*)+0xd) [0x7f73d6a25cf9] 2024-07-26T08:05:09.237 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 22: /lib64/libc.so.6(+0x89c02) [0x7f73d6089c02] 2024-07-26T08:05:09.237 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f73d610ec40] 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: -2> 2024-07-26T08:05:08.771+0000 7f73cfe00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: in thread 7f73cfe00640 thread_name:ms_dispatch 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f73d603e6f0] 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f73d608b94c] 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 3: raise() 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 4: abort() 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 5: /lib64/libc.so.6(+0x2871b) [0x7f73d602871b] 2024-07-26T08:05:09.238 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 6: /lib64/libc.so.6(+0x37386) [0x7f73d6037386] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56331eda36b9] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56331ef6180c] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56331eea12ca] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56331ee71753] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56331ee72a1b] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56331ee56752] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56331eda4a56] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56331eda8637] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56331eda8f5e] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56331ed8c03a] 2024-07-26T08:05:09.239 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f73d6b9b407] 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 18: (DispatchQueue::entry()+0x60f) [0x7f73d6b9bcf9] 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f73d6c785a1] 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 20: (Thread::entry_wrapper()+0x43) [0x7f73d6a25cdd] 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 21: (Thread::_entry_func(void*)+0xd) [0x7f73d6a25cf9] 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 22: /lib64/libc.so.6(+0x89c02) [0x7f73d6089c02] 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f73d610ec40] 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195449]: 2024-07-26T08:05:09.240 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:08 smithi097 podman[195629]: 2024-07-26 08:05:08.945860246 +0000 UTC m=+0.043410955 container died ef5969c23ae599c242089984d66b7b86aa684bb2f78e6ae5e28e95ab34490d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-07-26T08:05:09.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:09 smithi097 podman[195629]: 2024-07-26 08:05:09.639234082 +0000 UTC m=+0.736784828 container remove ef5969c23ae599c242089984d66b7b86aa684bb2f78e6ae5e28e95ab34490d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-07-26T08:05:09.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:09 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:05:10.073 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:09 smithi181 ceph-mon[35869]: pgmap v941: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:05:10.073 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:09 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:10.073 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:09 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:active 2024-07-26T08:05:10.073 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:09 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:05:10.073 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:09 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:10.074 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:09 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 347.603704 secs 2024-07-26T08:05:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:09 smithi115 ceph-mon[35364]: pgmap v941: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:05:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:09 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:09 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:active 2024-07-26T08:05:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:09 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:05:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:09 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:10.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:09 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 347.603704 secs 2024-07-26T08:05:10.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:09 smithi097 ceph-mon[32231]: pgmap v941: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:05:10.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:09 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:10.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:09 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/4060893273,v1:172.21.15.97:6837/4060893273] up:active 2024-07-26T08:05:10.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:09 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:05:10.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:09 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:10.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:09 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 347.603704 secs 2024-07-26T08:05:10.482 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:05:10 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:05:10.071+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:05:10.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:10 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T08:05:10.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:10 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.147s CPU time. 2024-07-26T08:05:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:11 smithi115 ceph-mon[35364]: pgmap v942: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 11 op/s 2024-07-26T08:05:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:11 smithi181 ceph-mon[35869]: pgmap v942: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 11 op/s 2024-07-26T08:05:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:11 smithi097 ceph-mon[32231]: pgmap v942: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 11 op/s 2024-07-26T08:05:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:13 smithi115 ceph-mon[35364]: pgmap v943: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:05:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:13 smithi181 ceph-mon[35869]: pgmap v943: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:05:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:13 smithi097 ceph-mon[32231]: pgmap v943: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:05:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:14 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 352.603801 secs 2024-07-26T08:05:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:14 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 352.603801 secs 2024-07-26T08:05:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:14 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 352.603801 secs 2024-07-26T08:05:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:15 smithi115 ceph-mon[35364]: pgmap v944: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 200 B/s wr, 11 op/s 2024-07-26T08:05:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:15 smithi181 ceph-mon[35869]: pgmap v944: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 200 B/s wr, 11 op/s 2024-07-26T08:05:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:15 smithi097 ceph-mon[32231]: pgmap v944: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 200 B/s wr, 11 op/s 2024-07-26T08:05:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:17.488+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:05:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:17.489+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:05:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:17.489+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:05:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:17.489+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:05:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:17.489+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:05:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:17 smithi115 ceph-mon[35364]: pgmap v945: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:17 smithi181 ceph-mon[35869]: pgmap v945: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:17 smithi097 ceph-mon[32231]: pgmap v945: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:19 smithi115 ceph-mon[35364]: pgmap v946: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:19 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 357.603940 secs 2024-07-26T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:19 smithi181 ceph-mon[35869]: pgmap v946: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:19 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 357.603940 secs 2024-07-26T08:05:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:19 smithi097 ceph-mon[32231]: pgmap v946: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:19 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 357.603940 secs 2024-07-26T08:05:20.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:20 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-07-26T08:05:20.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:20 smithi097 systemd[1]: Stopped Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:05:20.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:20 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.147s CPU time. 2024-07-26T08:05:20.658 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:20 smithi097 systemd[1]: Starting Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:05:20.920 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:20 smithi097 podman[195718]: 2024-07-26 08:05:20.561369897 +0000 UTC m=+0.033536673 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:05:20.920 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:20 smithi097 podman[195718]: 2024-07-26 08:05:20.674124052 +0000 UTC m=+0.146290864 container create c0211e40a02c26ede9f13510edc64b518fb00b01a41008efeaf442c8325b588f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-26T08:05:21.252 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:20 smithi097 podman[195718]: 2024-07-26 08:05:20.91923866 +0000 UTC m=+0.391405478 container init c0211e40a02c26ede9f13510edc64b518fb00b01a41008efeaf442c8325b588f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-07-26T08:05:21.252 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:20 smithi097 podman[195718]: 2024-07-26 08:05:20.930870326 +0000 UTC m=+0.403037141 container start c0211e40a02c26ede9f13510edc64b518fb00b01a41008efeaf442c8325b588f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T08:05:21.252 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:21 smithi097 bash[195718]: c0211e40a02c26ede9f13510edc64b518fb00b01a41008efeaf442c8325b588f 2024-07-26T08:05:21.252 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:21 smithi097 systemd[1]: Started Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:05:21.252 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mds[195737]: -- 172.21.15.97:0/452865251 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560f0c851860 con 0x560f109f3180 2024-07-26T08:05:21.252 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: starting mds.d at 2024-07-26T08:05:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: pgmap v947: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: daemon mds.d restarted 2024-07-26T08:05:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T08:05:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:boot 2024-07-26T08:05:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:05:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:05:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:21 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: pgmap v947: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: daemon mds.d restarted 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:boot 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:05:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:21 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: pgmap v947: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: daemon mds.d restarted 2024-07-26T08:05:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T08:05:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:boot 2024-07-26T08:05:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-07-26T08:05:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:05:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:21 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:23 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:resolve 2024-07-26T08:05:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:23 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:23 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:resolve 2024-07-26T08:05:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:23 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:23 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:resolve 2024-07-26T08:05:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:23 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:24 smithi115 ceph-mon[35364]: pgmap v949: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T08:05:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:24 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899984 2024-07-26T08:05:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:24 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:reconnect 2024-07-26T08:05:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:24 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:24 smithi181 ceph-mon[35869]: pgmap v949: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T08:05:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:24 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899984 2024-07-26T08:05:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:24 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:reconnect 2024-07-26T08:05:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:24 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-mon[32231]: pgmap v949: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T08:05:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00899984 2024-07-26T08:05:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:reconnect 2024-07-26T08:05:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:24.501 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:05:24.501 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: *** Caught signal (Aborted) ** 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: in thread 7f59fe400640 thread_name:ms_dispatch 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5a0463e6f0] 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5a0468b94c] 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 3: raise() 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 4: abort() 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5a0462871b] 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 6: /lib64/libc.so.6(+0x37386) [0x7f5a04637386] 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f3f68f56b9] 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f3f6ab380c] 2024-07-26T08:05:24.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f3f69f32ca] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f3f69c3753] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f3f69c4a1b] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f3f69a8752] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f3f68f6a56] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f3f68fa637] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f3f68faf5e] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f3f68de03a] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5a0519b407] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 18: (DispatchQueue::entry()+0x60f) [0x7f5a0519bcf9] 2024-07-26T08:05:24.503 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5a052785a1] 2024-07-26T08:05:24.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 20: (Thread::entry_wrapper()+0x43) [0x7f5a05025cdd] 2024-07-26T08:05:24.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5a05025cf9] 2024-07-26T08:05:24.504 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5a04689c02] 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5a0470ec40] 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2024-07-26T08:05:24.201+0000 7f59fe400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: in thread 7f59fe400640 thread_name:ms_dispatch 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5a0463e6f0] 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5a0468b94c] 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 3: raise() 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 4: abort() 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5a0462871b] 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 6: /lib64/libc.so.6(+0x37386) [0x7f5a04637386] 2024-07-26T08:05:24.505 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f3f68f56b9] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f3f6ab380c] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f3f69f32ca] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f3f69c3753] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f3f69c4a1b] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f3f69a8752] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f3f68f6a56] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f3f68fa637] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f3f68faf5e] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f3f68de03a] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5a0519b407] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 18: (DispatchQueue::entry()+0x60f) [0x7f5a0519bcf9] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5a052785a1] 2024-07-26T08:05:24.506 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 20: (Thread::entry_wrapper()+0x43) [0x7f5a05025cdd] 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5a05025cf9] 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5a04689c02] 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5a0470ec40] 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 0> 2024-07-26T08:05:24.201+0000 7f59fe400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: in thread 7f59fe400640 thread_name:ms_dispatch 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5a0463e6f0] 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5a0468b94c] 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 3: raise() 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 4: abort() 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5a0462871b] 2024-07-26T08:05:24.507 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 6: /lib64/libc.so.6(+0x37386) [0x7f5a04637386] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f3f68f56b9] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f3f6ab380c] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f3f69f32ca] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f3f69c3753] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f3f69c4a1b] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f3f69a8752] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f3f68f6a56] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f3f68fa637] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f3f68faf5e] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f3f68de03a] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5a0519b407] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 18: (DispatchQueue::entry()+0x60f) [0x7f5a0519bcf9] 2024-07-26T08:05:24.508 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5a052785a1] 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 20: (Thread::entry_wrapper()+0x43) [0x7f5a05025cdd] 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5a05025cf9] 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5a04689c02] 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5a0470ec40] 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: -2> 2024-07-26T08:05:24.201+0000 7f59fe400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: in thread 7f59fe400640 thread_name:ms_dispatch 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:24.509 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5a0463e6f0] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5a0468b94c] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 3: raise() 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 4: abort() 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5a0462871b] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 6: /lib64/libc.so.6(+0x37386) [0x7f5a04637386] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f3f68f56b9] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f3f6ab380c] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f3f69f32ca] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f3f69c3753] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f3f69c4a1b] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f3f69a8752] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f3f68f6a56] 2024-07-26T08:05:24.510 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f3f68fa637] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f3f68faf5e] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f3f68de03a] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5a0519b407] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 18: (DispatchQueue::entry()+0x60f) [0x7f5a0519bcf9] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5a052785a1] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 20: (Thread::entry_wrapper()+0x43) [0x7f5a05025cdd] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5a05025cf9] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5a04689c02] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5a0470ec40] 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[195591]: 2024-07-26T08:05:24.511 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:24 smithi097 podman[195771]: 2024-07-26 08:05:24.341994329 +0000 UTC m=+0.033116861 container died d4fb6ee685d00536838653a7bb889acb4dbd84f8c6585c908a3c9a157cbe316b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS) 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:25 smithi097 podman[195771]: 2024-07-26 08:05:25.065697073 +0000 UTC m=+0.756819643 container remove d4fb6ee685d00536838653a7bb889acb4dbd84f8c6585c908a3c9a157cbe316b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:25 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:rejoin 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:active 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:25.447 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: 1 slow requests, 1 included below; oldest blocked for > 362.604085 secs 2024-07-26T08:05:25.448 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: slow request 362.604084 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:05:25.448 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:rejoin 2024-07-26T08:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:active 2024-07-26T08:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: 1 slow requests, 1 included below; oldest blocked for > 362.604085 secs 2024-07-26T08:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: slow request 362.604084 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:05:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:05:25.482 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:05:25.072+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:rejoin 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/1706196819,v1:172.21.15.97:6835/1706196819] up:active 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: 1 slow requests, 1 included below; oldest blocked for > 362.604085 secs 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: slow request 362.604084 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:05:25.751 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:25 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:05:25.751 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:25 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 1.977s CPU time. 2024-07-26T08:05:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:26 smithi115 ceph-mon[35364]: pgmap v950: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:05:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:05:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:05:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:05:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:26 smithi181 ceph-mon[35869]: pgmap v950: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:05:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:05:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:05:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:05:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:26 smithi097 ceph-mon[32231]: pgmap v950: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:05:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:05:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:05:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:05:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:27 smithi115 ceph-mon[35364]: pgmap v951: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 108 B/s wr, 11 op/s 2024-07-26T08:05:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:27 smithi115 ceph-mon[35364]: pgmap v952: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 138 B/s wr, 12 op/s 2024-07-26T08:05:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:27 smithi181 ceph-mon[35869]: pgmap v951: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 108 B/s wr, 11 op/s 2024-07-26T08:05:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:27 smithi181 ceph-mon[35869]: pgmap v952: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 138 B/s wr, 12 op/s 2024-07-26T08:05:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:27 smithi097 ceph-mon[32231]: pgmap v951: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 108 B/s wr, 11 op/s 2024-07-26T08:05:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:27 smithi097 ceph-mon[32231]: pgmap v952: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 138 B/s wr, 12 op/s 2024-07-26T08:05:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:29 smithi115 ceph-mon[35364]: pgmap v953: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 276 B/s wr, 12 op/s 2024-07-26T08:05:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:29 smithi181 ceph-mon[35869]: pgmap v953: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 276 B/s wr, 12 op/s 2024-07-26T08:05:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:29 smithi097 ceph-mon[32231]: pgmap v953: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 276 B/s wr, 12 op/s 2024-07-26T08:05:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 367.604223 secs 2024-07-26T08:05:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 367.604223 secs 2024-07-26T08:05:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 367.604223 secs 2024-07-26T08:05:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:31 smithi097 ceph-mon[32231]: pgmap v954: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 235 B/s wr, 11 op/s 2024-07-26T08:05:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:31 smithi115 ceph-mon[35364]: pgmap v954: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 235 B/s wr, 11 op/s 2024-07-26T08:05:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:31 smithi181 ceph-mon[35869]: pgmap v954: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 235 B/s wr, 11 op/s 2024-07-26T08:05:32.493 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:32.492+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:05:32.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:32.493+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:05:32.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:32.493+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:05:32.495 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:32.493+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:05:32.495 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:32.493+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:05:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:33 smithi097 ceph-mon[32231]: pgmap v955: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 217 B/s wr, 12 op/s 2024-07-26T08:05:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:33 smithi115 ceph-mon[35364]: pgmap v955: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 217 B/s wr, 12 op/s 2024-07-26T08:05:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:33 smithi181 ceph-mon[35869]: pgmap v955: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 217 B/s wr, 12 op/s 2024-07-26T08:05:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:35 smithi097 ceph-mon[32231]: pgmap v956: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 10 op/s 2024-07-26T08:05:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 372.604373 secs 2024-07-26T08:05:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:35 smithi115 ceph-mon[35364]: pgmap v956: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 10 op/s 2024-07-26T08:05:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 372.604373 secs 2024-07-26T08:05:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:35 smithi181 ceph-mon[35869]: pgmap v956: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 217 B/s wr, 10 op/s 2024-07-26T08:05:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 372.604373 secs 2024-07-26T08:05:35.900 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:35 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-07-26T08:05:35.900 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:35 smithi097 systemd[1]: Stopped Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:05:35.901 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:35 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 1.977s CPU time. 2024-07-26T08:05:35.901 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:35 smithi097 systemd[1]: Starting Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968... 2024-07-26T08:05:36.156 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:35 smithi097 podman[195990]: 2024-07-26 08:05:35.803595656 +0000 UTC m=+0.036847142 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:05:36.156 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:35 smithi097 podman[195990]: 2024-07-26 08:05:35.93415501 +0000 UTC m=+0.167406547 container create 17a6f27414c40711955ac46fce9ab2f5972189199935132f968df60bf623ac9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T08:05:36.156 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:36 smithi097 podman[195990]: 2024-07-26 08:05:36.154198158 +0000 UTC m=+0.387449702 container init 17a6f27414c40711955ac46fce9ab2f5972189199935132f968df60bf623ac9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T08:05:36.501 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:36 smithi097 podman[195990]: 2024-07-26 08:05:36.164180335 +0000 UTC m=+0.397431882 container start 17a6f27414c40711955ac46fce9ab2f5972189199935132f968df60bf623ac9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-26T08:05:36.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:36 smithi097 bash[195990]: 17a6f27414c40711955ac46fce9ab2f5972189199935132f968df60bf623ac9b 2024-07-26T08:05:36.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:36 smithi097 systemd[1]: Started Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:05:36.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:36 smithi097 ceph-mds[196016]: -- 172.21.15.97:0/1035830476 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b873969860 con 0x55b877b0b600 2024-07-26T08:05:36.502 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:05:36 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: starting mds.j at 2024-07-26T08:05:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: pgmap v957: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: daemon mds.j restarted 2024-07-26T08:05:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T08:05:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:boot 2024-07-26T08:05:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:05:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:05:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:37 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: pgmap v957: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: daemon mds.j restarted 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:boot 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:37 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: pgmap v957: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: daemon mds.j restarted 2024-07-26T08:05:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:05:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T08:05:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:boot 2024-07-26T08:05:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T08:05:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: fsmap cephfs:4/5 {0=c=up:active,2=l=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:05:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:05:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:37 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:05:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:38 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:resolve 2024-07-26T08:05:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:38 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:38 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:resolve 2024-07-26T08:05:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:38 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:38 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:resolve 2024-07-26T08:05:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:38 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:05:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:39 smithi115 ceph-mon[35364]: pgmap v959: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:05:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:39 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:05:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:39 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:reconnect 2024-07-26T08:05:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:39 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:39 smithi181 ceph-mon[35869]: pgmap v959: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:05:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:39 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:05:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:39 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:reconnect 2024-07-26T08:05:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:39 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:39 smithi097 ceph-mon[32231]: pgmap v959: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:05:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:39 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:05:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:39 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:reconnect 2024-07-26T08:05:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:39 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:05:40.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:40 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:rejoin 2024-07-26T08:05:40.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:40 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:40.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:40 smithi115 ceph-mon[35364]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:05:40.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 377.604461 secs 2024-07-26T08:05:40.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:40 smithi115 ceph-mon[35364]: pgmap v960: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:05:40.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:40 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:rejoin 2024-07-26T08:05:40.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:40 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:40.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:40 smithi181 ceph-mon[35869]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:05:40.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 377.604461 secs 2024-07-26T08:05:40.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:40 smithi181 ceph-mon[35869]: pgmap v960: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:05:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:rejoin 2024-07-26T08:05:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:05:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-mon[32231]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T08:05:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 377.604461 secs 2024-07-26T08:05:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-mon[32231]: pgmap v960: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:05:40.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:05:40.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: *** Caught signal (Aborted) ** 2024-07-26T08:05:40.752 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: in thread 7fac17e00640 thread_name:ms_dispatch 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fac1e03e6f0] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fac1e08b94c] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 3: raise() 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 4: abort() 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 5: /lib64/libc.so.6(+0x2871b) [0x7fac1e02871b] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 6: /lib64/libc.so.6(+0x37386) [0x7fac1e037386] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560f030896b9] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560f0324780c] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560f031872ca] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560f03157753] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560f03158a1b] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560f0313c752] 2024-07-26T08:05:40.753 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560f0308aa56] 2024-07-26T08:05:40.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560f0308e637] 2024-07-26T08:05:40.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560f0308ef5e] 2024-07-26T08:05:40.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560f0307203a] 2024-07-26T08:05:40.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fac1eb9b407] 2024-07-26T08:05:40.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 18: (DispatchQueue::entry()+0x60f) [0x7fac1eb9bcf9] 2024-07-26T08:05:40.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fac1ec785a1] 2024-07-26T08:05:40.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 20: (Thread::entry_wrapper()+0x43) [0x7fac1ea25cdd] 2024-07-26T08:05:40.754 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 21: (Thread::_entry_func(void*)+0xd) [0x7fac1ea25cf9] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 22: /lib64/libc.so.6(+0x89c02) [0x7fac1e089c02] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fac1e10ec40] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2024-07-26T08:05:40.358+0000 7fac17e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: in thread 7fac17e00640 thread_name:ms_dispatch 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fac1e03e6f0] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fac1e08b94c] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 3: raise() 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 4: abort() 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 5: /lib64/libc.so.6(+0x2871b) [0x7fac1e02871b] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 6: /lib64/libc.so.6(+0x37386) [0x7fac1e037386] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560f030896b9] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560f0324780c] 2024-07-26T08:05:40.755 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560f031872ca] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560f03157753] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560f03158a1b] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560f0313c752] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560f0308aa56] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560f0308e637] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560f0308ef5e] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560f0307203a] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fac1eb9b407] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 18: (DispatchQueue::entry()+0x60f) [0x7fac1eb9bcf9] 2024-07-26T08:05:40.756 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fac1ec785a1] 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 20: (Thread::entry_wrapper()+0x43) [0x7fac1ea25cdd] 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 21: (Thread::_entry_func(void*)+0xd) [0x7fac1ea25cf9] 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 22: /lib64/libc.so.6(+0x89c02) [0x7fac1e089c02] 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fac1e10ec40] 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 0> 2024-07-26T08:05:40.358+0000 7fac17e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: in thread 7fac17e00640 thread_name:ms_dispatch 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fac1e03e6f0] 2024-07-26T08:05:40.757 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fac1e08b94c] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 3: raise() 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 4: abort() 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 5: /lib64/libc.so.6(+0x2871b) [0x7fac1e02871b] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 6: /lib64/libc.so.6(+0x37386) [0x7fac1e037386] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560f030896b9] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560f0324780c] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560f031872ca] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560f03157753] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560f03158a1b] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560f0313c752] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560f0308aa56] 2024-07-26T08:05:40.758 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560f0308e637] 2024-07-26T08:05:40.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560f0308ef5e] 2024-07-26T08:05:40.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560f0307203a] 2024-07-26T08:05:40.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fac1eb9b407] 2024-07-26T08:05:40.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 18: (DispatchQueue::entry()+0x60f) [0x7fac1eb9bcf9] 2024-07-26T08:05:40.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fac1ec785a1] 2024-07-26T08:05:40.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 20: (Thread::entry_wrapper()+0x43) [0x7fac1ea25cdd] 2024-07-26T08:05:40.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 21: (Thread::_entry_func(void*)+0xd) [0x7fac1ea25cf9] 2024-07-26T08:05:40.759 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 22: /lib64/libc.so.6(+0x89c02) [0x7fac1e089c02] 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fac1e10ec40] 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: -2> 2024-07-26T08:05:40.358+0000 7fac17e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: in thread 7fac17e00640 thread_name:ms_dispatch 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fac1e03e6f0] 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fac1e08b94c] 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 3: raise() 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 4: abort() 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 5: /lib64/libc.so.6(+0x2871b) [0x7fac1e02871b] 2024-07-26T08:05:40.760 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 6: /lib64/libc.so.6(+0x37386) [0x7fac1e037386] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560f030896b9] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560f0324780c] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560f031872ca] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560f03157753] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560f03158a1b] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560f0313c752] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560f0308aa56] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560f0308e637] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560f0308ef5e] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560f0307203a] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fac1eb9b407] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 18: (DispatchQueue::entry()+0x60f) [0x7fac1eb9bcf9] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fac1ec785a1] 2024-07-26T08:05:40.761 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 20: (Thread::entry_wrapper()+0x43) [0x7fac1ea25cdd] 2024-07-26T08:05:40.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 21: (Thread::_entry_func(void*)+0xd) [0x7fac1ea25cf9] 2024-07-26T08:05:40.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 22: /lib64/libc.so.6(+0x89c02) [0x7fac1e089c02] 2024-07-26T08:05:40.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fac1e10ec40] 2024-07-26T08:05:40.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:05:40.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d[195733]: 2024-07-26T08:05:40.762 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:40 smithi097 podman[196053]: 2024-07-26 08:05:40.491610059 +0000 UTC m=+0.042246183 container died c0211e40a02c26ede9f13510edc64b518fb00b01a41008efeaf442c8325b588f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T08:05:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:41 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:41 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:active 2024-07-26T08:05:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:41 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:05:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:41 smithi097 ceph-mon[32231]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:41.501 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:41 smithi097 podman[196053]: 2024-07-26 08:05:41.158183596 +0000 UTC m=+0.708819740 container remove c0211e40a02c26ede9f13510edc64b518fb00b01a41008efeaf442c8325b588f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-d, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:05:41.502 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:41 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:05:41.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:41 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:41.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:41 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:active 2024-07-26T08:05:41.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:41 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:05:41.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:41 smithi115 ceph-mon[35364]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:41 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:05:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:41 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6836/2214919839,v1:172.21.15.97:6837/2214919839] up:active 2024-07-26T08:05:41.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:41 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:05:41.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:41 smithi181 ceph-mon[35869]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:05:42.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:41 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T08:05:42.001 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:41 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.106s CPU time. 2024-07-26T08:05:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:42 smithi115 ceph-mon[35364]: pgmap v961: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:42 smithi181 ceph-mon[35869]: pgmap v961: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:42 smithi097 ceph-mon[32231]: pgmap v961: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:05:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:44 smithi115 ceph-mon[35364]: pgmap v962: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:05:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:44 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 382.604623 secs 2024-07-26T08:05:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:44 smithi181 ceph-mon[35869]: pgmap v962: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:05:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:44 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 382.604623 secs 2024-07-26T08:05:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:44 smithi097 ceph-mon[32231]: pgmap v962: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:05:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:44 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 382.604623 secs 2024-07-26T08:05:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:46 smithi115 ceph-mon[35364]: pgmap v963: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:05:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:46 smithi181 ceph-mon[35869]: pgmap v963: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:05:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:46 smithi097 ceph-mon[32231]: pgmap v963: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:05:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:47.497+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:05:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:47.497+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:05:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:47.497+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:05:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:47.497+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:05:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:05:47.497+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:05:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:48 smithi115 ceph-mon[35364]: pgmap v964: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:05:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:48 smithi181 ceph-mon[35869]: pgmap v964: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:05:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:48 smithi097 ceph-mon[32231]: pgmap v964: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 178 B/s wr, 11 op/s 2024-07-26T08:05:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:49 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 387.604784 secs 2024-07-26T08:05:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:49 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 387.604784 secs 2024-07-26T08:05:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:49 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 387.604784 secs 2024-07-26T08:05:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:50 smithi115 ceph-mon[35364]: pgmap v965: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:50 smithi181 ceph-mon[35869]: pgmap v965: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:50 smithi097 ceph-mon[32231]: pgmap v965: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:05:52.251 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:51 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-07-26T08:05:52.251 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:51 smithi097 systemd[1]: Stopped Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:05:52.251 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:51 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Consumed 2.106s CPU time. 2024-07-26T08:05:52.251 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:51 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-07-26T08:05:52.252 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:51 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T08:05:52.252 INFO:journalctl@ceph.mds.d.smithi097.stdout:Jul 26 08:05:51 smithi097 systemd[1]: Failed to start Ceph mds.d for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:05:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:52 smithi115 ceph-mon[35364]: pgmap v966: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:52 smithi181 ceph-mon[35869]: pgmap v966: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:52 smithi097 ceph-mon[32231]: pgmap v966: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:05:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:54 smithi115 ceph-mon[35364]: pgmap v967: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:05:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:54 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 392.604927 secs 2024-07-26T08:05:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:54 smithi181 ceph-mon[35869]: pgmap v967: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:05:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:54 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 392.604927 secs 2024-07-26T08:05:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:54 smithi097 ceph-mon[32231]: pgmap v967: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:05:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:54 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 392.604927 secs 2024-07-26T08:05:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:56 smithi115 ceph-mon[35364]: pgmap v968: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T08:05:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:56 smithi181 ceph-mon[35869]: pgmap v968: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T08:05:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:56 smithi097 ceph-mon[32231]: pgmap v968: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T08:05:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:58 smithi115 ceph-mon[35364]: pgmap v969: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T08:05:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:58 smithi181 ceph-mon[35869]: pgmap v969: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T08:05:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:58 smithi097 ceph-mon[32231]: pgmap v969: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T08:06:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:59 smithi115 ceph-mon[35364]: Replacing daemon mds.d as rank 1 with standby daemon mds.j 2024-07-26T08:06:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:59 smithi115 ceph-mon[35364]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-07-26T08:06:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:59 smithi115 ceph-mon[35364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:06:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:59 smithi115 ceph-mon[35364]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T08:06:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:59 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:06:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:05:59 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 397.605064 secs 2024-07-26T08:06:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:59 smithi181 ceph-mon[35869]: Replacing daemon mds.d as rank 1 with standby daemon mds.j 2024-07-26T08:06:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:59 smithi181 ceph-mon[35869]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-07-26T08:06:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:59 smithi181 ceph-mon[35869]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:06:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:59 smithi181 ceph-mon[35869]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T08:06:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:59 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:06:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:05:59 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 397.605064 secs 2024-07-26T08:06:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:59 smithi097 ceph-mon[32231]: Replacing daemon mds.d as rank 1 with standby daemon mds.j 2024-07-26T08:06:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:59 smithi097 ceph-mon[32231]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-07-26T08:06:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:59 smithi097 ceph-mon[32231]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:06:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:59 smithi097 ceph-mon[32231]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T08:06:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:59 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:06:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:05:59 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 397.605064 secs 2024-07-26T08:06:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:01 smithi181 ceph-mon[35869]: pgmap v971: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:06:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:01 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:resolve 2024-07-26T08:06:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:01 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:06:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:01 smithi097 ceph-mon[32231]: pgmap v971: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:06:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:01 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:resolve 2024-07-26T08:06:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:01 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:06:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:01 smithi115 ceph-mon[35364]: pgmap v971: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:06:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:01 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:resolve 2024-07-26T08:06:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:01 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:06:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:02.499+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:06:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:02.501+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:06:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:02.501+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:06:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:02.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:06:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:02.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:06:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:02 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:reconnect 2024-07-26T08:06:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:02 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:06:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:02 smithi115 ceph-mon[35364]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:06:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:02 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:reconnect 2024-07-26T08:06:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:02 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:06:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:02 smithi181 ceph-mon[35869]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:06:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:02 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:reconnect 2024-07-26T08:06:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:02 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:06:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:02 smithi097 ceph-mon[32231]: reconnect by client.15360 192.168.0.1:0/1832020397 after 0.00699988 2024-07-26T08:06:03.646 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-mon[32231]: pgmap v972: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:06:03.647 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:rejoin 2024-07-26T08:06:03.647 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-mon[32231]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:06:03.647 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-mon[32231]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:06:03.647 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:06:03.647 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: *** Caught signal (Aborted) ** 2024-07-26T08:06:03.647 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: in thread 7f93bd800640 thread_name:ms_dispatch 2024-07-26T08:06:03.647 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f93c3a3e6f0] 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f93c3a8b94c] 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 3: raise() 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 4: abort() 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 5: /lib64/libc.so.6(+0x2871b) [0x7f93c3a2871b] 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 6: /lib64/libc.so.6(+0x37386) [0x7f93c3a37386] 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b8625216b9] 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b8626df80c] 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b86261f2ca] 2024-07-26T08:06:03.648 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b8625ef753] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b8625f0a1b] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b8625d4752] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b862522a56] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b862526637] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b862526f5e] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b86250a03a] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f93c459b407] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 18: (DispatchQueue::entry()+0x60f) [0x7f93c459bcf9] 2024-07-26T08:06:03.649 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f93c46785a1] 2024-07-26T08:06:03.650 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 20: (Thread::entry_wrapper()+0x43) [0x7f93c4425cdd] 2024-07-26T08:06:03.650 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 21: (Thread::_entry_func(void*)+0xd) [0x7f93c4425cf9] 2024-07-26T08:06:03.650 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 22: /lib64/libc.so.6(+0x89c02) [0x7f93c3a89c02] 2024-07-26T08:06:03.650 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f93c3b0ec40] 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2024-07-26T08:06:03.267+0000 7f93bd800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: in thread 7f93bd800640 thread_name:ms_dispatch 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f93c3a3e6f0] 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f93c3a8b94c] 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 3: raise() 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 4: abort() 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 5: /lib64/libc.so.6(+0x2871b) [0x7f93c3a2871b] 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 6: /lib64/libc.so.6(+0x37386) [0x7f93c3a37386] 2024-07-26T08:06:03.651 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b8625216b9] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b8626df80c] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b86261f2ca] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b8625ef753] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b8625f0a1b] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b8625d4752] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b862522a56] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b862526637] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b862526f5e] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b86250a03a] 2024-07-26T08:06:03.652 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f93c459b407] 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 18: (DispatchQueue::entry()+0x60f) [0x7f93c459bcf9] 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f93c46785a1] 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 20: (Thread::entry_wrapper()+0x43) [0x7f93c4425cdd] 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 21: (Thread::_entry_func(void*)+0xd) [0x7f93c4425cf9] 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 22: /lib64/libc.so.6(+0x89c02) [0x7f93c3a89c02] 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f93c3b0ec40] 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 0> 2024-07-26T08:06:03.267+0000 7f93bd800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: in thread 7f93bd800640 thread_name:ms_dispatch 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:06:03.653 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f93c3a3e6f0] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f93c3a8b94c] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 3: raise() 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 4: abort() 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 5: /lib64/libc.so.6(+0x2871b) [0x7f93c3a2871b] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 6: /lib64/libc.so.6(+0x37386) [0x7f93c3a37386] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b8625216b9] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b8626df80c] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b86261f2ca] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b8625ef753] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b8625f0a1b] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b8625d4752] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b862522a56] 2024-07-26T08:06:03.654 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b862526637] 2024-07-26T08:06:03.655 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b862526f5e] 2024-07-26T08:06:03.655 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b86250a03a] 2024-07-26T08:06:03.655 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f93c459b407] 2024-07-26T08:06:03.655 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 18: (DispatchQueue::entry()+0x60f) [0x7f93c459bcf9] 2024-07-26T08:06:03.655 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f93c46785a1] 2024-07-26T08:06:03.655 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 20: (Thread::entry_wrapper()+0x43) [0x7f93c4425cdd] 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 21: (Thread::_entry_func(void*)+0xd) [0x7f93c4425cf9] 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 22: /lib64/libc.so.6(+0x89c02) [0x7f93c3a89c02] 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f93c3b0ec40] 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: -7> 2024-07-26T08:06:03.267+0000 7f93bd800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: in thread 7f93bd800640 thread_name:ms_dispatch 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f93c3a3e6f0] 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f93c3a8b94c] 2024-07-26T08:06:03.656 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 3: raise() 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 4: abort() 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 5: /lib64/libc.so.6(+0x2871b) [0x7f93c3a2871b] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 6: /lib64/libc.so.6(+0x37386) [0x7f93c3a37386] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b8625216b9] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b8626df80c] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b86261f2ca] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b8625ef753] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b8625f0a1b] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b8625d4752] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b862522a56] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b862526637] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b862526f5e] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b86250a03a] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f93c459b407] 2024-07-26T08:06:03.657 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 18: (DispatchQueue::entry()+0x60f) [0x7f93c459bcf9] 2024-07-26T08:06:03.658 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f93c46785a1] 2024-07-26T08:06:03.658 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 20: (Thread::entry_wrapper()+0x43) [0x7f93c4425cdd] 2024-07-26T08:06:03.658 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 21: (Thread::_entry_func(void*)+0xd) [0x7f93c4425cf9] 2024-07-26T08:06:03.658 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 22: /lib64/libc.so.6(+0x89c02) [0x7f93c3a89c02] 2024-07-26T08:06:03.658 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f93c3b0ec40] 2024-07-26T08:06:03.658 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:06:03.658 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j[196012]: 2024-07-26T08:06:03.658 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 podman[196100]: 2024-07-26 08:06:03.385296449 +0000 UTC m=+0.034877987 container died 17a6f27414c40711955ac46fce9ab2f5972189199935132f968df60bf623ac9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:06:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:03 smithi115 ceph-mon[35364]: pgmap v972: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:06:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:03 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:rejoin 2024-07-26T08:06:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:03 smithi115 ceph-mon[35364]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:06:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:03 smithi115 ceph-mon[35364]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:06:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:03 smithi181 ceph-mon[35869]: pgmap v972: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T08:06:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:03 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:rejoin 2024-07-26T08:06:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:03 smithi181 ceph-mon[35869]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:06:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:03 smithi181 ceph-mon[35869]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-07-26T08:06:04.251 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 podman[196100]: 2024-07-26 08:06:03.979891118 +0000 UTC m=+0.629472677 container remove 17a6f27414c40711955ac46fce9ab2f5972189199935132f968df60bf623ac9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-j, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:06:04.251 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:03 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:06:04.536 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:04 smithi097 ceph-mon[32231]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:06:04.536 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:04 smithi097 ceph-mon[32231]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:active 2024-07-26T08:06:04.536 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:04 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:06:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:04 smithi115 ceph-mon[35364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:06:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:04 smithi115 ceph-mon[35364]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:active 2024-07-26T08:06:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:04 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:06:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:04 smithi181 ceph-mon[35869]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:06:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:04 smithi181 ceph-mon[35869]: mds.? [v2:172.21.15.97:6834/4143030320,v1:172.21.15.97:6835/4143030320] up:active 2024-07-26T08:06:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:04 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:06:05.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:04 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:06:05.001 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:04 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.154s CPU time. 2024-07-26T08:06:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:05 smithi181 ceph-mon[35869]: pgmap v973: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:06:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 402.605217 secs 2024-07-26T08:06:05.483 INFO:journalctl@ceph.mds.c.smithi181.stdout:Jul 26 08:06:05 smithi181 ceph-9621a072-4b21-11ef-bcb6-c7b262605968-mds-c[68894]: 2024-07-26T08:06:05.078+0000 7fcfc3400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:06:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:05 smithi115 ceph-mon[35364]: pgmap v973: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:06:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 402.605217 secs 2024-07-26T08:06:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:05 smithi097 ceph-mon[32231]: pgmap v973: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:06:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 402.605217 secs 2024-07-26T08:06:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:07 smithi115 ceph-mon[35364]: pgmap v974: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:06:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:07 smithi181 ceph-mon[35869]: pgmap v974: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:06:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:07 smithi097 ceph-mon[32231]: pgmap v974: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:06:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:09 smithi115 ceph-mon[35364]: pgmap v975: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:06:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:09 smithi181 ceph-mon[35869]: pgmap v975: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:06:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:09 smithi097 ceph-mon[32231]: pgmap v975: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:06:10.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 407.605357 secs 2024-07-26T08:06:10.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 407.605357 secs 2024-07-26T08:06:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 407.605357 secs 2024-07-26T08:06:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:11 smithi115 ceph-mon[35364]: pgmap v976: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 193 B/s wr, 10 op/s 2024-07-26T08:06:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:11 smithi181 ceph-mon[35869]: pgmap v976: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 193 B/s wr, 10 op/s 2024-07-26T08:06:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:11 smithi097 ceph-mon[32231]: pgmap v976: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 193 B/s wr, 10 op/s 2024-07-26T08:06:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:12 smithi115 ceph-mon[35364]: pgmap v977: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T08:06:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:12 smithi181 ceph-mon[35869]: pgmap v977: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T08:06:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:12 smithi097 ceph-mon[32231]: pgmap v977: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T08:06:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:14 smithi115 ceph-mon[35364]: pgmap v978: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:06:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:14 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 412.605451 secs 2024-07-26T08:06:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:14 smithi181 ceph-mon[35869]: pgmap v978: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:06:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:14 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 412.605451 secs 2024-07-26T08:06:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:14 smithi097 ceph-mon[32231]: pgmap v978: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:06:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:14 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 412.605451 secs 2024-07-26T08:06:15.251 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:14 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-07-26T08:06:15.251 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:14 smithi097 systemd[1]: Stopped Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:06:15.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:14 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Consumed 2.154s CPU time. 2024-07-26T08:06:15.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:14 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-07-26T08:06:15.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:14 smithi097 systemd[1]: ceph-9621a072-4b21-11ef-bcb6-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:06:15.252 INFO:journalctl@ceph.mds.j.smithi097.stdout:Jul 26 08:06:14 smithi097 systemd[1]: Failed to start Ceph mds.j for 9621a072-4b21-11ef-bcb6-c7b262605968. 2024-07-26T08:06:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:16 smithi115 ceph-mon[35364]: pgmap v979: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:06:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:16 smithi181 ceph-mon[35869]: pgmap v979: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:06:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:16 smithi097 ceph-mon[32231]: pgmap v979: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:06:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:17.510+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:06:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:17.511+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:06:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:17.511+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:06:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:17.511+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:06:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:17.511+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:06:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:18 smithi115 ceph-mon[35364]: pgmap v980: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:18 smithi181 ceph-mon[35869]: pgmap v980: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:18 smithi097 ceph-mon[32231]: pgmap v980: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:20 smithi097 ceph-mon[32231]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:06:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 417.605597 secs 2024-07-26T08:06:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:20 smithi115 ceph-mon[35364]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:06:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 417.605597 secs 2024-07-26T08:06:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:20 smithi181 ceph-mon[35869]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:06:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 417.605597 secs 2024-07-26T08:06:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:21 smithi097 ceph-mon[32231]: pgmap v981: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:21 smithi115 ceph-mon[35364]: pgmap v981: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:21 smithi181 ceph-mon[35869]: pgmap v981: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:21.948+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0005a20 con 0x7ff3f01061a0 2024-07-26T08:06:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:21.949+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c0005a20 con 0x7ff3f01061a0 2024-07-26T08:06:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:21.950+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc005b10 con 0x55d25070b2f0 2024-07-26T08:06:21.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:21.952+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc005b10 con 0x55d25070b2f0 2024-07-26T08:06:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:23 smithi097 ceph-mon[32231]: pgmap v982: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:23 smithi115 ceph-mon[35364]: pgmap v982: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:23 smithi181 ceph-mon[35869]: pgmap v982: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:25 smithi097 ceph-mon[32231]: pgmap v983: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 422.605687 secs 2024-07-26T08:06:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:25 smithi115 ceph-mon[35364]: pgmap v983: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 422.605687 secs 2024-07-26T08:06:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:25 smithi181 ceph-mon[35869]: pgmap v983: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 422.605687 secs 2024-07-26T08:06:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:06:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:06:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: pgmap v984: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:06:27.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:06:27.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: pgmap v984: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:06:27.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:06:27.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:06:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: pgmap v984: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:06:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:06:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:06:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:06:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:28 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:06:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:28 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:06:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:28 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:06:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:28 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:06:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:28 smithi115 ceph-mon[35364]: pgmap v985: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:06:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:28 smithi115 ceph-mon[35364]: pgmap v986: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:06:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:28 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:06:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:28 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:06:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:28 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:06:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:28 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:06:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:28 smithi181 ceph-mon[35869]: pgmap v985: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:06:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:28 smithi181 ceph-mon[35869]: pgmap v986: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:06:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:28 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:06:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:28 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:06:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:28 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:06:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:28 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:06:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:28 smithi097 ceph-mon[32231]: pgmap v985: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:06:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:28 smithi097 ceph-mon[32231]: pgmap v986: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:06:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:30 smithi115 ceph-mon[35364]: pgmap v987: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 427.605810 secs 2024-07-26T08:06:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:30 smithi181 ceph-mon[35869]: pgmap v987: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 427.605810 secs 2024-07-26T08:06:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:30 smithi097 ceph-mon[32231]: pgmap v987: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 427.605810 secs 2024-07-26T08:06:32.516 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:32.514+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:06:32.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:32.516+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:06:32.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:32.516+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:06:32.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:32.516+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:06:32.518 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:32.516+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:06:32.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:32 smithi115 ceph-mon[35364]: pgmap v988: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:32 smithi181 ceph-mon[35869]: pgmap v988: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:32 smithi097 ceph-mon[32231]: pgmap v988: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:34 smithi115 ceph-mon[35364]: pgmap v989: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:34 smithi181 ceph-mon[35869]: pgmap v989: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:34 smithi097 ceph-mon[32231]: pgmap v989: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 432.605943 secs 2024-07-26T08:06:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 432.605943 secs 2024-07-26T08:06:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 432.605943 secs 2024-07-26T08:06:36.682 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:36 smithi097 ceph-mon[32231]: pgmap v990: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:36 smithi115 ceph-mon[35364]: pgmap v990: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:36 smithi181 ceph-mon[35869]: pgmap v990: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:38 smithi097 ceph-mon[32231]: pgmap v991: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:06:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:38 smithi115 ceph-mon[35364]: pgmap v991: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:06:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:38 smithi181 ceph-mon[35869]: pgmap v991: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:06:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:40 smithi115 ceph-mon[35364]: pgmap v992: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:06:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 437.606078 secs 2024-07-26T08:06:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:40 smithi181 ceph-mon[35869]: pgmap v992: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:06:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 437.606078 secs 2024-07-26T08:06:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:40 smithi097 ceph-mon[32231]: pgmap v992: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:06:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 437.606078 secs 2024-07-26T08:06:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:41 smithi115 ceph-mon[35364]: pgmap v993: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:41 smithi181 ceph-mon[35869]: pgmap v993: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:41 smithi097 ceph-mon[32231]: pgmap v993: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:44 smithi115 ceph-mon[35364]: pgmap v994: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:44 smithi181 ceph-mon[35869]: pgmap v994: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:44 smithi097 ceph-mon[32231]: pgmap v994: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 442.606222 secs 2024-07-26T08:06:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 442.606222 secs 2024-07-26T08:06:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 442.606222 secs 2024-07-26T08:06:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:46 smithi115 ceph-mon[35364]: pgmap v995: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:46 smithi181 ceph-mon[35869]: pgmap v995: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:46 smithi097 ceph-mon[32231]: pgmap v995: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:47.524+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:06:47.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:47.525+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:06:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:47.525+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:06:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:47.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:06:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:06:47.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:06:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:48 smithi115 ceph-mon[35364]: pgmap v996: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:48 smithi181 ceph-mon[35869]: pgmap v996: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:48 smithi097 ceph-mon[32231]: pgmap v996: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:50 smithi115 ceph-mon[35364]: pgmap v997: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:06:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 447.606359 secs 2024-07-26T08:06:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:50 smithi181 ceph-mon[35869]: pgmap v997: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:06:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 447.606359 secs 2024-07-26T08:06:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:50 smithi097 ceph-mon[32231]: pgmap v997: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:06:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 447.606359 secs 2024-07-26T08:06:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:51 smithi115 ceph-mon[35364]: pgmap v998: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:51 smithi181 ceph-mon[35869]: pgmap v998: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:51 smithi097 ceph-mon[32231]: pgmap v998: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:06:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:54 smithi115 ceph-mon[35364]: pgmap v999: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:54 smithi181 ceph-mon[35869]: pgmap v999: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:54 smithi097 ceph-mon[32231]: pgmap v999: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:06:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 452.606506 secs 2024-07-26T08:06:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 452.606506 secs 2024-07-26T08:06:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 452.606506 secs 2024-07-26T08:06:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:56 smithi115 ceph-mon[35364]: pgmap v1000: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:56 smithi181 ceph-mon[35869]: pgmap v1000: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:56 smithi097 ceph-mon[32231]: pgmap v1000: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:06:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:58 smithi115 ceph-mon[35364]: pgmap v1001: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:58 smithi181 ceph-mon[35869]: pgmap v1001: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:58 smithi097 ceph-mon[32231]: pgmap v1001: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:06:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:06:59 smithi115 ceph-mon[35364]: pgmap v1002: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:06:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:06:59 smithi181 ceph-mon[35869]: pgmap v1002: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:06:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:06:59 smithi097 ceph-mon[32231]: pgmap v1002: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 457.606654 secs 2024-07-26T08:07:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 457.606654 secs 2024-07-26T08:07:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 457.606654 secs 2024-07-26T08:07:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:01 smithi115 ceph-mon[35364]: pgmap v1003: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:01 smithi181 ceph-mon[35869]: pgmap v1003: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:01 smithi097 ceph-mon[32231]: pgmap v1003: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:02.529+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:07:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:02.530+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:07:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:02.530+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:07:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:02.530+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:07:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:02.530+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:07:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:04 smithi115 ceph-mon[35364]: pgmap v1004: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:04 smithi181 ceph-mon[35869]: pgmap v1004: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:04 smithi097 ceph-mon[32231]: pgmap v1004: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:05.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 462.606795 secs 2024-07-26T08:07:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 462.606795 secs 2024-07-26T08:07:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 462.606795 secs 2024-07-26T08:07:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:06 smithi115 ceph-mon[35364]: pgmap v1005: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:06 smithi181 ceph-mon[35869]: pgmap v1005: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:06 smithi097 ceph-mon[32231]: pgmap v1005: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:06.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:06.956+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0005d30 con 0x7ff3f01061a0 2024-07-26T08:07:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:06.956+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0005d30 con 0x7ff3f01061a0 2024-07-26T08:07:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:06.957+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc005e10 con 0x55d25070b2f0 2024-07-26T08:07:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:06.958+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc005e10 con 0x55d25070b2f0 2024-07-26T08:07:08.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:08 smithi115 ceph-mon[35364]: pgmap v1006: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:08 smithi181 ceph-mon[35869]: pgmap v1006: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:08 smithi097 ceph-mon[32231]: pgmap v1006: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:09 smithi115 ceph-mon[35364]: pgmap v1007: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:09 smithi181 ceph-mon[35869]: pgmap v1007: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:09 smithi097 ceph-mon[32231]: pgmap v1007: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:10.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 467.606965 secs 2024-07-26T08:07:10.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 467.606965 secs 2024-07-26T08:07:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 467.606965 secs 2024-07-26T08:07:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:11 smithi115 ceph-mon[35364]: pgmap v1008: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:11 smithi181 ceph-mon[35869]: pgmap v1008: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:11 smithi097 ceph-mon[32231]: pgmap v1008: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:14 smithi115 ceph-mon[35364]: pgmap v1009: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:14 smithi181 ceph-mon[35869]: pgmap v1009: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:14 smithi097 ceph-mon[32231]: pgmap v1009: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 472.607122 secs 2024-07-26T08:07:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 472.607122 secs 2024-07-26T08:07:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 472.607122 secs 2024-07-26T08:07:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:16 smithi115 ceph-mon[35364]: pgmap v1010: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:16 smithi181 ceph-mon[35869]: pgmap v1010: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:16 smithi097 ceph-mon[32231]: pgmap v1010: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:17.538+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:07:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:17.539+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:07:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:17.539+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:07:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:17.539+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:07:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:17.539+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:07:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:17 smithi115 ceph-mon[35364]: pgmap v1011: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:17 smithi181 ceph-mon[35869]: pgmap v1011: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:17 smithi097 ceph-mon[32231]: pgmap v1011: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:20 smithi115 ceph-mon[35364]: pgmap v1012: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 477.607271 secs 2024-07-26T08:07:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:20 smithi181 ceph-mon[35869]: pgmap v1012: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 477.607271 secs 2024-07-26T08:07:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:20 smithi097 ceph-mon[32231]: pgmap v1012: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 477.607271 secs 2024-07-26T08:07:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:22 smithi115 ceph-mon[35364]: pgmap v1013: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:22 smithi181 ceph-mon[35869]: pgmap v1013: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:22 smithi097 ceph-mon[32231]: pgmap v1013: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:24 smithi115 ceph-mon[35364]: pgmap v1014: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:24 smithi181 ceph-mon[35869]: pgmap v1014: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:24 smithi097 ceph-mon[32231]: pgmap v1014: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 482.607418 secs 2024-07-26T08:07:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 482.607418 secs 2024-07-26T08:07:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 482.607418 secs 2024-07-26T08:07:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:26 smithi115 ceph-mon[35364]: pgmap v1015: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:26 smithi181 ceph-mon[35869]: pgmap v1015: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:26 smithi097 ceph-mon[32231]: pgmap v1015: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:07:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:27 smithi115 ceph-mon[35364]: pgmap v1016: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:27 smithi181 ceph-mon[35869]: pgmap v1016: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:27.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:27 smithi097 ceph-mon[32231]: pgmap v1016: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:07:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:07:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:07:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: pgmap v1017: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: pgmap v1018: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:29.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: pgmap v1017: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:29.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: pgmap v1018: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:29.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:07:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:07:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:07:29.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:07:29.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: pgmap v1017: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:29.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: pgmap v1018: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:29.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:07:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 487.607524 secs 2024-07-26T08:07:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 487.607524 secs 2024-07-26T08:07:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 487.607524 secs 2024-07-26T08:07:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:31 smithi115 ceph-mon[35364]: pgmap v1019: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:31 smithi181 ceph-mon[35869]: pgmap v1019: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:31 smithi097 ceph-mon[32231]: pgmap v1019: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:32.544 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:32.542+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:07:32.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:32.544+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:07:32.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:32.544+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:07:32.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:32.544+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:07:32.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:32.544+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:07:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:34 smithi097 ceph-mon[32231]: pgmap v1020: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:34 smithi115 ceph-mon[35364]: pgmap v1020: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:34 smithi181 ceph-mon[35869]: pgmap v1020: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 492.607655 secs 2024-07-26T08:07:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 492.607655 secs 2024-07-26T08:07:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 492.607655 secs 2024-07-26T08:07:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:36 smithi115 ceph-mon[35364]: pgmap v1021: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:36 smithi181 ceph-mon[35869]: pgmap v1021: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:36 smithi097 ceph-mon[32231]: pgmap v1021: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:38 smithi115 ceph-mon[35364]: pgmap v1022: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:38 smithi181 ceph-mon[35869]: pgmap v1022: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:38 smithi097 ceph-mon[32231]: pgmap v1022: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:07:40.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:40 smithi115 ceph-mon[35364]: pgmap v1023: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:07:40.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 497.607787 secs 2024-07-26T08:07:40.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:40 smithi181 ceph-mon[35869]: pgmap v1023: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:07:40.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 497.607787 secs 2024-07-26T08:07:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:40 smithi097 ceph-mon[32231]: pgmap v1023: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:07:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 497.607787 secs 2024-07-26T08:07:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:42 smithi115 ceph-mon[35364]: pgmap v1024: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:42 smithi181 ceph-mon[35869]: pgmap v1024: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:42 smithi097 ceph-mon[32231]: pgmap v1024: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:44 smithi115 ceph-mon[35364]: pgmap v1025: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:44 smithi181 ceph-mon[35869]: pgmap v1025: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:44 smithi097 ceph-mon[32231]: pgmap v1025: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:07:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 502.607922 secs 2024-07-26T08:07:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 502.607922 secs 2024-07-26T08:07:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 502.607922 secs 2024-07-26T08:07:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:46 smithi115 ceph-mon[35364]: pgmap v1026: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:07:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:46 smithi181 ceph-mon[35869]: pgmap v1026: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:07:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:46 smithi097 ceph-mon[32231]: pgmap v1026: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:07:47.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:47.552+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:07:47.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:47.553+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:07:47.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:47.553+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:07:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:47.553+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:07:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:07:47.553+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:07:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:48 smithi115 ceph-mon[35364]: pgmap v1027: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:07:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:48 smithi181 ceph-mon[35869]: pgmap v1027: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:07:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:48 smithi097 ceph-mon[32231]: pgmap v1027: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:07:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:50 smithi115 ceph-mon[35364]: pgmap v1028: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:07:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 507.608078 secs 2024-07-26T08:07:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:50 smithi181 ceph-mon[35869]: pgmap v1028: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:07:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 507.608078 secs 2024-07-26T08:07:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:50 smithi097 ceph-mon[32231]: pgmap v1028: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:07:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 507.608078 secs 2024-07-26T08:07:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:52 smithi115 ceph-mon[35364]: pgmap v1029: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:52 smithi181 ceph-mon[35869]: pgmap v1029: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:52 smithi097 ceph-mon[32231]: pgmap v1029: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:07:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:54 smithi115 ceph-mon[35364]: pgmap v1030: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:07:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:54 smithi181 ceph-mon[35869]: pgmap v1030: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:07:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:54 smithi097 ceph-mon[32231]: pgmap v1030: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:07:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 512.608224 secs 2024-07-26T08:07:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 512.608224 secs 2024-07-26T08:07:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 512.608224 secs 2024-07-26T08:07:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:56 smithi115 ceph-mon[35364]: pgmap v1031: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:07:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:56 smithi181 ceph-mon[35869]: pgmap v1031: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:07:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:56 smithi097 ceph-mon[32231]: pgmap v1031: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:07:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:58 smithi115 ceph-mon[35364]: pgmap v1032: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:58 smithi181 ceph-mon[35869]: pgmap v1032: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:58 smithi097 ceph-mon[32231]: pgmap v1032: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:07:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:07:59 smithi115 ceph-mon[35364]: pgmap v1033: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:07:59 smithi181 ceph-mon[35869]: pgmap v1033: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:07:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:07:59 smithi097 ceph-mon[32231]: pgmap v1033: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 517.608365 secs 2024-07-26T08:08:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 517.608365 secs 2024-07-26T08:08:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 517.608365 secs 2024-07-26T08:08:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:01 smithi115 ceph-mon[35364]: pgmap v1034: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:01 smithi181 ceph-mon[35869]: pgmap v1034: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:01 smithi097 ceph-mon[32231]: pgmap v1034: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:02.557+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:08:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:02.558+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:08:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:02.558+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:08:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:02.558+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:08:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:02.558+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:08:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:04 smithi115 ceph-mon[35364]: pgmap v1035: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:04 smithi181 ceph-mon[35869]: pgmap v1035: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:04 smithi097 ceph-mon[32231]: pgmap v1035: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:05.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 522.608510 secs 2024-07-26T08:08:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 522.608510 secs 2024-07-26T08:08:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 522.608510 secs 2024-07-26T08:08:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:06 smithi115 ceph-mon[35364]: pgmap v1036: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:06 smithi181 ceph-mon[35869]: pgmap v1036: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:06 smithi097 ceph-mon[32231]: pgmap v1036: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:08 smithi115 ceph-mon[35364]: pgmap v1037: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:08 smithi181 ceph-mon[35869]: pgmap v1037: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:08 smithi097 ceph-mon[32231]: pgmap v1037: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:10.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:10 smithi181 ceph-mon[35869]: pgmap v1038: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:10.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 527.608600 secs 2024-07-26T08:08:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:10 smithi115 ceph-mon[35364]: pgmap v1038: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 527.608600 secs 2024-07-26T08:08:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:10 smithi097 ceph-mon[32231]: pgmap v1038: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 527.608600 secs 2024-07-26T08:08:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:12 smithi115 ceph-mon[35364]: pgmap v1039: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:12 smithi181 ceph-mon[35869]: pgmap v1039: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:12 smithi097 ceph-mon[32231]: pgmap v1039: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:14 smithi115 ceph-mon[35364]: pgmap v1040: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:14 smithi181 ceph-mon[35869]: pgmap v1040: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:14 smithi097 ceph-mon[32231]: pgmap v1040: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 532.608740 secs 2024-07-26T08:08:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 532.608740 secs 2024-07-26T08:08:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 532.608740 secs 2024-07-26T08:08:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:16 smithi115 ceph-mon[35364]: pgmap v1041: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:16 smithi181 ceph-mon[35869]: pgmap v1041: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:16 smithi097 ceph-mon[32231]: pgmap v1041: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:17.567+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:08:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:17.568+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:08:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:17.568+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:08:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:17.568+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:08:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:17.568+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:08:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:18 smithi115 ceph-mon[35364]: pgmap v1042: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:18 smithi181 ceph-mon[35869]: pgmap v1042: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:18 smithi097 ceph-mon[32231]: pgmap v1042: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:20 smithi115 ceph-mon[35364]: pgmap v1043: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 537.608881 secs 2024-07-26T08:08:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:20 smithi181 ceph-mon[35869]: pgmap v1043: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 537.608881 secs 2024-07-26T08:08:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:20 smithi097 ceph-mon[32231]: pgmap v1043: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 537.608881 secs 2024-07-26T08:08:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:22 smithi115 ceph-mon[35364]: pgmap v1044: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:22 smithi181 ceph-mon[35869]: pgmap v1044: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:22 smithi097 ceph-mon[32231]: pgmap v1044: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:24 smithi115 ceph-mon[35364]: pgmap v1045: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:24 smithi181 ceph-mon[35869]: pgmap v1045: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:24 smithi097 ceph-mon[32231]: pgmap v1045: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 542.609028 secs 2024-07-26T08:08:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 542.609028 secs 2024-07-26T08:08:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 542.609028 secs 2024-07-26T08:08:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:26 smithi115 ceph-mon[35364]: pgmap v1046: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:26 smithi181 ceph-mon[35869]: pgmap v1046: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:26 smithi097 ceph-mon[32231]: pgmap v1046: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:27 smithi115 ceph-mon[35364]: pgmap v1047: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:27 smithi181 ceph-mon[35869]: pgmap v1047: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:27 smithi097 ceph-mon[32231]: pgmap v1047: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:30 smithi115 ceph-mon[35364]: pgmap v1048: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:08:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 547.609146 secs 2024-07-26T08:08:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:30 smithi181 ceph-mon[35869]: pgmap v1048: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:08:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 547.609146 secs 2024-07-26T08:08:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:30 smithi097 ceph-mon[32231]: pgmap v1048: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:08:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 547.609146 secs 2024-07-26T08:08:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: pgmap v1049: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: pgmap v1050: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: pgmap v1051: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:08:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:08:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: pgmap v1049: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: pgmap v1050: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:08:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: pgmap v1051: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:08:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: pgmap v1049: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: pgmap v1050: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: pgmap v1051: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:08:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:32.572+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:08:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:32.573+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:08:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:32.573+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:08:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:32.573+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:08:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:32.573+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:08:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:34 smithi115 ceph-mon[35364]: pgmap v1052: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:34 smithi181 ceph-mon[35869]: pgmap v1052: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:34 smithi097 ceph-mon[32231]: pgmap v1052: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 552.609307 secs 2024-07-26T08:08:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 552.609307 secs 2024-07-26T08:08:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 552.609307 secs 2024-07-26T08:08:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:36 smithi115 ceph-mon[35364]: pgmap v1053: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:08:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:36 smithi181 ceph-mon[35869]: pgmap v1053: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:08:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:36 smithi097 ceph-mon[32231]: pgmap v1053: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:08:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:38 smithi115 ceph-mon[35364]: pgmap v1054: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T08:08:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:38 smithi181 ceph-mon[35869]: pgmap v1054: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T08:08:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:38 smithi097 ceph-mon[32231]: pgmap v1054: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T08:08:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:40 smithi115 ceph-mon[35364]: pgmap v1055: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:08:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 557.609466 secs 2024-07-26T08:08:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:40 smithi181 ceph-mon[35869]: pgmap v1055: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:08:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 557.609466 secs 2024-07-26T08:08:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:40 smithi097 ceph-mon[32231]: pgmap v1055: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:08:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 557.609466 secs 2024-07-26T08:08:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:42 smithi115 ceph-mon[35364]: pgmap v1056: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:08:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:42 smithi181 ceph-mon[35869]: pgmap v1056: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:08:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:42 smithi097 ceph-mon[32231]: pgmap v1056: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:08:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:44 smithi115 ceph-mon[35364]: pgmap v1057: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:44 smithi181 ceph-mon[35869]: pgmap v1057: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:44 smithi097 ceph-mon[32231]: pgmap v1057: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 562.609608 secs 2024-07-26T08:08:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 562.609608 secs 2024-07-26T08:08:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 562.609608 secs 2024-07-26T08:08:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:46 smithi115 ceph-mon[35364]: pgmap v1058: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:46 smithi181 ceph-mon[35869]: pgmap v1058: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:46 smithi097 ceph-mon[32231]: pgmap v1058: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:47.581+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:08:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:47.582+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:08:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:47.582+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:08:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:47.582+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:08:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:08:47.583+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:08:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:48 smithi115 ceph-mon[35364]: pgmap v1059: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:48 smithi181 ceph-mon[35869]: pgmap v1059: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:48 smithi097 ceph-mon[32231]: pgmap v1059: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:50 smithi115 ceph-mon[35364]: pgmap v1060: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 567.609749 secs 2024-07-26T08:08:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:50 smithi181 ceph-mon[35869]: pgmap v1060: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 567.609749 secs 2024-07-26T08:08:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:50 smithi097 ceph-mon[32231]: pgmap v1060: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:08:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 567.609749 secs 2024-07-26T08:08:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:52 smithi115 ceph-mon[35364]: pgmap v1061: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:52 smithi181 ceph-mon[35869]: pgmap v1061: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:52 smithi097 ceph-mon[32231]: pgmap v1061: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:08:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:54 smithi115 ceph-mon[35364]: pgmap v1062: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:54 smithi181 ceph-mon[35869]: pgmap v1062: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:54 smithi097 ceph-mon[32231]: pgmap v1062: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:08:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 572.609898 secs 2024-07-26T08:08:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:55 smithi115 ceph-mon[35364]: pgmap v1063: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 572.609898 secs 2024-07-26T08:08:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:55 smithi181 ceph-mon[35869]: pgmap v1063: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 572.609898 secs 2024-07-26T08:08:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:55 smithi097 ceph-mon[32231]: pgmap v1063: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:08:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:08:58 smithi115 ceph-mon[35364]: pgmap v1064: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:08:58 smithi181 ceph-mon[35869]: pgmap v1064: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:08:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:08:58 smithi097 ceph-mon[32231]: pgmap v1064: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:00 smithi115 ceph-mon[35364]: pgmap v1065: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 577.610045 secs 2024-07-26T08:09:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:00 smithi181 ceph-mon[35869]: pgmap v1065: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 577.610045 secs 2024-07-26T08:09:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:00 smithi097 ceph-mon[32231]: pgmap v1065: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 577.610045 secs 2024-07-26T08:09:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:02.585+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:09:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:02.586+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:09:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:02.586+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:09:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:02.586+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:09:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:02.586+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:09:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:02 smithi115 ceph-mon[35364]: pgmap v1066: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:02 smithi181 ceph-mon[35869]: pgmap v1066: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:02 smithi097 ceph-mon[32231]: pgmap v1066: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:04 smithi115 ceph-mon[35364]: pgmap v1067: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:04 smithi181 ceph-mon[35869]: pgmap v1067: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:04 smithi097 ceph-mon[32231]: pgmap v1067: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:05.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 582.610193 secs 2024-07-26T08:09:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 582.610193 secs 2024-07-26T08:09:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 582.610193 secs 2024-07-26T08:09:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:06 smithi115 ceph-mon[35364]: pgmap v1068: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:06 smithi181 ceph-mon[35869]: pgmap v1068: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:06 smithi097 ceph-mon[32231]: pgmap v1068: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:08 smithi115 ceph-mon[35364]: pgmap v1069: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:08 smithi181 ceph-mon[35869]: pgmap v1069: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:08 smithi097 ceph-mon[32231]: pgmap v1069: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:10 smithi115 ceph-mon[35364]: pgmap v1070: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 587.610366 secs 2024-07-26T08:09:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:10 smithi181 ceph-mon[35869]: pgmap v1070: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 587.610366 secs 2024-07-26T08:09:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:10 smithi097 ceph-mon[32231]: pgmap v1070: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 587.610366 secs 2024-07-26T08:09:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:12 smithi115 ceph-mon[35364]: pgmap v1071: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:12 smithi181 ceph-mon[35869]: pgmap v1071: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:12 smithi097 ceph-mon[32231]: pgmap v1071: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:14 smithi115 ceph-mon[35364]: pgmap v1072: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:14 smithi181 ceph-mon[35869]: pgmap v1072: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:14 smithi097 ceph-mon[32231]: pgmap v1072: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 592.610464 secs 2024-07-26T08:09:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 592.610464 secs 2024-07-26T08:09:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 592.610464 secs 2024-07-26T08:09:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:16 smithi115 ceph-mon[35364]: pgmap v1073: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:16 smithi181 ceph-mon[35869]: pgmap v1073: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:16 smithi097 ceph-mon[32231]: pgmap v1073: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:17.595 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:17.594+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:09:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:17.596+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:09:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:17.596+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:09:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:17.596+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:09:17.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:17.596+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:09:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:18 smithi115 ceph-mon[35364]: pgmap v1074: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:18 smithi181 ceph-mon[35869]: pgmap v1074: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:18 smithi097 ceph-mon[32231]: pgmap v1074: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:19 smithi115 ceph-mon[35364]: pgmap v1075: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:19 smithi181 ceph-mon[35869]: pgmap v1075: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:19 smithi097 ceph-mon[32231]: pgmap v1075: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 597.610564 secs 2024-07-26T08:09:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 597.610564 secs 2024-07-26T08:09:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 597.610564 secs 2024-07-26T08:09:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:21 smithi115 ceph-mon[35364]: pgmap v1076: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:21 smithi181 ceph-mon[35869]: pgmap v1076: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:21 smithi097 ceph-mon[32231]: pgmap v1076: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:24 smithi115 ceph-mon[35364]: pgmap v1077: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:24 smithi181 ceph-mon[35869]: pgmap v1077: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:24 smithi097 ceph-mon[32231]: pgmap v1077: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 602.610703 secs 2024-07-26T08:09:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 602.610703 secs 2024-07-26T08:09:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 602.610703 secs 2024-07-26T08:09:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:26 smithi115 ceph-mon[35364]: pgmap v1078: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:26 smithi181 ceph-mon[35869]: pgmap v1078: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:26 smithi097 ceph-mon[32231]: pgmap v1078: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:28 smithi115 ceph-mon[35364]: pgmap v1079: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:28 smithi181 ceph-mon[35869]: pgmap v1079: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:28 smithi097 ceph-mon[32231]: pgmap v1079: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:30 smithi115 ceph-mon[35364]: pgmap v1080: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 607.610839 secs 2024-07-26T08:09:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:30 smithi181 ceph-mon[35869]: pgmap v1080: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 607.610839 secs 2024-07-26T08:09:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:30 smithi097 ceph-mon[32231]: pgmap v1080: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 607.610839 secs 2024-07-26T08:09:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:32.600+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:09:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:32.601+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:09:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:32.601+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:09:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:32.601+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:09:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:32.601+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:09:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:32 smithi115 ceph-mon[35364]: pgmap v1081: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:09:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:32 smithi181 ceph-mon[35869]: pgmap v1081: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:09:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:32 smithi097 ceph-mon[32231]: pgmap v1081: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:09:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:09:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:09:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:09:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:09:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:09:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:09:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:09:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:34 smithi115 ceph-mon[35364]: pgmap v1082: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:34 smithi115 ceph-mon[35364]: pgmap v1083: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:34 smithi181 ceph-mon[35869]: pgmap v1082: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:34 smithi181 ceph-mon[35869]: pgmap v1083: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:34 smithi097 ceph-mon[32231]: pgmap v1082: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:34 smithi097 ceph-mon[32231]: pgmap v1083: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:35.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 612.610979 secs 2024-07-26T08:09:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 612.610979 secs 2024-07-26T08:09:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 612.610979 secs 2024-07-26T08:09:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:36 smithi115 ceph-mon[35364]: pgmap v1084: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:36 smithi181 ceph-mon[35869]: pgmap v1084: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:36 smithi097 ceph-mon[32231]: pgmap v1084: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:37 smithi115 ceph-mon[35364]: pgmap v1085: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:37 smithi181 ceph-mon[35869]: pgmap v1085: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:37 smithi097 ceph-mon[32231]: pgmap v1085: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:40 smithi115 ceph-mon[35364]: pgmap v1086: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 617.611145 secs 2024-07-26T08:09:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:40 smithi181 ceph-mon[35869]: pgmap v1086: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 617.611145 secs 2024-07-26T08:09:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:40 smithi097 ceph-mon[32231]: pgmap v1086: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 617.611145 secs 2024-07-26T08:09:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:42 smithi115 ceph-mon[35364]: pgmap v1087: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:42 smithi181 ceph-mon[35869]: pgmap v1087: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:42 smithi097 ceph-mon[32231]: pgmap v1087: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:09:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:44 smithi115 ceph-mon[35364]: pgmap v1088: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:09:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:44 smithi181 ceph-mon[35869]: pgmap v1088: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:09:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:44 smithi097 ceph-mon[32231]: pgmap v1088: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:09:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 622.611319 secs 2024-07-26T08:09:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 622.611319 secs 2024-07-26T08:09:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 622.611319 secs 2024-07-26T08:09:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:46 smithi115 ceph-mon[35364]: pgmap v1089: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:46 smithi181 ceph-mon[35869]: pgmap v1089: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:46 smithi097 ceph-mon[32231]: pgmap v1089: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:47.608+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:09:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:47.610+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:09:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:47.610+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:09:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:47.610+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:09:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:09:47.610+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:09:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:48 smithi115 ceph-mon[35364]: pgmap v1090: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:48 smithi181 ceph-mon[35869]: pgmap v1090: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:48 smithi097 ceph-mon[32231]: pgmap v1090: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:50 smithi115 ceph-mon[35364]: pgmap v1091: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 627.611457 secs 2024-07-26T08:09:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:50 smithi181 ceph-mon[35869]: pgmap v1091: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 627.611457 secs 2024-07-26T08:09:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:50 smithi097 ceph-mon[32231]: pgmap v1091: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 627.611457 secs 2024-07-26T08:09:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:52 smithi115 ceph-mon[35364]: pgmap v1092: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:52 smithi181 ceph-mon[35869]: pgmap v1092: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:52 smithi097 ceph-mon[32231]: pgmap v1092: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:09:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:54 smithi115 ceph-mon[35364]: pgmap v1093: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:54 smithi181 ceph-mon[35869]: pgmap v1093: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:54 smithi097 ceph-mon[32231]: pgmap v1093: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:09:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 632.611615 secs 2024-07-26T08:09:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 632.611615 secs 2024-07-26T08:09:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 632.611615 secs 2024-07-26T08:09:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:56 smithi115 ceph-mon[35364]: pgmap v1094: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:56 smithi181 ceph-mon[35869]: pgmap v1094: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:56 smithi097 ceph-mon[32231]: pgmap v1094: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:09:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:58 smithi115 ceph-mon[35364]: pgmap v1095: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:58 smithi181 ceph-mon[35869]: pgmap v1095: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:58 smithi097 ceph-mon[32231]: pgmap v1095: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:09:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:09:59 smithi115 ceph-mon[35364]: pgmap v1096: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:09:59 smithi181 ceph-mon[35869]: pgmap v1096: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:09:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:09:59 smithi097 ceph-mon[32231]: pgmap v1096: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 637.611754 secs 2024-07-26T08:10:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:00 smithi115 ceph-mon[35364]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:10:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:00 smithi115 ceph-mon[35364]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-07-26T08:10:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:00 smithi115 ceph-mon[35364]: daemon mds.f on smithi181 is in unknown state 2024-07-26T08:10:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:00 smithi115 ceph-mon[35364]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:10:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:00 smithi115 ceph-mon[35364]: mds.c(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T08:10:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 637.611754 secs 2024-07-26T08:10:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:00 smithi181 ceph-mon[35869]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:10:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:00 smithi181 ceph-mon[35869]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-07-26T08:10:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:00 smithi181 ceph-mon[35869]: daemon mds.f on smithi181 is in unknown state 2024-07-26T08:10:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:00 smithi181 ceph-mon[35869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:10:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:00 smithi181 ceph-mon[35869]: mds.c(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T08:10:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 637.611754 secs 2024-07-26T08:10:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:00 smithi097 ceph-mon[32231]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:10:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:00 smithi097 ceph-mon[32231]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-07-26T08:10:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:00 smithi097 ceph-mon[32231]: daemon mds.f on smithi181 is in unknown state 2024-07-26T08:10:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:00 smithi097 ceph-mon[32231]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:10:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:00 smithi097 ceph-mon[32231]: mds.c(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T08:10:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:01 smithi115 ceph-mon[35364]: pgmap v1097: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:01 smithi181 ceph-mon[35869]: pgmap v1097: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:01 smithi097 ceph-mon[32231]: pgmap v1097: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:02.614+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:10:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:02.615+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:10:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:02.615+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:10:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:02.615+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:10:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:02.615+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:10:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:04 smithi115 ceph-mon[35364]: pgmap v1098: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:04 smithi181 ceph-mon[35869]: pgmap v1098: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:04 smithi097 ceph-mon[32231]: pgmap v1098: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:05.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 642.611896 secs 2024-07-26T08:10:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 642.611896 secs 2024-07-26T08:10:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 642.611896 secs 2024-07-26T08:10:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:06 smithi115 ceph-mon[35364]: pgmap v1099: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:06 smithi181 ceph-mon[35869]: pgmap v1099: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:06 smithi097 ceph-mon[32231]: pgmap v1099: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:06.986+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0005fb0 con 0x7ff3f01061a0 2024-07-26T08:10:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:06.986+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006070 con 0x55d25070b2f0 2024-07-26T08:10:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:06.987+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0005fb0 con 0x7ff3f01061a0 2024-07-26T08:10:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:06.987+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006070 con 0x55d25070b2f0 2024-07-26T08:10:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:08 smithi115 ceph-mon[35364]: pgmap v1100: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:08 smithi181 ceph-mon[35869]: pgmap v1100: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:08 smithi097 ceph-mon[32231]: pgmap v1100: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:10 smithi115 ceph-mon[35364]: pgmap v1101: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 647.612032 secs 2024-07-26T08:10:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:10 smithi181 ceph-mon[35869]: pgmap v1101: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 647.612032 secs 2024-07-26T08:10:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:10 smithi097 ceph-mon[32231]: pgmap v1101: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 647.612032 secs 2024-07-26T08:10:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:12 smithi115 ceph-mon[35364]: pgmap v1102: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:12 smithi181 ceph-mon[35869]: pgmap v1102: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:12 smithi097 ceph-mon[32231]: pgmap v1102: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:14 smithi115 ceph-mon[35364]: pgmap v1103: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:14 smithi181 ceph-mon[35869]: pgmap v1103: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:14 smithi097 ceph-mon[32231]: pgmap v1103: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 652.612174 secs 2024-07-26T08:10:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 652.612174 secs 2024-07-26T08:10:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 652.612174 secs 2024-07-26T08:10:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:16 smithi115 ceph-mon[35364]: pgmap v1104: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:16 smithi181 ceph-mon[35869]: pgmap v1104: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:16 smithi097 ceph-mon[32231]: pgmap v1104: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:17.624+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:10:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:17.625+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:10:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:17.625+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:10:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:17.625+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:10:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:17.625+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:10:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:18 smithi115 ceph-mon[35364]: pgmap v1105: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:18 smithi181 ceph-mon[35869]: pgmap v1105: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:18 smithi097 ceph-mon[32231]: pgmap v1105: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:20 smithi115 ceph-mon[35364]: pgmap v1106: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 657.612316 secs 2024-07-26T08:10:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:20 smithi181 ceph-mon[35869]: pgmap v1106: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 657.612316 secs 2024-07-26T08:10:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:20 smithi097 ceph-mon[32231]: pgmap v1106: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 657.612316 secs 2024-07-26T08:10:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:21 smithi115 ceph-mon[35364]: pgmap v1107: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:21 smithi181 ceph-mon[35869]: pgmap v1107: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:21 smithi097 ceph-mon[32231]: pgmap v1107: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:24 smithi115 ceph-mon[35364]: pgmap v1108: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:24 smithi181 ceph-mon[35869]: pgmap v1108: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:24 smithi097 ceph-mon[32231]: pgmap v1108: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 662.612478 secs 2024-07-26T08:10:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 662.612478 secs 2024-07-26T08:10:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 662.612478 secs 2024-07-26T08:10:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:26 smithi115 ceph-mon[35364]: pgmap v1109: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:26 smithi181 ceph-mon[35869]: pgmap v1109: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:26 smithi097 ceph-mon[32231]: pgmap v1109: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:28 smithi115 ceph-mon[35364]: pgmap v1110: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:28 smithi181 ceph-mon[35869]: pgmap v1110: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:28 smithi097 ceph-mon[32231]: pgmap v1110: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:30 smithi115 ceph-mon[35364]: pgmap v1111: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 667.612618 secs 2024-07-26T08:10:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:30 smithi181 ceph-mon[35869]: pgmap v1111: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 667.612618 secs 2024-07-26T08:10:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:30 smithi097 ceph-mon[32231]: pgmap v1111: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 667.612618 secs 2024-07-26T08:10:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:32.628+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:10:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:32.629+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:10:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:32.629+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:10:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:32.629+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:10:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:32.630+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:10:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:32 smithi115 ceph-mon[35364]: pgmap v1112: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:32 smithi181 ceph-mon[35869]: pgmap v1112: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:32 smithi097 ceph-mon[32231]: pgmap v1112: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:34.702 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:34 smithi115 ceph-mon[35364]: pgmap v1113: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:34.702 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:10:34.706 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:34 smithi097 ceph-mon[32231]: pgmap v1113: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:34.706 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:10:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:34 smithi181 ceph-mon[35869]: pgmap v1113: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:10:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 672.612763 secs 2024-07-26T08:10:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 672.612763 secs 2024-07-26T08:10:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 672.612763 secs 2024-07-26T08:10:36.723 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:36 smithi115 ceph-mon[35364]: pgmap v1114: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:36 smithi181 ceph-mon[35869]: pgmap v1114: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:36 smithi097 ceph-mon[32231]: pgmap v1114: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:37 smithi115 ceph-mon[35364]: pgmap v1115: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:37 smithi181 ceph-mon[35869]: pgmap v1115: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:38.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:37 smithi097 ceph-mon[32231]: pgmap v1115: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:10:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:10:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:38 smithi115 ceph-mon[35364]: pgmap v1116: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:38 smithi115 ceph-mon[35364]: pgmap v1117: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:10:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:38 smithi115 ceph-mon[35364]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:10:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:10:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:10:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:38 smithi181 ceph-mon[35869]: pgmap v1116: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:38 smithi181 ceph-mon[35869]: pgmap v1117: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:10:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:38 smithi181 ceph-mon[35869]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:10:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:10:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:10:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:38 smithi097 ceph-mon[32231]: pgmap v1116: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:38 smithi097 ceph-mon[32231]: pgmap v1117: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:10:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:10:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:38 smithi097 ceph-mon[32231]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:10:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:39 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 677.612925 secs 2024-07-26T08:10:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:39 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 677.612925 secs 2024-07-26T08:10:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:39 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 677.612925 secs 2024-07-26T08:10:41.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:40 smithi115 ceph-mon[35364]: pgmap v1118: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T08:10:41.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:40 smithi181 ceph-mon[35869]: pgmap v1118: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T08:10:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:40 smithi097 ceph-mon[32231]: pgmap v1118: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T08:10:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:43 smithi115 ceph-mon[35364]: pgmap v1119: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:10:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:43 smithi181 ceph-mon[35869]: pgmap v1119: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:10:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:43 smithi097 ceph-mon[32231]: pgmap v1119: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:10:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:45 smithi115 ceph-mon[35364]: pgmap v1120: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:10:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 682.613068 secs 2024-07-26T08:10:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:45 smithi181 ceph-mon[35869]: pgmap v1120: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:10:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 682.613068 secs 2024-07-26T08:10:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:45 smithi097 ceph-mon[32231]: pgmap v1120: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:10:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 682.613068 secs 2024-07-26T08:10:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:47 smithi115 ceph-mon[35364]: pgmap v1121: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:10:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:47 smithi181 ceph-mon[35869]: pgmap v1121: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:10:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:47 smithi097 ceph-mon[32231]: pgmap v1121: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:10:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:47.639+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:10:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:47.640+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:10:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:47.640+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:10:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:47.640+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:10:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:10:47.640+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:10:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:49 smithi115 ceph-mon[35364]: pgmap v1122: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:10:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:49 smithi181 ceph-mon[35869]: pgmap v1122: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:10:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:49 smithi097 ceph-mon[32231]: pgmap v1122: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:10:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 687.613204 secs 2024-07-26T08:10:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 687.613204 secs 2024-07-26T08:10:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 687.613204 secs 2024-07-26T08:10:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:51 smithi115 ceph-mon[35364]: pgmap v1123: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:51 smithi181 ceph-mon[35869]: pgmap v1123: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:51 smithi097 ceph-mon[32231]: pgmap v1123: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:10:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:53 smithi115 ceph-mon[35364]: pgmap v1124: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:53 smithi181 ceph-mon[35869]: pgmap v1124: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:53 smithi097 ceph-mon[32231]: pgmap v1124: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:10:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:55 smithi115 ceph-mon[35364]: pgmap v1125: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 692.613342 secs 2024-07-26T08:10:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:55 smithi181 ceph-mon[35869]: pgmap v1125: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 692.613342 secs 2024-07-26T08:10:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:55 smithi097 ceph-mon[32231]: pgmap v1125: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:10:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 692.613342 secs 2024-07-26T08:10:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:57 smithi115 ceph-mon[35364]: pgmap v1126: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:57 smithi181 ceph-mon[35869]: pgmap v1126: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:57 smithi097 ceph-mon[32231]: pgmap v1126: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:10:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:10:59 smithi115 ceph-mon[35364]: pgmap v1127: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:10:59 smithi181 ceph-mon[35869]: pgmap v1127: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:10:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:10:59 smithi097 ceph-mon[32231]: pgmap v1127: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 697.613486 secs 2024-07-26T08:11:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 697.613486 secs 2024-07-26T08:11:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 697.613486 secs 2024-07-26T08:11:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:01 smithi115 ceph-mon[35364]: pgmap v1128: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:01 smithi181 ceph-mon[35869]: pgmap v1128: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:01 smithi097 ceph-mon[32231]: pgmap v1128: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:02.644+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:11:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:02.645+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:11:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:02.645+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:11:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:02.645+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:11:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:02.645+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:11:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:03 smithi115 ceph-mon[35364]: pgmap v1129: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:11:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:03 smithi181 ceph-mon[35869]: pgmap v1129: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:11:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:03 smithi097 ceph-mon[32231]: pgmap v1129: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:11:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:05 smithi115 ceph-mon[35364]: pgmap v1130: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:11:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 702.613587 secs 2024-07-26T08:11:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:05 smithi181 ceph-mon[35869]: pgmap v1130: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:11:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 702.613587 secs 2024-07-26T08:11:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:05 smithi097 ceph-mon[32231]: pgmap v1130: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:11:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 702.613587 secs 2024-07-26T08:11:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:07 smithi115 ceph-mon[35364]: pgmap v1131: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:07 smithi181 ceph-mon[35869]: pgmap v1131: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:07 smithi097 ceph-mon[32231]: pgmap v1131: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:09 smithi115 ceph-mon[35364]: pgmap v1132: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:11:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:09 smithi181 ceph-mon[35869]: pgmap v1132: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:11:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:09 smithi097 ceph-mon[32231]: pgmap v1132: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:11:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 707.613682 secs 2024-07-26T08:11:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 707.613682 secs 2024-07-26T08:11:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 707.613682 secs 2024-07-26T08:11:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:11 smithi115 ceph-mon[35364]: pgmap v1133: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:11:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:11 smithi181 ceph-mon[35869]: pgmap v1133: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:11:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:11 smithi097 ceph-mon[32231]: pgmap v1133: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:11:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:13 smithi181 ceph-mon[35869]: pgmap v1134: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:13 smithi097 ceph-mon[32231]: pgmap v1134: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:13 smithi115 ceph-mon[35364]: pgmap v1134: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:15 smithi115 ceph-mon[35364]: pgmap v1135: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:11:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 712.613819 secs 2024-07-26T08:11:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:15 smithi181 ceph-mon[35869]: pgmap v1135: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:11:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 712.613819 secs 2024-07-26T08:11:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:15 smithi097 ceph-mon[32231]: pgmap v1135: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:11:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 712.613819 secs 2024-07-26T08:11:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:17.654+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:11:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:17.655+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:11:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:17.655+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:11:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:17.655+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:11:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:17.655+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:11:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:17 smithi115 ceph-mon[35364]: pgmap v1136: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:11:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:17 smithi181 ceph-mon[35869]: pgmap v1136: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:11:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:17 smithi097 ceph-mon[32231]: pgmap v1136: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:11:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:18 smithi115 ceph-mon[35364]: pgmap v1137: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:11:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:18 smithi181 ceph-mon[35869]: pgmap v1137: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:11:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:18 smithi097 ceph-mon[32231]: pgmap v1137: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:11:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:19 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 717.613953 secs 2024-07-26T08:11:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:19 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 717.613953 secs 2024-07-26T08:11:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:19 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 717.613953 secs 2024-07-26T08:11:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:20 smithi115 ceph-mon[35364]: pgmap v1138: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:11:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:20 smithi181 ceph-mon[35869]: pgmap v1138: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:11:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:20 smithi097 ceph-mon[32231]: pgmap v1138: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:11:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:23 smithi115 ceph-mon[35364]: pgmap v1139: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:23 smithi181 ceph-mon[35869]: pgmap v1139: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:23 smithi097 ceph-mon[32231]: pgmap v1139: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:25 smithi115 ceph-mon[35364]: pgmap v1140: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:11:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:25 smithi115 ceph-mon[35364]: 1 slow requests, 1 included below; oldest blocked for > 722.614105 secs 2024-07-26T08:11:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:25 smithi115 ceph-mon[35364]: slow request 722.614105 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:11:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:25 smithi181 ceph-mon[35869]: pgmap v1140: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:11:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:25 smithi181 ceph-mon[35869]: 1 slow requests, 1 included below; oldest blocked for > 722.614105 secs 2024-07-26T08:11:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:25 smithi181 ceph-mon[35869]: slow request 722.614105 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:11:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:25 smithi097 ceph-mon[32231]: pgmap v1140: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:11:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:25 smithi097 ceph-mon[32231]: 1 slow requests, 1 included below; oldest blocked for > 722.614105 secs 2024-07-26T08:11:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:25 smithi097 ceph-mon[32231]: slow request 722.614105 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:11:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:27 smithi115 ceph-mon[35364]: pgmap v1141: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:27 smithi181 ceph-mon[35869]: pgmap v1141: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:27 smithi097 ceph-mon[32231]: pgmap v1141: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:29 smithi115 ceph-mon[35364]: pgmap v1142: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:29 smithi181 ceph-mon[35869]: pgmap v1142: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:29 smithi097 ceph-mon[32231]: pgmap v1142: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 727.614186 secs 2024-07-26T08:11:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 727.614186 secs 2024-07-26T08:11:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 727.614186 secs 2024-07-26T08:11:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:31 smithi115 ceph-mon[35364]: pgmap v1143: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:31 smithi181 ceph-mon[35869]: pgmap v1143: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:31 smithi097 ceph-mon[32231]: pgmap v1143: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:32.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:32.658+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:11:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:32.659+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:11:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:32.659+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:11:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:32.659+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:11:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:32.659+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:11:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:33 smithi115 ceph-mon[35364]: pgmap v1144: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:11:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:33 smithi181 ceph-mon[35869]: pgmap v1144: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:11:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:33 smithi097 ceph-mon[32231]: pgmap v1144: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:11:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:35 smithi115 ceph-mon[35364]: pgmap v1145: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:11:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 732.614335 secs 2024-07-26T08:11:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:35 smithi181 ceph-mon[35869]: pgmap v1145: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:11:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 732.614335 secs 2024-07-26T08:11:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:35 smithi097 ceph-mon[32231]: pgmap v1145: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:11:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 732.614335 secs 2024-07-26T08:11:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:37 smithi115 ceph-mon[35364]: pgmap v1146: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:37 smithi181 ceph-mon[35869]: pgmap v1146: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:37 smithi097 ceph-mon[32231]: pgmap v1146: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:39 smithi115 ceph-mon[35364]: pgmap v1147: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:11:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:39 smithi181 ceph-mon[35869]: pgmap v1147: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:11:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:39 smithi097 ceph-mon[32231]: pgmap v1147: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:11:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:11:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:11:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:11:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 737.614480 secs 2024-07-26T08:11:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:11:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:11:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:11:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 737.614480 secs 2024-07-26T08:11:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:11:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:11:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:11:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 737.614480 secs 2024-07-26T08:11:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:41 smithi115 ceph-mon[35364]: pgmap v1148: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:11:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:41 smithi115 ceph-mon[35364]: pgmap v1149: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:41 smithi181 ceph-mon[35869]: pgmap v1148: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:11:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:41 smithi181 ceph-mon[35869]: pgmap v1149: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:41 smithi097 ceph-mon[32231]: pgmap v1148: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:11:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:41 smithi097 ceph-mon[32231]: pgmap v1149: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:43 smithi115 ceph-mon[35364]: pgmap v1150: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:43 smithi181 ceph-mon[35869]: pgmap v1150: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:43 smithi097 ceph-mon[32231]: pgmap v1150: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:45 smithi115 ceph-mon[35364]: pgmap v1151: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:11:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 742.614619 secs 2024-07-26T08:11:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:45 smithi181 ceph-mon[35869]: pgmap v1151: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:11:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 742.614619 secs 2024-07-26T08:11:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:45 smithi097 ceph-mon[32231]: pgmap v1151: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:11:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 742.614619 secs 2024-07-26T08:11:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:47 smithi115 ceph-mon[35364]: pgmap v1152: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:47 smithi181 ceph-mon[35869]: pgmap v1152: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:47 smithi097 ceph-mon[32231]: pgmap v1152: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:47.668+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:11:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:47.669+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:11:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:47.669+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:11:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:47.669+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:11:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:11:47.670+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:11:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:49 smithi115 ceph-mon[35364]: pgmap v1153: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:49 smithi181 ceph-mon[35869]: pgmap v1153: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:49 smithi097 ceph-mon[32231]: pgmap v1153: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:11:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 747.614778 secs 2024-07-26T08:11:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 747.614778 secs 2024-07-26T08:11:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 747.614778 secs 2024-07-26T08:11:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:51 smithi097 ceph-mon[32231]: pgmap v1154: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:11:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:51 smithi115 ceph-mon[35364]: pgmap v1154: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:11:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:51 smithi181 ceph-mon[35869]: pgmap v1154: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:11:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:53 smithi097 ceph-mon[32231]: pgmap v1155: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:53 smithi115 ceph-mon[35364]: pgmap v1155: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:53 smithi181 ceph-mon[35869]: pgmap v1155: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:11:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:55 smithi115 ceph-mon[35364]: pgmap v1156: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 752.614920 secs 2024-07-26T08:11:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:55 smithi181 ceph-mon[35869]: pgmap v1156: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 752.614920 secs 2024-07-26T08:11:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:55 smithi097 ceph-mon[32231]: pgmap v1156: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:11:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 752.614920 secs 2024-07-26T08:11:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:57 smithi115 ceph-mon[35364]: pgmap v1157: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:57 smithi181 ceph-mon[35869]: pgmap v1157: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:57 smithi097 ceph-mon[32231]: pgmap v1157: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:11:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:11:59 smithi115 ceph-mon[35364]: pgmap v1158: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:11:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:11:59 smithi181 ceph-mon[35869]: pgmap v1158: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:11:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:11:59 smithi097 ceph-mon[32231]: pgmap v1158: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:12:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 757.615058 secs 2024-07-26T08:12:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 757.615058 secs 2024-07-26T08:12:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 757.615058 secs 2024-07-26T08:12:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:01 smithi115 ceph-mon[35364]: pgmap v1159: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:01 smithi181 ceph-mon[35869]: pgmap v1159: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:01 smithi097 ceph-mon[32231]: pgmap v1159: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:02.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:02.674+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:12:02.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:02.675+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:12:02.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:02.675+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:12:02.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:02.675+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:12:02.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:02.675+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:12:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:03 smithi115 ceph-mon[35364]: pgmap v1160: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:03 smithi181 ceph-mon[35869]: pgmap v1160: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:03 smithi097 ceph-mon[32231]: pgmap v1160: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:05.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:05 smithi097 ceph-mon[32231]: pgmap v1161: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:05.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 762.615193 secs 2024-07-26T08:12:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:05 smithi115 ceph-mon[35364]: pgmap v1161: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 762.615193 secs 2024-07-26T08:12:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:05 smithi181 ceph-mon[35869]: pgmap v1161: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 762.615193 secs 2024-07-26T08:12:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:07 smithi115 ceph-mon[35364]: pgmap v1162: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:07 smithi181 ceph-mon[35869]: pgmap v1162: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:07 smithi097 ceph-mon[32231]: pgmap v1162: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:09.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:09 smithi115 ceph-mon[35364]: pgmap v1163: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:09 smithi181 ceph-mon[35869]: pgmap v1163: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:09 smithi097 ceph-mon[32231]: pgmap v1163: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 767.615333 secs 2024-07-26T08:12:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 767.615333 secs 2024-07-26T08:12:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 767.615333 secs 2024-07-26T08:12:11.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:11 smithi115 ceph-mon[35364]: pgmap v1164: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:11.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:11 smithi181 ceph-mon[35869]: pgmap v1164: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:11 smithi097 ceph-mon[32231]: pgmap v1164: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:13 smithi115 ceph-mon[35364]: pgmap v1165: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:13 smithi181 ceph-mon[35869]: pgmap v1165: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:13 smithi097 ceph-mon[32231]: pgmap v1165: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:15 smithi115 ceph-mon[35364]: pgmap v1166: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 772.615477 secs 2024-07-26T08:12:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:15 smithi181 ceph-mon[35869]: pgmap v1166: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 772.615477 secs 2024-07-26T08:12:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:15 smithi097 ceph-mon[32231]: pgmap v1166: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 772.615477 secs 2024-07-26T08:12:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:17.684+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:12:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:17.685+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:12:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:17.685+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:12:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:17.685+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:12:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:17.685+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:12:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:17 smithi115 ceph-mon[35364]: pgmap v1167: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:17 smithi181 ceph-mon[35869]: pgmap v1167: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:17 smithi097 ceph-mon[32231]: pgmap v1167: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:19 smithi115 ceph-mon[35364]: pgmap v1168: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:19 smithi181 ceph-mon[35869]: pgmap v1168: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:19 smithi097 ceph-mon[32231]: pgmap v1168: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 777.615564 secs 2024-07-26T08:12:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 777.615564 secs 2024-07-26T08:12:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 777.615564 secs 2024-07-26T08:12:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:21 smithi115 ceph-mon[35364]: pgmap v1169: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:21 smithi181 ceph-mon[35869]: pgmap v1169: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:21 smithi097 ceph-mon[32231]: pgmap v1169: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:23 smithi115 ceph-mon[35364]: pgmap v1170: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:23 smithi181 ceph-mon[35869]: pgmap v1170: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:23 smithi097 ceph-mon[32231]: pgmap v1170: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:25 smithi115 ceph-mon[35364]: pgmap v1171: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 782.615712 secs 2024-07-26T08:12:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:25 smithi181 ceph-mon[35869]: pgmap v1171: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 782.615712 secs 2024-07-26T08:12:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:25 smithi097 ceph-mon[32231]: pgmap v1171: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 782.615712 secs 2024-07-26T08:12:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:27 smithi115 ceph-mon[35364]: pgmap v1172: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:27 smithi181 ceph-mon[35869]: pgmap v1172: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:27 smithi097 ceph-mon[32231]: pgmap v1172: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:29 smithi115 ceph-mon[35364]: pgmap v1173: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:29 smithi181 ceph-mon[35869]: pgmap v1173: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:29 smithi097 ceph-mon[32231]: pgmap v1173: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 787.615853 secs 2024-07-26T08:12:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 787.615853 secs 2024-07-26T08:12:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 787.615853 secs 2024-07-26T08:12:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:31 smithi115 ceph-mon[35364]: pgmap v1174: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:31 smithi181 ceph-mon[35869]: pgmap v1174: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:31 smithi097 ceph-mon[32231]: pgmap v1174: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:32.689+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:12:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:32.691+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:12:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:32.691+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:12:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:32.691+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:12:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:32.691+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:12:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:32 smithi115 ceph-mon[35364]: pgmap v1175: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:32 smithi181 ceph-mon[35869]: pgmap v1175: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:32 smithi097 ceph-mon[32231]: pgmap v1175: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:12:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:34 smithi115 ceph-mon[35364]: pgmap v1176: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:34 smithi181 ceph-mon[35869]: pgmap v1176: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:34 smithi097 ceph-mon[32231]: pgmap v1176: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:35.695 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 792.615997 secs 2024-07-26T08:12:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 792.615997 secs 2024-07-26T08:12:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 792.615997 secs 2024-07-26T08:12:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:36 smithi115 ceph-mon[35364]: pgmap v1177: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:36 smithi181 ceph-mon[35869]: pgmap v1177: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:36 smithi097 ceph-mon[32231]: pgmap v1177: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:38 smithi115 ceph-mon[35364]: pgmap v1178: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:38 smithi181 ceph-mon[35869]: pgmap v1178: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:38 smithi097 ceph-mon[32231]: pgmap v1178: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:40 smithi115 ceph-mon[35364]: pgmap v1179: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 797.616157 secs 2024-07-26T08:12:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:12:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:40 smithi181 ceph-mon[35869]: pgmap v1179: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 797.616157 secs 2024-07-26T08:12:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:12:41.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:40 smithi097 ceph-mon[32231]: pgmap v1179: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:12:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 797.616157 secs 2024-07-26T08:12:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:12:41.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:12:41.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:12:41.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:41 smithi115 ceph-mon[35364]: pgmap v1180: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:12:41.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:41 smithi115 ceph-mon[35364]: pgmap v1181: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:41.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:12:41.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:12:41.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:12:41.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:41 smithi181 ceph-mon[35869]: pgmap v1180: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:12:41.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:41 smithi181 ceph-mon[35869]: pgmap v1181: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:41.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:12:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:12:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:12:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:41 smithi097 ceph-mon[32231]: pgmap v1180: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:12:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:41 smithi097 ceph-mon[32231]: pgmap v1181: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:12:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:43 smithi097 ceph-mon[32231]: pgmap v1182: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:43 smithi115 ceph-mon[35364]: pgmap v1182: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:43 smithi181 ceph-mon[35869]: pgmap v1182: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:44 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 802.616308 secs 2024-07-26T08:12:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 802.616308 secs 2024-07-26T08:12:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 802.616308 secs 2024-07-26T08:12:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:46 smithi115 ceph-mon[35364]: pgmap v1183: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:46 smithi181 ceph-mon[35869]: pgmap v1183: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:46 smithi097 ceph-mon[32231]: pgmap v1183: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:47.700+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:12:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:47.701+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:12:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:47.701+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:12:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:47.701+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:12:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:12:47.701+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:12:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:48 smithi115 ceph-mon[35364]: pgmap v1184: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:48 smithi181 ceph-mon[35869]: pgmap v1184: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:48 smithi097 ceph-mon[32231]: pgmap v1184: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:50 smithi115 ceph-mon[35364]: pgmap v1185: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:12:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 807.616421 secs 2024-07-26T08:12:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:50 smithi181 ceph-mon[35869]: pgmap v1185: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:12:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 807.616421 secs 2024-07-26T08:12:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:50 smithi097 ceph-mon[32231]: pgmap v1185: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:12:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 807.616421 secs 2024-07-26T08:12:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:52 smithi115 ceph-mon[35364]: pgmap v1186: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:12:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:52 smithi181 ceph-mon[35869]: pgmap v1186: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:12:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:52 smithi097 ceph-mon[32231]: pgmap v1186: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:12:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:54 smithi115 ceph-mon[35364]: pgmap v1187: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:54 smithi181 ceph-mon[35869]: pgmap v1187: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:54 smithi097 ceph-mon[32231]: pgmap v1187: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:12:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 812.616566 secs 2024-07-26T08:12:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 812.616566 secs 2024-07-26T08:12:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 812.616566 secs 2024-07-26T08:12:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:56 smithi115 ceph-mon[35364]: pgmap v1188: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:56 smithi181 ceph-mon[35869]: pgmap v1188: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:56 smithi097 ceph-mon[32231]: pgmap v1188: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:12:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:12:58 smithi115 ceph-mon[35364]: pgmap v1189: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:12:58 smithi181 ceph-mon[35869]: pgmap v1189: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:12:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:12:58 smithi097 ceph-mon[32231]: pgmap v1189: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:00 smithi115 ceph-mon[35364]: pgmap v1190: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 817.617266 secs 2024-07-26T08:13:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:00 smithi181 ceph-mon[35869]: pgmap v1190: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 817.617266 secs 2024-07-26T08:13:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:00 smithi097 ceph-mon[32231]: pgmap v1190: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 817.617266 secs 2024-07-26T08:13:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:02 smithi115 ceph-mon[35364]: pgmap v1191: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:02 smithi181 ceph-mon[35869]: pgmap v1191: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:02 smithi097 ceph-mon[32231]: pgmap v1191: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:02.706 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:02.705+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:13:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:02.706+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:13:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:02.706+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:13:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:02.706+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:13:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:02.707+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:13:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:04 smithi115 ceph-mon[35364]: pgmap v1192: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:04 smithi181 ceph-mon[35869]: pgmap v1192: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:04 smithi097 ceph-mon[32231]: pgmap v1192: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:05.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 822.617372 secs 2024-07-26T08:13:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 822.617372 secs 2024-07-26T08:13:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 822.617372 secs 2024-07-26T08:13:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:06 smithi115 ceph-mon[35364]: pgmap v1193: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:13:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:06 smithi181 ceph-mon[35869]: pgmap v1193: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:13:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:06 smithi097 ceph-mon[32231]: pgmap v1193: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:13:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.879+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T08:13:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.879+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T08:13:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.879+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b6160 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000bc80 tx=0x7ff3e0005f30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:13:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.879+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b6160 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000bc80 tx=0x7ff3e0005f30 comp rx=0 tx=0).stop 2024-07-26T08:13:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.879+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b6160 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T08:13:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.879+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b6160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:13:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.879+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b6160 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:13:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.879+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 0x7ff3d40c0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:13:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.880+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 0x7ff3d40c0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:13:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:06.882+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 0x7ff3d40c0000 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff3e001fef0 tx=0x7ff3e0038000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:13:07.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:07.018+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:13:07.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:07.018+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006330 con 0x55d25070b2f0 2024-07-26T08:13:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:07.018+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006330 con 0x55d25070b2f0 2024-07-26T08:13:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:07.018+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d8097040 con 0x7ff3f01061a0 2024-07-26T08:13:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:08 smithi115 ceph-mon[35364]: pgmap v1194: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:13:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:08 smithi181 ceph-mon[35869]: pgmap v1194: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:13:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:08 smithi097 ceph-mon[32231]: pgmap v1194: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:13:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:10 smithi115 ceph-mon[35364]: pgmap v1195: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:13:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 827.617518 secs 2024-07-26T08:13:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:10 smithi181 ceph-mon[35869]: pgmap v1195: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:13:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 827.617518 secs 2024-07-26T08:13:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:10 smithi097 ceph-mon[32231]: pgmap v1195: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:13:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 827.617518 secs 2024-07-26T08:13:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:12 smithi115 ceph-mon[35364]: pgmap v1196: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T08:13:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:12 smithi181 ceph-mon[35869]: pgmap v1196: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T08:13:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:12 smithi097 ceph-mon[32231]: pgmap v1196: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T08:13:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:14 smithi115 ceph-mon[35364]: pgmap v1197: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:13:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:14 smithi181 ceph-mon[35869]: pgmap v1197: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:13:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:14 smithi097 ceph-mon[32231]: pgmap v1197: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:13:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 832.617684 secs 2024-07-26T08:13:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 832.617684 secs 2024-07-26T08:13:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 832.617684 secs 2024-07-26T08:13:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:16 smithi115 ceph-mon[35364]: pgmap v1198: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:13:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:16 smithi181 ceph-mon[35869]: pgmap v1198: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:13:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:16 smithi097 ceph-mon[32231]: pgmap v1198: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:13:17.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:17.715+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:13:17.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:17.716+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:13:17.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:17.716+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:13:17.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:17.716+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:13:17.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:17.716+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:13:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:18 smithi115 ceph-mon[35364]: pgmap v1199: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:18 smithi181 ceph-mon[35869]: pgmap v1199: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:18 smithi097 ceph-mon[32231]: pgmap v1199: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:20 smithi115 ceph-mon[35364]: pgmap v1200: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 837.617828 secs 2024-07-26T08:13:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:20 smithi181 ceph-mon[35869]: pgmap v1200: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 837.617828 secs 2024-07-26T08:13:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:20 smithi097 ceph-mon[32231]: pgmap v1200: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 837.617828 secs 2024-07-26T08:13:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:22 smithi115 ceph-mon[35364]: pgmap v1201: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:22 smithi181 ceph-mon[35869]: pgmap v1201: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:22 smithi097 ceph-mon[32231]: pgmap v1201: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:24 smithi115 ceph-mon[35364]: pgmap v1202: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:24 smithi181 ceph-mon[35869]: pgmap v1202: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:24 smithi097 ceph-mon[32231]: pgmap v1202: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 842.617982 secs 2024-07-26T08:13:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 842.617982 secs 2024-07-26T08:13:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 842.617982 secs 2024-07-26T08:13:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:26 smithi115 ceph-mon[35364]: pgmap v1203: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:26 smithi181 ceph-mon[35869]: pgmap v1203: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:26 smithi097 ceph-mon[32231]: pgmap v1203: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:28 smithi115 ceph-mon[35364]: pgmap v1204: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:28 smithi181 ceph-mon[35869]: pgmap v1204: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:28 smithi097 ceph-mon[32231]: pgmap v1204: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:30 smithi115 ceph-mon[35364]: pgmap v1205: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 847.618124 secs 2024-07-26T08:13:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:30 smithi181 ceph-mon[35869]: pgmap v1205: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 847.618124 secs 2024-07-26T08:13:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:30 smithi097 ceph-mon[32231]: pgmap v1205: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 847.618124 secs 2024-07-26T08:13:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:32 smithi115 ceph-mon[35364]: pgmap v1206: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:32 smithi181 ceph-mon[35869]: pgmap v1206: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:32 smithi097 ceph-mon[32231]: pgmap v1206: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:32.720+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:13:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:32.721+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:13:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:32.721+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:13:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:32.721+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:13:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:32.721+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:13:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:34 smithi115 ceph-mon[35364]: pgmap v1207: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:34 smithi181 ceph-mon[35869]: pgmap v1207: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:34 smithi097 ceph-mon[32231]: pgmap v1207: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 852.618275 secs 2024-07-26T08:13:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 852.618275 secs 2024-07-26T08:13:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 852.618275 secs 2024-07-26T08:13:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:36 smithi115 ceph-mon[35364]: pgmap v1208: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:36 smithi181 ceph-mon[35869]: pgmap v1208: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:36 smithi097 ceph-mon[32231]: pgmap v1208: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:38 smithi181 ceph-mon[35869]: pgmap v1209: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:38 smithi097 ceph-mon[32231]: pgmap v1209: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:38 smithi115 ceph-mon[35364]: pgmap v1209: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:40 smithi097 ceph-mon[32231]: pgmap v1210: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 857.618411 secs 2024-07-26T08:13:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:40 smithi115 ceph-mon[35364]: pgmap v1210: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 857.618411 secs 2024-07-26T08:13:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:40 smithi181 ceph-mon[35869]: pgmap v1210: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 857.618411 secs 2024-07-26T08:13:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:42 smithi097 ceph-mon[32231]: pgmap v1211: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:13:42.526 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:42 smithi115 ceph-mon[35364]: pgmap v1211: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:42.526 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:13:42.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:42 smithi181 ceph-mon[35869]: pgmap v1211: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T08:13:42.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:13:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:13:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:13:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:43 smithi115 ceph-mon[35364]: pgmap v1212: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:13:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:43 smithi115 ceph-mon[35364]: pgmap v1213: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:13:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:13:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:13:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:13:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:43 smithi181 ceph-mon[35869]: pgmap v1212: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:13:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:43 smithi181 ceph-mon[35869]: pgmap v1213: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:13:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:13:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:13:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:13:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:43 smithi097 ceph-mon[32231]: pgmap v1212: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:13:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:43 smithi097 ceph-mon[32231]: pgmap v1213: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:13:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:13:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:44 smithi115 ceph-mon[35364]: pgmap v1214: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:44.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:44 smithi181 ceph-mon[35869]: pgmap v1214: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:44 smithi097 ceph-mon[32231]: pgmap v1214: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 862.618559 secs 2024-07-26T08:13:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 862.618559 secs 2024-07-26T08:13:45.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 862.618559 secs 2024-07-26T08:13:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:46 smithi181 ceph-mon[35869]: pgmap v1215: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:46 smithi097 ceph-mon[32231]: pgmap v1215: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:46 smithi115 ceph-mon[35364]: pgmap v1215: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:47.730+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:13:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:47.731+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:13:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:47.731+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:13:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:47.731+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:13:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:13:47.731+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:13:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:48 smithi115 ceph-mon[35364]: pgmap v1216: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:48 smithi181 ceph-mon[35869]: pgmap v1216: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:48 smithi097 ceph-mon[32231]: pgmap v1216: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:13:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:49 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 867.618698 secs 2024-07-26T08:13:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:49 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 867.618698 secs 2024-07-26T08:13:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:49 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 867.618698 secs 2024-07-26T08:13:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:50 smithi115 ceph-mon[35364]: pgmap v1217: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:50 smithi181 ceph-mon[35869]: pgmap v1217: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:50 smithi097 ceph-mon[32231]: pgmap v1217: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:52 smithi115 ceph-mon[35364]: pgmap v1218: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:13:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:52 smithi181 ceph-mon[35869]: pgmap v1218: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:13:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:52 smithi097 ceph-mon[32231]: pgmap v1218: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:13:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:54 smithi115 ceph-mon[35364]: pgmap v1219: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:13:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:54 smithi181 ceph-mon[35869]: pgmap v1219: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:13:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:54 smithi097 ceph-mon[32231]: pgmap v1219: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:13:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 872.618841 secs 2024-07-26T08:13:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 872.618841 secs 2024-07-26T08:13:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 872.618841 secs 2024-07-26T08:13:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:56 smithi097 ceph-mon[32231]: pgmap v1220: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:56 smithi115 ceph-mon[35364]: pgmap v1220: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:56 smithi181 ceph-mon[35869]: pgmap v1220: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:13:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:58 smithi115 ceph-mon[35364]: pgmap v1221: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:58 smithi181 ceph-mon[35869]: pgmap v1221: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:13:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:58 smithi097 ceph-mon[32231]: pgmap v1221: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:13:59 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 877.618977 secs 2024-07-26T08:14:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:13:59 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 877.618977 secs 2024-07-26T08:14:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:13:59 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 877.618977 secs 2024-07-26T08:14:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:00 smithi115 ceph-mon[35364]: pgmap v1222: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:00 smithi181 ceph-mon[35869]: pgmap v1222: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:00 smithi097 ceph-mon[32231]: pgmap v1222: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:02 smithi181 ceph-mon[35869]: pgmap v1223: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:02.734+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:14:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:02.735+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:14:02.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:02.736+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:14:02.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:02.736+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:14:02.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:02.736+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:14:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:02 smithi097 ceph-mon[32231]: pgmap v1223: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:02 smithi115 ceph-mon[35364]: pgmap v1223: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:04 smithi181 ceph-mon[35869]: pgmap v1224: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:04 smithi097 ceph-mon[32231]: pgmap v1224: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:04 smithi115 ceph-mon[35364]: pgmap v1224: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 882.619138 secs 2024-07-26T08:14:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 882.619138 secs 2024-07-26T08:14:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 882.619138 secs 2024-07-26T08:14:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:06 smithi097 ceph-mon[32231]: pgmap v1225: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:06 smithi115 ceph-mon[35364]: pgmap v1225: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:06 smithi181 ceph-mon[35869]: pgmap v1225: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:08 smithi181 ceph-mon[35869]: pgmap v1226: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:08 smithi097 ceph-mon[32231]: pgmap v1226: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:08 smithi115 ceph-mon[35364]: pgmap v1226: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:09 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 887.619298 secs 2024-07-26T08:14:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:09 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 887.619298 secs 2024-07-26T08:14:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:09 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 887.619298 secs 2024-07-26T08:14:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:10 smithi115 ceph-mon[35364]: pgmap v1227: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:10 smithi181 ceph-mon[35869]: pgmap v1227: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:10 smithi097 ceph-mon[32231]: pgmap v1227: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:12 smithi181 ceph-mon[35869]: pgmap v1228: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:12 smithi097 ceph-mon[32231]: pgmap v1228: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:12 smithi115 ceph-mon[35364]: pgmap v1228: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:14 smithi181 ceph-mon[35869]: pgmap v1229: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:14 smithi097 ceph-mon[32231]: pgmap v1229: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:14 smithi115 ceph-mon[35364]: pgmap v1229: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 892.619443 secs 2024-07-26T08:14:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 892.619443 secs 2024-07-26T08:14:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 892.619443 secs 2024-07-26T08:14:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:16 smithi097 ceph-mon[32231]: pgmap v1230: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:16 smithi115 ceph-mon[35364]: pgmap v1230: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:16 smithi181 ceph-mon[35869]: pgmap v1230: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:17.738 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:17.737+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:14:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:17.738+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:14:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:17.738+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:14:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:17.738+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:14:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:17.738+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:14:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:18 smithi181 ceph-mon[35869]: pgmap v1231: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:18 smithi097 ceph-mon[32231]: pgmap v1231: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:18 smithi115 ceph-mon[35364]: pgmap v1231: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:19 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 897.619578 secs 2024-07-26T08:14:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:19 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 897.619578 secs 2024-07-26T08:14:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:19 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 897.619578 secs 2024-07-26T08:14:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:20 smithi115 ceph-mon[35364]: pgmap v1232: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:20 smithi181 ceph-mon[35869]: pgmap v1232: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:20 smithi097 ceph-mon[32231]: pgmap v1232: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:22 smithi181 ceph-mon[35869]: pgmap v1233: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:22 smithi097 ceph-mon[32231]: pgmap v1233: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:22 smithi115 ceph-mon[35364]: pgmap v1233: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:24 smithi181 ceph-mon[35869]: pgmap v1234: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:24 smithi097 ceph-mon[32231]: pgmap v1234: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:24 smithi115 ceph-mon[35364]: pgmap v1234: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 902.619758 secs 2024-07-26T08:14:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 902.619758 secs 2024-07-26T08:14:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 902.619758 secs 2024-07-26T08:14:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:26 smithi097 ceph-mon[32231]: pgmap v1235: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:26 smithi115 ceph-mon[35364]: pgmap v1235: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:26 smithi181 ceph-mon[35869]: pgmap v1235: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:28 smithi181 ceph-mon[35869]: pgmap v1236: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:28 smithi097 ceph-mon[32231]: pgmap v1236: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:28 smithi115 ceph-mon[35364]: pgmap v1236: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:29 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 907.619895 secs 2024-07-26T08:14:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:29 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 907.619895 secs 2024-07-26T08:14:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:29 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 907.619895 secs 2024-07-26T08:14:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:30 smithi115 ceph-mon[35364]: pgmap v1237: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:30 smithi181 ceph-mon[35869]: pgmap v1237: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:30 smithi097 ceph-mon[32231]: pgmap v1237: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:32 smithi181 ceph-mon[35869]: pgmap v1238: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:32.739+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:14:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:32.741+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:14:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:32.741+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:14:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:32.741+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:14:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:32.741+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:14:32.758 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:32 smithi097 ceph-mon[32231]: pgmap v1238: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:32 smithi115 ceph-mon[35364]: pgmap v1238: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:34 smithi181 ceph-mon[35869]: pgmap v1239: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:34 smithi097 ceph-mon[32231]: pgmap v1239: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:34 smithi115 ceph-mon[35364]: pgmap v1239: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:14:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 912.620034 secs 2024-07-26T08:14:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 912.620034 secs 2024-07-26T08:14:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 912.620034 secs 2024-07-26T08:14:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:36 smithi097 ceph-mon[32231]: pgmap v1240: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:36 smithi115 ceph-mon[35364]: pgmap v1240: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:36 smithi181 ceph-mon[35869]: pgmap v1240: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:37.034+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:14:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:37.034+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:14:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:37.035+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:14:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:37.035+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:14:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:38 smithi181 ceph-mon[35869]: pgmap v1241: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:38 smithi097 ceph-mon[32231]: pgmap v1241: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:38 smithi115 ceph-mon[35364]: pgmap v1241: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:39 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 917.620179 secs 2024-07-26T08:14:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:39 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 917.620179 secs 2024-07-26T08:14:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:39 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 917.620179 secs 2024-07-26T08:14:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:40 smithi115 ceph-mon[35364]: pgmap v1242: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:40 smithi181 ceph-mon[35869]: pgmap v1242: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:40 smithi097 ceph-mon[32231]: pgmap v1242: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:42 smithi181 ceph-mon[35869]: pgmap v1243: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:42 smithi097 ceph-mon[32231]: pgmap v1243: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:42 smithi115 ceph-mon[35364]: pgmap v1243: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:14:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:14:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:14:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:14:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:14:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:14:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:44 smithi097 ceph-mon[32231]: pgmap v1244: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:14:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:44 smithi097 ceph-mon[32231]: pgmap v1245: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:14:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:14:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:14:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:14:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:44 smithi115 ceph-mon[35364]: pgmap v1244: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:14:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:44 smithi115 ceph-mon[35364]: pgmap v1245: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:14:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:14:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:14:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:14:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:44 smithi181 ceph-mon[35869]: pgmap v1244: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:14:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:44 smithi181 ceph-mon[35869]: pgmap v1245: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:14:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:14:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 922.620341 secs 2024-07-26T08:14:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 922.620341 secs 2024-07-26T08:14:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 922.620341 secs 2024-07-26T08:14:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:46 smithi115 ceph-mon[35364]: pgmap v1246: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:46 smithi181 ceph-mon[35869]: pgmap v1246: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:46 smithi097 ceph-mon[32231]: pgmap v1246: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:47.743+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:14:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:47.745+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:14:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:47.745+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:14:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:47.745+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:14:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:14:47.745+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:14:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:48 smithi115 ceph-mon[35364]: pgmap v1247: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:48 smithi181 ceph-mon[35869]: pgmap v1247: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:48 smithi097 ceph-mon[32231]: pgmap v1247: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:49 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 927.620468 secs 2024-07-26T08:14:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:49 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 927.620468 secs 2024-07-26T08:14:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:49 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 927.620468 secs 2024-07-26T08:14:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:50 smithi115 ceph-mon[35364]: pgmap v1248: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:50 smithi181 ceph-mon[35869]: pgmap v1248: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:50 smithi097 ceph-mon[32231]: pgmap v1248: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:14:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:52 smithi115 ceph-mon[35364]: pgmap v1249: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:52 smithi181 ceph-mon[35869]: pgmap v1249: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:52 smithi097 ceph-mon[32231]: pgmap v1249: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:14:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:54 smithi115 ceph-mon[35364]: pgmap v1250: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:14:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:54 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 932.620611 secs 2024-07-26T08:14:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:54 smithi181 ceph-mon[35869]: pgmap v1250: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:14:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:54 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 932.620611 secs 2024-07-26T08:14:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:54 smithi097 ceph-mon[32231]: pgmap v1250: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:14:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:54 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 932.620611 secs 2024-07-26T08:14:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:56 smithi115 ceph-mon[35364]: pgmap v1251: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:14:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:56 smithi181 ceph-mon[35869]: pgmap v1251: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:14:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:56 smithi097 ceph-mon[32231]: pgmap v1251: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:14:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:14:58 smithi115 ceph-mon[35364]: pgmap v1252: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:14:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:14:58 smithi181 ceph-mon[35869]: pgmap v1252: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:14:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:14:58 smithi097 ceph-mon[32231]: pgmap v1252: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:15:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 937.620759 secs 2024-07-26T08:15:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:00 smithi097 ceph-mon[32231]: pgmap v1253: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 937.620759 secs 2024-07-26T08:15:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:00 smithi115 ceph-mon[35364]: pgmap v1253: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 937.620759 secs 2024-07-26T08:15:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:00 smithi181 ceph-mon[35869]: pgmap v1253: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:02.748+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:15:02.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:02.749+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:15:02.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:02.749+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:15:02.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:02.749+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:15:02.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:02.749+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:15:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:02 smithi115 ceph-mon[35364]: pgmap v1254: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:15:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:02 smithi181 ceph-mon[35869]: pgmap v1254: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:15:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:02 smithi097 ceph-mon[32231]: pgmap v1254: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:15:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:04 smithi115 ceph-mon[35364]: pgmap v1255: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:04 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 942.620907 secs 2024-07-26T08:15:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:04 smithi181 ceph-mon[35869]: pgmap v1255: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:04 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 942.620907 secs 2024-07-26T08:15:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:04 smithi097 ceph-mon[32231]: pgmap v1255: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:04 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 942.620907 secs 2024-07-26T08:15:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:06 smithi115 ceph-mon[35364]: pgmap v1256: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:06 smithi181 ceph-mon[35869]: pgmap v1256: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:06 smithi097 ceph-mon[32231]: pgmap v1256: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:15:09.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:08 smithi115 ceph-mon[35364]: pgmap v1257: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:08 smithi181 ceph-mon[35869]: pgmap v1257: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:08 smithi097 ceph-mon[32231]: pgmap v1257: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:09 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 947.621046 secs 2024-07-26T08:15:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:09 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 947.621046 secs 2024-07-26T08:15:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:09 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 947.621046 secs 2024-07-26T08:15:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:10 smithi115 ceph-mon[35364]: pgmap v1258: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:10 smithi181 ceph-mon[35869]: pgmap v1258: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:10 smithi097 ceph-mon[32231]: pgmap v1258: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:12 smithi115 ceph-mon[35364]: pgmap v1259: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:12 smithi181 ceph-mon[35869]: pgmap v1259: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:12 smithi097 ceph-mon[32231]: pgmap v1259: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:14 smithi115 ceph-mon[35364]: pgmap v1260: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:14 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 952.621180 secs 2024-07-26T08:15:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:14 smithi181 ceph-mon[35869]: pgmap v1260: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:14 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 952.621180 secs 2024-07-26T08:15:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:14 smithi097 ceph-mon[32231]: pgmap v1260: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:14 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 952.621180 secs 2024-07-26T08:15:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:16 smithi115 ceph-mon[35364]: pgmap v1261: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:16 smithi181 ceph-mon[35869]: pgmap v1261: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:16 smithi097 ceph-mon[32231]: pgmap v1261: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:17.758+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:15:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:17.759+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:15:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:17.760+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:15:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:17.760+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:15:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:17.760+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:15:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:18 smithi115 ceph-mon[35364]: pgmap v1262: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:18 smithi181 ceph-mon[35869]: pgmap v1262: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:18 smithi097 ceph-mon[32231]: pgmap v1262: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:19 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 957.621324 secs 2024-07-26T08:15:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:19 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 957.621324 secs 2024-07-26T08:15:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:19 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 957.621324 secs 2024-07-26T08:15:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:20 smithi097 ceph-mon[32231]: pgmap v1263: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:20 smithi115 ceph-mon[35364]: pgmap v1263: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:20 smithi181 ceph-mon[35869]: pgmap v1263: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:22 smithi097 ceph-mon[32231]: pgmap v1264: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:22 smithi115 ceph-mon[35364]: pgmap v1264: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:22 smithi181 ceph-mon[35869]: pgmap v1264: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:24 smithi097 ceph-mon[32231]: pgmap v1265: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:24 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 962.621461 secs 2024-07-26T08:15:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:25 smithi115 ceph-mon[35364]: pgmap v1265: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 962.621461 secs 2024-07-26T08:15:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:24 smithi181 ceph-mon[35869]: pgmap v1265: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 962.621461 secs 2024-07-26T08:15:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:27 smithi115 ceph-mon[35364]: pgmap v1266: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:27 smithi181 ceph-mon[35869]: pgmap v1266: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:27 smithi097 ceph-mon[32231]: pgmap v1266: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:29 smithi115 ceph-mon[35364]: pgmap v1267: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:29 smithi181 ceph-mon[35869]: pgmap v1267: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:29 smithi097 ceph-mon[32231]: pgmap v1267: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 967.621591 secs 2024-07-26T08:15:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 967.621591 secs 2024-07-26T08:15:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 967.621591 secs 2024-07-26T08:15:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:31 smithi115 ceph-mon[35364]: pgmap v1268: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:31 smithi181 ceph-mon[35869]: pgmap v1268: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:31 smithi097 ceph-mon[32231]: pgmap v1268: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:32.764+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:15:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:32.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:15:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:32.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:15:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:32.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:15:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:32.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:15:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:33 smithi115 ceph-mon[35364]: pgmap v1269: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:33 smithi181 ceph-mon[35869]: pgmap v1269: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:33 smithi097 ceph-mon[32231]: pgmap v1269: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:35.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:35 smithi097 ceph-mon[32231]: pgmap v1270: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:35.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 972.621679 secs 2024-07-26T08:15:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:35 smithi115 ceph-mon[35364]: pgmap v1270: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 972.621679 secs 2024-07-26T08:15:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:35 smithi181 ceph-mon[35869]: pgmap v1270: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 972.621679 secs 2024-07-26T08:15:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:37 smithi115 ceph-mon[35364]: pgmap v1271: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:15:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:37 smithi181 ceph-mon[35869]: pgmap v1271: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:15:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:37 smithi097 ceph-mon[32231]: pgmap v1271: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:15:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:39 smithi115 ceph-mon[35364]: pgmap v1272: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:39 smithi181 ceph-mon[35869]: pgmap v1272: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:39 smithi097 ceph-mon[32231]: pgmap v1272: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 977.621806 secs 2024-07-26T08:15:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 977.621806 secs 2024-07-26T08:15:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 977.621806 secs 2024-07-26T08:15:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:41 smithi115 ceph-mon[35364]: pgmap v1273: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:41 smithi181 ceph-mon[35869]: pgmap v1273: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:41 smithi097 ceph-mon[32231]: pgmap v1273: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:43 smithi115 ceph-mon[35364]: pgmap v1274: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:43 smithi181 ceph-mon[35869]: pgmap v1274: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:43 smithi097 ceph-mon[32231]: pgmap v1274: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:15:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:45 smithi115 ceph-mon[35364]: pgmap v1275: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:15:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 982.621935 secs 2024-07-26T08:15:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:45 smithi181 ceph-mon[35869]: pgmap v1275: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:15:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 982.621935 secs 2024-07-26T08:15:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:45 smithi097 ceph-mon[32231]: pgmap v1275: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:15:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:15:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 982.621935 secs 2024-07-26T08:15:47.417 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:47 smithi097 ceph-mon[32231]: pgmap v1276: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:15:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:47 smithi115 ceph-mon[35364]: pgmap v1276: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:15:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:47 smithi181 ceph-mon[35869]: pgmap v1276: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:15:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:47.773+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:15:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:47.775+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:15:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:47.775+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:15:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:47.775+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:15:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:15:47.775+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:15:49.387 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:49 smithi097 ceph-mon[32231]: pgmap v1277: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:49 smithi115 ceph-mon[35364]: pgmap v1277: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:49 smithi181 ceph-mon[35869]: pgmap v1277: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:15:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 987.622075 secs 2024-07-26T08:15:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 987.622075 secs 2024-07-26T08:15:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 987.622075 secs 2024-07-26T08:15:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:51 smithi115 ceph-mon[35364]: pgmap v1278: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:15:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:15:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:51 smithi115 ceph-mon[35364]: pgmap v1279: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:15:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:51 smithi115 ceph-mon[35364]: pgmap v1280: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T08:15:51.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:51 smithi181 ceph-mon[35869]: pgmap v1278: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:15:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:15:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:51 smithi181 ceph-mon[35869]: pgmap v1279: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:15:51.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:51 smithi181 ceph-mon[35869]: pgmap v1280: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T08:15:51.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:51 smithi097 ceph-mon[32231]: pgmap v1278: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:15:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:15:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:15:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:51 smithi097 ceph-mon[32231]: pgmap v1279: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:15:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:51 smithi097 ceph-mon[32231]: pgmap v1280: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T08:15:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:15:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:53 smithi115 ceph-mon[35364]: pgmap v1281: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T08:15:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:53 smithi181 ceph-mon[35869]: pgmap v1281: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T08:15:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:53 smithi097 ceph-mon[32231]: pgmap v1281: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T08:15:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:55 smithi115 ceph-mon[35364]: pgmap v1282: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:15:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 992.622287 secs 2024-07-26T08:15:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:55 smithi181 ceph-mon[35869]: pgmap v1282: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:15:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 992.622287 secs 2024-07-26T08:15:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:55 smithi097 ceph-mon[32231]: pgmap v1282: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:15:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 992.622287 secs 2024-07-26T08:15:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:57 smithi115 ceph-mon[35364]: pgmap v1283: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T08:15:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:57 smithi181 ceph-mon[35869]: pgmap v1283: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T08:15:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:57 smithi097 ceph-mon[32231]: pgmap v1283: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T08:15:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:15:59 smithi115 ceph-mon[35364]: pgmap v1284: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T08:15:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:15:59 smithi181 ceph-mon[35869]: pgmap v1284: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T08:15:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:15:59 smithi097 ceph-mon[32231]: pgmap v1284: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T08:16:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 997.622446 secs 2024-07-26T08:16:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 997.622446 secs 2024-07-26T08:16:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 997.622446 secs 2024-07-26T08:16:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:01 smithi115 ceph-mon[35364]: pgmap v1285: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:16:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:01 smithi181 ceph-mon[35869]: pgmap v1285: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:16:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:01 smithi097 ceph-mon[32231]: pgmap v1285: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:16:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:02.778+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:16:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:02.779+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:16:02.781 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:02.779+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:16:02.781 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:02.779+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:16:02.781 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:02.779+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:16:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:03 smithi115 ceph-mon[35364]: pgmap v1286: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:16:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:03 smithi181 ceph-mon[35869]: pgmap v1286: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:16:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:03 smithi097 ceph-mon[32231]: pgmap v1286: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:16:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:05 smithi115 ceph-mon[35364]: pgmap v1287: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:16:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1002.622543 secs 2024-07-26T08:16:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:05 smithi181 ceph-mon[35869]: pgmap v1287: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:16:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1002.622543 secs 2024-07-26T08:16:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:05 smithi097 ceph-mon[32231]: pgmap v1287: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:16:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1002.622543 secs 2024-07-26T08:16:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:07.050+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:16:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:07.050+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:16:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:07.051+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:16:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:07.051+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:16:07.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:07 smithi115 ceph-mon[35364]: pgmap v1288: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:07 smithi181 ceph-mon[35869]: pgmap v1288: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:07 smithi097 ceph-mon[32231]: pgmap v1288: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:09 smithi115 ceph-mon[35364]: pgmap v1289: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:09 smithi181 ceph-mon[35869]: pgmap v1289: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:09 smithi097 ceph-mon[32231]: pgmap v1289: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1007.622690 secs 2024-07-26T08:16:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1007.622690 secs 2024-07-26T08:16:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1007.622690 secs 2024-07-26T08:16:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:11 smithi115 ceph-mon[35364]: pgmap v1290: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:11 smithi181 ceph-mon[35869]: pgmap v1290: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:11 smithi097 ceph-mon[32231]: pgmap v1290: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:13 smithi115 ceph-mon[35364]: pgmap v1291: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:13 smithi181 ceph-mon[35869]: pgmap v1291: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:13 smithi097 ceph-mon[32231]: pgmap v1291: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:15 smithi115 ceph-mon[35364]: pgmap v1292: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1012.622826 secs 2024-07-26T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:15 smithi181 ceph-mon[35869]: pgmap v1292: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1012.622826 secs 2024-07-26T08:16:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:15 smithi097 ceph-mon[32231]: pgmap v1292: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1012.622826 secs 2024-07-26T08:16:17.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:17 smithi097 ceph-mon[32231]: pgmap v1293: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:17 smithi115 ceph-mon[35364]: pgmap v1293: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:17 smithi181 ceph-mon[35869]: pgmap v1293: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:17.788+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:16:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:17.789+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:16:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:17.789+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:16:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:17.789+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:16:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:17.789+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:16:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:19 smithi115 ceph-mon[35364]: pgmap v1294: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:19 smithi181 ceph-mon[35869]: pgmap v1294: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:19 smithi097 ceph-mon[32231]: pgmap v1294: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1017.623002 secs 2024-07-26T08:16:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1017.623002 secs 2024-07-26T08:16:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1017.623002 secs 2024-07-26T08:16:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:21 smithi115 ceph-mon[35364]: pgmap v1295: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:21 smithi181 ceph-mon[35869]: pgmap v1295: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:21 smithi097 ceph-mon[32231]: pgmap v1295: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:23 smithi115 ceph-mon[35364]: pgmap v1296: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:23 smithi181 ceph-mon[35869]: pgmap v1296: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:23 smithi097 ceph-mon[32231]: pgmap v1296: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:25 smithi115 ceph-mon[35364]: pgmap v1297: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1022.623146 secs 2024-07-26T08:16:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:25 smithi181 ceph-mon[35869]: pgmap v1297: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1022.623146 secs 2024-07-26T08:16:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:25 smithi097 ceph-mon[32231]: pgmap v1297: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1022.623146 secs 2024-07-26T08:16:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:27 smithi115 ceph-mon[35364]: pgmap v1298: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:27 smithi181 ceph-mon[35869]: pgmap v1298: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:27 smithi097 ceph-mon[32231]: pgmap v1298: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:29 smithi115 ceph-mon[35364]: pgmap v1299: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:29 smithi181 ceph-mon[35869]: pgmap v1299: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:29 smithi097 ceph-mon[32231]: pgmap v1299: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1027.623288 secs 2024-07-26T08:16:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1027.623288 secs 2024-07-26T08:16:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1027.623288 secs 2024-07-26T08:16:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:31 smithi115 ceph-mon[35364]: pgmap v1300: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:31 smithi181 ceph-mon[35869]: pgmap v1300: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:31 smithi097 ceph-mon[32231]: pgmap v1300: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:32.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:32.793+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:16:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:32.794+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:16:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:32.794+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:16:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:32.794+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:16:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:32.794+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:16:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:33 smithi115 ceph-mon[35364]: pgmap v1301: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:33 smithi181 ceph-mon[35869]: pgmap v1301: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:33 smithi097 ceph-mon[32231]: pgmap v1301: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:34 smithi115 ceph-mon[35364]: pgmap v1302: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:34 smithi181 ceph-mon[35869]: pgmap v1302: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:34 smithi097 ceph-mon[32231]: pgmap v1302: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:16:35.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1032.623384 secs 2024-07-26T08:16:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1032.623384 secs 2024-07-26T08:16:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1032.623384 secs 2024-07-26T08:16:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:36 smithi115 ceph-mon[35364]: pgmap v1303: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:36 smithi181 ceph-mon[35869]: pgmap v1303: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:36 smithi097 ceph-mon[32231]: pgmap v1303: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:39 smithi115 ceph-mon[35364]: pgmap v1304: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:39 smithi181 ceph-mon[35869]: pgmap v1304: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:39 smithi097 ceph-mon[32231]: pgmap v1304: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:16:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1037.623483 secs 2024-07-26T08:16:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1037.623483 secs 2024-07-26T08:16:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1037.623483 secs 2024-07-26T08:16:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:41 smithi115 ceph-mon[35364]: pgmap v1305: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:41 smithi181 ceph-mon[35869]: pgmap v1305: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:41 smithi097 ceph-mon[32231]: pgmap v1305: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:16:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:43 smithi115 ceph-mon[35364]: pgmap v1306: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:43 smithi181 ceph-mon[35869]: pgmap v1306: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:43 smithi097 ceph-mon[32231]: pgmap v1306: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:16:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:45 smithi115 ceph-mon[35364]: pgmap v1307: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:16:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1042.623631 secs 2024-07-26T08:16:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:45 smithi181 ceph-mon[35869]: pgmap v1307: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:16:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1042.623631 secs 2024-07-26T08:16:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:45 smithi097 ceph-mon[32231]: pgmap v1307: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:16:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1042.623631 secs 2024-07-26T08:16:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:47 smithi115 ceph-mon[35364]: pgmap v1308: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:47 smithi181 ceph-mon[35869]: pgmap v1308: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:47 smithi097 ceph-mon[32231]: pgmap v1308: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:47.800+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:16:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:47.801+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:47.801+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:47.801+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:16:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:16:47.801+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:16:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:49 smithi115 ceph-mon[35364]: pgmap v1309: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:16:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:49 smithi181 ceph-mon[35869]: pgmap v1309: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:16:49.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:49 smithi097 ceph-mon[32231]: pgmap v1309: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:16:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1047.623768 secs 2024-07-26T08:16:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1047.623768 secs 2024-07-26T08:16:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1047.623768 secs 2024-07-26T08:16:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:51 smithi115 ceph-mon[35364]: pgmap v1310: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:16:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:16:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:51 smithi181 ceph-mon[35869]: pgmap v1310: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:16:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:16:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:51 smithi097 ceph-mon[32231]: pgmap v1310: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:16:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:16:52.630 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.630 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.630 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.631 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.681 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.682 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.682 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.682 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:53.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:53 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:16:53.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:53 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:16:53.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:53 smithi115 ceph-mon[35364]: pgmap v1311: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:16:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:53 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:16:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:53 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:16:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:53 smithi181 ceph-mon[35869]: pgmap v1311: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:16:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:53 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:16:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:53 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:16:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:53 smithi097 ceph-mon[32231]: pgmap v1311: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:16:55.683 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:55 smithi181 ceph-mon[35869]: pgmap v1312: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:16:55.683 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1052.623903 secs 2024-07-26T08:16:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:55 smithi115 ceph-mon[35364]: pgmap v1312: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:16:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1052.623903 secs 2024-07-26T08:16:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:55 smithi097 ceph-mon[32231]: pgmap v1312: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:16:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1052.623903 secs 2024-07-26T08:16:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: pgmap v1313: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:16:57.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:16:57.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:16:57.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:16:57.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: pgmap v1313: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:16:57.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:16:57.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:16:57.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:16:57.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: pgmap v1313: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:16:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:57.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:16:57.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:16:57.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:16:57.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:16:57.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:16:57.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:16:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:58 smithi115 ceph-mon[35364]: pgmap v1314: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:16:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:16:58 smithi115 ceph-mon[35364]: pgmap v1315: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:16:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:58 smithi181 ceph-mon[35869]: pgmap v1314: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:16:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:16:58 smithi181 ceph-mon[35869]: pgmap v1315: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:16:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:58 smithi097 ceph-mon[32231]: pgmap v1314: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:16:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:16:58 smithi097 ceph-mon[32231]: pgmap v1315: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:17:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:00 smithi115 ceph-mon[35364]: pgmap v1316: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:17:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1057.624032 secs 2024-07-26T08:17:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:00 smithi181 ceph-mon[35869]: pgmap v1316: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:17:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1057.624032 secs 2024-07-26T08:17:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:00 smithi097 ceph-mon[32231]: pgmap v1316: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:17:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1057.624032 secs 2024-07-26T08:17:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:02 smithi115 ceph-mon[35364]: pgmap v1317: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:17:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:02 smithi181 ceph-mon[35869]: pgmap v1317: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:17:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:02 smithi097 ceph-mon[32231]: pgmap v1317: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:17:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:02.805+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:17:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:02.806+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:17:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:02.806+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:17:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:02.806+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:17:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:02.806+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:17:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:04 smithi115 ceph-mon[35364]: pgmap v1318: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:17:04.769 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:04 smithi097 ceph-mon[32231]: pgmap v1318: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:17:04.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:04 smithi181 ceph-mon[35869]: pgmap v1318: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:17:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1062.624170 secs 2024-07-26T08:17:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1062.624170 secs 2024-07-26T08:17:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1062.624170 secs 2024-07-26T08:17:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:06 smithi115 ceph-mon[35364]: pgmap v1319: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:17:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:06 smithi181 ceph-mon[35869]: pgmap v1319: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:17:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:06 smithi097 ceph-mon[32231]: pgmap v1319: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:17:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:07 smithi097 ceph-mon[32231]: pgmap v1320: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:17:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:07 smithi115 ceph-mon[35364]: pgmap v1320: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:17:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:07 smithi181 ceph-mon[35869]: pgmap v1320: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:17:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:09 smithi097 ceph-mon[32231]: pgmap v1321: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:09 smithi115 ceph-mon[35364]: pgmap v1321: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:09 smithi181 ceph-mon[35869]: pgmap v1321: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1067.624306 secs 2024-07-26T08:17:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1067.624306 secs 2024-07-26T08:17:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1067.624306 secs 2024-07-26T08:17:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:11 smithi115 ceph-mon[35364]: pgmap v1322: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:11 smithi181 ceph-mon[35869]: pgmap v1322: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:11 smithi097 ceph-mon[32231]: pgmap v1322: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:13.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:13 smithi115 ceph-mon[35364]: pgmap v1323: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:13 smithi181 ceph-mon[35869]: pgmap v1323: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:13 smithi097 ceph-mon[32231]: pgmap v1323: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:15 smithi115 ceph-mon[35364]: pgmap v1324: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1072.624440 secs 2024-07-26T08:17:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:15 smithi181 ceph-mon[35869]: pgmap v1324: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1072.624440 secs 2024-07-26T08:17:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:15 smithi097 ceph-mon[32231]: pgmap v1324: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1072.624440 secs 2024-07-26T08:17:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:17.814+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:17:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:17.816+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:17:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:17.816+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:17:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:17.816+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:17:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:17.816+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:17:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:17 smithi115 ceph-mon[35364]: pgmap v1325: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:17 smithi181 ceph-mon[35869]: pgmap v1325: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:17 smithi097 ceph-mon[32231]: pgmap v1325: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:19 smithi115 ceph-mon[35364]: pgmap v1326: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:19 smithi181 ceph-mon[35869]: pgmap v1326: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:19 smithi097 ceph-mon[32231]: pgmap v1326: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1077.624580 secs 2024-07-26T08:17:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1077.624580 secs 2024-07-26T08:17:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1077.624580 secs 2024-07-26T08:17:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:21 smithi115 ceph-mon[35364]: pgmap v1327: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:21 smithi181 ceph-mon[35869]: pgmap v1327: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:21 smithi097 ceph-mon[32231]: pgmap v1327: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:23 smithi115 ceph-mon[35364]: pgmap v1328: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:23 smithi181 ceph-mon[35869]: pgmap v1328: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:23 smithi097 ceph-mon[32231]: pgmap v1328: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:25 smithi181 ceph-mon[35869]: pgmap v1329: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1082.624722 secs 2024-07-26T08:17:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:25 smithi115 ceph-mon[35364]: pgmap v1329: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1082.624722 secs 2024-07-26T08:17:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:25 smithi097 ceph-mon[32231]: pgmap v1329: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1082.624722 secs 2024-07-26T08:17:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:27 smithi115 ceph-mon[35364]: pgmap v1330: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:27 smithi181 ceph-mon[35869]: pgmap v1330: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:27 smithi097 ceph-mon[32231]: pgmap v1330: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:29 smithi115 ceph-mon[35364]: pgmap v1331: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:29 smithi181 ceph-mon[35869]: pgmap v1331: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:29 smithi097 ceph-mon[32231]: pgmap v1331: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1087.624884 secs 2024-07-26T08:17:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1087.624884 secs 2024-07-26T08:17:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1087.624884 secs 2024-07-26T08:17:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:31 smithi115 ceph-mon[35364]: pgmap v1332: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:31 smithi181 ceph-mon[35869]: pgmap v1332: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:31 smithi097 ceph-mon[32231]: pgmap v1332: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:32.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:32.820+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:17:32.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:32.821+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:17:32.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:32.821+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:17:32.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:32.821+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:17:32.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:32.821+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:17:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:33 smithi115 ceph-mon[35364]: pgmap v1333: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:33 smithi181 ceph-mon[35869]: pgmap v1333: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:33 smithi097 ceph-mon[32231]: pgmap v1333: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:35 smithi115 ceph-mon[35364]: pgmap v1334: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1092.625008 secs 2024-07-26T08:17:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:35 smithi181 ceph-mon[35869]: pgmap v1334: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1092.625008 secs 2024-07-26T08:17:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:35 smithi097 ceph-mon[32231]: pgmap v1334: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1092.625008 secs 2024-07-26T08:17:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:37 smithi097 ceph-mon[32231]: pgmap v1335: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:37 smithi115 ceph-mon[35364]: pgmap v1335: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:37 smithi181 ceph-mon[35869]: pgmap v1335: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:39 smithi115 ceph-mon[35364]: pgmap v1336: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:39 smithi181 ceph-mon[35869]: pgmap v1336: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:39 smithi097 ceph-mon[32231]: pgmap v1336: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1097.625162 secs 2024-07-26T08:17:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1097.625162 secs 2024-07-26T08:17:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1097.625162 secs 2024-07-26T08:17:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:41 smithi115 ceph-mon[35364]: pgmap v1337: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:41 smithi181 ceph-mon[35869]: pgmap v1337: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:41 smithi097 ceph-mon[32231]: pgmap v1337: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:43 smithi115 ceph-mon[35364]: pgmap v1338: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:43.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:43 smithi181 ceph-mon[35869]: pgmap v1338: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:43 smithi097 ceph-mon[32231]: pgmap v1338: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:45 smithi115 ceph-mon[35364]: pgmap v1339: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1102.625331 secs 2024-07-26T08:17:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:45 smithi181 ceph-mon[35869]: pgmap v1339: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1102.625331 secs 2024-07-26T08:17:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:45 smithi097 ceph-mon[32231]: pgmap v1339: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1102.625331 secs 2024-07-26T08:17:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:47.826+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:17:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:47.827+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:17:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:47.827+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:17:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:47.827+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:17:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:17:47.827+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:17:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:47 smithi115 ceph-mon[35364]: pgmap v1340: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:47 smithi181 ceph-mon[35869]: pgmap v1340: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:47 smithi097 ceph-mon[32231]: pgmap v1340: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:49 smithi115 ceph-mon[35364]: pgmap v1341: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:49 smithi181 ceph-mon[35869]: pgmap v1341: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:49 smithi097 ceph-mon[32231]: pgmap v1341: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1107.625469 secs 2024-07-26T08:17:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1107.625469 secs 2024-07-26T08:17:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1107.625469 secs 2024-07-26T08:17:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:51 smithi115 ceph-mon[35364]: pgmap v1342: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:51 smithi181 ceph-mon[35869]: pgmap v1342: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:51 smithi097 ceph-mon[32231]: pgmap v1342: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:53 smithi115 ceph-mon[35364]: pgmap v1343: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:53 smithi181 ceph-mon[35869]: pgmap v1343: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:53 smithi097 ceph-mon[32231]: pgmap v1343: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:55 smithi115 ceph-mon[35364]: pgmap v1344: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1112.625606 secs 2024-07-26T08:17:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:55 smithi181 ceph-mon[35869]: pgmap v1344: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1112.625606 secs 2024-07-26T08:17:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:55 smithi097 ceph-mon[32231]: pgmap v1344: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:17:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1112.625606 secs 2024-07-26T08:17:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:57 smithi115 ceph-mon[35364]: pgmap v1345: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:17:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:57 smithi181 ceph-mon[35869]: pgmap v1345: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:17:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:57 smithi097 ceph-mon[32231]: pgmap v1345: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:17:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:17:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:17:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:17:58.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:17:58.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:17:58.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: pgmap v1346: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:17:58.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: pgmap v1347: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:58.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:17:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: pgmap v1346: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: pgmap v1347: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:17:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: pgmap v1346: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: pgmap v1347: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:17:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:18:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:17:59 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1117.625740 secs 2024-07-26T08:18:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:17:59 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1117.625740 secs 2024-07-26T08:18:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:17:59 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1117.625740 secs 2024-07-26T08:18:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:00 smithi115 ceph-mon[35364]: pgmap v1348: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:00 smithi181 ceph-mon[35869]: pgmap v1348: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:00 smithi097 ceph-mon[32231]: pgmap v1348: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:02.831+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:18:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:02.832+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:18:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:02.832+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:18:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:02.832+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:18:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:02.832+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:18:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:03 smithi115 ceph-mon[35364]: pgmap v1349: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:18:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:03 smithi181 ceph-mon[35869]: pgmap v1349: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:18:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:03 smithi097 ceph-mon[32231]: pgmap v1349: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:18:05.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:05 smithi097 ceph-mon[32231]: pgmap v1350: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:05.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1122.625829 secs 2024-07-26T08:18:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:05 smithi115 ceph-mon[35364]: pgmap v1350: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1122.625829 secs 2024-07-26T08:18:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:05 smithi181 ceph-mon[35869]: pgmap v1350: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1122.625829 secs 2024-07-26T08:18:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:07 smithi115 ceph-mon[35364]: pgmap v1351: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:07 smithi181 ceph-mon[35869]: pgmap v1351: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:07 smithi097 ceph-mon[32231]: pgmap v1351: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:09 smithi115 ceph-mon[35364]: pgmap v1352: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:18:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:09 smithi181 ceph-mon[35869]: pgmap v1352: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:18:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:09 smithi097 ceph-mon[32231]: pgmap v1352: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:18:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1127.625966 secs 2024-07-26T08:18:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1127.625966 secs 2024-07-26T08:18:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1127.625966 secs 2024-07-26T08:18:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:11 smithi115 ceph-mon[35364]: pgmap v1353: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:11 smithi181 ceph-mon[35869]: pgmap v1353: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:11 smithi097 ceph-mon[32231]: pgmap v1353: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:13 smithi115 ceph-mon[35364]: pgmap v1354: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:13 smithi181 ceph-mon[35869]: pgmap v1354: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:13 smithi097 ceph-mon[32231]: pgmap v1354: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:15 smithi115 ceph-mon[35364]: pgmap v1355: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1132.626102 secs 2024-07-26T08:18:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:15 smithi181 ceph-mon[35869]: pgmap v1355: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1132.626102 secs 2024-07-26T08:18:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:15 smithi097 ceph-mon[32231]: pgmap v1355: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1132.626102 secs 2024-07-26T08:18:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:17 smithi115 ceph-mon[35364]: pgmap v1356: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:17 smithi181 ceph-mon[35869]: pgmap v1356: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:17 smithi097 ceph-mon[32231]: pgmap v1356: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:17.834+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:18:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:17.835+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:18:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:17.835+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:18:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:17.835+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:18:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:17.835+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:18:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:19 smithi115 ceph-mon[35364]: pgmap v1357: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:19 smithi181 ceph-mon[35869]: pgmap v1357: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:19 smithi097 ceph-mon[32231]: pgmap v1357: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1137.626294 secs 2024-07-26T08:18:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1137.626294 secs 2024-07-26T08:18:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1137.626294 secs 2024-07-26T08:18:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:21 smithi115 ceph-mon[35364]: pgmap v1358: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:21 smithi181 ceph-mon[35869]: pgmap v1358: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:21 smithi097 ceph-mon[32231]: pgmap v1358: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:23 smithi115 ceph-mon[35364]: pgmap v1359: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:23 smithi181 ceph-mon[35869]: pgmap v1359: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:23 smithi097 ceph-mon[32231]: pgmap v1359: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:25 smithi115 ceph-mon[35364]: pgmap v1360: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1142.626453 secs 2024-07-26T08:18:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:25 smithi181 ceph-mon[35869]: pgmap v1360: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1142.626453 secs 2024-07-26T08:18:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:25 smithi097 ceph-mon[32231]: pgmap v1360: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1142.626453 secs 2024-07-26T08:18:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:27 smithi115 ceph-mon[35364]: pgmap v1361: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:27 smithi181 ceph-mon[35869]: pgmap v1361: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:27 smithi097 ceph-mon[32231]: pgmap v1361: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:29 smithi115 ceph-mon[35364]: pgmap v1362: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:29 smithi181 ceph-mon[35869]: pgmap v1362: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:29 smithi097 ceph-mon[32231]: pgmap v1362: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1147.626584 secs 2024-07-26T08:18:30.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1147.626584 secs 2024-07-26T08:18:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1147.626584 secs 2024-07-26T08:18:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:31 smithi115 ceph-mon[35364]: pgmap v1363: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:31 smithi181 ceph-mon[35869]: pgmap v1363: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:31 smithi097 ceph-mon[32231]: pgmap v1363: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:32.838+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:18:32.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:32.841+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:18:32.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:32.841+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:18:32.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:32.841+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:18:32.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:32.841+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:18:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:33 smithi115 ceph-mon[35364]: pgmap v1364: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:33 smithi181 ceph-mon[35869]: pgmap v1364: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:33 smithi097 ceph-mon[32231]: pgmap v1364: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:35 smithi115 ceph-mon[35364]: pgmap v1365: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1152.626720 secs 2024-07-26T08:18:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:35 smithi181 ceph-mon[35869]: pgmap v1365: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1152.626720 secs 2024-07-26T08:18:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:35 smithi097 ceph-mon[32231]: pgmap v1365: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1152.626720 secs 2024-07-26T08:18:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:37 smithi115 ceph-mon[35364]: pgmap v1366: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:37 smithi181 ceph-mon[35869]: pgmap v1366: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:37 smithi097 ceph-mon[32231]: pgmap v1366: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:39 smithi115 ceph-mon[35364]: pgmap v1367: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:39 smithi181 ceph-mon[35869]: pgmap v1367: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:39 smithi097 ceph-mon[32231]: pgmap v1367: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:40.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1157.626857 secs 2024-07-26T08:18:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1157.626857 secs 2024-07-26T08:18:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1157.626857 secs 2024-07-26T08:18:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:41 smithi115 ceph-mon[35364]: pgmap v1368: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:41 smithi181 ceph-mon[35869]: pgmap v1368: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:41 smithi097 ceph-mon[32231]: pgmap v1368: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:43 smithi115 ceph-mon[35364]: pgmap v1369: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:43 smithi181 ceph-mon[35869]: pgmap v1369: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:43 smithi097 ceph-mon[32231]: pgmap v1369: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:45 smithi115 ceph-mon[35364]: pgmap v1370: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1162.626962 secs 2024-07-26T08:18:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:45 smithi181 ceph-mon[35869]: pgmap v1370: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1162.626962 secs 2024-07-26T08:18:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:45 smithi097 ceph-mon[32231]: pgmap v1370: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1162.626962 secs 2024-07-26T08:18:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:47 smithi115 ceph-mon[35364]: pgmap v1371: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:47 smithi181 ceph-mon[35869]: pgmap v1371: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:47 smithi097 ceph-mon[32231]: pgmap v1371: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:47.849+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:18:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:47.850+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:18:47.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:47.850+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:18:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:47.850+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:18:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:18:47.850+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:18:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:49 smithi181 ceph-mon[35869]: pgmap v1372: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:49 smithi097 ceph-mon[32231]: pgmap v1372: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:49 smithi115 ceph-mon[35364]: pgmap v1372: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1167.627098 secs 2024-07-26T08:18:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1167.627098 secs 2024-07-26T08:18:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1167.627098 secs 2024-07-26T08:18:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:51 smithi097 ceph-mon[32231]: pgmap v1373: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:51 smithi115 ceph-mon[35364]: pgmap v1373: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:51 smithi181 ceph-mon[35869]: pgmap v1373: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:18:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:53 smithi097 ceph-mon[32231]: pgmap v1374: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:53 smithi115 ceph-mon[35364]: pgmap v1374: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:53 smithi181 ceph-mon[35869]: pgmap v1374: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:18:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:55 smithi115 ceph-mon[35364]: pgmap v1375: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1172.627280 secs 2024-07-26T08:18:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:55 smithi181 ceph-mon[35869]: pgmap v1375: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1172.627280 secs 2024-07-26T08:18:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:55 smithi097 ceph-mon[32231]: pgmap v1375: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:18:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1172.627280 secs 2024-07-26T08:18:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:57 smithi115 ceph-mon[35364]: pgmap v1376: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:57 smithi181 ceph-mon[35869]: pgmap v1376: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:57 smithi097 ceph-mon[32231]: pgmap v1376: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:18:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:59 smithi115 ceph-mon[35364]: pgmap v1377: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:18:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:18:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:18:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:18:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:59 smithi181 ceph-mon[35869]: pgmap v1377: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:18:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:18:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:18:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:18:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:59 smithi097 ceph-mon[32231]: pgmap v1377: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:18:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:18:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:18:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:18:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:19:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1177.627415 secs 2024-07-26T08:19:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:19:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:19:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:19:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1177.627415 secs 2024-07-26T08:19:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:19:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:19:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:19:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1177.627415 secs 2024-07-26T08:19:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:19:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:19:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:19:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:01 smithi115 ceph-mon[35364]: pgmap v1378: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:01 smithi181 ceph-mon[35869]: pgmap v1378: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:01.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:01 smithi097 ceph-mon[32231]: pgmap v1378: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:02.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:02.854+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:19:02.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:02.856+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:19:02.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:02.856+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:19:02.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:02.856+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:19:02.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:02.856+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:19:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:03 smithi115 ceph-mon[35364]: pgmap v1379: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:19:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:03 smithi181 ceph-mon[35869]: pgmap v1379: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:19:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:03 smithi097 ceph-mon[32231]: pgmap v1379: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:19:05.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:05 smithi097 ceph-mon[32231]: pgmap v1380: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:05.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1182.627511 secs 2024-07-26T08:19:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:05 smithi115 ceph-mon[35364]: pgmap v1380: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1182.627511 secs 2024-07-26T08:19:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:05 smithi181 ceph-mon[35869]: pgmap v1380: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1182.627511 secs 2024-07-26T08:19:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:07.082+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:19:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:07.082+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:19:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:07.082+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:19:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:07.082+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:19:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:07 smithi115 ceph-mon[35364]: pgmap v1381: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:19:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:07 smithi181 ceph-mon[35869]: pgmap v1381: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:19:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:07 smithi097 ceph-mon[32231]: pgmap v1381: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:19:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:09 smithi115 ceph-mon[35364]: pgmap v1382: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:09 smithi181 ceph-mon[35869]: pgmap v1382: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:09 smithi097 ceph-mon[32231]: pgmap v1382: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1187.627642 secs 2024-07-26T08:19:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1187.627642 secs 2024-07-26T08:19:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1187.627642 secs 2024-07-26T08:19:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:11 smithi115 ceph-mon[35364]: pgmap v1383: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:11 smithi181 ceph-mon[35869]: pgmap v1383: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:11 smithi097 ceph-mon[32231]: pgmap v1383: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:13 smithi115 ceph-mon[35364]: pgmap v1384: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:13 smithi181 ceph-mon[35869]: pgmap v1384: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:13 smithi097 ceph-mon[32231]: pgmap v1384: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:15 smithi115 ceph-mon[35364]: pgmap v1385: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1192.627800 secs 2024-07-26T08:19:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:15 smithi181 ceph-mon[35869]: pgmap v1385: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1192.627800 secs 2024-07-26T08:19:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:15 smithi097 ceph-mon[32231]: pgmap v1385: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1192.627800 secs 2024-07-26T08:19:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:17 smithi115 ceph-mon[35364]: pgmap v1386: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:17 smithi181 ceph-mon[35869]: pgmap v1386: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:17 smithi097 ceph-mon[32231]: pgmap v1386: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:17.865+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:19:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:17.867+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:19:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:17.867+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:19:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:17.867+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:19:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:17.867+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:19:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:19 smithi115 ceph-mon[35364]: pgmap v1387: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:19 smithi181 ceph-mon[35869]: pgmap v1387: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:19 smithi097 ceph-mon[32231]: pgmap v1387: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1197.627954 secs 2024-07-26T08:19:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1197.627954 secs 2024-07-26T08:19:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1197.627954 secs 2024-07-26T08:19:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:21 smithi115 ceph-mon[35364]: pgmap v1388: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:21 smithi181 ceph-mon[35869]: pgmap v1388: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:21 smithi097 ceph-mon[32231]: pgmap v1388: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:23 smithi115 ceph-mon[35364]: pgmap v1389: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:23 smithi181 ceph-mon[35869]: pgmap v1389: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:23 smithi097 ceph-mon[32231]: pgmap v1389: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:25 smithi115 ceph-mon[35364]: pgmap v1390: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1202.628124 secs 2024-07-26T08:19:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:25 smithi181 ceph-mon[35869]: pgmap v1390: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1202.628124 secs 2024-07-26T08:19:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:25 smithi097 ceph-mon[32231]: pgmap v1390: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1202.628124 secs 2024-07-26T08:19:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:27 smithi115 ceph-mon[35364]: pgmap v1391: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:27 smithi181 ceph-mon[35869]: pgmap v1391: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:27 smithi097 ceph-mon[32231]: pgmap v1391: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:29 smithi115 ceph-mon[35364]: pgmap v1392: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:29 smithi181 ceph-mon[35869]: pgmap v1392: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:29 smithi097 ceph-mon[32231]: pgmap v1392: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1207.628308 secs 2024-07-26T08:19:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:30 smithi115 ceph-mon[35364]: pgmap v1393: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1207.628308 secs 2024-07-26T08:19:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:30 smithi181 ceph-mon[35869]: pgmap v1393: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1207.628308 secs 2024-07-26T08:19:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:30 smithi097 ceph-mon[32231]: pgmap v1393: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:32.870+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:19:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:32.872+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:19:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:32.872+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:19:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:32.872+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:19:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:32.872+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:19:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:33 smithi115 ceph-mon[35364]: pgmap v1394: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:33.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:33 smithi181 ceph-mon[35869]: pgmap v1394: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:33 smithi097 ceph-mon[32231]: pgmap v1394: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:35.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:35 smithi097 ceph-mon[32231]: pgmap v1395: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:35.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1212.628449 secs 2024-07-26T08:19:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:35 smithi115 ceph-mon[35364]: pgmap v1395: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1212.628449 secs 2024-07-26T08:19:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:35 smithi181 ceph-mon[35869]: pgmap v1395: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1212.628449 secs 2024-07-26T08:19:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:37 smithi115 ceph-mon[35364]: pgmap v1396: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:37 smithi181 ceph-mon[35869]: pgmap v1396: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:37 smithi097 ceph-mon[32231]: pgmap v1396: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:39 smithi115 ceph-mon[35364]: pgmap v1397: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:39 smithi181 ceph-mon[35869]: pgmap v1397: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:39 smithi097 ceph-mon[32231]: pgmap v1397: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1217.628591 secs 2024-07-26T08:19:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1217.628591 secs 2024-07-26T08:19:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1217.628591 secs 2024-07-26T08:19:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:41 smithi115 ceph-mon[35364]: pgmap v1398: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:41 smithi181 ceph-mon[35869]: pgmap v1398: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:41 smithi097 ceph-mon[32231]: pgmap v1398: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:43 smithi115 ceph-mon[35364]: pgmap v1399: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:43 smithi181 ceph-mon[35869]: pgmap v1399: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:43 smithi097 ceph-mon[32231]: pgmap v1399: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:45 smithi115 ceph-mon[35364]: pgmap v1400: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1222.628759 secs 2024-07-26T08:19:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:45 smithi181 ceph-mon[35869]: pgmap v1400: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1222.628759 secs 2024-07-26T08:19:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:45 smithi097 ceph-mon[32231]: pgmap v1400: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1222.628759 secs 2024-07-26T08:19:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:47 smithi115 ceph-mon[35364]: pgmap v1401: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:47 smithi181 ceph-mon[35869]: pgmap v1401: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:47 smithi097 ceph-mon[32231]: pgmap v1401: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:47.881+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:19:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:47.882+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:19:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:47.882+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:19:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:47.882+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:19:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:19:47.883+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:19:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:49 smithi115 ceph-mon[35364]: pgmap v1402: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:49 smithi181 ceph-mon[35869]: pgmap v1402: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:49 smithi097 ceph-mon[32231]: pgmap v1402: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1227.628916 secs 2024-07-26T08:19:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1227.628916 secs 2024-07-26T08:19:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1227.628916 secs 2024-07-26T08:19:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:51 smithi115 ceph-mon[35364]: pgmap v1403: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:51 smithi181 ceph-mon[35869]: pgmap v1403: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:51 smithi097 ceph-mon[32231]: pgmap v1403: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:19:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:53 smithi115 ceph-mon[35364]: pgmap v1404: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:53 smithi181 ceph-mon[35869]: pgmap v1404: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:53 smithi097 ceph-mon[32231]: pgmap v1404: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:19:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:55 smithi115 ceph-mon[35364]: pgmap v1405: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1232.629059 secs 2024-07-26T08:19:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:55 smithi181 ceph-mon[35869]: pgmap v1405: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1232.629059 secs 2024-07-26T08:19:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:55 smithi097 ceph-mon[32231]: pgmap v1405: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:19:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1232.629059 secs 2024-07-26T08:19:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:57 smithi115 ceph-mon[35364]: pgmap v1406: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:57 smithi181 ceph-mon[35869]: pgmap v1406: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:57 smithi097 ceph-mon[32231]: pgmap v1406: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:19:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:19:59 smithi115 ceph-mon[35364]: pgmap v1407: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:19:59 smithi181 ceph-mon[35869]: pgmap v1407: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:19:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:19:59 smithi097 ceph-mon[32231]: pgmap v1407: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1237.629184 secs 2024-07-26T08:20:00.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:20:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T08:20:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: daemon mds.j on smithi097 is in error state 2024-07-26T08:20:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: daemon mds.d on smithi097 is in error state 2024-07-26T08:20:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: daemon mds.e on smithi115 is in error state 2024-07-26T08:20:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: daemon mds.f on smithi181 is in error state 2024-07-26T08:20:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:20:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:00 smithi115 ceph-mon[35364]: mds.c(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T08:20:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1237.629184 secs 2024-07-26T08:20:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:20:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T08:20:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: daemon mds.j on smithi097 is in error state 2024-07-26T08:20:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: daemon mds.d on smithi097 is in error state 2024-07-26T08:20:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: daemon mds.e on smithi115 is in error state 2024-07-26T08:20:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: daemon mds.f on smithi181 is in error state 2024-07-26T08:20:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:20:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:00 smithi181 ceph-mon[35869]: mds.c(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1237.629184 secs 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: daemon mds.j on smithi097 is in error state 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: daemon mds.d on smithi097 is in error state 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: daemon mds.e on smithi115 is in error state 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: daemon mds.f on smithi181 is in error state 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:00 smithi097 ceph-mon[32231]: mds.c(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T08:20:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:01 smithi115 ceph-mon[35364]: pgmap v1408: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:20:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:01 smithi181 ceph-mon[35869]: pgmap v1408: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:20:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:01 smithi097 ceph-mon[32231]: pgmap v1408: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:01.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:20:01.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.505 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:01.505 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:20:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:20:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:20:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:20:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:20:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:20:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:20:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:02.887+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:20:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:02.888+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:20:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:02.888+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:20:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:02.888+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:20:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:02.888+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:20:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:03 smithi115 ceph-mon[35364]: pgmap v1409: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:03 smithi115 ceph-mon[35364]: pgmap v1410: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:20:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:03 smithi181 ceph-mon[35869]: pgmap v1409: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:03 smithi181 ceph-mon[35869]: pgmap v1410: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:20:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:03 smithi097 ceph-mon[32231]: pgmap v1409: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:03 smithi097 ceph-mon[32231]: pgmap v1410: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:20:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:05 smithi115 ceph-mon[35364]: pgmap v1411: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:20:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1242.629356 secs 2024-07-26T08:20:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:05 smithi181 ceph-mon[35869]: pgmap v1411: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:20:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1242.629356 secs 2024-07-26T08:20:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:05 smithi097 ceph-mon[32231]: pgmap v1411: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:20:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1242.629356 secs 2024-07-26T08:20:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:07 smithi115 ceph-mon[35364]: pgmap v1412: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:20:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:07 smithi181 ceph-mon[35869]: pgmap v1412: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:20:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:07 smithi097 ceph-mon[32231]: pgmap v1412: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:20:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:09 smithi115 ceph-mon[35364]: pgmap v1413: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:20:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:09 smithi181 ceph-mon[35869]: pgmap v1413: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:20:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:09 smithi097 ceph-mon[32231]: pgmap v1413: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:20:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1247.629516 secs 2024-07-26T08:20:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1247.629516 secs 2024-07-26T08:20:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1247.629516 secs 2024-07-26T08:20:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:11 smithi115 ceph-mon[35364]: pgmap v1414: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:11 smithi181 ceph-mon[35869]: pgmap v1414: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:11 smithi097 ceph-mon[32231]: pgmap v1414: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:13 smithi115 ceph-mon[35364]: pgmap v1415: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:13 smithi181 ceph-mon[35869]: pgmap v1415: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:13 smithi097 ceph-mon[32231]: pgmap v1415: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:15 smithi115 ceph-mon[35364]: pgmap v1416: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:20:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1252.629652 secs 2024-07-26T08:20:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:15 smithi181 ceph-mon[35869]: pgmap v1416: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:20:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1252.629652 secs 2024-07-26T08:20:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:15 smithi097 ceph-mon[32231]: pgmap v1416: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:20:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1252.629652 secs 2024-07-26T08:20:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:17 smithi115 ceph-mon[35364]: pgmap v1417: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:20:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:17 smithi181 ceph-mon[35869]: pgmap v1417: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:20:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:17 smithi097 ceph-mon[32231]: pgmap v1417: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:20:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:17.890+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:20:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:17.891+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:20:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:17.891+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:20:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:17.891+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:20:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:17.891+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:20:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:19 smithi115 ceph-mon[35364]: pgmap v1418: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:20:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:19 smithi181 ceph-mon[35869]: pgmap v1418: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:20:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:19 smithi097 ceph-mon[32231]: pgmap v1418: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:20:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1257.629795 secs 2024-07-26T08:20:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1257.629795 secs 2024-07-26T08:20:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1257.629795 secs 2024-07-26T08:20:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:21 smithi181 ceph-mon[35869]: pgmap v1419: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:20:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:21 smithi097 ceph-mon[32231]: pgmap v1419: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:20:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:21 smithi115 ceph-mon[35364]: pgmap v1419: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:20:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:23 smithi097 ceph-mon[32231]: pgmap v1420: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:23 smithi115 ceph-mon[35364]: pgmap v1420: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:23 smithi181 ceph-mon[35869]: pgmap v1420: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:25 smithi097 ceph-mon[32231]: pgmap v1421: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1262.629949 secs 2024-07-26T08:20:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:25 smithi115 ceph-mon[35364]: pgmap v1421: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1262.629949 secs 2024-07-26T08:20:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:25 smithi181 ceph-mon[35869]: pgmap v1421: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1262.629949 secs 2024-07-26T08:20:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:27 smithi097 ceph-mon[32231]: pgmap v1422: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:27 smithi115 ceph-mon[35364]: pgmap v1422: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:27 smithi181 ceph-mon[35869]: pgmap v1422: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:29 smithi115 ceph-mon[35364]: pgmap v1423: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:29 smithi181 ceph-mon[35869]: pgmap v1423: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:29 smithi097 ceph-mon[32231]: pgmap v1423: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1267.630079 secs 2024-07-26T08:20:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1267.630079 secs 2024-07-26T08:20:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1267.630079 secs 2024-07-26T08:20:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:31 smithi115 ceph-mon[35364]: pgmap v1424: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:31 smithi181 ceph-mon[35869]: pgmap v1424: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:31 smithi097 ceph-mon[32231]: pgmap v1424: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:32.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:32.894+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:20:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:32.895+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:20:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:32.895+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:20:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:32.895+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:20:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:32.895+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:20:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:33 smithi115 ceph-mon[35364]: pgmap v1425: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:33 smithi181 ceph-mon[35869]: pgmap v1425: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:33 smithi097 ceph-mon[32231]: pgmap v1425: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:35.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:35 smithi097 ceph-mon[32231]: pgmap v1426: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:35.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1272.630224 secs 2024-07-26T08:20:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:35 smithi115 ceph-mon[35364]: pgmap v1426: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1272.630224 secs 2024-07-26T08:20:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:35 smithi181 ceph-mon[35869]: pgmap v1426: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1272.630224 secs 2024-07-26T08:20:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:37 smithi115 ceph-mon[35364]: pgmap v1427: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:37 smithi181 ceph-mon[35869]: pgmap v1427: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:37 smithi097 ceph-mon[32231]: pgmap v1427: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:39 smithi115 ceph-mon[35364]: pgmap v1428: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:39 smithi181 ceph-mon[35869]: pgmap v1428: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:39 smithi097 ceph-mon[32231]: pgmap v1428: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1277.630372 secs 2024-07-26T08:20:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1277.630372 secs 2024-07-26T08:20:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1277.630372 secs 2024-07-26T08:20:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:41 smithi115 ceph-mon[35364]: pgmap v1429: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:41 smithi181 ceph-mon[35869]: pgmap v1429: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:41 smithi097 ceph-mon[32231]: pgmap v1429: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:43 smithi115 ceph-mon[35364]: pgmap v1430: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:43 smithi181 ceph-mon[35869]: pgmap v1430: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:43 smithi097 ceph-mon[32231]: pgmap v1430: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:45 smithi115 ceph-mon[35364]: pgmap v1431: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1282.630514 secs 2024-07-26T08:20:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:45 smithi181 ceph-mon[35869]: pgmap v1431: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1282.630514 secs 2024-07-26T08:20:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:45 smithi097 ceph-mon[32231]: pgmap v1431: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1282.630514 secs 2024-07-26T08:20:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:47 smithi115 ceph-mon[35364]: pgmap v1432: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:47 smithi181 ceph-mon[35869]: pgmap v1432: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:47 smithi097 ceph-mon[32231]: pgmap v1432: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:47.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:47.904+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:20:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:47.905+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:20:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:47.905+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:20:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:47.905+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:20:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:20:47.905+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:20:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:49 smithi115 ceph-mon[35364]: pgmap v1433: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:49 smithi181 ceph-mon[35869]: pgmap v1433: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:49 smithi097 ceph-mon[32231]: pgmap v1433: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1287.630651 secs 2024-07-26T08:20:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1287.630651 secs 2024-07-26T08:20:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1287.630651 secs 2024-07-26T08:20:51.677 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:51 smithi097 ceph-mon[32231]: pgmap v1434: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:51 smithi115 ceph-mon[35364]: pgmap v1434: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:51 smithi181 ceph-mon[35869]: pgmap v1434: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:20:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:53 smithi115 ceph-mon[35364]: pgmap v1435: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:53 smithi181 ceph-mon[35869]: pgmap v1435: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:53 smithi097 ceph-mon[32231]: pgmap v1435: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:20:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:55 smithi115 ceph-mon[35364]: pgmap v1436: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1292.630744 secs 2024-07-26T08:20:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:55 smithi181 ceph-mon[35869]: pgmap v1436: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1292.630744 secs 2024-07-26T08:20:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:55 smithi097 ceph-mon[32231]: pgmap v1436: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:20:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1292.630744 secs 2024-07-26T08:20:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:57 smithi115 ceph-mon[35364]: pgmap v1437: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:57 smithi181 ceph-mon[35869]: pgmap v1437: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:57 smithi097 ceph-mon[32231]: pgmap v1437: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:20:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:20:59 smithi115 ceph-mon[35364]: pgmap v1438: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:20:59 smithi181 ceph-mon[35869]: pgmap v1438: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:20:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:20:59 smithi097 ceph-mon[32231]: pgmap v1438: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1297.630886 secs 2024-07-26T08:21:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1297.630886 secs 2024-07-26T08:21:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1297.630886 secs 2024-07-26T08:21:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:01 smithi115 ceph-mon[35364]: pgmap v1439: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:01 smithi181 ceph-mon[35869]: pgmap v1439: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:01 smithi097 ceph-mon[32231]: pgmap v1439: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:21:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:21:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:21:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:02.908+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:21:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:02.909+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:21:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:02.909+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:21:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:02.909+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:21:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:02.909+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:21:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:03 smithi115 ceph-mon[35364]: pgmap v1440: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:03 smithi181 ceph-mon[35869]: pgmap v1440: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:03 smithi097 ceph-mon[32231]: pgmap v1440: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:05 smithi115 ceph-mon[35364]: pgmap v1441: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1302.630973 secs 2024-07-26T08:21:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:05 smithi181 ceph-mon[35869]: pgmap v1441: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1302.630973 secs 2024-07-26T08:21:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:05 smithi097 ceph-mon[32231]: pgmap v1441: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1302.630973 secs 2024-07-26T08:21:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:06 smithi115 ceph-mon[35364]: pgmap v1442: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:21:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:21:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:21:06.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:06.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:06.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:06 smithi181 ceph-mon[35869]: pgmap v1442: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:21:06.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:21:06.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:21:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:06 smithi097 ceph-mon[32231]: pgmap v1442: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:21:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:21:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:21:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:07 smithi115 ceph-mon[35364]: pgmap v1443: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:21:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:07 smithi115 ceph-mon[35364]: pgmap v1444: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:21:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:07 smithi181 ceph-mon[35869]: pgmap v1443: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:21:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:07 smithi181 ceph-mon[35869]: pgmap v1444: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:21:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:07 smithi097 ceph-mon[32231]: pgmap v1443: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:21:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:07 smithi097 ceph-mon[32231]: pgmap v1444: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:21:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:21:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:09 smithi115 ceph-mon[35364]: pgmap v1445: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:21:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:09 smithi181 ceph-mon[35869]: pgmap v1445: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:21:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:09 smithi097 ceph-mon[32231]: pgmap v1445: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:21:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1307.631069 secs 2024-07-26T08:21:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1307.631069 secs 2024-07-26T08:21:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1307.631069 secs 2024-07-26T08:21:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:11 smithi115 ceph-mon[35364]: pgmap v1446: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:11 smithi181 ceph-mon[35869]: pgmap v1446: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:11 smithi097 ceph-mon[32231]: pgmap v1446: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:13 smithi115 ceph-mon[35364]: pgmap v1447: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:21:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:13 smithi181 ceph-mon[35869]: pgmap v1447: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:21:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:13 smithi097 ceph-mon[32231]: pgmap v1447: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:21:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:15 smithi115 ceph-mon[35364]: pgmap v1448: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1312.631202 secs 2024-07-26T08:21:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:15 smithi181 ceph-mon[35869]: pgmap v1448: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1312.631202 secs 2024-07-26T08:21:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:15 smithi097 ceph-mon[32231]: pgmap v1448: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:21:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1312.631202 secs 2024-07-26T08:21:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:17.918+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:21:17.920 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:17.919+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:21:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:17.919+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:21:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:17.919+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:21:17.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:17.919+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:21:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:17 smithi115 ceph-mon[35364]: pgmap v1449: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:21:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:17 smithi181 ceph-mon[35869]: pgmap v1449: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:21:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:17 smithi097 ceph-mon[32231]: pgmap v1449: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:21:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:19 smithi115 ceph-mon[35364]: pgmap v1450: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:19 smithi181 ceph-mon[35869]: pgmap v1450: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:19 smithi097 ceph-mon[32231]: pgmap v1450: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1317.631336 secs 2024-07-26T08:21:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1317.631336 secs 2024-07-26T08:21:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1317.631336 secs 2024-07-26T08:21:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:21 smithi115 ceph-mon[35364]: pgmap v1451: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:21 smithi181 ceph-mon[35869]: pgmap v1451: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:21 smithi097 ceph-mon[32231]: pgmap v1451: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:23 smithi115 ceph-mon[35364]: pgmap v1452: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:23 smithi181 ceph-mon[35869]: pgmap v1452: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:23 smithi097 ceph-mon[32231]: pgmap v1452: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:25 smithi115 ceph-mon[35364]: pgmap v1453: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1322.631476 secs 2024-07-26T08:21:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:25 smithi181 ceph-mon[35869]: pgmap v1453: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1322.631476 secs 2024-07-26T08:21:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:25 smithi097 ceph-mon[32231]: pgmap v1453: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1322.631476 secs 2024-07-26T08:21:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:27 smithi115 ceph-mon[35364]: pgmap v1454: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:27 smithi181 ceph-mon[35869]: pgmap v1454: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:27 smithi097 ceph-mon[32231]: pgmap v1454: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:29 smithi115 ceph-mon[35364]: pgmap v1455: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:29 smithi181 ceph-mon[35869]: pgmap v1455: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:29 smithi097 ceph-mon[32231]: pgmap v1455: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1327.631617 secs 2024-07-26T08:21:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1327.631617 secs 2024-07-26T08:21:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1327.631617 secs 2024-07-26T08:21:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:31 smithi115 ceph-mon[35364]: pgmap v1456: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:31 smithi181 ceph-mon[35869]: pgmap v1456: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:31 smithi097 ceph-mon[32231]: pgmap v1456: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:32.923+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:21:32.926 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:32.924+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:21:32.926 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:32.924+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:21:32.926 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:32.924+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:21:32.926 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:32.924+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:21:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:33 smithi115 ceph-mon[35364]: pgmap v1457: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:33 smithi181 ceph-mon[35869]: pgmap v1457: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:33 smithi097 ceph-mon[32231]: pgmap v1457: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:35 smithi115 ceph-mon[35364]: pgmap v1458: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1332.631757 secs 2024-07-26T08:21:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:35 smithi181 ceph-mon[35869]: pgmap v1458: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1332.631757 secs 2024-07-26T08:21:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:35 smithi097 ceph-mon[32231]: pgmap v1458: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1332.631757 secs 2024-07-26T08:21:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:37 smithi181 ceph-mon[35869]: pgmap v1459: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:37 smithi097 ceph-mon[32231]: pgmap v1459: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:37 smithi115 ceph-mon[35364]: pgmap v1459: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:39 smithi097 ceph-mon[32231]: pgmap v1460: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:39 smithi115 ceph-mon[35364]: pgmap v1460: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:39 smithi181 ceph-mon[35869]: pgmap v1460: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1337.631894 secs 2024-07-26T08:21:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1337.631894 secs 2024-07-26T08:21:41.282 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1337.631894 secs 2024-07-26T08:21:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:41 smithi115 ceph-mon[35364]: pgmap v1461: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:41 smithi181 ceph-mon[35869]: pgmap v1461: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:41 smithi097 ceph-mon[32231]: pgmap v1461: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:44.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:43 smithi115 ceph-mon[35364]: pgmap v1462: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:44.035 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:43 smithi181 ceph-mon[35869]: pgmap v1462: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:43 smithi097 ceph-mon[32231]: pgmap v1462: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:45 smithi115 ceph-mon[35364]: pgmap v1463: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1342.632050 secs 2024-07-26T08:21:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:45 smithi181 ceph-mon[35869]: pgmap v1463: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1342.632050 secs 2024-07-26T08:21:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:45 smithi097 ceph-mon[32231]: pgmap v1463: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1342.632050 secs 2024-07-26T08:21:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:47.930+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:21:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:47.931+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:21:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:47.931+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:21:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:47.931+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:21:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:21:47.931+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:21:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:47 smithi115 ceph-mon[35364]: pgmap v1464: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:47 smithi181 ceph-mon[35869]: pgmap v1464: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:47 smithi097 ceph-mon[32231]: pgmap v1464: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:49 smithi115 ceph-mon[35364]: pgmap v1465: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:49 smithi181 ceph-mon[35869]: pgmap v1465: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:49 smithi097 ceph-mon[32231]: pgmap v1465: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1347.632188 secs 2024-07-26T08:21:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1347.632188 secs 2024-07-26T08:21:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1347.632188 secs 2024-07-26T08:21:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:51 smithi115 ceph-mon[35364]: pgmap v1466: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:51 smithi181 ceph-mon[35869]: pgmap v1466: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:51 smithi097 ceph-mon[32231]: pgmap v1466: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:53 smithi115 ceph-mon[35364]: pgmap v1467: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:53 smithi181 ceph-mon[35869]: pgmap v1467: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:53 smithi097 ceph-mon[32231]: pgmap v1467: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:21:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:55 smithi115 ceph-mon[35364]: pgmap v1468: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1352.632351 secs 2024-07-26T08:21:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:55 smithi181 ceph-mon[35869]: pgmap v1468: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1352.632351 secs 2024-07-26T08:21:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:55 smithi097 ceph-mon[32231]: pgmap v1468: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:21:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1352.632351 secs 2024-07-26T08:21:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:57 smithi115 ceph-mon[35364]: pgmap v1469: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:57 smithi181 ceph-mon[35869]: pgmap v1469: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:21:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:57 smithi097 ceph-mon[32231]: pgmap v1469: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:21:59 smithi115 ceph-mon[35364]: pgmap v1470: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:21:59 smithi181 ceph-mon[35869]: pgmap v1470: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:21:59 smithi097 ceph-mon[32231]: pgmap v1470: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1357.632489 secs 2024-07-26T08:22:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1357.632489 secs 2024-07-26T08:22:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1357.632489 secs 2024-07-26T08:22:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:01 smithi115 ceph-mon[35364]: pgmap v1471: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:22:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:01 smithi181 ceph-mon[35869]: pgmap v1471: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:22:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:01 smithi097 ceph-mon[32231]: pgmap v1471: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:22:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:02.934+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:22:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:02.936+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:22:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:02.936+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:22:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:02.936+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:22:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:02.936+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:22:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:03 smithi115 ceph-mon[35364]: pgmap v1472: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:22:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:03 smithi181 ceph-mon[35869]: pgmap v1472: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:22:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:03 smithi097 ceph-mon[32231]: pgmap v1472: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:22:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:05 smithi115 ceph-mon[35364]: pgmap v1473: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:22:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1362.632629 secs 2024-07-26T08:22:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:05 smithi181 ceph-mon[35869]: pgmap v1473: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:22:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1362.632629 secs 2024-07-26T08:22:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:05 smithi097 ceph-mon[32231]: pgmap v1473: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:22:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1362.632629 secs 2024-07-26T08:22:07.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:07.112+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:22:07.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:07.112+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:22:07.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:07.113+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:22:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:07.113+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:22:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:07 smithi115 ceph-mon[35364]: pgmap v1474: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:22:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:22:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:07 smithi181 ceph-mon[35869]: pgmap v1474: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:22:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:22:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:07 smithi097 ceph-mon[32231]: pgmap v1474: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:22:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:22:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:22:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:22:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:08 smithi115 ceph-mon[35364]: pgmap v1475: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:22:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:08 smithi115 ceph-mon[35364]: pgmap v1476: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:22:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:22:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:22:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:08 smithi181 ceph-mon[35869]: pgmap v1475: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:22:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:08 smithi181 ceph-mon[35869]: pgmap v1476: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:22:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:22:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:22:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:08 smithi097 ceph-mon[32231]: pgmap v1475: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:22:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:08 smithi097 ceph-mon[32231]: pgmap v1476: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:22:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:09 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1367.632773 secs 2024-07-26T08:22:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:09 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1367.632773 secs 2024-07-26T08:22:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:09 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1367.632773 secs 2024-07-26T08:22:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:10 smithi115 ceph-mon[35364]: pgmap v1477: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:22:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:10 smithi181 ceph-mon[35869]: pgmap v1477: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:22:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:10 smithi097 ceph-mon[32231]: pgmap v1477: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:22:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:13 smithi115 ceph-mon[35364]: pgmap v1478: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:22:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:13 smithi181 ceph-mon[35869]: pgmap v1478: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:22:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:13 smithi097 ceph-mon[32231]: pgmap v1478: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:22:15.313 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:15 smithi181 ceph-mon[35869]: pgmap v1479: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:15.314 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1372.632858 secs 2024-07-26T08:22:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:15 smithi115 ceph-mon[35364]: pgmap v1479: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1372.632858 secs 2024-07-26T08:22:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:15 smithi097 ceph-mon[32231]: pgmap v1479: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1372.632858 secs 2024-07-26T08:22:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:17 smithi181 ceph-mon[35869]: pgmap v1480: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:22:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:17 smithi115 ceph-mon[35364]: pgmap v1480: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:22:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:17 smithi097 ceph-mon[32231]: pgmap v1480: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:22:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:17.938+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:22:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:17.939+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:22:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:17.939+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:22:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:17.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:22:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:17.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:22:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:19 smithi115 ceph-mon[35364]: pgmap v1481: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:22:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:19 smithi181 ceph-mon[35869]: pgmap v1481: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:22:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:19 smithi097 ceph-mon[32231]: pgmap v1481: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:22:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1377.633000 secs 2024-07-26T08:22:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1377.633000 secs 2024-07-26T08:22:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1377.633000 secs 2024-07-26T08:22:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:21 smithi115 ceph-mon[35364]: pgmap v1482: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:21 smithi181 ceph-mon[35869]: pgmap v1482: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:21 smithi097 ceph-mon[32231]: pgmap v1482: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:23 smithi115 ceph-mon[35364]: pgmap v1483: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:23 smithi181 ceph-mon[35869]: pgmap v1483: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:23 smithi097 ceph-mon[32231]: pgmap v1483: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:25 smithi115 ceph-mon[35364]: pgmap v1484: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1382.633092 secs 2024-07-26T08:22:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:25 smithi181 ceph-mon[35869]: pgmap v1484: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1382.633092 secs 2024-07-26T08:22:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:25 smithi097 ceph-mon[32231]: pgmap v1484: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1382.633092 secs 2024-07-26T08:22:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:27 smithi181 ceph-mon[35869]: pgmap v1485: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:27 smithi115 ceph-mon[35364]: pgmap v1485: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:27 smithi097 ceph-mon[32231]: pgmap v1485: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:29 smithi115 ceph-mon[35364]: pgmap v1486: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:29 smithi181 ceph-mon[35869]: pgmap v1486: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:29 smithi097 ceph-mon[32231]: pgmap v1486: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1387.633226 secs 2024-07-26T08:22:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1387.633226 secs 2024-07-26T08:22:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1387.633226 secs 2024-07-26T08:22:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:31 smithi115 ceph-mon[35364]: pgmap v1487: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:31 smithi181 ceph-mon[35869]: pgmap v1487: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:31 smithi097 ceph-mon[32231]: pgmap v1487: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:32.943+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:22:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:32.944+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:22:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:32.944+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:22:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:32.944+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:22:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:32.944+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:22:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:33 smithi115 ceph-mon[35364]: pgmap v1488: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:33 smithi181 ceph-mon[35869]: pgmap v1488: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:33 smithi097 ceph-mon[32231]: pgmap v1488: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:35 smithi115 ceph-mon[35364]: pgmap v1489: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1392.633339 secs 2024-07-26T08:22:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:35 smithi181 ceph-mon[35869]: pgmap v1489: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1392.633339 secs 2024-07-26T08:22:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:35 smithi097 ceph-mon[32231]: pgmap v1489: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1392.633339 secs 2024-07-26T08:22:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:37 smithi115 ceph-mon[35364]: pgmap v1490: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:37 smithi181 ceph-mon[35869]: pgmap v1490: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:37 smithi097 ceph-mon[32231]: pgmap v1490: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:39 smithi115 ceph-mon[35364]: pgmap v1491: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:39 smithi181 ceph-mon[35869]: pgmap v1491: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:39.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:39 smithi097 ceph-mon[32231]: pgmap v1491: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1397.633430 secs 2024-07-26T08:22:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1397.633430 secs 2024-07-26T08:22:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1397.633430 secs 2024-07-26T08:22:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:41 smithi115 ceph-mon[35364]: pgmap v1492: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:41 smithi181 ceph-mon[35869]: pgmap v1492: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:41 smithi097 ceph-mon[32231]: pgmap v1492: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:43 smithi115 ceph-mon[35364]: pgmap v1493: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:43 smithi181 ceph-mon[35869]: pgmap v1493: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:43 smithi097 ceph-mon[32231]: pgmap v1493: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:45 smithi115 ceph-mon[35364]: pgmap v1494: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1402.633529 secs 2024-07-26T08:22:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:45 smithi181 ceph-mon[35869]: pgmap v1494: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1402.633529 secs 2024-07-26T08:22:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:45 smithi097 ceph-mon[32231]: pgmap v1494: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1402.633529 secs 2024-07-26T08:22:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:47 smithi115 ceph-mon[35364]: pgmap v1495: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:47 smithi181 ceph-mon[35869]: pgmap v1495: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:47 smithi097 ceph-mon[32231]: pgmap v1495: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:47.953+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:22:47.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:47.954+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:22:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:47.954+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:22:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:47.954+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:22:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:47.954+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:22:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:49 smithi115 ceph-mon[35364]: pgmap v1496: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:49 smithi181 ceph-mon[35869]: pgmap v1496: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:49 smithi097 ceph-mon[32231]: pgmap v1496: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1407.633692 secs 2024-07-26T08:22:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1407.633692 secs 2024-07-26T08:22:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1407.633692 secs 2024-07-26T08:22:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:51 smithi115 ceph-mon[35364]: pgmap v1497: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:51 smithi181 ceph-mon[35869]: pgmap v1497: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:51 smithi097 ceph-mon[32231]: pgmap v1497: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:22:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:52.120+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:22:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:52.120+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:22:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:52.121+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:22:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:22:52.121+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:22:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:53 smithi181 ceph-mon[35869]: pgmap v1498: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:53 smithi097 ceph-mon[32231]: pgmap v1498: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:53 smithi115 ceph-mon[35364]: pgmap v1498: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:22:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:55 smithi097 ceph-mon[32231]: pgmap v1499: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1412.633826 secs 2024-07-26T08:22:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:55 smithi115 ceph-mon[35364]: pgmap v1499: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1412.633826 secs 2024-07-26T08:22:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:55 smithi181 ceph-mon[35869]: pgmap v1499: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:22:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1412.633826 secs 2024-07-26T08:22:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:57 smithi097 ceph-mon[32231]: pgmap v1500: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:57 smithi115 ceph-mon[35364]: pgmap v1500: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:57 smithi181 ceph-mon[35869]: pgmap v1500: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:22:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:22:59 smithi097 ceph-mon[32231]: pgmap v1501: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:22:59 smithi115 ceph-mon[35364]: pgmap v1501: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:22:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:22:59 smithi181 ceph-mon[35869]: pgmap v1501: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1417.633981 secs 2024-07-26T08:23:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1417.633981 secs 2024-07-26T08:23:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1417.633981 secs 2024-07-26T08:23:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:01 smithi115 ceph-mon[35364]: pgmap v1502: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:01 smithi181 ceph-mon[35869]: pgmap v1502: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:01 smithi097 ceph-mon[32231]: pgmap v1502: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:02.958+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:23:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:02.961+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:23:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:02.961+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:23:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:02.961+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:23:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:02.961+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:23:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:03 smithi115 ceph-mon[35364]: pgmap v1503: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:03 smithi181 ceph-mon[35869]: pgmap v1503: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:03.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:03 smithi097 ceph-mon[32231]: pgmap v1503: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:05 smithi115 ceph-mon[35364]: pgmap v1504: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:23:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1422.634128 secs 2024-07-26T08:23:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:05 smithi181 ceph-mon[35869]: pgmap v1504: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:23:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1422.634128 secs 2024-07-26T08:23:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:05 smithi097 ceph-mon[32231]: pgmap v1504: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:23:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1422.634128 secs 2024-07-26T08:23:07.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:07 smithi097 ceph-mon[32231]: pgmap v1505: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:23:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:07 smithi115 ceph-mon[35364]: pgmap v1505: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:23:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:07 smithi181 ceph-mon[35869]: pgmap v1505: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:23:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:09 smithi115 ceph-mon[35364]: pgmap v1506: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:09 smithi181 ceph-mon[35869]: pgmap v1506: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:09 smithi097 ceph-mon[32231]: pgmap v1506: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:23:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:23:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:10 smithi115 ceph-mon[35364]: pgmap v1507: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:23:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1427.634301 secs 2024-07-26T08:23:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:23:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:23:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:10 smithi181 ceph-mon[35869]: pgmap v1507: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:23:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1427.634301 secs 2024-07-26T08:23:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:23:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:23:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:10 smithi097 ceph-mon[32231]: pgmap v1507: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:23:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1427.634301 secs 2024-07-26T08:23:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:12 smithi115 ceph-mon[35364]: pgmap v1508: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:23:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:12 smithi181 ceph-mon[35869]: pgmap v1508: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:23:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:12 smithi097 ceph-mon[32231]: pgmap v1508: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:23:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:14 smithi115 ceph-mon[35364]: pgmap v1509: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:14 smithi181 ceph-mon[35869]: pgmap v1509: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:14 smithi097 ceph-mon[32231]: pgmap v1509: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:15.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1432.634450 secs 2024-07-26T08:23:15.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1432.634450 secs 2024-07-26T08:23:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1432.634450 secs 2024-07-26T08:23:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:16 smithi115 ceph-mon[35364]: pgmap v1510: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:23:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:16 smithi181 ceph-mon[35869]: pgmap v1510: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:23:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:16 smithi097 ceph-mon[32231]: pgmap v1510: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:23:17.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:17.969+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:23:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:17.970+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:23:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:17.970+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:23:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:17.970+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:23:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:17.971+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:23:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:18 smithi115 ceph-mon[35364]: pgmap v1511: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:23:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:18 smithi181 ceph-mon[35869]: pgmap v1511: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:23:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:18 smithi097 ceph-mon[32231]: pgmap v1511: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:23:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:19 smithi115 ceph-mon[35364]: pgmap v1512: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:19 smithi181 ceph-mon[35869]: pgmap v1512: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:19 smithi097 ceph-mon[32231]: pgmap v1512: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:23:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1437.634597 secs 2024-07-26T08:23:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1437.634597 secs 2024-07-26T08:23:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1437.634597 secs 2024-07-26T08:23:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:21 smithi115 ceph-mon[35364]: pgmap v1513: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:21 smithi181 ceph-mon[35869]: pgmap v1513: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:21 smithi097 ceph-mon[32231]: pgmap v1513: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:24 smithi115 ceph-mon[35364]: pgmap v1514: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:24 smithi181 ceph-mon[35869]: pgmap v1514: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:24 smithi097 ceph-mon[32231]: pgmap v1514: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:25 smithi115 ceph-mon[35364]: 1 slow requests, 1 included below; oldest blocked for > 1442.634736 secs 2024-07-26T08:23:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:25 smithi115 ceph-mon[35364]: slow request 1442.634735 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:23:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:25 smithi181 ceph-mon[35869]: 1 slow requests, 1 included below; oldest blocked for > 1442.634736 secs 2024-07-26T08:23:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:25 smithi181 ceph-mon[35869]: slow request 1442.634735 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:23:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:25 smithi097 ceph-mon[32231]: 1 slow requests, 1 included below; oldest blocked for > 1442.634736 secs 2024-07-26T08:23:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:25 smithi097 ceph-mon[32231]: slow request 1442.634735 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:23:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:26 smithi115 ceph-mon[35364]: pgmap v1515: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:26 smithi181 ceph-mon[35869]: pgmap v1515: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:26 smithi097 ceph-mon[32231]: pgmap v1515: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:28 smithi115 ceph-mon[35364]: pgmap v1516: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:28 smithi181 ceph-mon[35869]: pgmap v1516: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:28 smithi097 ceph-mon[32231]: pgmap v1516: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:30 smithi115 ceph-mon[35364]: pgmap v1517: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1447.634876 secs 2024-07-26T08:23:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:30 smithi181 ceph-mon[35869]: pgmap v1517: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1447.634876 secs 2024-07-26T08:23:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:30 smithi097 ceph-mon[32231]: pgmap v1517: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1447.634876 secs 2024-07-26T08:23:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:31 smithi115 ceph-mon[35364]: pgmap v1518: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:31 smithi181 ceph-mon[35869]: pgmap v1518: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:31 smithi097 ceph-mon[32231]: pgmap v1518: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:32.975+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:23:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:32.976+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:23:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:32.976+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:23:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:32.976+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:23:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:32.976+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:23:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:34 smithi115 ceph-mon[35364]: pgmap v1519: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:34 smithi181 ceph-mon[35869]: pgmap v1519: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:34 smithi097 ceph-mon[32231]: pgmap v1519: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1452.635025 secs 2024-07-26T08:23:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1452.635025 secs 2024-07-26T08:23:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1452.635025 secs 2024-07-26T08:23:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:36 smithi115 ceph-mon[35364]: pgmap v1520: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:36 smithi181 ceph-mon[35869]: pgmap v1520: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:36 smithi097 ceph-mon[32231]: pgmap v1520: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:38.804 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:38 smithi097 ceph-mon[32231]: pgmap v1521: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:38 smithi115 ceph-mon[35364]: pgmap v1521: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:38 smithi181 ceph-mon[35869]: pgmap v1521: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:39 smithi115 ceph-mon[35364]: pgmap v1522: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:39 smithi181 ceph-mon[35869]: pgmap v1522: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:39 smithi097 ceph-mon[32231]: pgmap v1522: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1457.635187 secs 2024-07-26T08:23:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1457.635187 secs 2024-07-26T08:23:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1457.635187 secs 2024-07-26T08:23:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:41 smithi115 ceph-mon[35364]: pgmap v1523: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:41 smithi181 ceph-mon[35869]: pgmap v1523: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:41 smithi097 ceph-mon[32231]: pgmap v1523: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:44 smithi115 ceph-mon[35364]: pgmap v1524: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:44 smithi181 ceph-mon[35869]: pgmap v1524: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:44 smithi097 ceph-mon[32231]: pgmap v1524: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:45.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1462.635338 secs 2024-07-26T08:23:45.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1462.635338 secs 2024-07-26T08:23:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1462.635338 secs 2024-07-26T08:23:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:46 smithi115 ceph-mon[35364]: pgmap v1525: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:46 smithi181 ceph-mon[35869]: pgmap v1525: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:46 smithi097 ceph-mon[32231]: pgmap v1525: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:47.985+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:23:47.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:47.986+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:23:47.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:47.986+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:23:47.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:47.986+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:23:47.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:23:47.986+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:23:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:48 smithi115 ceph-mon[35364]: pgmap v1526: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:48 smithi181 ceph-mon[35869]: pgmap v1526: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:48 smithi097 ceph-mon[32231]: pgmap v1526: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:49 smithi115 ceph-mon[35364]: pgmap v1527: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:49 smithi181 ceph-mon[35869]: pgmap v1527: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:49 smithi097 ceph-mon[32231]: pgmap v1527: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1467.635480 secs 2024-07-26T08:23:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1467.635480 secs 2024-07-26T08:23:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1467.635480 secs 2024-07-26T08:23:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:51 smithi115 ceph-mon[35364]: pgmap v1528: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:51 smithi181 ceph-mon[35869]: pgmap v1528: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:51 smithi097 ceph-mon[32231]: pgmap v1528: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:23:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:54 smithi115 ceph-mon[35364]: pgmap v1529: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:54 smithi181 ceph-mon[35869]: pgmap v1529: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:54 smithi097 ceph-mon[32231]: pgmap v1529: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:23:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1472.635620 secs 2024-07-26T08:23:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1472.635620 secs 2024-07-26T08:23:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1472.635620 secs 2024-07-26T08:23:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:56 smithi115 ceph-mon[35364]: pgmap v1530: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:56 smithi181 ceph-mon[35869]: pgmap v1530: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:56 smithi097 ceph-mon[32231]: pgmap v1530: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:23:57 smithi115 ceph-mon[35364]: pgmap v1531: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:23:57 smithi181 ceph-mon[35869]: pgmap v1531: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:23:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:23:57 smithi097 ceph-mon[32231]: pgmap v1531: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:00 smithi115 ceph-mon[35364]: pgmap v1532: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1477.635778 secs 2024-07-26T08:24:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:00 smithi181 ceph-mon[35869]: pgmap v1532: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1477.635778 secs 2024-07-26T08:24:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:00 smithi097 ceph-mon[32231]: pgmap v1532: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1477.635778 secs 2024-07-26T08:24:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:02 smithi115 ceph-mon[35364]: pgmap v1533: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:02 smithi181 ceph-mon[35869]: pgmap v1533: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:02.989+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:24:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:02.990+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:24:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:02.990+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:24:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:02.990+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:24:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:02.990+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:24:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:02 smithi097 ceph-mon[32231]: pgmap v1533: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:04 smithi115 ceph-mon[35364]: pgmap v1534: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:04 smithi181 ceph-mon[35869]: pgmap v1534: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:04 smithi097 ceph-mon[32231]: pgmap v1534: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1482.635914 secs 2024-07-26T08:24:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1482.635914 secs 2024-07-26T08:24:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1482.635914 secs 2024-07-26T08:24:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:06 smithi115 ceph-mon[35364]: pgmap v1535: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:06 smithi181 ceph-mon[35869]: pgmap v1535: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:06 smithi097 ceph-mon[32231]: pgmap v1535: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:07 smithi115 ceph-mon[35364]: pgmap v1536: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:07 smithi181 ceph-mon[35869]: pgmap v1536: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:07 smithi097 ceph-mon[32231]: pgmap v1536: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:10 smithi115 ceph-mon[35364]: pgmap v1537: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1487.636054 secs 2024-07-26T08:24:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:10 smithi181 ceph-mon[35869]: pgmap v1537: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1487.636054 secs 2024-07-26T08:24:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:10 smithi097 ceph-mon[32231]: pgmap v1537: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1487.636054 secs 2024-07-26T08:24:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:24:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:24:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:24:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:12 smithi115 ceph-mon[35364]: pgmap v1538: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:12 smithi181 ceph-mon[35869]: pgmap v1538: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:12 smithi097 ceph-mon[32231]: pgmap v1538: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:13 smithi115 ceph-mon[35364]: pgmap v1539: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:24:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:13 smithi181 ceph-mon[35869]: pgmap v1539: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:24:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:13 smithi097 ceph-mon[32231]: pgmap v1539: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:24:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1492.636210 secs 2024-07-26T08:24:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:15 smithi115 ceph-mon[35364]: pgmap v1540: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:24:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1492.636210 secs 2024-07-26T08:24:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:15 smithi181 ceph-mon[35869]: pgmap v1540: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:24:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1492.636210 secs 2024-07-26T08:24:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:15 smithi097 ceph-mon[32231]: pgmap v1540: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:24:17.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:17.993+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:24:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:17.994+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:24:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:17.994+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:24:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:17.994+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:24:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:17.994+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:24:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:17 smithi115 ceph-mon[35364]: pgmap v1541: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:17 smithi181 ceph-mon[35869]: pgmap v1541: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:17 smithi097 ceph-mon[32231]: pgmap v1541: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:19 smithi115 ceph-mon[35364]: pgmap v1542: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:24:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:19 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1497.636350 secs 2024-07-26T08:24:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:19 smithi181 ceph-mon[35869]: pgmap v1542: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:24:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:19 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1497.636350 secs 2024-07-26T08:24:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:19 smithi097 ceph-mon[32231]: pgmap v1542: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:24:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:19 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1497.636350 secs 2024-07-26T08:24:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:21 smithi115 ceph-mon[35364]: pgmap v1543: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:21 smithi181 ceph-mon[35869]: pgmap v1543: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:21 smithi097 ceph-mon[32231]: pgmap v1543: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:24:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:23 smithi181 ceph-mon[35869]: pgmap v1544: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:23 smithi097 ceph-mon[32231]: pgmap v1544: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:23 smithi115 ceph-mon[35364]: pgmap v1544: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:24 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1502.636492 secs 2024-07-26T08:24:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:24 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1502.636492 secs 2024-07-26T08:24:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:24 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1502.636492 secs 2024-07-26T08:24:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:25 smithi097 ceph-mon[32231]: pgmap v1545: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:25 smithi115 ceph-mon[35364]: pgmap v1545: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:25 smithi181 ceph-mon[35869]: pgmap v1545: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:28 smithi115 ceph-mon[35364]: pgmap v1546: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:28 smithi181 ceph-mon[35869]: pgmap v1546: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:28 smithi097 ceph-mon[32231]: pgmap v1546: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:30 smithi115 ceph-mon[35364]: pgmap v1547: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1507.636629 secs 2024-07-26T08:24:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:30 smithi181 ceph-mon[35869]: pgmap v1547: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1507.636629 secs 2024-07-26T08:24:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:30 smithi097 ceph-mon[32231]: pgmap v1547: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1507.636629 secs 2024-07-26T08:24:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:32 smithi115 ceph-mon[35364]: pgmap v1548: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:32 smithi181 ceph-mon[35869]: pgmap v1548: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:32 smithi097 ceph-mon[32231]: pgmap v1548: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:32.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:32.998+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:24:33.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:32.999+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:24:33.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:32.999+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:24:33.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:32.999+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:24:33.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:32.999+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:24:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:34 smithi115 ceph-mon[35364]: pgmap v1549: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:34 smithi181 ceph-mon[35869]: pgmap v1549: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:34 smithi097 ceph-mon[32231]: pgmap v1549: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1512.636768 secs 2024-07-26T08:24:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1512.636768 secs 2024-07-26T08:24:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1512.636768 secs 2024-07-26T08:24:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:36 smithi115 ceph-mon[35364]: pgmap v1550: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:36 smithi181 ceph-mon[35869]: pgmap v1550: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:36 smithi097 ceph-mon[32231]: pgmap v1550: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:38 smithi115 ceph-mon[35364]: pgmap v1551: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:38 smithi181 ceph-mon[35869]: pgmap v1551: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:38 smithi097 ceph-mon[32231]: pgmap v1551: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:40.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:40 smithi097 ceph-mon[32231]: pgmap v1552: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:40.306 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1517.636907 secs 2024-07-26T08:24:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:40 smithi115 ceph-mon[35364]: pgmap v1552: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1517.636907 secs 2024-07-26T08:24:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:40 smithi181 ceph-mon[35869]: pgmap v1552: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1517.636907 secs 2024-07-26T08:24:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:42 smithi115 ceph-mon[35364]: pgmap v1553: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:42 smithi181 ceph-mon[35869]: pgmap v1553: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:42 smithi097 ceph-mon[32231]: pgmap v1553: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:44 smithi115 ceph-mon[35364]: pgmap v1554: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:44 smithi181 ceph-mon[35869]: pgmap v1554: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:44 smithi097 ceph-mon[32231]: pgmap v1554: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:45.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1522.637068 secs 2024-07-26T08:24:45.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1522.637068 secs 2024-07-26T08:24:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1522.637068 secs 2024-07-26T08:24:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:46 smithi115 ceph-mon[35364]: pgmap v1555: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:46 smithi181 ceph-mon[35869]: pgmap v1555: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:46 smithi097 ceph-mon[32231]: pgmap v1555: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:48.008+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:24:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:48.009+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:24:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:48.009+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:24:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:48.009+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:24:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:24:48.009+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:24:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:48 smithi115 ceph-mon[35364]: pgmap v1556: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:48 smithi181 ceph-mon[35869]: pgmap v1556: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:48 smithi097 ceph-mon[32231]: pgmap v1556: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:50 smithi115 ceph-mon[35364]: pgmap v1557: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1527.637221 secs 2024-07-26T08:24:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:50 smithi181 ceph-mon[35869]: pgmap v1557: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1527.637221 secs 2024-07-26T08:24:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:50 smithi097 ceph-mon[32231]: pgmap v1557: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:24:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1527.637221 secs 2024-07-26T08:24:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:52 smithi115 ceph-mon[35364]: pgmap v1558: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:52 smithi181 ceph-mon[35869]: pgmap v1558: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:52 smithi097 ceph-mon[32231]: pgmap v1558: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:24:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:54 smithi115 ceph-mon[35364]: pgmap v1559: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:54 smithi181 ceph-mon[35869]: pgmap v1559: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:54 smithi097 ceph-mon[32231]: pgmap v1559: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:24:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1532.637362 secs 2024-07-26T08:24:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1532.637362 secs 2024-07-26T08:24:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1532.637362 secs 2024-07-26T08:24:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:56 smithi115 ceph-mon[35364]: pgmap v1560: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:56 smithi181 ceph-mon[35869]: pgmap v1560: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:56 smithi097 ceph-mon[32231]: pgmap v1560: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:24:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:24:58 smithi115 ceph-mon[35364]: pgmap v1561: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:24:58 smithi181 ceph-mon[35869]: pgmap v1561: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:24:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:24:58 smithi097 ceph-mon[32231]: pgmap v1561: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:25:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:00 smithi115 ceph-mon[35364]: pgmap v1562: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1537.637500 secs 2024-07-26T08:25:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:00 smithi181 ceph-mon[35869]: pgmap v1562: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1537.637500 secs 2024-07-26T08:25:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:00 smithi097 ceph-mon[32231]: pgmap v1562: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1537.637500 secs 2024-07-26T08:25:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:02 smithi115 ceph-mon[35364]: pgmap v1563: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:02 smithi181 ceph-mon[35869]: pgmap v1563: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:02 smithi097 ceph-mon[32231]: pgmap v1563: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:03.010+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:03.011+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:25:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:03.011+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:25:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:03.011+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:25:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:03.011+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:25:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:04 smithi115 ceph-mon[35364]: pgmap v1564: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:25:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:04 smithi181 ceph-mon[35869]: pgmap v1564: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:25:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:04 smithi097 ceph-mon[32231]: pgmap v1564: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:25:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1542.637649 secs 2024-07-26T08:25:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1542.637649 secs 2024-07-26T08:25:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1542.637649 secs 2024-07-26T08:25:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:06 smithi115 ceph-mon[35364]: pgmap v1565: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:25:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:06 smithi181 ceph-mon[35869]: pgmap v1565: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:25:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:06 smithi097 ceph-mon[32231]: pgmap v1565: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:25:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:07.145+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:25:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:07.145+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:25:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:07.146+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:25:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:07.146+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:25:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:08 smithi115 ceph-mon[35364]: pgmap v1566: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:25:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:08 smithi181 ceph-mon[35869]: pgmap v1566: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:25:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:08 smithi097 ceph-mon[32231]: pgmap v1566: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:25:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:10 smithi115 ceph-mon[35364]: pgmap v1567: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1547.637790 secs 2024-07-26T08:25:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:10 smithi181 ceph-mon[35869]: pgmap v1567: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1547.637790 secs 2024-07-26T08:25:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:10 smithi097 ceph-mon[32231]: pgmap v1567: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1547.637790 secs 2024-07-26T08:25:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:12 smithi115 ceph-mon[35364]: pgmap v1568: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:12 smithi181 ceph-mon[35869]: pgmap v1568: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:12 smithi097 ceph-mon[32231]: pgmap v1568: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:13 smithi115 ceph-mon[35364]: pgmap v1569: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:25:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:13 smithi115 ceph-mon[35364]: pgmap v1570: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:25:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:25:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:13 smithi181 ceph-mon[35869]: pgmap v1569: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:25:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:13 smithi181 ceph-mon[35869]: pgmap v1570: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:25:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:25:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:13 smithi097 ceph-mon[32231]: pgmap v1569: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:25:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:13 smithi097 ceph-mon[32231]: pgmap v1570: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:25:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:25:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:15 smithi115 ceph-mon[35364]: pgmap v1571: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:25:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1552.637933 secs 2024-07-26T08:25:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:15 smithi181 ceph-mon[35869]: pgmap v1571: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:25:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1552.637933 secs 2024-07-26T08:25:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:15 smithi097 ceph-mon[32231]: pgmap v1571: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:25:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1552.637933 secs 2024-07-26T08:25:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:17 smithi115 ceph-mon[35364]: pgmap v1572: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:17 smithi181 ceph-mon[35869]: pgmap v1572: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:17 smithi097 ceph-mon[32231]: pgmap v1572: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:18.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:18.018+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:18.019+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:25:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:18.019+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:25:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:18.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:25:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:18.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:25:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:19 smithi115 ceph-mon[35364]: pgmap v1573: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:25:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:19 smithi181 ceph-mon[35869]: pgmap v1573: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:25:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:19 smithi097 ceph-mon[32231]: pgmap v1573: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:25:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1557.638073 secs 2024-07-26T08:25:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1557.638073 secs 2024-07-26T08:25:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1557.638073 secs 2024-07-26T08:25:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:21 smithi115 ceph-mon[35364]: pgmap v1574: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:21 smithi181 ceph-mon[35869]: pgmap v1574: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:21 smithi097 ceph-mon[32231]: pgmap v1574: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:23 smithi115 ceph-mon[35364]: pgmap v1575: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:25:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:23 smithi181 ceph-mon[35869]: pgmap v1575: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:25:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:23 smithi097 ceph-mon[32231]: pgmap v1575: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:25:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:25 smithi115 ceph-mon[35364]: pgmap v1576: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1562.638284 secs 2024-07-26T08:25:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:25 smithi181 ceph-mon[35869]: pgmap v1576: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1562.638284 secs 2024-07-26T08:25:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:25 smithi097 ceph-mon[32231]: pgmap v1576: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1562.638284 secs 2024-07-26T08:25:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:27 smithi115 ceph-mon[35364]: pgmap v1577: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:27 smithi181 ceph-mon[35869]: pgmap v1577: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:27 smithi097 ceph-mon[32231]: pgmap v1577: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:29 smithi115 ceph-mon[35364]: pgmap v1578: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:25:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:29 smithi181 ceph-mon[35869]: pgmap v1578: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:25:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:29 smithi097 ceph-mon[32231]: pgmap v1578: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:25:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1567.638433 secs 2024-07-26T08:25:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1567.638433 secs 2024-07-26T08:25:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1567.638433 secs 2024-07-26T08:25:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:31 smithi115 ceph-mon[35364]: pgmap v1579: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:25:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:31 smithi181 ceph-mon[35869]: pgmap v1579: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:25:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:31 smithi097 ceph-mon[32231]: pgmap v1579: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:25:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:33.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:33.024+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:33.024+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:25:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:33.024+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:25:33.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:33.024+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:25:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:33 smithi115 ceph-mon[35364]: pgmap v1580: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:25:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:33 smithi181 ceph-mon[35869]: pgmap v1580: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:25:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:33 smithi097 ceph-mon[32231]: pgmap v1580: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:25:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:35 smithi115 ceph-mon[35364]: pgmap v1581: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1572.638580 secs 2024-07-26T08:25:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:35 smithi181 ceph-mon[35869]: pgmap v1581: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1572.638580 secs 2024-07-26T08:25:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:35 smithi097 ceph-mon[32231]: pgmap v1581: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:25:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1572.638580 secs 2024-07-26T08:25:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:37 smithi115 ceph-mon[35364]: pgmap v1582: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:25:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:37 smithi181 ceph-mon[35869]: pgmap v1582: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:25:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:37 smithi097 ceph-mon[32231]: pgmap v1582: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:25:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:39 smithi115 ceph-mon[35364]: pgmap v1583: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:25:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:39 smithi181 ceph-mon[35869]: pgmap v1583: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:25:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:39 smithi097 ceph-mon[32231]: pgmap v1583: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:25:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1577.638726 secs 2024-07-26T08:25:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1577.638726 secs 2024-07-26T08:25:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1577.638726 secs 2024-07-26T08:25:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:41 smithi115 ceph-mon[35364]: pgmap v1584: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:25:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:41 smithi181 ceph-mon[35869]: pgmap v1584: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:25:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:41 smithi097 ceph-mon[32231]: pgmap v1584: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:25:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:43 smithi115 ceph-mon[35364]: pgmap v1585: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:25:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:43 smithi181 ceph-mon[35869]: pgmap v1585: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:25:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:43 smithi097 ceph-mon[32231]: pgmap v1585: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:25:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:45 smithi181 ceph-mon[35869]: pgmap v1586: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:25:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1582.638885 secs 2024-07-26T08:25:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:45 smithi097 ceph-mon[32231]: pgmap v1586: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:25:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1582.638885 secs 2024-07-26T08:25:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:45 smithi115 ceph-mon[35364]: pgmap v1586: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:25:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1582.638885 secs 2024-07-26T08:25:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:47 smithi181 ceph-mon[35869]: pgmap v1587: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:47 smithi097 ceph-mon[32231]: pgmap v1587: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:47 smithi115 ceph-mon[35364]: pgmap v1587: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:48.032+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:48.033+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:25:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:48.033+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:25:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:48.033+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:25:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:25:48.033+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:25:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:49 smithi097 ceph-mon[32231]: pgmap v1588: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:25:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:49 smithi115 ceph-mon[35364]: pgmap v1588: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:25:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:49 smithi181 ceph-mon[35869]: pgmap v1588: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:25:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1587.639027 secs 2024-07-26T08:25:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1587.639027 secs 2024-07-26T08:25:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1587.639027 secs 2024-07-26T08:25:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:51 smithi115 ceph-mon[35364]: pgmap v1589: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:25:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:51 smithi181 ceph-mon[35869]: pgmap v1589: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:25:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:51 smithi097 ceph-mon[32231]: pgmap v1589: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:25:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:53 smithi115 ceph-mon[35364]: pgmap v1590: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:25:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:53 smithi181 ceph-mon[35869]: pgmap v1590: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:25:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:53 smithi097 ceph-mon[32231]: pgmap v1590: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:25:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:55 smithi115 ceph-mon[35364]: pgmap v1591: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1592.639201 secs 2024-07-26T08:25:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:55 smithi181 ceph-mon[35869]: pgmap v1591: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1592.639201 secs 2024-07-26T08:25:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:55 smithi097 ceph-mon[32231]: pgmap v1591: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:25:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1592.639201 secs 2024-07-26T08:25:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:57 smithi115 ceph-mon[35364]: pgmap v1592: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:57 smithi181 ceph-mon[35869]: pgmap v1592: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:57 smithi097 ceph-mon[32231]: pgmap v1592: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:25:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:25:59 smithi115 ceph-mon[35364]: pgmap v1593: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:25:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:25:59 smithi181 ceph-mon[35869]: pgmap v1593: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:25:59 smithi097 ceph-mon[32231]: pgmap v1593: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1597.639341 secs 2024-07-26T08:26:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1597.639341 secs 2024-07-26T08:26:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1597.639341 secs 2024-07-26T08:26:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:01 smithi115 ceph-mon[35364]: pgmap v1594: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:01 smithi181 ceph-mon[35869]: pgmap v1594: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:01 smithi097 ceph-mon[32231]: pgmap v1594: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:03.035+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:03.036+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:26:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:03.036+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:26:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:03.036+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:26:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:03.036+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:26:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:03 smithi115 ceph-mon[35364]: pgmap v1595: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:26:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:03 smithi181 ceph-mon[35869]: pgmap v1595: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:26:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:03 smithi097 ceph-mon[32231]: pgmap v1595: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:26:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:05 smithi115 ceph-mon[35364]: pgmap v1596: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:26:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1602.639479 secs 2024-07-26T08:26:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:05 smithi181 ceph-mon[35869]: pgmap v1596: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:26:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1602.639479 secs 2024-07-26T08:26:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:05 smithi097 ceph-mon[32231]: pgmap v1596: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:26:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1602.639479 secs 2024-07-26T08:26:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:07 smithi115 ceph-mon[35364]: pgmap v1597: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:26:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:07 smithi181 ceph-mon[35869]: pgmap v1597: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:26:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:07 smithi097 ceph-mon[32231]: pgmap v1597: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:26:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:09 smithi115 ceph-mon[35364]: pgmap v1598: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:09 smithi181 ceph-mon[35869]: pgmap v1598: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:09 smithi097 ceph-mon[32231]: pgmap v1598: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:10.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1607.639615 secs 2024-07-26T08:26:10.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:10 smithi097 ceph-mon[32231]: pgmap v1599: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1607.639615 secs 2024-07-26T08:26:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:10 smithi115 ceph-mon[35364]: pgmap v1599: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1607.639615 secs 2024-07-26T08:26:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:10 smithi181 ceph-mon[35869]: pgmap v1599: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:13 smithi115 ceph-mon[35364]: pgmap v1600: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:13 smithi181 ceph-mon[35869]: pgmap v1600: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:13 smithi097 ceph-mon[32231]: pgmap v1600: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:26:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:26:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:26:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:15 smithi115 ceph-mon[35364]: pgmap v1601: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:26:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:15 smithi115 ceph-mon[35364]: pgmap v1602: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:26:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1612.639756 secs 2024-07-26T08:26:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:15 smithi181 ceph-mon[35869]: pgmap v1601: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:26:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:15 smithi181 ceph-mon[35869]: pgmap v1602: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:26:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1612.639756 secs 2024-07-26T08:26:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:15 smithi097 ceph-mon[32231]: pgmap v1601: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:26:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:15 smithi097 ceph-mon[32231]: pgmap v1602: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:26:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1612.639756 secs 2024-07-26T08:26:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:17 smithi115 ceph-mon[35364]: pgmap v1603: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:17 smithi181 ceph-mon[35869]: pgmap v1603: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:17 smithi097 ceph-mon[32231]: pgmap v1603: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:18.044+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:18.046+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:26:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:18.046+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:26:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:18.046+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:26:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:18.046+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:26:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:19 smithi115 ceph-mon[35364]: pgmap v1604: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:19 smithi181 ceph-mon[35869]: pgmap v1604: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:19 smithi097 ceph-mon[32231]: pgmap v1604: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1617.639899 secs 2024-07-26T08:26:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1617.639899 secs 2024-07-26T08:26:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1617.639899 secs 2024-07-26T08:26:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:21 smithi115 ceph-mon[35364]: pgmap v1605: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:21 smithi181 ceph-mon[35869]: pgmap v1605: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:21 smithi097 ceph-mon[32231]: pgmap v1605: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:23 smithi115 ceph-mon[35364]: pgmap v1606: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:23 smithi181 ceph-mon[35869]: pgmap v1606: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:23 smithi097 ceph-mon[32231]: pgmap v1606: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:25 smithi115 ceph-mon[35364]: pgmap v1607: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:26:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1622.640054 secs 2024-07-26T08:26:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:25 smithi181 ceph-mon[35869]: pgmap v1607: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:26:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1622.640054 secs 2024-07-26T08:26:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:25 smithi097 ceph-mon[32231]: pgmap v1607: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:26:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1622.640054 secs 2024-07-26T08:26:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:27 smithi115 ceph-mon[35364]: pgmap v1608: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:27 smithi181 ceph-mon[35869]: pgmap v1608: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:27 smithi097 ceph-mon[32231]: pgmap v1608: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:29 smithi115 ceph-mon[35364]: pgmap v1609: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:29 smithi181 ceph-mon[35869]: pgmap v1609: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:29 smithi097 ceph-mon[32231]: pgmap v1609: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1627.640137 secs 2024-07-26T08:26:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1627.640137 secs 2024-07-26T08:26:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1627.640137 secs 2024-07-26T08:26:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:31 smithi115 ceph-mon[35364]: pgmap v1610: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:31 smithi181 ceph-mon[35869]: pgmap v1610: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:31 smithi097 ceph-mon[32231]: pgmap v1610: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:33.050+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:33.051+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:33.051+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:33.051+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:33.052+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:26:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:33 smithi115 ceph-mon[35364]: pgmap v1611: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:33 smithi181 ceph-mon[35869]: pgmap v1611: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:33.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:33 smithi097 ceph-mon[32231]: pgmap v1611: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:35 smithi181 ceph-mon[35869]: pgmap v1612: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1632.640258 secs 2024-07-26T08:26:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:35 smithi097 ceph-mon[32231]: pgmap v1612: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1632.640258 secs 2024-07-26T08:26:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:35 smithi115 ceph-mon[35364]: pgmap v1612: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1632.640258 secs 2024-07-26T08:26:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:37 smithi097 ceph-mon[32231]: pgmap v1613: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:37 smithi115 ceph-mon[35364]: pgmap v1613: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:37 smithi181 ceph-mon[35869]: pgmap v1613: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:39 smithi097 ceph-mon[32231]: pgmap v1614: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:39 smithi115 ceph-mon[35364]: pgmap v1614: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:39 smithi181 ceph-mon[35869]: pgmap v1614: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1637.640378 secs 2024-07-26T08:26:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1637.640378 secs 2024-07-26T08:26:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1637.640378 secs 2024-07-26T08:26:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:41 smithi115 ceph-mon[35364]: pgmap v1615: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:41 smithi181 ceph-mon[35869]: pgmap v1615: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:41 smithi097 ceph-mon[32231]: pgmap v1615: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:43 smithi115 ceph-mon[35364]: pgmap v1616: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:43 smithi181 ceph-mon[35869]: pgmap v1616: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:43 smithi097 ceph-mon[32231]: pgmap v1616: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:45.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:45 smithi115 ceph-mon[35364]: pgmap v1617: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:45.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1642.640525 secs 2024-07-26T08:26:45.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:45 smithi181 ceph-mon[35869]: pgmap v1617: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:45.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1642.640525 secs 2024-07-26T08:26:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:45 smithi097 ceph-mon[32231]: pgmap v1617: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1642.640525 secs 2024-07-26T08:26:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:47 smithi115 ceph-mon[35364]: pgmap v1618: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:47 smithi181 ceph-mon[35869]: pgmap v1618: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:47 smithi097 ceph-mon[32231]: pgmap v1618: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:48.056+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:48.057+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:26:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:48.057+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:26:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:48.057+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:26:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:26:48.057+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:26:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:49 smithi115 ceph-mon[35364]: pgmap v1619: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:49 smithi181 ceph-mon[35869]: pgmap v1619: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:49 smithi097 ceph-mon[32231]: pgmap v1619: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1647.640671 secs 2024-07-26T08:26:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1647.640671 secs 2024-07-26T08:26:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1647.640671 secs 2024-07-26T08:26:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:51 smithi115 ceph-mon[35364]: pgmap v1620: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:51 smithi181 ceph-mon[35869]: pgmap v1620: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:51 smithi097 ceph-mon[32231]: pgmap v1620: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:53 smithi115 ceph-mon[35364]: pgmap v1621: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:53 smithi181 ceph-mon[35869]: pgmap v1621: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:53 smithi097 ceph-mon[32231]: pgmap v1621: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:26:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:55 smithi115 ceph-mon[35364]: pgmap v1622: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1652.640813 secs 2024-07-26T08:26:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:55 smithi181 ceph-mon[35869]: pgmap v1622: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1652.640813 secs 2024-07-26T08:26:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:55 smithi097 ceph-mon[32231]: pgmap v1622: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:26:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1652.640813 secs 2024-07-26T08:26:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:57 smithi115 ceph-mon[35364]: pgmap v1623: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:57 smithi181 ceph-mon[35869]: pgmap v1623: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:57 smithi097 ceph-mon[32231]: pgmap v1623: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:26:59 smithi115 ceph-mon[35364]: pgmap v1624: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:26:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:26:59 smithi181 ceph-mon[35869]: pgmap v1624: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:26:59 smithi097 ceph-mon[32231]: pgmap v1624: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1657.640949 secs 2024-07-26T08:27:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1657.640949 secs 2024-07-26T08:27:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1657.640949 secs 2024-07-26T08:27:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:01 smithi115 ceph-mon[35364]: pgmap v1625: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:01 smithi181 ceph-mon[35869]: pgmap v1625: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:01 smithi097 ceph-mon[32231]: pgmap v1625: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:02 smithi115 ceph-mon[35364]: pgmap v1626: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:02 smithi181 ceph-mon[35869]: pgmap v1626: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:02 smithi097 ceph-mon[32231]: pgmap v1626: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:03.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:03.061+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:03.063+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:27:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:03.063+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:27:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:03.063+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:27:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:03.063+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:27:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:04 smithi115 ceph-mon[35364]: pgmap v1627: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:04 smithi181 ceph-mon[35869]: pgmap v1627: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:04 smithi097 ceph-mon[32231]: pgmap v1627: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1662.641090 secs 2024-07-26T08:27:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1662.641090 secs 2024-07-26T08:27:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1662.641090 secs 2024-07-26T08:27:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:06 smithi115 ceph-mon[35364]: pgmap v1628: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:06 smithi181 ceph-mon[35869]: pgmap v1628: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:06 smithi097 ceph-mon[32231]: pgmap v1628: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:08 smithi115 ceph-mon[35364]: pgmap v1629: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:08 smithi181 ceph-mon[35869]: pgmap v1629: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:08 smithi097 ceph-mon[32231]: pgmap v1629: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1667.641272 secs 2024-07-26T08:27:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:10 smithi115 ceph-mon[35364]: pgmap v1630: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1667.641272 secs 2024-07-26T08:27:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:10 smithi181 ceph-mon[35869]: pgmap v1630: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1667.641272 secs 2024-07-26T08:27:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:10 smithi097 ceph-mon[32231]: pgmap v1630: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:12 smithi115 ceph-mon[35364]: pgmap v1631: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:12 smithi181 ceph-mon[35869]: pgmap v1631: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:12 smithi097 ceph-mon[32231]: pgmap v1631: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:14 smithi115 ceph-mon[35364]: pgmap v1632: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:14 smithi181 ceph-mon[35869]: pgmap v1632: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:14 smithi097 ceph-mon[32231]: pgmap v1632: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1672.641376 secs 2024-07-26T08:27:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: pgmap v1633: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: pgmap v1634: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:16.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:27:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1672.641376 secs 2024-07-26T08:27:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: pgmap v1633: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: pgmap v1634: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1672.641376 secs 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:27:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:27:16.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:27:16.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:16.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:16.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: pgmap v1633: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:27:16.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: pgmap v1634: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:16.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:27:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:18.066+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:18.067+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:27:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:18.067+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:27:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:18.067+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:27:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:18.067+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:27:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:18 smithi097 ceph-mon[32231]: pgmap v1635: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:27:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:18 smithi115 ceph-mon[35364]: pgmap v1635: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:27:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:18 smithi181 ceph-mon[35869]: pgmap v1635: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:27:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:20 smithi115 ceph-mon[35364]: pgmap v1636: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:27:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1677.641465 secs 2024-07-26T08:27:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:20 smithi181 ceph-mon[35869]: pgmap v1636: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:27:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1677.641465 secs 2024-07-26T08:27:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:20 smithi097 ceph-mon[32231]: pgmap v1636: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:27:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1677.641465 secs 2024-07-26T08:27:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:22 smithi115 ceph-mon[35364]: pgmap v1637: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:27:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:22 smithi181 ceph-mon[35869]: pgmap v1637: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:27:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:22 smithi097 ceph-mon[32231]: pgmap v1637: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:27:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:24 smithi115 ceph-mon[35364]: pgmap v1638: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:27:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:24 smithi181 ceph-mon[35869]: pgmap v1638: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:27:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:24 smithi097 ceph-mon[32231]: pgmap v1638: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:27:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1682.641608 secs 2024-07-26T08:27:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1682.641608 secs 2024-07-26T08:27:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1682.641608 secs 2024-07-26T08:27:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:26 smithi115 ceph-mon[35364]: pgmap v1639: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:27:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:26 smithi181 ceph-mon[35869]: pgmap v1639: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:27:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:26 smithi097 ceph-mon[32231]: pgmap v1639: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:27:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:28 smithi115 ceph-mon[35364]: pgmap v1640: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:27:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:28 smithi181 ceph-mon[35869]: pgmap v1640: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:27:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:28 smithi097 ceph-mon[32231]: pgmap v1640: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:27:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:30 smithi115 ceph-mon[35364]: pgmap v1641: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:27:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1687.641699 secs 2024-07-26T08:27:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:30 smithi181 ceph-mon[35869]: pgmap v1641: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:27:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1687.641699 secs 2024-07-26T08:27:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:30 smithi097 ceph-mon[32231]: pgmap v1641: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:27:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1687.641699 secs 2024-07-26T08:27:32.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:32 smithi115 ceph-mon[35364]: pgmap v1642: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:32 smithi181 ceph-mon[35869]: pgmap v1642: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:32 smithi097 ceph-mon[32231]: pgmap v1642: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:33.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:33.071+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:33.072+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:27:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:33.072+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:27:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:33.072+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:27:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:33.072+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:27:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:34 smithi115 ceph-mon[35364]: pgmap v1643: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:34 smithi181 ceph-mon[35869]: pgmap v1643: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:34 smithi097 ceph-mon[32231]: pgmap v1643: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1692.641845 secs 2024-07-26T08:27:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1692.641845 secs 2024-07-26T08:27:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1692.641845 secs 2024-07-26T08:27:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:36 smithi115 ceph-mon[35364]: pgmap v1644: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:36 smithi181 ceph-mon[35869]: pgmap v1644: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:36 smithi097 ceph-mon[32231]: pgmap v1644: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:38 smithi115 ceph-mon[35364]: pgmap v1645: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:38 smithi181 ceph-mon[35869]: pgmap v1645: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:38 smithi097 ceph-mon[32231]: pgmap v1645: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:40 smithi115 ceph-mon[35364]: pgmap v1646: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:27:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1697.642003 secs 2024-07-26T08:27:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:40 smithi181 ceph-mon[35869]: pgmap v1646: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:27:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1697.642003 secs 2024-07-26T08:27:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:40 smithi097 ceph-mon[32231]: pgmap v1646: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:27:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1697.642003 secs 2024-07-26T08:27:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:42 smithi115 ceph-mon[35364]: pgmap v1647: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:42 smithi181 ceph-mon[35869]: pgmap v1647: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:42 smithi097 ceph-mon[32231]: pgmap v1647: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:44 smithi115 ceph-mon[35364]: pgmap v1648: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:44 smithi181 ceph-mon[35869]: pgmap v1648: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:44 smithi097 ceph-mon[32231]: pgmap v1648: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:45.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1702.642167 secs 2024-07-26T08:27:45.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1702.642167 secs 2024-07-26T08:27:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1702.642167 secs 2024-07-26T08:27:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:46 smithi115 ceph-mon[35364]: pgmap v1649: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:46 smithi181 ceph-mon[35869]: pgmap v1649: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:46 smithi097 ceph-mon[32231]: pgmap v1649: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:48.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:48.074+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:48.076+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:27:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:48.076+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:27:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:48.076+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:27:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:27:48.076+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:27:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:48 smithi115 ceph-mon[35364]: pgmap v1650: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:48 smithi181 ceph-mon[35869]: pgmap v1650: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:48 smithi097 ceph-mon[32231]: pgmap v1650: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:27:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:50 smithi115 ceph-mon[35364]: pgmap v1651: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:27:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1707.642330 secs 2024-07-26T08:27:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:50 smithi181 ceph-mon[35869]: pgmap v1651: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:27:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1707.642330 secs 2024-07-26T08:27:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:50 smithi097 ceph-mon[32231]: pgmap v1651: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:27:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1707.642330 secs 2024-07-26T08:27:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:52 smithi115 ceph-mon[35364]: pgmap v1652: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:52 smithi181 ceph-mon[35869]: pgmap v1652: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:52 smithi097 ceph-mon[32231]: pgmap v1652: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:27:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:54 smithi115 ceph-mon[35364]: pgmap v1653: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:54 smithi181 ceph-mon[35869]: pgmap v1653: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:54 smithi097 ceph-mon[32231]: pgmap v1653: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:27:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1712.642475 secs 2024-07-26T08:27:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1712.642475 secs 2024-07-26T08:27:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1712.642475 secs 2024-07-26T08:27:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:56 smithi115 ceph-mon[35364]: pgmap v1654: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:56 smithi181 ceph-mon[35869]: pgmap v1654: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:56 smithi097 ceph-mon[32231]: pgmap v1654: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:27:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:27:58 smithi115 ceph-mon[35364]: pgmap v1655: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:27:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:27:58 smithi181 ceph-mon[35869]: pgmap v1655: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:27:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:27:58 smithi097 ceph-mon[32231]: pgmap v1655: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:28:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:00 smithi115 ceph-mon[35364]: pgmap v1656: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:28:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1717.642619 secs 2024-07-26T08:28:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:00 smithi181 ceph-mon[35869]: pgmap v1656: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:28:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1717.642619 secs 2024-07-26T08:28:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:00 smithi097 ceph-mon[32231]: pgmap v1656: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:28:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1717.642619 secs 2024-07-26T08:28:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:02 smithi115 ceph-mon[35364]: pgmap v1657: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:02 smithi181 ceph-mon[35869]: pgmap v1657: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:02 smithi097 ceph-mon[32231]: pgmap v1657: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:03.078+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:03.079+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:28:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:03.079+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:28:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:03.079+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:28:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:03.079+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:28:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:04 smithi115 ceph-mon[35364]: pgmap v1658: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:04 smithi181 ceph-mon[35869]: pgmap v1658: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:04.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:04 smithi097 ceph-mon[32231]: pgmap v1658: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1722.642751 secs 2024-07-26T08:28:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1722.642751 secs 2024-07-26T08:28:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1722.642751 secs 2024-07-26T08:28:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:06 smithi115 ceph-mon[35364]: pgmap v1659: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:06 smithi181 ceph-mon[35869]: pgmap v1659: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:06 smithi097 ceph-mon[32231]: pgmap v1659: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.881+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 msgr2=0x7ff3d40c0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T08:28:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.881+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 msgr2=0x7ff3d40c0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T08:28:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.881+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 0x7ff3d40c0000 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff3e001fef0 tx=0x7ff3e0038000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:28:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.881+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 0x7ff3d40c0000 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff3e001fef0 tx=0x7ff3e0038000 comp rx=0 tx=0).stop 2024-07-26T08:28:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.881+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 msgr2=0x7ff3d40c0000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:28:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.881+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdbe0 0x7ff3d40c0000 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.881+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.882+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:06.883+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0540 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e003b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:28:07.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:07.177+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:28:07.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:07.177+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:28:07.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:07.177+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:28:07.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:07.178+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:28:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:08 smithi115 ceph-mon[35364]: pgmap v1660: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:08 smithi181 ceph-mon[35869]: pgmap v1660: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:08 smithi097 ceph-mon[32231]: pgmap v1660: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:10 smithi181 ceph-mon[35869]: pgmap v1661: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:28:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1727.642895 secs 2024-07-26T08:28:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:10 smithi097 ceph-mon[32231]: pgmap v1661: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:28:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1727.642895 secs 2024-07-26T08:28:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:10 smithi115 ceph-mon[35364]: pgmap v1661: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:28:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1727.642895 secs 2024-07-26T08:28:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:12 smithi181 ceph-mon[35869]: pgmap v1662: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:12 smithi097 ceph-mon[32231]: pgmap v1662: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:12 smithi115 ceph-mon[35364]: pgmap v1662: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:14 smithi097 ceph-mon[32231]: pgmap v1663: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:14 smithi115 ceph-mon[35364]: pgmap v1663: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:14 smithi181 ceph-mon[35869]: pgmap v1663: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1732.643038 secs 2024-07-26T08:28:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:15 smithi115 ceph-mon[35364]: pgmap v1664: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1732.643038 secs 2024-07-26T08:28:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:15 smithi181 ceph-mon[35869]: pgmap v1664: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1732.643038 secs 2024-07-26T08:28:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:15 smithi097 ceph-mon[32231]: pgmap v1664: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:28:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:28:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:28:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:18.082+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:18.083+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:28:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:18.083+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:28:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:18.083+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:28:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:18.083+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:28:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:17 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:28:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:17 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:28:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:17 smithi115 ceph-mon[35364]: pgmap v1665: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:28:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:17 smithi115 ceph-mon[35364]: pgmap v1666: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:17 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:28:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:17 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:28:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:17 smithi181 ceph-mon[35869]: pgmap v1665: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:28:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:17 smithi181 ceph-mon[35869]: pgmap v1666: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:17 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:28:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:17 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:28:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:17 smithi097 ceph-mon[32231]: pgmap v1665: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:28:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:17 smithi097 ceph-mon[32231]: pgmap v1666: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:19 smithi115 ceph-mon[35364]: pgmap v1667: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:19 smithi181 ceph-mon[35869]: pgmap v1667: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:19 smithi097 ceph-mon[32231]: pgmap v1667: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:28:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1737.643200 secs 2024-07-26T08:28:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1737.643200 secs 2024-07-26T08:28:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1737.643200 secs 2024-07-26T08:28:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:21 smithi115 ceph-mon[35364]: pgmap v1668: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:21 smithi181 ceph-mon[35869]: pgmap v1668: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:21 smithi097 ceph-mon[32231]: pgmap v1668: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:23 smithi115 ceph-mon[35364]: pgmap v1669: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:23 smithi181 ceph-mon[35869]: pgmap v1669: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:23 smithi097 ceph-mon[32231]: pgmap v1669: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1742.643353 secs 2024-07-26T08:28:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:25 smithi115 ceph-mon[35364]: pgmap v1670: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1742.643353 secs 2024-07-26T08:28:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:25 smithi181 ceph-mon[35869]: pgmap v1670: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1742.643353 secs 2024-07-26T08:28:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:25 smithi097 ceph-mon[32231]: pgmap v1670: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:27 smithi115 ceph-mon[35364]: pgmap v1671: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:28:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:27 smithi181 ceph-mon[35869]: pgmap v1671: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:28:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:27 smithi097 ceph-mon[32231]: pgmap v1671: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:28:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:29 smithi115 ceph-mon[35364]: pgmap v1672: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:29 smithi181 ceph-mon[35869]: pgmap v1672: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:29 smithi097 ceph-mon[32231]: pgmap v1672: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1747.643498 secs 2024-07-26T08:28:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1747.643498 secs 2024-07-26T08:28:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1747.643498 secs 2024-07-26T08:28:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:31 smithi115 ceph-mon[35364]: pgmap v1673: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:31 smithi181 ceph-mon[35869]: pgmap v1673: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:31 smithi097 ceph-mon[32231]: pgmap v1673: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:33.086+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:33.087+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:28:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:33.087+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:28:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:33.087+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:28:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:33.087+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:28:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:33 smithi115 ceph-mon[35364]: pgmap v1674: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:33 smithi181 ceph-mon[35869]: pgmap v1674: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:33 smithi097 ceph-mon[32231]: pgmap v1674: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:34 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1752.643640 secs 2024-07-26T08:28:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:34 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1752.643640 secs 2024-07-26T08:28:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:34 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1752.643640 secs 2024-07-26T08:28:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:35 smithi097 ceph-mon[32231]: pgmap v1675: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:35 smithi115 ceph-mon[35364]: pgmap v1675: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:35 smithi181 ceph-mon[35869]: pgmap v1675: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:37 smithi097 ceph-mon[32231]: pgmap v1676: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:37 smithi115 ceph-mon[35364]: pgmap v1676: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:37 smithi181 ceph-mon[35869]: pgmap v1676: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:39 smithi097 ceph-mon[32231]: pgmap v1677: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:39 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1757.643782 secs 2024-07-26T08:28:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:39 smithi115 ceph-mon[35364]: pgmap v1677: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:39 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1757.643782 secs 2024-07-26T08:28:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:39 smithi181 ceph-mon[35869]: pgmap v1677: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:39 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1757.643782 secs 2024-07-26T08:28:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:42 smithi115 ceph-mon[35364]: pgmap v1678: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:42 smithi181 ceph-mon[35869]: pgmap v1678: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:42 smithi097 ceph-mon[32231]: pgmap v1678: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:44 smithi115 ceph-mon[35364]: pgmap v1679: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:44 smithi181 ceph-mon[35869]: pgmap v1679: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:44 smithi097 ceph-mon[32231]: pgmap v1679: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1762.643956 secs 2024-07-26T08:28:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:45 smithi115 ceph-mon[35364]: pgmap v1680: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1762.643956 secs 2024-07-26T08:28:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:45 smithi181 ceph-mon[35869]: pgmap v1680: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1762.643956 secs 2024-07-26T08:28:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:45 smithi097 ceph-mon[32231]: pgmap v1680: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:48.096+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:48.097+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:28:48.099 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:48.097+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:28:48.099 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:48.097+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:28:48.099 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:28:48.097+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:28:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:47 smithi115 ceph-mon[35364]: pgmap v1681: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:47 smithi181 ceph-mon[35869]: pgmap v1681: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:47 smithi097 ceph-mon[32231]: pgmap v1681: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:49 smithi115 ceph-mon[35364]: pgmap v1682: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:49 smithi181 ceph-mon[35869]: pgmap v1682: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:49 smithi097 ceph-mon[32231]: pgmap v1682: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1767.644096 secs 2024-07-26T08:28:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1767.644096 secs 2024-07-26T08:28:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1767.644096 secs 2024-07-26T08:28:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:51 smithi115 ceph-mon[35364]: pgmap v1683: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:51 smithi181 ceph-mon[35869]: pgmap v1683: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:51 smithi097 ceph-mon[32231]: pgmap v1683: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:53 smithi115 ceph-mon[35364]: pgmap v1684: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:53 smithi181 ceph-mon[35869]: pgmap v1684: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:53 smithi097 ceph-mon[32231]: pgmap v1684: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:28:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1772.644278 secs 2024-07-26T08:28:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:55 smithi115 ceph-mon[35364]: pgmap v1685: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1772.644278 secs 2024-07-26T08:28:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:55 smithi181 ceph-mon[35869]: pgmap v1685: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1772.644278 secs 2024-07-26T08:28:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:55 smithi097 ceph-mon[32231]: pgmap v1685: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:28:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:57 smithi115 ceph-mon[35364]: pgmap v1686: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:57 smithi181 ceph-mon[35869]: pgmap v1686: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:28:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:57 smithi097 ceph-mon[32231]: pgmap v1686: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:28:59 smithi115 ceph-mon[35364]: pgmap v1687: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:28:59 smithi181 ceph-mon[35869]: pgmap v1687: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:28:59 smithi097 ceph-mon[32231]: pgmap v1687: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1777.644422 secs 2024-07-26T08:29:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1777.644422 secs 2024-07-26T08:29:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1777.644422 secs 2024-07-26T08:29:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:01 smithi115 ceph-mon[35364]: pgmap v1688: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:01 smithi181 ceph-mon[35869]: pgmap v1688: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:01 smithi097 ceph-mon[32231]: pgmap v1688: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:03.101+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:03.103+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:29:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:03.103+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:29:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:03.103+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:29:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:03.104+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:29:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:03 smithi115 ceph-mon[35364]: pgmap v1689: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:03 smithi181 ceph-mon[35869]: pgmap v1689: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:03 smithi097 ceph-mon[32231]: pgmap v1689: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1782.644564 secs 2024-07-26T08:29:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:05 smithi115 ceph-mon[35364]: pgmap v1690: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1782.644564 secs 2024-07-26T08:29:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:05 smithi181 ceph-mon[35869]: pgmap v1690: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1782.644564 secs 2024-07-26T08:29:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:05 smithi097 ceph-mon[32231]: pgmap v1690: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:07 smithi115 ceph-mon[35364]: pgmap v1691: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:07 smithi181 ceph-mon[35869]: pgmap v1691: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:07 smithi097 ceph-mon[32231]: pgmap v1691: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:09 smithi115 ceph-mon[35364]: pgmap v1692: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:09 smithi181 ceph-mon[35869]: pgmap v1692: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:09 smithi097 ceph-mon[32231]: pgmap v1692: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1787.644703 secs 2024-07-26T08:29:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1787.644703 secs 2024-07-26T08:29:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1787.644703 secs 2024-07-26T08:29:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:11 smithi097 ceph-mon[32231]: pgmap v1693: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:11 smithi115 ceph-mon[35364]: pgmap v1693: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:11 smithi181 ceph-mon[35869]: pgmap v1693: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:13 smithi097 ceph-mon[32231]: pgmap v1694: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:13 smithi115 ceph-mon[35364]: pgmap v1694: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:13 smithi181 ceph-mon[35869]: pgmap v1694: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1792.644862 secs 2024-07-26T08:29:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1792.644862 secs 2024-07-26T08:29:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1792.644862 secs 2024-07-26T08:29:16.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:16 smithi115 ceph-mon[35364]: pgmap v1695: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:16.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:16 smithi181 ceph-mon[35869]: pgmap v1695: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:16 smithi097 ceph-mon[32231]: pgmap v1695: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:18.113+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:18.114+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:29:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:18.114+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:29:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:18.114+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:29:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:18.114+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:29:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:18 smithi115 ceph-mon[35364]: pgmap v1696: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:18 smithi181 ceph-mon[35869]: pgmap v1696: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:18 smithi097 ceph-mon[32231]: pgmap v1696: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:29:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:20 smithi115 ceph-mon[35364]: pgmap v1697: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:20 smithi115 ceph-mon[35364]: pgmap v1698: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1797.644995 secs 2024-07-26T08:29:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:20 smithi181 ceph-mon[35869]: pgmap v1697: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:20 smithi181 ceph-mon[35869]: pgmap v1698: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1797.644995 secs 2024-07-26T08:29:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:20 smithi097 ceph-mon[32231]: pgmap v1697: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:20 smithi097 ceph-mon[32231]: pgmap v1698: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1797.644995 secs 2024-07-26T08:29:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:22 smithi115 ceph-mon[35364]: pgmap v1699: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:22 smithi181 ceph-mon[35869]: pgmap v1699: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:22 smithi097 ceph-mon[32231]: pgmap v1699: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:24 smithi115 ceph-mon[35364]: pgmap v1700: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:24 smithi181 ceph-mon[35869]: pgmap v1700: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:24 smithi097 ceph-mon[32231]: pgmap v1700: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1802.645145 secs 2024-07-26T08:29:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1802.645145 secs 2024-07-26T08:29:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1802.645145 secs 2024-07-26T08:29:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:26 smithi115 ceph-mon[35364]: pgmap v1701: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:26 smithi181 ceph-mon[35869]: pgmap v1701: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:26 smithi097 ceph-mon[32231]: pgmap v1701: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:28 smithi115 ceph-mon[35364]: pgmap v1702: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:28 smithi181 ceph-mon[35869]: pgmap v1702: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:28 smithi097 ceph-mon[32231]: pgmap v1702: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:29:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:30 smithi115 ceph-mon[35364]: pgmap v1703: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:29:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1807.645319 secs 2024-07-26T08:29:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:30 smithi181 ceph-mon[35869]: pgmap v1703: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:29:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1807.645319 secs 2024-07-26T08:29:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:30 smithi097 ceph-mon[32231]: pgmap v1703: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:29:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1807.645319 secs 2024-07-26T08:29:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:32 smithi115 ceph-mon[35364]: pgmap v1704: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:32 smithi181 ceph-mon[35869]: pgmap v1704: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:32 smithi097 ceph-mon[32231]: pgmap v1704: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:33.117+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:33.118+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:29:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:33.118+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:29:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:33.118+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:29:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:33.118+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:29:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:34 smithi115 ceph-mon[35364]: pgmap v1705: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:34 smithi181 ceph-mon[35869]: pgmap v1705: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:34 smithi097 ceph-mon[32231]: pgmap v1705: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1812.645424 secs 2024-07-26T08:29:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1812.645424 secs 2024-07-26T08:29:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1812.645424 secs 2024-07-26T08:29:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:36 smithi115 ceph-mon[35364]: pgmap v1706: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:36 smithi181 ceph-mon[35869]: pgmap v1706: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:36 smithi097 ceph-mon[32231]: pgmap v1706: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:38 smithi115 ceph-mon[35364]: pgmap v1707: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:38 smithi181 ceph-mon[35869]: pgmap v1707: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:38 smithi097 ceph-mon[32231]: pgmap v1707: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:40 smithi115 ceph-mon[35364]: pgmap v1708: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1817.645576 secs 2024-07-26T08:29:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:40 smithi181 ceph-mon[35869]: pgmap v1708: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1817.645576 secs 2024-07-26T08:29:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:40 smithi097 ceph-mon[32231]: pgmap v1708: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1817.645576 secs 2024-07-26T08:29:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:42 smithi115 ceph-mon[35364]: pgmap v1709: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:42 smithi181 ceph-mon[35869]: pgmap v1709: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:42 smithi097 ceph-mon[32231]: pgmap v1709: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:44 smithi115 ceph-mon[35364]: pgmap v1710: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:44 smithi181 ceph-mon[35869]: pgmap v1710: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:44.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:44 smithi097 ceph-mon[32231]: pgmap v1710: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1822.645718 secs 2024-07-26T08:29:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1822.645718 secs 2024-07-26T08:29:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1822.645718 secs 2024-07-26T08:29:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:46 smithi115 ceph-mon[35364]: pgmap v1711: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:46 smithi181 ceph-mon[35869]: pgmap v1711: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:46 smithi097 ceph-mon[32231]: pgmap v1711: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:48.124+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:48.125+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:29:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:48.125+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:29:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:48.125+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:29:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:29:48.125+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:29:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:48 smithi115 ceph-mon[35364]: pgmap v1712: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:48 smithi181 ceph-mon[35869]: pgmap v1712: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:48 smithi097 ceph-mon[32231]: pgmap v1712: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:50 smithi115 ceph-mon[35364]: pgmap v1713: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1827.645874 secs 2024-07-26T08:29:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:50 smithi181 ceph-mon[35869]: pgmap v1713: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1827.645874 secs 2024-07-26T08:29:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:50 smithi097 ceph-mon[32231]: pgmap v1713: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1827.645874 secs 2024-07-26T08:29:52.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:52 smithi115 ceph-mon[35364]: pgmap v1714: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:52 smithi181 ceph-mon[35869]: pgmap v1714: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:52 smithi097 ceph-mon[32231]: pgmap v1714: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:54 smithi115 ceph-mon[35364]: pgmap v1715: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:54 smithi181 ceph-mon[35869]: pgmap v1715: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:54 smithi097 ceph-mon[32231]: pgmap v1715: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:29:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1832.646037 secs 2024-07-26T08:29:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1832.646037 secs 2024-07-26T08:29:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1832.646037 secs 2024-07-26T08:29:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:56 smithi115 ceph-mon[35364]: pgmap v1716: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:56 smithi181 ceph-mon[35869]: pgmap v1716: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:56 smithi097 ceph-mon[32231]: pgmap v1716: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:29:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:29:58 smithi115 ceph-mon[35364]: pgmap v1717: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:29:58 smithi181 ceph-mon[35869]: pgmap v1717: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:29:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:29:58 smithi097 ceph-mon[32231]: pgmap v1717: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:00 smithi115 ceph-mon[35364]: pgmap v1718: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1837.646181 secs 2024-07-26T08:30:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:30:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:00 smithi181 ceph-mon[35869]: pgmap v1718: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1837.646181 secs 2024-07-26T08:30:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:30:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:00 smithi097 ceph-mon[32231]: pgmap v1718: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1837.646181 secs 2024-07-26T08:30:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:30:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:02 smithi115 ceph-mon[35364]: pgmap v1719: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:02 smithi181 ceph-mon[35869]: pgmap v1719: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:02 smithi097 ceph-mon[32231]: pgmap v1719: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:03.131 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:03.129+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:03.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:03.130+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:30:03.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:03.130+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:30:03.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:03.130+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:30:03.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:03.130+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:30:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:04 smithi181 ceph-mon[35869]: pgmap v1720: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:04 smithi097 ceph-mon[32231]: pgmap v1720: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:04 smithi115 ceph-mon[35364]: pgmap v1720: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1842.646327 secs 2024-07-26T08:30:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1842.646327 secs 2024-07-26T08:30:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1842.646327 secs 2024-07-26T08:30:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:06 smithi097 ceph-mon[32231]: pgmap v1721: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:06 smithi115 ceph-mon[35364]: pgmap v1721: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:06 smithi181 ceph-mon[35869]: pgmap v1721: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:08 smithi097 ceph-mon[32231]: pgmap v1722: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:08 smithi115 ceph-mon[35364]: pgmap v1722: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:08 smithi181 ceph-mon[35869]: pgmap v1722: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:10 smithi115 ceph-mon[35364]: pgmap v1723: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1847.646419 secs 2024-07-26T08:30:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:10 smithi181 ceph-mon[35869]: pgmap v1723: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1847.646419 secs 2024-07-26T08:30:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:10 smithi097 ceph-mon[32231]: pgmap v1723: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1847.646419 secs 2024-07-26T08:30:12.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:12 smithi097 ceph-mon[32231]: pgmap v1724: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:12 smithi115 ceph-mon[35364]: pgmap v1724: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:12 smithi181 ceph-mon[35869]: pgmap v1724: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:14 smithi097 ceph-mon[32231]: pgmap v1725: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:14 smithi115 ceph-mon[35364]: pgmap v1725: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:14 smithi181 ceph-mon[35869]: pgmap v1725: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1852.646517 secs 2024-07-26T08:30:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1852.646517 secs 2024-07-26T08:30:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1852.646517 secs 2024-07-26T08:30:16.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:16 smithi115 ceph-mon[35364]: pgmap v1726: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:16.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:16 smithi181 ceph-mon[35869]: pgmap v1726: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:16 smithi097 ceph-mon[32231]: pgmap v1726: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:18.139+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:18.140+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:30:18.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:18.140+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:30:18.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:18.140+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:30:18.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:18.140+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:30:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:18 smithi115 ceph-mon[35364]: pgmap v1727: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:18 smithi181 ceph-mon[35869]: pgmap v1727: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:18 smithi097 ceph-mon[32231]: pgmap v1727: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:20 smithi115 ceph-mon[35364]: pgmap v1728: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1857.646617 secs 2024-07-26T08:30:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:20 smithi181 ceph-mon[35869]: pgmap v1728: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1857.646617 secs 2024-07-26T08:30:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:20 smithi097 ceph-mon[32231]: pgmap v1728: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:30:20.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1857.646617 secs 2024-07-26T08:30:20.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:20.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:30:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:22 smithi115 ceph-mon[35364]: pgmap v1729: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:22 smithi115 ceph-mon[35364]: pgmap v1730: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:22 smithi115 ceph-mon[35364]: pgmap v1731: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:30:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:22 smithi181 ceph-mon[35869]: pgmap v1729: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:22 smithi181 ceph-mon[35869]: pgmap v1730: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:22 smithi181 ceph-mon[35869]: pgmap v1731: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:30:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:22 smithi097 ceph-mon[32231]: pgmap v1729: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:22 smithi097 ceph-mon[32231]: pgmap v1730: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:22 smithi097 ceph-mon[32231]: pgmap v1731: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:30:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:24 smithi115 ceph-mon[35364]: pgmap v1732: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:30:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:24 smithi181 ceph-mon[35869]: pgmap v1732: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:30:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:24 smithi097 ceph-mon[32231]: pgmap v1732: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:30:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1862.646757 secs 2024-07-26T08:30:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1862.646757 secs 2024-07-26T08:30:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1862.646757 secs 2024-07-26T08:30:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:26 smithi115 ceph-mon[35364]: pgmap v1733: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:30:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:26 smithi181 ceph-mon[35869]: pgmap v1733: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:30:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:26 smithi097 ceph-mon[32231]: pgmap v1733: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:30:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:28 smithi115 ceph-mon[35364]: pgmap v1734: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:30:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:28 smithi181 ceph-mon[35869]: pgmap v1734: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:30:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:28 smithi097 ceph-mon[32231]: pgmap v1734: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:30:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:30 smithi115 ceph-mon[35364]: pgmap v1735: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:30:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1867.646902 secs 2024-07-26T08:30:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:30 smithi181 ceph-mon[35869]: pgmap v1735: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:30:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1867.646902 secs 2024-07-26T08:30:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:30 smithi097 ceph-mon[32231]: pgmap v1735: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:30:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1867.646902 secs 2024-07-26T08:30:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:32 smithi115 ceph-mon[35364]: pgmap v1736: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:32 smithi181 ceph-mon[35869]: pgmap v1736: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:32 smithi097 ceph-mon[32231]: pgmap v1736: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:33.144+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:33.145+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:30:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:33.146+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:30:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:33.146+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:30:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:33.146+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:30:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:34 smithi115 ceph-mon[35364]: pgmap v1737: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:34 smithi181 ceph-mon[35869]: pgmap v1737: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:34 smithi097 ceph-mon[32231]: pgmap v1737: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:30:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1872.647080 secs 2024-07-26T08:30:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1872.647080 secs 2024-07-26T08:30:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1872.647080 secs 2024-07-26T08:30:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:36 smithi115 ceph-mon[35364]: pgmap v1738: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:36 smithi181 ceph-mon[35869]: pgmap v1738: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:36.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:36 smithi097 ceph-mon[32231]: pgmap v1738: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:38 smithi115 ceph-mon[35364]: pgmap v1739: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:30:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:38 smithi181 ceph-mon[35869]: pgmap v1739: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:30:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:38 smithi097 ceph-mon[32231]: pgmap v1739: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:30:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:40 smithi115 ceph-mon[35364]: pgmap v1740: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1877.647219 secs 2024-07-26T08:30:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:40 smithi181 ceph-mon[35869]: pgmap v1740: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1877.647219 secs 2024-07-26T08:30:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:40 smithi097 ceph-mon[32231]: pgmap v1740: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1877.647219 secs 2024-07-26T08:30:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:42 smithi115 ceph-mon[35364]: pgmap v1741: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:30:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:42 smithi181 ceph-mon[35869]: pgmap v1741: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:30:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:42 smithi097 ceph-mon[32231]: pgmap v1741: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:30:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:44 smithi115 ceph-mon[35364]: pgmap v1742: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:30:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:44 smithi181 ceph-mon[35869]: pgmap v1742: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:30:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:44 smithi097 ceph-mon[32231]: pgmap v1742: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:30:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1882.647362 secs 2024-07-26T08:30:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1882.647362 secs 2024-07-26T08:30:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1882.647362 secs 2024-07-26T08:30:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:46 smithi115 ceph-mon[35364]: pgmap v1743: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:46 smithi181 ceph-mon[35869]: pgmap v1743: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:46 smithi097 ceph-mon[32231]: pgmap v1743: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:30:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:48.155+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:48.157+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:30:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:48.157+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:30:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:48.157+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:30:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:30:48.157+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:30:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:48 smithi115 ceph-mon[35364]: pgmap v1744: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:48 smithi181 ceph-mon[35869]: pgmap v1744: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:48 smithi097 ceph-mon[32231]: pgmap v1744: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:30:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:50 smithi115 ceph-mon[35364]: pgmap v1745: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:30:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1887.647494 secs 2024-07-26T08:30:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:50 smithi181 ceph-mon[35869]: pgmap v1745: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:30:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1887.647494 secs 2024-07-26T08:30:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:50 smithi097 ceph-mon[32231]: pgmap v1745: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:30:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1887.647494 secs 2024-07-26T08:30:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:52 smithi115 ceph-mon[35364]: pgmap v1746: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:52 smithi181 ceph-mon[35869]: pgmap v1746: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:52 smithi097 ceph-mon[32231]: pgmap v1746: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:54 smithi115 ceph-mon[35364]: pgmap v1747: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:54 smithi181 ceph-mon[35869]: pgmap v1747: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:54 smithi097 ceph-mon[32231]: pgmap v1747: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:30:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1892.647630 secs 2024-07-26T08:30:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1892.647630 secs 2024-07-26T08:30:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1892.647630 secs 2024-07-26T08:30:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:56 smithi181 ceph-mon[35869]: pgmap v1748: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:30:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:56 smithi097 ceph-mon[32231]: pgmap v1748: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:30:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:56 smithi115 ceph-mon[35364]: pgmap v1748: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:30:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:30:58 smithi097 ceph-mon[32231]: pgmap v1749: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:30:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:30:58 smithi115 ceph-mon[35364]: pgmap v1749: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:30:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:30:58 smithi181 ceph-mon[35869]: pgmap v1749: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:31:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:00 smithi097 ceph-mon[32231]: pgmap v1750: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:31:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1897.647766 secs 2024-07-26T08:31:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:00 smithi115 ceph-mon[35364]: pgmap v1750: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:31:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1897.647766 secs 2024-07-26T08:31:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:00 smithi181 ceph-mon[35869]: pgmap v1750: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:31:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1897.647766 secs 2024-07-26T08:31:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:02 smithi097 ceph-mon[32231]: pgmap v1751: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:02 smithi115 ceph-mon[35364]: pgmap v1751: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:02 smithi181 ceph-mon[35869]: pgmap v1751: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:03.162 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:03.161+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:03.162+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:31:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:03.162+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:31:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:03.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:31:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:03.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:31:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:04 smithi115 ceph-mon[35364]: pgmap v1752: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:04 smithi181 ceph-mon[35869]: pgmap v1752: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:04 smithi097 ceph-mon[32231]: pgmap v1752: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1902.647906 secs 2024-07-26T08:31:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1902.647906 secs 2024-07-26T08:31:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1902.647906 secs 2024-07-26T08:31:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:06 smithi115 ceph-mon[35364]: pgmap v1753: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:06 smithi181 ceph-mon[35869]: pgmap v1753: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:06 smithi097 ceph-mon[32231]: pgmap v1753: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:07.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:07.209+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:31:07.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:07.209+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:31:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:07.209+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T08:31:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:07.210+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:31:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:07.210+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T08:31:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:07.210+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:31:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:07.210+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:31:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:07.210+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:31:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:08 smithi115 ceph-mon[35364]: pgmap v1754: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:08 smithi181 ceph-mon[35869]: pgmap v1754: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:08 smithi097 ceph-mon[32231]: pgmap v1754: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:10 smithi115 ceph-mon[35364]: pgmap v1755: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1907.648041 secs 2024-07-26T08:31:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:10 smithi181 ceph-mon[35869]: pgmap v1755: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1907.648041 secs 2024-07-26T08:31:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:10 smithi097 ceph-mon[32231]: pgmap v1755: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1907.648041 secs 2024-07-26T08:31:12.804 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:12 smithi097 ceph-mon[32231]: pgmap v1756: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:12 smithi115 ceph-mon[35364]: pgmap v1756: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:12 smithi181 ceph-mon[35869]: pgmap v1756: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:14 smithi115 ceph-mon[35364]: pgmap v1757: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:14 smithi181 ceph-mon[35869]: pgmap v1757: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:14 smithi097 ceph-mon[32231]: pgmap v1757: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1912.648187 secs 2024-07-26T08:31:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:15 smithi115 ceph-mon[35364]: pgmap v1758: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1912.648187 secs 2024-07-26T08:31:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:15 smithi181 ceph-mon[35869]: pgmap v1758: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1912.648187 secs 2024-07-26T08:31:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:15 smithi097 ceph-mon[32231]: pgmap v1758: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:18.171+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:18.173+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:31:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:18.173+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:31:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:18.173+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:31:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:18.173+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:31:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:17 smithi115 ceph-mon[35364]: pgmap v1759: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:17 smithi181 ceph-mon[35869]: pgmap v1759: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:17 smithi097 ceph-mon[32231]: pgmap v1759: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:19 smithi115 ceph-mon[35364]: pgmap v1760: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:19 smithi181 ceph-mon[35869]: pgmap v1760: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:19 smithi097 ceph-mon[32231]: pgmap v1760: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1917.648333 secs 2024-07-26T08:31:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1917.648333 secs 2024-07-26T08:31:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1917.648333 secs 2024-07-26T08:31:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:21 smithi115 ceph-mon[35364]: pgmap v1761: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:21 smithi181 ceph-mon[35869]: pgmap v1761: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:21 smithi097 ceph-mon[32231]: pgmap v1761: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:23 smithi115 ceph-mon[35364]: pgmap v1762: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:23 smithi181 ceph-mon[35869]: pgmap v1762: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:23 smithi097 ceph-mon[32231]: pgmap v1762: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1922.648429 secs 2024-07-26T08:31:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1922.648429 secs 2024-07-26T08:31:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1922.648429 secs 2024-07-26T08:31:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:26 smithi115 ceph-mon[35364]: pgmap v1763: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:26 smithi115 ceph-mon[35364]: pgmap v1764: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:31:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:26 smithi115 ceph-mon[35364]: pgmap v1765: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:31:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:26 smithi181 ceph-mon[35869]: pgmap v1763: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:26 smithi181 ceph-mon[35869]: pgmap v1764: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:31:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:26 smithi181 ceph-mon[35869]: pgmap v1765: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:31:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:26 smithi097 ceph-mon[32231]: pgmap v1763: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:26 smithi097 ceph-mon[32231]: pgmap v1764: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:31:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:26 smithi097 ceph-mon[32231]: pgmap v1765: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:31:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:31:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:28 smithi115 ceph-mon[35364]: pgmap v1766: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T08:31:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:28 smithi181 ceph-mon[35869]: pgmap v1766: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T08:31:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:28 smithi097 ceph-mon[32231]: pgmap v1766: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T08:31:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:30 smithi115 ceph-mon[35364]: pgmap v1767: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:31:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1927.648524 secs 2024-07-26T08:31:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:30 smithi181 ceph-mon[35869]: pgmap v1767: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:31:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1927.648524 secs 2024-07-26T08:31:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:30 smithi097 ceph-mon[32231]: pgmap v1767: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:31:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1927.648524 secs 2024-07-26T08:31:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:32 smithi115 ceph-mon[35364]: pgmap v1768: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:31:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:32 smithi181 ceph-mon[35869]: pgmap v1768: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:31:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:32 smithi097 ceph-mon[32231]: pgmap v1768: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:31:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:33.177+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:33.179+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:31:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:33.179+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:31:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:33.179+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:31:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:33.179+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:31:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:34 smithi115 ceph-mon[35364]: pgmap v1769: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T08:31:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:34 smithi181 ceph-mon[35869]: pgmap v1769: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T08:31:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:34 smithi097 ceph-mon[32231]: pgmap v1769: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T08:31:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1932.648626 secs 2024-07-26T08:31:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1932.648626 secs 2024-07-26T08:31:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1932.648626 secs 2024-07-26T08:31:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:36 smithi115 ceph-mon[35364]: pgmap v1770: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:31:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:36 smithi181 ceph-mon[35869]: pgmap v1770: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:31:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:36 smithi097 ceph-mon[32231]: pgmap v1770: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:31:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:38 smithi115 ceph-mon[35364]: pgmap v1771: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:38 smithi181 ceph-mon[35869]: pgmap v1771: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:38 smithi097 ceph-mon[32231]: pgmap v1771: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:40 smithi115 ceph-mon[35364]: pgmap v1772: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1937.648769 secs 2024-07-26T08:31:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:40 smithi181 ceph-mon[35869]: pgmap v1772: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1937.648769 secs 2024-07-26T08:31:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:40 smithi097 ceph-mon[32231]: pgmap v1772: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1937.648769 secs 2024-07-26T08:31:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:42 smithi115 ceph-mon[35364]: pgmap v1773: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:42 smithi181 ceph-mon[35869]: pgmap v1773: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:42 smithi097 ceph-mon[32231]: pgmap v1773: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:43 smithi115 ceph-mon[35364]: pgmap v1774: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:43 smithi181 ceph-mon[35869]: pgmap v1774: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:43 smithi097 ceph-mon[32231]: pgmap v1774: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1942.648909 secs 2024-07-26T08:31:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1942.648909 secs 2024-07-26T08:31:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1942.648909 secs 2024-07-26T08:31:46.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:46 smithi115 ceph-mon[35364]: pgmap v1775: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:46.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:46 smithi181 ceph-mon[35869]: pgmap v1775: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:46 smithi097 ceph-mon[32231]: pgmap v1775: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:48.188+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:48.189+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:31:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:48.189+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:31:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:48.189+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:31:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:31:48.190+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:31:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:48 smithi115 ceph-mon[35364]: pgmap v1776: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:48 smithi181 ceph-mon[35869]: pgmap v1776: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:48 smithi097 ceph-mon[32231]: pgmap v1776: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:50 smithi115 ceph-mon[35364]: pgmap v1777: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1947.649066 secs 2024-07-26T08:31:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:50 smithi181 ceph-mon[35869]: pgmap v1777: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1947.649066 secs 2024-07-26T08:31:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:50 smithi097 ceph-mon[32231]: pgmap v1777: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1947.649066 secs 2024-07-26T08:31:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:52 smithi115 ceph-mon[35364]: pgmap v1778: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:52 smithi181 ceph-mon[35869]: pgmap v1778: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:52 smithi097 ceph-mon[32231]: pgmap v1778: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:31:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:54 smithi115 ceph-mon[35364]: pgmap v1779: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:54 smithi181 ceph-mon[35869]: pgmap v1779: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:54 smithi097 ceph-mon[32231]: pgmap v1779: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1952.649209 secs 2024-07-26T08:31:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1952.649209 secs 2024-07-26T08:31:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1952.649209 secs 2024-07-26T08:31:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:56 smithi115 ceph-mon[35364]: pgmap v1780: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:56 smithi181 ceph-mon[35869]: pgmap v1780: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:56 smithi097 ceph-mon[32231]: pgmap v1780: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:31:58.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:31:58 smithi115 ceph-mon[35364]: pgmap v1781: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:31:58 smithi181 ceph-mon[35869]: pgmap v1781: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:31:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:31:58 smithi097 ceph-mon[32231]: pgmap v1781: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:00 smithi115 ceph-mon[35364]: pgmap v1782: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1957.649356 secs 2024-07-26T08:32:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:00 smithi181 ceph-mon[35869]: pgmap v1782: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1957.649356 secs 2024-07-26T08:32:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:00 smithi097 ceph-mon[32231]: pgmap v1782: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1957.649356 secs 2024-07-26T08:32:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:01 smithi115 ceph-mon[35364]: pgmap v1783: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:01 smithi181 ceph-mon[35869]: pgmap v1783: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:01 smithi097 ceph-mon[32231]: pgmap v1783: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:03.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:03.193+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:03.194+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:32:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:03.194+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:32:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:03.194+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:32:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:03.194+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:32:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:04 smithi115 ceph-mon[35364]: pgmap v1784: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:04 smithi181 ceph-mon[35869]: pgmap v1784: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:04 smithi097 ceph-mon[32231]: pgmap v1784: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1962.649487 secs 2024-07-26T08:32:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1962.649487 secs 2024-07-26T08:32:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1962.649487 secs 2024-07-26T08:32:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:06 smithi115 ceph-mon[35364]: pgmap v1785: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:06 smithi181 ceph-mon[35869]: pgmap v1785: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:06 smithi097 ceph-mon[32231]: pgmap v1785: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:08 smithi115 ceph-mon[35364]: pgmap v1786: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:08 smithi181 ceph-mon[35869]: pgmap v1786: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:08 smithi097 ceph-mon[32231]: pgmap v1786: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:10 smithi115 ceph-mon[35364]: pgmap v1787: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1967.649638 secs 2024-07-26T08:32:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:10 smithi181 ceph-mon[35869]: pgmap v1787: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1967.649638 secs 2024-07-26T08:32:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:10 smithi097 ceph-mon[32231]: pgmap v1787: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1967.649638 secs 2024-07-26T08:32:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:12 smithi115 ceph-mon[35364]: pgmap v1788: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:12 smithi181 ceph-mon[35869]: pgmap v1788: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:12 smithi097 ceph-mon[32231]: pgmap v1788: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:14 smithi115 ceph-mon[35364]: pgmap v1789: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:14 smithi181 ceph-mon[35869]: pgmap v1789: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:14 smithi097 ceph-mon[32231]: pgmap v1789: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1972.649730 secs 2024-07-26T08:32:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:15 smithi181 ceph-mon[35869]: pgmap v1790: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1972.649730 secs 2024-07-26T08:32:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:15 smithi097 ceph-mon[32231]: pgmap v1790: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1972.649730 secs 2024-07-26T08:32:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:15 smithi115 ceph-mon[35364]: pgmap v1790: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:18.204 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:18.202+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:18.203+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:32:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:18.203+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:32:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:18.203+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:32:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:18.204+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:32:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:18 smithi115 ceph-mon[35364]: pgmap v1791: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:18 smithi181 ceph-mon[35869]: pgmap v1791: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:18 smithi097 ceph-mon[32231]: pgmap v1791: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:20 smithi115 ceph-mon[35364]: pgmap v1792: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1977.649912 secs 2024-07-26T08:32:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:20 smithi181 ceph-mon[35869]: pgmap v1792: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1977.649912 secs 2024-07-26T08:32:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:20 smithi097 ceph-mon[32231]: pgmap v1792: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1977.649912 secs 2024-07-26T08:32:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:22 smithi115 ceph-mon[35364]: pgmap v1793: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:22 smithi181 ceph-mon[35869]: pgmap v1793: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:22 smithi097 ceph-mon[32231]: pgmap v1793: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:24 smithi115 ceph-mon[35364]: pgmap v1794: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:24 smithi181 ceph-mon[35869]: pgmap v1794: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:24 smithi097 ceph-mon[32231]: pgmap v1794: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1982.650078 secs 2024-07-26T08:32:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1982.650078 secs 2024-07-26T08:32:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1982.650078 secs 2024-07-26T08:32:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:26 smithi115 ceph-mon[35364]: pgmap v1795: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:26 smithi181 ceph-mon[35869]: pgmap v1795: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:26 smithi097 ceph-mon[32231]: pgmap v1795: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:32:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:32:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:32:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:28 smithi115 ceph-mon[35364]: pgmap v1796: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:32:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:28 smithi181 ceph-mon[35869]: pgmap v1796: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:32:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:28 smithi097 ceph-mon[32231]: pgmap v1796: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:32:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:29 smithi115 ceph-mon[35364]: pgmap v1797: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:32:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:29 smithi181 ceph-mon[35869]: pgmap v1797: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:32:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:29 smithi097 ceph-mon[32231]: pgmap v1797: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T08:32:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1987.650212 secs 2024-07-26T08:32:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1987.650212 secs 2024-07-26T08:32:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1987.650212 secs 2024-07-26T08:32:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:31 smithi115 ceph-mon[35364]: pgmap v1798: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:32:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:31 smithi181 ceph-mon[35869]: pgmap v1798: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:32:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:31 smithi097 ceph-mon[32231]: pgmap v1798: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:32:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:33.208+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:33.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:32:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:33.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:32:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:33.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:32:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:33.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:32:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:34 smithi115 ceph-mon[35364]: pgmap v1799: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:32:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:34 smithi181 ceph-mon[35869]: pgmap v1799: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:32:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:34 smithi097 ceph-mon[32231]: pgmap v1799: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:32:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1992.650334 secs 2024-07-26T08:32:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1992.650334 secs 2024-07-26T08:32:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1992.650334 secs 2024-07-26T08:32:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:36 smithi115 ceph-mon[35364]: pgmap v1800: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:32:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:36 smithi181 ceph-mon[35869]: pgmap v1800: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:32:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:36 smithi097 ceph-mon[32231]: pgmap v1800: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:32:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:38 smithi115 ceph-mon[35364]: pgmap v1801: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:32:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:38 smithi181 ceph-mon[35869]: pgmap v1801: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:32:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:38 smithi097 ceph-mon[32231]: pgmap v1801: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:32:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:40 smithi115 ceph-mon[35364]: pgmap v1802: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:32:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 1997.650476 secs 2024-07-26T08:32:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:40 smithi181 ceph-mon[35869]: pgmap v1802: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:32:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 1997.650476 secs 2024-07-26T08:32:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:40 smithi097 ceph-mon[32231]: pgmap v1802: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:32:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 1997.650476 secs 2024-07-26T08:32:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:42 smithi115 ceph-mon[35364]: pgmap v1803: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:32:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:42 smithi181 ceph-mon[35869]: pgmap v1803: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:32:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:42 smithi097 ceph-mon[32231]: pgmap v1803: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:32:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:44 smithi115 ceph-mon[35364]: pgmap v1804: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:44 smithi181 ceph-mon[35869]: pgmap v1804: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:44 smithi097 ceph-mon[32231]: pgmap v1804: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2002.650566 secs 2024-07-26T08:32:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2002.650566 secs 2024-07-26T08:32:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2002.650566 secs 2024-07-26T08:32:46.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:46 smithi115 ceph-mon[35364]: pgmap v1805: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:46.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:46 smithi181 ceph-mon[35869]: pgmap v1805: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:46 smithi097 ceph-mon[32231]: pgmap v1805: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:48.214+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:48.214+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:32:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:48.214+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:32:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:48.214+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:32:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:32:48.214+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:32:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:48 smithi115 ceph-mon[35364]: pgmap v1806: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:48 smithi181 ceph-mon[35869]: pgmap v1806: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:48 smithi097 ceph-mon[32231]: pgmap v1806: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:50 smithi115 ceph-mon[35364]: pgmap v1807: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:32:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2007.650710 secs 2024-07-26T08:32:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:50 smithi181 ceph-mon[35869]: pgmap v1807: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:32:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2007.650710 secs 2024-07-26T08:32:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:50 smithi097 ceph-mon[32231]: pgmap v1807: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:32:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2007.650710 secs 2024-07-26T08:32:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:52 smithi115 ceph-mon[35364]: pgmap v1808: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:32:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:52 smithi181 ceph-mon[35869]: pgmap v1808: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:32:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:52 smithi097 ceph-mon[32231]: pgmap v1808: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:32:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:54 smithi115 ceph-mon[35364]: pgmap v1809: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:54 smithi181 ceph-mon[35869]: pgmap v1809: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:54 smithi097 ceph-mon[32231]: pgmap v1809: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:32:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2012.650852 secs 2024-07-26T08:32:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2012.650852 secs 2024-07-26T08:32:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2012.650852 secs 2024-07-26T08:32:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:56 smithi115 ceph-mon[35364]: pgmap v1810: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:56 smithi181 ceph-mon[35869]: pgmap v1810: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:56 smithi097 ceph-mon[32231]: pgmap v1810: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:32:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:32:58 smithi115 ceph-mon[35364]: pgmap v1811: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:32:58 smithi181 ceph-mon[35869]: pgmap v1811: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:32:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:32:58 smithi097 ceph-mon[32231]: pgmap v1811: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:00 smithi115 ceph-mon[35364]: pgmap v1812: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2017.650941 secs 2024-07-26T08:33:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:00 smithi181 ceph-mon[35869]: pgmap v1812: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2017.650941 secs 2024-07-26T08:33:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:00 smithi097 ceph-mon[32231]: pgmap v1812: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2017.650941 secs 2024-07-26T08:33:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:02 smithi115 ceph-mon[35364]: pgmap v1813: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:02 smithi181 ceph-mon[35869]: pgmap v1813: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:02 smithi097 ceph-mon[32231]: pgmap v1813: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:03.218+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:03.219+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:33:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:03.219+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:33:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:03.219+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:33:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:03.219+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:33:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:04 smithi115 ceph-mon[35364]: pgmap v1814: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:04 smithi181 ceph-mon[35869]: pgmap v1814: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:04 smithi097 ceph-mon[32231]: pgmap v1814: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2022.651097 secs 2024-07-26T08:33:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2022.651097 secs 2024-07-26T08:33:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2022.651097 secs 2024-07-26T08:33:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:06 smithi115 ceph-mon[35364]: pgmap v1815: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:06 smithi181 ceph-mon[35869]: pgmap v1815: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:06 smithi097 ceph-mon[32231]: pgmap v1815: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:08 smithi115 ceph-mon[35364]: pgmap v1816: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:08 smithi181 ceph-mon[35869]: pgmap v1816: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:08 smithi097 ceph-mon[32231]: pgmap v1816: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:10 smithi115 ceph-mon[35364]: pgmap v1817: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2027.651224 secs 2024-07-26T08:33:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:10 smithi181 ceph-mon[35869]: pgmap v1817: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2027.651224 secs 2024-07-26T08:33:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:10 smithi097 ceph-mon[32231]: pgmap v1817: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2027.651224 secs 2024-07-26T08:33:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:12 smithi115 ceph-mon[35364]: pgmap v1818: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:12 smithi181 ceph-mon[35869]: pgmap v1818: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:12 smithi097 ceph-mon[32231]: pgmap v1818: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:14 smithi115 ceph-mon[35364]: pgmap v1819: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:14 smithi181 ceph-mon[35869]: pgmap v1819: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:14 smithi097 ceph-mon[32231]: pgmap v1819: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2032.651378 secs 2024-07-26T08:33:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2032.651378 secs 2024-07-26T08:33:15.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2032.651378 secs 2024-07-26T08:33:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:16 smithi181 ceph-mon[35869]: pgmap v1820: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:16.528 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:16 smithi097 ceph-mon[32231]: pgmap v1820: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:16.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:16 smithi115 ceph-mon[35364]: pgmap v1820: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:18.229+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:18.230+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:33:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:18.230+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:33:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:18.230+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:33:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:18.230+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:33:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:18 smithi097 ceph-mon[32231]: pgmap v1821: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:18 smithi115 ceph-mon[35364]: pgmap v1821: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:18 smithi181 ceph-mon[35869]: pgmap v1821: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:20 smithi097 ceph-mon[32231]: pgmap v1822: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2037.651507 secs 2024-07-26T08:33:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:20 smithi115 ceph-mon[35364]: pgmap v1822: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2037.651507 secs 2024-07-26T08:33:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:20 smithi181 ceph-mon[35869]: pgmap v1822: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2037.651507 secs 2024-07-26T08:33:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:22 smithi097 ceph-mon[32231]: pgmap v1823: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:22 smithi115 ceph-mon[35364]: pgmap v1823: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:22 smithi181 ceph-mon[35869]: pgmap v1823: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:24.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:24 smithi115 ceph-mon[35364]: pgmap v1824: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:24 smithi181 ceph-mon[35869]: pgmap v1824: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:24 smithi097 ceph-mon[32231]: pgmap v1824: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2042.651646 secs 2024-07-26T08:33:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2042.651646 secs 2024-07-26T08:33:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2042.651646 secs 2024-07-26T08:33:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:26 smithi115 ceph-mon[35364]: pgmap v1825: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:26 smithi181 ceph-mon[35869]: pgmap v1825: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:26 smithi097 ceph-mon[32231]: pgmap v1825: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:28 smithi115 ceph-mon[35364]: pgmap v1826: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:28 smithi181 ceph-mon[35869]: pgmap v1826: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:28 smithi097 ceph-mon[32231]: pgmap v1826: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:29 smithi115 ceph-mon[35364]: pgmap v1827: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:33:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:33:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:29 smithi181 ceph-mon[35869]: pgmap v1827: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:33:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:33:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:29 smithi097 ceph-mon[32231]: pgmap v1827: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:33:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:33:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2047.651778 secs 2024-07-26T08:33:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2047.651778 secs 2024-07-26T08:33:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2047.651778 secs 2024-07-26T08:33:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:31 smithi115 ceph-mon[35364]: pgmap v1828: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:33:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:31 smithi181 ceph-mon[35869]: pgmap v1828: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:33:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:31 smithi097 ceph-mon[32231]: pgmap v1828: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:33:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:33.234+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:33.235+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:33:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:33.235+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:33:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:33.235+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:33:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:33.235+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:33:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:33 smithi115 ceph-mon[35364]: pgmap v1829: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:33:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:33 smithi181 ceph-mon[35869]: pgmap v1829: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:33:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:33 smithi097 ceph-mon[32231]: pgmap v1829: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:33:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:35 smithi115 ceph-mon[35364]: pgmap v1830: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:33:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2052.651881 secs 2024-07-26T08:33:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:35 smithi181 ceph-mon[35869]: pgmap v1830: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:33:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2052.651881 secs 2024-07-26T08:33:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:35 smithi097 ceph-mon[32231]: pgmap v1830: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:33:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2052.651881 secs 2024-07-26T08:33:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:37 smithi115 ceph-mon[35364]: pgmap v1831: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:33:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:37 smithi181 ceph-mon[35869]: pgmap v1831: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:33:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:37 smithi097 ceph-mon[32231]: pgmap v1831: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:33:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:39 smithi115 ceph-mon[35364]: pgmap v1832: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:33:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:39 smithi181 ceph-mon[35869]: pgmap v1832: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:33:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:39 smithi097 ceph-mon[32231]: pgmap v1832: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:33:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2057.652027 secs 2024-07-26T08:33:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2057.652027 secs 2024-07-26T08:33:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2057.652027 secs 2024-07-26T08:33:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:41 smithi181 ceph-mon[35869]: pgmap v1833: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:41 smithi097 ceph-mon[32231]: pgmap v1833: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:41 smithi115 ceph-mon[35364]: pgmap v1833: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:43 smithi097 ceph-mon[32231]: pgmap v1834: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:43 smithi115 ceph-mon[35364]: pgmap v1834: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:43 smithi181 ceph-mon[35869]: pgmap v1834: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:45 smithi097 ceph-mon[32231]: pgmap v1835: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2062.652174 secs 2024-07-26T08:33:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:45 smithi115 ceph-mon[35364]: pgmap v1835: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2062.652174 secs 2024-07-26T08:33:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:45 smithi181 ceph-mon[35869]: pgmap v1835: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2062.652174 secs 2024-07-26T08:33:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:47 smithi097 ceph-mon[32231]: pgmap v1836: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:47.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:47 smithi115 ceph-mon[35364]: pgmap v1836: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:47.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:47 smithi181 ceph-mon[35869]: pgmap v1836: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:48.245+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:48.246+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:33:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:48.246+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:33:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:48.246+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:33:48.248 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:33:48.246+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:33:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:49 smithi115 ceph-mon[35364]: pgmap v1837: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:49 smithi181 ceph-mon[35869]: pgmap v1837: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:49 smithi097 ceph-mon[32231]: pgmap v1837: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2067.652315 secs 2024-07-26T08:33:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2067.652315 secs 2024-07-26T08:33:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2067.652315 secs 2024-07-26T08:33:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:51 smithi115 ceph-mon[35364]: pgmap v1838: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:51.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:51 smithi181 ceph-mon[35869]: pgmap v1838: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:51 smithi097 ceph-mon[32231]: pgmap v1838: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:33:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:53 smithi115 ceph-mon[35364]: pgmap v1839: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:53 smithi181 ceph-mon[35869]: pgmap v1839: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:53 smithi097 ceph-mon[32231]: pgmap v1839: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:33:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:55 smithi115 ceph-mon[35364]: pgmap v1840: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2072.652458 secs 2024-07-26T08:33:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:55 smithi181 ceph-mon[35869]: pgmap v1840: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2072.652458 secs 2024-07-26T08:33:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:55 smithi097 ceph-mon[32231]: pgmap v1840: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:33:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2072.652458 secs 2024-07-26T08:33:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:57 smithi115 ceph-mon[35364]: pgmap v1841: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:57 smithi181 ceph-mon[35869]: pgmap v1841: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:57 smithi097 ceph-mon[32231]: pgmap v1841: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:33:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:33:59 smithi115 ceph-mon[35364]: pgmap v1842: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:33:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:33:59 smithi181 ceph-mon[35869]: pgmap v1842: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:33:59 smithi097 ceph-mon[32231]: pgmap v1842: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2077.652595 secs 2024-07-26T08:34:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2077.652595 secs 2024-07-26T08:34:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2077.652595 secs 2024-07-26T08:34:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:01 smithi115 ceph-mon[35364]: pgmap v1843: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:01 smithi181 ceph-mon[35869]: pgmap v1843: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:01 smithi097 ceph-mon[32231]: pgmap v1843: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:03.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:03.249+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:03.250+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:34:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:03.250+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:34:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:03.250+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:34:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:03.250+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:34:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:03 smithi115 ceph-mon[35364]: pgmap v1844: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:03 smithi181 ceph-mon[35869]: pgmap v1844: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:03 smithi097 ceph-mon[32231]: pgmap v1844: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:05 smithi115 ceph-mon[35364]: pgmap v1845: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2082.652750 secs 2024-07-26T08:34:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:05 smithi181 ceph-mon[35869]: pgmap v1845: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2082.652750 secs 2024-07-26T08:34:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:05 smithi097 ceph-mon[32231]: pgmap v1845: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2082.652750 secs 2024-07-26T08:34:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:07.240+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:34:07.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:07.240+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:34:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:07.241+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:34:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:07.241+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:34:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:07 smithi115 ceph-mon[35364]: pgmap v1846: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:07 smithi181 ceph-mon[35869]: pgmap v1846: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:07 smithi097 ceph-mon[32231]: pgmap v1846: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:09 smithi115 ceph-mon[35364]: pgmap v1847: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:09 smithi181 ceph-mon[35869]: pgmap v1847: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:09 smithi097 ceph-mon[32231]: pgmap v1847: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2087.652913 secs 2024-07-26T08:34:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2087.652913 secs 2024-07-26T08:34:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2087.652913 secs 2024-07-26T08:34:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:11 smithi115 ceph-mon[35364]: pgmap v1848: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:11 smithi181 ceph-mon[35869]: pgmap v1848: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:11 smithi097 ceph-mon[32231]: pgmap v1848: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:13 smithi115 ceph-mon[35364]: pgmap v1849: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:34:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:13 smithi181 ceph-mon[35869]: pgmap v1849: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:34:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:13 smithi097 ceph-mon[32231]: pgmap v1849: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:34:15.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:15 smithi115 ceph-mon[35364]: pgmap v1850: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:15.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2092.653055 secs 2024-07-26T08:34:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:15 smithi181 ceph-mon[35869]: pgmap v1850: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2092.653055 secs 2024-07-26T08:34:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:15 smithi097 ceph-mon[32231]: pgmap v1850: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2092.653055 secs 2024-07-26T08:34:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:17 smithi115 ceph-mon[35364]: pgmap v1851: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:17 smithi181 ceph-mon[35869]: pgmap v1851: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:17 smithi097 ceph-mon[32231]: pgmap v1851: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:18.259+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:18.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:18.260+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:34:18.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:18.260+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:34:18.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:18.260+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:34:18.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:18.260+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:34:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:19 smithi115 ceph-mon[35364]: pgmap v1852: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:19 smithi181 ceph-mon[35869]: pgmap v1852: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:19 smithi097 ceph-mon[32231]: pgmap v1852: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2097.653217 secs 2024-07-26T08:34:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2097.653217 secs 2024-07-26T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2097.653217 secs 2024-07-26T08:34:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:21 smithi115 ceph-mon[35364]: pgmap v1853: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:21 smithi181 ceph-mon[35869]: pgmap v1853: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:21 smithi097 ceph-mon[32231]: pgmap v1853: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:23 smithi115 ceph-mon[35364]: pgmap v1854: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:23 smithi181 ceph-mon[35869]: pgmap v1854: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:23 smithi097 ceph-mon[32231]: pgmap v1854: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:25 smithi115 ceph-mon[35364]: pgmap v1855: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2102.653289 secs 2024-07-26T08:34:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:25 smithi181 ceph-mon[35869]: pgmap v1855: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2102.653289 secs 2024-07-26T08:34:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:25 smithi097 ceph-mon[32231]: pgmap v1855: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:34:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2102.653289 secs 2024-07-26T08:34:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:27 smithi115 ceph-mon[35364]: pgmap v1856: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:27 smithi181 ceph-mon[35869]: pgmap v1856: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:27 smithi097 ceph-mon[32231]: pgmap v1856: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:29 smithi115 ceph-mon[35364]: pgmap v1857: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:29 smithi181 ceph-mon[35869]: pgmap v1857: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:29 smithi097 ceph-mon[32231]: pgmap v1857: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2107.653388 secs 2024-07-26T08:34:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:34:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:34:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:34:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2107.653388 secs 2024-07-26T08:34:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:34:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:34:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:34:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2107.653388 secs 2024-07-26T08:34:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:34:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:34:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:34:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:31 smithi115 ceph-mon[35364]: pgmap v1858: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:34:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:31 smithi115 ceph-mon[35364]: pgmap v1859: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:31 smithi181 ceph-mon[35869]: pgmap v1858: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:34:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:31 smithi181 ceph-mon[35869]: pgmap v1859: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:31 smithi097 ceph-mon[32231]: pgmap v1858: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:34:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:31 smithi097 ceph-mon[32231]: pgmap v1859: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:32 smithi115 ceph-mon[35364]: pgmap v1860: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:32 smithi181 ceph-mon[35869]: pgmap v1860: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:32 smithi097 ceph-mon[32231]: pgmap v1860: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:33.264+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:33.265+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:34:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:33.265+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:34:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:33.265+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:34:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:33.265+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:34:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:34 smithi115 ceph-mon[35364]: pgmap v1861: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:34 smithi181 ceph-mon[35869]: pgmap v1861: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:34 smithi097 ceph-mon[32231]: pgmap v1861: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:34:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2112.653531 secs 2024-07-26T08:34:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2112.653531 secs 2024-07-26T08:34:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2112.653531 secs 2024-07-26T08:34:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:36 smithi115 ceph-mon[35364]: pgmap v1862: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:36 smithi181 ceph-mon[35869]: pgmap v1862: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:36 smithi097 ceph-mon[32231]: pgmap v1862: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:38 smithi115 ceph-mon[35364]: pgmap v1863: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:38 smithi181 ceph-mon[35869]: pgmap v1863: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:38 smithi097 ceph-mon[32231]: pgmap v1863: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2117.653666 secs 2024-07-26T08:34:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:40 smithi115 ceph-mon[35364]: pgmap v1864: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:34:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2117.653666 secs 2024-07-26T08:34:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:40 smithi181 ceph-mon[35869]: pgmap v1864: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:34:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2117.653666 secs 2024-07-26T08:34:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:40 smithi097 ceph-mon[32231]: pgmap v1864: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:34:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:42 smithi115 ceph-mon[35364]: pgmap v1865: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:42 smithi181 ceph-mon[35869]: pgmap v1865: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:42 smithi097 ceph-mon[32231]: pgmap v1865: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:44 smithi115 ceph-mon[35364]: pgmap v1866: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:44 smithi181 ceph-mon[35869]: pgmap v1866: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:44 smithi097 ceph-mon[32231]: pgmap v1866: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2122.653819 secs 2024-07-26T08:34:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2122.653819 secs 2024-07-26T08:34:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2122.653819 secs 2024-07-26T08:34:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:46 smithi115 ceph-mon[35364]: pgmap v1867: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:46 smithi181 ceph-mon[35869]: pgmap v1867: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:46 smithi097 ceph-mon[32231]: pgmap v1867: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:48.273+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:48.273+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:34:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:48.273+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:34:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:48.273+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:34:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:34:48.273+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:34:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:48 smithi115 ceph-mon[35364]: pgmap v1868: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:48 smithi181 ceph-mon[35869]: pgmap v1868: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:48 smithi097 ceph-mon[32231]: pgmap v1868: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2127.653964 secs 2024-07-26T08:34:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:50 smithi115 ceph-mon[35364]: pgmap v1869: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2127.653964 secs 2024-07-26T08:34:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:50 smithi181 ceph-mon[35869]: pgmap v1869: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2127.653964 secs 2024-07-26T08:34:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:50 smithi097 ceph-mon[32231]: pgmap v1869: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:52 smithi115 ceph-mon[35364]: pgmap v1870: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:52 smithi181 ceph-mon[35869]: pgmap v1870: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:52 smithi097 ceph-mon[32231]: pgmap v1870: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:34:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:54 smithi181 ceph-mon[35869]: pgmap v1871: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:54 smithi097 ceph-mon[32231]: pgmap v1871: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:54 smithi115 ceph-mon[35364]: pgmap v1871: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:34:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2132.654105 secs 2024-07-26T08:34:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2132.654105 secs 2024-07-26T08:34:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2132.654105 secs 2024-07-26T08:34:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:56 smithi097 ceph-mon[32231]: pgmap v1872: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:56 smithi115 ceph-mon[35364]: pgmap v1872: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:56 smithi181 ceph-mon[35869]: pgmap v1872: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:34:58 smithi097 ceph-mon[32231]: pgmap v1873: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:34:58 smithi115 ceph-mon[35364]: pgmap v1873: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:34:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:34:58 smithi181 ceph-mon[35869]: pgmap v1873: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2137.654298 secs 2024-07-26T08:35:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:01 smithi115 ceph-mon[35364]: pgmap v1874: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2137.654298 secs 2024-07-26T08:35:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:01 smithi181 ceph-mon[35869]: pgmap v1874: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2137.654298 secs 2024-07-26T08:35:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:01 smithi097 ceph-mon[32231]: pgmap v1874: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:03.277+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:03.278+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:35:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:03.278+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:35:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:03.278+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:35:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:03.278+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:35:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:03 smithi115 ceph-mon[35364]: pgmap v1875: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:03 smithi181 ceph-mon[35869]: pgmap v1875: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:03 smithi097 ceph-mon[32231]: pgmap v1875: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:05 smithi115 ceph-mon[35364]: pgmap v1876: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:05 smithi181 ceph-mon[35869]: pgmap v1876: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:05 smithi097 ceph-mon[32231]: pgmap v1876: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2142.654431 secs 2024-07-26T08:35:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2142.654431 secs 2024-07-26T08:35:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2142.654431 secs 2024-07-26T08:35:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:07 smithi115 ceph-mon[35364]: pgmap v1877: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:07 smithi181 ceph-mon[35869]: pgmap v1877: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:07 smithi097 ceph-mon[32231]: pgmap v1877: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:09 smithi115 ceph-mon[35364]: pgmap v1878: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:09 smithi181 ceph-mon[35869]: pgmap v1878: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:09 smithi097 ceph-mon[32231]: pgmap v1878: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2147.654593 secs 2024-07-26T08:35:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2147.654593 secs 2024-07-26T08:35:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2147.654593 secs 2024-07-26T08:35:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:11 smithi115 ceph-mon[35364]: pgmap v1879: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:11 smithi181 ceph-mon[35869]: pgmap v1879: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:11 smithi097 ceph-mon[32231]: pgmap v1879: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:13 smithi115 ceph-mon[35364]: pgmap v1880: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:13 smithi181 ceph-mon[35869]: pgmap v1880: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:13 smithi097 ceph-mon[32231]: pgmap v1880: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:15 smithi115 ceph-mon[35364]: pgmap v1881: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2152.654677 secs 2024-07-26T08:35:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:15 smithi181 ceph-mon[35869]: pgmap v1881: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2152.654677 secs 2024-07-26T08:35:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:15 smithi097 ceph-mon[32231]: pgmap v1881: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2152.654677 secs 2024-07-26T08:35:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:17 smithi115 ceph-mon[35364]: pgmap v1882: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:17 smithi181 ceph-mon[35869]: pgmap v1882: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:17 smithi097 ceph-mon[32231]: pgmap v1882: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:18.287+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:18.289+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:35:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:18.289+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:35:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:18.289+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:35:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:18.289+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:35:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:19 smithi115 ceph-mon[35364]: pgmap v1883: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:19 smithi181 ceph-mon[35869]: pgmap v1883: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:19 smithi097 ceph-mon[32231]: pgmap v1883: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2157.654821 secs 2024-07-26T08:35:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2157.654821 secs 2024-07-26T08:35:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2157.654821 secs 2024-07-26T08:35:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:21 smithi115 ceph-mon[35364]: pgmap v1884: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:21 smithi181 ceph-mon[35869]: pgmap v1884: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:21 smithi097 ceph-mon[32231]: pgmap v1884: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:23 smithi115 ceph-mon[35364]: pgmap v1885: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:23 smithi181 ceph-mon[35869]: pgmap v1885: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:23 smithi097 ceph-mon[32231]: pgmap v1885: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:25 smithi115 ceph-mon[35364]: pgmap v1886: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2162.654957 secs 2024-07-26T08:35:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:25 smithi181 ceph-mon[35869]: pgmap v1886: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2162.654957 secs 2024-07-26T08:35:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:25 smithi097 ceph-mon[32231]: pgmap v1886: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2162.654957 secs 2024-07-26T08:35:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:27 smithi115 ceph-mon[35364]: pgmap v1887: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:27 smithi181 ceph-mon[35869]: pgmap v1887: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:27 smithi097 ceph-mon[32231]: pgmap v1887: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:29 smithi115 ceph-mon[35364]: pgmap v1888: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:29 smithi181 ceph-mon[35869]: pgmap v1888: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:29 smithi097 ceph-mon[32231]: pgmap v1888: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2167.655071 secs 2024-07-26T08:35:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2167.655071 secs 2024-07-26T08:35:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2167.655071 secs 2024-07-26T08:35:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:31 smithi115 ceph-mon[35364]: pgmap v1889: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:35:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:31 smithi181 ceph-mon[35869]: pgmap v1889: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:35:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:31 smithi097 ceph-mon[32231]: pgmap v1889: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:35:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:32 smithi115 ceph-mon[35364]: pgmap v1890: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:35:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:32 smithi115 ceph-mon[35364]: pgmap v1891: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:35:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:35:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:32 smithi181 ceph-mon[35869]: pgmap v1890: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:35:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:32 smithi181 ceph-mon[35869]: pgmap v1891: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:35:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:35:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:32 smithi097 ceph-mon[32231]: pgmap v1890: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:35:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:32 smithi097 ceph-mon[32231]: pgmap v1891: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:35:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:35:33.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:33.293+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:33.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:33.294+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:35:33.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:33.294+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:35:33.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:33.295+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:35:33.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:33.295+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:35:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:34 smithi115 ceph-mon[35364]: pgmap v1892: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:35:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:34 smithi181 ceph-mon[35869]: pgmap v1892: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:35:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:34 smithi097 ceph-mon[32231]: pgmap v1892: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:35:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2172.655211 secs 2024-07-26T08:35:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2172.655211 secs 2024-07-26T08:35:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2172.655211 secs 2024-07-26T08:35:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:36 smithi115 ceph-mon[35364]: pgmap v1893: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:35:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:36 smithi181 ceph-mon[35869]: pgmap v1893: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:35:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:36 smithi097 ceph-mon[32231]: pgmap v1893: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:35:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:38 smithi115 ceph-mon[35364]: pgmap v1894: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:35:38.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:38 smithi181 ceph-mon[35869]: pgmap v1894: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:35:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:38 smithi097 ceph-mon[32231]: pgmap v1894: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:35:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:40 smithi115 ceph-mon[35364]: pgmap v1895: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:35:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2177.655369 secs 2024-07-26T08:35:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:40 smithi181 ceph-mon[35869]: pgmap v1895: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:35:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2177.655369 secs 2024-07-26T08:35:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:40 smithi097 ceph-mon[32231]: pgmap v1895: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:35:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2177.655369 secs 2024-07-26T08:35:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:42 smithi115 ceph-mon[35364]: pgmap v1896: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:35:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:42 smithi181 ceph-mon[35869]: pgmap v1896: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:35:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:42 smithi097 ceph-mon[32231]: pgmap v1896: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:35:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:44 smithi115 ceph-mon[35364]: pgmap v1897: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:44 smithi181 ceph-mon[35869]: pgmap v1897: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:44 smithi097 ceph-mon[32231]: pgmap v1897: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:35:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2182.655510 secs 2024-07-26T08:35:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2182.655510 secs 2024-07-26T08:35:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2182.655510 secs 2024-07-26T08:35:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:46 smithi115 ceph-mon[35364]: pgmap v1898: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:35:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:46 smithi181 ceph-mon[35869]: pgmap v1898: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:35:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:46 smithi097 ceph-mon[32231]: pgmap v1898: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:35:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:48.304+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:48.306+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:35:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:48.306+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:35:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:48.306+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:35:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:35:48.306+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:35:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:48 smithi115 ceph-mon[35364]: pgmap v1899: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:35:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:48 smithi181 ceph-mon[35869]: pgmap v1899: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:35:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:48 smithi097 ceph-mon[32231]: pgmap v1899: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:35:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:50 smithi115 ceph-mon[35364]: pgmap v1900: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:35:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2187.655651 secs 2024-07-26T08:35:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:50 smithi181 ceph-mon[35869]: pgmap v1900: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:35:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2187.655651 secs 2024-07-26T08:35:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:50 smithi097 ceph-mon[32231]: pgmap v1900: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:35:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2187.655651 secs 2024-07-26T08:35:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:52 smithi115 ceph-mon[35364]: pgmap v1901: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:35:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:52 smithi181 ceph-mon[35869]: pgmap v1901: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:35:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:52 smithi097 ceph-mon[32231]: pgmap v1901: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:35:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:54 smithi115 ceph-mon[35364]: pgmap v1902: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:35:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:54 smithi181 ceph-mon[35869]: pgmap v1902: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:35:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:54 smithi097 ceph-mon[32231]: pgmap v1902: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:35:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2192.655748 secs 2024-07-26T08:35:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2192.655748 secs 2024-07-26T08:35:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2192.655748 secs 2024-07-26T08:35:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:56 smithi115 ceph-mon[35364]: pgmap v1903: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:35:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:56 smithi181 ceph-mon[35869]: pgmap v1903: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:35:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:56 smithi097 ceph-mon[32231]: pgmap v1903: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:35:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:35:58 smithi115 ceph-mon[35364]: pgmap v1904: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:35:58 smithi181 ceph-mon[35869]: pgmap v1904: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:35:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:35:58 smithi097 ceph-mon[32231]: pgmap v1904: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:00 smithi115 ceph-mon[35364]: pgmap v1905: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2197.655886 secs 2024-07-26T08:36:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:00 smithi181 ceph-mon[35869]: pgmap v1905: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2197.655886 secs 2024-07-26T08:36:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:00 smithi097 ceph-mon[32231]: pgmap v1905: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2197.655886 secs 2024-07-26T08:36:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:02 smithi115 ceph-mon[35364]: pgmap v1906: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:02 smithi181 ceph-mon[35869]: pgmap v1906: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:02 smithi097 ceph-mon[32231]: pgmap v1906: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:03.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:03.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:03.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:03.312+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:36:03.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:03.312+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:36:03.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:03.312+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:36:03.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:03.312+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:36:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:04 smithi115 ceph-mon[35364]: pgmap v1907: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:04 smithi181 ceph-mon[35869]: pgmap v1907: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:04 smithi097 ceph-mon[32231]: pgmap v1907: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2202.656054 secs 2024-07-26T08:36:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2202.656054 secs 2024-07-26T08:36:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2202.656054 secs 2024-07-26T08:36:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:06 smithi115 ceph-mon[35364]: pgmap v1908: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:06 smithi181 ceph-mon[35869]: pgmap v1908: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:06 smithi097 ceph-mon[32231]: pgmap v1908: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:08 smithi115 ceph-mon[35364]: pgmap v1909: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:08 smithi181 ceph-mon[35869]: pgmap v1909: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:08 smithi097 ceph-mon[32231]: pgmap v1909: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:10 smithi115 ceph-mon[35364]: pgmap v1910: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2207.656199 secs 2024-07-26T08:36:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:10 smithi181 ceph-mon[35869]: pgmap v1910: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2207.656199 secs 2024-07-26T08:36:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:10 smithi097 ceph-mon[32231]: pgmap v1910: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2207.656199 secs 2024-07-26T08:36:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:12 smithi181 ceph-mon[35869]: pgmap v1911: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:12 smithi097 ceph-mon[32231]: pgmap v1911: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:12 smithi115 ceph-mon[35364]: pgmap v1911: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:14 smithi097 ceph-mon[32231]: pgmap v1912: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:14 smithi115 ceph-mon[35364]: pgmap v1912: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:14 smithi181 ceph-mon[35869]: pgmap v1912: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2212.656342 secs 2024-07-26T08:36:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2212.656342 secs 2024-07-26T08:36:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2212.656342 secs 2024-07-26T08:36:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:16 smithi097 ceph-mon[32231]: pgmap v1913: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:36:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:16 smithi115 ceph-mon[35364]: pgmap v1913: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:36:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:16 smithi181 ceph-mon[35869]: pgmap v1913: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:36:18.323 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:18.321+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:18.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:18.322+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:36:18.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:18.323+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:36:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:18.323+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:36:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:18.323+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:36:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:18 smithi097 ceph-mon[32231]: pgmap v1914: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:36:18.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:18 smithi115 ceph-mon[35364]: pgmap v1914: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:36:18.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:18 smithi181 ceph-mon[35869]: pgmap v1914: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:36:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:20 smithi115 ceph-mon[35364]: pgmap v1915: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:36:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2217.656490 secs 2024-07-26T08:36:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:20 smithi181 ceph-mon[35869]: pgmap v1915: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:36:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2217.656490 secs 2024-07-26T08:36:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:20 smithi097 ceph-mon[32231]: pgmap v1915: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:36:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2217.656490 secs 2024-07-26T08:36:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:22 smithi115 ceph-mon[35364]: pgmap v1916: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:36:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:22 smithi181 ceph-mon[35869]: pgmap v1916: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:36:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:22 smithi097 ceph-mon[32231]: pgmap v1916: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:36:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:24 smithi115 ceph-mon[35364]: pgmap v1917: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:24 smithi181 ceph-mon[35869]: pgmap v1917: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:24 smithi097 ceph-mon[32231]: pgmap v1917: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2222.656634 secs 2024-07-26T08:36:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2222.656634 secs 2024-07-26T08:36:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2222.656634 secs 2024-07-26T08:36:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:26 smithi115 ceph-mon[35364]: pgmap v1918: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:26 smithi181 ceph-mon[35869]: pgmap v1918: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:26 smithi097 ceph-mon[32231]: pgmap v1918: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:28 smithi115 ceph-mon[35364]: pgmap v1919: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:28 smithi181 ceph-mon[35869]: pgmap v1919: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:28 smithi097 ceph-mon[32231]: pgmap v1919: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:30 smithi115 ceph-mon[35364]: pgmap v1920: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2227.656779 secs 2024-07-26T08:36:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:30 smithi181 ceph-mon[35869]: pgmap v1920: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2227.656779 secs 2024-07-26T08:36:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:30 smithi097 ceph-mon[32231]: pgmap v1920: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:36:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2227.656779 secs 2024-07-26T08:36:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:32 smithi115 ceph-mon[35364]: pgmap v1921: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:32 smithi181 ceph-mon[35869]: pgmap v1921: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:32 smithi097 ceph-mon[32231]: pgmap v1921: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:33.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:33.328+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:33.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:33.328+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:36:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:33.328+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:36:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:33.328+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:36:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:33.328+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:36:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:33 smithi115 ceph-mon[35364]: pgmap v1922: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:36:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:33 smithi115 ceph-mon[35364]: pgmap v1923: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:36:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:33 smithi181 ceph-mon[35869]: pgmap v1922: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:36:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:33 smithi181 ceph-mon[35869]: pgmap v1923: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:36:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:33 smithi097 ceph-mon[32231]: pgmap v1922: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:36:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:33 smithi097 ceph-mon[32231]: pgmap v1923: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:36:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:34 smithi115 ceph-mon[35364]: pgmap v1924: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:34 smithi181 ceph-mon[35869]: pgmap v1924: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:34 smithi097 ceph-mon[32231]: pgmap v1924: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2232.656863 secs 2024-07-26T08:36:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2232.656863 secs 2024-07-26T08:36:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2232.656863 secs 2024-07-26T08:36:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:36 smithi115 ceph-mon[35364]: pgmap v1925: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:36 smithi181 ceph-mon[35869]: pgmap v1925: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:36 smithi097 ceph-mon[32231]: pgmap v1925: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:36:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:38 smithi115 ceph-mon[35364]: pgmap v1926: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:38 smithi181 ceph-mon[35869]: pgmap v1926: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:38 smithi097 ceph-mon[32231]: pgmap v1926: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:36:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2237.657003 secs 2024-07-26T08:36:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2237.657003 secs 2024-07-26T08:36:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2237.657003 secs 2024-07-26T08:36:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:41 smithi115 ceph-mon[35364]: pgmap v1927: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:36:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:41 smithi181 ceph-mon[35869]: pgmap v1927: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:36:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:41 smithi097 ceph-mon[32231]: pgmap v1927: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:36:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:42 smithi115 ceph-mon[35364]: pgmap v1928: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:42 smithi181 ceph-mon[35869]: pgmap v1928: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:42 smithi097 ceph-mon[32231]: pgmap v1928: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:36:45.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:44 smithi097 ceph-mon[32231]: pgmap v1929: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:44 smithi115 ceph-mon[35364]: pgmap v1929: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:44 smithi181 ceph-mon[35869]: pgmap v1929: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2242.657142 secs 2024-07-26T08:36:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2242.657142 secs 2024-07-26T08:36:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2242.657142 secs 2024-07-26T08:36:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:46 smithi115 ceph-mon[35364]: pgmap v1930: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:46 smithi181 ceph-mon[35869]: pgmap v1930: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:46 smithi097 ceph-mon[32231]: pgmap v1930: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:48.337+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:48.338+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:36:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:48.338+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:36:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:48.338+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:36:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:36:48.338+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:36:49.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:48 smithi115 ceph-mon[35364]: pgmap v1931: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:49.035 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:48 smithi181 ceph-mon[35869]: pgmap v1931: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:48 smithi097 ceph-mon[32231]: pgmap v1931: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2247.657282 secs 2024-07-26T08:36:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2247.657282 secs 2024-07-26T08:36:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2247.657282 secs 2024-07-26T08:36:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:51 smithi115 ceph-mon[35364]: pgmap v1932: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:51 smithi181 ceph-mon[35869]: pgmap v1932: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:51 smithi097 ceph-mon[32231]: pgmap v1932: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:52 smithi115 ceph-mon[35364]: pgmap v1933: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:52 smithi181 ceph-mon[35869]: pgmap v1933: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:52 smithi097 ceph-mon[32231]: pgmap v1933: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:36:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:54 smithi115 ceph-mon[35364]: pgmap v1934: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:54 smithi181 ceph-mon[35869]: pgmap v1934: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:54 smithi097 ceph-mon[32231]: pgmap v1934: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2252.657369 secs 2024-07-26T08:36:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2252.657369 secs 2024-07-26T08:36:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2252.657369 secs 2024-07-26T08:36:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:56 smithi115 ceph-mon[35364]: pgmap v1935: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:56 smithi181 ceph-mon[35869]: pgmap v1935: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:56 smithi097 ceph-mon[32231]: pgmap v1935: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:36:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:36:58 smithi115 ceph-mon[35364]: pgmap v1936: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:36:58 smithi181 ceph-mon[35869]: pgmap v1936: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:36:58 smithi097 ceph-mon[32231]: pgmap v1936: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2257.657471 secs 2024-07-26T08:37:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2257.657471 secs 2024-07-26T08:37:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2257.657471 secs 2024-07-26T08:37:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:01 smithi115 ceph-mon[35364]: pgmap v1937: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:01 smithi181 ceph-mon[35869]: pgmap v1937: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:01 smithi097 ceph-mon[32231]: pgmap v1937: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:02 smithi115 ceph-mon[35364]: pgmap v1938: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:02 smithi181 ceph-mon[35869]: pgmap v1938: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:02 smithi097 ceph-mon[32231]: pgmap v1938: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:03.342+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:03.343+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:37:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:03.343+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:37:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:03.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:37:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:03.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:37:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:04 smithi115 ceph-mon[35364]: pgmap v1939: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:04 smithi181 ceph-mon[35869]: pgmap v1939: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:04 smithi097 ceph-mon[32231]: pgmap v1939: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2262.657554 secs 2024-07-26T08:37:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2262.657554 secs 2024-07-26T08:37:06.255 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2262.657554 secs 2024-07-26T08:37:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:06 smithi115 ceph-mon[35364]: pgmap v1940: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:06 smithi181 ceph-mon[35869]: pgmap v1940: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:06 smithi097 ceph-mon[32231]: pgmap v1940: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:07.272+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:37:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:07.272+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:37:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:07.273+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:37:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:07.273+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:37:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:08 smithi115 ceph-mon[35364]: pgmap v1941: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:08 smithi181 ceph-mon[35869]: pgmap v1941: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:08 smithi097 ceph-mon[32231]: pgmap v1941: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2267.657691 secs 2024-07-26T08:37:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2267.657691 secs 2024-07-26T08:37:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2267.657691 secs 2024-07-26T08:37:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:11 smithi115 ceph-mon[35364]: pgmap v1942: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:11 smithi181 ceph-mon[35869]: pgmap v1942: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:11 smithi097 ceph-mon[32231]: pgmap v1942: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:12 smithi115 ceph-mon[35364]: pgmap v1943: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:12 smithi181 ceph-mon[35869]: pgmap v1943: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:12 smithi097 ceph-mon[32231]: pgmap v1943: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:15.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:14 smithi097 ceph-mon[32231]: pgmap v1944: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:14 smithi115 ceph-mon[35364]: pgmap v1944: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:14 smithi181 ceph-mon[35869]: pgmap v1944: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2272.657829 secs 2024-07-26T08:37:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2272.657829 secs 2024-07-26T08:37:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2272.657829 secs 2024-07-26T08:37:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:16 smithi115 ceph-mon[35364]: pgmap v1945: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:16 smithi181 ceph-mon[35869]: pgmap v1945: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:16 smithi097 ceph-mon[32231]: pgmap v1945: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:18.353+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:18.354+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:37:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:18.354+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:37:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:18.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:37:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:18.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:37:19.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:18 smithi115 ceph-mon[35364]: pgmap v1946: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:19.034 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:18 smithi181 ceph-mon[35869]: pgmap v1946: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:18 smithi097 ceph-mon[32231]: pgmap v1946: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2277.657966 secs 2024-07-26T08:37:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2277.657966 secs 2024-07-26T08:37:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2277.657966 secs 2024-07-26T08:37:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:21 smithi115 ceph-mon[35364]: pgmap v1947: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:21 smithi181 ceph-mon[35869]: pgmap v1947: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:21 smithi097 ceph-mon[32231]: pgmap v1947: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:22 smithi115 ceph-mon[35364]: pgmap v1948: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:22 smithi181 ceph-mon[35869]: pgmap v1948: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:22 smithi097 ceph-mon[32231]: pgmap v1948: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:24 smithi115 ceph-mon[35364]: pgmap v1949: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:24 smithi181 ceph-mon[35869]: pgmap v1949: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:24 smithi097 ceph-mon[32231]: pgmap v1949: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2282.658107 secs 2024-07-26T08:37:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2282.658107 secs 2024-07-26T08:37:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2282.658107 secs 2024-07-26T08:37:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:26 smithi115 ceph-mon[35364]: pgmap v1950: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:26 smithi181 ceph-mon[35869]: pgmap v1950: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:26 smithi097 ceph-mon[32231]: pgmap v1950: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:37:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:28 smithi115 ceph-mon[35364]: pgmap v1951: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:28 smithi181 ceph-mon[35869]: pgmap v1951: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:28 smithi097 ceph-mon[32231]: pgmap v1951: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2287.658265 secs 2024-07-26T08:37:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2287.658265 secs 2024-07-26T08:37:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2287.658265 secs 2024-07-26T08:37:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:31 smithi115 ceph-mon[35364]: pgmap v1952: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:31 smithi181 ceph-mon[35869]: pgmap v1952: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:31 smithi097 ceph-mon[32231]: pgmap v1952: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:32 smithi115 ceph-mon[35364]: pgmap v1953: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:32 smithi181 ceph-mon[35869]: pgmap v1953: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:32 smithi097 ceph-mon[32231]: pgmap v1953: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:33.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:33.357+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:33.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:33.358+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:37:33.360 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:33.358+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:37:33.360 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:33.358+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:37:33.360 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:33.358+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:37:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:35.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:35.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: pgmap v1954: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:37:35.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: pgmap v1954: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:37:35.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:35.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:37:35.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:37:35.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:35.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:35.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: pgmap v1954: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:37:35.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:37:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2292.658435 secs 2024-07-26T08:37:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2292.658435 secs 2024-07-26T08:37:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2292.658435 secs 2024-07-26T08:37:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:36 smithi115 ceph-mon[35364]: pgmap v1955: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:37:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:36 smithi181 ceph-mon[35869]: pgmap v1955: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:37:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:36 smithi097 ceph-mon[32231]: pgmap v1955: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:37:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:39 smithi115 ceph-mon[35364]: pgmap v1956: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:39 smithi181 ceph-mon[35869]: pgmap v1956: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:39 smithi097 ceph-mon[32231]: pgmap v1956: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2297.658599 secs 2024-07-26T08:37:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2297.658599 secs 2024-07-26T08:37:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2297.658599 secs 2024-07-26T08:37:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:41 smithi115 ceph-mon[35364]: pgmap v1957: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:41 smithi181 ceph-mon[35869]: pgmap v1957: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:41 smithi097 ceph-mon[32231]: pgmap v1957: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:43 smithi115 ceph-mon[35364]: pgmap v1958: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:43 smithi181 ceph-mon[35869]: pgmap v1958: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:43 smithi097 ceph-mon[32231]: pgmap v1958: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:45 smithi115 ceph-mon[35364]: pgmap v1959: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2302.658744 secs 2024-07-26T08:37:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:45 smithi181 ceph-mon[35869]: pgmap v1959: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2302.658744 secs 2024-07-26T08:37:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:45 smithi097 ceph-mon[32231]: pgmap v1959: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2302.658744 secs 2024-07-26T08:37:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:47 smithi115 ceph-mon[35364]: pgmap v1960: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:47 smithi181 ceph-mon[35869]: pgmap v1960: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:47 smithi097 ceph-mon[32231]: pgmap v1960: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:48.364+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:48.365+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:37:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:48.365+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:37:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:48.365+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:37:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:37:48.365+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:37:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:49 smithi115 ceph-mon[35364]: pgmap v1961: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:49 smithi181 ceph-mon[35869]: pgmap v1961: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:49 smithi097 ceph-mon[32231]: pgmap v1961: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:37:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2307.658843 secs 2024-07-26T08:37:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2307.658843 secs 2024-07-26T08:37:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2307.658843 secs 2024-07-26T08:37:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:51 smithi115 ceph-mon[35364]: pgmap v1962: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:51 smithi181 ceph-mon[35869]: pgmap v1962: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:51 smithi097 ceph-mon[32231]: pgmap v1962: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:53 smithi115 ceph-mon[35364]: pgmap v1963: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:53 smithi181 ceph-mon[35869]: pgmap v1963: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:53 smithi097 ceph-mon[32231]: pgmap v1963: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:55 smithi115 ceph-mon[35364]: pgmap v1964: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2312.658937 secs 2024-07-26T08:37:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:55 smithi181 ceph-mon[35869]: pgmap v1964: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2312.658937 secs 2024-07-26T08:37:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:55 smithi097 ceph-mon[32231]: pgmap v1964: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2312.658937 secs 2024-07-26T08:37:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:57 smithi115 ceph-mon[35364]: pgmap v1965: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:57 smithi181 ceph-mon[35869]: pgmap v1965: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:57 smithi097 ceph-mon[32231]: pgmap v1965: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:37:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:37:59 smithi115 ceph-mon[35364]: pgmap v1966: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:37:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:37:59 smithi181 ceph-mon[35869]: pgmap v1966: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:37:59 smithi097 ceph-mon[32231]: pgmap v1966: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2317.659079 secs 2024-07-26T08:38:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2317.659079 secs 2024-07-26T08:38:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2317.659079 secs 2024-07-26T08:38:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:01 smithi181 ceph-mon[35869]: pgmap v1967: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:01 smithi097 ceph-mon[32231]: pgmap v1967: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:01 smithi115 ceph-mon[35364]: pgmap v1967: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:02 smithi097 ceph-mon[32231]: pgmap v1968: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:02 smithi115 ceph-mon[35364]: pgmap v1968: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:02 smithi181 ceph-mon[35869]: pgmap v1968: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:03.369+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:03.370+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:38:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:03.370+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:38:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:03.370+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:38:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:03.370+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:38:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:05 smithi115 ceph-mon[35364]: pgmap v1969: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2322.659170 secs 2024-07-26T08:38:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:05 smithi181 ceph-mon[35869]: pgmap v1969: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2322.659170 secs 2024-07-26T08:38:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:05 smithi097 ceph-mon[32231]: pgmap v1969: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2322.659170 secs 2024-07-26T08:38:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:07 smithi115 ceph-mon[35364]: pgmap v1970: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:07 smithi181 ceph-mon[35869]: pgmap v1970: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:07 smithi097 ceph-mon[32231]: pgmap v1970: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:09 smithi115 ceph-mon[35364]: pgmap v1971: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:09 smithi181 ceph-mon[35869]: pgmap v1971: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:09 smithi097 ceph-mon[32231]: pgmap v1971: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2327.659344 secs 2024-07-26T08:38:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2327.659344 secs 2024-07-26T08:38:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2327.659344 secs 2024-07-26T08:38:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:11 smithi115 ceph-mon[35364]: pgmap v1972: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:11 smithi181 ceph-mon[35869]: pgmap v1972: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:11 smithi097 ceph-mon[32231]: pgmap v1972: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:13 smithi115 ceph-mon[35364]: pgmap v1973: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:13 smithi181 ceph-mon[35869]: pgmap v1973: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:13 smithi097 ceph-mon[32231]: pgmap v1973: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:15 smithi115 ceph-mon[35364]: pgmap v1974: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2332.659490 secs 2024-07-26T08:38:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:15 smithi181 ceph-mon[35869]: pgmap v1974: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2332.659490 secs 2024-07-26T08:38:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:15 smithi097 ceph-mon[32231]: pgmap v1974: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2332.659490 secs 2024-07-26T08:38:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:17 smithi115 ceph-mon[35364]: pgmap v1975: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:17 smithi181 ceph-mon[35869]: pgmap v1975: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:17 smithi097 ceph-mon[32231]: pgmap v1975: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:18.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:18.380+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:38:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:18.380+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:38:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:18.380+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:38:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:18.380+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:38:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:19 smithi115 ceph-mon[35364]: pgmap v1976: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:19 smithi181 ceph-mon[35869]: pgmap v1976: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:19 smithi097 ceph-mon[32231]: pgmap v1976: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2337.659648 secs 2024-07-26T08:38:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2337.659648 secs 2024-07-26T08:38:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2337.659648 secs 2024-07-26T08:38:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:21 smithi115 ceph-mon[35364]: pgmap v1977: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:21 smithi181 ceph-mon[35869]: pgmap v1977: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:21 smithi097 ceph-mon[32231]: pgmap v1977: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:23 smithi115 ceph-mon[35364]: pgmap v1978: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:23 smithi181 ceph-mon[35869]: pgmap v1978: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:23 smithi097 ceph-mon[32231]: pgmap v1978: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:25 smithi181 ceph-mon[35869]: pgmap v1979: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2342.659795 secs 2024-07-26T08:38:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:25 smithi097 ceph-mon[32231]: pgmap v1979: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2342.659795 secs 2024-07-26T08:38:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:25 smithi115 ceph-mon[35364]: pgmap v1979: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2342.659795 secs 2024-07-26T08:38:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:27 smithi181 ceph-mon[35869]: pgmap v1980: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:27 smithi097 ceph-mon[32231]: pgmap v1980: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:27 smithi115 ceph-mon[35364]: pgmap v1980: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:28 smithi097 ceph-mon[32231]: pgmap v1981: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:28 smithi115 ceph-mon[35364]: pgmap v1981: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:28 smithi181 ceph-mon[35869]: pgmap v1981: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2347.659956 secs 2024-07-26T08:38:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2347.659956 secs 2024-07-26T08:38:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2347.659956 secs 2024-07-26T08:38:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:31 smithi115 ceph-mon[35364]: pgmap v1982: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:31 smithi181 ceph-mon[35869]: pgmap v1982: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:31 smithi097 ceph-mon[32231]: pgmap v1982: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:33.383+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:33.385+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:38:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:33.385+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:38:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:33.385+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:38:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:33.385+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:38:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:33 smithi115 ceph-mon[35364]: pgmap v1983: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:33 smithi181 ceph-mon[35869]: pgmap v1983: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:33 smithi097 ceph-mon[32231]: pgmap v1983: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:35 smithi115 ceph-mon[35364]: pgmap v1984: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2352.660113 secs 2024-07-26T08:38:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:35 smithi181 ceph-mon[35869]: pgmap v1984: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2352.660113 secs 2024-07-26T08:38:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:35 smithi097 ceph-mon[32231]: pgmap v1984: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2352.660113 secs 2024-07-26T08:38:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: pgmap v1985: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: pgmap v1986: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:37.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: pgmap v1985: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: pgmap v1986: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:38:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:38.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:38.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: pgmap v1985: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:38:38.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: pgmap v1986: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:38.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:38:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:39 smithi115 ceph-mon[35364]: pgmap v1987: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:39 smithi181 ceph-mon[35869]: pgmap v1987: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:39 smithi097 ceph-mon[32231]: pgmap v1987: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2357.660203 secs 2024-07-26T08:38:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2357.660203 secs 2024-07-26T08:38:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2357.660203 secs 2024-07-26T08:38:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:41 smithi115 ceph-mon[35364]: pgmap v1988: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:41 smithi181 ceph-mon[35869]: pgmap v1988: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:41 smithi097 ceph-mon[32231]: pgmap v1988: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:43 smithi115 ceph-mon[35364]: pgmap v1989: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:38:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:43 smithi181 ceph-mon[35869]: pgmap v1989: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:38:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:43 smithi097 ceph-mon[32231]: pgmap v1989: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:38:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:45 smithi115 ceph-mon[35364]: pgmap v1990: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:38:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2362.660295 secs 2024-07-26T08:38:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:45 smithi181 ceph-mon[35869]: pgmap v1990: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:38:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2362.660295 secs 2024-07-26T08:38:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:45 smithi097 ceph-mon[32231]: pgmap v1990: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:38:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2362.660295 secs 2024-07-26T08:38:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:47 smithi115 ceph-mon[35364]: pgmap v1991: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:47 smithi181 ceph-mon[35869]: pgmap v1991: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:47 smithi097 ceph-mon[32231]: pgmap v1991: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:38:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:48.394+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:48.395+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:38:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:48.395+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:38:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:48.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:38:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:38:48.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:38:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:49 smithi115 ceph-mon[35364]: pgmap v1992: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:49 smithi181 ceph-mon[35869]: pgmap v1992: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:49 smithi097 ceph-mon[32231]: pgmap v1992: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2367.660430 secs 2024-07-26T08:38:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2367.660430 secs 2024-07-26T08:38:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2367.660430 secs 2024-07-26T08:38:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:51 smithi115 ceph-mon[35364]: pgmap v1993: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:51 smithi181 ceph-mon[35869]: pgmap v1993: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:51 smithi097 ceph-mon[32231]: pgmap v1993: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:53 smithi115 ceph-mon[35364]: pgmap v1994: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:53 smithi181 ceph-mon[35869]: pgmap v1994: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:53 smithi097 ceph-mon[32231]: pgmap v1994: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:55 smithi115 ceph-mon[35364]: pgmap v1995: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2372.660568 secs 2024-07-26T08:38:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:55 smithi181 ceph-mon[35869]: pgmap v1995: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2372.660568 secs 2024-07-26T08:38:56.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:55 smithi097 ceph-mon[32231]: pgmap v1995: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:56.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2372.660568 secs 2024-07-26T08:38:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:57 smithi115 ceph-mon[35364]: pgmap v1996: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:57 smithi181 ceph-mon[35869]: pgmap v1996: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:57 smithi097 ceph-mon[32231]: pgmap v1996: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:38:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:38:59 smithi115 ceph-mon[35364]: pgmap v1997: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:38:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:38:59 smithi181 ceph-mon[35869]: pgmap v1997: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:38:59 smithi097 ceph-mon[32231]: pgmap v1997: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2377.660648 secs 2024-07-26T08:39:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2377.660648 secs 2024-07-26T08:39:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2377.660648 secs 2024-07-26T08:39:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:01 smithi097 ceph-mon[32231]: pgmap v1998: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:01 smithi115 ceph-mon[35364]: pgmap v1998: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:01 smithi181 ceph-mon[35869]: pgmap v1998: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:02 smithi115 ceph-mon[35364]: pgmap v1999: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:02 smithi181 ceph-mon[35869]: pgmap v1999: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:02 smithi097 ceph-mon[32231]: pgmap v1999: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:03.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:03.399+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:03.400+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:39:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:03.400+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:39:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:03.400+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:39:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:03.401+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:39:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:05 smithi115 ceph-mon[35364]: pgmap v2000: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2382.660797 secs 2024-07-26T08:39:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:05 smithi181 ceph-mon[35869]: pgmap v2000: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2382.660797 secs 2024-07-26T08:39:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:05 smithi097 ceph-mon[32231]: pgmap v2000: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2382.660797 secs 2024-07-26T08:39:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:07 smithi115 ceph-mon[35364]: pgmap v2001: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:07 smithi181 ceph-mon[35869]: pgmap v2001: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:07 smithi097 ceph-mon[32231]: pgmap v2001: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:09 smithi115 ceph-mon[35364]: pgmap v2002: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:09 smithi181 ceph-mon[35869]: pgmap v2002: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:09 smithi097 ceph-mon[32231]: pgmap v2002: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2387.660931 secs 2024-07-26T08:39:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2387.660931 secs 2024-07-26T08:39:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2387.660931 secs 2024-07-26T08:39:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:11 smithi115 ceph-mon[35364]: pgmap v2003: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:11 smithi181 ceph-mon[35869]: pgmap v2003: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:11 smithi097 ceph-mon[32231]: pgmap v2003: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:13 smithi115 ceph-mon[35364]: pgmap v2004: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:13 smithi181 ceph-mon[35869]: pgmap v2004: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:13 smithi097 ceph-mon[32231]: pgmap v2004: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:15 smithi115 ceph-mon[35364]: pgmap v2005: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2392.661027 secs 2024-07-26T08:39:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:15 smithi181 ceph-mon[35869]: pgmap v2005: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2392.661027 secs 2024-07-26T08:39:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:15 smithi097 ceph-mon[32231]: pgmap v2005: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2392.661027 secs 2024-07-26T08:39:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:17 smithi115 ceph-mon[35364]: pgmap v2006: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:17 smithi181 ceph-mon[35869]: pgmap v2006: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:17 smithi097 ceph-mon[32231]: pgmap v2006: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:18.412 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:18.409+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:18.413 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:18.411+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:39:18.413 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:18.411+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:39:18.413 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:18.411+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:39:18.413 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:18.411+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:39:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:19 smithi115 ceph-mon[35364]: pgmap v2007: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:19 smithi181 ceph-mon[35869]: pgmap v2007: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:19 smithi097 ceph-mon[32231]: pgmap v2007: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2397.661157 secs 2024-07-26T08:39:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2397.661157 secs 2024-07-26T08:39:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2397.661157 secs 2024-07-26T08:39:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:21 smithi115 ceph-mon[35364]: pgmap v2008: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:21 smithi181 ceph-mon[35869]: pgmap v2008: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:21 smithi097 ceph-mon[32231]: pgmap v2008: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:22.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:22.296+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:39:22.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:22.296+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:39:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:22.297+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:39:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:22.297+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:39:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:23 smithi181 ceph-mon[35869]: pgmap v2009: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:23 smithi097 ceph-mon[32231]: pgmap v2009: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:23 smithi115 ceph-mon[35364]: pgmap v2009: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:25 smithi097 ceph-mon[32231]: pgmap v2010: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2402.661300 secs 2024-07-26T08:39:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:25 smithi115 ceph-mon[35364]: pgmap v2010: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2402.661300 secs 2024-07-26T08:39:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:25 smithi181 ceph-mon[35869]: pgmap v2010: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2402.661300 secs 2024-07-26T08:39:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:27 smithi097 ceph-mon[32231]: pgmap v2011: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:27 smithi115 ceph-mon[35364]: pgmap v2011: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:27 smithi181 ceph-mon[35869]: pgmap v2011: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:28 smithi115 ceph-mon[35364]: pgmap v2012: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:28 smithi181 ceph-mon[35869]: pgmap v2012: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:28 smithi097 ceph-mon[32231]: pgmap v2012: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2407.661430 secs 2024-07-26T08:39:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2407.661430 secs 2024-07-26T08:39:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2407.661430 secs 2024-07-26T08:39:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:31 smithi115 ceph-mon[35364]: pgmap v2013: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:31 smithi181 ceph-mon[35869]: pgmap v2013: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:31 smithi097 ceph-mon[32231]: pgmap v2013: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:33.416+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:33.417+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:39:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:33.417+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:39:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:33.417+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:39:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:33.417+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:39:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:33 smithi115 ceph-mon[35364]: pgmap v2014: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:33 smithi181 ceph-mon[35869]: pgmap v2014: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:33 smithi097 ceph-mon[32231]: pgmap v2014: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:35 smithi115 ceph-mon[35364]: pgmap v2015: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2412.661567 secs 2024-07-26T08:39:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:35 smithi181 ceph-mon[35869]: pgmap v2015: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2412.661567 secs 2024-07-26T08:39:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:35 smithi097 ceph-mon[32231]: pgmap v2015: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2412.661567 secs 2024-07-26T08:39:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:37 smithi115 ceph-mon[35364]: pgmap v2016: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:37 smithi181 ceph-mon[35869]: pgmap v2016: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:37 smithi097 ceph-mon[32231]: pgmap v2016: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:39:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:39 smithi115 ceph-mon[35364]: pgmap v2017: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:39 smithi115 ceph-mon[35364]: pgmap v2018: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:39 smithi181 ceph-mon[35869]: pgmap v2017: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:39 smithi181 ceph-mon[35869]: pgmap v2018: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:39 smithi097 ceph-mon[32231]: pgmap v2017: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:39 smithi097 ceph-mon[32231]: pgmap v2018: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2417.661704 secs 2024-07-26T08:39:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2417.661704 secs 2024-07-26T08:39:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2417.661704 secs 2024-07-26T08:39:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:41 smithi115 ceph-mon[35364]: pgmap v2019: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:41 smithi181 ceph-mon[35869]: pgmap v2019: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:41 smithi097 ceph-mon[32231]: pgmap v2019: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:43 smithi115 ceph-mon[35364]: pgmap v2020: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:43 smithi181 ceph-mon[35869]: pgmap v2020: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:43 smithi097 ceph-mon[32231]: pgmap v2020: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:45 smithi115 ceph-mon[35364]: pgmap v2021: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2422.661850 secs 2024-07-26T08:39:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:45 smithi181 ceph-mon[35869]: pgmap v2021: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2422.661850 secs 2024-07-26T08:39:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:45 smithi097 ceph-mon[32231]: pgmap v2021: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2422.661850 secs 2024-07-26T08:39:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:47 smithi181 ceph-mon[35869]: pgmap v2022: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:47 smithi097 ceph-mon[32231]: pgmap v2022: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:47 smithi115 ceph-mon[35364]: pgmap v2022: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:39:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:48.426+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:48.427+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:39:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:48.427+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:39:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:48.427+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:39:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:39:48.427+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:39:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:49 smithi181 ceph-mon[35869]: pgmap v2023: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:39:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:49 smithi097 ceph-mon[32231]: pgmap v2023: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:39:50.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:49 smithi115 ceph-mon[35364]: pgmap v2023: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:39:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2427.661989 secs 2024-07-26T08:39:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2427.661989 secs 2024-07-26T08:39:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2427.661989 secs 2024-07-26T08:39:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:51 smithi097 ceph-mon[32231]: pgmap v2024: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:51 smithi115 ceph-mon[35364]: pgmap v2024: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:51 smithi181 ceph-mon[35869]: pgmap v2024: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:52 smithi115 ceph-mon[35364]: pgmap v2025: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:52 smithi181 ceph-mon[35869]: pgmap v2025: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:52 smithi097 ceph-mon[32231]: pgmap v2025: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:55 smithi115 ceph-mon[35364]: pgmap v2026: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2432.662152 secs 2024-07-26T08:39:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:55 smithi181 ceph-mon[35869]: pgmap v2026: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2432.662152 secs 2024-07-26T08:39:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:55 smithi097 ceph-mon[32231]: pgmap v2026: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2432.662152 secs 2024-07-26T08:39:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:57 smithi115 ceph-mon[35364]: pgmap v2027: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:57 smithi181 ceph-mon[35869]: pgmap v2027: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:57 smithi097 ceph-mon[32231]: pgmap v2027: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:39:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:39:59 smithi115 ceph-mon[35364]: pgmap v2028: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:39:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:39:59 smithi181 ceph-mon[35869]: pgmap v2028: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:39:59 smithi097 ceph-mon[32231]: pgmap v2028: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2437.662299 secs 2024-07-26T08:40:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:40:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2437.662299 secs 2024-07-26T08:40:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:40:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2437.662299 secs 2024-07-26T08:40:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:40:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:01 smithi115 ceph-mon[35364]: pgmap v2029: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:01 smithi181 ceph-mon[35869]: pgmap v2029: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:01 smithi097 ceph-mon[32231]: pgmap v2029: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:03.432+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:03.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:03.432+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:40:03.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:03.432+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:40:03.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:03.432+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:40:03.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:03.432+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:40:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:03 smithi115 ceph-mon[35364]: pgmap v2030: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:03 smithi181 ceph-mon[35869]: pgmap v2030: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:03 smithi097 ceph-mon[32231]: pgmap v2030: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:05 smithi115 ceph-mon[35364]: pgmap v2031: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2442.662443 secs 2024-07-26T08:40:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:05 smithi181 ceph-mon[35869]: pgmap v2031: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2442.662443 secs 2024-07-26T08:40:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:05 smithi097 ceph-mon[32231]: pgmap v2031: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2442.662443 secs 2024-07-26T08:40:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:07.304+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:40:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:07.304+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:40:07.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:07.305+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:40:07.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:07.305+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:40:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:07 smithi115 ceph-mon[35364]: pgmap v2032: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:07 smithi181 ceph-mon[35869]: pgmap v2032: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:07 smithi097 ceph-mon[32231]: pgmap v2032: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:09 smithi115 ceph-mon[35364]: pgmap v2033: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:09 smithi181 ceph-mon[35869]: pgmap v2033: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:09 smithi097 ceph-mon[32231]: pgmap v2033: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2447.662587 secs 2024-07-26T08:40:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2447.662587 secs 2024-07-26T08:40:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2447.662587 secs 2024-07-26T08:40:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:11 smithi181 ceph-mon[35869]: pgmap v2034: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:11 smithi097 ceph-mon[32231]: pgmap v2034: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:11 smithi115 ceph-mon[35364]: pgmap v2034: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:13 smithi097 ceph-mon[32231]: pgmap v2035: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:13 smithi115 ceph-mon[35364]: pgmap v2035: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:13 smithi181 ceph-mon[35869]: pgmap v2035: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:15 smithi097 ceph-mon[32231]: pgmap v2036: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2452.662729 secs 2024-07-26T08:40:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:15 smithi115 ceph-mon[35364]: pgmap v2036: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2452.662729 secs 2024-07-26T08:40:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:15 smithi181 ceph-mon[35869]: pgmap v2036: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2452.662729 secs 2024-07-26T08:40:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:17 smithi097 ceph-mon[32231]: pgmap v2037: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:17 smithi115 ceph-mon[35364]: pgmap v2037: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:17 smithi181 ceph-mon[35869]: pgmap v2037: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:18.436+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:18.437+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:40:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:18.437+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:40:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:18.437+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:40:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:18.437+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:40:19.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:18 smithi181 ceph-mon[35869]: pgmap v2038: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:18 smithi115 ceph-mon[35364]: pgmap v2038: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:18 smithi097 ceph-mon[32231]: pgmap v2038: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2457.662886 secs 2024-07-26T08:40:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2457.662886 secs 2024-07-26T08:40:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2457.662886 secs 2024-07-26T08:40:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:21 smithi115 ceph-mon[35364]: pgmap v2039: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:21 smithi181 ceph-mon[35869]: pgmap v2039: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:21 smithi097 ceph-mon[32231]: pgmap v2039: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:23 smithi115 ceph-mon[35364]: pgmap v2040: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:23 smithi181 ceph-mon[35869]: pgmap v2040: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:23 smithi097 ceph-mon[32231]: pgmap v2040: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:25 smithi115 ceph-mon[35364]: pgmap v2041: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2462.663029 secs 2024-07-26T08:40:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:25 smithi181 ceph-mon[35869]: pgmap v2041: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2462.663029 secs 2024-07-26T08:40:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:25 smithi097 ceph-mon[32231]: pgmap v2041: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2462.663029 secs 2024-07-26T08:40:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:27 smithi115 ceph-mon[35364]: pgmap v2042: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:27 smithi181 ceph-mon[35869]: pgmap v2042: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:27 smithi097 ceph-mon[32231]: pgmap v2042: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:29 smithi115 ceph-mon[35364]: pgmap v2043: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:29 smithi181 ceph-mon[35869]: pgmap v2043: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:29 smithi097 ceph-mon[32231]: pgmap v2043: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2467.663183 secs 2024-07-26T08:40:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2467.663183 secs 2024-07-26T08:40:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2467.663183 secs 2024-07-26T08:40:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:31 smithi115 ceph-mon[35364]: pgmap v2044: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:31 smithi181 ceph-mon[35869]: pgmap v2044: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:31 smithi097 ceph-mon[32231]: pgmap v2044: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:33.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:33.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:33.443+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:40:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:33.443+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:40:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:33.443+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:40:33.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:33.443+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:40:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:33 smithi115 ceph-mon[35364]: pgmap v2045: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:33 smithi181 ceph-mon[35869]: pgmap v2045: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:33 smithi097 ceph-mon[32231]: pgmap v2045: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:35 smithi181 ceph-mon[35869]: pgmap v2046: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2472.663322 secs 2024-07-26T08:40:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:35 smithi097 ceph-mon[32231]: pgmap v2046: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2472.663322 secs 2024-07-26T08:40:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:35 smithi115 ceph-mon[35364]: pgmap v2046: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2472.663322 secs 2024-07-26T08:40:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:37 smithi181 ceph-mon[35869]: pgmap v2047: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:37 smithi097 ceph-mon[32231]: pgmap v2047: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:37 smithi115 ceph-mon[35364]: pgmap v2047: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:39 smithi097 ceph-mon[32231]: pgmap v2048: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:39 smithi115 ceph-mon[35364]: pgmap v2048: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:39 smithi181 ceph-mon[35869]: pgmap v2048: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2477.663414 secs 2024-07-26T08:40:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2477.663414 secs 2024-07-26T08:40:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2477.663414 secs 2024-07-26T08:40:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:40:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:41 smithi115 ceph-mon[35364]: pgmap v2049: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:41 smithi115 ceph-mon[35364]: pgmap v2050: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:41 smithi181 ceph-mon[35869]: pgmap v2049: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:41 smithi181 ceph-mon[35869]: pgmap v2050: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:41 smithi097 ceph-mon[32231]: pgmap v2049: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:41 smithi097 ceph-mon[32231]: pgmap v2050: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:42 smithi115 ceph-mon[35364]: pgmap v2051: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:42 smithi181 ceph-mon[35869]: pgmap v2051: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:42 smithi097 ceph-mon[32231]: pgmap v2051: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:45.956 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:45 smithi097 ceph-mon[32231]: pgmap v2052: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:45.956 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2482.663547 secs 2024-07-26T08:40:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:45 smithi115 ceph-mon[35364]: pgmap v2052: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2482.663547 secs 2024-07-26T08:40:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:45 smithi181 ceph-mon[35869]: pgmap v2052: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2482.663547 secs 2024-07-26T08:40:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:47 smithi115 ceph-mon[35364]: pgmap v2053: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:47 smithi181 ceph-mon[35869]: pgmap v2053: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:47 smithi097 ceph-mon[32231]: pgmap v2053: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:48.452+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:48.455 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:48.453+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:40:48.455 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:48.453+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:40:48.455 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:48.453+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:40:48.455 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:40:48.454+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:40:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:49 smithi115 ceph-mon[35364]: pgmap v2054: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:49 smithi181 ceph-mon[35869]: pgmap v2054: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:49 smithi097 ceph-mon[32231]: pgmap v2054: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:40:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2487.663686 secs 2024-07-26T08:40:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2487.663686 secs 2024-07-26T08:40:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2487.663686 secs 2024-07-26T08:40:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:51 smithi115 ceph-mon[35364]: pgmap v2055: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:40:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:51 smithi181 ceph-mon[35869]: pgmap v2055: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:40:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:51 smithi097 ceph-mon[32231]: pgmap v2055: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:40:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:53 smithi115 ceph-mon[35364]: pgmap v2056: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:53 smithi181 ceph-mon[35869]: pgmap v2056: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:53 smithi097 ceph-mon[32231]: pgmap v2056: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:40:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:55 smithi115 ceph-mon[35364]: pgmap v2057: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2492.663773 secs 2024-07-26T08:40:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:55 smithi181 ceph-mon[35869]: pgmap v2057: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2492.663773 secs 2024-07-26T08:40:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:55 smithi097 ceph-mon[32231]: pgmap v2057: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2492.663773 secs 2024-07-26T08:40:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:57 smithi115 ceph-mon[35364]: pgmap v2058: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:57 smithi181 ceph-mon[35869]: pgmap v2058: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:57 smithi097 ceph-mon[32231]: pgmap v2058: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:40:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:40:59 smithi115 ceph-mon[35364]: pgmap v2059: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:40:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:40:59 smithi181 ceph-mon[35869]: pgmap v2059: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:41:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:40:59 smithi097 ceph-mon[32231]: pgmap v2059: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:41:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2497.663943 secs 2024-07-26T08:41:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2497.663943 secs 2024-07-26T08:41:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2497.663943 secs 2024-07-26T08:41:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:01 smithi115 ceph-mon[35364]: pgmap v2060: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:41:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:01 smithi181 ceph-mon[35869]: pgmap v2060: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:41:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:01 smithi097 ceph-mon[32231]: pgmap v2060: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:41:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:03.457+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:03.458+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:41:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:03.458+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:41:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:03.458+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:41:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:03.458+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:03 smithi115 ceph-mon[35364]: pgmap v2061: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:41:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:03 smithi181 ceph-mon[35869]: pgmap v2061: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:41:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:03 smithi097 ceph-mon[32231]: pgmap v2061: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:41:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:05 smithi115 ceph-mon[35364]: pgmap v2062: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2502.664030 secs 2024-07-26T08:41:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:05 smithi181 ceph-mon[35869]: pgmap v2062: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2502.664030 secs 2024-07-26T08:41:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:05 smithi097 ceph-mon[32231]: pgmap v2062: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2502.664030 secs 2024-07-26T08:41:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:07 smithi181 ceph-mon[35869]: pgmap v2063: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:07 smithi097 ceph-mon[32231]: pgmap v2063: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:07 smithi115 ceph-mon[35364]: pgmap v2063: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:09 smithi097 ceph-mon[32231]: pgmap v2064: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:09 smithi115 ceph-mon[35364]: pgmap v2064: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:09 smithi181 ceph-mon[35869]: pgmap v2064: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2507.664177 secs 2024-07-26T08:41:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2507.664177 secs 2024-07-26T08:41:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2507.664177 secs 2024-07-26T08:41:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:11 smithi097 ceph-mon[32231]: pgmap v2065: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:11 smithi115 ceph-mon[35364]: pgmap v2065: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:11 smithi181 ceph-mon[35869]: pgmap v2065: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:13 smithi115 ceph-mon[35364]: pgmap v2066: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:13 smithi181 ceph-mon[35869]: pgmap v2066: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:13 smithi097 ceph-mon[32231]: pgmap v2066: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:16.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:15 smithi097 ceph-mon[32231]: pgmap v2067: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:16.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2512.664326 secs 2024-07-26T08:41:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:15 smithi115 ceph-mon[35364]: pgmap v2067: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2512.664326 secs 2024-07-26T08:41:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:15 smithi181 ceph-mon[35869]: pgmap v2067: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2512.664326 secs 2024-07-26T08:41:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:16 smithi115 ceph-mon[35364]: pgmap v2068: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:16 smithi181 ceph-mon[35869]: pgmap v2068: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:16 smithi097 ceph-mon[32231]: pgmap v2068: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:18.466+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:18.468+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:41:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:18.468+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:41:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:18.468+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:41:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:18.468+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:19.937 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:19 smithi181 ceph-mon[35869]: pgmap v2069: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:19 smithi115 ceph-mon[35364]: pgmap v2069: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:19 smithi097 ceph-mon[32231]: pgmap v2069: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2517.664462 secs 2024-07-26T08:41:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2517.664462 secs 2024-07-26T08:41:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2517.664462 secs 2024-07-26T08:41:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:21 smithi115 ceph-mon[35364]: pgmap v2070: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:21 smithi181 ceph-mon[35869]: pgmap v2070: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:21 smithi097 ceph-mon[32231]: pgmap v2070: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:23 smithi115 ceph-mon[35364]: pgmap v2071: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:23 smithi181 ceph-mon[35869]: pgmap v2071: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:23 smithi097 ceph-mon[32231]: pgmap v2071: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:25 smithi115 ceph-mon[35364]: pgmap v2072: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2522.664605 secs 2024-07-26T08:41:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:25 smithi181 ceph-mon[35869]: pgmap v2072: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2522.664605 secs 2024-07-26T08:41:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:25 smithi097 ceph-mon[32231]: pgmap v2072: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2522.664605 secs 2024-07-26T08:41:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:27 smithi115 ceph-mon[35364]: pgmap v2073: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:27 smithi181 ceph-mon[35869]: pgmap v2073: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:27 smithi097 ceph-mon[32231]: pgmap v2073: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:29 smithi115 ceph-mon[35364]: pgmap v2074: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:29 smithi181 ceph-mon[35869]: pgmap v2074: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:29 smithi097 ceph-mon[32231]: pgmap v2074: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2527.664738 secs 2024-07-26T08:41:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2527.664738 secs 2024-07-26T08:41:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2527.664738 secs 2024-07-26T08:41:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:31 smithi115 ceph-mon[35364]: pgmap v2075: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:31 smithi181 ceph-mon[35869]: pgmap v2075: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:31 smithi097 ceph-mon[32231]: pgmap v2075: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:41:33.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:33.472+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:33.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:33.474+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:41:33.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:33.474+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:41:33.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:33.474+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:41:33.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:33.474+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:33 smithi115 ceph-mon[35364]: pgmap v2076: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:33 smithi181 ceph-mon[35869]: pgmap v2076: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:33 smithi097 ceph-mon[32231]: pgmap v2076: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:35 smithi181 ceph-mon[35869]: pgmap v2077: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:41:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2532.664890 secs 2024-07-26T08:41:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:35 smithi097 ceph-mon[32231]: pgmap v2077: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:41:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2532.664890 secs 2024-07-26T08:41:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:35 smithi115 ceph-mon[35364]: pgmap v2077: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:41:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2532.664890 secs 2024-07-26T08:41:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:37 smithi097 ceph-mon[32231]: pgmap v2078: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:41:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:37 smithi115 ceph-mon[35364]: pgmap v2078: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:41:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:37 smithi181 ceph-mon[35869]: pgmap v2078: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:41:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:39 smithi097 ceph-mon[32231]: pgmap v2079: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:39 smithi115 ceph-mon[35364]: pgmap v2079: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:39 smithi181 ceph-mon[35869]: pgmap v2079: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:41.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2537.665026 secs 2024-07-26T08:41:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2537.665026 secs 2024-07-26T08:41:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2537.665026 secs 2024-07-26T08:41:42.161 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:41 smithi115 ceph-mon[35364]: pgmap v2080: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:41:42.161 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:42.181 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:41 smithi181 ceph-mon[35869]: pgmap v2080: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:41:42.181 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:42.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:41 smithi097 ceph-mon[32231]: pgmap v2080: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:41:42.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:43 smithi115 ceph-mon[35364]: pgmap v2081: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:43 smithi181 ceph-mon[35869]: pgmap v2081: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:43 smithi097 ceph-mon[32231]: pgmap v2081: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:44 smithi115 ceph-mon[35364]: pgmap v2082: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:44 smithi181 ceph-mon[35869]: pgmap v2082: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:44 smithi097 ceph-mon[32231]: pgmap v2082: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:41:46.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2542.665123 secs 2024-07-26T08:41:46.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:41:46.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:41:46.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:45 smithi097 ceph-mon[32231]: pgmap v2083: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:46.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:45 smithi097 ceph-mon[32231]: pgmap v2084: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:41:46.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2542.665123 secs 2024-07-26T08:41:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:41:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:41:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:45 smithi115 ceph-mon[35364]: pgmap v2083: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:45 smithi115 ceph-mon[35364]: pgmap v2084: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:41:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2542.665123 secs 2024-07-26T08:41:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:41:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:41:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:45 smithi181 ceph-mon[35869]: pgmap v2083: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:41:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:45 smithi181 ceph-mon[35869]: pgmap v2084: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:41:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:41:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:48.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:48.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:41:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:48.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:41:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:48.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:41:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:41:48.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:48 smithi115 ceph-mon[35364]: pgmap v2085: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:41:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:48 smithi181 ceph-mon[35869]: pgmap v2085: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:41:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:48 smithi097 ceph-mon[32231]: pgmap v2085: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:41:50.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:50 smithi181 ceph-mon[35869]: pgmap v2086: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:41:50.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2547.665267 secs 2024-07-26T08:41:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:50 smithi115 ceph-mon[35364]: pgmap v2086: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:41:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2547.665267 secs 2024-07-26T08:41:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:50 smithi097 ceph-mon[32231]: pgmap v2086: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:41:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2547.665267 secs 2024-07-26T08:41:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:52 smithi181 ceph-mon[35869]: pgmap v2087: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:41:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:52 smithi115 ceph-mon[35364]: pgmap v2087: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:41:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:52 smithi097 ceph-mon[32231]: pgmap v2087: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T08:41:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:54 smithi115 ceph-mon[35364]: pgmap v2088: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:41:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:54 smithi181 ceph-mon[35869]: pgmap v2088: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:41:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:54 smithi097 ceph-mon[32231]: pgmap v2088: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:41:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2552.665415 secs 2024-07-26T08:41:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2552.665415 secs 2024-07-26T08:41:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2552.665415 secs 2024-07-26T08:41:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:56 smithi115 ceph-mon[35364]: pgmap v2089: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:56 smithi181 ceph-mon[35869]: pgmap v2089: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:56 smithi097 ceph-mon[32231]: pgmap v2089: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:41:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:41:58 smithi115 ceph-mon[35364]: pgmap v2090: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:41:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:41:58 smithi181 ceph-mon[35869]: pgmap v2090: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:41:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:41:58 smithi097 ceph-mon[32231]: pgmap v2090: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:00 smithi115 ceph-mon[35364]: pgmap v2091: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2557.665560 secs 2024-07-26T08:42:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:00 smithi181 ceph-mon[35869]: pgmap v2091: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2557.665560 secs 2024-07-26T08:42:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:00 smithi097 ceph-mon[32231]: pgmap v2091: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2557.665560 secs 2024-07-26T08:42:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:02 smithi181 ceph-mon[35869]: pgmap v2092: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:02 smithi115 ceph-mon[35364]: pgmap v2092: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:02 smithi097 ceph-mon[32231]: pgmap v2092: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:03.485+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:42:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:03.486+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:42:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:03.486+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:42:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:03.486+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:42:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:03.486+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:04 smithi115 ceph-mon[35364]: pgmap v2093: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:04 smithi181 ceph-mon[35869]: pgmap v2093: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:04 smithi097 ceph-mon[32231]: pgmap v2093: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2562.665702 secs 2024-07-26T08:42:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2562.665702 secs 2024-07-26T08:42:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2562.665702 secs 2024-07-26T08:42:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:06 smithi115 ceph-mon[35364]: pgmap v2094: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:06 smithi181 ceph-mon[35869]: pgmap v2094: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:06 smithi097 ceph-mon[32231]: pgmap v2094: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:08 smithi115 ceph-mon[35364]: pgmap v2095: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:08 smithi181 ceph-mon[35869]: pgmap v2095: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:08 smithi097 ceph-mon[32231]: pgmap v2095: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:10 smithi115 ceph-mon[35364]: pgmap v2096: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2567.665786 secs 2024-07-26T08:42:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:10 smithi181 ceph-mon[35869]: pgmap v2096: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2567.665786 secs 2024-07-26T08:42:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:10 smithi097 ceph-mon[32231]: pgmap v2096: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2567.665786 secs 2024-07-26T08:42:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:12 smithi115 ceph-mon[35364]: pgmap v2097: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:12 smithi181 ceph-mon[35869]: pgmap v2097: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:12 smithi097 ceph-mon[32231]: pgmap v2097: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:14 smithi115 ceph-mon[35364]: pgmap v2098: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:14 smithi181 ceph-mon[35869]: pgmap v2098: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:14 smithi097 ceph-mon[32231]: pgmap v2098: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2572.665930 secs 2024-07-26T08:42:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2572.665930 secs 2024-07-26T08:42:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2572.665930 secs 2024-07-26T08:42:16.655 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:16 smithi097 ceph-mon[32231]: pgmap v2099: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:16 smithi115 ceph-mon[35364]: pgmap v2099: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:16 smithi181 ceph-mon[35869]: pgmap v2099: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:18.489+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:42:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:18.490+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:42:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:18.490+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:42:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:18.490+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:42:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:18.490+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:18 smithi115 ceph-mon[35364]: pgmap v2100: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:18 smithi181 ceph-mon[35869]: pgmap v2100: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:18 smithi097 ceph-mon[32231]: pgmap v2100: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:20 smithi115 ceph-mon[35364]: pgmap v2101: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2577.666062 secs 2024-07-26T08:42:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:20 smithi181 ceph-mon[35869]: pgmap v2101: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2577.666062 secs 2024-07-26T08:42:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:20 smithi097 ceph-mon[32231]: pgmap v2101: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2577.666062 secs 2024-07-26T08:42:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:22 smithi115 ceph-mon[35364]: pgmap v2102: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:22 smithi181 ceph-mon[35869]: pgmap v2102: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:22 smithi097 ceph-mon[32231]: pgmap v2102: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:24 smithi115 ceph-mon[35364]: pgmap v2103: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:24 smithi181 ceph-mon[35869]: pgmap v2103: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:24 smithi097 ceph-mon[32231]: pgmap v2103: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2582.666196 secs 2024-07-26T08:42:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2582.666196 secs 2024-07-26T08:42:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2582.666196 secs 2024-07-26T08:42:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:26 smithi115 ceph-mon[35364]: pgmap v2104: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:26 smithi181 ceph-mon[35869]: pgmap v2104: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:26 smithi097 ceph-mon[32231]: pgmap v2104: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:28 smithi115 ceph-mon[35364]: pgmap v2105: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:28 smithi181 ceph-mon[35869]: pgmap v2105: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:28 smithi097 ceph-mon[32231]: pgmap v2105: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:30 smithi115 ceph-mon[35364]: pgmap v2106: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2587.666279 secs 2024-07-26T08:42:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:30 smithi181 ceph-mon[35869]: pgmap v2106: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2587.666279 secs 2024-07-26T08:42:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:30 smithi097 ceph-mon[32231]: pgmap v2106: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2587.666279 secs 2024-07-26T08:42:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:32 smithi115 ceph-mon[35364]: pgmap v2107: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:32 smithi181 ceph-mon[35869]: pgmap v2107: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:32 smithi097 ceph-mon[32231]: pgmap v2107: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:33.494+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:42:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:33.495+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:42:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:33.495+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:42:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:33.495+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:42:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:33.495+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:34 smithi097 ceph-mon[32231]: pgmap v2108: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:34 smithi115 ceph-mon[35364]: pgmap v2108: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:34 smithi181 ceph-mon[35869]: pgmap v2108: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2592.666377 secs 2024-07-26T08:42:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2592.666377 secs 2024-07-26T08:42:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2592.666377 secs 2024-07-26T08:42:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:36 smithi115 ceph-mon[35364]: pgmap v2109: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:36 smithi181 ceph-mon[35869]: pgmap v2109: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:36 smithi097 ceph-mon[32231]: pgmap v2109: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:38 smithi115 ceph-mon[35364]: pgmap v2110: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:38 smithi181 ceph-mon[35869]: pgmap v2110: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:38 smithi097 ceph-mon[32231]: pgmap v2110: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:42:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:40 smithi115 ceph-mon[35364]: pgmap v2111: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2597.666471 secs 2024-07-26T08:42:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:40 smithi181 ceph-mon[35869]: pgmap v2111: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2597.666471 secs 2024-07-26T08:42:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:40 smithi097 ceph-mon[32231]: pgmap v2111: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:42:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2597.666471 secs 2024-07-26T08:42:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:42 smithi115 ceph-mon[35364]: pgmap v2112: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:42 smithi181 ceph-mon[35869]: pgmap v2112: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:42 smithi097 ceph-mon[32231]: pgmap v2112: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:42:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:44 smithi115 ceph-mon[35364]: pgmap v2113: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:44 smithi181 ceph-mon[35869]: pgmap v2113: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:44 smithi097 ceph-mon[32231]: pgmap v2113: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2602.666621 secs 2024-07-26T08:42:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2602.666621 secs 2024-07-26T08:42:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2602.666621 secs 2024-07-26T08:42:46.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:46 smithi097 ceph-mon[32231]: pgmap v2114: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:46.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:42:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:46 smithi115 ceph-mon[35364]: pgmap v2114: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:42:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:46 smithi181 ceph-mon[35869]: pgmap v2114: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:42:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:47 smithi115 ceph-mon[35364]: pgmap v2115: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:42:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:47 smithi115 ceph-mon[35364]: pgmap v2116: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:42:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:42:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:47 smithi181 ceph-mon[35869]: pgmap v2115: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:42:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:47 smithi181 ceph-mon[35869]: pgmap v2116: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:42:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:42:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:47 smithi097 ceph-mon[32231]: pgmap v2115: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:42:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:47 smithi097 ceph-mon[32231]: pgmap v2116: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:42:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:48.498+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:48.499+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:48.499+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:48.499+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:42:48.499+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:49 smithi115 ceph-mon[35364]: pgmap v2117: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:42:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:49 smithi181 ceph-mon[35869]: pgmap v2117: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:42:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:49 smithi097 ceph-mon[32231]: pgmap v2117: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:42:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2607.666710 secs 2024-07-26T08:42:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2607.666710 secs 2024-07-26T08:42:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2607.666710 secs 2024-07-26T08:42:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:51 smithi115 ceph-mon[35364]: pgmap v2118: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:51 smithi181 ceph-mon[35869]: pgmap v2118: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:51 smithi097 ceph-mon[32231]: pgmap v2118: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:42:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:53 smithi115 ceph-mon[35364]: pgmap v2119: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:53 smithi181 ceph-mon[35869]: pgmap v2119: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:53 smithi097 ceph-mon[32231]: pgmap v2119: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:42:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:55 smithi181 ceph-mon[35869]: pgmap v2120: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:42:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2612.666851 secs 2024-07-26T08:42:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:55 smithi097 ceph-mon[32231]: pgmap v2120: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:42:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2612.666851 secs 2024-07-26T08:42:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:55 smithi115 ceph-mon[35364]: pgmap v2120: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:42:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2612.666851 secs 2024-07-26T08:42:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:57 smithi097 ceph-mon[32231]: pgmap v2121: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:42:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:57 smithi115 ceph-mon[35364]: pgmap v2121: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:42:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:57 smithi181 ceph-mon[35869]: pgmap v2121: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:43:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:42:59 smithi097 ceph-mon[32231]: pgmap v2122: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:43:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:42:59 smithi115 ceph-mon[35364]: pgmap v2122: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:43:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:42:59 smithi181 ceph-mon[35869]: pgmap v2122: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:43:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2617.667010 secs 2024-07-26T08:43:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2617.667010 secs 2024-07-26T08:43:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2617.667010 secs 2024-07-26T08:43:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:01 smithi115 ceph-mon[35364]: pgmap v2123: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:43:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:01 smithi181 ceph-mon[35869]: pgmap v2123: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:43:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:01 smithi097 ceph-mon[32231]: pgmap v2123: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:43:03.504 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:03.502+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:43:03.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:03.504+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:43:03.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:03.504+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:43:03.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:03.504+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:43:03.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:03.504+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:03 smithi115 ceph-mon[35364]: pgmap v2124: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:03 smithi181 ceph-mon[35869]: pgmap v2124: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:03 smithi097 ceph-mon[32231]: pgmap v2124: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:05 smithi115 ceph-mon[35364]: pgmap v2125: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2622.667165 secs 2024-07-26T08:43:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:05 smithi181 ceph-mon[35869]: pgmap v2125: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2622.667165 secs 2024-07-26T08:43:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:05 smithi097 ceph-mon[32231]: pgmap v2125: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2622.667165 secs 2024-07-26T08:43:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:06.883+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40c0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000322) for more than 900000000 us, fault. 2024-07-26T08:43:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:06.883+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0540 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e003b000 comp rx=0 tx=0).stop 2024-07-26T08:43:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:06.883+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40c0540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:43:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:06.883+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0540 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:43:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:06.883+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:43:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:06.884+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:43:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:06.886+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0a80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e0058000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:43:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:07.336+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:43:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:07.336+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:43:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:07.337+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:43:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:07.337+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:43:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:07 smithi115 ceph-mon[35364]: pgmap v2126: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:07 smithi181 ceph-mon[35869]: pgmap v2126: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:07 smithi097 ceph-mon[32231]: pgmap v2126: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:09 smithi115 ceph-mon[35364]: pgmap v2127: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:09 smithi181 ceph-mon[35869]: pgmap v2127: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:09 smithi097 ceph-mon[32231]: pgmap v2127: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2627.667285 secs 2024-07-26T08:43:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2627.667285 secs 2024-07-26T08:43:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2627.667285 secs 2024-07-26T08:43:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:11 smithi115 ceph-mon[35364]: pgmap v2128: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:11 smithi181 ceph-mon[35869]: pgmap v2128: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:11 smithi097 ceph-mon[32231]: pgmap v2128: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:12 smithi115 ceph-mon[35364]: pgmap v2129: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:12 smithi181 ceph-mon[35869]: pgmap v2129: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:12 smithi097 ceph-mon[32231]: pgmap v2129: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:15 smithi115 ceph-mon[35364]: pgmap v2130: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2632.667422 secs 2024-07-26T08:43:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:15 smithi181 ceph-mon[35869]: pgmap v2130: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2632.667422 secs 2024-07-26T08:43:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:15 smithi097 ceph-mon[32231]: pgmap v2130: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2632.667422 secs 2024-07-26T08:43:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:17 smithi115 ceph-mon[35364]: pgmap v2131: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:17 smithi181 ceph-mon[35869]: pgmap v2131: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:17 smithi097 ceph-mon[32231]: pgmap v2131: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:18.506+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:43:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:18.507+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:43:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:18.507+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:43:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:18.507+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:43:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:18.507+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:19 smithi181 ceph-mon[35869]: pgmap v2132: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:19 smithi097 ceph-mon[32231]: pgmap v2132: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:19 smithi115 ceph-mon[35364]: pgmap v2132: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2637.667578 secs 2024-07-26T08:43:21.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2637.667578 secs 2024-07-26T08:43:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2637.667578 secs 2024-07-26T08:43:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:21 smithi097 ceph-mon[32231]: pgmap v2133: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:21 smithi115 ceph-mon[35364]: pgmap v2133: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:21 smithi181 ceph-mon[35869]: pgmap v2133: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:23 smithi115 ceph-mon[35364]: pgmap v2134: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:23 smithi181 ceph-mon[35869]: pgmap v2134: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:23 smithi097 ceph-mon[32231]: pgmap v2134: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:25 smithi115 ceph-mon[35364]: pgmap v2135: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2642.667719 secs 2024-07-26T08:43:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:25 smithi181 ceph-mon[35869]: pgmap v2135: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2642.667719 secs 2024-07-26T08:43:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:25 smithi097 ceph-mon[32231]: pgmap v2135: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2642.667719 secs 2024-07-26T08:43:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:27 smithi115 ceph-mon[35364]: pgmap v2136: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:27 smithi181 ceph-mon[35869]: pgmap v2136: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:27 smithi097 ceph-mon[32231]: pgmap v2136: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:43:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:29 smithi115 ceph-mon[35364]: pgmap v2137: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:29 smithi181 ceph-mon[35869]: pgmap v2137: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:29 smithi097 ceph-mon[32231]: pgmap v2137: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2647.667804 secs 2024-07-26T08:43:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2647.667804 secs 2024-07-26T08:43:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2647.667804 secs 2024-07-26T08:43:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:31 smithi115 ceph-mon[35364]: pgmap v2138: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:31 smithi181 ceph-mon[35869]: pgmap v2138: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:31 smithi097 ceph-mon[32231]: pgmap v2138: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:32 smithi115 ceph-mon[35364]: pgmap v2139: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:32 smithi181 ceph-mon[35869]: pgmap v2139: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:32 smithi097 ceph-mon[32231]: pgmap v2139: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:33.511+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:43:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:33.512+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:43:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:33.512+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:43:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:33.512+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:43:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:33.512+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:35 smithi181 ceph-mon[35869]: pgmap v2140: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2652.667943 secs 2024-07-26T08:43:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:35 smithi097 ceph-mon[32231]: pgmap v2140: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2652.667943 secs 2024-07-26T08:43:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:35 smithi115 ceph-mon[35364]: pgmap v2140: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2652.667943 secs 2024-07-26T08:43:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:37 smithi097 ceph-mon[32231]: pgmap v2141: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:43:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:37 smithi115 ceph-mon[35364]: pgmap v2141: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:43:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:37 smithi181 ceph-mon[35869]: pgmap v2141: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:43:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:39 smithi097 ceph-mon[32231]: pgmap v2142: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:39 smithi115 ceph-mon[35364]: pgmap v2142: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:39 smithi181 ceph-mon[35869]: pgmap v2142: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2657.668081 secs 2024-07-26T08:43:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2657.668081 secs 2024-07-26T08:43:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2657.668081 secs 2024-07-26T08:43:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:41 smithi115 ceph-mon[35364]: pgmap v2143: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:41 smithi181 ceph-mon[35869]: pgmap v2143: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:41 smithi097 ceph-mon[32231]: pgmap v2143: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:43 smithi115 ceph-mon[35364]: pgmap v2144: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:43 smithi181 ceph-mon[35869]: pgmap v2144: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:43 smithi097 ceph-mon[32231]: pgmap v2144: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:43:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:45 smithi115 ceph-mon[35364]: pgmap v2145: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2662.668222 secs 2024-07-26T08:43:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:45 smithi181 ceph-mon[35869]: pgmap v2145: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2662.668222 secs 2024-07-26T08:43:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:45 smithi097 ceph-mon[32231]: pgmap v2145: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:43:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2662.668222 secs 2024-07-26T08:43:48.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:47 smithi097 ceph-mon[32231]: pgmap v2146: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:43:48.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:47 smithi115 ceph-mon[35364]: pgmap v2146: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:43:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:47 smithi181 ceph-mon[35869]: pgmap v2146: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:43:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:48.519+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:43:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:48.520+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:43:48.522 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:48.520+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:43:48.522 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:48.520+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:43:48.522 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:43:48.520+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:43:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:43:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:43:50.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:49 smithi181 ceph-mon[35869]: pgmap v2147: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:43:50.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:49 smithi181 ceph-mon[35869]: pgmap v2148: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:49 smithi115 ceph-mon[35364]: pgmap v2147: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:43:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:49 smithi115 ceph-mon[35364]: pgmap v2148: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:49 smithi097 ceph-mon[32231]: pgmap v2147: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:43:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:49 smithi097 ceph-mon[32231]: pgmap v2148: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:51.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2667.668341 secs 2024-07-26T08:43:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2667.668341 secs 2024-07-26T08:43:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2667.668341 secs 2024-07-26T08:43:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:51 smithi115 ceph-mon[35364]: pgmap v2149: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:51 smithi181 ceph-mon[35869]: pgmap v2149: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:51 smithi097 ceph-mon[32231]: pgmap v2149: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:52 smithi115 ceph-mon[35364]: pgmap v2150: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:43:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:52 smithi181 ceph-mon[35869]: pgmap v2150: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:43:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:52 smithi097 ceph-mon[32231]: pgmap v2150: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:43:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:55 smithi115 ceph-mon[35364]: pgmap v2151: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:43:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:55 smithi181 ceph-mon[35869]: pgmap v2151: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:43:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:55 smithi097 ceph-mon[32231]: pgmap v2151: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:43:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2672.668494 secs 2024-07-26T08:43:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2672.668494 secs 2024-07-26T08:43:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2672.668494 secs 2024-07-26T08:43:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:57 smithi115 ceph-mon[35364]: pgmap v2152: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:57 smithi181 ceph-mon[35869]: pgmap v2152: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:57 smithi097 ceph-mon[32231]: pgmap v2152: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:43:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:43:59 smithi115 ceph-mon[35364]: pgmap v2153: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:43:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:43:59 smithi181 ceph-mon[35869]: pgmap v2153: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:43:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:43:59 smithi097 ceph-mon[32231]: pgmap v2153: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:44:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2677.668631 secs 2024-07-26T08:44:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:01 smithi115 ceph-mon[35364]: pgmap v2154: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2677.668631 secs 2024-07-26T08:44:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:01 smithi181 ceph-mon[35869]: pgmap v2154: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2677.668631 secs 2024-07-26T08:44:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:01 smithi097 ceph-mon[32231]: pgmap v2154: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:03 smithi115 ceph-mon[35364]: pgmap v2155: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:03 smithi181 ceph-mon[35869]: pgmap v2155: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:03 smithi097 ceph-mon[32231]: pgmap v2155: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:03.524+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:44:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:03.525+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:44:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:03.525+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:44:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:03.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:44:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:03.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:05 smithi115 ceph-mon[35364]: pgmap v2156: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:05 smithi181 ceph-mon[35869]: pgmap v2156: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:05 smithi097 ceph-mon[32231]: pgmap v2156: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2682.668763 secs 2024-07-26T08:44:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2682.668763 secs 2024-07-26T08:44:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2682.668763 secs 2024-07-26T08:44:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:07 smithi115 ceph-mon[35364]: pgmap v2157: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:07 smithi181 ceph-mon[35869]: pgmap v2157: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:07 smithi097 ceph-mon[32231]: pgmap v2157: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:09 smithi115 ceph-mon[35364]: pgmap v2158: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:09 smithi181 ceph-mon[35869]: pgmap v2158: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:09 smithi097 ceph-mon[32231]: pgmap v2158: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2687.668900 secs 2024-07-26T08:44:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2687.668900 secs 2024-07-26T08:44:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2687.668900 secs 2024-07-26T08:44:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:11 smithi115 ceph-mon[35364]: pgmap v2159: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:11 smithi181 ceph-mon[35869]: pgmap v2159: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:11 smithi097 ceph-mon[32231]: pgmap v2159: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:13 smithi115 ceph-mon[35364]: pgmap v2160: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:13 smithi181 ceph-mon[35869]: pgmap v2160: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:13 smithi097 ceph-mon[32231]: pgmap v2160: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:15 smithi115 ceph-mon[35364]: pgmap v2161: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2692.669032 secs 2024-07-26T08:44:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:15 smithi181 ceph-mon[35869]: pgmap v2161: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2692.669032 secs 2024-07-26T08:44:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:15 smithi097 ceph-mon[32231]: pgmap v2161: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2692.669032 secs 2024-07-26T08:44:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:17 smithi115 ceph-mon[35364]: pgmap v2162: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:17 smithi181 ceph-mon[35869]: pgmap v2162: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:17 smithi097 ceph-mon[32231]: pgmap v2162: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:18.535+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:44:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:18.536+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:44:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:18.536+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:44:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:18.536+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:44:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:18.536+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:19.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:19 smithi097 ceph-mon[32231]: pgmap v2163: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:19 smithi115 ceph-mon[35364]: pgmap v2163: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:19 smithi181 ceph-mon[35869]: pgmap v2163: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:20.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2697.669166 secs 2024-07-26T08:44:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2697.669166 secs 2024-07-26T08:44:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2697.669166 secs 2024-07-26T08:44:21.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:21 smithi115 ceph-mon[35364]: pgmap v2164: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:44:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:21 smithi181 ceph-mon[35869]: pgmap v2164: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:44:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:21 smithi097 ceph-mon[32231]: pgmap v2164: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:44:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:23 smithi115 ceph-mon[35364]: pgmap v2165: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:44:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:23 smithi181 ceph-mon[35869]: pgmap v2165: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:44:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:23 smithi097 ceph-mon[32231]: pgmap v2165: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:44:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:25 smithi115 ceph-mon[35364]: pgmap v2166: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:44:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2702.669308 secs 2024-07-26T08:44:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:25 smithi181 ceph-mon[35869]: pgmap v2166: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:44:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2702.669308 secs 2024-07-26T08:44:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:25 smithi097 ceph-mon[32231]: pgmap v2166: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:44:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2702.669308 secs 2024-07-26T08:44:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:27 smithi115 ceph-mon[35364]: pgmap v2167: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:27 smithi181 ceph-mon[35869]: pgmap v2167: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:27 smithi097 ceph-mon[32231]: pgmap v2167: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:29 smithi115 ceph-mon[35364]: pgmap v2168: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:44:29.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:29 smithi181 ceph-mon[35869]: pgmap v2168: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:44:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:29 smithi097 ceph-mon[32231]: pgmap v2168: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:44:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2707.669403 secs 2024-07-26T08:44:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2707.669403 secs 2024-07-26T08:44:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2707.669403 secs 2024-07-26T08:44:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:31 smithi115 ceph-mon[35364]: pgmap v2169: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:44:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:31 smithi181 ceph-mon[35869]: pgmap v2169: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:44:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:31 smithi097 ceph-mon[32231]: pgmap v2169: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:44:33.542 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:33.540+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:44:33.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:33.542+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:44:33.544 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:33.542+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:44:33.544 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:33.542+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:44:33.544 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:33.542+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:33.677 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:33 smithi115 ceph-mon[35364]: pgmap v2170: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:33 smithi181 ceph-mon[35869]: pgmap v2170: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:33 smithi097 ceph-mon[32231]: pgmap v2170: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:35 smithi115 ceph-mon[35364]: pgmap v2171: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2712.669537 secs 2024-07-26T08:44:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:35 smithi181 ceph-mon[35869]: pgmap v2171: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2712.669537 secs 2024-07-26T08:44:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:35 smithi097 ceph-mon[32231]: pgmap v2171: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2712.669537 secs 2024-07-26T08:44:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:37 smithi115 ceph-mon[35364]: pgmap v2172: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:37.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:37 smithi181 ceph-mon[35869]: pgmap v2172: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:37 smithi097 ceph-mon[32231]: pgmap v2172: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:39 smithi115 ceph-mon[35364]: pgmap v2173: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:39 smithi181 ceph-mon[35869]: pgmap v2173: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:39 smithi097 ceph-mon[32231]: pgmap v2173: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2717.669630 secs 2024-07-26T08:44:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2717.669630 secs 2024-07-26T08:44:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2717.669630 secs 2024-07-26T08:44:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:41 smithi115 ceph-mon[35364]: pgmap v2174: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:41 smithi181 ceph-mon[35869]: pgmap v2174: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:41 smithi097 ceph-mon[32231]: pgmap v2174: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:43 smithi115 ceph-mon[35364]: pgmap v2175: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:43 smithi181 ceph-mon[35869]: pgmap v2175: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:43 smithi097 ceph-mon[32231]: pgmap v2175: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:44:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:45 smithi115 ceph-mon[35364]: pgmap v2176: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2722.669782 secs 2024-07-26T08:44:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:45 smithi181 ceph-mon[35869]: pgmap v2176: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2722.669782 secs 2024-07-26T08:44:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:45 smithi097 ceph-mon[32231]: pgmap v2176: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:44:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2722.669782 secs 2024-07-26T08:44:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:47 smithi115 ceph-mon[35364]: pgmap v2177: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:47 smithi181 ceph-mon[35869]: pgmap v2177: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:47 smithi097 ceph-mon[32231]: pgmap v2177: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:48.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:48.550+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:44:48.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:48.551+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:44:48.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:48.551+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:44:48.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:48.551+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:44:48.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:44:48.551+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:49 smithi115 ceph-mon[35364]: pgmap v2178: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:49 smithi181 ceph-mon[35869]: pgmap v2178: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:49 smithi097 ceph-mon[32231]: pgmap v2178: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:50 smithi115 ceph-mon[35364]: pgmap v2179: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:44:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:50 smithi115 ceph-mon[35364]: pgmap v2180: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:44:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2727.669918 secs 2024-07-26T08:44:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:50 smithi181 ceph-mon[35869]: pgmap v2179: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:44:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:50 smithi181 ceph-mon[35869]: pgmap v2180: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:44:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2727.669918 secs 2024-07-26T08:44:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:50 smithi097 ceph-mon[32231]: pgmap v2179: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:44:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:50 smithi097 ceph-mon[32231]: pgmap v2180: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:44:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2727.669918 secs 2024-07-26T08:44:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:52 smithi115 ceph-mon[35364]: pgmap v2181: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:52 smithi181 ceph-mon[35869]: pgmap v2181: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:52 smithi097 ceph-mon[32231]: pgmap v2181: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:44:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:54 smithi115 ceph-mon[35364]: pgmap v2182: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:54 smithi181 ceph-mon[35869]: pgmap v2182: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:54 smithi097 ceph-mon[32231]: pgmap v2182: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:44:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2732.670054 secs 2024-07-26T08:44:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2732.670054 secs 2024-07-26T08:44:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2732.670054 secs 2024-07-26T08:44:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:56 smithi115 ceph-mon[35364]: pgmap v2183: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:56 smithi181 ceph-mon[35869]: pgmap v2183: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:56 smithi097 ceph-mon[32231]: pgmap v2183: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:44:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:44:58 smithi115 ceph-mon[35364]: pgmap v2184: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:44:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:44:58 smithi181 ceph-mon[35869]: pgmap v2184: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:44:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:44:58 smithi097 ceph-mon[32231]: pgmap v2184: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:45:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:00 smithi115 ceph-mon[35364]: pgmap v2185: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:45:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2737.670189 secs 2024-07-26T08:45:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:00 smithi181 ceph-mon[35869]: pgmap v2185: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:45:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2737.670189 secs 2024-07-26T08:45:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:00 smithi097 ceph-mon[32231]: pgmap v2185: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:45:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2737.670189 secs 2024-07-26T08:45:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:02 smithi115 ceph-mon[35364]: pgmap v2186: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:02 smithi181 ceph-mon[35869]: pgmap v2186: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:02 smithi097 ceph-mon[32231]: pgmap v2186: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:03.556+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:45:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:03.557+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:45:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:03.557+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:45:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:03.557+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:45:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:03.557+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:04 smithi115 ceph-mon[35364]: pgmap v2187: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:04 smithi181 ceph-mon[35869]: pgmap v2187: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:04 smithi097 ceph-mon[32231]: pgmap v2187: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2742.670338 secs 2024-07-26T08:45:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2742.670338 secs 2024-07-26T08:45:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2742.670338 secs 2024-07-26T08:45:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:06 smithi115 ceph-mon[35364]: pgmap v2188: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:06 smithi181 ceph-mon[35869]: pgmap v2188: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:06 smithi097 ceph-mon[32231]: pgmap v2188: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:08 smithi115 ceph-mon[35364]: pgmap v2189: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:08 smithi181 ceph-mon[35869]: pgmap v2189: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:08 smithi097 ceph-mon[32231]: pgmap v2189: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:10 smithi115 ceph-mon[35364]: pgmap v2190: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2747.670483 secs 2024-07-26T08:45:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:10 smithi181 ceph-mon[35869]: pgmap v2190: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2747.670483 secs 2024-07-26T08:45:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:10 smithi097 ceph-mon[32231]: pgmap v2190: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2747.670483 secs 2024-07-26T08:45:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:12 smithi115 ceph-mon[35364]: pgmap v2191: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:12 smithi181 ceph-mon[35869]: pgmap v2191: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:12 smithi097 ceph-mon[32231]: pgmap v2191: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:14 smithi115 ceph-mon[35364]: pgmap v2192: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:14 smithi181 ceph-mon[35869]: pgmap v2192: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:14 smithi097 ceph-mon[32231]: pgmap v2192: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2752.670656 secs 2024-07-26T08:45:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2752.670656 secs 2024-07-26T08:45:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2752.670656 secs 2024-07-26T08:45:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:16 smithi115 ceph-mon[35364]: pgmap v2193: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:16 smithi181 ceph-mon[35869]: pgmap v2193: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:16 smithi097 ceph-mon[32231]: pgmap v2193: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:18.566+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:45:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:18.568+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:45:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:18.568+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:45:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:18.568+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:45:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:18.568+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:18 smithi115 ceph-mon[35364]: pgmap v2194: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:18 smithi181 ceph-mon[35869]: pgmap v2194: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:18 smithi097 ceph-mon[32231]: pgmap v2194: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:20 smithi115 ceph-mon[35364]: pgmap v2195: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2757.670795 secs 2024-07-26T08:45:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:20 smithi181 ceph-mon[35869]: pgmap v2195: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2757.670795 secs 2024-07-26T08:45:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:20 smithi097 ceph-mon[32231]: pgmap v2195: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2757.670795 secs 2024-07-26T08:45:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:22 smithi115 ceph-mon[35364]: pgmap v2196: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:22 smithi181 ceph-mon[35869]: pgmap v2196: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:22 smithi097 ceph-mon[32231]: pgmap v2196: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:24 smithi115 ceph-mon[35364]: pgmap v2197: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:24 smithi181 ceph-mon[35869]: pgmap v2197: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:24 smithi097 ceph-mon[32231]: pgmap v2197: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2762.670886 secs 2024-07-26T08:45:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2762.670886 secs 2024-07-26T08:45:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2762.670886 secs 2024-07-26T08:45:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:26 smithi115 ceph-mon[35364]: pgmap v2198: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:26 smithi181 ceph-mon[35869]: pgmap v2198: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:26 smithi097 ceph-mon[32231]: pgmap v2198: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:28 smithi115 ceph-mon[35364]: pgmap v2199: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:45:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:28 smithi181 ceph-mon[35869]: pgmap v2199: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:45:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:28 smithi097 ceph-mon[32231]: pgmap v2199: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:45:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:30 smithi115 ceph-mon[35364]: pgmap v2200: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2767.671017 secs 2024-07-26T08:45:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:30 smithi181 ceph-mon[35869]: pgmap v2200: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2767.671017 secs 2024-07-26T08:45:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:30 smithi097 ceph-mon[32231]: pgmap v2200: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2767.671017 secs 2024-07-26T08:45:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:32 smithi115 ceph-mon[35364]: pgmap v2201: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:32 smithi181 ceph-mon[35869]: pgmap v2201: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:32 smithi097 ceph-mon[32231]: pgmap v2201: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:33.573+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:45:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:33.574+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:45:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:33.574+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:45:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:33.574+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:45:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:33.574+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:34 smithi115 ceph-mon[35364]: pgmap v2202: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:34 smithi181 ceph-mon[35869]: pgmap v2202: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:34 smithi097 ceph-mon[32231]: pgmap v2202: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2772.671155 secs 2024-07-26T08:45:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2772.671155 secs 2024-07-26T08:45:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2772.671155 secs 2024-07-26T08:45:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:36 smithi097 ceph-mon[32231]: pgmap v2203: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:36 smithi115 ceph-mon[35364]: pgmap v2203: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:36 smithi181 ceph-mon[35869]: pgmap v2203: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:38 smithi097 ceph-mon[32231]: pgmap v2204: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:38 smithi115 ceph-mon[35364]: pgmap v2204: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:38 smithi181 ceph-mon[35869]: pgmap v2204: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:40 smithi097 ceph-mon[32231]: pgmap v2205: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2777.671291 secs 2024-07-26T08:45:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:40 smithi115 ceph-mon[35364]: pgmap v2205: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2777.671291 secs 2024-07-26T08:45:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:40 smithi181 ceph-mon[35869]: pgmap v2205: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2777.671291 secs 2024-07-26T08:45:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:42 smithi115 ceph-mon[35364]: pgmap v2206: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:42 smithi181 ceph-mon[35869]: pgmap v2206: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:42 smithi097 ceph-mon[32231]: pgmap v2206: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:44 smithi115 ceph-mon[35364]: pgmap v2207: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:44 smithi181 ceph-mon[35869]: pgmap v2207: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:44 smithi097 ceph-mon[32231]: pgmap v2207: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:45:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2782.671455 secs 2024-07-26T08:45:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2782.671455 secs 2024-07-26T08:45:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2782.671455 secs 2024-07-26T08:45:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:46 smithi115 ceph-mon[35364]: pgmap v2208: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:46 smithi181 ceph-mon[35869]: pgmap v2208: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:46 smithi097 ceph-mon[32231]: pgmap v2208: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:45:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:48.582+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:45:48.585 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:48.584+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:45:48.585 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:48.584+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:45:48.585 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:48.584+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:45:48.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:45:48.584+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:48 smithi115 ceph-mon[35364]: pgmap v2209: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:48 smithi181 ceph-mon[35869]: pgmap v2209: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:48 smithi097 ceph-mon[32231]: pgmap v2209: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:51.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:50 smithi097 ceph-mon[32231]: pgmap v2210: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:51.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2787.671605 secs 2024-07-26T08:45:51.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:50 smithi115 ceph-mon[35364]: pgmap v2210: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2787.671605 secs 2024-07-26T08:45:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:50 smithi181 ceph-mon[35869]: pgmap v2210: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2787.671605 secs 2024-07-26T08:45:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:53.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:52 smithi097 ceph-mon[32231]: pgmap v2211: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:52 smithi115 ceph-mon[35364]: pgmap v2211: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:52 smithi181 ceph-mon[35869]: pgmap v2211: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:45:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:54 smithi097 ceph-mon[32231]: pgmap v2212: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:45:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:54 smithi115 ceph-mon[35364]: pgmap v2212: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:45:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:54 smithi181 ceph-mon[35869]: pgmap v2212: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:45:55.991 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2792.671752 secs 2024-07-26T08:45:55.992 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:55.992 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:55.992 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:55.992 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:55.992 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2792.671752 secs 2024-07-26T08:45:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2792.671752 secs 2024-07-26T08:45:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:45:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:56 smithi115 ceph-mon[35364]: pgmap v2213: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:45:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:56 smithi115 ceph-mon[35364]: pgmap v2214: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:45:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:56 smithi115 ceph-mon[35364]: pgmap v2215: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:45:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:56 smithi181 ceph-mon[35869]: pgmap v2213: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:45:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:56 smithi181 ceph-mon[35869]: pgmap v2214: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:45:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:56 smithi181 ceph-mon[35869]: pgmap v2215: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:45:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:56 smithi097 ceph-mon[32231]: pgmap v2213: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:45:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:56 smithi097 ceph-mon[32231]: pgmap v2214: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T08:45:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:56 smithi097 ceph-mon[32231]: pgmap v2215: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:45:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:45:57 smithi115 ceph-mon[35364]: pgmap v2216: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:45:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:45:57 smithi181 ceph-mon[35869]: pgmap v2216: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:45:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:45:57 smithi097 ceph-mon[32231]: pgmap v2216: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:46:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2797.671881 secs 2024-07-26T08:46:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:00 smithi115 ceph-mon[35364]: pgmap v2217: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2797.671881 secs 2024-07-26T08:46:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:00 smithi181 ceph-mon[35869]: pgmap v2217: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2797.671881 secs 2024-07-26T08:46:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:00 smithi097 ceph-mon[32231]: pgmap v2217: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:02 smithi115 ceph-mon[35364]: pgmap v2218: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:46:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:02 smithi181 ceph-mon[35869]: pgmap v2218: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:46:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:02 smithi097 ceph-mon[32231]: pgmap v2218: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:46:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:03.589+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:46:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:03.590+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:46:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:03.590+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:46:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:03.590+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:46:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:03.590+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:04 smithi115 ceph-mon[35364]: pgmap v2219: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:46:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:04 smithi181 ceph-mon[35869]: pgmap v2219: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:46:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:04 smithi097 ceph-mon[32231]: pgmap v2219: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:46:06.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2802.672039 secs 2024-07-26T08:46:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2802.672039 secs 2024-07-26T08:46:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2802.672039 secs 2024-07-26T08:46:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:06 smithi115 ceph-mon[35364]: pgmap v2220: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:46:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:06 smithi181 ceph-mon[35869]: pgmap v2220: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:46:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:06 smithi097 ceph-mon[32231]: pgmap v2220: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:46:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:07.368+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:46:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:07.368+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:46:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:07.368+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:46:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:07.370+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:46:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:07 smithi115 ceph-mon[35364]: pgmap v2221: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:08.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:07 smithi181 ceph-mon[35869]: pgmap v2221: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:07 smithi097 ceph-mon[32231]: pgmap v2221: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2807.672191 secs 2024-07-26T08:46:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:10 smithi115 ceph-mon[35364]: pgmap v2222: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2807.672191 secs 2024-07-26T08:46:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:10 smithi181 ceph-mon[35869]: pgmap v2222: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2807.672191 secs 2024-07-26T08:46:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:10 smithi097 ceph-mon[32231]: pgmap v2222: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:12 smithi115 ceph-mon[35364]: pgmap v2223: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:12 smithi181 ceph-mon[35869]: pgmap v2223: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:12 smithi097 ceph-mon[32231]: pgmap v2223: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:14 smithi115 ceph-mon[35364]: pgmap v2224: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:46:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:14 smithi181 ceph-mon[35869]: pgmap v2224: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:46:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:14 smithi097 ceph-mon[32231]: pgmap v2224: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:46:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2812.672357 secs 2024-07-26T08:46:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2812.672357 secs 2024-07-26T08:46:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2812.672357 secs 2024-07-26T08:46:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:16 smithi115 ceph-mon[35364]: pgmap v2225: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:16 smithi181 ceph-mon[35869]: pgmap v2225: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:16 smithi097 ceph-mon[32231]: pgmap v2225: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:17 smithi115 ceph-mon[35364]: pgmap v2226: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:46:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:17 smithi181 ceph-mon[35869]: pgmap v2226: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:46:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:17 smithi097 ceph-mon[32231]: pgmap v2226: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:46:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:18.594+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:46:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:18.595+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:46:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:18.595+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:46:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:18.595+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:46:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:18.595+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2817.672513 secs 2024-07-26T08:46:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:20 smithi115 ceph-mon[35364]: pgmap v2227: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2817.672513 secs 2024-07-26T08:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:20 smithi181 ceph-mon[35869]: pgmap v2227: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:46:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2817.672513 secs 2024-07-26T08:46:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:20 smithi097 ceph-mon[32231]: pgmap v2227: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:46:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:22 smithi115 ceph-mon[35364]: pgmap v2228: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:46:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:22 smithi181 ceph-mon[35869]: pgmap v2228: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:46:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:22 smithi097 ceph-mon[32231]: pgmap v2228: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:46:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:24 smithi097 ceph-mon[32231]: pgmap v2229: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:46:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:24 smithi115 ceph-mon[35364]: pgmap v2229: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:46:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:24 smithi181 ceph-mon[35869]: pgmap v2229: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:46:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2822.672680 secs 2024-07-26T08:46:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2822.672680 secs 2024-07-26T08:46:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2822.672680 secs 2024-07-26T08:46:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:26 smithi115 ceph-mon[35364]: pgmap v2230: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:26 smithi181 ceph-mon[35869]: pgmap v2230: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:26 smithi097 ceph-mon[32231]: pgmap v2230: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:27 smithi115 ceph-mon[35364]: pgmap v2231: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:27 smithi181 ceph-mon[35869]: pgmap v2231: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:27 smithi097 ceph-mon[32231]: pgmap v2231: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2827.672842 secs 2024-07-26T08:46:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:30 smithi115 ceph-mon[35364]: pgmap v2232: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2827.672842 secs 2024-07-26T08:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:30 smithi181 ceph-mon[35869]: pgmap v2232: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2827.672842 secs 2024-07-26T08:46:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:30 smithi097 ceph-mon[32231]: pgmap v2232: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:32 smithi115 ceph-mon[35364]: pgmap v2233: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:32 smithi181 ceph-mon[35869]: pgmap v2233: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:32 smithi097 ceph-mon[32231]: pgmap v2233: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:33.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:33.598+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:46:33.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:33.599+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:46:33.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:33.599+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:46:33.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:33.599+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:46:33.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:33.600+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:35.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:34 smithi097 ceph-mon[32231]: pgmap v2234: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:34 smithi115 ceph-mon[35364]: pgmap v2234: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:34 smithi181 ceph-mon[35869]: pgmap v2234: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2832.673001 secs 2024-07-26T08:46:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2832.673001 secs 2024-07-26T08:46:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2832.673001 secs 2024-07-26T08:46:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:36 smithi115 ceph-mon[35364]: pgmap v2235: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:36 smithi181 ceph-mon[35869]: pgmap v2235: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:36 smithi097 ceph-mon[32231]: pgmap v2235: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:37 smithi115 ceph-mon[35364]: pgmap v2236: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:37 smithi181 ceph-mon[35869]: pgmap v2236: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:37 smithi097 ceph-mon[32231]: pgmap v2236: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2837.673155 secs 2024-07-26T08:46:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:40 smithi115 ceph-mon[35364]: pgmap v2237: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2837.673155 secs 2024-07-26T08:46:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:40 smithi181 ceph-mon[35869]: pgmap v2237: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2837.673155 secs 2024-07-26T08:46:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:40 smithi097 ceph-mon[32231]: pgmap v2237: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:42 smithi115 ceph-mon[35364]: pgmap v2238: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:42 smithi181 ceph-mon[35869]: pgmap v2238: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:42 smithi097 ceph-mon[32231]: pgmap v2238: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:44 smithi115 ceph-mon[35364]: pgmap v2239: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:44 smithi181 ceph-mon[35869]: pgmap v2239: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:44 smithi097 ceph-mon[32231]: pgmap v2239: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2842.673335 secs 2024-07-26T08:46:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2842.673335 secs 2024-07-26T08:46:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2842.673335 secs 2024-07-26T08:46:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:46 smithi115 ceph-mon[35364]: pgmap v2240: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:46 smithi181 ceph-mon[35869]: pgmap v2240: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:46 smithi097 ceph-mon[32231]: pgmap v2240: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:47 smithi115 ceph-mon[35364]: pgmap v2241: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:47 smithi181 ceph-mon[35869]: pgmap v2241: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:47 smithi097 ceph-mon[32231]: pgmap v2241: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:48.610+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:46:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:48.611+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:46:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:48.611+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:46:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:48.611+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:46:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:46:48.611+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2847.673471 secs 2024-07-26T08:46:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:50 smithi115 ceph-mon[35364]: pgmap v2242: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2847.673471 secs 2024-07-26T08:46:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:50 smithi181 ceph-mon[35869]: pgmap v2242: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2847.673471 secs 2024-07-26T08:46:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:50 smithi097 ceph-mon[32231]: pgmap v2242: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:46:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:52 smithi115 ceph-mon[35364]: pgmap v2243: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:52 smithi181 ceph-mon[35869]: pgmap v2243: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:52 smithi097 ceph-mon[32231]: pgmap v2243: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:46:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:54 smithi115 ceph-mon[35364]: pgmap v2244: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:54 smithi181 ceph-mon[35869]: pgmap v2244: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:54 smithi097 ceph-mon[32231]: pgmap v2244: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:46:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2852.673639 secs 2024-07-26T08:46:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2852.673639 secs 2024-07-26T08:46:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2852.673639 secs 2024-07-26T08:46:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:56 smithi115 ceph-mon[35364]: pgmap v2245: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:56 smithi181 ceph-mon[35869]: pgmap v2245: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:56 smithi097 ceph-mon[32231]: pgmap v2245: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:46:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:58.117 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:46:57 smithi181 ceph-mon[35869]: pgmap v2246: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:46:57 smithi115 ceph-mon[35364]: pgmap v2246: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:46:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:46:57 smithi097 ceph-mon[32231]: pgmap v2246: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:01.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2857.673730 secs 2024-07-26T08:47:01.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:00 smithi115 ceph-mon[35364]: pgmap v2247: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:01.226 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2857.673730 secs 2024-07-26T08:47:01.227 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:00 smithi181 ceph-mon[35869]: pgmap v2247: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2857.673730 secs 2024-07-26T08:47:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:00 smithi097 ceph-mon[32231]: pgmap v2247: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:02 smithi115 ceph-mon[35364]: pgmap v2248: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:02 smithi181 ceph-mon[35869]: pgmap v2248: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:02 smithi097 ceph-mon[32231]: pgmap v2248: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:47:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:03.615+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:47:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:03.617+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:47:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:03.617+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:47:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:03.617+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:47:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:03.617+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:03 smithi115 ceph-mon[35364]: pgmap v2249: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:47:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:03 smithi115 ceph-mon[35364]: pgmap v2250: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T08:47:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:03 smithi181 ceph-mon[35869]: pgmap v2249: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:47:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:03 smithi181 ceph-mon[35869]: pgmap v2250: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T08:47:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:03 smithi097 ceph-mon[32231]: pgmap v2249: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T08:47:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:03 smithi097 ceph-mon[32231]: pgmap v2250: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T08:47:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:04 smithi115 ceph-mon[35364]: pgmap v2251: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:04 smithi181 ceph-mon[35869]: pgmap v2251: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:04 smithi097 ceph-mon[32231]: pgmap v2251: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2862.673887 secs 2024-07-26T08:47:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2862.673887 secs 2024-07-26T08:47:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2862.673887 secs 2024-07-26T08:47:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:06 smithi115 ceph-mon[35364]: pgmap v2252: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T08:47:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:06 smithi181 ceph-mon[35869]: pgmap v2252: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T08:47:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:06 smithi097 ceph-mon[32231]: pgmap v2252: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T08:47:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:09 smithi115 ceph-mon[35364]: pgmap v2253: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:47:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:09 smithi181 ceph-mon[35869]: pgmap v2253: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:47:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:09 smithi097 ceph-mon[32231]: pgmap v2253: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:47:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2867.674047 secs 2024-07-26T08:47:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:11 smithi115 ceph-mon[35364]: pgmap v2254: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T08:47:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2867.674047 secs 2024-07-26T08:47:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:11 smithi181 ceph-mon[35869]: pgmap v2254: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T08:47:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2867.674047 secs 2024-07-26T08:47:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:11 smithi097 ceph-mon[32231]: pgmap v2254: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T08:47:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:13 smithi115 ceph-mon[35364]: pgmap v2255: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:47:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:13 smithi181 ceph-mon[35869]: pgmap v2255: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:47:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:13 smithi097 ceph-mon[32231]: pgmap v2255: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:47:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:15 smithi115 ceph-mon[35364]: pgmap v2256: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:15 smithi181 ceph-mon[35869]: pgmap v2256: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:15 smithi097 ceph-mon[32231]: pgmap v2256: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2872.674144 secs 2024-07-26T08:47:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2872.674144 secs 2024-07-26T08:47:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2872.674144 secs 2024-07-26T08:47:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:17 smithi115 ceph-mon[35364]: pgmap v2257: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:17 smithi181 ceph-mon[35869]: pgmap v2257: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:17 smithi097 ceph-mon[32231]: pgmap v2257: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:18.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:18.627+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:47:18.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:18.627+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:47:18.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:18.627+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:47:18.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:18.627+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:47:18.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:18.627+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:19 smithi115 ceph-mon[35364]: pgmap v2258: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:19 smithi181 ceph-mon[35869]: pgmap v2258: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:19 smithi097 ceph-mon[32231]: pgmap v2258: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2877.674277 secs 2024-07-26T08:47:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2877.674277 secs 2024-07-26T08:47:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2877.674277 secs 2024-07-26T08:47:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:21 smithi115 ceph-mon[35364]: pgmap v2259: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:21 smithi181 ceph-mon[35869]: pgmap v2259: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:21 smithi097 ceph-mon[32231]: pgmap v2259: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:23 smithi115 ceph-mon[35364]: pgmap v2260: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:23 smithi181 ceph-mon[35869]: pgmap v2260: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:23 smithi097 ceph-mon[32231]: pgmap v2260: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:25 smithi181 ceph-mon[35869]: pgmap v2261: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:25 smithi181 ceph-mon[35869]: 1 slow requests, 1 included below; oldest blocked for > 2882.674356 secs 2024-07-26T08:47:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:25 smithi181 ceph-mon[35869]: slow request 2882.674355 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:47:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:25 smithi115 ceph-mon[35364]: pgmap v2261: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:25 smithi115 ceph-mon[35364]: 1 slow requests, 1 included below; oldest blocked for > 2882.674356 secs 2024-07-26T08:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:25 smithi115 ceph-mon[35364]: slow request 2882.674355 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:47:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:25 smithi097 ceph-mon[32231]: pgmap v2261: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:25 smithi097 ceph-mon[32231]: 1 slow requests, 1 included below; oldest blocked for > 2882.674356 secs 2024-07-26T08:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:25 smithi097 ceph-mon[32231]: slow request 2882.674355 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:47:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:27 smithi115 ceph-mon[35364]: pgmap v2262: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:27 smithi181 ceph-mon[35869]: pgmap v2262: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:27 smithi097 ceph-mon[32231]: pgmap v2262: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:29 smithi181 ceph-mon[35869]: pgmap v2263: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:29 smithi097 ceph-mon[32231]: pgmap v2263: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:29 smithi115 ceph-mon[35364]: pgmap v2263: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2887.674523 secs 2024-07-26T08:47:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2887.674523 secs 2024-07-26T08:47:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2887.674523 secs 2024-07-26T08:47:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:31 smithi097 ceph-mon[32231]: pgmap v2264: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:31 smithi115 ceph-mon[35364]: pgmap v2264: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:31 smithi181 ceph-mon[35869]: pgmap v2264: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:33 smithi097 ceph-mon[32231]: pgmap v2265: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:47:33.632 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:33.630+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:47:33.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:33.632+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:47:33.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:33.632+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:47:33.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:33.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:47:33.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:33.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:33 smithi115 ceph-mon[35364]: pgmap v2265: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:47:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:33 smithi181 ceph-mon[35869]: pgmap v2265: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:47:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:35 smithi115 ceph-mon[35364]: pgmap v2266: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2892.674620 secs 2024-07-26T08:47:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:35 smithi181 ceph-mon[35869]: pgmap v2266: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2892.674620 secs 2024-07-26T08:47:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:35 smithi097 ceph-mon[32231]: pgmap v2266: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2892.674620 secs 2024-07-26T08:47:37.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:37.383+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:47:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:37.384+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:47:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:37.384+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:47:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:37.384+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:47:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:37 smithi115 ceph-mon[35364]: pgmap v2267: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:37 smithi181 ceph-mon[35869]: pgmap v2267: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:37 smithi097 ceph-mon[32231]: pgmap v2267: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:39 smithi115 ceph-mon[35364]: pgmap v2268: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:39 smithi181 ceph-mon[35869]: pgmap v2268: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:39 smithi097 ceph-mon[32231]: pgmap v2268: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2897.674776 secs 2024-07-26T08:47:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2897.674776 secs 2024-07-26T08:47:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2897.674776 secs 2024-07-26T08:47:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:41 smithi115 ceph-mon[35364]: pgmap v2269: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:41 smithi181 ceph-mon[35869]: pgmap v2269: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:41 smithi097 ceph-mon[32231]: pgmap v2269: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:43 smithi115 ceph-mon[35364]: pgmap v2270: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:43 smithi181 ceph-mon[35869]: pgmap v2270: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:43 smithi097 ceph-mon[32231]: pgmap v2270: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:45 smithi115 ceph-mon[35364]: pgmap v2271: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2902.674908 secs 2024-07-26T08:47:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:45 smithi181 ceph-mon[35869]: pgmap v2271: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2902.674908 secs 2024-07-26T08:47:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:45 smithi097 ceph-mon[32231]: pgmap v2271: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2902.674908 secs 2024-07-26T08:47:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:47 smithi115 ceph-mon[35364]: pgmap v2272: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:47 smithi181 ceph-mon[35869]: pgmap v2272: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:47 smithi097 ceph-mon[32231]: pgmap v2272: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:48.641+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:47:48.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:48.642+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:47:48.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:48.642+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:47:48.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:48.642+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:47:48.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:47:48.642+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:49 smithi115 ceph-mon[35364]: pgmap v2273: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:49 smithi181 ceph-mon[35869]: pgmap v2273: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:49 smithi097 ceph-mon[32231]: pgmap v2273: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2907.675064 secs 2024-07-26T08:47:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2907.675064 secs 2024-07-26T08:47:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2907.675064 secs 2024-07-26T08:47:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:51 smithi115 ceph-mon[35364]: pgmap v2274: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:51 smithi181 ceph-mon[35869]: pgmap v2274: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:51 smithi097 ceph-mon[32231]: pgmap v2274: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:53 smithi115 ceph-mon[35364]: pgmap v2275: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:53 smithi181 ceph-mon[35869]: pgmap v2275: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:53 smithi097 ceph-mon[32231]: pgmap v2275: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:55 smithi115 ceph-mon[35364]: pgmap v2276: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2912.675219 secs 2024-07-26T08:47:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:55 smithi181 ceph-mon[35869]: pgmap v2276: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2912.675219 secs 2024-07-26T08:47:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:55 smithi097 ceph-mon[32231]: pgmap v2276: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2912.675219 secs 2024-07-26T08:47:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:57 smithi115 ceph-mon[35364]: pgmap v2277: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:57 smithi181 ceph-mon[35869]: pgmap v2277: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:57 smithi097 ceph-mon[32231]: pgmap v2277: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:47:59 smithi115 ceph-mon[35364]: pgmap v2278: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:47:59 smithi181 ceph-mon[35869]: pgmap v2278: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:47:59 smithi097 ceph-mon[32231]: pgmap v2278: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2917.675380 secs 2024-07-26T08:48:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2917.675380 secs 2024-07-26T08:48:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2917.675380 secs 2024-07-26T08:48:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:01 smithi115 ceph-mon[35364]: pgmap v2279: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:01 smithi181 ceph-mon[35869]: pgmap v2279: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:01 smithi097 ceph-mon[32231]: pgmap v2279: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:03.645+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:48:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:03.646+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:48:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:03.646+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:48:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:03.646+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:48:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:03.646+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:03 smithi115 ceph-mon[35364]: pgmap v2280: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:48:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:03 smithi181 ceph-mon[35869]: pgmap v2280: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:48:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:03 smithi097 ceph-mon[32231]: pgmap v2280: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:48:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: pgmap v2281: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: pgmap v2282: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:48:04.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:48:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: pgmap v2281: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:04.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: pgmap v2282: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:48:04.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:48:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: pgmap v2281: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: pgmap v2282: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:48:04.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:48:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2922.675540 secs 2024-07-26T08:48:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2922.675540 secs 2024-07-26T08:48:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2922.675540 secs 2024-07-26T08:48:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:06 smithi115 ceph-mon[35364]: pgmap v2283: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:06 smithi181 ceph-mon[35869]: pgmap v2283: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:06 smithi097 ceph-mon[32231]: pgmap v2283: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:08 smithi115 ceph-mon[35364]: pgmap v2284: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:08 smithi181 ceph-mon[35869]: pgmap v2284: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:08 smithi097 ceph-mon[32231]: pgmap v2284: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:10 smithi115 ceph-mon[35364]: pgmap v2285: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2927.675703 secs 2024-07-26T08:48:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:10 smithi181 ceph-mon[35869]: pgmap v2285: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2927.675703 secs 2024-07-26T08:48:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:10 smithi097 ceph-mon[32231]: pgmap v2285: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2927.675703 secs 2024-07-26T08:48:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:12 smithi115 ceph-mon[35364]: pgmap v2286: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:12 smithi181 ceph-mon[35869]: pgmap v2286: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:12 smithi097 ceph-mon[32231]: pgmap v2286: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:14 smithi115 ceph-mon[35364]: pgmap v2287: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:48:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:14 smithi181 ceph-mon[35869]: pgmap v2287: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:48:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:14 smithi097 ceph-mon[32231]: pgmap v2287: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:48:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2932.675866 secs 2024-07-26T08:48:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2932.675866 secs 2024-07-26T08:48:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2932.675866 secs 2024-07-26T08:48:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:16 smithi115 ceph-mon[35364]: pgmap v2288: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:16 smithi181 ceph-mon[35869]: pgmap v2288: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:16 smithi097 ceph-mon[32231]: pgmap v2288: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:18.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:18.657+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:48:18.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:18.659+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:48:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:18.659+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:48:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:18.659+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:48:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:18.659+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:18 smithi115 ceph-mon[35364]: pgmap v2289: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:18 smithi181 ceph-mon[35869]: pgmap v2289: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:18 smithi097 ceph-mon[32231]: pgmap v2289: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:20 smithi115 ceph-mon[35364]: pgmap v2290: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2937.676003 secs 2024-07-26T08:48:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:20 smithi181 ceph-mon[35869]: pgmap v2290: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2937.676003 secs 2024-07-26T08:48:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:20 smithi097 ceph-mon[32231]: pgmap v2290: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2937.676003 secs 2024-07-26T08:48:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:22 smithi115 ceph-mon[35364]: pgmap v2291: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:22 smithi181 ceph-mon[35869]: pgmap v2291: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:22 smithi097 ceph-mon[32231]: pgmap v2291: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:24 smithi115 ceph-mon[35364]: pgmap v2292: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:24 smithi181 ceph-mon[35869]: pgmap v2292: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:24 smithi097 ceph-mon[32231]: pgmap v2292: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2942.676133 secs 2024-07-26T08:48:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2942.676133 secs 2024-07-26T08:48:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2942.676133 secs 2024-07-26T08:48:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:26 smithi115 ceph-mon[35364]: pgmap v2293: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:26 smithi181 ceph-mon[35869]: pgmap v2293: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:26 smithi097 ceph-mon[32231]: pgmap v2293: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:28 smithi115 ceph-mon[35364]: pgmap v2294: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:28 smithi181 ceph-mon[35869]: pgmap v2294: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:28 smithi097 ceph-mon[32231]: pgmap v2294: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:30 smithi115 ceph-mon[35364]: pgmap v2295: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2947.676325 secs 2024-07-26T08:48:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:30 smithi181 ceph-mon[35869]: pgmap v2295: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2947.676325 secs 2024-07-26T08:48:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:30 smithi097 ceph-mon[32231]: pgmap v2295: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2947.676325 secs 2024-07-26T08:48:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:32 smithi115 ceph-mon[35364]: pgmap v2296: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:32 smithi181 ceph-mon[35869]: pgmap v2296: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:32 smithi097 ceph-mon[32231]: pgmap v2296: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:33.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:33.663+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:48:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:33.665+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:48:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:33.665+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:48:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:33.665+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:48:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:33.665+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:34 smithi181 ceph-mon[35869]: pgmap v2297: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:34 smithi097 ceph-mon[32231]: pgmap v2297: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:34 smithi115 ceph-mon[35364]: pgmap v2297: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2952.676417 secs 2024-07-26T08:48:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2952.676417 secs 2024-07-26T08:48:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2952.676417 secs 2024-07-26T08:48:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:36 smithi097 ceph-mon[32231]: pgmap v2298: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:36 smithi115 ceph-mon[35364]: pgmap v2298: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:36 smithi181 ceph-mon[35869]: pgmap v2298: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:38 smithi097 ceph-mon[32231]: pgmap v2299: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:38 smithi115 ceph-mon[35364]: pgmap v2299: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:39.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:38 smithi181 ceph-mon[35869]: pgmap v2299: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:40 smithi115 ceph-mon[35364]: pgmap v2300: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2957.676495 secs 2024-07-26T08:48:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:40 smithi181 ceph-mon[35869]: pgmap v2300: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2957.676495 secs 2024-07-26T08:48:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:40 smithi097 ceph-mon[32231]: pgmap v2300: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2957.676495 secs 2024-07-26T08:48:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:42 smithi115 ceph-mon[35364]: pgmap v2301: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:42 smithi181 ceph-mon[35869]: pgmap v2301: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:42 smithi097 ceph-mon[32231]: pgmap v2301: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:44 smithi115 ceph-mon[35364]: pgmap v2302: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:44 smithi181 ceph-mon[35869]: pgmap v2302: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:44 smithi097 ceph-mon[32231]: pgmap v2302: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2962.676575 secs 2024-07-26T08:48:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2962.676575 secs 2024-07-26T08:48:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2962.676575 secs 2024-07-26T08:48:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:46 smithi115 ceph-mon[35364]: pgmap v2303: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:46 smithi181 ceph-mon[35869]: pgmap v2303: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:46 smithi097 ceph-mon[32231]: pgmap v2303: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:48.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:48.674+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:48:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:48.675+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:48:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:48.675+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:48:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:48.675+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:48:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:48:48.675+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:48 smithi115 ceph-mon[35364]: pgmap v2304: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:48 smithi181 ceph-mon[35869]: pgmap v2304: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:48 smithi097 ceph-mon[32231]: pgmap v2304: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:50 smithi115 ceph-mon[35364]: pgmap v2305: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2967.676653 secs 2024-07-26T08:48:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:50 smithi181 ceph-mon[35869]: pgmap v2305: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2967.676653 secs 2024-07-26T08:48:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:50 smithi097 ceph-mon[32231]: pgmap v2305: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2967.676653 secs 2024-07-26T08:48:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:52 smithi115 ceph-mon[35364]: pgmap v2306: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:52 smithi181 ceph-mon[35869]: pgmap v2306: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:52 smithi097 ceph-mon[32231]: pgmap v2306: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:54 smithi115 ceph-mon[35364]: pgmap v2307: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:54 smithi181 ceph-mon[35869]: pgmap v2307: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:54 smithi097 ceph-mon[32231]: pgmap v2307: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2972.676789 secs 2024-07-26T08:48:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2972.676789 secs 2024-07-26T08:48:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2972.676789 secs 2024-07-26T08:48:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:56 smithi115 ceph-mon[35364]: pgmap v2308: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:56 smithi181 ceph-mon[35869]: pgmap v2308: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:56 smithi097 ceph-mon[32231]: pgmap v2308: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:48:58 smithi115 ceph-mon[35364]: pgmap v2309: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:48:58 smithi181 ceph-mon[35869]: pgmap v2309: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:48:58 smithi097 ceph-mon[32231]: pgmap v2309: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:00 smithi115 ceph-mon[35364]: pgmap v2310: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2977.676921 secs 2024-07-26T08:49:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:00 smithi181 ceph-mon[35869]: pgmap v2310: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2977.676921 secs 2024-07-26T08:49:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:00 smithi097 ceph-mon[32231]: pgmap v2310: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2977.676921 secs 2024-07-26T08:49:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:02 smithi115 ceph-mon[35364]: pgmap v2311: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:02 smithi181 ceph-mon[35869]: pgmap v2311: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:02 smithi097 ceph-mon[32231]: pgmap v2311: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:03.679+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:49:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:03.680+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:49:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:03.680+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:49:03.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:03.680+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:49:03.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:03.681+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:04 smithi115 ceph-mon[35364]: pgmap v2312: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:04 smithi181 ceph-mon[35869]: pgmap v2312: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:04 smithi097 ceph-mon[32231]: pgmap v2312: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2982.677011 secs 2024-07-26T08:49:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:06.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:49:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2982.677011 secs 2024-07-26T08:49:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:49:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2982.677011 secs 2024-07-26T08:49:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:49:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:49:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:06 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:49:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:06 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:49:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:06 smithi115 ceph-mon[35364]: pgmap v2313: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:49:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:06 smithi115 ceph-mon[35364]: pgmap v2314: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:06 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:49:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:06 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:49:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:06 smithi181 ceph-mon[35869]: pgmap v2313: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:49:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:06 smithi181 ceph-mon[35869]: pgmap v2314: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:06 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:49:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:06 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:49:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:06 smithi097 ceph-mon[32231]: pgmap v2313: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:49:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:06 smithi097 ceph-mon[32231]: pgmap v2314: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:07.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:07.399+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:49:07.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:07.399+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:49:07.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:07.399+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:49:07.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:07.399+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:49:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:07 smithi115 ceph-mon[35364]: pgmap v2315: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:07 smithi181 ceph-mon[35869]: pgmap v2315: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:07 smithi097 ceph-mon[32231]: pgmap v2315: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:10 smithi115 ceph-mon[35364]: pgmap v2316: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:10 smithi181 ceph-mon[35869]: pgmap v2316: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:10 smithi097 ceph-mon[32231]: pgmap v2316: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2987.677119 secs 2024-07-26T08:49:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2987.677119 secs 2024-07-26T08:49:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2987.677119 secs 2024-07-26T08:49:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:12 smithi115 ceph-mon[35364]: pgmap v2317: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:12 smithi181 ceph-mon[35869]: pgmap v2317: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:12 smithi097 ceph-mon[32231]: pgmap v2317: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:14 smithi115 ceph-mon[35364]: pgmap v2318: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:14 smithi181 ceph-mon[35869]: pgmap v2318: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:14 smithi097 ceph-mon[32231]: pgmap v2318: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2992.677304 secs 2024-07-26T08:49:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2992.677304 secs 2024-07-26T08:49:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2992.677304 secs 2024-07-26T08:49:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:16 smithi115 ceph-mon[35364]: pgmap v2319: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:49:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:16 smithi181 ceph-mon[35869]: pgmap v2319: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:49:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:16 smithi097 ceph-mon[32231]: pgmap v2319: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:49:18.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:18.689+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:49:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:18.690+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:49:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:18.690+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:49:18.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:18.690+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:49:18.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:18.691+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:18 smithi115 ceph-mon[35364]: pgmap v2320: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:18 smithi181 ceph-mon[35869]: pgmap v2320: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:18 smithi097 ceph-mon[32231]: pgmap v2320: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:20 smithi181 ceph-mon[35869]: pgmap v2321: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 2997.677433 secs 2024-07-26T08:49:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:20 smithi097 ceph-mon[32231]: pgmap v2321: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 2997.677433 secs 2024-07-26T08:49:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:20 smithi115 ceph-mon[35364]: pgmap v2321: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 2997.677433 secs 2024-07-26T08:49:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:22 smithi097 ceph-mon[32231]: pgmap v2322: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:22 smithi115 ceph-mon[35364]: pgmap v2322: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:22 smithi181 ceph-mon[35869]: pgmap v2322: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:24 smithi097 ceph-mon[32231]: pgmap v2323: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:24 smithi115 ceph-mon[35364]: pgmap v2323: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:24 smithi181 ceph-mon[35869]: pgmap v2323: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:25.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3002.677569 secs 2024-07-26T08:49:25.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3002.677569 secs 2024-07-26T08:49:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3002.677569 secs 2024-07-26T08:49:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:26 smithi115 ceph-mon[35364]: pgmap v2324: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:26 smithi181 ceph-mon[35869]: pgmap v2324: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:26 smithi097 ceph-mon[32231]: pgmap v2324: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:28 smithi115 ceph-mon[35364]: pgmap v2325: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:28 smithi181 ceph-mon[35869]: pgmap v2325: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:28 smithi097 ceph-mon[32231]: pgmap v2325: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:30 smithi115 ceph-mon[35364]: pgmap v2326: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3007.677665 secs 2024-07-26T08:49:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:30 smithi181 ceph-mon[35869]: pgmap v2326: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3007.677665 secs 2024-07-26T08:49:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:30 smithi097 ceph-mon[32231]: pgmap v2326: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3007.677665 secs 2024-07-26T08:49:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:32 smithi115 ceph-mon[35364]: pgmap v2327: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:32 smithi181 ceph-mon[35869]: pgmap v2327: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:32 smithi097 ceph-mon[32231]: pgmap v2327: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:33.697 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:33.695+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:49:33.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:33.697+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:49:33.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:33.697+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:49:33.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:33.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:49:33.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:33.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:34 smithi115 ceph-mon[35364]: pgmap v2328: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:49:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:34 smithi181 ceph-mon[35869]: pgmap v2328: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:49:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:34 smithi097 ceph-mon[32231]: pgmap v2328: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:49:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3012.677801 secs 2024-07-26T08:49:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3012.677801 secs 2024-07-26T08:49:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3012.677801 secs 2024-07-26T08:49:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:36 smithi115 ceph-mon[35364]: pgmap v2329: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:49:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:36 smithi181 ceph-mon[35869]: pgmap v2329: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:49:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:36 smithi097 ceph-mon[32231]: pgmap v2329: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:49:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:38 smithi115 ceph-mon[35364]: pgmap v2330: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:38 smithi181 ceph-mon[35869]: pgmap v2330: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:38 smithi097 ceph-mon[32231]: pgmap v2330: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:40 smithi115 ceph-mon[35364]: pgmap v2331: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:49:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3017.677935 secs 2024-07-26T08:49:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:40 smithi181 ceph-mon[35869]: pgmap v2331: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:49:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3017.677935 secs 2024-07-26T08:49:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:40 smithi097 ceph-mon[32231]: pgmap v2331: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:49:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3017.677935 secs 2024-07-26T08:49:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:42 smithi115 ceph-mon[35364]: pgmap v2332: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:42 smithi181 ceph-mon[35869]: pgmap v2332: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:42 smithi097 ceph-mon[32231]: pgmap v2332: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:44 smithi115 ceph-mon[35364]: pgmap v2333: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:44 smithi181 ceph-mon[35869]: pgmap v2333: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:44 smithi097 ceph-mon[32231]: pgmap v2333: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3022.678077 secs 2024-07-26T08:49:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3022.678077 secs 2024-07-26T08:49:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3022.678077 secs 2024-07-26T08:49:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:46 smithi115 ceph-mon[35364]: pgmap v2334: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:46 smithi181 ceph-mon[35869]: pgmap v2334: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:46 smithi097 ceph-mon[32231]: pgmap v2334: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:48.701+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:49:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:48.702+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:49:48.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:48.702+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:49:48.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:48.702+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:49:48.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:49:48.702+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:48 smithi115 ceph-mon[35364]: pgmap v2335: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:48 smithi181 ceph-mon[35869]: pgmap v2335: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:48 smithi097 ceph-mon[32231]: pgmap v2335: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:50 smithi115 ceph-mon[35364]: pgmap v2336: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3027.678215 secs 2024-07-26T08:49:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:50 smithi181 ceph-mon[35869]: pgmap v2336: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3027.678215 secs 2024-07-26T08:49:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:50 smithi097 ceph-mon[32231]: pgmap v2336: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:49:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3027.678215 secs 2024-07-26T08:49:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:52 smithi115 ceph-mon[35364]: pgmap v2337: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:52 smithi181 ceph-mon[35869]: pgmap v2337: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:52 smithi097 ceph-mon[32231]: pgmap v2337: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:49:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:54 smithi115 ceph-mon[35364]: pgmap v2338: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:54 smithi181 ceph-mon[35869]: pgmap v2338: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:54 smithi097 ceph-mon[32231]: pgmap v2338: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:55.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3032.678355 secs 2024-07-26T08:49:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3032.678355 secs 2024-07-26T08:49:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3032.678355 secs 2024-07-26T08:49:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:56 smithi115 ceph-mon[35364]: pgmap v2339: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:56 smithi181 ceph-mon[35869]: pgmap v2339: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:56 smithi097 ceph-mon[32231]: pgmap v2339: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:49:58 smithi115 ceph-mon[35364]: pgmap v2340: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:49:58 smithi181 ceph-mon[35869]: pgmap v2340: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:49:58 smithi097 ceph-mon[32231]: pgmap v2340: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:00 smithi115 ceph-mon[35364]: pgmap v2341: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3037.678494 secs 2024-07-26T08:50:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:50:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:00 smithi181 ceph-mon[35869]: pgmap v2341: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3037.678494 secs 2024-07-26T08:50:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:50:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:00 smithi097 ceph-mon[32231]: pgmap v2341: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3037.678494 secs 2024-07-26T08:50:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:50:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:02 smithi115 ceph-mon[35364]: pgmap v2342: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:02 smithi181 ceph-mon[35869]: pgmap v2342: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:02 smithi097 ceph-mon[32231]: pgmap v2342: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:03.706+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:50:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:03.706+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:50:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:03.706+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:50:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:03.706+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:50:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:03.706+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:04 smithi115 ceph-mon[35364]: pgmap v2343: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:04 smithi181 ceph-mon[35869]: pgmap v2343: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:04 smithi097 ceph-mon[32231]: pgmap v2343: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3042.678596 secs 2024-07-26T08:50:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3042.678596 secs 2024-07-26T08:50:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3042.678596 secs 2024-07-26T08:50:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:06 smithi115 ceph-mon[35364]: pgmap v2344: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:06 smithi181 ceph-mon[35869]: pgmap v2344: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:06 smithi097 ceph-mon[32231]: pgmap v2344: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:50:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:08 smithi115 ceph-mon[35364]: pgmap v2345: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:08 smithi115 ceph-mon[35364]: pgmap v2346: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:08 smithi181 ceph-mon[35869]: pgmap v2345: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:08 smithi181 ceph-mon[35869]: pgmap v2346: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:08 smithi097 ceph-mon[32231]: pgmap v2345: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:08 smithi097 ceph-mon[32231]: pgmap v2346: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:10 smithi115 ceph-mon[35364]: pgmap v2347: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3047.678726 secs 2024-07-26T08:50:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:10 smithi181 ceph-mon[35869]: pgmap v2347: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3047.678726 secs 2024-07-26T08:50:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:10 smithi097 ceph-mon[32231]: pgmap v2347: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3047.678726 secs 2024-07-26T08:50:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:12 smithi181 ceph-mon[35869]: pgmap v2348: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:12 smithi097 ceph-mon[32231]: pgmap v2348: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:12 smithi115 ceph-mon[35364]: pgmap v2348: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:14 smithi097 ceph-mon[32231]: pgmap v2349: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:14 smithi115 ceph-mon[35364]: pgmap v2349: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:14 smithi181 ceph-mon[35869]: pgmap v2349: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3052.678859 secs 2024-07-26T08:50:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3052.678859 secs 2024-07-26T08:50:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3052.678859 secs 2024-07-26T08:50:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:16 smithi115 ceph-mon[35364]: pgmap v2350: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:16 smithi181 ceph-mon[35869]: pgmap v2350: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:16 smithi097 ceph-mon[32231]: pgmap v2350: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:18.716+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:50:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:18.717+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:50:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:18.717+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:50:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:18.717+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:50:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:18.717+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:18 smithi115 ceph-mon[35364]: pgmap v2351: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:18 smithi181 ceph-mon[35869]: pgmap v2351: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:18 smithi097 ceph-mon[32231]: pgmap v2351: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:20 smithi115 ceph-mon[35364]: pgmap v2352: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3057.678994 secs 2024-07-26T08:50:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:20 smithi181 ceph-mon[35869]: pgmap v2352: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3057.678994 secs 2024-07-26T08:50:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:20 smithi097 ceph-mon[32231]: pgmap v2352: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3057.678994 secs 2024-07-26T08:50:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:22 smithi115 ceph-mon[35364]: pgmap v2353: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:22 smithi181 ceph-mon[35869]: pgmap v2353: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:22 smithi097 ceph-mon[32231]: pgmap v2353: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:25.029 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:24 smithi115 ceph-mon[35364]: pgmap v2354: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:25.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:24 smithi181 ceph-mon[35869]: pgmap v2354: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:25.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:24 smithi097 ceph-mon[32231]: pgmap v2354: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3062.679134 secs 2024-07-26T08:50:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3062.679134 secs 2024-07-26T08:50:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3062.679134 secs 2024-07-26T08:50:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:26 smithi115 ceph-mon[35364]: pgmap v2355: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:26 smithi181 ceph-mon[35869]: pgmap v2355: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:26 smithi097 ceph-mon[32231]: pgmap v2355: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:28 smithi115 ceph-mon[35364]: pgmap v2356: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:28 smithi181 ceph-mon[35869]: pgmap v2356: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:28 smithi097 ceph-mon[32231]: pgmap v2356: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:30 smithi115 ceph-mon[35364]: pgmap v2357: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:50:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3067.679287 secs 2024-07-26T08:50:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:30 smithi181 ceph-mon[35869]: pgmap v2357: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:50:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3067.679287 secs 2024-07-26T08:50:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:30 smithi097 ceph-mon[32231]: pgmap v2357: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:50:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3067.679287 secs 2024-07-26T08:50:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:32 smithi115 ceph-mon[35364]: pgmap v2358: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:50:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:32 smithi181 ceph-mon[35869]: pgmap v2358: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:50:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:32 smithi097 ceph-mon[32231]: pgmap v2358: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:50:33.724 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:33.722+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:50:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:33.723+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:50:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:33.723+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:50:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:33.723+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:50:33.725 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:33.723+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:34 smithi115 ceph-mon[35364]: pgmap v2359: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:34 smithi181 ceph-mon[35869]: pgmap v2359: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:34 smithi097 ceph-mon[32231]: pgmap v2359: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3072.679420 secs 2024-07-26T08:50:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3072.679420 secs 2024-07-26T08:50:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3072.679420 secs 2024-07-26T08:50:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:36 smithi115 ceph-mon[35364]: pgmap v2360: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:36 smithi181 ceph-mon[35869]: pgmap v2360: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:36 smithi097 ceph-mon[32231]: pgmap v2360: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:50:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:38 smithi115 ceph-mon[35364]: pgmap v2361: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:50:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:38 smithi181 ceph-mon[35869]: pgmap v2361: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:50:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:38 smithi097 ceph-mon[32231]: pgmap v2361: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:50:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:40 smithi115 ceph-mon[35364]: pgmap v2362: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:50:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3077.679553 secs 2024-07-26T08:50:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:40 smithi181 ceph-mon[35869]: pgmap v2362: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:50:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3077.679553 secs 2024-07-26T08:50:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:40 smithi097 ceph-mon[32231]: pgmap v2362: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:50:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3077.679553 secs 2024-07-26T08:50:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:42 smithi115 ceph-mon[35364]: pgmap v2363: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:42 smithi181 ceph-mon[35869]: pgmap v2363: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:42 smithi097 ceph-mon[32231]: pgmap v2363: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:44 smithi115 ceph-mon[35364]: pgmap v2364: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:44 smithi181 ceph-mon[35869]: pgmap v2364: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:44 smithi097 ceph-mon[32231]: pgmap v2364: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3082.679689 secs 2024-07-26T08:50:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3082.679689 secs 2024-07-26T08:50:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3082.679689 secs 2024-07-26T08:50:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:46 smithi115 ceph-mon[35364]: pgmap v2365: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:46 smithi181 ceph-mon[35869]: pgmap v2365: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:46 smithi097 ceph-mon[32231]: pgmap v2365: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:47 smithi115 ceph-mon[35364]: pgmap v2366: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:47 smithi181 ceph-mon[35869]: pgmap v2366: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:47 smithi097 ceph-mon[32231]: pgmap v2366: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:48.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:48.733+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:50:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:48.734+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:50:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:48.734+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:50:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:48.734+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:50:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:50:48.734+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:50 smithi115 ceph-mon[35364]: pgmap v2367: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:50 smithi181 ceph-mon[35869]: pgmap v2367: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:50 smithi097 ceph-mon[32231]: pgmap v2367: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3087.679820 secs 2024-07-26T08:50:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3087.679820 secs 2024-07-26T08:50:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3087.679820 secs 2024-07-26T08:50:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:52 smithi115 ceph-mon[35364]: pgmap v2368: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:52 smithi181 ceph-mon[35869]: pgmap v2368: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:52 smithi097 ceph-mon[32231]: pgmap v2368: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:54 smithi115 ceph-mon[35364]: pgmap v2369: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:54 smithi181 ceph-mon[35869]: pgmap v2369: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:54 smithi097 ceph-mon[32231]: pgmap v2369: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3092.679955 secs 2024-07-26T08:50:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:56 smithi115 ceph-mon[35364]: pgmap v2370: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3092.679955 secs 2024-07-26T08:50:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:56 smithi181 ceph-mon[35869]: pgmap v2370: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3092.679955 secs 2024-07-26T08:50:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:56 smithi097 ceph-mon[32231]: pgmap v2370: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:50:58 smithi115 ceph-mon[35364]: pgmap v2371: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:50:58 smithi181 ceph-mon[35869]: pgmap v2371: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:50:58 smithi097 ceph-mon[32231]: pgmap v2371: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:00 smithi115 ceph-mon[35364]: pgmap v2372: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:00 smithi181 ceph-mon[35869]: pgmap v2372: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:00 smithi097 ceph-mon[32231]: pgmap v2372: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3097.680088 secs 2024-07-26T08:51:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3097.680088 secs 2024-07-26T08:51:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3097.680088 secs 2024-07-26T08:51:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:02 smithi115 ceph-mon[35364]: pgmap v2373: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:02 smithi181 ceph-mon[35869]: pgmap v2373: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:02 smithi097 ceph-mon[32231]: pgmap v2373: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:03.737+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:51:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:03.738+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:51:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:03.738+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:51:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:03.738+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:51:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:03.739+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:04 smithi115 ceph-mon[35364]: pgmap v2374: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:04 smithi181 ceph-mon[35869]: pgmap v2374: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:04 smithi097 ceph-mon[32231]: pgmap v2374: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3102.680225 secs 2024-07-26T08:51:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:06 smithi115 ceph-mon[35364]: pgmap v2375: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3102.680225 secs 2024-07-26T08:51:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:06 smithi181 ceph-mon[35869]: pgmap v2375: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3102.680225 secs 2024-07-26T08:51:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:06 smithi097 ceph-mon[32231]: pgmap v2375: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:08 smithi115 ceph-mon[35364]: pgmap v2376: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:08 smithi181 ceph-mon[35869]: pgmap v2376: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:08 smithi097 ceph-mon[32231]: pgmap v2376: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:51:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:10 smithi115 ceph-mon[35364]: pgmap v2377: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:10.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:10 smithi115 ceph-mon[35364]: pgmap v2378: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:10.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:10 smithi181 ceph-mon[35869]: pgmap v2377: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:10.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:10 smithi181 ceph-mon[35869]: pgmap v2378: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:10 smithi097 ceph-mon[32231]: pgmap v2377: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:10 smithi097 ceph-mon[32231]: pgmap v2378: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3107.680341 secs 2024-07-26T08:51:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3107.680341 secs 2024-07-26T08:51:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3107.680341 secs 2024-07-26T08:51:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:12 smithi181 ceph-mon[35869]: pgmap v2379: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:12 smithi097 ceph-mon[32231]: pgmap v2379: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:12 smithi115 ceph-mon[35364]: pgmap v2379: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:14 smithi097 ceph-mon[32231]: pgmap v2380: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:14 smithi115 ceph-mon[35364]: pgmap v2380: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:14 smithi181 ceph-mon[35869]: pgmap v2380: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3112.680505 secs 2024-07-26T08:51:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3112.680505 secs 2024-07-26T08:51:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3112.680505 secs 2024-07-26T08:51:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:16 smithi115 ceph-mon[35364]: pgmap v2381: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:16 smithi181 ceph-mon[35869]: pgmap v2381: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:16 smithi097 ceph-mon[32231]: pgmap v2381: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:18 smithi115 ceph-mon[35364]: pgmap v2382: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:18 smithi181 ceph-mon[35869]: pgmap v2382: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:18.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:18.749+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:51:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:18 smithi097 ceph-mon[32231]: pgmap v2382: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:51:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:18.750+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:51:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:18.750+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:51:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:18.750+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:51:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:18.751+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:20.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:20 smithi115 ceph-mon[35364]: pgmap v2383: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:51:20.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3117.680645 secs 2024-07-26T08:51:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:20 smithi181 ceph-mon[35869]: pgmap v2383: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:51:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3117.680645 secs 2024-07-26T08:51:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:20 smithi097 ceph-mon[32231]: pgmap v2383: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:51:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3117.680645 secs 2024-07-26T08:51:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:22 smithi115 ceph-mon[35364]: pgmap v2384: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:22 smithi181 ceph-mon[35869]: pgmap v2384: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:22 smithi097 ceph-mon[32231]: pgmap v2384: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:24 smithi115 ceph-mon[35364]: pgmap v2385: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:24 smithi181 ceph-mon[35869]: pgmap v2385: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:24 smithi097 ceph-mon[32231]: pgmap v2385: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3122.680789 secs 2024-07-26T08:51:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3122.680789 secs 2024-07-26T08:51:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3122.680789 secs 2024-07-26T08:51:26.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:26 smithi115 ceph-mon[35364]: pgmap v2386: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:51:26.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:26 smithi181 ceph-mon[35869]: pgmap v2386: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:51:26.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:26 smithi097 ceph-mon[32231]: pgmap v2386: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:51:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:28 smithi115 ceph-mon[35364]: pgmap v2387: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:51:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:28 smithi181 ceph-mon[35869]: pgmap v2387: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:51:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:28 smithi097 ceph-mon[32231]: pgmap v2387: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:51:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:30 smithi115 ceph-mon[35364]: pgmap v2388: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:51:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3127.680921 secs 2024-07-26T08:51:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:30 smithi181 ceph-mon[35869]: pgmap v2388: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:51:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3127.680921 secs 2024-07-26T08:51:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:30 smithi097 ceph-mon[32231]: pgmap v2388: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:51:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3127.680921 secs 2024-07-26T08:51:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:32 smithi115 ceph-mon[35364]: pgmap v2389: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:32 smithi181 ceph-mon[35869]: pgmap v2389: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:32 smithi097 ceph-mon[32231]: pgmap v2389: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:33.753+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:33.754+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:33.754+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:33.754+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:33.754+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:34 smithi115 ceph-mon[35364]: pgmap v2390: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:51:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:34 smithi181 ceph-mon[35869]: pgmap v2390: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:51:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:34 smithi097 ceph-mon[32231]: pgmap v2390: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:51:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3132.681085 secs 2024-07-26T08:51:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3132.681085 secs 2024-07-26T08:51:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3132.681085 secs 2024-07-26T08:51:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:36 smithi115 ceph-mon[35364]: pgmap v2391: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:51:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:36 smithi181 ceph-mon[35869]: pgmap v2391: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:51:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:36 smithi097 ceph-mon[32231]: pgmap v2391: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:51:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:38 smithi115 ceph-mon[35364]: pgmap v2392: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:38 smithi181 ceph-mon[35869]: pgmap v2392: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:38 smithi097 ceph-mon[32231]: pgmap v2392: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:40 smithi115 ceph-mon[35364]: pgmap v2393: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3137.681223 secs 2024-07-26T08:51:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:40 smithi181 ceph-mon[35869]: pgmap v2393: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3137.681223 secs 2024-07-26T08:51:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:40 smithi097 ceph-mon[32231]: pgmap v2393: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3137.681223 secs 2024-07-26T08:51:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:42 smithi115 ceph-mon[35364]: pgmap v2394: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:42 smithi181 ceph-mon[35869]: pgmap v2394: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:42 smithi097 ceph-mon[32231]: pgmap v2394: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:44 smithi115 ceph-mon[35364]: pgmap v2395: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:44 smithi181 ceph-mon[35869]: pgmap v2395: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:44 smithi097 ceph-mon[32231]: pgmap v2395: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3142.681368 secs 2024-07-26T08:51:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3142.681368 secs 2024-07-26T08:51:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3142.681368 secs 2024-07-26T08:51:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:46 smithi115 ceph-mon[35364]: pgmap v2396: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:46 smithi181 ceph-mon[35869]: pgmap v2396: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:46 smithi097 ceph-mon[32231]: pgmap v2396: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:48 smithi115 ceph-mon[35364]: pgmap v2397: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:48 smithi181 ceph-mon[35869]: pgmap v2397: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:48 smithi097 ceph-mon[32231]: pgmap v2397: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:48.763+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:51:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:48.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:51:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:48.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:51:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:48.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:51:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:51:48.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:50 smithi115 ceph-mon[35364]: pgmap v2398: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3147.681515 secs 2024-07-26T08:51:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:50 smithi181 ceph-mon[35869]: pgmap v2398: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3147.681515 secs 2024-07-26T08:51:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:50 smithi097 ceph-mon[32231]: pgmap v2398: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3147.681515 secs 2024-07-26T08:51:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:52 smithi115 ceph-mon[35364]: pgmap v2399: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:52 smithi181 ceph-mon[35869]: pgmap v2399: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:52 smithi097 ceph-mon[32231]: pgmap v2399: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:54 smithi115 ceph-mon[35364]: pgmap v2400: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:54 smithi181 ceph-mon[35869]: pgmap v2400: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:54 smithi097 ceph-mon[32231]: pgmap v2400: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3152.681612 secs 2024-07-26T08:51:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3152.681612 secs 2024-07-26T08:51:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3152.681612 secs 2024-07-26T08:51:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:56 smithi115 ceph-mon[35364]: pgmap v2401: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:56 smithi181 ceph-mon[35869]: pgmap v2401: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:56 smithi097 ceph-mon[32231]: pgmap v2401: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:51:58 smithi115 ceph-mon[35364]: pgmap v2402: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:51:58 smithi181 ceph-mon[35869]: pgmap v2402: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:51:58 smithi097 ceph-mon[32231]: pgmap v2402: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:00 smithi115 ceph-mon[35364]: pgmap v2403: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3157.681704 secs 2024-07-26T08:52:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:00 smithi181 ceph-mon[35869]: pgmap v2403: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3157.681704 secs 2024-07-26T08:52:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:00 smithi097 ceph-mon[32231]: pgmap v2403: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3157.681704 secs 2024-07-26T08:52:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:02 smithi115 ceph-mon[35364]: pgmap v2404: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:02 smithi181 ceph-mon[35869]: pgmap v2404: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:02 smithi097 ceph-mon[32231]: pgmap v2404: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:03.769+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:52:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:03.770+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:52:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:03.770+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:52:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:03.770+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:52:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:03.770+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:04 smithi115 ceph-mon[35364]: pgmap v2405: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:04 smithi181 ceph-mon[35869]: pgmap v2405: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:04 smithi097 ceph-mon[32231]: pgmap v2405: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3162.681843 secs 2024-07-26T08:52:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3162.681843 secs 2024-07-26T08:52:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3162.681843 secs 2024-07-26T08:52:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:06 smithi181 ceph-mon[35869]: pgmap v2406: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:06 smithi097 ceph-mon[32231]: pgmap v2406: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:06 smithi115 ceph-mon[35364]: pgmap v2406: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:07.429+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:52:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:07.429+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:52:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:07.430+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:52:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:07.430+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:52:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:08 smithi097 ceph-mon[32231]: pgmap v2407: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:08 smithi115 ceph-mon[35364]: pgmap v2407: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:08 smithi181 ceph-mon[35869]: pgmap v2407: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:10 smithi097 ceph-mon[32231]: pgmap v2408: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3167.681981 secs 2024-07-26T08:52:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:10 smithi115 ceph-mon[35364]: pgmap v2408: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3167.681981 secs 2024-07-26T08:52:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:10 smithi181 ceph-mon[35869]: pgmap v2408: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3167.681981 secs 2024-07-26T08:52:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:12 smithi115 ceph-mon[35364]: pgmap v2409: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:12 smithi181 ceph-mon[35869]: pgmap v2409: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:12 smithi097 ceph-mon[32231]: pgmap v2409: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:14 smithi115 ceph-mon[35364]: pgmap v2410: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:14 smithi115 ceph-mon[35364]: pgmap v2411: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:14 smithi115 ceph-mon[35364]: pgmap v2412: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:14 smithi181 ceph-mon[35869]: pgmap v2410: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:14 smithi181 ceph-mon[35869]: pgmap v2411: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:14 smithi181 ceph-mon[35869]: pgmap v2412: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:14 smithi097 ceph-mon[32231]: pgmap v2410: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:14 smithi097 ceph-mon[32231]: pgmap v2411: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:14 smithi097 ceph-mon[32231]: pgmap v2412: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:52:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3172.682135 secs 2024-07-26T08:52:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3172.682135 secs 2024-07-26T08:52:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3172.682135 secs 2024-07-26T08:52:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:16 smithi115 ceph-mon[35364]: pgmap v2413: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:52:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:16 smithi181 ceph-mon[35869]: pgmap v2413: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:52:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:16 smithi097 ceph-mon[32231]: pgmap v2413: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T08:52:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:18.780+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:52:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:18.781+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:52:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:18.781+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:52:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:18.781+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:52:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:18.781+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:18 smithi115 ceph-mon[35364]: pgmap v2414: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:18 smithi181 ceph-mon[35869]: pgmap v2414: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:18 smithi097 ceph-mon[32231]: pgmap v2414: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3177.682272 secs 2024-07-26T08:52:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:20 smithi115 ceph-mon[35364]: pgmap v2415: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T08:52:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3177.682272 secs 2024-07-26T08:52:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:20 smithi181 ceph-mon[35869]: pgmap v2415: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T08:52:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3177.682272 secs 2024-07-26T08:52:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:20 smithi097 ceph-mon[32231]: pgmap v2415: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T08:52:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:22 smithi115 ceph-mon[35364]: pgmap v2416: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:52:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:22 smithi181 ceph-mon[35869]: pgmap v2416: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:52:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:22 smithi097 ceph-mon[32231]: pgmap v2416: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T08:52:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:24 smithi115 ceph-mon[35364]: pgmap v2417: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:24 smithi181 ceph-mon[35869]: pgmap v2417: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:24 smithi097 ceph-mon[32231]: pgmap v2417: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:26.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3182.682414 secs 2024-07-26T08:52:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3182.682414 secs 2024-07-26T08:52:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3182.682414 secs 2024-07-26T08:52:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:26 smithi115 ceph-mon[35364]: pgmap v2418: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:26 smithi181 ceph-mon[35869]: pgmap v2418: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:26 smithi097 ceph-mon[32231]: pgmap v2418: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:28 smithi115 ceph-mon[35364]: pgmap v2419: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:28 smithi181 ceph-mon[35869]: pgmap v2419: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:28 smithi097 ceph-mon[32231]: pgmap v2419: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3187.682553 secs 2024-07-26T08:52:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:30 smithi115 ceph-mon[35364]: pgmap v2420: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3187.682553 secs 2024-07-26T08:52:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:30 smithi181 ceph-mon[35869]: pgmap v2420: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3187.682553 secs 2024-07-26T08:52:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:30 smithi097 ceph-mon[32231]: pgmap v2420: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:31 smithi115 ceph-mon[35364]: pgmap v2421: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:31 smithi181 ceph-mon[35869]: pgmap v2421: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:31 smithi097 ceph-mon[32231]: pgmap v2421: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:33.785+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:52:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:33.786+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:52:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:33.786+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:52:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:33.786+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:52:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:33.786+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:34 smithi115 ceph-mon[35364]: pgmap v2422: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:34 smithi181 ceph-mon[35869]: pgmap v2422: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:34 smithi097 ceph-mon[32231]: pgmap v2422: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3192.682701 secs 2024-07-26T08:52:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3192.682701 secs 2024-07-26T08:52:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3192.682701 secs 2024-07-26T08:52:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:36 smithi115 ceph-mon[35364]: pgmap v2423: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:36 smithi181 ceph-mon[35869]: pgmap v2423: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:36 smithi097 ceph-mon[32231]: pgmap v2423: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:37 smithi115 ceph-mon[35364]: pgmap v2424: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:37 smithi181 ceph-mon[35869]: pgmap v2424: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:37 smithi097 ceph-mon[32231]: pgmap v2424: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3197.682839 secs 2024-07-26T08:52:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:40 smithi115 ceph-mon[35364]: pgmap v2425: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3197.682839 secs 2024-07-26T08:52:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:40 smithi181 ceph-mon[35869]: pgmap v2425: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3197.682839 secs 2024-07-26T08:52:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:40 smithi097 ceph-mon[32231]: pgmap v2425: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:41 smithi115 ceph-mon[35364]: pgmap v2426: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:41 smithi181 ceph-mon[35869]: pgmap v2426: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:41 smithi097 ceph-mon[32231]: pgmap v2426: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:44 smithi115 ceph-mon[35364]: pgmap v2427: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:45.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:44 smithi181 ceph-mon[35869]: pgmap v2427: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:44 smithi097 ceph-mon[32231]: pgmap v2427: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3202.682981 secs 2024-07-26T08:52:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:45 smithi115 ceph-mon[35364]: pgmap v2428: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3202.682981 secs 2024-07-26T08:52:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:45 smithi181 ceph-mon[35869]: pgmap v2428: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:46.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3202.682981 secs 2024-07-26T08:52:46.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:45 smithi097 ceph-mon[32231]: pgmap v2428: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:48.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:48.792+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:52:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:48.794+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:52:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:48.794+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:52:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:48.794+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:52:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:52:48.794+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:48 smithi115 ceph-mon[35364]: pgmap v2429: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:48 smithi181 ceph-mon[35869]: pgmap v2429: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:48 smithi097 ceph-mon[32231]: pgmap v2429: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:49 smithi115 ceph-mon[35364]: pgmap v2430: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:49 smithi181 ceph-mon[35869]: pgmap v2430: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:49 smithi097 ceph-mon[32231]: pgmap v2430: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3207.683124 secs 2024-07-26T08:52:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3207.683124 secs 2024-07-26T08:52:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3207.683124 secs 2024-07-26T08:52:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:51 smithi115 ceph-mon[35364]: pgmap v2431: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:51 smithi181 ceph-mon[35869]: pgmap v2431: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:51 smithi097 ceph-mon[32231]: pgmap v2431: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:53 smithi115 ceph-mon[35364]: pgmap v2432: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:54.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:53 smithi181 ceph-mon[35869]: pgmap v2432: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:53 smithi097 ceph-mon[32231]: pgmap v2432: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:52:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3212.683267 secs 2024-07-26T08:52:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3212.683267 secs 2024-07-26T08:52:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3212.683267 secs 2024-07-26T08:52:56.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:56 smithi115 ceph-mon[35364]: pgmap v2433: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:56.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:56 smithi181 ceph-mon[35869]: pgmap v2433: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:56.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:56 smithi097 ceph-mon[32231]: pgmap v2433: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:52:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:57 smithi115 ceph-mon[35364]: pgmap v2434: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:57 smithi181 ceph-mon[35869]: pgmap v2434: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:57 smithi097 ceph-mon[32231]: pgmap v2434: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:52:59 smithi115 ceph-mon[35364]: pgmap v2435: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:52:59 smithi181 ceph-mon[35869]: pgmap v2435: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:52:59 smithi097 ceph-mon[32231]: pgmap v2435: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3217.683406 secs 2024-07-26T08:53:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3217.683406 secs 2024-07-26T08:53:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3217.683406 secs 2024-07-26T08:53:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:01 smithi097 ceph-mon[32231]: pgmap v2436: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:01 smithi115 ceph-mon[35364]: pgmap v2436: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:01 smithi181 ceph-mon[35869]: pgmap v2436: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:03.796+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:53:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:03.797+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:53:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:03.797+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:53:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:03.797+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:53:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:03.797+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:03 smithi115 ceph-mon[35364]: pgmap v2437: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:03 smithi181 ceph-mon[35869]: pgmap v2437: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:03 smithi097 ceph-mon[32231]: pgmap v2437: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3222.683546 secs 2024-07-26T08:53:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3222.683546 secs 2024-07-26T08:53:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3222.683546 secs 2024-07-26T08:53:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:06 smithi115 ceph-mon[35364]: pgmap v2438: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:06 smithi181 ceph-mon[35869]: pgmap v2438: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:06 smithi097 ceph-mon[32231]: pgmap v2438: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:07 smithi115 ceph-mon[35364]: pgmap v2439: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:07 smithi181 ceph-mon[35869]: pgmap v2439: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:07 smithi097 ceph-mon[32231]: pgmap v2439: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:09 smithi115 ceph-mon[35364]: pgmap v2440: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:09 smithi181 ceph-mon[35869]: pgmap v2440: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:09 smithi097 ceph-mon[32231]: pgmap v2440: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3227.683677 secs 2024-07-26T08:53:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3227.683677 secs 2024-07-26T08:53:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3227.683677 secs 2024-07-26T08:53:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:11 smithi097 ceph-mon[32231]: pgmap v2441: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:11 smithi115 ceph-mon[35364]: pgmap v2441: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:11 smithi181 ceph-mon[35869]: pgmap v2441: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:13 smithi115 ceph-mon[35364]: pgmap v2442: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:13 smithi181 ceph-mon[35869]: pgmap v2442: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:13 smithi097 ceph-mon[32231]: pgmap v2442: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3232.683757 secs 2024-07-26T08:53:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:15 smithi097 ceph-mon[32231]: pgmap v2443: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:15 smithi097 ceph-mon[32231]: pgmap v2444: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:53:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:53:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3232.683757 secs 2024-07-26T08:53:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:15 smithi115 ceph-mon[35364]: pgmap v2443: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:15 smithi115 ceph-mon[35364]: pgmap v2444: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:53:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:53:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3232.683757 secs 2024-07-26T08:53:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:15 smithi181 ceph-mon[35869]: pgmap v2443: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:15 smithi181 ceph-mon[35869]: pgmap v2444: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:53:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:53:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:18 smithi115 ceph-mon[35364]: pgmap v2445: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:53:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:18 smithi181 ceph-mon[35869]: pgmap v2445: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:53:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:18 smithi097 ceph-mon[32231]: pgmap v2445: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:53:18.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:18.807+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:53:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:18.808+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:53:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:18.808+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:53:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:18.808+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:53:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:18.808+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:20 smithi115 ceph-mon[35364]: pgmap v2446: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3237.683909 secs 2024-07-26T08:53:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:20 smithi181 ceph-mon[35869]: pgmap v2446: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3237.683909 secs 2024-07-26T08:53:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:20 smithi097 ceph-mon[32231]: pgmap v2446: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3237.683909 secs 2024-07-26T08:53:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:22 smithi115 ceph-mon[35364]: pgmap v2447: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:22 smithi181 ceph-mon[35869]: pgmap v2447: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:22 smithi097 ceph-mon[32231]: pgmap v2447: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:24 smithi115 ceph-mon[35364]: pgmap v2448: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:53:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:24 smithi181 ceph-mon[35869]: pgmap v2448: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:53:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:24 smithi097 ceph-mon[32231]: pgmap v2448: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:53:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3242.684049 secs 2024-07-26T08:53:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3242.684049 secs 2024-07-26T08:53:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3242.684049 secs 2024-07-26T08:53:26.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:26 smithi115 ceph-mon[35364]: pgmap v2449: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:26.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:26 smithi181 ceph-mon[35869]: pgmap v2449: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:26.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:26 smithi097 ceph-mon[32231]: pgmap v2449: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:28 smithi115 ceph-mon[35364]: pgmap v2450: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:28 smithi181 ceph-mon[35869]: pgmap v2450: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:28 smithi097 ceph-mon[32231]: pgmap v2450: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:30 smithi115 ceph-mon[35364]: pgmap v2451: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3247.684187 secs 2024-07-26T08:53:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:30 smithi181 ceph-mon[35869]: pgmap v2451: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3247.684187 secs 2024-07-26T08:53:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:30 smithi097 ceph-mon[32231]: pgmap v2451: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3247.684187 secs 2024-07-26T08:53:32.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:32 smithi115 ceph-mon[35364]: pgmap v2452: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:32 smithi181 ceph-mon[35869]: pgmap v2452: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:32 smithi097 ceph-mon[32231]: pgmap v2452: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:33.810+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:53:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:33.811+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:53:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:33.811+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:53:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:33.811+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:53:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:33.811+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:34 smithi115 ceph-mon[35364]: pgmap v2453: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:34 smithi181 ceph-mon[35869]: pgmap v2453: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:34 smithi097 ceph-mon[32231]: pgmap v2453: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3252.684324 secs 2024-07-26T08:53:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3252.684324 secs 2024-07-26T08:53:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3252.684324 secs 2024-07-26T08:53:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:36 smithi115 ceph-mon[35364]: pgmap v2454: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:36 smithi181 ceph-mon[35869]: pgmap v2454: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:36 smithi097 ceph-mon[32231]: pgmap v2454: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:38 smithi115 ceph-mon[35364]: pgmap v2455: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:38 smithi181 ceph-mon[35869]: pgmap v2455: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:38 smithi097 ceph-mon[32231]: pgmap v2455: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:40 smithi115 ceph-mon[35364]: pgmap v2456: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3257.684462 secs 2024-07-26T08:53:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:40 smithi181 ceph-mon[35869]: pgmap v2456: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3257.684462 secs 2024-07-26T08:53:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:40 smithi097 ceph-mon[32231]: pgmap v2456: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3257.684462 secs 2024-07-26T08:53:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:42 smithi115 ceph-mon[35364]: pgmap v2457: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:42 smithi181 ceph-mon[35869]: pgmap v2457: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:42 smithi097 ceph-mon[32231]: pgmap v2457: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:44 smithi115 ceph-mon[35364]: pgmap v2458: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:44 smithi181 ceph-mon[35869]: pgmap v2458: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:44 smithi097 ceph-mon[32231]: pgmap v2458: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3262.684606 secs 2024-07-26T08:53:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3262.684606 secs 2024-07-26T08:53:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3262.684606 secs 2024-07-26T08:53:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:46 smithi115 ceph-mon[35364]: pgmap v2459: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:46 smithi181 ceph-mon[35869]: pgmap v2459: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:46 smithi097 ceph-mon[32231]: pgmap v2459: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:48 smithi115 ceph-mon[35364]: pgmap v2460: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:48 smithi181 ceph-mon[35869]: pgmap v2460: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:48 smithi097 ceph-mon[32231]: pgmap v2460: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:48.819+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:53:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:48.821+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:53:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:48.821+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:53:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:48.821+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:53:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:53:48.821+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:50 smithi115 ceph-mon[35364]: pgmap v2461: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3267.684761 secs 2024-07-26T08:53:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:50 smithi181 ceph-mon[35869]: pgmap v2461: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3267.684761 secs 2024-07-26T08:53:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:50 smithi097 ceph-mon[32231]: pgmap v2461: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:53:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3267.684761 secs 2024-07-26T08:53:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:52 smithi115 ceph-mon[35364]: pgmap v2462: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:52 smithi181 ceph-mon[35869]: pgmap v2462: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:52 smithi097 ceph-mon[32231]: pgmap v2462: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:54 smithi115 ceph-mon[35364]: pgmap v2463: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:54 smithi181 ceph-mon[35869]: pgmap v2463: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:54 smithi097 ceph-mon[32231]: pgmap v2463: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3272.684921 secs 2024-07-26T08:53:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3272.684921 secs 2024-07-26T08:53:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3272.684921 secs 2024-07-26T08:53:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:56 smithi115 ceph-mon[35364]: pgmap v2464: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:56 smithi181 ceph-mon[35869]: pgmap v2464: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:56 smithi097 ceph-mon[32231]: pgmap v2464: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:53:58 smithi115 ceph-mon[35364]: pgmap v2465: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:53:58 smithi181 ceph-mon[35869]: pgmap v2465: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:53:58 smithi097 ceph-mon[32231]: pgmap v2465: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:00 smithi181 ceph-mon[35869]: pgmap v2466: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3277.685062 secs 2024-07-26T08:54:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:00 smithi097 ceph-mon[32231]: pgmap v2466: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3277.685062 secs 2024-07-26T08:54:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:00 smithi115 ceph-mon[35364]: pgmap v2466: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3277.685062 secs 2024-07-26T08:54:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:02 smithi097 ceph-mon[32231]: pgmap v2467: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:02 smithi115 ceph-mon[35364]: pgmap v2467: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:02 smithi181 ceph-mon[35869]: pgmap v2467: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:03.826+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:54:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:03.827+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:54:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:03.827+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:54:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:03.827+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:54:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:03.827+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:04 smithi097 ceph-mon[32231]: pgmap v2468: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:04 smithi115 ceph-mon[35364]: pgmap v2468: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:04 smithi181 ceph-mon[35869]: pgmap v2468: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3282.685210 secs 2024-07-26T08:54:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3282.685210 secs 2024-07-26T08:54:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3282.685210 secs 2024-07-26T08:54:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:06 smithi115 ceph-mon[35364]: pgmap v2469: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:06 smithi181 ceph-mon[35869]: pgmap v2469: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:06 smithi097 ceph-mon[32231]: pgmap v2469: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:08 smithi115 ceph-mon[35364]: pgmap v2470: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:08 smithi181 ceph-mon[35869]: pgmap v2470: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:08 smithi097 ceph-mon[32231]: pgmap v2470: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:10 smithi115 ceph-mon[35364]: pgmap v2471: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3287.685362 secs 2024-07-26T08:54:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:10 smithi181 ceph-mon[35869]: pgmap v2471: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3287.685362 secs 2024-07-26T08:54:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:10 smithi097 ceph-mon[32231]: pgmap v2471: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3287.685362 secs 2024-07-26T08:54:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:12 smithi115 ceph-mon[35364]: pgmap v2472: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:12 smithi181 ceph-mon[35869]: pgmap v2472: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:12 smithi097 ceph-mon[32231]: pgmap v2472: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:14 smithi115 ceph-mon[35364]: pgmap v2473: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:14 smithi181 ceph-mon[35869]: pgmap v2473: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:14 smithi097 ceph-mon[32231]: pgmap v2473: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3292.685511 secs 2024-07-26T08:54:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3292.685511 secs 2024-07-26T08:54:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3292.685511 secs 2024-07-26T08:54:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:16 smithi115 ceph-mon[35364]: pgmap v2474: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:16 smithi181 ceph-mon[35869]: pgmap v2474: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:16 smithi097 ceph-mon[32231]: pgmap v2474: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:17 smithi115 ceph-mon[35364]: pgmap v2475: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:54:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:17 smithi115 ceph-mon[35364]: pgmap v2476: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:54:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:54:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:17 smithi181 ceph-mon[35869]: pgmap v2475: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:54:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:17 smithi181 ceph-mon[35869]: pgmap v2476: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:54:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:54:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:17 smithi097 ceph-mon[32231]: pgmap v2475: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:54:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:17 smithi097 ceph-mon[32231]: pgmap v2476: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:54:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:54:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:18.837+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:54:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:18.838+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:54:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:18.838+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:54:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:18.838+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:54:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:18.838+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:19 smithi115 ceph-mon[35364]: pgmap v2477: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:19 smithi181 ceph-mon[35869]: pgmap v2477: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:19 smithi097 ceph-mon[32231]: pgmap v2477: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3297.685669 secs 2024-07-26T08:54:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3297.685669 secs 2024-07-26T08:54:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3297.685669 secs 2024-07-26T08:54:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:21 smithi097 ceph-mon[32231]: pgmap v2478: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:54:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:21 smithi115 ceph-mon[35364]: pgmap v2478: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:54:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:21 smithi181 ceph-mon[35869]: pgmap v2478: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:54:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:23 smithi097 ceph-mon[32231]: pgmap v2479: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:54:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:23 smithi115 ceph-mon[35364]: pgmap v2479: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:54:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:23 smithi181 ceph-mon[35869]: pgmap v2479: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:54:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:25 smithi097 ceph-mon[32231]: pgmap v2480: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:54:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3302.685828 secs 2024-07-26T08:54:26.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:25 smithi115 ceph-mon[35364]: pgmap v2480: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:54:26.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3302.685828 secs 2024-07-26T08:54:26.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:25 smithi181 ceph-mon[35869]: pgmap v2480: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T08:54:26.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3302.685828 secs 2024-07-26T08:54:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:27 smithi115 ceph-mon[35364]: pgmap v2481: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:27 smithi181 ceph-mon[35869]: pgmap v2481: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:27 smithi097 ceph-mon[32231]: pgmap v2481: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:29 smithi115 ceph-mon[35364]: pgmap v2482: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:29 smithi181 ceph-mon[35869]: pgmap v2482: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:29 smithi097 ceph-mon[32231]: pgmap v2482: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3307.685989 secs 2024-07-26T08:54:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3307.685989 secs 2024-07-26T08:54:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3307.685989 secs 2024-07-26T08:54:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:31 smithi115 ceph-mon[35364]: pgmap v2483: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:31 smithi181 ceph-mon[35869]: pgmap v2483: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:31 smithi097 ceph-mon[32231]: pgmap v2483: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:33.842+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:54:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:33.843+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:54:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:33.843+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:54:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:33.843+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:54:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:33.844+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:33 smithi115 ceph-mon[35364]: pgmap v2484: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:33 smithi181 ceph-mon[35869]: pgmap v2484: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:33 smithi097 ceph-mon[32231]: pgmap v2484: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:35 smithi115 ceph-mon[35364]: pgmap v2485: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3312.686133 secs 2024-07-26T08:54:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:35 smithi181 ceph-mon[35869]: pgmap v2485: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3312.686133 secs 2024-07-26T08:54:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:35 smithi097 ceph-mon[32231]: pgmap v2485: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3312.686133 secs 2024-07-26T08:54:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:37 smithi115 ceph-mon[35364]: pgmap v2486: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:37 smithi181 ceph-mon[35869]: pgmap v2486: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:37 smithi097 ceph-mon[32231]: pgmap v2486: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:39 smithi115 ceph-mon[35364]: pgmap v2487: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:39 smithi181 ceph-mon[35869]: pgmap v2487: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:39 smithi097 ceph-mon[32231]: pgmap v2487: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3317.686270 secs 2024-07-26T08:54:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3317.686270 secs 2024-07-26T08:54:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3317.686270 secs 2024-07-26T08:54:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:41 smithi115 ceph-mon[35364]: pgmap v2488: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:41 smithi181 ceph-mon[35869]: pgmap v2488: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:41 smithi097 ceph-mon[32231]: pgmap v2488: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:43 smithi115 ceph-mon[35364]: pgmap v2489: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:43 smithi181 ceph-mon[35869]: pgmap v2489: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:43 smithi097 ceph-mon[32231]: pgmap v2489: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:45 smithi115 ceph-mon[35364]: pgmap v2490: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:54:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3322.686434 secs 2024-07-26T08:54:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:45 smithi181 ceph-mon[35869]: pgmap v2490: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:54:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3322.686434 secs 2024-07-26T08:54:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:45 smithi097 ceph-mon[32231]: pgmap v2490: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:54:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3322.686434 secs 2024-07-26T08:54:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:47 smithi115 ceph-mon[35364]: pgmap v2491: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:54:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:47 smithi181 ceph-mon[35869]: pgmap v2491: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:54:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:47 smithi097 ceph-mon[32231]: pgmap v2491: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:54:48.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:48.849+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:54:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:48.851+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:54:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:48.851+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:54:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:48.851+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:54:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:54:48.851+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:49 smithi115 ceph-mon[35364]: pgmap v2492: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:54:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:49 smithi181 ceph-mon[35869]: pgmap v2492: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:54:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:49 smithi097 ceph-mon[32231]: pgmap v2492: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:54:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3327.686595 secs 2024-07-26T08:54:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3327.686595 secs 2024-07-26T08:54:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3327.686595 secs 2024-07-26T08:54:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:51 smithi115 ceph-mon[35364]: pgmap v2493: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:51 smithi181 ceph-mon[35869]: pgmap v2493: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:51 smithi097 ceph-mon[32231]: pgmap v2493: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:53 smithi115 ceph-mon[35364]: pgmap v2494: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:53 smithi181 ceph-mon[35869]: pgmap v2494: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:53 smithi097 ceph-mon[32231]: pgmap v2494: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:55 smithi115 ceph-mon[35364]: pgmap v2495: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3332.686693 secs 2024-07-26T08:54:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:55 smithi181 ceph-mon[35869]: pgmap v2495: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3332.686693 secs 2024-07-26T08:54:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:55 smithi097 ceph-mon[32231]: pgmap v2495: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3332.686693 secs 2024-07-26T08:54:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:57 smithi115 ceph-mon[35364]: pgmap v2496: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:57 smithi181 ceph-mon[35869]: pgmap v2496: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:57 smithi097 ceph-mon[32231]: pgmap v2496: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:54:59 smithi115 ceph-mon[35364]: pgmap v2497: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:54:59 smithi181 ceph-mon[35869]: pgmap v2497: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:54:59 smithi097 ceph-mon[32231]: pgmap v2497: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3337.686829 secs 2024-07-26T08:55:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3337.686829 secs 2024-07-26T08:55:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3337.686829 secs 2024-07-26T08:55:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:01 smithi115 ceph-mon[35364]: pgmap v2498: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:01 smithi181 ceph-mon[35869]: pgmap v2498: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:01 smithi097 ceph-mon[32231]: pgmap v2498: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:03.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:03.856+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:55:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:03.857+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:55:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:03.857+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:55:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:03.857+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:55:03.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:03.857+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:03 smithi115 ceph-mon[35364]: pgmap v2499: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:03 smithi181 ceph-mon[35869]: pgmap v2499: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:03 smithi097 ceph-mon[32231]: pgmap v2499: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:05 smithi115 ceph-mon[35364]: pgmap v2500: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3342.686963 secs 2024-07-26T08:55:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:05 smithi181 ceph-mon[35869]: pgmap v2500: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3342.686963 secs 2024-07-26T08:55:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:05 smithi097 ceph-mon[32231]: pgmap v2500: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3342.686963 secs 2024-07-26T08:55:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:07.461+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:55:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:07.461+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:55:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:07.462+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:55:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:07.462+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:55:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:07 smithi115 ceph-mon[35364]: pgmap v2501: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:07 smithi181 ceph-mon[35869]: pgmap v2501: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:07 smithi097 ceph-mon[32231]: pgmap v2501: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:09 smithi115 ceph-mon[35364]: pgmap v2502: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:09 smithi181 ceph-mon[35869]: pgmap v2502: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:09 smithi097 ceph-mon[32231]: pgmap v2502: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3347.687101 secs 2024-07-26T08:55:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3347.687101 secs 2024-07-26T08:55:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3347.687101 secs 2024-07-26T08:55:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:11 smithi115 ceph-mon[35364]: pgmap v2503: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:11 smithi181 ceph-mon[35869]: pgmap v2503: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:11 smithi097 ceph-mon[32231]: pgmap v2503: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:12 smithi115 ceph-mon[35364]: pgmap v2504: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:12 smithi181 ceph-mon[35869]: pgmap v2504: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:12 smithi097 ceph-mon[32231]: pgmap v2504: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:15 smithi097 ceph-mon[32231]: pgmap v2505: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3352.687201 secs 2024-07-26T08:55:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:15 smithi115 ceph-mon[35364]: pgmap v2505: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3352.687201 secs 2024-07-26T08:55:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:15 smithi181 ceph-mon[35869]: pgmap v2505: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3352.687201 secs 2024-07-26T08:55:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:17 smithi115 ceph-mon[35364]: pgmap v2506: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:17 smithi181 ceph-mon[35869]: pgmap v2506: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:17 smithi097 ceph-mon[32231]: pgmap v2506: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:18.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:18.867+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:55:18.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:18.868+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:55:18.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:18.868+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:55:18.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:18.868+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:55:18.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:18.868+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:55:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:55:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:55:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:19 smithi115 ceph-mon[35364]: pgmap v2507: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:55:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:19 smithi115 ceph-mon[35364]: pgmap v2508: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:19 smithi181 ceph-mon[35869]: pgmap v2507: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:55:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:19 smithi181 ceph-mon[35869]: pgmap v2508: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:19 smithi097 ceph-mon[32231]: pgmap v2507: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:55:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:19 smithi097 ceph-mon[32231]: pgmap v2508: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3357.687343 secs 2024-07-26T08:55:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3357.687343 secs 2024-07-26T08:55:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3357.687343 secs 2024-07-26T08:55:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:21 smithi115 ceph-mon[35364]: pgmap v2509: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:21 smithi181 ceph-mon[35869]: pgmap v2509: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:21 smithi097 ceph-mon[32231]: pgmap v2509: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:23 smithi115 ceph-mon[35364]: pgmap v2510: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:55:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:23 smithi181 ceph-mon[35869]: pgmap v2510: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:55:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:23 smithi097 ceph-mon[32231]: pgmap v2510: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:55:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:25 smithi115 ceph-mon[35364]: pgmap v2511: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3362.687431 secs 2024-07-26T08:55:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:25 smithi181 ceph-mon[35869]: pgmap v2511: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3362.687431 secs 2024-07-26T08:55:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:25 smithi097 ceph-mon[32231]: pgmap v2511: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3362.687431 secs 2024-07-26T08:55:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:27 smithi115 ceph-mon[35364]: pgmap v2512: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:27 smithi181 ceph-mon[35869]: pgmap v2512: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:27 smithi097 ceph-mon[32231]: pgmap v2512: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:29 smithi115 ceph-mon[35364]: pgmap v2513: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:55:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:29 smithi181 ceph-mon[35869]: pgmap v2513: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:55:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:29 smithi097 ceph-mon[32231]: pgmap v2513: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:55:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3367.687570 secs 2024-07-26T08:55:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3367.687570 secs 2024-07-26T08:55:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3367.687570 secs 2024-07-26T08:55:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:31 smithi115 ceph-mon[35364]: pgmap v2514: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:31 smithi181 ceph-mon[35869]: pgmap v2514: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:31 smithi097 ceph-mon[32231]: pgmap v2514: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:33.872+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:55:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:33.873+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:55:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:33.873+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:55:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:33.873+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:55:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:33.873+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:33 smithi115 ceph-mon[35364]: pgmap v2515: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:33 smithi181 ceph-mon[35869]: pgmap v2515: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:33 smithi097 ceph-mon[32231]: pgmap v2515: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:35 smithi115 ceph-mon[35364]: pgmap v2516: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:55:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3372.687654 secs 2024-07-26T08:55:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:35 smithi181 ceph-mon[35869]: pgmap v2516: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:55:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3372.687654 secs 2024-07-26T08:55:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:35 smithi097 ceph-mon[32231]: pgmap v2516: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:55:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3372.687654 secs 2024-07-26T08:55:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:37 smithi115 ceph-mon[35364]: pgmap v2517: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:37 smithi181 ceph-mon[35869]: pgmap v2517: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:37 smithi097 ceph-mon[32231]: pgmap v2517: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:39 smithi115 ceph-mon[35364]: pgmap v2518: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:39 smithi181 ceph-mon[35869]: pgmap v2518: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:39 smithi097 ceph-mon[32231]: pgmap v2518: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3377.687827 secs 2024-07-26T08:55:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3377.687827 secs 2024-07-26T08:55:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3377.687827 secs 2024-07-26T08:55:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:41 smithi115 ceph-mon[35364]: pgmap v2519: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:41 smithi181 ceph-mon[35869]: pgmap v2519: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:41 smithi097 ceph-mon[32231]: pgmap v2519: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:43 smithi115 ceph-mon[35364]: pgmap v2520: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:43 smithi181 ceph-mon[35869]: pgmap v2520: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:43 smithi097 ceph-mon[32231]: pgmap v2520: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:45 smithi115 ceph-mon[35364]: pgmap v2521: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:55:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3382.687990 secs 2024-07-26T08:55:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:45 smithi181 ceph-mon[35869]: pgmap v2521: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:55:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3382.687990 secs 2024-07-26T08:55:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:45 smithi097 ceph-mon[32231]: pgmap v2521: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:55:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3382.687990 secs 2024-07-26T08:55:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:47 smithi115 ceph-mon[35364]: pgmap v2522: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:47 smithi181 ceph-mon[35869]: pgmap v2522: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:47 smithi097 ceph-mon[32231]: pgmap v2522: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:55:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:48.878+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:55:48.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:48.879+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:55:48.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:48.879+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:55:48.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:48.879+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:55:48.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:48.879+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:49 smithi115 ceph-mon[35364]: pgmap v2523: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:49 smithi181 ceph-mon[35869]: pgmap v2523: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:49 smithi097 ceph-mon[32231]: pgmap v2523: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3387.688140 secs 2024-07-26T08:55:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3387.688140 secs 2024-07-26T08:55:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3387.688140 secs 2024-07-26T08:55:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:51 smithi115 ceph-mon[35364]: pgmap v2524: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:51 smithi181 ceph-mon[35869]: pgmap v2524: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:51 smithi097 ceph-mon[32231]: pgmap v2524: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:52.468+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:55:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:52.468+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:55:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:52.469+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:55:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:55:52.469+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:55:54.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:53 smithi115 ceph-mon[35364]: pgmap v2525: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:53 smithi181 ceph-mon[35869]: pgmap v2525: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:53 smithi097 ceph-mon[32231]: pgmap v2525: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:55 smithi115 ceph-mon[35364]: pgmap v2526: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:55:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3392.688338 secs 2024-07-26T08:55:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:55 smithi181 ceph-mon[35869]: pgmap v2526: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:55:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3392.688338 secs 2024-07-26T08:55:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:55 smithi097 ceph-mon[32231]: pgmap v2526: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:55:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3392.688338 secs 2024-07-26T08:55:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:57 smithi115 ceph-mon[35364]: pgmap v2527: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:55:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:57 smithi181 ceph-mon[35869]: pgmap v2527: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:55:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:57 smithi097 ceph-mon[32231]: pgmap v2527: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:56:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:55:59 smithi115 ceph-mon[35364]: pgmap v2528: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:55:59 smithi181 ceph-mon[35869]: pgmap v2528: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:55:59 smithi097 ceph-mon[32231]: pgmap v2528: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3397.688495 secs 2024-07-26T08:56:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3397.688495 secs 2024-07-26T08:56:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3397.688495 secs 2024-07-26T08:56:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:01 smithi115 ceph-mon[35364]: pgmap v2529: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:56:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:01 smithi181 ceph-mon[35869]: pgmap v2529: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:56:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:01 smithi097 ceph-mon[32231]: pgmap v2529: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:56:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:02 smithi181 ceph-mon[35869]: pgmap v2530: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:02 smithi097 ceph-mon[32231]: pgmap v2530: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:02 smithi115 ceph-mon[35364]: pgmap v2530: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:03.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:03.882+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:56:03.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:03.883+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:56:03.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:03.883+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:56:03.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:03.883+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:56:03.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:03.883+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:05 smithi115 ceph-mon[35364]: pgmap v2531: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:05 smithi181 ceph-mon[35869]: pgmap v2531: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:05 smithi097 ceph-mon[32231]: pgmap v2531: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3402.688634 secs 2024-07-26T08:56:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3402.688634 secs 2024-07-26T08:56:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3402.688634 secs 2024-07-26T08:56:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:07 smithi115 ceph-mon[35364]: pgmap v2532: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:56:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:07 smithi181 ceph-mon[35869]: pgmap v2532: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:56:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:07 smithi097 ceph-mon[32231]: pgmap v2532: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:56:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:09 smithi115 ceph-mon[35364]: pgmap v2533: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:09 smithi181 ceph-mon[35869]: pgmap v2533: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:09 smithi097 ceph-mon[32231]: pgmap v2533: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3407.688774 secs 2024-07-26T08:56:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:11 smithi115 ceph-mon[35364]: pgmap v2534: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3407.688774 secs 2024-07-26T08:56:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:11 smithi181 ceph-mon[35869]: pgmap v2534: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3407.688774 secs 2024-07-26T08:56:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:11 smithi097 ceph-mon[32231]: pgmap v2534: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:13 smithi115 ceph-mon[35364]: pgmap v2535: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:13 smithi181 ceph-mon[35869]: pgmap v2535: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:13 smithi097 ceph-mon[32231]: pgmap v2535: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:15 smithi181 ceph-mon[35869]: pgmap v2536: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:56:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:15 smithi097 ceph-mon[32231]: pgmap v2536: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:56:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:15 smithi115 ceph-mon[35364]: pgmap v2536: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:56:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3412.688915 secs 2024-07-26T08:56:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3412.688915 secs 2024-07-26T08:56:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3412.688915 secs 2024-07-26T08:56:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:17 smithi115 ceph-mon[35364]: pgmap v2537: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:56:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:17 smithi181 ceph-mon[35869]: pgmap v2537: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:56:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:17 smithi097 ceph-mon[32231]: pgmap v2537: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:56:18.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:18.892+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:56:18.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:18.893+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:56:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:18.893+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:56:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:18.893+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:56:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:18.893+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:19 smithi115 ceph-mon[35364]: pgmap v2538: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:56:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:19 smithi181 ceph-mon[35869]: pgmap v2538: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:19.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:56:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:19 smithi097 ceph-mon[32231]: pgmap v2538: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:56:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:20 smithi115 ceph-mon[35364]: pgmap v2539: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3417.689060 secs 2024-07-26T08:56:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:20 smithi181 ceph-mon[35869]: pgmap v2539: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3417.689060 secs 2024-07-26T08:56:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:20 smithi097 ceph-mon[32231]: pgmap v2539: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3417.689060 secs 2024-07-26T08:56:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:22 smithi115 ceph-mon[35364]: pgmap v2540: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:56:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:22 smithi181 ceph-mon[35869]: pgmap v2540: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:56:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:22 smithi097 ceph-mon[32231]: pgmap v2540: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:56:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:24 smithi115 ceph-mon[35364]: pgmap v2541: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:24 smithi181 ceph-mon[35869]: pgmap v2541: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:24 smithi097 ceph-mon[32231]: pgmap v2541: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3422.689219 secs 2024-07-26T08:56:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3422.689219 secs 2024-07-26T08:56:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3422.689219 secs 2024-07-26T08:56:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:26 smithi115 ceph-mon[35364]: pgmap v2542: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:26 smithi181 ceph-mon[35869]: pgmap v2542: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:26 smithi097 ceph-mon[32231]: pgmap v2542: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:28 smithi115 ceph-mon[35364]: pgmap v2543: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:56:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:28 smithi181 ceph-mon[35869]: pgmap v2543: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:56:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:28 smithi097 ceph-mon[32231]: pgmap v2543: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T08:56:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:30 smithi097 ceph-mon[32231]: pgmap v2544: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3427.689362 secs 2024-07-26T08:56:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:30 smithi115 ceph-mon[35364]: pgmap v2544: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3427.689362 secs 2024-07-26T08:56:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:30 smithi181 ceph-mon[35869]: pgmap v2544: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3427.689362 secs 2024-07-26T08:56:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:32 smithi115 ceph-mon[35364]: pgmap v2545: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:32 smithi181 ceph-mon[35869]: pgmap v2545: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:32 smithi097 ceph-mon[32231]: pgmap v2545: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:33.895+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:56:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:33.896+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:56:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:33.896+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:56:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:33.896+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:56:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:33.896+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:34 smithi115 ceph-mon[35364]: pgmap v2546: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:34 smithi181 ceph-mon[35869]: pgmap v2546: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:34 smithi097 ceph-mon[32231]: pgmap v2546: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3432.689497 secs 2024-07-26T08:56:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3432.689497 secs 2024-07-26T08:56:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3432.689497 secs 2024-07-26T08:56:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:36 smithi097 ceph-mon[32231]: pgmap v2547: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:36 smithi115 ceph-mon[35364]: pgmap v2547: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:36 smithi181 ceph-mon[35869]: pgmap v2547: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:38 smithi097 ceph-mon[32231]: pgmap v2548: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:38 smithi115 ceph-mon[35364]: pgmap v2548: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:38 smithi181 ceph-mon[35869]: pgmap v2548: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:40 smithi097 ceph-mon[32231]: pgmap v2549: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:56:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3437.689639 secs 2024-07-26T08:56:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:40 smithi115 ceph-mon[35364]: pgmap v2549: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:56:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3437.689639 secs 2024-07-26T08:56:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:40 smithi181 ceph-mon[35869]: pgmap v2549: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:56:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3437.689639 secs 2024-07-26T08:56:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:42 smithi115 ceph-mon[35364]: pgmap v2550: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:42 smithi181 ceph-mon[35869]: pgmap v2550: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:42 smithi097 ceph-mon[32231]: pgmap v2550: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:44 smithi115 ceph-mon[35364]: pgmap v2551: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:56:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:44 smithi181 ceph-mon[35869]: pgmap v2551: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:56:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:44 smithi097 ceph-mon[32231]: pgmap v2551: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:56:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3442.689781 secs 2024-07-26T08:56:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3442.689781 secs 2024-07-26T08:56:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3442.689781 secs 2024-07-26T08:56:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:46 smithi115 ceph-mon[35364]: pgmap v2552: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:46 smithi181 ceph-mon[35869]: pgmap v2552: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:46 smithi097 ceph-mon[32231]: pgmap v2552: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:56:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:48.905+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:56:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:48.907+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:56:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:48.907+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:56:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:48.907+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:56:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:56:48.907+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:48 smithi115 ceph-mon[35364]: pgmap v2553: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:48 smithi181 ceph-mon[35869]: pgmap v2553: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:48 smithi097 ceph-mon[32231]: pgmap v2553: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:50 smithi115 ceph-mon[35364]: pgmap v2554: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3447.689919 secs 2024-07-26T08:56:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:50 smithi181 ceph-mon[35869]: pgmap v2554: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3447.689919 secs 2024-07-26T08:56:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:50 smithi097 ceph-mon[32231]: pgmap v2554: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3447.689919 secs 2024-07-26T08:56:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:52 smithi115 ceph-mon[35364]: pgmap v2555: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:52 smithi181 ceph-mon[35869]: pgmap v2555: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:52 smithi097 ceph-mon[32231]: pgmap v2555: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:55.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:54 smithi115 ceph-mon[35364]: pgmap v2556: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:54 smithi181 ceph-mon[35869]: pgmap v2556: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:54 smithi097 ceph-mon[32231]: pgmap v2556: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:56.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3452.690066 secs 2024-07-26T08:56:56.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3452.690066 secs 2024-07-26T08:56:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3452.690066 secs 2024-07-26T08:56:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:56 smithi115 ceph-mon[35364]: pgmap v2557: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:56 smithi181 ceph-mon[35869]: pgmap v2557: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:56 smithi097 ceph-mon[32231]: pgmap v2557: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:56:58 smithi115 ceph-mon[35364]: pgmap v2558: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:56:58 smithi181 ceph-mon[35869]: pgmap v2558: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:56:58 smithi097 ceph-mon[32231]: pgmap v2558: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:00 smithi115 ceph-mon[35364]: pgmap v2559: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3457.690207 secs 2024-07-26T08:57:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:00 smithi181 ceph-mon[35869]: pgmap v2559: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3457.690207 secs 2024-07-26T08:57:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:00 smithi097 ceph-mon[32231]: pgmap v2559: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3457.690207 secs 2024-07-26T08:57:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:02 smithi115 ceph-mon[35364]: pgmap v2560: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:02 smithi181 ceph-mon[35869]: pgmap v2560: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:02 smithi097 ceph-mon[32231]: pgmap v2560: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:03.912+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:57:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:03.913+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:57:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:03.913+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:57:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:03.913+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:57:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:03.913+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:04 smithi115 ceph-mon[35364]: pgmap v2561: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:04 smithi181 ceph-mon[35869]: pgmap v2561: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:04 smithi097 ceph-mon[32231]: pgmap v2561: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3462.690355 secs 2024-07-26T08:57:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3462.690355 secs 2024-07-26T08:57:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3462.690355 secs 2024-07-26T08:57:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:06 smithi115 ceph-mon[35364]: pgmap v2562: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:06 smithi181 ceph-mon[35869]: pgmap v2562: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:06 smithi097 ceph-mon[32231]: pgmap v2562: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:08 smithi115 ceph-mon[35364]: pgmap v2563: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:08 smithi181 ceph-mon[35869]: pgmap v2563: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:08 smithi097 ceph-mon[32231]: pgmap v2563: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:10 smithi115 ceph-mon[35364]: pgmap v2564: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3467.690453 secs 2024-07-26T08:57:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:10 smithi181 ceph-mon[35869]: pgmap v2564: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3467.690453 secs 2024-07-26T08:57:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:10 smithi097 ceph-mon[32231]: pgmap v2564: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3467.690453 secs 2024-07-26T08:57:13.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:12 smithi115 ceph-mon[35364]: pgmap v2565: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:12 smithi181 ceph-mon[35869]: pgmap v2565: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:12 smithi097 ceph-mon[32231]: pgmap v2565: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:14 smithi115 ceph-mon[35364]: pgmap v2566: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:14 smithi181 ceph-mon[35869]: pgmap v2566: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:14 smithi097 ceph-mon[32231]: pgmap v2566: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3472.690615 secs 2024-07-26T08:57:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3472.690615 secs 2024-07-26T08:57:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3472.690615 secs 2024-07-26T08:57:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:16 smithi115 ceph-mon[35364]: pgmap v2567: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-07-26T08:57:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:16 smithi181 ceph-mon[35869]: pgmap v2567: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-07-26T08:57:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:16 smithi097 ceph-mon[32231]: pgmap v2567: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-07-26T08:57:18.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:18.916+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:57:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:18.917+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:57:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:18.918+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:57:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:18.918+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:57:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:18.918+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:18 smithi115 ceph-mon[35364]: pgmap v2568: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:57:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:18 smithi181 ceph-mon[35869]: pgmap v2568: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:57:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:18 smithi097 ceph-mon[32231]: pgmap v2568: 97 pgs: 97 active+clean; 210 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:57:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:20 smithi115 ceph-mon[35364]: pgmap v2569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 151 KiB/s wr, 8 op/s 2024-07-26T08:57:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3477.690754 secs 2024-07-26T08:57:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:57:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:20 smithi181 ceph-mon[35869]: pgmap v2569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 151 KiB/s wr, 8 op/s 2024-07-26T08:57:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3477.690754 secs 2024-07-26T08:57:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:57:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:20 smithi097 ceph-mon[32231]: pgmap v2569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 151 KiB/s wr, 8 op/s 2024-07-26T08:57:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3477.690754 secs 2024-07-26T08:57:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:57:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:21 smithi115 ceph-mon[35364]: pgmap v2570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2024-07-26T08:57:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:21 smithi115 ceph-mon[35364]: pgmap v2571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2024-07-26T08:57:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:21 smithi181 ceph-mon[35869]: pgmap v2570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2024-07-26T08:57:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:21 smithi181 ceph-mon[35869]: pgmap v2571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2024-07-26T08:57:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:21 smithi097 ceph-mon[32231]: pgmap v2570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2024-07-26T08:57:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:21 smithi097 ceph-mon[32231]: pgmap v2571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2024-07-26T08:57:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:23 smithi115 ceph-mon[35364]: pgmap v2572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 8 op/s 2024-07-26T08:57:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:23 smithi181 ceph-mon[35869]: pgmap v2572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 8 op/s 2024-07-26T08:57:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:23 smithi097 ceph-mon[32231]: pgmap v2572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 8 op/s 2024-07-26T08:57:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3482.690841 secs 2024-07-26T08:57:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:25 smithi115 ceph-mon[35364]: pgmap v2573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 8 op/s 2024-07-26T08:57:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3482.690841 secs 2024-07-26T08:57:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:25 smithi181 ceph-mon[35869]: pgmap v2573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 8 op/s 2024-07-26T08:57:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3482.690841 secs 2024-07-26T08:57:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:25 smithi097 ceph-mon[32231]: pgmap v2573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 8 op/s 2024-07-26T08:57:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:27 smithi115 ceph-mon[35364]: pgmap v2574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 192 KiB/s wr, 7 op/s 2024-07-26T08:57:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:27 smithi181 ceph-mon[35869]: pgmap v2574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 192 KiB/s wr, 7 op/s 2024-07-26T08:57:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:27 smithi097 ceph-mon[32231]: pgmap v2574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 192 KiB/s wr, 7 op/s 2024-07-26T08:57:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:29 smithi115 ceph-mon[35364]: pgmap v2575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:29 smithi181 ceph-mon[35869]: pgmap v2575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:29 smithi097 ceph-mon[32231]: pgmap v2575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3487.690999 secs 2024-07-26T08:57:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3487.690999 secs 2024-07-26T08:57:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3487.690999 secs 2024-07-26T08:57:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:31 smithi181 ceph-mon[35869]: pgmap v2576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:57:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:31 smithi097 ceph-mon[32231]: pgmap v2576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:57:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:31 smithi115 ceph-mon[35364]: pgmap v2576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:57:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:32 smithi097 ceph-mon[32231]: pgmap v2577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:32 smithi115 ceph-mon[35364]: pgmap v2577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:32 smithi181 ceph-mon[35869]: pgmap v2577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:33.920+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:57:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:33.922+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:57:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:33.922+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:57:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:33.922+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:57:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:33.922+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:35 smithi115 ceph-mon[35364]: pgmap v2578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:35 smithi181 ceph-mon[35869]: pgmap v2578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:35 smithi097 ceph-mon[32231]: pgmap v2578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3492.691161 secs 2024-07-26T08:57:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3492.691161 secs 2024-07-26T08:57:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3492.691161 secs 2024-07-26T08:57:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:37 smithi115 ceph-mon[35364]: pgmap v2579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:37 smithi181 ceph-mon[35869]: pgmap v2579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:37 smithi097 ceph-mon[32231]: pgmap v2579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:39 smithi115 ceph-mon[35364]: pgmap v2580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:57:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:39 smithi181 ceph-mon[35869]: pgmap v2580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:57:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:39 smithi097 ceph-mon[32231]: pgmap v2580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:57:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3497.691336 secs 2024-07-26T08:57:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3497.691336 secs 2024-07-26T08:57:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3497.691336 secs 2024-07-26T08:57:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:41 smithi115 ceph-mon[35364]: pgmap v2581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:41 smithi181 ceph-mon[35869]: pgmap v2581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:41 smithi097 ceph-mon[32231]: pgmap v2581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:43 smithi115 ceph-mon[35364]: pgmap v2582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:43 smithi181 ceph-mon[35869]: pgmap v2582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:43 smithi097 ceph-mon[32231]: pgmap v2582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:45 smithi115 ceph-mon[35364]: pgmap v2583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:45 smithi181 ceph-mon[35869]: pgmap v2583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:45 smithi097 ceph-mon[32231]: pgmap v2583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3502.691437 secs 2024-07-26T08:57:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3502.691437 secs 2024-07-26T08:57:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3502.691437 secs 2024-07-26T08:57:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:47 smithi115 ceph-mon[35364]: pgmap v2584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:47 smithi181 ceph-mon[35869]: pgmap v2584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:47 smithi097 ceph-mon[32231]: pgmap v2584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:48.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:48.932+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:57:48.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:48.933+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:57:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:48.933+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:57:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:48.933+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:57:48.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:57:48.933+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:49 smithi115 ceph-mon[35364]: pgmap v2585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:57:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:49 smithi181 ceph-mon[35869]: pgmap v2585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:57:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:49 smithi097 ceph-mon[32231]: pgmap v2585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:57:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3507.691573 secs 2024-07-26T08:57:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3507.691573 secs 2024-07-26T08:57:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3507.691573 secs 2024-07-26T08:57:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:51 smithi115 ceph-mon[35364]: pgmap v2586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:51 smithi181 ceph-mon[35869]: pgmap v2586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:51 smithi097 ceph-mon[32231]: pgmap v2586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:53 smithi115 ceph-mon[35364]: pgmap v2587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:53 smithi181 ceph-mon[35869]: pgmap v2587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:53 smithi097 ceph-mon[32231]: pgmap v2587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:55 smithi115 ceph-mon[35364]: pgmap v2588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:55 smithi181 ceph-mon[35869]: pgmap v2588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:55 smithi097 ceph-mon[32231]: pgmap v2588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:56.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3512.691718 secs 2024-07-26T08:57:56.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3512.691718 secs 2024-07-26T08:57:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3512.691718 secs 2024-07-26T08:57:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:57 smithi115 ceph-mon[35364]: pgmap v2589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:57 smithi181 ceph-mon[35869]: pgmap v2589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:57 smithi097 ceph-mon[32231]: pgmap v2589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:57:59 smithi115 ceph-mon[35364]: pgmap v2590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:57:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:57:59 smithi181 ceph-mon[35869]: pgmap v2590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:57:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:57:59 smithi097 ceph-mon[32231]: pgmap v2590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3517.691858 secs 2024-07-26T08:58:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3517.691858 secs 2024-07-26T08:58:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3517.691858 secs 2024-07-26T08:58:01.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:01 smithi115 ceph-mon[35364]: pgmap v2591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:01 smithi181 ceph-mon[35869]: pgmap v2591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:01 smithi097 ceph-mon[32231]: pgmap v2591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:03 smithi115 ceph-mon[35364]: pgmap v2592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:03 smithi181 ceph-mon[35869]: pgmap v2592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:03 smithi097 ceph-mon[32231]: pgmap v2592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:03.937+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:58:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:03.939+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:58:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:03.939+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:58:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:03.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:58:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:03.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:05 smithi115 ceph-mon[35364]: pgmap v2593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:05 smithi181 ceph-mon[35869]: pgmap v2593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:05 smithi097 ceph-mon[32231]: pgmap v2593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3522.691999 secs 2024-07-26T08:58:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3522.691999 secs 2024-07-26T08:58:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3522.691999 secs 2024-07-26T08:58:06.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:06.888+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 msgr2=0x7ff3d40c0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900001301) for more than 900000000 us, fault. 2024-07-26T08:58:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:06.888+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0a80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e0058000 comp rx=0 tx=0).stop 2024-07-26T08:58:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:06.888+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 msgr2=0x7ff3d40c0a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:58:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:06.888+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0a80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:58:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:06.888+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:58:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:06.888+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:58:06.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:06.890+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0fc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff3e001fcb0 tx=0x7ff3e0071000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:58:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:07 smithi115 ceph-mon[35364]: pgmap v2594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:07 smithi181 ceph-mon[35869]: pgmap v2594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:07.493 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:07.492+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:58:07.493 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:07.492+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:58:07.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:07.493+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T08:58:07.495 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:07.493+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T08:58:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:07 smithi097 ceph-mon[32231]: pgmap v2594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:09 smithi115 ceph-mon[35364]: pgmap v2595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:09 smithi181 ceph-mon[35869]: pgmap v2595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:09 smithi097 ceph-mon[32231]: pgmap v2595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:10.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3527.692137 secs 2024-07-26T08:58:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3527.692137 secs 2024-07-26T08:58:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3527.692137 secs 2024-07-26T08:58:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:11 smithi115 ceph-mon[35364]: pgmap v2596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:11 smithi181 ceph-mon[35869]: pgmap v2596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:11 smithi097 ceph-mon[32231]: pgmap v2596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:13 smithi115 ceph-mon[35364]: pgmap v2597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:13 smithi181 ceph-mon[35869]: pgmap v2597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:13 smithi097 ceph-mon[32231]: pgmap v2597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:15 smithi115 ceph-mon[35364]: pgmap v2598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:15.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:15 smithi181 ceph-mon[35869]: pgmap v2598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:15 smithi097 ceph-mon[32231]: pgmap v2598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3532.692285 secs 2024-07-26T08:58:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3532.692285 secs 2024-07-26T08:58:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3532.692285 secs 2024-07-26T08:58:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:17 smithi115 ceph-mon[35364]: pgmap v2599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:17 smithi181 ceph-mon[35869]: pgmap v2599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:17 smithi097 ceph-mon[32231]: pgmap v2599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:18.947+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:58:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:18.948+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:58:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:18.948+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:58:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:18.948+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:58:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:18.948+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:19 smithi115 ceph-mon[35364]: pgmap v2600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:19 smithi181 ceph-mon[35869]: pgmap v2600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:19 smithi097 ceph-mon[32231]: pgmap v2600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3537.692423 secs 2024-07-26T08:58:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3537.692423 secs 2024-07-26T08:58:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3537.692423 secs 2024-07-26T08:58:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:21 smithi115 ceph-mon[35364]: pgmap v2601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:21 smithi181 ceph-mon[35869]: pgmap v2601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:21 smithi097 ceph-mon[32231]: pgmap v2601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:22.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:22.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:58:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:23 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:58:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:23 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:58:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:23 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:58:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:23 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:58:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:23 smithi115 ceph-mon[35364]: pgmap v2602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T08:58:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:23 smithi115 ceph-mon[35364]: pgmap v2603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:58:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:23 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:58:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:23 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:58:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:23 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:58:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:23 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:58:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:23 smithi181 ceph-mon[35869]: pgmap v2602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T08:58:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:23 smithi181 ceph-mon[35869]: pgmap v2603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:58:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:23 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T08:58:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:23 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T08:58:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:23 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T08:58:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:23 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T08:58:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:23 smithi097 ceph-mon[32231]: pgmap v2602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T08:58:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:23 smithi097 ceph-mon[32231]: pgmap v2603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:58:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:25 smithi115 ceph-mon[35364]: pgmap v2604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3542.692565 secs 2024-07-26T08:58:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:25 smithi181 ceph-mon[35869]: pgmap v2604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3542.692565 secs 2024-07-26T08:58:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:25 smithi097 ceph-mon[32231]: pgmap v2604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3542.692565 secs 2024-07-26T08:58:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:27 smithi115 ceph-mon[35364]: pgmap v2605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:27 smithi181 ceph-mon[35869]: pgmap v2605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:27 smithi097 ceph-mon[32231]: pgmap v2605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:29 smithi181 ceph-mon[35869]: pgmap v2606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:58:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:29 smithi097 ceph-mon[32231]: pgmap v2606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:58:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:29 smithi115 ceph-mon[35364]: pgmap v2606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T08:58:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3547.692656 secs 2024-07-26T08:58:30.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3547.692656 secs 2024-07-26T08:58:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3547.692656 secs 2024-07-26T08:58:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:31 smithi097 ceph-mon[32231]: pgmap v2607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:31 smithi115 ceph-mon[35364]: pgmap v2607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:31 smithi181 ceph-mon[35869]: pgmap v2607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:33.953+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:58:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:33.954+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:58:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:33.954+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:58:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:33.954+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:58:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:33.954+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:33 smithi115 ceph-mon[35364]: pgmap v2608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:58:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:33 smithi181 ceph-mon[35869]: pgmap v2608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:58:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:33 smithi097 ceph-mon[32231]: pgmap v2608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:58:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:35 smithi115 ceph-mon[35364]: pgmap v2609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3552.692805 secs 2024-07-26T08:58:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:35 smithi181 ceph-mon[35869]: pgmap v2609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3552.692805 secs 2024-07-26T08:58:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:35 smithi097 ceph-mon[32231]: pgmap v2609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3552.692805 secs 2024-07-26T08:58:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:37 smithi115 ceph-mon[35364]: pgmap v2610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:37 smithi181 ceph-mon[35869]: pgmap v2610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:37 smithi097 ceph-mon[32231]: pgmap v2610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:39 smithi115 ceph-mon[35364]: pgmap v2611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:39 smithi181 ceph-mon[35869]: pgmap v2611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:39 smithi097 ceph-mon[32231]: pgmap v2611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3557.692946 secs 2024-07-26T08:58:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3557.692946 secs 2024-07-26T08:58:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3557.692946 secs 2024-07-26T08:58:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:41 smithi115 ceph-mon[35364]: pgmap v2612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:41 smithi181 ceph-mon[35869]: pgmap v2612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:41 smithi097 ceph-mon[32231]: pgmap v2612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:43 smithi115 ceph-mon[35364]: pgmap v2613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:43 smithi181 ceph-mon[35869]: pgmap v2613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:43 smithi097 ceph-mon[32231]: pgmap v2613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:45 smithi115 ceph-mon[35364]: pgmap v2614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3562.693122 secs 2024-07-26T08:58:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:45 smithi181 ceph-mon[35869]: pgmap v2614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3562.693122 secs 2024-07-26T08:58:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:45 smithi097 ceph-mon[32231]: pgmap v2614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3562.693122 secs 2024-07-26T08:58:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:47 smithi115 ceph-mon[35364]: pgmap v2615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:47 smithi181 ceph-mon[35869]: pgmap v2615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:47 smithi097 ceph-mon[32231]: pgmap v2615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:48.963+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:58:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:48.965+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:58:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:48.965+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:58:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:48.965+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:58:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:58:48.965+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:49 smithi115 ceph-mon[35364]: pgmap v2616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:49 smithi181 ceph-mon[35869]: pgmap v2616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:49 smithi097 ceph-mon[32231]: pgmap v2616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3567.693286 secs 2024-07-26T08:58:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3567.693286 secs 2024-07-26T08:58:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3567.693286 secs 2024-07-26T08:58:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:51 smithi115 ceph-mon[35364]: pgmap v2617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:51 smithi181 ceph-mon[35869]: pgmap v2617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:51 smithi097 ceph-mon[32231]: pgmap v2617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:53 smithi115 ceph-mon[35364]: pgmap v2618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:53 smithi181 ceph-mon[35869]: pgmap v2618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:53 smithi097 ceph-mon[32231]: pgmap v2618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:55 smithi115 ceph-mon[35364]: pgmap v2619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3572.693417 secs 2024-07-26T08:58:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:55 smithi181 ceph-mon[35869]: pgmap v2619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3572.693417 secs 2024-07-26T08:58:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:55 smithi097 ceph-mon[32231]: pgmap v2619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3572.693417 secs 2024-07-26T08:58:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:57 smithi115 ceph-mon[35364]: pgmap v2620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:57 smithi181 ceph-mon[35869]: pgmap v2620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:57 smithi097 ceph-mon[32231]: pgmap v2620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:58:59 smithi115 ceph-mon[35364]: pgmap v2621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:58:59 smithi181 ceph-mon[35869]: pgmap v2621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:58:59 smithi097 ceph-mon[32231]: pgmap v2621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:00.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3577.693564 secs 2024-07-26T08:59:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3577.693564 secs 2024-07-26T08:59:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3577.693564 secs 2024-07-26T08:59:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:01 smithi115 ceph-mon[35364]: pgmap v2622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:01 smithi181 ceph-mon[35869]: pgmap v2622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:01 smithi097 ceph-mon[32231]: pgmap v2622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:03.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:03.969+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:59:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:03.970+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:59:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:03.970+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:59:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:03.970+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:59:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:03.971+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:03 smithi115 ceph-mon[35364]: pgmap v2623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:03 smithi181 ceph-mon[35869]: pgmap v2623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:03 smithi097 ceph-mon[32231]: pgmap v2623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:05 smithi115 ceph-mon[35364]: pgmap v2624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:59:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3582.693703 secs 2024-07-26T08:59:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:05 smithi181 ceph-mon[35869]: pgmap v2624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:59:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3582.693703 secs 2024-07-26T08:59:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:05 smithi097 ceph-mon[32231]: pgmap v2624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:59:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3582.693703 secs 2024-07-26T08:59:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:07 smithi115 ceph-mon[35364]: pgmap v2625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:07 smithi181 ceph-mon[35869]: pgmap v2625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:07 smithi097 ceph-mon[32231]: pgmap v2625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:09 smithi115 ceph-mon[35364]: pgmap v2626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:09 smithi181 ceph-mon[35869]: pgmap v2626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:09 smithi097 ceph-mon[32231]: pgmap v2626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3587.693858 secs 2024-07-26T08:59:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3587.693858 secs 2024-07-26T08:59:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3587.693858 secs 2024-07-26T08:59:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:11 smithi115 ceph-mon[35364]: pgmap v2627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:11 smithi181 ceph-mon[35869]: pgmap v2627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:11 smithi097 ceph-mon[32231]: pgmap v2627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:13 smithi115 ceph-mon[35364]: pgmap v2628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:13 smithi181 ceph-mon[35869]: pgmap v2628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:13 smithi097 ceph-mon[32231]: pgmap v2628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:15 smithi115 ceph-mon[35364]: pgmap v2629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:59:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3592.693949 secs 2024-07-26T08:59:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:15 smithi181 ceph-mon[35869]: pgmap v2629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:59:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3592.693949 secs 2024-07-26T08:59:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:15 smithi097 ceph-mon[32231]: pgmap v2629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:59:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3592.693949 secs 2024-07-26T08:59:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:17 smithi115 ceph-mon[35364]: pgmap v2630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:17 smithi181 ceph-mon[35869]: pgmap v2630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:17 smithi097 ceph-mon[32231]: pgmap v2630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:18.643 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() TimeoutError 2024-07-26T08:59:18.877 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() TimeoutError 2024-07-26T08:59:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:18.979+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:59:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:18.980+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:59:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:18.980+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:59:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:18.980+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:59:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:18.981+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:19 smithi115 ceph-mon[35364]: pgmap v2631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:19 smithi181 ceph-mon[35869]: pgmap v2631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:19 smithi097 ceph-mon[32231]: pgmap v2631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3597.694076 secs 2024-07-26T08:59:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3597.694076 secs 2024-07-26T08:59:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3597.694076 secs 2024-07-26T08:59:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:21 smithi115 ceph-mon[35364]: pgmap v2632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:21 smithi181 ceph-mon[35869]: pgmap v2632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:21 smithi097 ceph-mon[32231]: pgmap v2632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:23 smithi181 ceph-mon[35869]: pgmap v2633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:23 smithi097 ceph-mon[32231]: pgmap v2633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:23 smithi115 ceph-mon[35364]: pgmap v2633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:59:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:59:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:25.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:25.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: pgmap v2634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:59:25.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: pgmap v2635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:25.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:59:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:59:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:59:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: pgmap v2634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:59:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: pgmap v2635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:25.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:59:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:59:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T08:59:25.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T08:59:25.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:25.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:25.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: pgmap v2634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:59:25.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: pgmap v2635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:59:25.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T08:59:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3602.694263 secs 2024-07-26T08:59:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3602.694263 secs 2024-07-26T08:59:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3602.694263 secs 2024-07-26T08:59:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:26 smithi097 ceph-mon[32231]: pgmap v2636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:27.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:26 smithi115 ceph-mon[35364]: pgmap v2636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:27.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:26 smithi181 ceph-mon[35869]: pgmap v2636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:28 smithi115 ceph-mon[35364]: pgmap v2637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:28 smithi181 ceph-mon[35869]: pgmap v2637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:28 smithi097 ceph-mon[32231]: pgmap v2637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3607.694399 secs 2024-07-26T08:59:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:30 smithi115 ceph-mon[35364]: pgmap v2638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3607.694399 secs 2024-07-26T08:59:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:30 smithi181 ceph-mon[35869]: pgmap v2638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3607.694399 secs 2024-07-26T08:59:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:30 smithi097 ceph-mon[32231]: pgmap v2638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:32 smithi115 ceph-mon[35364]: pgmap v2639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:32 smithi181 ceph-mon[35869]: pgmap v2639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:32 smithi097 ceph-mon[32231]: pgmap v2639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:33.985+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:59:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:33.986+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:59:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:33.986+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:59:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:33.986+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:59:33.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:33.987+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:34 smithi115 ceph-mon[35364]: pgmap v2640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:59:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:34 smithi181 ceph-mon[35869]: pgmap v2640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:59:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:34 smithi097 ceph-mon[32231]: pgmap v2640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:59:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3612.694541 secs 2024-07-26T08:59:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3612.694541 secs 2024-07-26T08:59:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3612.694541 secs 2024-07-26T08:59:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:36 smithi115 ceph-mon[35364]: pgmap v2641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:36 smithi181 ceph-mon[35869]: pgmap v2641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:36 smithi097 ceph-mon[32231]: pgmap v2641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:38 smithi115 ceph-mon[35364]: pgmap v2642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:38 smithi181 ceph-mon[35869]: pgmap v2642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:38 smithi097 ceph-mon[32231]: pgmap v2642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3617.694681 secs 2024-07-26T08:59:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:40 smithi181 ceph-mon[35869]: pgmap v2643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3617.694681 secs 2024-07-26T08:59:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:40 smithi097 ceph-mon[32231]: pgmap v2643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3617.694681 secs 2024-07-26T08:59:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:40 smithi115 ceph-mon[35364]: pgmap v2643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:42 smithi097 ceph-mon[32231]: pgmap v2644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:42 smithi115 ceph-mon[35364]: pgmap v2644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:42 smithi181 ceph-mon[35869]: pgmap v2644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:44 smithi097 ceph-mon[32231]: pgmap v2645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:44 smithi115 ceph-mon[35364]: pgmap v2645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:44 smithi181 ceph-mon[35869]: pgmap v2645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3622.694765 secs 2024-07-26T08:59:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3622.694765 secs 2024-07-26T08:59:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3622.694765 secs 2024-07-26T08:59:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:47 smithi115 ceph-mon[35364]: pgmap v2646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:47 smithi181 ceph-mon[35869]: pgmap v2646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:47 smithi097 ceph-mon[32231]: pgmap v2646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:48 smithi115 ceph-mon[35364]: pgmap v2647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:48 smithi181 ceph-mon[35869]: pgmap v2647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:48 smithi097 ceph-mon[32231]: pgmap v2647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:48.997+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:59:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:48.998+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T08:59:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:48.998+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:59:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:48.998+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:59:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T08:59:48.998+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3627.694895 secs 2024-07-26T08:59:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:50 smithi115 ceph-mon[35364]: pgmap v2648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:59:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3627.694895 secs 2024-07-26T08:59:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:50 smithi181 ceph-mon[35869]: pgmap v2648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:59:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3627.694895 secs 2024-07-26T08:59:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:50 smithi097 ceph-mon[32231]: pgmap v2648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:59:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:52 smithi115 ceph-mon[35364]: pgmap v2649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:59:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:52 smithi181 ceph-mon[35869]: pgmap v2649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:59:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:52 smithi097 ceph-mon[32231]: pgmap v2649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:59:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:54 smithi115 ceph-mon[35364]: pgmap v2650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:59:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:54 smithi181 ceph-mon[35869]: pgmap v2650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:59:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:54 smithi097 ceph-mon[32231]: pgmap v2650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:59:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3632.694983 secs 2024-07-26T08:59:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3632.694983 secs 2024-07-26T08:59:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3632.694983 secs 2024-07-26T08:59:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:56 smithi115 ceph-mon[35364]: pgmap v2651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:59:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:56 smithi181 ceph-mon[35869]: pgmap v2651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:59:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:56 smithi097 ceph-mon[32231]: pgmap v2651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:59:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 08:59:58 smithi181 ceph-mon[35869]: pgmap v2652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:59:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 08:59:58 smithi097 ceph-mon[32231]: pgmap v2652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:59:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 08:59:58 smithi115 ceph-mon[35364]: pgmap v2652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:00:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3637.695120 secs 2024-07-26T09:00:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:00 smithi097 ceph-mon[32231]: pgmap v2653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:00:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:00:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3637.695120 secs 2024-07-26T09:00:01.484 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:00 smithi115 ceph-mon[35364]: pgmap v2653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:00:01.484 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:00:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3637.695120 secs 2024-07-26T09:00:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:00 smithi181 ceph-mon[35869]: pgmap v2653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:00:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:00:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:02 smithi097 ceph-mon[32231]: pgmap v2654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:02 smithi115 ceph-mon[35364]: pgmap v2654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:02 smithi181 ceph-mon[35869]: pgmap v2654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:04.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:04.001+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:00:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:04.002+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:00:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:04.002+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:00:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:04.002+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:00:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:04.002+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:04 smithi097 ceph-mon[32231]: pgmap v2655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:04 smithi115 ceph-mon[35364]: pgmap v2655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:04 smithi181 ceph-mon[35869]: pgmap v2655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3642.695286 secs 2024-07-26T09:00:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3642.695286 secs 2024-07-26T09:00:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3642.695286 secs 2024-07-26T09:00:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:07 smithi115 ceph-mon[35364]: pgmap v2656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:00:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:07 smithi181 ceph-mon[35869]: pgmap v2656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:00:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:07 smithi097 ceph-mon[32231]: pgmap v2656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:00:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:08 smithi115 ceph-mon[35364]: pgmap v2657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:08 smithi181 ceph-mon[35869]: pgmap v2657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:08 smithi097 ceph-mon[32231]: pgmap v2657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3647.695441 secs 2024-07-26T09:00:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:10 smithi115 ceph-mon[35364]: pgmap v2658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3647.695441 secs 2024-07-26T09:00:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:10 smithi181 ceph-mon[35869]: pgmap v2658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3647.695441 secs 2024-07-26T09:00:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:10 smithi097 ceph-mon[32231]: pgmap v2658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:12 smithi115 ceph-mon[35364]: pgmap v2659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:12 smithi181 ceph-mon[35869]: pgmap v2659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:12 smithi097 ceph-mon[32231]: pgmap v2659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:14 smithi115 ceph-mon[35364]: pgmap v2660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:14 smithi181 ceph-mon[35869]: pgmap v2660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:14 smithi097 ceph-mon[32231]: pgmap v2660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3652.695581 secs 2024-07-26T09:00:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3652.695581 secs 2024-07-26T09:00:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3652.695581 secs 2024-07-26T09:00:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:16 smithi115 ceph-mon[35364]: pgmap v2661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:00:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:16 smithi181 ceph-mon[35869]: pgmap v2661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:00:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:16 smithi097 ceph-mon[32231]: pgmap v2661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:00:19.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:19.011+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:00:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:19.012+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:00:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:19.012+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:00:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:19.012+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:00:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:19.013+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:18 smithi181 ceph-mon[35869]: pgmap v2662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:18 smithi097 ceph-mon[32231]: pgmap v2662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:18 smithi115 ceph-mon[35364]: pgmap v2662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3657.695726 secs 2024-07-26T09:00:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:20 smithi097 ceph-mon[32231]: pgmap v2663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3657.695726 secs 2024-07-26T09:00:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:20 smithi115 ceph-mon[35364]: pgmap v2663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3657.695726 secs 2024-07-26T09:00:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:20 smithi181 ceph-mon[35869]: pgmap v2663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:23 smithi115 ceph-mon[35364]: pgmap v2664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:23 smithi181 ceph-mon[35869]: pgmap v2664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:23 smithi097 ceph-mon[32231]: pgmap v2664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:24 smithi115 ceph-mon[35364]: pgmap v2665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:24 smithi181 ceph-mon[35869]: pgmap v2665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:24 smithi097 ceph-mon[32231]: pgmap v2665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:00:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3662.695867 secs 2024-07-26T09:00:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:26 smithi115 ceph-mon[35364]: pgmap v2666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:00:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:26 smithi115 ceph-mon[35364]: pgmap v2667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:26.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3662.695867 secs 2024-07-26T09:00:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:26 smithi181 ceph-mon[35869]: pgmap v2666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:00:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:26 smithi181 ceph-mon[35869]: pgmap v2667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3662.695867 secs 2024-07-26T09:00:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:26 smithi097 ceph-mon[32231]: pgmap v2666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:00:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:26 smithi097 ceph-mon[32231]: pgmap v2667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:00:29.178 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:28 smithi115 ceph-mon[35364]: pgmap v2668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:29.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:28 smithi181 ceph-mon[35869]: pgmap v2668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:28 smithi097 ceph-mon[32231]: pgmap v2668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3667.696004 secs 2024-07-26T09:00:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:30 smithi115 ceph-mon[35364]: pgmap v2669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:31.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3667.696004 secs 2024-07-26T09:00:31.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:30 smithi181 ceph-mon[35869]: pgmap v2669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3667.696004 secs 2024-07-26T09:00:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:30 smithi097 ceph-mon[32231]: pgmap v2669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:32 smithi115 ceph-mon[35364]: pgmap v2670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:32 smithi181 ceph-mon[35869]: pgmap v2670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:33.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:32 smithi097 ceph-mon[32231]: pgmap v2670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:34.018+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:00:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:34.019+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:00:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:34.019+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:00:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:34.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:00:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:34.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:34 smithi115 ceph-mon[35364]: pgmap v2671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:00:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:34 smithi181 ceph-mon[35869]: pgmap v2671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:00:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:34 smithi097 ceph-mon[32231]: pgmap v2671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:00:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3672.696143 secs 2024-07-26T09:00:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3672.696143 secs 2024-07-26T09:00:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3672.696143 secs 2024-07-26T09:00:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:36 smithi115 ceph-mon[35364]: pgmap v2672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:36 smithi181 ceph-mon[35869]: pgmap v2672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:36 smithi097 ceph-mon[32231]: pgmap v2672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:38 smithi115 ceph-mon[35364]: pgmap v2673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:38 smithi181 ceph-mon[35869]: pgmap v2673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:38 smithi097 ceph-mon[32231]: pgmap v2673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3677.696319 secs 2024-07-26T09:00:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:40 smithi115 ceph-mon[35364]: pgmap v2674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3677.696319 secs 2024-07-26T09:00:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:40 smithi181 ceph-mon[35869]: pgmap v2674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3677.696319 secs 2024-07-26T09:00:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:40 smithi097 ceph-mon[32231]: pgmap v2674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:42 smithi115 ceph-mon[35364]: pgmap v2675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:42 smithi181 ceph-mon[35869]: pgmap v2675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:42 smithi097 ceph-mon[32231]: pgmap v2675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:44 smithi115 ceph-mon[35364]: pgmap v2676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:44 smithi181 ceph-mon[35869]: pgmap v2676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:44 smithi097 ceph-mon[32231]: pgmap v2676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3682.696409 secs 2024-07-26T09:00:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3682.696409 secs 2024-07-26T09:00:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3682.696409 secs 2024-07-26T09:00:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:46 smithi181 ceph-mon[35869]: pgmap v2677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:46 smithi097 ceph-mon[32231]: pgmap v2677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:46 smithi115 ceph-mon[35364]: pgmap v2677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:49.029+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:00:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:49.030+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:00:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:49.030+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:00:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:49.030+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:00:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:00:49.030+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:48 smithi097 ceph-mon[32231]: pgmap v2678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:48 smithi115 ceph-mon[35364]: pgmap v2678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:48 smithi181 ceph-mon[35869]: pgmap v2678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3687.696539 secs 2024-07-26T09:00:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:50 smithi097 ceph-mon[32231]: pgmap v2679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3687.696539 secs 2024-07-26T09:00:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:50 smithi115 ceph-mon[35364]: pgmap v2679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3687.696539 secs 2024-07-26T09:00:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:50 smithi181 ceph-mon[35869]: pgmap v2679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:52 smithi097 ceph-mon[32231]: pgmap v2680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:53 smithi115 ceph-mon[35364]: pgmap v2680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:52 smithi181 ceph-mon[35869]: pgmap v2680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:55 smithi115 ceph-mon[35364]: pgmap v2681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:55 smithi181 ceph-mon[35869]: pgmap v2681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:55 smithi097 ceph-mon[32231]: pgmap v2681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:00:56.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3692.696704 secs 2024-07-26T09:00:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3692.696704 secs 2024-07-26T09:00:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3692.696704 secs 2024-07-26T09:00:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:57 smithi115 ceph-mon[35364]: pgmap v2682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:57 smithi181 ceph-mon[35869]: pgmap v2682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:57 smithi097 ceph-mon[32231]: pgmap v2682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:00:58 smithi115 ceph-mon[35364]: pgmap v2683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:00:58 smithi181 ceph-mon[35869]: pgmap v2683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:00:58 smithi097 ceph-mon[32231]: pgmap v2683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:01.178 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3697.696842 secs 2024-07-26T09:01:01.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:00 smithi115 ceph-mon[35364]: pgmap v2684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:01.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3697.696842 secs 2024-07-26T09:01:01.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:00 smithi181 ceph-mon[35869]: pgmap v2684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:01.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3697.696842 secs 2024-07-26T09:01:01.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:00 smithi097 ceph-mon[32231]: pgmap v2684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:02 smithi115 ceph-mon[35364]: pgmap v2685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:02 smithi181 ceph-mon[35869]: pgmap v2685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:02 smithi097 ceph-mon[32231]: pgmap v2685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:04.034+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:01:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:04.035+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:01:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:04.035+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:01:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:04.035+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:01:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:04.035+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:04 smithi115 ceph-mon[35364]: pgmap v2686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:04 smithi181 ceph-mon[35869]: pgmap v2686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:04 smithi097 ceph-mon[32231]: pgmap v2686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3702.696979 secs 2024-07-26T09:01:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3702.696979 secs 2024-07-26T09:01:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3702.696979 secs 2024-07-26T09:01:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:06 smithi115 ceph-mon[35364]: pgmap v2687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:06 smithi181 ceph-mon[35869]: pgmap v2687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:06 smithi097 ceph-mon[32231]: pgmap v2687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:07.523+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:01:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:07.523+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T09:01:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:07.524+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T09:01:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:07.525+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:01:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:08 smithi115 ceph-mon[35364]: pgmap v2688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:08 smithi181 ceph-mon[35869]: pgmap v2688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:08 smithi097 ceph-mon[32231]: pgmap v2688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3707.697133 secs 2024-07-26T09:01:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:10 smithi115 ceph-mon[35364]: pgmap v2689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3707.697133 secs 2024-07-26T09:01:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:10 smithi181 ceph-mon[35869]: pgmap v2689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3707.697133 secs 2024-07-26T09:01:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:10 smithi097 ceph-mon[32231]: pgmap v2689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:12 smithi115 ceph-mon[35364]: pgmap v2690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:12 smithi181 ceph-mon[35869]: pgmap v2690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:12 smithi097 ceph-mon[32231]: pgmap v2690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:14 smithi181 ceph-mon[35869]: pgmap v2691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:14 smithi097 ceph-mon[32231]: pgmap v2691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:14 smithi115 ceph-mon[35364]: pgmap v2691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3712.697276 secs 2024-07-26T09:01:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3712.697276 secs 2024-07-26T09:01:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3712.697276 secs 2024-07-26T09:01:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:16 smithi097 ceph-mon[32231]: pgmap v2692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:16 smithi115 ceph-mon[35364]: pgmap v2692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:16 smithi181 ceph-mon[35869]: pgmap v2692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:19.043+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:01:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:19.044+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:01:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:19.044+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:01:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:19.044+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:01:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:19.044+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:18 smithi097 ceph-mon[32231]: pgmap v2693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:19 smithi115 ceph-mon[35364]: pgmap v2693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:19 smithi181 ceph-mon[35869]: pgmap v2693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3717.697435 secs 2024-07-26T09:01:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:21 smithi115 ceph-mon[35364]: pgmap v2694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3717.697435 secs 2024-07-26T09:01:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:21 smithi181 ceph-mon[35869]: pgmap v2694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3717.697435 secs 2024-07-26T09:01:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:21 smithi097 ceph-mon[32231]: pgmap v2694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:23 smithi115 ceph-mon[35364]: pgmap v2695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:23 smithi181 ceph-mon[35869]: pgmap v2695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:23 smithi097 ceph-mon[32231]: pgmap v2695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:25 smithi115 ceph-mon[35364]: pgmap v2696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:25 smithi181 ceph-mon[35869]: pgmap v2696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:25 smithi097 ceph-mon[32231]: pgmap v2696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3722.697583 secs 2024-07-26T09:01:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:26 smithi115 ceph-mon[35364]: pgmap v2697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3722.697583 secs 2024-07-26T09:01:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:26 smithi181 ceph-mon[35869]: pgmap v2697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3722.697583 secs 2024-07-26T09:01:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:26 smithi097 ceph-mon[32231]: pgmap v2697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:28 smithi115 ceph-mon[35364]: pgmap v2698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:28 smithi115 ceph-mon[35364]: pgmap v2699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:28 smithi181 ceph-mon[35869]: pgmap v2698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:28 smithi181 ceph-mon[35869]: pgmap v2699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:28 smithi097 ceph-mon[32231]: pgmap v2698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:28 smithi097 ceph-mon[32231]: pgmap v2699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:01:31.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3727.697751 secs 2024-07-26T09:01:31.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:30 smithi115 ceph-mon[35364]: pgmap v2700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:31.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3727.697751 secs 2024-07-26T09:01:31.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:30 smithi181 ceph-mon[35869]: pgmap v2700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:31.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3727.697751 secs 2024-07-26T09:01:31.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:30 smithi097 ceph-mon[32231]: pgmap v2700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:32 smithi115 ceph-mon[35364]: pgmap v2701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:32 smithi181 ceph-mon[35869]: pgmap v2701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:32 smithi097 ceph-mon[32231]: pgmap v2701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:34.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:34.048+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:01:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:34.050+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:01:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:34.050+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:01:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:34.050+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:01:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:34.050+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:34 smithi115 ceph-mon[35364]: pgmap v2702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:34 smithi181 ceph-mon[35869]: pgmap v2702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:34 smithi097 ceph-mon[32231]: pgmap v2702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3732.697895 secs 2024-07-26T09:01:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3732.697895 secs 2024-07-26T09:01:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3732.697895 secs 2024-07-26T09:01:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:36 smithi115 ceph-mon[35364]: pgmap v2703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:36 smithi181 ceph-mon[35869]: pgmap v2703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:36 smithi097 ceph-mon[32231]: pgmap v2703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:01:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:38 smithi115 ceph-mon[35364]: pgmap v2704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:38 smithi181 ceph-mon[35869]: pgmap v2704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:38 smithi097 ceph-mon[32231]: pgmap v2704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3737.698032 secs 2024-07-26T09:01:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:40 smithi115 ceph-mon[35364]: pgmap v2705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3737.698032 secs 2024-07-26T09:01:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:40 smithi181 ceph-mon[35869]: pgmap v2705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3737.698032 secs 2024-07-26T09:01:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:40 smithi097 ceph-mon[32231]: pgmap v2705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:42 smithi115 ceph-mon[35364]: pgmap v2706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:42 smithi181 ceph-mon[35869]: pgmap v2706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:42 smithi097 ceph-mon[32231]: pgmap v2706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:44 smithi115 ceph-mon[35364]: pgmap v2707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:44 smithi181 ceph-mon[35869]: pgmap v2707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:44 smithi097 ceph-mon[32231]: pgmap v2707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3742.698178 secs 2024-07-26T09:01:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3742.698178 secs 2024-07-26T09:01:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3742.698178 secs 2024-07-26T09:01:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:46 smithi181 ceph-mon[35869]: pgmap v2708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:46 smithi097 ceph-mon[32231]: pgmap v2708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:46 smithi115 ceph-mon[35364]: pgmap v2708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:49.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:49.059+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:01:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:49.061+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:01:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:49.061+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:01:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:49.061+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:01:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:01:49.061+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:48 smithi097 ceph-mon[32231]: pgmap v2709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:48 smithi115 ceph-mon[35364]: pgmap v2709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:48 smithi181 ceph-mon[35869]: pgmap v2709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3747.698273 secs 2024-07-26T09:01:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:50 smithi097 ceph-mon[32231]: pgmap v2710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3747.698273 secs 2024-07-26T09:01:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:50 smithi115 ceph-mon[35364]: pgmap v2710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3747.698273 secs 2024-07-26T09:01:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:50 smithi181 ceph-mon[35869]: pgmap v2710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:53 smithi115 ceph-mon[35364]: pgmap v2711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:53 smithi181 ceph-mon[35869]: pgmap v2711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:53 smithi097 ceph-mon[32231]: pgmap v2711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:55 smithi115 ceph-mon[35364]: pgmap v2712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:55 smithi181 ceph-mon[35869]: pgmap v2712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:55 smithi097 ceph-mon[32231]: pgmap v2712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3752.698353 secs 2024-07-26T09:01:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3752.698353 secs 2024-07-26T09:01:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3752.698353 secs 2024-07-26T09:01:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:57 smithi115 ceph-mon[35364]: pgmap v2713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:57 smithi181 ceph-mon[35869]: pgmap v2713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:57 smithi097 ceph-mon[32231]: pgmap v2713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:01:58 smithi115 ceph-mon[35364]: pgmap v2714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:01:58 smithi181 ceph-mon[35869]: pgmap v2714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:01:58 smithi097 ceph-mon[32231]: pgmap v2714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:01.178 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3757.698490 secs 2024-07-26T09:02:01.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:00 smithi115 ceph-mon[35364]: pgmap v2715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:01.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3757.698490 secs 2024-07-26T09:02:01.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:00 smithi181 ceph-mon[35869]: pgmap v2715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:01.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3757.698490 secs 2024-07-26T09:02:01.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:00 smithi097 ceph-mon[32231]: pgmap v2715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:02 smithi115 ceph-mon[35364]: pgmap v2716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:02 smithi181 ceph-mon[35869]: pgmap v2716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:02 smithi097 ceph-mon[32231]: pgmap v2716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:04.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:04.066+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:02:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:04.067+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:02:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:04.067+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:02:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:04.067+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:02:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:04.067+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:04 smithi115 ceph-mon[35364]: pgmap v2717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:04 smithi181 ceph-mon[35869]: pgmap v2717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:04 smithi097 ceph-mon[32231]: pgmap v2717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3762.698629 secs 2024-07-26T09:02:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3762.698629 secs 2024-07-26T09:02:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3762.698629 secs 2024-07-26T09:02:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:06 smithi115 ceph-mon[35364]: pgmap v2718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:06 smithi181 ceph-mon[35869]: pgmap v2718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:06 smithi097 ceph-mon[32231]: pgmap v2718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:08 smithi115 ceph-mon[35364]: pgmap v2719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:08 smithi181 ceph-mon[35869]: pgmap v2719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:08 smithi097 ceph-mon[32231]: pgmap v2719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3767.698758 secs 2024-07-26T09:02:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:10 smithi115 ceph-mon[35364]: pgmap v2720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3767.698758 secs 2024-07-26T09:02:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:10 smithi181 ceph-mon[35869]: pgmap v2720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3767.698758 secs 2024-07-26T09:02:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:10 smithi097 ceph-mon[32231]: pgmap v2720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:13.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:12 smithi181 ceph-mon[35869]: pgmap v2721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:12 smithi097 ceph-mon[32231]: pgmap v2721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:12 smithi115 ceph-mon[35364]: pgmap v2721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:14 smithi097 ceph-mon[32231]: pgmap v2722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:14 smithi115 ceph-mon[35364]: pgmap v2722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:14 smithi181 ceph-mon[35869]: pgmap v2722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3772.698855 secs 2024-07-26T09:02:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3772.698855 secs 2024-07-26T09:02:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3772.698855 secs 2024-07-26T09:02:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:16 smithi097 ceph-mon[32231]: pgmap v2723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:16 smithi115 ceph-mon[35364]: pgmap v2723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:16 smithi181 ceph-mon[35869]: pgmap v2723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:19.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:19.077+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:02:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:19.078+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:02:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:19.078+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:02:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:19.078+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:02:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:19.078+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:19 smithi115 ceph-mon[35364]: pgmap v2724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:19 smithi181 ceph-mon[35869]: pgmap v2724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:19 smithi097 ceph-mon[32231]: pgmap v2724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3777.699000 secs 2024-07-26T09:02:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:21 smithi115 ceph-mon[35364]: pgmap v2725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3777.699000 secs 2024-07-26T09:02:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:21 smithi181 ceph-mon[35869]: pgmap v2725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3777.699000 secs 2024-07-26T09:02:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:21 smithi097 ceph-mon[32231]: pgmap v2725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:23 smithi115 ceph-mon[35364]: pgmap v2726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:23 smithi181 ceph-mon[35869]: pgmap v2726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:23 smithi097 ceph-mon[32231]: pgmap v2726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:25 smithi115 ceph-mon[35364]: pgmap v2727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:25 smithi181 ceph-mon[35869]: pgmap v2727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:25 smithi097 ceph-mon[32231]: pgmap v2727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3782.699171 secs 2024-07-26T09:02:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3782.699171 secs 2024-07-26T09:02:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3782.699171 secs 2024-07-26T09:02:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:27 smithi115 ceph-mon[35364]: pgmap v2728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:27 smithi181 ceph-mon[35869]: pgmap v2728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:27 smithi097 ceph-mon[32231]: pgmap v2728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:28 smithi115 ceph-mon[35364]: pgmap v2729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:28 smithi181 ceph-mon[35869]: pgmap v2729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:28 smithi097 ceph-mon[32231]: pgmap v2729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:29.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:29.468 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:29.475 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:30 smithi115 ceph-mon[35364]: pgmap v2730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:30 smithi181 ceph-mon[35869]: pgmap v2730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:30 smithi097 ceph-mon[32231]: pgmap v2730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:31.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3787.699348 secs 2024-07-26T09:02:31.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3787.699348 secs 2024-07-26T09:02:31.405 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3787.699348 secs 2024-07-26T09:02:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:32 smithi097 ceph-mon[32231]: pgmap v2731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:32 smithi115 ceph-mon[35364]: pgmap v2731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:32.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:32 smithi181 ceph-mon[35869]: pgmap v2731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:33.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:33.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:33 smithi097 ceph-mon[32231]: pgmap v2732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:33.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:33 smithi115 ceph-mon[35364]: pgmap v2732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:33 smithi181 ceph-mon[35869]: pgmap v2732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:02:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:34.083+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:02:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:34.084+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:02:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:34.084+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:02:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:34.084+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:02:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:34.084+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:35 smithi181 ceph-mon[35869]: pgmap v2733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3792.699442 secs 2024-07-26T09:02:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:35 smithi097 ceph-mon[32231]: pgmap v2733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3792.699442 secs 2024-07-26T09:02:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:35 smithi115 ceph-mon[35364]: pgmap v2733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3792.699442 secs 2024-07-26T09:02:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:37 smithi115 ceph-mon[35364]: pgmap v2734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:37 smithi181 ceph-mon[35869]: pgmap v2734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:37 smithi097 ceph-mon[32231]: pgmap v2734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:39 smithi115 ceph-mon[35364]: pgmap v2735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:02:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:39 smithi181 ceph-mon[35869]: pgmap v2735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:02:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:39 smithi097 ceph-mon[32231]: pgmap v2735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:02:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3797.699582 secs 2024-07-26T09:02:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3797.699582 secs 2024-07-26T09:02:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3797.699582 secs 2024-07-26T09:02:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:41 smithi115 ceph-mon[35364]: pgmap v2736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:41 smithi181 ceph-mon[35869]: pgmap v2736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:41 smithi097 ceph-mon[32231]: pgmap v2736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:43 smithi115 ceph-mon[35364]: pgmap v2737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:43 smithi181 ceph-mon[35869]: pgmap v2737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:43 smithi097 ceph-mon[32231]: pgmap v2737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:45 smithi115 ceph-mon[35364]: pgmap v2738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3802.699677 secs 2024-07-26T09:02:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:45 smithi181 ceph-mon[35869]: pgmap v2738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3802.699677 secs 2024-07-26T09:02:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:45 smithi097 ceph-mon[32231]: pgmap v2738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3802.699677 secs 2024-07-26T09:02:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:47 smithi115 ceph-mon[35364]: pgmap v2739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:47 smithi181 ceph-mon[35869]: pgmap v2739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:47 smithi097 ceph-mon[32231]: pgmap v2739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:49.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:02:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:49.094+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:02:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:49.094+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:02:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:49.094+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:02:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:02:49.095+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:49 smithi115 ceph-mon[35364]: pgmap v2740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:49 smithi181 ceph-mon[35869]: pgmap v2740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:49 smithi097 ceph-mon[32231]: pgmap v2740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3807.699814 secs 2024-07-26T09:02:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3807.699814 secs 2024-07-26T09:02:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3807.699814 secs 2024-07-26T09:02:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:51 smithi115 ceph-mon[35364]: pgmap v2741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:51 smithi181 ceph-mon[35869]: pgmap v2741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:51 smithi097 ceph-mon[32231]: pgmap v2741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:53 smithi115 ceph-mon[35364]: pgmap v2742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:53 smithi181 ceph-mon[35869]: pgmap v2742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:53 smithi097 ceph-mon[32231]: pgmap v2742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:55 smithi115 ceph-mon[35364]: pgmap v2743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3812.699973 secs 2024-07-26T09:02:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:55 smithi181 ceph-mon[35869]: pgmap v2743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3812.699973 secs 2024-07-26T09:02:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:55 smithi097 ceph-mon[32231]: pgmap v2743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3812.699973 secs 2024-07-26T09:02:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:57 smithi115 ceph-mon[35364]: pgmap v2744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:57 smithi181 ceph-mon[35869]: pgmap v2744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:57 smithi097 ceph-mon[32231]: pgmap v2744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:02:59 smithi115 ceph-mon[35364]: pgmap v2745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:02:59 smithi181 ceph-mon[35869]: pgmap v2745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:02:59 smithi097 ceph-mon[32231]: pgmap v2745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3817.700129 secs 2024-07-26T09:03:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3817.700129 secs 2024-07-26T09:03:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3817.700129 secs 2024-07-26T09:03:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:01 smithi115 ceph-mon[35364]: pgmap v2746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:01 smithi181 ceph-mon[35869]: pgmap v2746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:01 smithi097 ceph-mon[32231]: pgmap v2746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:04.100+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:03:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:04.101+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:03:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:04.101+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:03:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:04.101+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:03:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:04.101+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:03 smithi115 ceph-mon[35364]: pgmap v2747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:03 smithi181 ceph-mon[35869]: pgmap v2747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:03 smithi097 ceph-mon[32231]: pgmap v2747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:05 smithi115 ceph-mon[35364]: pgmap v2748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3822.700272 secs 2024-07-26T09:03:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:05 smithi181 ceph-mon[35869]: pgmap v2748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3822.700272 secs 2024-07-26T09:03:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:05 smithi097 ceph-mon[32231]: pgmap v2748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3822.700272 secs 2024-07-26T09:03:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:07 smithi115 ceph-mon[35364]: pgmap v2749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:07 smithi181 ceph-mon[35869]: pgmap v2749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:07 smithi097 ceph-mon[32231]: pgmap v2749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:09 smithi115 ceph-mon[35364]: pgmap v2750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:09 smithi181 ceph-mon[35869]: pgmap v2750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:09 smithi097 ceph-mon[32231]: pgmap v2750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3827.700413 secs 2024-07-26T09:03:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3827.700413 secs 2024-07-26T09:03:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3827.700413 secs 2024-07-26T09:03:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:11 smithi181 ceph-mon[35869]: pgmap v2751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:11 smithi097 ceph-mon[32231]: pgmap v2751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:11 smithi115 ceph-mon[35364]: pgmap v2751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:13 smithi097 ceph-mon[32231]: pgmap v2752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:13 smithi115 ceph-mon[35364]: pgmap v2752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:13 smithi181 ceph-mon[35869]: pgmap v2752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:15 smithi097 ceph-mon[32231]: pgmap v2753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3832.700560 secs 2024-07-26T09:03:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:15 smithi115 ceph-mon[35364]: pgmap v2753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3832.700560 secs 2024-07-26T09:03:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:15 smithi181 ceph-mon[35869]: pgmap v2753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3832.700560 secs 2024-07-26T09:03:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:17 smithi097 ceph-mon[32231]: pgmap v2754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:17 smithi115 ceph-mon[35364]: pgmap v2754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:17 smithi181 ceph-mon[35869]: pgmap v2754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:19.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:19.111+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:03:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:19.112+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:03:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:19.112+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:03:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:19.112+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:03:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:19.112+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:20 smithi115 ceph-mon[35364]: pgmap v2755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:20 smithi181 ceph-mon[35869]: pgmap v2755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:20 smithi097 ceph-mon[32231]: pgmap v2755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3837.700652 secs 2024-07-26T09:03:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3837.700652 secs 2024-07-26T09:03:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3837.700652 secs 2024-07-26T09:03:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:22 smithi115 ceph-mon[35364]: pgmap v2756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:22 smithi181 ceph-mon[35869]: pgmap v2756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:22 smithi097 ceph-mon[32231]: pgmap v2756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:24 smithi115 ceph-mon[35364]: pgmap v2757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:24 smithi181 ceph-mon[35869]: pgmap v2757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:24 smithi097 ceph-mon[32231]: pgmap v2757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:26 smithi115 ceph-mon[35364]: pgmap v2758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3842.700785 secs 2024-07-26T09:03:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:26 smithi181 ceph-mon[35869]: pgmap v2758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3842.700785 secs 2024-07-26T09:03:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:26 smithi097 ceph-mon[32231]: pgmap v2758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3842.700785 secs 2024-07-26T09:03:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:28 smithi115 ceph-mon[35364]: pgmap v2759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:28 smithi181 ceph-mon[35869]: pgmap v2759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:28 smithi097 ceph-mon[32231]: pgmap v2759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:30 smithi115 ceph-mon[35364]: pgmap v2760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:30 smithi181 ceph-mon[35869]: pgmap v2760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:30 smithi097 ceph-mon[32231]: pgmap v2760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:31.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3847.700937 secs 2024-07-26T09:03:31.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:31 smithi115 ceph-mon[35364]: pgmap v2761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:31.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3847.700937 secs 2024-07-26T09:03:31.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:31 smithi181 ceph-mon[35869]: pgmap v2761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:31.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3847.700937 secs 2024-07-26T09:03:31.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:31 smithi097 ceph-mon[32231]: pgmap v2761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:34.118 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:34.116+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:03:34.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:34.117+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:03:34.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:34.117+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:03:34.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:34.117+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:03:34.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:34.117+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:33 smithi115 ceph-mon[35364]: pgmap v2762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:34.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:34.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:33 smithi181 ceph-mon[35869]: pgmap v2762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:34.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:33 smithi097 ceph-mon[32231]: pgmap v2762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:03:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:03:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:03:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:35 smithi115 ceph-mon[35364]: pgmap v2763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:03:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3852.701108 secs 2024-07-26T09:03:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:35 smithi181 ceph-mon[35869]: pgmap v2763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:03:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3852.701108 secs 2024-07-26T09:03:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:35 smithi097 ceph-mon[32231]: pgmap v2763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:03:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3852.701108 secs 2024-07-26T09:03:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:37 smithi115 ceph-mon[35364]: pgmap v2764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:37 smithi181 ceph-mon[35869]: pgmap v2764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:37 smithi097 ceph-mon[32231]: pgmap v2764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:39 smithi115 ceph-mon[35364]: pgmap v2765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:03:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:39 smithi181 ceph-mon[35869]: pgmap v2765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:03:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:39 smithi097 ceph-mon[32231]: pgmap v2765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:03:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3857.701292 secs 2024-07-26T09:03:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3857.701292 secs 2024-07-26T09:03:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3857.701292 secs 2024-07-26T09:03:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:41 smithi115 ceph-mon[35364]: pgmap v2766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:03:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:41 smithi181 ceph-mon[35869]: pgmap v2766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:03:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:41 smithi097 ceph-mon[32231]: pgmap v2766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:03:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:43 smithi115 ceph-mon[35364]: pgmap v2767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:43 smithi181 ceph-mon[35869]: pgmap v2767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:43 smithi097 ceph-mon[32231]: pgmap v2767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:45 smithi115 ceph-mon[35364]: pgmap v2768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:03:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3862.701390 secs 2024-07-26T09:03:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:45 smithi181 ceph-mon[35869]: pgmap v2768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:03:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3862.701390 secs 2024-07-26T09:03:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:45 smithi097 ceph-mon[32231]: pgmap v2768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:03:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3862.701390 secs 2024-07-26T09:03:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:47 smithi115 ceph-mon[35364]: pgmap v2769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:03:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:47 smithi181 ceph-mon[35869]: pgmap v2769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:03:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:47 smithi097 ceph-mon[32231]: pgmap v2769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:03:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:49.127+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:03:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:49.128+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:03:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:49.128+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:03:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:49.128+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:03:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:03:49.128+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:49 smithi115 ceph-mon[35364]: pgmap v2770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:49 smithi181 ceph-mon[35869]: pgmap v2770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:49 smithi097 ceph-mon[32231]: pgmap v2770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3867.701525 secs 2024-07-26T09:03:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3867.701525 secs 2024-07-26T09:03:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3867.701525 secs 2024-07-26T09:03:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:51 smithi115 ceph-mon[35364]: pgmap v2771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:51 smithi181 ceph-mon[35869]: pgmap v2771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:51 smithi097 ceph-mon[32231]: pgmap v2771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:53 smithi115 ceph-mon[35364]: pgmap v2772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:53 smithi181 ceph-mon[35869]: pgmap v2772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:53 smithi097 ceph-mon[32231]: pgmap v2772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:55 smithi115 ceph-mon[35364]: pgmap v2773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:03:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3872.701691 secs 2024-07-26T09:03:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:55 smithi181 ceph-mon[35869]: pgmap v2773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:03:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3872.701691 secs 2024-07-26T09:03:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:55 smithi097 ceph-mon[32231]: pgmap v2773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:03:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3872.701691 secs 2024-07-26T09:03:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:57 smithi115 ceph-mon[35364]: pgmap v2774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:03:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:57 smithi181 ceph-mon[35869]: pgmap v2774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:03:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:57 smithi097 ceph-mon[32231]: pgmap v2774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:03:59 smithi115 ceph-mon[35364]: pgmap v2775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:03:59 smithi181 ceph-mon[35869]: pgmap v2775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:03:59 smithi097 ceph-mon[32231]: pgmap v2775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3877.701836 secs 2024-07-26T09:04:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3877.701836 secs 2024-07-26T09:04:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3877.701836 secs 2024-07-26T09:04:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:01 smithi115 ceph-mon[35364]: pgmap v2776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:01 smithi181 ceph-mon[35869]: pgmap v2776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:01 smithi097 ceph-mon[32231]: pgmap v2776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:04.130+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:04:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:04.131+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:04:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:04.131+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:04:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:04.131+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:04:04.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:04.131+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:03 smithi115 ceph-mon[35364]: pgmap v2777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:03 smithi181 ceph-mon[35869]: pgmap v2777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:03 smithi097 ceph-mon[32231]: pgmap v2777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:05 smithi115 ceph-mon[35364]: pgmap v2778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3882.701992 secs 2024-07-26T09:04:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:05 smithi181 ceph-mon[35869]: pgmap v2778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3882.701992 secs 2024-07-26T09:04:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:05 smithi097 ceph-mon[32231]: pgmap v2778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3882.701992 secs 2024-07-26T09:04:07.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:07.557+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T09:04:07.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:07.557+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:04:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:07.557+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T09:04:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:07.557+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:04:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:07.557+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:04:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:07.558+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:04:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:07.558+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:04:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:07.559+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:04:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:07 smithi115 ceph-mon[35364]: pgmap v2779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:07 smithi181 ceph-mon[35869]: pgmap v2779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:07 smithi097 ceph-mon[32231]: pgmap v2779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:09 smithi115 ceph-mon[35364]: pgmap v2780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:09 smithi181 ceph-mon[35869]: pgmap v2780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:09 smithi097 ceph-mon[32231]: pgmap v2780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3887.702123 secs 2024-07-26T09:04:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3887.702123 secs 2024-07-26T09:04:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3887.702123 secs 2024-07-26T09:04:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:11 smithi115 ceph-mon[35364]: pgmap v2781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:11 smithi181 ceph-mon[35869]: pgmap v2781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:11 smithi097 ceph-mon[32231]: pgmap v2781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:13 smithi115 ceph-mon[35364]: pgmap v2782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:13 smithi181 ceph-mon[35869]: pgmap v2782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:13 smithi097 ceph-mon[32231]: pgmap v2782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:15 smithi115 ceph-mon[35364]: pgmap v2783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3892.702268 secs 2024-07-26T09:04:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:15 smithi181 ceph-mon[35869]: pgmap v2783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3892.702268 secs 2024-07-26T09:04:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:15 smithi097 ceph-mon[32231]: pgmap v2783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3892.702268 secs 2024-07-26T09:04:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:17 smithi181 ceph-mon[35869]: pgmap v2784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:17 smithi097 ceph-mon[32231]: pgmap v2784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:17 smithi115 ceph-mon[35364]: pgmap v2784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:19.138+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:04:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:19.139+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:04:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:19.139+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:04:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:19.139+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:04:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:19.139+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:19 smithi181 ceph-mon[35869]: pgmap v2785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:19 smithi097 ceph-mon[32231]: pgmap v2785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:19 smithi115 ceph-mon[35364]: pgmap v2785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3897.702414 secs 2024-07-26T09:04:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3897.702414 secs 2024-07-26T09:04:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3897.702414 secs 2024-07-26T09:04:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:21 smithi097 ceph-mon[32231]: pgmap v2786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:21 smithi115 ceph-mon[35364]: pgmap v2786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:21 smithi181 ceph-mon[35869]: pgmap v2786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:24 smithi115 ceph-mon[35364]: pgmap v2787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:24 smithi181 ceph-mon[35869]: pgmap v2787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:24 smithi097 ceph-mon[32231]: pgmap v2787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:26 smithi115 ceph-mon[35364]: pgmap v2788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3902.702555 secs 2024-07-26T09:04:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:26 smithi181 ceph-mon[35869]: pgmap v2788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3902.702555 secs 2024-07-26T09:04:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:26 smithi097 ceph-mon[32231]: pgmap v2788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3902.702555 secs 2024-07-26T09:04:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:28 smithi115 ceph-mon[35364]: pgmap v2789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:28 smithi181 ceph-mon[35869]: pgmap v2789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:28 smithi097 ceph-mon[32231]: pgmap v2789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:30 smithi115 ceph-mon[35364]: pgmap v2790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:30 smithi181 ceph-mon[35869]: pgmap v2790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:30 smithi097 ceph-mon[32231]: pgmap v2790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:31.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3907.702693 secs 2024-07-26T09:04:31.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3907.702693 secs 2024-07-26T09:04:31.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3907.702693 secs 2024-07-26T09:04:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:32 smithi115 ceph-mon[35364]: pgmap v2791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:32 smithi181 ceph-mon[35869]: pgmap v2791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:32 smithi097 ceph-mon[32231]: pgmap v2791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:34.142+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:04:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:34.143+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:04:34.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:34.143+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:04:34.146 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:34.143+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:04:34.146 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:34.144+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:34 smithi115 ceph-mon[35364]: pgmap v2792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:34 smithi181 ceph-mon[35869]: pgmap v2792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:34 smithi097 ceph-mon[32231]: pgmap v2792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:36 smithi115 ceph-mon[35364]: pgmap v2793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3912.702804 secs 2024-07-26T09:04:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:04:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:36 smithi181 ceph-mon[35869]: pgmap v2793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3912.702804 secs 2024-07-26T09:04:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:04:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:36 smithi097 ceph-mon[32231]: pgmap v2793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:04:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3912.702804 secs 2024-07-26T09:04:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:04:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:37 smithi115 ceph-mon[35364]: pgmap v2794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:04:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:37 smithi115 ceph-mon[35364]: pgmap v2795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:37 smithi181 ceph-mon[35869]: pgmap v2794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:04:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:37 smithi181 ceph-mon[35869]: pgmap v2795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:37 smithi097 ceph-mon[32231]: pgmap v2794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:04:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:37 smithi097 ceph-mon[32231]: pgmap v2795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:38 smithi115 ceph-mon[35364]: pgmap v2796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:38 smithi181 ceph-mon[35869]: pgmap v2796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:38 smithi097 ceph-mon[32231]: pgmap v2796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:40 smithi115 ceph-mon[35364]: pgmap v2797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3917.702943 secs 2024-07-26T09:04:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:40 smithi181 ceph-mon[35869]: pgmap v2797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3917.702943 secs 2024-07-26T09:04:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:40 smithi097 ceph-mon[32231]: pgmap v2797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3917.702943 secs 2024-07-26T09:04:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:42 smithi115 ceph-mon[35364]: pgmap v2798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:42 smithi181 ceph-mon[35869]: pgmap v2798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:42 smithi097 ceph-mon[32231]: pgmap v2798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:44 smithi115 ceph-mon[35364]: pgmap v2799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:44 smithi181 ceph-mon[35869]: pgmap v2799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:44 smithi097 ceph-mon[32231]: pgmap v2799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:04:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3922.703083 secs 2024-07-26T09:04:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3922.703083 secs 2024-07-26T09:04:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3922.703083 secs 2024-07-26T09:04:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:46 smithi115 ceph-mon[35364]: pgmap v2800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:46 smithi181 ceph-mon[35869]: pgmap v2800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:46 smithi097 ceph-mon[32231]: pgmap v2800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:48 smithi115 ceph-mon[35364]: pgmap v2801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:48 smithi181 ceph-mon[35869]: pgmap v2801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:48 smithi097 ceph-mon[32231]: pgmap v2801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:49.154+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:04:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:49.155+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:04:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:49.155+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:04:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:49.155+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:04:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:04:49.156+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:50 smithi115 ceph-mon[35364]: pgmap v2802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3927.703223 secs 2024-07-26T09:04:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:50 smithi181 ceph-mon[35869]: pgmap v2802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3927.703223 secs 2024-07-26T09:04:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:50 smithi097 ceph-mon[32231]: pgmap v2802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3927.703223 secs 2024-07-26T09:04:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:52 smithi115 ceph-mon[35364]: pgmap v2803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:52 smithi181 ceph-mon[35869]: pgmap v2803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:52 smithi097 ceph-mon[32231]: pgmap v2803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:54 smithi115 ceph-mon[35364]: pgmap v2804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:54 smithi181 ceph-mon[35869]: pgmap v2804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:54 smithi097 ceph-mon[32231]: pgmap v2804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3932.703368 secs 2024-07-26T09:04:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3932.703368 secs 2024-07-26T09:04:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3932.703368 secs 2024-07-26T09:04:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:56 smithi115 ceph-mon[35364]: pgmap v2805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:56 smithi181 ceph-mon[35869]: pgmap v2805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:56 smithi097 ceph-mon[32231]: pgmap v2805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:04:58 smithi115 ceph-mon[35364]: pgmap v2806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:04:58 smithi181 ceph-mon[35869]: pgmap v2806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:04:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:04:58 smithi097 ceph-mon[32231]: pgmap v2806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:00 smithi115 ceph-mon[35364]: pgmap v2807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3937.703461 secs 2024-07-26T09:05:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:00 smithi181 ceph-mon[35869]: pgmap v2807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3937.703461 secs 2024-07-26T09:05:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:00 smithi097 ceph-mon[32231]: pgmap v2807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3937.703461 secs 2024-07-26T09:05:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:02 smithi181 ceph-mon[35869]: pgmap v2808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:05:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:02 smithi097 ceph-mon[32231]: pgmap v2808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:05:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:02 smithi115 ceph-mon[35364]: pgmap v2808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:05:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:04.161+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:05:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:04.162+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:05:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:04.162+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:05:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:04.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:05:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:04.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:04 smithi097 ceph-mon[32231]: pgmap v2809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:04 smithi115 ceph-mon[35364]: pgmap v2809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:04 smithi181 ceph-mon[35869]: pgmap v2809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3942.703602 secs 2024-07-26T09:05:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3942.703602 secs 2024-07-26T09:05:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3942.703602 secs 2024-07-26T09:05:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:06 smithi097 ceph-mon[32231]: pgmap v2810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:06 smithi115 ceph-mon[35364]: pgmap v2810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:06 smithi181 ceph-mon[35869]: pgmap v2810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:08 smithi115 ceph-mon[35364]: pgmap v2811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:08 smithi181 ceph-mon[35869]: pgmap v2811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:08 smithi097 ceph-mon[32231]: pgmap v2811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:10 smithi115 ceph-mon[35364]: pgmap v2812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3947.703758 secs 2024-07-26T09:05:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:10 smithi181 ceph-mon[35869]: pgmap v2812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3947.703758 secs 2024-07-26T09:05:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:10 smithi097 ceph-mon[32231]: pgmap v2812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3947.703758 secs 2024-07-26T09:05:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:12 smithi115 ceph-mon[35364]: pgmap v2813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:12 smithi181 ceph-mon[35869]: pgmap v2813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:12 smithi097 ceph-mon[32231]: pgmap v2813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:14 smithi115 ceph-mon[35364]: pgmap v2814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:14 smithi181 ceph-mon[35869]: pgmap v2814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:14 smithi097 ceph-mon[32231]: pgmap v2814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3952.703843 secs 2024-07-26T09:05:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3952.703843 secs 2024-07-26T09:05:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3952.703843 secs 2024-07-26T09:05:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:16 smithi115 ceph-mon[35364]: pgmap v2815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:16 smithi181 ceph-mon[35869]: pgmap v2815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:16 smithi097 ceph-mon[32231]: pgmap v2815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:19.172+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:05:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:19.173+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:05:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:19.173+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:05:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:19.173+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:05:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:19.173+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:18 smithi115 ceph-mon[35364]: pgmap v2816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:18 smithi181 ceph-mon[35869]: pgmap v2816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:18 smithi097 ceph-mon[32231]: pgmap v2816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:20 smithi115 ceph-mon[35364]: pgmap v2817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3957.703921 secs 2024-07-26T09:05:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:20 smithi181 ceph-mon[35869]: pgmap v2817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3957.703921 secs 2024-07-26T09:05:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:20 smithi097 ceph-mon[32231]: pgmap v2817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3957.703921 secs 2024-07-26T09:05:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:22 smithi115 ceph-mon[35364]: pgmap v2818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:22 smithi181 ceph-mon[35869]: pgmap v2818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:22 smithi097 ceph-mon[32231]: pgmap v2818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:24 smithi115 ceph-mon[35364]: pgmap v2819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:24 smithi181 ceph-mon[35869]: pgmap v2819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:24 smithi097 ceph-mon[32231]: pgmap v2819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3962.704060 secs 2024-07-26T09:05:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3962.704060 secs 2024-07-26T09:05:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3962.704060 secs 2024-07-26T09:05:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:26 smithi115 ceph-mon[35364]: pgmap v2820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:26 smithi181 ceph-mon[35869]: pgmap v2820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:26 smithi097 ceph-mon[32231]: pgmap v2820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:28 smithi115 ceph-mon[35364]: pgmap v2821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:28 smithi181 ceph-mon[35869]: pgmap v2821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:28 smithi097 ceph-mon[32231]: pgmap v2821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:05:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:30 smithi115 ceph-mon[35364]: pgmap v2822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3967.704271 secs 2024-07-26T09:05:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:30 smithi181 ceph-mon[35869]: pgmap v2822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3967.704271 secs 2024-07-26T09:05:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:30 smithi097 ceph-mon[32231]: pgmap v2822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3967.704271 secs 2024-07-26T09:05:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:32 smithi115 ceph-mon[35364]: pgmap v2823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:32 smithi181 ceph-mon[35869]: pgmap v2823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:32 smithi097 ceph-mon[32231]: pgmap v2823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:34.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:34.177+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:05:34.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:34.178+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:05:34.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:34.178+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:05:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:34.178+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:05:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:34.178+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:34 smithi115 ceph-mon[35364]: pgmap v2824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:34 smithi181 ceph-mon[35869]: pgmap v2824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:34 smithi097 ceph-mon[32231]: pgmap v2824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3972.704408 secs 2024-07-26T09:05:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3972.704408 secs 2024-07-26T09:05:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3972.704408 secs 2024-07-26T09:05:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:36 smithi115 ceph-mon[35364]: pgmap v2825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:36 smithi181 ceph-mon[35869]: pgmap v2825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:36 smithi097 ceph-mon[32231]: pgmap v2825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:05:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:05:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:37 smithi115 ceph-mon[35364]: pgmap v2826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:05:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:37 smithi115 ceph-mon[35364]: pgmap v2827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:05:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:05:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:05:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:05:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:37 smithi181 ceph-mon[35869]: pgmap v2826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:05:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:37 smithi181 ceph-mon[35869]: pgmap v2827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:05:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:05:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:05:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:05:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:37 smithi097 ceph-mon[32231]: pgmap v2826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:05:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:37 smithi097 ceph-mon[32231]: pgmap v2827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:05:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:05:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:40 smithi115 ceph-mon[35364]: pgmap v2828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:05:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:40 smithi181 ceph-mon[35869]: pgmap v2828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:05:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:40 smithi097 ceph-mon[32231]: pgmap v2828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:05:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3977.704549 secs 2024-07-26T09:05:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3977.704549 secs 2024-07-26T09:05:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3977.704549 secs 2024-07-26T09:05:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:42 smithi115 ceph-mon[35364]: pgmap v2829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:42 smithi181 ceph-mon[35869]: pgmap v2829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:42 smithi097 ceph-mon[32231]: pgmap v2829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:43.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:43 smithi115 ceph-mon[35364]: pgmap v2830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:05:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:43 smithi181 ceph-mon[35869]: pgmap v2830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:05:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:43 smithi097 ceph-mon[32231]: pgmap v2830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:05:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3982.704693 secs 2024-07-26T09:05:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:46 smithi115 ceph-mon[35364]: pgmap v2831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:05:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3982.704693 secs 2024-07-26T09:05:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:46 smithi181 ceph-mon[35869]: pgmap v2831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:05:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3982.704693 secs 2024-07-26T09:05:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:46 smithi097 ceph-mon[32231]: pgmap v2831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:05:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:48 smithi115 ceph-mon[35364]: pgmap v2832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:48 smithi181 ceph-mon[35869]: pgmap v2832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:48 smithi097 ceph-mon[32231]: pgmap v2832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:49.188+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:05:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:49.189+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:05:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:49.189+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:05:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:49.189+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:05:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:05:49.189+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:50 smithi115 ceph-mon[35364]: pgmap v2833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:05:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:50 smithi181 ceph-mon[35869]: pgmap v2833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:05:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:50 smithi097 ceph-mon[32231]: pgmap v2833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:05:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3987.704849 secs 2024-07-26T09:05:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:51 smithi115 ceph-mon[35364]: pgmap v2834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:05:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3987.704849 secs 2024-07-26T09:05:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:51 smithi181 ceph-mon[35869]: pgmap v2834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:05:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3987.704849 secs 2024-07-26T09:05:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:51 smithi097 ceph-mon[32231]: pgmap v2834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:05:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:54 smithi115 ceph-mon[35364]: pgmap v2835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:54 smithi181 ceph-mon[35869]: pgmap v2835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:54 smithi097 ceph-mon[32231]: pgmap v2835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:05:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3992.704976 secs 2024-07-26T09:05:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:56 smithi115 ceph-mon[35364]: pgmap v2836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3992.704976 secs 2024-07-26T09:05:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:56 smithi181 ceph-mon[35869]: pgmap v2836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3992.704976 secs 2024-07-26T09:05:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:56 smithi097 ceph-mon[32231]: pgmap v2836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:05:57 smithi115 ceph-mon[35364]: pgmap v2837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:05:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:05:57 smithi181 ceph-mon[35869]: pgmap v2837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:05:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:05:57 smithi097 ceph-mon[32231]: pgmap v2837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:06:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:00 smithi115 ceph-mon[35364]: pgmap v2838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:06:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:00 smithi181 ceph-mon[35869]: pgmap v2838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:06:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:00 smithi097 ceph-mon[32231]: pgmap v2838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:06:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 3997.705112 secs 2024-07-26T09:06:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 3997.705112 secs 2024-07-26T09:06:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 3997.705112 secs 2024-07-26T09:06:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:02 smithi115 ceph-mon[35364]: pgmap v2839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:02 smithi181 ceph-mon[35869]: pgmap v2839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:02 smithi097 ceph-mon[32231]: pgmap v2839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:03.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:03 smithi115 ceph-mon[35364]: pgmap v2840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:03.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:03 smithi181 ceph-mon[35869]: pgmap v2840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:03.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:03 smithi097 ceph-mon[32231]: pgmap v2840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:04.195+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:06:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:04.195+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:06:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:04.195+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:06:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:04.195+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:06:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:04.195+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4002.705291 secs 2024-07-26T09:06:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:06 smithi115 ceph-mon[35364]: pgmap v2841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4002.705291 secs 2024-07-26T09:06:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:06 smithi181 ceph-mon[35869]: pgmap v2841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4002.705291 secs 2024-07-26T09:06:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:06 smithi097 ceph-mon[32231]: pgmap v2841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:08 smithi115 ceph-mon[35364]: pgmap v2842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:08 smithi181 ceph-mon[35869]: pgmap v2842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:08 smithi097 ceph-mon[32231]: pgmap v2842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:09 smithi115 ceph-mon[35364]: pgmap v2843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:09 smithi181 ceph-mon[35869]: pgmap v2843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:09 smithi097 ceph-mon[32231]: pgmap v2843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4007.705436 secs 2024-07-26T09:06:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4007.705436 secs 2024-07-26T09:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4007.705436 secs 2024-07-26T09:06:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:11 smithi115 ceph-mon[35364]: pgmap v2844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:11 smithi181 ceph-mon[35869]: pgmap v2844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:11 smithi097 ceph-mon[32231]: pgmap v2844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:14 smithi115 ceph-mon[35364]: pgmap v2845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:14 smithi181 ceph-mon[35869]: pgmap v2845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:14 smithi097 ceph-mon[32231]: pgmap v2845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:15 smithi115 ceph-mon[35364]: pgmap v2846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:15 smithi181 ceph-mon[35869]: pgmap v2846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:15 smithi097 ceph-mon[32231]: pgmap v2846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4012.705577 secs 2024-07-26T09:06:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4012.705577 secs 2024-07-26T09:06:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4012.705577 secs 2024-07-26T09:06:17.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:17 smithi115 ceph-mon[35364]: pgmap v2847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:17 smithi181 ceph-mon[35869]: pgmap v2847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:17 smithi097 ceph-mon[32231]: pgmap v2847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:19.204+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:06:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:19.205+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:06:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:19.205+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:06:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:19.205+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:06:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:19.205+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:20 smithi115 ceph-mon[35364]: pgmap v2848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:20 smithi181 ceph-mon[35869]: pgmap v2848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:20 smithi097 ceph-mon[32231]: pgmap v2848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4017.705716 secs 2024-07-26T09:06:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4017.705716 secs 2024-07-26T09:06:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4017.705716 secs 2024-07-26T09:06:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:22 smithi115 ceph-mon[35364]: pgmap v2849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:22 smithi181 ceph-mon[35869]: pgmap v2849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:22 smithi097 ceph-mon[32231]: pgmap v2849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:23 smithi115 ceph-mon[35364]: pgmap v2850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:23 smithi181 ceph-mon[35869]: pgmap v2850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:23 smithi097 ceph-mon[32231]: pgmap v2850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4022.705800 secs 2024-07-26T09:06:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:26 smithi115 ceph-mon[35364]: pgmap v2851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4022.705800 secs 2024-07-26T09:06:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:26 smithi181 ceph-mon[35869]: pgmap v2851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4022.705800 secs 2024-07-26T09:06:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:26 smithi097 ceph-mon[32231]: pgmap v2851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:27 smithi115 ceph-mon[35364]: pgmap v2852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:27 smithi181 ceph-mon[35869]: pgmap v2852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:27 smithi097 ceph-mon[32231]: pgmap v2852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:30 smithi115 ceph-mon[35364]: pgmap v2853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:30 smithi181 ceph-mon[35869]: pgmap v2853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:30 smithi097 ceph-mon[32231]: pgmap v2853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:06:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4027.705956 secs 2024-07-26T09:06:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:31 smithi115 ceph-mon[35364]: pgmap v2854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4027.705956 secs 2024-07-26T09:06:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:31 smithi181 ceph-mon[35869]: pgmap v2854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4027.705956 secs 2024-07-26T09:06:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:31 smithi097 ceph-mon[32231]: pgmap v2854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:33 smithi115 ceph-mon[35364]: pgmap v2855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:33 smithi181 ceph-mon[35869]: pgmap v2855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:33 smithi097 ceph-mon[32231]: pgmap v2855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:34.212 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:34.210+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:06:34.213 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:34.211+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:06:34.213 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:34.211+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:06:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:34.211+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:06:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:34.212+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:35 smithi115 ceph-mon[35364]: pgmap v2856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:35 smithi181 ceph-mon[35869]: pgmap v2856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:35 smithi097 ceph-mon[32231]: pgmap v2856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4032.706123 secs 2024-07-26T09:06:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4032.706123 secs 2024-07-26T09:06:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4032.706123 secs 2024-07-26T09:06:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:37 smithi115 ceph-mon[35364]: pgmap v2857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:37 smithi181 ceph-mon[35869]: pgmap v2857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:37 smithi097 ceph-mon[32231]: pgmap v2857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:06:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:06:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:06:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:39 smithi115 ceph-mon[35364]: pgmap v2858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:06:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:39 smithi115 ceph-mon[35364]: pgmap v2859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:06:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:06:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:39 smithi181 ceph-mon[35869]: pgmap v2858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:06:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:39 smithi181 ceph-mon[35869]: pgmap v2859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:06:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:06:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:39 smithi097 ceph-mon[32231]: pgmap v2858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:06:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:39 smithi097 ceph-mon[32231]: pgmap v2859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:06:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:06:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4037.706272 secs 2024-07-26T09:06:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4037.706272 secs 2024-07-26T09:06:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4037.706272 secs 2024-07-26T09:06:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:41 smithi115 ceph-mon[35364]: pgmap v2860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:41 smithi181 ceph-mon[35869]: pgmap v2860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:41 smithi097 ceph-mon[32231]: pgmap v2860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:43 smithi115 ceph-mon[35364]: pgmap v2861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:43 smithi181 ceph-mon[35869]: pgmap v2861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:43 smithi097 ceph-mon[32231]: pgmap v2861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:45 smithi115 ceph-mon[35364]: pgmap v2862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4042.706406 secs 2024-07-26T09:06:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:45 smithi181 ceph-mon[35869]: pgmap v2862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:45.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4042.706406 secs 2024-07-26T09:06:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:45 smithi097 ceph-mon[32231]: pgmap v2862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4042.706406 secs 2024-07-26T09:06:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:47 smithi115 ceph-mon[35364]: pgmap v2863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:47 smithi181 ceph-mon[35869]: pgmap v2863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:47 smithi097 ceph-mon[32231]: pgmap v2863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:49.221+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:06:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:49.223+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:06:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:49.223+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:06:49.226 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:49.223+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:06:49.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:06:49.223+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:49 smithi115 ceph-mon[35364]: pgmap v2864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:06:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:49 smithi181 ceph-mon[35869]: pgmap v2864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:06:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:49 smithi097 ceph-mon[32231]: pgmap v2864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:06:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4047.706556 secs 2024-07-26T09:06:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4047.706556 secs 2024-07-26T09:06:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4047.706556 secs 2024-07-26T09:06:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:51 smithi115 ceph-mon[35364]: pgmap v2865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:51 smithi181 ceph-mon[35869]: pgmap v2865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:51 smithi097 ceph-mon[32231]: pgmap v2865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:53 smithi115 ceph-mon[35364]: pgmap v2866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:53 smithi181 ceph-mon[35869]: pgmap v2866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:53 smithi097 ceph-mon[32231]: pgmap v2866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:06:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:55 smithi115 ceph-mon[35364]: pgmap v2867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4052.706698 secs 2024-07-26T09:06:55.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:55 smithi181 ceph-mon[35869]: pgmap v2867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:55.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4052.706698 secs 2024-07-26T09:06:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:55 smithi097 ceph-mon[32231]: pgmap v2867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4052.706698 secs 2024-07-26T09:06:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:57 smithi115 ceph-mon[35364]: pgmap v2868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:57 smithi181 ceph-mon[35869]: pgmap v2868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:57 smithi097 ceph-mon[32231]: pgmap v2868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:06:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:06:59 smithi115 ceph-mon[35364]: pgmap v2869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:06:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:06:59 smithi181 ceph-mon[35869]: pgmap v2869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:07:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:06:59 smithi097 ceph-mon[32231]: pgmap v2869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:07:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4057.706844 secs 2024-07-26T09:07:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4057.706844 secs 2024-07-26T09:07:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4057.706844 secs 2024-07-26T09:07:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:01 smithi115 ceph-mon[35364]: pgmap v2870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:07:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:01 smithi181 ceph-mon[35869]: pgmap v2870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:07:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:01 smithi097 ceph-mon[32231]: pgmap v2870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:07:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:03 smithi115 ceph-mon[35364]: pgmap v2871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:07:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:03 smithi181 ceph-mon[35869]: pgmap v2871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:07:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:03 smithi097 ceph-mon[32231]: pgmap v2871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:07:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:04.227+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:07:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:04.229+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:07:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:04.229+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:07:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:04.229+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:07:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:04.229+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:05 smithi115 ceph-mon[35364]: pgmap v2872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4062.706999 secs 2024-07-26T09:07:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:05 smithi181 ceph-mon[35869]: pgmap v2872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4062.706999 secs 2024-07-26T09:07:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:05 smithi097 ceph-mon[32231]: pgmap v2872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4062.706999 secs 2024-07-26T09:07:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:07.588+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:07:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:07.588+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:07:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:07.588+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:07:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:07.588+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:07:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:07 smithi115 ceph-mon[35364]: pgmap v2873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:07 smithi181 ceph-mon[35869]: pgmap v2873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:07 smithi097 ceph-mon[32231]: pgmap v2873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:09 smithi115 ceph-mon[35364]: pgmap v2874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:09 smithi181 ceph-mon[35869]: pgmap v2874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:09 smithi097 ceph-mon[32231]: pgmap v2874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4067.707134 secs 2024-07-26T09:07:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4067.707134 secs 2024-07-26T09:07:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4067.707134 secs 2024-07-26T09:07:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:11 smithi115 ceph-mon[35364]: pgmap v2875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:11 smithi181 ceph-mon[35869]: pgmap v2875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:11 smithi097 ceph-mon[32231]: pgmap v2875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:13 smithi115 ceph-mon[35364]: pgmap v2876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:13 smithi181 ceph-mon[35869]: pgmap v2876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:13 smithi097 ceph-mon[32231]: pgmap v2876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:15 smithi115 ceph-mon[35364]: pgmap v2877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4072.707279 secs 2024-07-26T09:07:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:15 smithi181 ceph-mon[35869]: pgmap v2877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4072.707279 secs 2024-07-26T09:07:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:15 smithi097 ceph-mon[32231]: pgmap v2877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4072.707279 secs 2024-07-26T09:07:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:17 smithi115 ceph-mon[35364]: pgmap v2878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:17 smithi181 ceph-mon[35869]: pgmap v2878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:17 smithi097 ceph-mon[32231]: pgmap v2878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:19.238+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:07:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:19.239+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:07:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:19.239+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:07:19.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:19.239+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:07:19.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:19.239+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:19 smithi115 ceph-mon[35364]: pgmap v2879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:19 smithi181 ceph-mon[35869]: pgmap v2879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:19 smithi097 ceph-mon[32231]: pgmap v2879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4077.707422 secs 2024-07-26T09:07:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4077.707422 secs 2024-07-26T09:07:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4077.707422 secs 2024-07-26T09:07:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:21 smithi115 ceph-mon[35364]: pgmap v2880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:21 smithi181 ceph-mon[35869]: pgmap v2880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:21 smithi097 ceph-mon[32231]: pgmap v2880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:23 smithi115 ceph-mon[35364]: pgmap v2881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:23 smithi181 ceph-mon[35869]: pgmap v2881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:23 smithi097 ceph-mon[32231]: pgmap v2881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:25 smithi115 ceph-mon[35364]: pgmap v2882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4082.707569 secs 2024-07-26T09:07:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:25 smithi181 ceph-mon[35869]: pgmap v2882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4082.707569 secs 2024-07-26T09:07:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:25 smithi097 ceph-mon[32231]: pgmap v2882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4082.707569 secs 2024-07-26T09:07:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:27 smithi115 ceph-mon[35364]: pgmap v2883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:27 smithi181 ceph-mon[35869]: pgmap v2883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:27 smithi097 ceph-mon[32231]: pgmap v2883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:29 smithi115 ceph-mon[35364]: pgmap v2884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:29 smithi181 ceph-mon[35869]: pgmap v2884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:29 smithi097 ceph-mon[32231]: pgmap v2884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4087.707698 secs 2024-07-26T09:07:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4087.707698 secs 2024-07-26T09:07:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4087.707698 secs 2024-07-26T09:07:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:31 smithi097 ceph-mon[32231]: pgmap v2885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:31 smithi115 ceph-mon[35364]: pgmap v2885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:31 smithi181 ceph-mon[35869]: pgmap v2885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:33 smithi097 ceph-mon[32231]: pgmap v2886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:34.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:33 smithi115 ceph-mon[35364]: pgmap v2886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:34.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:33 smithi181 ceph-mon[35869]: pgmap v2886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:34.244+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:07:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:34.245+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:07:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:34.245+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:07:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:34.245+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:07:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:34.245+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:35 smithi115 ceph-mon[35364]: pgmap v2887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4092.707855 secs 2024-07-26T09:07:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:35 smithi181 ceph-mon[35869]: pgmap v2887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4092.707855 secs 2024-07-26T09:07:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:35 smithi097 ceph-mon[32231]: pgmap v2887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4092.707855 secs 2024-07-26T09:07:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:37 smithi115 ceph-mon[35364]: pgmap v2888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:37 smithi181 ceph-mon[35869]: pgmap v2888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:37 smithi097 ceph-mon[32231]: pgmap v2888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:39 smithi115 ceph-mon[35364]: pgmap v2889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:39 smithi181 ceph-mon[35869]: pgmap v2889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:39 smithi097 ceph-mon[32231]: pgmap v2889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4097.707996 secs 2024-07-26T09:07:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4097.707996 secs 2024-07-26T09:07:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4097.707996 secs 2024-07-26T09:07:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:07:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:41 smithi115 ceph-mon[35364]: pgmap v2890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:41 smithi115 ceph-mon[35364]: pgmap v2891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:41 smithi181 ceph-mon[35869]: pgmap v2890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:41 smithi181 ceph-mon[35869]: pgmap v2891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:41 smithi097 ceph-mon[32231]: pgmap v2890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:41 smithi097 ceph-mon[32231]: pgmap v2891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:43 smithi115 ceph-mon[35364]: pgmap v2892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:44.265 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:43 smithi097 ceph-mon[32231]: pgmap v2892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:44.265 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:43 smithi181 ceph-mon[35869]: pgmap v2892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:45 smithi115 ceph-mon[35364]: pgmap v2893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4102.708130 secs 2024-07-26T09:07:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:45 smithi181 ceph-mon[35869]: pgmap v2893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4102.708130 secs 2024-07-26T09:07:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:45 smithi097 ceph-mon[32231]: pgmap v2893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4102.708130 secs 2024-07-26T09:07:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:47 smithi115 ceph-mon[35364]: pgmap v2894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:47 smithi181 ceph-mon[35869]: pgmap v2894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:47 smithi097 ceph-mon[32231]: pgmap v2894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:49.252+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:07:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:49.253+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:07:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:49.253+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:07:49.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:49.253+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:07:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:07:49.253+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:49 smithi115 ceph-mon[35364]: pgmap v2895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:49 smithi181 ceph-mon[35869]: pgmap v2895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:49 smithi097 ceph-mon[32231]: pgmap v2895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:07:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4107.708275 secs 2024-07-26T09:07:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4107.708275 secs 2024-07-26T09:07:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4107.708275 secs 2024-07-26T09:07:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:51 smithi115 ceph-mon[35364]: pgmap v2896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:07:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:51 smithi181 ceph-mon[35869]: pgmap v2896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:07:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:51 smithi097 ceph-mon[32231]: pgmap v2896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:07:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:53 smithi115 ceph-mon[35364]: pgmap v2897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:53 smithi181 ceph-mon[35869]: pgmap v2897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:53 smithi097 ceph-mon[32231]: pgmap v2897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:55 smithi115 ceph-mon[35364]: pgmap v2898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4112.708408 secs 2024-07-26T09:07:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:55 smithi181 ceph-mon[35869]: pgmap v2898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4112.708408 secs 2024-07-26T09:07:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:55 smithi097 ceph-mon[32231]: pgmap v2898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4112.708408 secs 2024-07-26T09:07:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:57 smithi115 ceph-mon[35364]: pgmap v2899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:57 smithi181 ceph-mon[35869]: pgmap v2899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:07:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:57 smithi097 ceph-mon[32231]: pgmap v2899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:07:59 smithi115 ceph-mon[35364]: pgmap v2900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:07:59 smithi181 ceph-mon[35869]: pgmap v2900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:07:59 smithi097 ceph-mon[32231]: pgmap v2900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4117.708545 secs 2024-07-26T09:08:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4117.708545 secs 2024-07-26T09:08:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4117.708545 secs 2024-07-26T09:08:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:01 smithi115 ceph-mon[35364]: pgmap v2901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:01 smithi181 ceph-mon[35869]: pgmap v2901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:01 smithi097 ceph-mon[32231]: pgmap v2901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:03 smithi115 ceph-mon[35364]: pgmap v2902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:03 smithi181 ceph-mon[35869]: pgmap v2902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:03 smithi097 ceph-mon[32231]: pgmap v2902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:04.258+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:08:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:04.259+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:08:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:04.259+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:08:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:04.259+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:08:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:04.260+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:05 smithi115 ceph-mon[35364]: pgmap v2903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4122.708707 secs 2024-07-26T09:08:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:05 smithi181 ceph-mon[35869]: pgmap v2903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4122.708707 secs 2024-07-26T09:08:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:05 smithi097 ceph-mon[32231]: pgmap v2903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4122.708707 secs 2024-07-26T09:08:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:07 smithi115 ceph-mon[35364]: pgmap v2904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:07 smithi181 ceph-mon[35869]: pgmap v2904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:07 smithi097 ceph-mon[32231]: pgmap v2904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:09 smithi115 ceph-mon[35364]: pgmap v2905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:10.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:09 smithi181 ceph-mon[35869]: pgmap v2905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:09 smithi097 ceph-mon[32231]: pgmap v2905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4127.708847 secs 2024-07-26T09:08:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4127.708847 secs 2024-07-26T09:08:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4127.708847 secs 2024-07-26T09:08:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:11 smithi115 ceph-mon[35364]: pgmap v2906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:11 smithi181 ceph-mon[35869]: pgmap v2906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:11 smithi097 ceph-mon[32231]: pgmap v2906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:13 smithi115 ceph-mon[35364]: pgmap v2907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:13 smithi181 ceph-mon[35869]: pgmap v2907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:13 smithi097 ceph-mon[32231]: pgmap v2907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:15 smithi115 ceph-mon[35364]: pgmap v2908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4132.708971 secs 2024-07-26T09:08:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:15 smithi181 ceph-mon[35869]: pgmap v2908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4132.708971 secs 2024-07-26T09:08:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:15 smithi097 ceph-mon[32231]: pgmap v2908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4132.708971 secs 2024-07-26T09:08:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:17 smithi115 ceph-mon[35364]: pgmap v2909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:17 smithi181 ceph-mon[35869]: pgmap v2909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:17 smithi097 ceph-mon[32231]: pgmap v2909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:19.263+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:08:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:19.264+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:08:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:19.264+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:08:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:19.264+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:08:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:19.264+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:19 smithi181 ceph-mon[35869]: pgmap v2910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:19 smithi097 ceph-mon[32231]: pgmap v2910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:19 smithi115 ceph-mon[35364]: pgmap v2910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4137.709122 secs 2024-07-26T09:08:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4137.709122 secs 2024-07-26T09:08:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4137.709122 secs 2024-07-26T09:08:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:21 smithi097 ceph-mon[32231]: pgmap v2911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:21 smithi115 ceph-mon[35364]: pgmap v2911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:21 smithi181 ceph-mon[35869]: pgmap v2911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:24 smithi115 ceph-mon[35364]: pgmap v2912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:24 smithi181 ceph-mon[35869]: pgmap v2912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:24 smithi097 ceph-mon[32231]: pgmap v2912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:26 smithi115 ceph-mon[35364]: pgmap v2913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4142.709295 secs 2024-07-26T09:08:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:26 smithi181 ceph-mon[35869]: pgmap v2913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4142.709295 secs 2024-07-26T09:08:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:26 smithi097 ceph-mon[32231]: pgmap v2913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4142.709295 secs 2024-07-26T09:08:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:28 smithi115 ceph-mon[35364]: pgmap v2914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:28 smithi181 ceph-mon[35869]: pgmap v2914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:28 smithi097 ceph-mon[32231]: pgmap v2914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:30 smithi115 ceph-mon[35364]: pgmap v2915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:30 smithi181 ceph-mon[35869]: pgmap v2915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:30 smithi097 ceph-mon[32231]: pgmap v2915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4147.709437 secs 2024-07-26T09:08:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4147.709437 secs 2024-07-26T09:08:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4147.709437 secs 2024-07-26T09:08:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:32 smithi115 ceph-mon[35364]: pgmap v2916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:32 smithi181 ceph-mon[35869]: pgmap v2916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:32 smithi097 ceph-mon[32231]: pgmap v2916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:34.269+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:08:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:34.270+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:08:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:34.270+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:08:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:34.270+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:08:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:34.270+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:34.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:34 smithi097 ceph-mon[32231]: pgmap v2917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:34.305 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:34 smithi115 ceph-mon[35364]: pgmap v2917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:34 smithi181 ceph-mon[35869]: pgmap v2917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:36.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:36 smithi115 ceph-mon[35364]: pgmap v2918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:36.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4152.709600 secs 2024-07-26T09:08:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:36 smithi181 ceph-mon[35869]: pgmap v2918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4152.709600 secs 2024-07-26T09:08:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:36 smithi097 ceph-mon[32231]: pgmap v2918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4152.709600 secs 2024-07-26T09:08:38.480 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:38 smithi115 ceph-mon[35364]: pgmap v2919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:38 smithi181 ceph-mon[35869]: pgmap v2919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:38 smithi097 ceph-mon[32231]: pgmap v2919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:40 smithi115 ceph-mon[35364]: pgmap v2920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:40 smithi181 ceph-mon[35869]: pgmap v2920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:40 smithi097 ceph-mon[32231]: pgmap v2920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4157.709759 secs 2024-07-26T09:08:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4157.709759 secs 2024-07-26T09:08:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4157.709759 secs 2024-07-26T09:08:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:42 smithi115 ceph-mon[35364]: pgmap v2921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:42 smithi181 ceph-mon[35869]: pgmap v2921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:42 smithi097 ceph-mon[32231]: pgmap v2921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:08:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:43.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:08:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:08:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:43.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:08:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:44 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:08:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:44 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:08:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:44 smithi115 ceph-mon[35364]: pgmap v2922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:44 smithi115 ceph-mon[35364]: pgmap v2923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:44 smithi115 ceph-mon[35364]: pgmap v2924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:08:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:44 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:08:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:44 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:08:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:44 smithi181 ceph-mon[35869]: pgmap v2922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:44 smithi181 ceph-mon[35869]: pgmap v2923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:44 smithi181 ceph-mon[35869]: pgmap v2924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:08:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:44 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:08:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:44 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:08:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:44 smithi097 ceph-mon[32231]: pgmap v2922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:44 smithi097 ceph-mon[32231]: pgmap v2923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:44 smithi097 ceph-mon[32231]: pgmap v2924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:08:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:46 smithi115 ceph-mon[35364]: pgmap v2925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:08:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4162.709919 secs 2024-07-26T09:08:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:46 smithi181 ceph-mon[35869]: pgmap v2925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:08:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4162.709919 secs 2024-07-26T09:08:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:46 smithi097 ceph-mon[32231]: pgmap v2925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:08:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4162.709919 secs 2024-07-26T09:08:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:48 smithi115 ceph-mon[35364]: pgmap v2926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:08:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:48 smithi181 ceph-mon[35869]: pgmap v2926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:08:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:48 smithi097 ceph-mon[32231]: pgmap v2926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:08:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:49.280+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:08:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:49.282+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:08:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:49.282+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:08:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:49.282+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:08:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:08:49.282+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:50 smithi115 ceph-mon[35364]: pgmap v2927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:08:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:50 smithi181 ceph-mon[35869]: pgmap v2927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:08:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:50 smithi097 ceph-mon[32231]: pgmap v2927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:08:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4167.710059 secs 2024-07-26T09:08:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4167.710059 secs 2024-07-26T09:08:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4167.710059 secs 2024-07-26T09:08:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:52 smithi115 ceph-mon[35364]: pgmap v2928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:08:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:52 smithi181 ceph-mon[35869]: pgmap v2928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:08:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:52 smithi097 ceph-mon[32231]: pgmap v2928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:08:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:53 smithi115 ceph-mon[35364]: pgmap v2929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:53 smithi181 ceph-mon[35869]: pgmap v2929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:53 smithi097 ceph-mon[32231]: pgmap v2929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:55 smithi115 ceph-mon[35364]: pgmap v2930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4172.710212 secs 2024-07-26T09:08:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:55 smithi181 ceph-mon[35869]: pgmap v2930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4172.710212 secs 2024-07-26T09:08:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:55 smithi097 ceph-mon[32231]: pgmap v2930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4172.710212 secs 2024-07-26T09:08:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:57 smithi115 ceph-mon[35364]: pgmap v2931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:57 smithi181 ceph-mon[35869]: pgmap v2931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:57 smithi097 ceph-mon[32231]: pgmap v2931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:08:59 smithi115 ceph-mon[35364]: pgmap v2932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:08:59 smithi181 ceph-mon[35869]: pgmap v2932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:08:59 smithi097 ceph-mon[32231]: pgmap v2932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4177.710358 secs 2024-07-26T09:09:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4177.710358 secs 2024-07-26T09:09:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4177.710358 secs 2024-07-26T09:09:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:01 smithi115 ceph-mon[35364]: pgmap v2933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:01 smithi181 ceph-mon[35869]: pgmap v2933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:01 smithi097 ceph-mon[32231]: pgmap v2933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:03 smithi115 ceph-mon[35364]: pgmap v2934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:03 smithi181 ceph-mon[35869]: pgmap v2934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:03 smithi097 ceph-mon[32231]: pgmap v2934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:04.285+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:09:04.289 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:04.287+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:09:04.289 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:04.287+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:09:04.289 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:04.287+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:09:04.289 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:04.287+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:05 smithi115 ceph-mon[35364]: pgmap v2935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4182.710498 secs 2024-07-26T09:09:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:05 smithi181 ceph-mon[35869]: pgmap v2935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:05.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4182.710498 secs 2024-07-26T09:09:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:05 smithi097 ceph-mon[32231]: pgmap v2935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4182.710498 secs 2024-07-26T09:09:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:07 smithi115 ceph-mon[35364]: pgmap v2936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:07 smithi181 ceph-mon[35869]: pgmap v2936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:07 smithi097 ceph-mon[32231]: pgmap v2936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:09 smithi115 ceph-mon[35364]: pgmap v2937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:09 smithi181 ceph-mon[35869]: pgmap v2937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:09 smithi097 ceph-mon[32231]: pgmap v2937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4187.710636 secs 2024-07-26T09:09:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4187.710636 secs 2024-07-26T09:09:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4187.710636 secs 2024-07-26T09:09:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:11 smithi115 ceph-mon[35364]: pgmap v2938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:11 smithi181 ceph-mon[35869]: pgmap v2938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:11 smithi097 ceph-mon[32231]: pgmap v2938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:13 smithi115 ceph-mon[35364]: pgmap v2939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:13 smithi181 ceph-mon[35869]: pgmap v2939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:13 smithi097 ceph-mon[32231]: pgmap v2939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:15 smithi115 ceph-mon[35364]: pgmap v2940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4192.710797 secs 2024-07-26T09:09:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:15 smithi181 ceph-mon[35869]: pgmap v2940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4192.710797 secs 2024-07-26T09:09:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:15 smithi097 ceph-mon[32231]: pgmap v2940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4192.710797 secs 2024-07-26T09:09:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:17 smithi115 ceph-mon[35364]: pgmap v2941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:17 smithi181 ceph-mon[35869]: pgmap v2941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:17 smithi097 ceph-mon[32231]: pgmap v2941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:19.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:19.297+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:09:19.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:19.298+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:09:19.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:19.298+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:09:19.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:19.298+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:09:19.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:19.298+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:19 smithi115 ceph-mon[35364]: pgmap v2942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:19 smithi181 ceph-mon[35869]: pgmap v2942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:19 smithi097 ceph-mon[32231]: pgmap v2942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4197.710928 secs 2024-07-26T09:09:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4197.710928 secs 2024-07-26T09:09:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4197.710928 secs 2024-07-26T09:09:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:22 smithi115 ceph-mon[35364]: pgmap v2943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:22 smithi181 ceph-mon[35869]: pgmap v2943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:22 smithi097 ceph-mon[32231]: pgmap v2943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:24 smithi115 ceph-mon[35364]: pgmap v2944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:24 smithi181 ceph-mon[35869]: pgmap v2944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:24 smithi097 ceph-mon[32231]: pgmap v2944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:26 smithi115 ceph-mon[35364]: pgmap v2945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4202.711083 secs 2024-07-26T09:09:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:26 smithi181 ceph-mon[35869]: pgmap v2945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4202.711083 secs 2024-07-26T09:09:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:26 smithi097 ceph-mon[32231]: pgmap v2945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4202.711083 secs 2024-07-26T09:09:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:28 smithi115 ceph-mon[35364]: pgmap v2946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:28 smithi181 ceph-mon[35869]: pgmap v2946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:28 smithi097 ceph-mon[32231]: pgmap v2946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:30 smithi115 ceph-mon[35364]: pgmap v2947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:30 smithi181 ceph-mon[35869]: pgmap v2947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:30 smithi097 ceph-mon[32231]: pgmap v2947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4207.711292 secs 2024-07-26T09:09:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4207.711292 secs 2024-07-26T09:09:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4207.711292 secs 2024-07-26T09:09:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:32 smithi115 ceph-mon[35364]: pgmap v2948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:32 smithi181 ceph-mon[35869]: pgmap v2948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:32 smithi097 ceph-mon[32231]: pgmap v2948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:34.304+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:09:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:34.305+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:09:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:34.305+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:09:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:34.305+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:09:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:34.305+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:34 smithi115 ceph-mon[35364]: pgmap v2949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:34 smithi181 ceph-mon[35869]: pgmap v2949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:34 smithi097 ceph-mon[32231]: pgmap v2949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:36.307 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:36 smithi097 ceph-mon[32231]: pgmap v2950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:36.307 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4212.711462 secs 2024-07-26T09:09:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:36 smithi115 ceph-mon[35364]: pgmap v2950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4212.711462 secs 2024-07-26T09:09:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:36 smithi181 ceph-mon[35869]: pgmap v2950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4212.711462 secs 2024-07-26T09:09:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:38 smithi115 ceph-mon[35364]: pgmap v2951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:38 smithi181 ceph-mon[35869]: pgmap v2951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:38 smithi097 ceph-mon[32231]: pgmap v2951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:40 smithi115 ceph-mon[35364]: pgmap v2952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:40 smithi181 ceph-mon[35869]: pgmap v2952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:40 smithi097 ceph-mon[32231]: pgmap v2952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4217.711637 secs 2024-07-26T09:09:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4217.711637 secs 2024-07-26T09:09:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4217.711637 secs 2024-07-26T09:09:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:42 smithi115 ceph-mon[35364]: pgmap v2953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:42.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:42 smithi181 ceph-mon[35869]: pgmap v2953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:42 smithi097 ceph-mon[32231]: pgmap v2953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:44 smithi115 ceph-mon[35364]: pgmap v2954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:44.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:44.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:09:44.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:44 smithi181 ceph-mon[35869]: pgmap v2954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:44.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:44.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:44.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:09:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:44 smithi097 ceph-mon[32231]: pgmap v2954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:09:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:44.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:09:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:45 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:09:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:45 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:09:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:45 smithi115 ceph-mon[35364]: pgmap v2955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:45 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:09:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:45 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:09:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:45 smithi181 ceph-mon[35869]: pgmap v2955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:45 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:09:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:45 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:09:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:45 smithi097 ceph-mon[32231]: pgmap v2955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4222.711734 secs 2024-07-26T09:09:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4222.711734 secs 2024-07-26T09:09:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4222.711734 secs 2024-07-26T09:09:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:47 smithi115 ceph-mon[35364]: pgmap v2956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:09:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:47 smithi181 ceph-mon[35869]: pgmap v2956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:09:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:47 smithi097 ceph-mon[32231]: pgmap v2956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:09:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:48 smithi115 ceph-mon[35364]: pgmap v2957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:09:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:48 smithi181 ceph-mon[35869]: pgmap v2957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:09:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:48 smithi097 ceph-mon[32231]: pgmap v2957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:09:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:49.315+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:09:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:49.316+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:09:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:49.316+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:09:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:49.316+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:09:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:09:49.316+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4227.711874 secs 2024-07-26T09:09:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:50 smithi097 ceph-mon[32231]: pgmap v2958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:09:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4227.711874 secs 2024-07-26T09:09:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:50 smithi115 ceph-mon[35364]: pgmap v2958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:09:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4227.711874 secs 2024-07-26T09:09:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:50 smithi181 ceph-mon[35869]: pgmap v2958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:09:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:52 smithi097 ceph-mon[32231]: pgmap v2959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:52 smithi115 ceph-mon[35364]: pgmap v2959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:52 smithi181 ceph-mon[35869]: pgmap v2959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:54 smithi097 ceph-mon[32231]: pgmap v2960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:54 smithi115 ceph-mon[35364]: pgmap v2960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:54 smithi181 ceph-mon[35869]: pgmap v2960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:09:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4232.712016 secs 2024-07-26T09:09:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4232.712016 secs 2024-07-26T09:09:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4232.712016 secs 2024-07-26T09:09:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:57 smithi115 ceph-mon[35364]: pgmap v2961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:57 smithi181 ceph-mon[35869]: pgmap v2961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:57 smithi097 ceph-mon[32231]: pgmap v2961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:59.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:09:59 smithi115 ceph-mon[35364]: pgmap v2962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:09:59 smithi181 ceph-mon[35869]: pgmap v2962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:09:59 smithi097 ceph-mon[32231]: pgmap v2962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:10:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:10:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:10:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4237.712153 secs 2024-07-26T09:10:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:01 smithi115 ceph-mon[35364]: pgmap v2963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4237.712153 secs 2024-07-26T09:10:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:01 smithi181 ceph-mon[35869]: pgmap v2963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4237.712153 secs 2024-07-26T09:10:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:01 smithi097 ceph-mon[32231]: pgmap v2963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:03 smithi115 ceph-mon[35364]: pgmap v2964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:03 smithi181 ceph-mon[35869]: pgmap v2964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:03 smithi097 ceph-mon[32231]: pgmap v2964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:04.320+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:10:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:04.321+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:10:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:04.321+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:10:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:04.321+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:10:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:04.322+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:05 smithi115 ceph-mon[35364]: pgmap v2965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:05 smithi181 ceph-mon[35869]: pgmap v2965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:05 smithi097 ceph-mon[32231]: pgmap v2965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:06.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4242.712294 secs 2024-07-26T09:10:06.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4242.712294 secs 2024-07-26T09:10:06.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4242.712294 secs 2024-07-26T09:10:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:07 smithi115 ceph-mon[35364]: pgmap v2966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:07 smithi181 ceph-mon[35869]: pgmap v2966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:07 smithi097 ceph-mon[32231]: pgmap v2966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:07.620+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:10:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:07.620+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:10:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:07.621+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:10:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:07.621+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:10:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:09 smithi115 ceph-mon[35364]: pgmap v2967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:09 smithi181 ceph-mon[35869]: pgmap v2967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:09 smithi097 ceph-mon[32231]: pgmap v2967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4247.712381 secs 2024-07-26T09:10:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:11 smithi115 ceph-mon[35364]: pgmap v2968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4247.712381 secs 2024-07-26T09:10:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:11 smithi181 ceph-mon[35869]: pgmap v2968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4247.712381 secs 2024-07-26T09:10:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:11 smithi097 ceph-mon[32231]: pgmap v2968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:13 smithi115 ceph-mon[35364]: pgmap v2969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:10:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:13 smithi181 ceph-mon[35869]: pgmap v2969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:10:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:13 smithi097 ceph-mon[32231]: pgmap v2969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:10:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:15 smithi115 ceph-mon[35364]: pgmap v2970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:15 smithi181 ceph-mon[35869]: pgmap v2970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:15 smithi097 ceph-mon[32231]: pgmap v2970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4252.712510 secs 2024-07-26T09:10:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4252.712510 secs 2024-07-26T09:10:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4252.712510 secs 2024-07-26T09:10:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:17 smithi115 ceph-mon[35364]: pgmap v2971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:10:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:17 smithi181 ceph-mon[35869]: pgmap v2971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:10:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:17 smithi097 ceph-mon[32231]: pgmap v2971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:10:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:19.331+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:10:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:19.333+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:10:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:19.333+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:10:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:19.333+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:10:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:19.333+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:19.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:19 smithi115 ceph-mon[35364]: pgmap v2972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:10:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:19 smithi181 ceph-mon[35869]: pgmap v2972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:10:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:19 smithi097 ceph-mon[32231]: pgmap v2972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:10:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4257.712666 secs 2024-07-26T09:10:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:21 smithi115 ceph-mon[35364]: pgmap v2973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4257.712666 secs 2024-07-26T09:10:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:21 smithi181 ceph-mon[35869]: pgmap v2973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4257.712666 secs 2024-07-26T09:10:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:21 smithi097 ceph-mon[32231]: pgmap v2973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:10:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:23 smithi115 ceph-mon[35364]: pgmap v2974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:23 smithi181 ceph-mon[35869]: pgmap v2974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:23 smithi097 ceph-mon[32231]: pgmap v2974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:24 smithi115 ceph-mon[35364]: pgmap v2975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:24 smithi181 ceph-mon[35869]: pgmap v2975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:24 smithi097 ceph-mon[32231]: pgmap v2975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4262.712755 secs 2024-07-26T09:10:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4262.712755 secs 2024-07-26T09:10:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4262.712755 secs 2024-07-26T09:10:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:26 smithi115 ceph-mon[35364]: pgmap v2976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:26 smithi181 ceph-mon[35869]: pgmap v2976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:26 smithi097 ceph-mon[32231]: pgmap v2976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:29 smithi115 ceph-mon[35364]: pgmap v2977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:29 smithi181 ceph-mon[35869]: pgmap v2977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:29 smithi097 ceph-mon[32231]: pgmap v2977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4267.712893 secs 2024-07-26T09:10:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:31 smithi115 ceph-mon[35364]: pgmap v2978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4267.712893 secs 2024-07-26T09:10:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:31 smithi181 ceph-mon[35869]: pgmap v2978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4267.712893 secs 2024-07-26T09:10:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:31 smithi097 ceph-mon[32231]: pgmap v2978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:33 smithi115 ceph-mon[35364]: pgmap v2979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:33 smithi181 ceph-mon[35869]: pgmap v2979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:33 smithi097 ceph-mon[32231]: pgmap v2979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:34.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:34.338+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:10:34.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:34.339+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:10:34.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:34.339+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:10:34.341 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:34.339+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:10:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:34.339+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:35 smithi115 ceph-mon[35364]: pgmap v2980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:35 smithi181 ceph-mon[35869]: pgmap v2980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:35 smithi097 ceph-mon[32231]: pgmap v2980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:36.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4272.713048 secs 2024-07-26T09:10:36.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4272.713048 secs 2024-07-26T09:10:36.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4272.713048 secs 2024-07-26T09:10:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:37 smithi115 ceph-mon[35364]: pgmap v2981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:37 smithi181 ceph-mon[35869]: pgmap v2981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:37 smithi097 ceph-mon[32231]: pgmap v2981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:39 smithi115 ceph-mon[35364]: pgmap v2982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:39 smithi181 ceph-mon[35869]: pgmap v2982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:39 smithi097 ceph-mon[32231]: pgmap v2982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4277.713199 secs 2024-07-26T09:10:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:41 smithi115 ceph-mon[35364]: pgmap v2983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4277.713199 secs 2024-07-26T09:10:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:41 smithi181 ceph-mon[35869]: pgmap v2983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4277.713199 secs 2024-07-26T09:10:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:41 smithi097 ceph-mon[32231]: pgmap v2983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:43 smithi115 ceph-mon[35364]: pgmap v2984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:43 smithi181 ceph-mon[35869]: pgmap v2984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:43 smithi097 ceph-mon[32231]: pgmap v2984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:45 smithi115 ceph-mon[35364]: pgmap v2985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:45 smithi181 ceph-mon[35869]: pgmap v2985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:45.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:45.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:45 smithi097 ceph-mon[32231]: pgmap v2985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4282.713288 secs 2024-07-26T09:10:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4282.713288 secs 2024-07-26T09:10:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4282.713288 secs 2024-07-26T09:10:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:10:47.459 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:47 smithi181 ceph-mon[35869]: pgmap v2986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:47.459 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:47 smithi181 ceph-mon[35869]: pgmap v2987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:47 smithi115 ceph-mon[35364]: pgmap v2986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:47 smithi115 ceph-mon[35364]: pgmap v2987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:47 smithi097 ceph-mon[32231]: pgmap v2986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:47 smithi097 ceph-mon[32231]: pgmap v2987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:49.341+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:10:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:49.342+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:10:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:49.342+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:10:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:49.342+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:10:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:10:49.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:49 smithi115 ceph-mon[35364]: pgmap v2988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:49 smithi181 ceph-mon[35869]: pgmap v2988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:49 smithi097 ceph-mon[32231]: pgmap v2988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4287.713426 secs 2024-07-26T09:10:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:51 smithi115 ceph-mon[35364]: pgmap v2989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4287.713426 secs 2024-07-26T09:10:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:51 smithi181 ceph-mon[35869]: pgmap v2989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4287.713426 secs 2024-07-26T09:10:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:51 smithi097 ceph-mon[32231]: pgmap v2989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:53.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:53 smithi115 ceph-mon[35364]: pgmap v2990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:53 smithi181 ceph-mon[35869]: pgmap v2990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:53 smithi097 ceph-mon[32231]: pgmap v2990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:55 smithi115 ceph-mon[35364]: pgmap v2991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:55 smithi181 ceph-mon[35869]: pgmap v2991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:55 smithi097 ceph-mon[32231]: pgmap v2991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:10:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4292.713571 secs 2024-07-26T09:10:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4292.713571 secs 2024-07-26T09:10:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4292.713571 secs 2024-07-26T09:10:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:57 smithi115 ceph-mon[35364]: pgmap v2992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:10:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:57 smithi181 ceph-mon[35869]: pgmap v2992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:10:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:57 smithi097 ceph-mon[32231]: pgmap v2992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:10:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:10:59 smithi115 ceph-mon[35364]: pgmap v2993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:10:59 smithi181 ceph-mon[35869]: pgmap v2993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:10:59 smithi097 ceph-mon[32231]: pgmap v2993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4297.713724 secs 2024-07-26T09:11:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:01 smithi115 ceph-mon[35364]: pgmap v2994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4297.713724 secs 2024-07-26T09:11:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:01 smithi181 ceph-mon[35869]: pgmap v2994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4297.713724 secs 2024-07-26T09:11:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:01 smithi097 ceph-mon[32231]: pgmap v2994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:02 smithi115 ceph-mon[35364]: pgmap v2995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:02 smithi181 ceph-mon[35869]: pgmap v2995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:02 smithi097 ceph-mon[32231]: pgmap v2995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:04.348+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:11:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:04.349+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:11:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:04.349+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:11:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:04.349+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:11:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:04.349+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:04 smithi115 ceph-mon[35364]: pgmap v2996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:04 smithi181 ceph-mon[35869]: pgmap v2996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:04 smithi097 ceph-mon[32231]: pgmap v2996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4302.713875 secs 2024-07-26T09:11:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4302.713875 secs 2024-07-26T09:11:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4302.713875 secs 2024-07-26T09:11:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:06 smithi115 ceph-mon[35364]: pgmap v2997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:06 smithi181 ceph-mon[35869]: pgmap v2997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:06 smithi097 ceph-mon[32231]: pgmap v2997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:09.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:08 smithi115 ceph-mon[35364]: pgmap v2998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:08 smithi181 ceph-mon[35869]: pgmap v2998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:08 smithi097 ceph-mon[32231]: pgmap v2998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4307.714013 secs 2024-07-26T09:11:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:10 smithi115 ceph-mon[35364]: pgmap v2999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4307.714013 secs 2024-07-26T09:11:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:10 smithi181 ceph-mon[35869]: pgmap v2999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4307.714013 secs 2024-07-26T09:11:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:10 smithi097 ceph-mon[32231]: pgmap v2999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:12 smithi115 ceph-mon[35364]: pgmap v3000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:11:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:12 smithi181 ceph-mon[35869]: pgmap v3000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:11:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:12 smithi097 ceph-mon[32231]: pgmap v3000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:11:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:14 smithi115 ceph-mon[35364]: pgmap v3001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:14 smithi181 ceph-mon[35869]: pgmap v3001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:14 smithi097 ceph-mon[32231]: pgmap v3001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4312.714148 secs 2024-07-26T09:11:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4312.714148 secs 2024-07-26T09:11:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4312.714148 secs 2024-07-26T09:11:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:16 smithi115 ceph-mon[35364]: pgmap v3002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:16 smithi181 ceph-mon[35869]: pgmap v3002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:16 smithi097 ceph-mon[32231]: pgmap v3002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:19.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:18 smithi181 ceph-mon[35869]: pgmap v3003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:18 smithi097 ceph-mon[32231]: pgmap v3003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:19.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:19.357+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:11:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:19.359+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:11:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:19.359+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:11:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:19.359+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:11:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:19.359+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:18 smithi115 ceph-mon[35364]: pgmap v3003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4317.714284 secs 2024-07-26T09:11:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:20 smithi181 ceph-mon[35869]: pgmap v3004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4317.714284 secs 2024-07-26T09:11:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:20 smithi097 ceph-mon[32231]: pgmap v3004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4317.714284 secs 2024-07-26T09:11:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:20 smithi115 ceph-mon[35364]: pgmap v3004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:22 smithi097 ceph-mon[32231]: pgmap v3005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:11:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:22 smithi115 ceph-mon[35364]: pgmap v3005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:11:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:22 smithi181 ceph-mon[35869]: pgmap v3005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:11:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:24 smithi097 ceph-mon[32231]: pgmap v3006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:24 smithi115 ceph-mon[35364]: pgmap v3006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:24 smithi181 ceph-mon[35869]: pgmap v3006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4322.714450 secs 2024-07-26T09:11:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4322.714450 secs 2024-07-26T09:11:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4322.714450 secs 2024-07-26T09:11:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:27 smithi115 ceph-mon[35364]: pgmap v3007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:27 smithi181 ceph-mon[35869]: pgmap v3007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:27 smithi097 ceph-mon[32231]: pgmap v3007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:11:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:29 smithi115 ceph-mon[35364]: pgmap v3008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:29 smithi181 ceph-mon[35869]: pgmap v3008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:29 smithi097 ceph-mon[32231]: pgmap v3008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:11:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4327.714569 secs 2024-07-26T09:11:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:31 smithi115 ceph-mon[35364]: pgmap v3009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4327.714569 secs 2024-07-26T09:11:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:31 smithi181 ceph-mon[35869]: pgmap v3009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4327.714569 secs 2024-07-26T09:11:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:31 smithi097 ceph-mon[32231]: pgmap v3009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:33 smithi115 ceph-mon[35364]: pgmap v3010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:33 smithi181 ceph-mon[35869]: pgmap v3010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:33 smithi097 ceph-mon[32231]: pgmap v3010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:34.364+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:11:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:34.365+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:11:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:34.365+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:11:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:34.365+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:11:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:34.365+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:35 smithi115 ceph-mon[35364]: pgmap v3011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:35 smithi181 ceph-mon[35869]: pgmap v3011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:35 smithi097 ceph-mon[32231]: pgmap v3011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:36.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4332.714706 secs 2024-07-26T09:11:36.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4332.714706 secs 2024-07-26T09:11:36.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4332.714706 secs 2024-07-26T09:11:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:37 smithi115 ceph-mon[35364]: pgmap v3012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:37 smithi181 ceph-mon[35869]: pgmap v3012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:37 smithi097 ceph-mon[32231]: pgmap v3012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:39 smithi115 ceph-mon[35364]: pgmap v3013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:39 smithi181 ceph-mon[35869]: pgmap v3013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:39 smithi097 ceph-mon[32231]: pgmap v3013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4337.714837 secs 2024-07-26T09:11:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:41 smithi115 ceph-mon[35364]: pgmap v3014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4337.714837 secs 2024-07-26T09:11:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:41 smithi181 ceph-mon[35869]: pgmap v3014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4337.714837 secs 2024-07-26T09:11:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:41 smithi097 ceph-mon[32231]: pgmap v3014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:43 smithi115 ceph-mon[35364]: pgmap v3015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:43 smithi181 ceph-mon[35869]: pgmap v3015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:43 smithi097 ceph-mon[32231]: pgmap v3015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:45 smithi115 ceph-mon[35364]: pgmap v3016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:45 smithi181 ceph-mon[35869]: pgmap v3016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:45 smithi097 ceph-mon[32231]: pgmap v3016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4342.714979 secs 2024-07-26T09:11:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4342.714979 secs 2024-07-26T09:11:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4342.714979 secs 2024-07-26T09:11:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:47 smithi115 ceph-mon[35364]: pgmap v3017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:47 smithi181 ceph-mon[35869]: pgmap v3017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:47 smithi097 ceph-mon[32231]: pgmap v3017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:47.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:11:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:49.373+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:11:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:49.375+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:11:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:49.375+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:11:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:49.375+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:11:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:11:49.376+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:49 smithi115 ceph-mon[35364]: pgmap v3018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:49 smithi115 ceph-mon[35364]: pgmap v3019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:49 smithi181 ceph-mon[35869]: pgmap v3018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:49 smithi181 ceph-mon[35869]: pgmap v3019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:49 smithi097 ceph-mon[32231]: pgmap v3018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:49 smithi097 ceph-mon[32231]: pgmap v3019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4347.715112 secs 2024-07-26T09:11:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:51 smithi115 ceph-mon[35364]: pgmap v3020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4347.715112 secs 2024-07-26T09:11:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:51 smithi181 ceph-mon[35869]: pgmap v3020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4347.715112 secs 2024-07-26T09:11:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:51 smithi097 ceph-mon[32231]: pgmap v3020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:53 smithi115 ceph-mon[35364]: pgmap v3021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:53 smithi181 ceph-mon[35869]: pgmap v3021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:53 smithi097 ceph-mon[32231]: pgmap v3021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:55 smithi115 ceph-mon[35364]: pgmap v3022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:55 smithi181 ceph-mon[35869]: pgmap v3022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:55 smithi097 ceph-mon[32231]: pgmap v3022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4352.715292 secs 2024-07-26T09:11:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4352.715292 secs 2024-07-26T09:11:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4352.715292 secs 2024-07-26T09:11:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:57 smithi115 ceph-mon[35364]: pgmap v3023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:57 smithi181 ceph-mon[35869]: pgmap v3023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:57 smithi097 ceph-mon[32231]: pgmap v3023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:11:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:11:58 smithi115 ceph-mon[35364]: pgmap v3024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:11:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:11:58 smithi181 ceph-mon[35869]: pgmap v3024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:11:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:11:58 smithi097 ceph-mon[32231]: pgmap v3024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4357.715419 secs 2024-07-26T09:12:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:00 smithi115 ceph-mon[35364]: pgmap v3025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4357.715419 secs 2024-07-26T09:12:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:00 smithi181 ceph-mon[35869]: pgmap v3025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4357.715419 secs 2024-07-26T09:12:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:00 smithi097 ceph-mon[32231]: pgmap v3025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:02 smithi115 ceph-mon[35364]: pgmap v3026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:02 smithi181 ceph-mon[35869]: pgmap v3026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:02 smithi097 ceph-mon[32231]: pgmap v3026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:04.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:04.381+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:12:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:04.383+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:12:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:04.383+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:12:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:04.383+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:12:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:04.383+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:04 smithi115 ceph-mon[35364]: pgmap v3027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:04 smithi181 ceph-mon[35869]: pgmap v3027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:04 smithi097 ceph-mon[32231]: pgmap v3027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4362.715571 secs 2024-07-26T09:12:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4362.715571 secs 2024-07-26T09:12:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4362.715571 secs 2024-07-26T09:12:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:06 smithi115 ceph-mon[35364]: pgmap v3028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:06 smithi181 ceph-mon[35869]: pgmap v3028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:06 smithi097 ceph-mon[32231]: pgmap v3028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:08 smithi115 ceph-mon[35364]: pgmap v3029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:12:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:08 smithi181 ceph-mon[35869]: pgmap v3029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:12:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:08 smithi097 ceph-mon[32231]: pgmap v3029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:12:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4367.715729 secs 2024-07-26T09:12:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:10 smithi115 ceph-mon[35364]: pgmap v3030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4367.715729 secs 2024-07-26T09:12:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:10 smithi181 ceph-mon[35869]: pgmap v3030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4367.715729 secs 2024-07-26T09:12:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:10 smithi097 ceph-mon[32231]: pgmap v3030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:12 smithi115 ceph-mon[35364]: pgmap v3031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:12:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:12 smithi181 ceph-mon[35869]: pgmap v3031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:12:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:12 smithi097 ceph-mon[32231]: pgmap v3031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:12:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:14 smithi115 ceph-mon[35364]: pgmap v3032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:14 smithi181 ceph-mon[35869]: pgmap v3032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:14 smithi097 ceph-mon[32231]: pgmap v3032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4372.715874 secs 2024-07-26T09:12:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4372.715874 secs 2024-07-26T09:12:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4372.715874 secs 2024-07-26T09:12:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:17 smithi115 ceph-mon[35364]: pgmap v3033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:17.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:17 smithi181 ceph-mon[35869]: pgmap v3033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:17 smithi097 ceph-mon[32231]: pgmap v3033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:19.393+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:12:19.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:19.395+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:12:19.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:19.395+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:12:19.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:19.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:12:19.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:19.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:19 smithi115 ceph-mon[35364]: pgmap v3034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:19 smithi181 ceph-mon[35869]: pgmap v3034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:19 smithi097 ceph-mon[32231]: pgmap v3034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4377.716006 secs 2024-07-26T09:12:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:21 smithi115 ceph-mon[35364]: pgmap v3035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4377.716006 secs 2024-07-26T09:12:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:21 smithi181 ceph-mon[35869]: pgmap v3035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4377.716006 secs 2024-07-26T09:12:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:21 smithi097 ceph-mon[32231]: pgmap v3035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:22.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:22.644+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:12:22.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:22.644+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:12:22.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:22.645+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:12:22.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:22.645+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:12:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:23 smithi115 ceph-mon[35364]: pgmap v3036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:23 smithi181 ceph-mon[35869]: pgmap v3036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:23 smithi097 ceph-mon[32231]: pgmap v3036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:25 smithi115 ceph-mon[35364]: pgmap v3037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:25 smithi181 ceph-mon[35869]: pgmap v3037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:25 smithi097 ceph-mon[32231]: pgmap v3037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4382.716145 secs 2024-07-26T09:12:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4382.716145 secs 2024-07-26T09:12:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4382.716145 secs 2024-07-26T09:12:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:27 smithi115 ceph-mon[35364]: pgmap v3038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:27 smithi181 ceph-mon[35869]: pgmap v3038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:27 smithi097 ceph-mon[32231]: pgmap v3038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:29 smithi115 ceph-mon[35364]: pgmap v3039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:29 smithi181 ceph-mon[35869]: pgmap v3039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:29 smithi097 ceph-mon[32231]: pgmap v3039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4387.716273 secs 2024-07-26T09:12:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:31 smithi115 ceph-mon[35364]: pgmap v3040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4387.716273 secs 2024-07-26T09:12:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:31 smithi181 ceph-mon[35869]: pgmap v3040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4387.716273 secs 2024-07-26T09:12:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:31 smithi097 ceph-mon[32231]: pgmap v3040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:33 smithi115 ceph-mon[35364]: pgmap v3041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:33 smithi181 ceph-mon[35869]: pgmap v3041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:33 smithi097 ceph-mon[32231]: pgmap v3041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:34.399+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:12:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:34.401+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:12:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:34.401+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:12:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:34.401+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:12:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:34.401+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:35 smithi115 ceph-mon[35364]: pgmap v3042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:35 smithi181 ceph-mon[35869]: pgmap v3042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:35 smithi097 ceph-mon[32231]: pgmap v3042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4392.716411 secs 2024-07-26T09:12:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:36 smithi115 ceph-mon[35364]: pgmap v3043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4392.716411 secs 2024-07-26T09:12:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:36 smithi181 ceph-mon[35869]: pgmap v3043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4392.716411 secs 2024-07-26T09:12:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:36 smithi097 ceph-mon[32231]: pgmap v3043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:38 smithi115 ceph-mon[35364]: pgmap v3044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:38 smithi181 ceph-mon[35869]: pgmap v3044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:38 smithi097 ceph-mon[32231]: pgmap v3044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4397.716554 secs 2024-07-26T09:12:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:40 smithi115 ceph-mon[35364]: pgmap v3045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4397.716554 secs 2024-07-26T09:12:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:40 smithi181 ceph-mon[35869]: pgmap v3045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4397.716554 secs 2024-07-26T09:12:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:40 smithi097 ceph-mon[32231]: pgmap v3045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:42 smithi115 ceph-mon[35364]: pgmap v3046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:42 smithi181 ceph-mon[35869]: pgmap v3046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:42 smithi097 ceph-mon[32231]: pgmap v3046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:44 smithi115 ceph-mon[35364]: pgmap v3047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:44 smithi181 ceph-mon[35869]: pgmap v3047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:44 smithi097 ceph-mon[32231]: pgmap v3047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4402.716650 secs 2024-07-26T09:12:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4402.716650 secs 2024-07-26T09:12:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4402.716650 secs 2024-07-26T09:12:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:46 smithi115 ceph-mon[35364]: pgmap v3048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:46 smithi181 ceph-mon[35869]: pgmap v3048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:46 smithi097 ceph-mon[32231]: pgmap v3048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:48 smithi115 ceph-mon[35364]: pgmap v3049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:48 smithi181 ceph-mon[35869]: pgmap v3049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:48 smithi097 ceph-mon[32231]: pgmap v3049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:49.411+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:49.412+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:49.412+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:49.412+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:12:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:12:49.412+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4407.716788 secs 2024-07-26T09:12:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:50 smithi115 ceph-mon[35364]: pgmap v3050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4407.716788 secs 2024-07-26T09:12:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:50 smithi181 ceph-mon[35869]: pgmap v3050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4407.716788 secs 2024-07-26T09:12:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:50 smithi097 ceph-mon[32231]: pgmap v3050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:52 smithi115 ceph-mon[35364]: pgmap v3051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:52 smithi181 ceph-mon[35869]: pgmap v3051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:52 smithi097 ceph-mon[32231]: pgmap v3051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:53 smithi115 ceph-mon[35364]: pgmap v3052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:53 smithi115 ceph-mon[35364]: pgmap v3053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T09:12:53.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:53 smithi181 ceph-mon[35869]: pgmap v3052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:53 smithi181 ceph-mon[35869]: pgmap v3053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T09:12:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:53 smithi097 ceph-mon[32231]: pgmap v3052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:53 smithi097 ceph-mon[32231]: pgmap v3053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T09:12:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:12:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:55 smithi115 ceph-mon[35364]: pgmap v3054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4412.716877 secs 2024-07-26T09:12:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:55 smithi181 ceph-mon[35869]: pgmap v3054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4412.716877 secs 2024-07-26T09:12:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:55 smithi097 ceph-mon[32231]: pgmap v3054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4412.716877 secs 2024-07-26T09:12:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:12:58 smithi115 ceph-mon[35364]: pgmap v3055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:12:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:12:58 smithi181 ceph-mon[35869]: pgmap v3055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:12:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:12:58 smithi097 ceph-mon[32231]: pgmap v3055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:13:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:00 smithi115 ceph-mon[35364]: pgmap v3056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T09:13:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:00 smithi181 ceph-mon[35869]: pgmap v3056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T09:13:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:00 smithi097 ceph-mon[32231]: pgmap v3056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T09:13:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4417.717000 secs 2024-07-26T09:13:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4417.717000 secs 2024-07-26T09:13:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4417.717000 secs 2024-07-26T09:13:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:02 smithi115 ceph-mon[35364]: pgmap v3057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:13:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:02 smithi181 ceph-mon[35869]: pgmap v3057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:13:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:02 smithi097 ceph-mon[32231]: pgmap v3057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:13:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:04.417+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:13:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:04.418+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:13:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:04.418+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:13:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:04.418+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:13:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:04.418+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:04 smithi115 ceph-mon[35364]: pgmap v3058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:13:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:04 smithi181 ceph-mon[35869]: pgmap v3058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:13:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:04 smithi097 ceph-mon[32231]: pgmap v3058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:13:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:06 smithi115 ceph-mon[35364]: pgmap v3059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4422.717140 secs 2024-07-26T09:13:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:06 smithi181 ceph-mon[35869]: pgmap v3059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4422.717140 secs 2024-07-26T09:13:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:06 smithi097 ceph-mon[32231]: pgmap v3059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4422.717140 secs 2024-07-26T09:13:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:06.890+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40c0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000350) for more than 900000000 us, fault. 2024-07-26T09:13:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:06.890+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0fc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff3e001fcb0 tx=0x7ff3e0071000 comp rx=0 tx=0).stop 2024-07-26T09:13:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:06.890+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40c0fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:13:06.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:06.890+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40c0fc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:13:06.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:06.890+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:13:06.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:06.891+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:13:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:06.892+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0830 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e006a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:13:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:07.652+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:13:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:07.652+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:13:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:07.653+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:13:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:07.654+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:13:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:08 smithi097 ceph-mon[32231]: pgmap v3060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:08 smithi115 ceph-mon[35364]: pgmap v3060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:08 smithi181 ceph-mon[35869]: pgmap v3060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:10 smithi115 ceph-mon[35364]: pgmap v3061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:10 smithi181 ceph-mon[35869]: pgmap v3061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:10 smithi097 ceph-mon[32231]: pgmap v3061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4427.717273 secs 2024-07-26T09:13:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4427.717273 secs 2024-07-26T09:13:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4427.717273 secs 2024-07-26T09:13:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:12 smithi115 ceph-mon[35364]: pgmap v3062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:12 smithi181 ceph-mon[35869]: pgmap v3062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:12 smithi097 ceph-mon[32231]: pgmap v3062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:13 smithi115 ceph-mon[35364]: pgmap v3063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:13.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:13 smithi181 ceph-mon[35869]: pgmap v3063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:13 smithi097 ceph-mon[32231]: pgmap v3063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:15 smithi115 ceph-mon[35364]: pgmap v3064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4432.717419 secs 2024-07-26T09:13:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:15 smithi181 ceph-mon[35869]: pgmap v3064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4432.717419 secs 2024-07-26T09:13:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:15 smithi097 ceph-mon[32231]: pgmap v3064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4432.717419 secs 2024-07-26T09:13:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:17 smithi115 ceph-mon[35364]: pgmap v3065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:17.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:17 smithi181 ceph-mon[35869]: pgmap v3065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:17 smithi097 ceph-mon[32231]: pgmap v3065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:19.427+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:13:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:19.428+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:13:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:19.428+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:13:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:19.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:13:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:19.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:19 smithi115 ceph-mon[35364]: pgmap v3066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:19 smithi181 ceph-mon[35869]: pgmap v3066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:19 smithi097 ceph-mon[32231]: pgmap v3066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4437.717507 secs 2024-07-26T09:13:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4437.717507 secs 2024-07-26T09:13:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4437.717507 secs 2024-07-26T09:13:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:21 smithi181 ceph-mon[35869]: pgmap v3067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:21 smithi097 ceph-mon[32231]: pgmap v3067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:21 smithi115 ceph-mon[35364]: pgmap v3067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:23 smithi097 ceph-mon[32231]: pgmap v3068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:23 smithi115 ceph-mon[35364]: pgmap v3068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:23 smithi181 ceph-mon[35869]: pgmap v3068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:25 smithi097 ceph-mon[32231]: pgmap v3069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:26.008 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4442.717654 secs 2024-07-26T09:13:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:25 smithi115 ceph-mon[35364]: pgmap v3069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4442.717654 secs 2024-07-26T09:13:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:25 smithi181 ceph-mon[35869]: pgmap v3069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4442.717654 secs 2024-07-26T09:13:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:27 smithi097 ceph-mon[32231]: pgmap v3070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:27 smithi115 ceph-mon[35364]: pgmap v3070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:27 smithi181 ceph-mon[35869]: pgmap v3070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:29 smithi115 ceph-mon[35364]: pgmap v3071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:29 smithi181 ceph-mon[35869]: pgmap v3071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:29 smithi097 ceph-mon[32231]: pgmap v3071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4447.717787 secs 2024-07-26T09:13:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4447.717787 secs 2024-07-26T09:13:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4447.717787 secs 2024-07-26T09:13:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:31 smithi115 ceph-mon[35364]: pgmap v3072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:31 smithi181 ceph-mon[35869]: pgmap v3072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:31 smithi097 ceph-mon[32231]: pgmap v3072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:33 smithi115 ceph-mon[35364]: pgmap v3073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:33 smithi181 ceph-mon[35869]: pgmap v3073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:33 smithi097 ceph-mon[32231]: pgmap v3073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:34.432+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:13:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:34.433+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:13:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:34.433+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:13:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:34.433+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:13:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:34.433+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:36.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:35 smithi097 ceph-mon[32231]: pgmap v3074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:36.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4452.717927 secs 2024-07-26T09:13:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:35 smithi115 ceph-mon[35364]: pgmap v3074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4452.717927 secs 2024-07-26T09:13:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:35 smithi181 ceph-mon[35869]: pgmap v3074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4452.717927 secs 2024-07-26T09:13:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:37 smithi115 ceph-mon[35364]: pgmap v3075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:37 smithi181 ceph-mon[35869]: pgmap v3075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:37 smithi097 ceph-mon[32231]: pgmap v3075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:39 smithi115 ceph-mon[35364]: pgmap v3076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:13:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:39 smithi181 ceph-mon[35869]: pgmap v3076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:13:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:39 smithi097 ceph-mon[32231]: pgmap v3076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:13:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4457.718062 secs 2024-07-26T09:13:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4457.718062 secs 2024-07-26T09:13:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4457.718062 secs 2024-07-26T09:13:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:41 smithi115 ceph-mon[35364]: pgmap v3077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:41 smithi181 ceph-mon[35869]: pgmap v3077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:41 smithi097 ceph-mon[32231]: pgmap v3077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:43 smithi115 ceph-mon[35364]: pgmap v3078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:13:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:43 smithi181 ceph-mon[35869]: pgmap v3078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:13:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:43 smithi097 ceph-mon[32231]: pgmap v3078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:13:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:45 smithi115 ceph-mon[35364]: pgmap v3079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4462.718204 secs 2024-07-26T09:13:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:45 smithi181 ceph-mon[35869]: pgmap v3079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4462.718204 secs 2024-07-26T09:13:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:45 smithi097 ceph-mon[32231]: pgmap v3079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:13:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4462.718204 secs 2024-07-26T09:13:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:47 smithi115 ceph-mon[35364]: pgmap v3080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:47 smithi181 ceph-mon[35869]: pgmap v3080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:47 smithi097 ceph-mon[32231]: pgmap v3080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:49.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:13:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:49.442+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:13:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:49.442+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:13:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:49.442+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:13:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:13:49.442+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:49 smithi115 ceph-mon[35364]: pgmap v3081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:49 smithi181 ceph-mon[35869]: pgmap v3081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:49 smithi097 ceph-mon[32231]: pgmap v3081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4467.718336 secs 2024-07-26T09:13:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4467.718336 secs 2024-07-26T09:13:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4467.718336 secs 2024-07-26T09:13:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:51 smithi115 ceph-mon[35364]: pgmap v3082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:51 smithi181 ceph-mon[35869]: pgmap v3082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:51 smithi097 ceph-mon[32231]: pgmap v3082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:53 smithi115 ceph-mon[35364]: pgmap v3083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:53 smithi181 ceph-mon[35869]: pgmap v3083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:54.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:53 smithi097 ceph-mon[32231]: pgmap v3083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:54 smithi115 ceph-mon[35364]: pgmap v3084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:13:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:54 smithi115 ceph-mon[35364]: pgmap v3085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:55.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:13:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:54 smithi181 ceph-mon[35869]: pgmap v3084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:13:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:54 smithi181 ceph-mon[35869]: pgmap v3085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:13:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:54 smithi097 ceph-mon[32231]: pgmap v3084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:13:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:54 smithi097 ceph-mon[32231]: pgmap v3085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:13:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4472.718479 secs 2024-07-26T09:13:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4472.718479 secs 2024-07-26T09:13:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4472.718479 secs 2024-07-26T09:13:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:56 smithi115 ceph-mon[35364]: pgmap v3086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:56 smithi181 ceph-mon[35869]: pgmap v3086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:57.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:56 smithi097 ceph-mon[32231]: pgmap v3086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:13:59 smithi115 ceph-mon[35364]: pgmap v3087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:13:59 smithi181 ceph-mon[35869]: pgmap v3087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:13:59 smithi097 ceph-mon[32231]: pgmap v3087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4477.718617 secs 2024-07-26T09:14:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:01 smithi115 ceph-mon[35364]: pgmap v3088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4477.718617 secs 2024-07-26T09:14:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:01 smithi181 ceph-mon[35869]: pgmap v3088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4477.718617 secs 2024-07-26T09:14:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:01 smithi097 ceph-mon[32231]: pgmap v3088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:03 smithi115 ceph-mon[35364]: pgmap v3089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:14:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:03 smithi181 ceph-mon[35869]: pgmap v3089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:14:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:03 smithi097 ceph-mon[32231]: pgmap v3089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:14:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:04.447+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:14:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:04.448+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:14:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:04.448+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:14:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:04.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:14:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:04.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:05 smithi115 ceph-mon[35364]: pgmap v3090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:05 smithi181 ceph-mon[35869]: pgmap v3090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:05 smithi097 ceph-mon[32231]: pgmap v3090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4482.718759 secs 2024-07-26T09:14:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4482.718759 secs 2024-07-26T09:14:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4482.718759 secs 2024-07-26T09:14:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:07 smithi115 ceph-mon[35364]: pgmap v3091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:07.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:07 smithi181 ceph-mon[35869]: pgmap v3091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:07 smithi097 ceph-mon[32231]: pgmap v3091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:08 smithi115 ceph-mon[35364]: pgmap v3092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:08 smithi181 ceph-mon[35869]: pgmap v3092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:08 smithi097 ceph-mon[32231]: pgmap v3092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4487.718892 secs 2024-07-26T09:14:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:11 smithi115 ceph-mon[35364]: pgmap v3093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4487.718892 secs 2024-07-26T09:14:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:11 smithi181 ceph-mon[35869]: pgmap v3093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4487.718892 secs 2024-07-26T09:14:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:11 smithi097 ceph-mon[32231]: pgmap v3093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:13 smithi115 ceph-mon[35364]: pgmap v3094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:14:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:13 smithi181 ceph-mon[35869]: pgmap v3094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:14:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:13 smithi097 ceph-mon[32231]: pgmap v3094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:14:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:15 smithi115 ceph-mon[35364]: pgmap v3095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:15 smithi181 ceph-mon[35869]: pgmap v3095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:15 smithi097 ceph-mon[32231]: pgmap v3095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4492.719027 secs 2024-07-26T09:14:16.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4492.719027 secs 2024-07-26T09:14:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4492.719027 secs 2024-07-26T09:14:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:17 smithi115 ceph-mon[35364]: pgmap v3096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:17 smithi181 ceph-mon[35869]: pgmap v3096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:17 smithi097 ceph-mon[32231]: pgmap v3096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:18 smithi115 ceph-mon[35364]: pgmap v3097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:18 smithi181 ceph-mon[35869]: pgmap v3097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:18 smithi097 ceph-mon[32231]: pgmap v3097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:19.457+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:14:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:19.459+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:14:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:19.459+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:14:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:19.459+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:14:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:19.459+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4497.719186 secs 2024-07-26T09:14:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:21 smithi115 ceph-mon[35364]: pgmap v3098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4497.719186 secs 2024-07-26T09:14:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:21 smithi181 ceph-mon[35869]: pgmap v3098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4497.719186 secs 2024-07-26T09:14:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:21 smithi097 ceph-mon[32231]: pgmap v3098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:23 smithi115 ceph-mon[35364]: pgmap v3099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:23 smithi181 ceph-mon[35869]: pgmap v3099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:23 smithi097 ceph-mon[32231]: pgmap v3099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:25 smithi115 ceph-mon[35364]: pgmap v3100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:25 smithi181 ceph-mon[35869]: pgmap v3100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:25 smithi097 ceph-mon[32231]: pgmap v3100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4502.719362 secs 2024-07-26T09:14:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4502.719362 secs 2024-07-26T09:14:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4502.719362 secs 2024-07-26T09:14:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:27 smithi115 ceph-mon[35364]: pgmap v3101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:27 smithi181 ceph-mon[35869]: pgmap v3101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:27 smithi097 ceph-mon[32231]: pgmap v3101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:28 smithi115 ceph-mon[35364]: pgmap v3102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:28 smithi181 ceph-mon[35869]: pgmap v3102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:28 smithi097 ceph-mon[32231]: pgmap v3102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4507.719481 secs 2024-07-26T09:14:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:31 smithi115 ceph-mon[35364]: pgmap v3103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4507.719481 secs 2024-07-26T09:14:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:31 smithi181 ceph-mon[35869]: pgmap v3103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4507.719481 secs 2024-07-26T09:14:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:31 smithi097 ceph-mon[32231]: pgmap v3103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:33 smithi115 ceph-mon[35364]: pgmap v3104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:33.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:33 smithi181 ceph-mon[35869]: pgmap v3104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:33 smithi097 ceph-mon[32231]: pgmap v3104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:34.463+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:14:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:34.464+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:14:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:34.464+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:14:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:34.464+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:14:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:34.465+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:34 smithi115 ceph-mon[35364]: pgmap v3105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:34 smithi181 ceph-mon[35869]: pgmap v3105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:34 smithi097 ceph-mon[32231]: pgmap v3105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4512.719624 secs 2024-07-26T09:14:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4512.719624 secs 2024-07-26T09:14:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4512.719624 secs 2024-07-26T09:14:36.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:36 smithi115 ceph-mon[35364]: pgmap v3106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:36.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:36 smithi181 ceph-mon[35869]: pgmap v3106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:36 smithi097 ceph-mon[32231]: pgmap v3106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:39 smithi115 ceph-mon[35364]: pgmap v3107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:39 smithi181 ceph-mon[35869]: pgmap v3107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:39 smithi097 ceph-mon[32231]: pgmap v3107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4517.719761 secs 2024-07-26T09:14:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:41 smithi115 ceph-mon[35364]: pgmap v3108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4517.719761 secs 2024-07-26T09:14:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:41 smithi181 ceph-mon[35869]: pgmap v3108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4517.719761 secs 2024-07-26T09:14:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:41 smithi097 ceph-mon[32231]: pgmap v3108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:43 smithi115 ceph-mon[35364]: pgmap v3109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:43 smithi181 ceph-mon[35869]: pgmap v3109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:43 smithi097 ceph-mon[32231]: pgmap v3109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:44 smithi097 ceph-mon[32231]: pgmap v3110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:44 smithi115 ceph-mon[35364]: pgmap v3110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:44 smithi181 ceph-mon[35869]: pgmap v3110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4522.719855 secs 2024-07-26T09:14:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4522.719855 secs 2024-07-26T09:14:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4522.719855 secs 2024-07-26T09:14:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:46 smithi115 ceph-mon[35364]: pgmap v3111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:46 smithi181 ceph-mon[35869]: pgmap v3111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:46 smithi097 ceph-mon[32231]: pgmap v3111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:14:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:49 smithi115 ceph-mon[35364]: pgmap v3112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:49.474+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:14:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:49.475+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:14:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:49.475+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:14:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:49.475+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:14:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:14:49.475+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:49 smithi181 ceph-mon[35869]: pgmap v3112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:49 smithi097 ceph-mon[32231]: pgmap v3112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4527.719992 secs 2024-07-26T09:14:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:51 smithi115 ceph-mon[35364]: pgmap v3113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4527.719992 secs 2024-07-26T09:14:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:51 smithi181 ceph-mon[35869]: pgmap v3113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4527.719992 secs 2024-07-26T09:14:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:51 smithi097 ceph-mon[32231]: pgmap v3113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:53 smithi115 ceph-mon[35364]: pgmap v3114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:53 smithi181 ceph-mon[35869]: pgmap v3114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:53 smithi097 ceph-mon[32231]: pgmap v3114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:54 smithi097 ceph-mon[32231]: pgmap v3115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:54 smithi115 ceph-mon[35364]: pgmap v3115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:54 smithi181 ceph-mon[35869]: pgmap v3115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:14:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4532.720131 secs 2024-07-26T09:14:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:56 smithi115 ceph-mon[35364]: pgmap v3116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:56 smithi115 ceph-mon[35364]: pgmap v3117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:56.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:14:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4532.720131 secs 2024-07-26T09:14:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:56 smithi181 ceph-mon[35869]: pgmap v3116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:56 smithi181 ceph-mon[35869]: pgmap v3117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:14:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4532.720131 secs 2024-07-26T09:14:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:56 smithi097 ceph-mon[32231]: pgmap v3116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:56 smithi097 ceph-mon[32231]: pgmap v3117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:14:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:14:58 smithi115 ceph-mon[35364]: pgmap v3118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:14:58 smithi181 ceph-mon[35869]: pgmap v3118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:14:58 smithi097 ceph-mon[32231]: pgmap v3118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:15:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:00 smithi115 ceph-mon[35364]: pgmap v3119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:15:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4537.720272 secs 2024-07-26T09:15:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:00 smithi181 ceph-mon[35869]: pgmap v3119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:15:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4537.720272 secs 2024-07-26T09:15:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:00 smithi097 ceph-mon[32231]: pgmap v3119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:15:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4537.720272 secs 2024-07-26T09:15:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:02 smithi115 ceph-mon[35364]: pgmap v3120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:15:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:02 smithi181 ceph-mon[35869]: pgmap v3120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:15:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:02 smithi097 ceph-mon[32231]: pgmap v3120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:15:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:04.480+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:15:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:04.482+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:15:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:04.482+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:15:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:04.482+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:15:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:04.482+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:04.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:04 smithi115 ceph-mon[35364]: pgmap v3121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:15:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:04 smithi181 ceph-mon[35869]: pgmap v3121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:15:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:04 smithi097 ceph-mon[32231]: pgmap v3121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:15:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4542.720417 secs 2024-07-26T09:15:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4542.720417 secs 2024-07-26T09:15:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4542.720417 secs 2024-07-26T09:15:06.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:06 smithi115 ceph-mon[35364]: pgmap v3122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:15:06.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:06 smithi181 ceph-mon[35869]: pgmap v3122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:15:06.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:06 smithi097 ceph-mon[32231]: pgmap v3122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:15:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:08 smithi115 ceph-mon[35364]: pgmap v3123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:08 smithi181 ceph-mon[35869]: pgmap v3123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:08 smithi097 ceph-mon[32231]: pgmap v3123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:10 smithi115 ceph-mon[35364]: pgmap v3124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4547.720556 secs 2024-07-26T09:15:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:10 smithi181 ceph-mon[35869]: pgmap v3124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4547.720556 secs 2024-07-26T09:15:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:10 smithi097 ceph-mon[32231]: pgmap v3124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4547.720556 secs 2024-07-26T09:15:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:12 smithi115 ceph-mon[35364]: pgmap v3125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:12 smithi181 ceph-mon[35869]: pgmap v3125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:12 smithi097 ceph-mon[32231]: pgmap v3125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:14 smithi115 ceph-mon[35364]: pgmap v3126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:14 smithi181 ceph-mon[35869]: pgmap v3126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:14 smithi097 ceph-mon[32231]: pgmap v3126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4552.720702 secs 2024-07-26T09:15:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4552.720702 secs 2024-07-26T09:15:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4552.720702 secs 2024-07-26T09:15:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:17 smithi115 ceph-mon[35364]: pgmap v3127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:17 smithi181 ceph-mon[35869]: pgmap v3127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:17 smithi097 ceph-mon[32231]: pgmap v3127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:19 smithi115 ceph-mon[35364]: pgmap v3128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:19 smithi181 ceph-mon[35869]: pgmap v3128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:19.492+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:15:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:19.493+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:15:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:19.493+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:15:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:19.493+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:15:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:19.494+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:19 smithi097 ceph-mon[32231]: pgmap v3128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:21 smithi115 ceph-mon[35364]: pgmap v3129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4557.720793 secs 2024-07-26T09:15:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:21 smithi181 ceph-mon[35869]: pgmap v3129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4557.720793 secs 2024-07-26T09:15:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:21 smithi097 ceph-mon[32231]: pgmap v3129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4557.720793 secs 2024-07-26T09:15:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:23 smithi115 ceph-mon[35364]: pgmap v3130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:15:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:23 smithi181 ceph-mon[35869]: pgmap v3130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:15:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:23 smithi097 ceph-mon[32231]: pgmap v3130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:15:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:25 smithi115 ceph-mon[35364]: pgmap v3131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:25 smithi181 ceph-mon[35869]: pgmap v3131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:25 smithi097 ceph-mon[32231]: pgmap v3131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4562.720886 secs 2024-07-26T09:15:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4562.720886 secs 2024-07-26T09:15:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4562.720886 secs 2024-07-26T09:15:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:27 smithi115 ceph-mon[35364]: pgmap v3132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:15:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:27 smithi181 ceph-mon[35869]: pgmap v3132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:15:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:27 smithi097 ceph-mon[32231]: pgmap v3132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:15:28.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:28 smithi181 ceph-mon[35869]: pgmap v3133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:15:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:28 smithi097 ceph-mon[32231]: pgmap v3133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:15:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:28 smithi115 ceph-mon[35364]: pgmap v3133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:15:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:30 smithi115 ceph-mon[35364]: pgmap v3134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4567.721024 secs 2024-07-26T09:15:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:30 smithi181 ceph-mon[35869]: pgmap v3134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4567.721024 secs 2024-07-26T09:15:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:30 smithi097 ceph-mon[32231]: pgmap v3134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4567.721024 secs 2024-07-26T09:15:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:32 smithi115 ceph-mon[35364]: pgmap v3135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:32 smithi181 ceph-mon[35869]: pgmap v3135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:32 smithi097 ceph-mon[32231]: pgmap v3135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:34.500+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:15:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:34.501+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:15:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:34.501+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:15:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:34.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:15:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:34.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:34 smithi115 ceph-mon[35364]: pgmap v3136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:34 smithi181 ceph-mon[35869]: pgmap v3136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:34 smithi097 ceph-mon[32231]: pgmap v3136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4572.721168 secs 2024-07-26T09:15:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4572.721168 secs 2024-07-26T09:15:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4572.721168 secs 2024-07-26T09:15:36.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:36 smithi115 ceph-mon[35364]: pgmap v3137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:36.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:36 smithi181 ceph-mon[35869]: pgmap v3137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:36.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:36 smithi097 ceph-mon[32231]: pgmap v3137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:38 smithi115 ceph-mon[35364]: pgmap v3138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:38 smithi181 ceph-mon[35869]: pgmap v3138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:38 smithi097 ceph-mon[32231]: pgmap v3138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:40 smithi115 ceph-mon[35364]: pgmap v3139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4577.721307 secs 2024-07-26T09:15:40.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:40 smithi181 ceph-mon[35869]: pgmap v3139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:40.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4577.721307 secs 2024-07-26T09:15:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:40 smithi097 ceph-mon[32231]: pgmap v3139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4577.721307 secs 2024-07-26T09:15:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:42 smithi115 ceph-mon[35364]: pgmap v3140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:42 smithi181 ceph-mon[35869]: pgmap v3140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:42 smithi097 ceph-mon[32231]: pgmap v3140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:44 smithi115 ceph-mon[35364]: pgmap v3141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:44 smithi181 ceph-mon[35869]: pgmap v3141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:44 smithi097 ceph-mon[32231]: pgmap v3141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4582.721451 secs 2024-07-26T09:15:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4582.721451 secs 2024-07-26T09:15:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4582.721451 secs 2024-07-26T09:15:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:46 smithi115 ceph-mon[35364]: pgmap v3142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:46 smithi181 ceph-mon[35869]: pgmap v3142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:46 smithi097 ceph-mon[32231]: pgmap v3142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:48 smithi115 ceph-mon[35364]: pgmap v3143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:48 smithi181 ceph-mon[35869]: pgmap v3143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:48 smithi097 ceph-mon[32231]: pgmap v3143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:49.505+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:15:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:49.507+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:15:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:49.507+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:15:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:49.507+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:15:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:15:49.507+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:50 smithi115 ceph-mon[35364]: pgmap v3144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4587.721596 secs 2024-07-26T09:15:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:50 smithi181 ceph-mon[35869]: pgmap v3144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4587.721596 secs 2024-07-26T09:15:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:50 smithi097 ceph-mon[32231]: pgmap v3144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4587.721596 secs 2024-07-26T09:15:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:52 smithi115 ceph-mon[35364]: pgmap v3145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:52 smithi181 ceph-mon[35869]: pgmap v3145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:52 smithi097 ceph-mon[32231]: pgmap v3145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:54 smithi115 ceph-mon[35364]: pgmap v3146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:54 smithi181 ceph-mon[35869]: pgmap v3146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:54 smithi097 ceph-mon[32231]: pgmap v3146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4592.721736 secs 2024-07-26T09:15:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4592.721736 secs 2024-07-26T09:15:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4592.721736 secs 2024-07-26T09:15:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:56 smithi115 ceph-mon[35364]: pgmap v3147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:56 smithi181 ceph-mon[35869]: pgmap v3147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:56 smithi097 ceph-mon[32231]: pgmap v3147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:15:58 smithi115 ceph-mon[35364]: pgmap v3148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:58.976 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:15:58 smithi097 ceph-mon[32231]: pgmap v3148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:15:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:15:58 smithi181 ceph-mon[35869]: pgmap v3148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:16:00.968 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:00 smithi097 ceph-mon[32231]: pgmap v3149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:00.968 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4597.721877 secs 2024-07-26T09:16:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:00 smithi115 ceph-mon[35364]: pgmap v3149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4597.721877 secs 2024-07-26T09:16:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:00 smithi181 ceph-mon[35869]: pgmap v3149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4597.721877 secs 2024-07-26T09:16:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:02 smithi115 ceph-mon[35364]: pgmap v3150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:03.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:03.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:03.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:03.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:03.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:02 smithi115 ceph-mon[35364]: pgmap v3151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:16:03.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:02 smithi115 ceph-mon[35364]: pgmap v3152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:03.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:03.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:02 smithi181 ceph-mon[35869]: pgmap v3150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:03.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:03.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:03.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:03.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:03.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:02 smithi181 ceph-mon[35869]: pgmap v3151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:16:03.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:02 smithi181 ceph-mon[35869]: pgmap v3152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:03.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:02 smithi097 ceph-mon[32231]: pgmap v3150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:03.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:03.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:03.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:03.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:02 smithi097 ceph-mon[32231]: pgmap v3151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:16:03.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:02 smithi097 ceph-mon[32231]: pgmap v3152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:03.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:16:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:04.512+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:16:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:04.514+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:16:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:04.514+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:16:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:04.514+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:16:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:04.514+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:04 smithi097 ceph-mon[32231]: pgmap v3153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:04 smithi115 ceph-mon[35364]: pgmap v3153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:04 smithi181 ceph-mon[35869]: pgmap v3153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4602.722011 secs 2024-07-26T09:16:06.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4602.722011 secs 2024-07-26T09:16:06.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4602.722011 secs 2024-07-26T09:16:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:07 smithi115 ceph-mon[35364]: pgmap v3154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T09:16:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:07 smithi181 ceph-mon[35869]: pgmap v3154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T09:16:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:07 smithi097 ceph-mon[32231]: pgmap v3154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T09:16:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:07.684+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:16:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:07.684+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:16:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:07.685+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T09:16:07.688 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:07.685+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:16:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:09 smithi115 ceph-mon[35364]: pgmap v3155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:09 smithi181 ceph-mon[35869]: pgmap v3155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:09 smithi097 ceph-mon[32231]: pgmap v3155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4607.722143 secs 2024-07-26T09:16:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:11 smithi115 ceph-mon[35364]: pgmap v3156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T09:16:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4607.722143 secs 2024-07-26T09:16:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:11 smithi181 ceph-mon[35869]: pgmap v3156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T09:16:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4607.722143 secs 2024-07-26T09:16:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:11 smithi097 ceph-mon[32231]: pgmap v3156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T09:16:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:13 smithi115 ceph-mon[35364]: pgmap v3157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:16:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:13 smithi181 ceph-mon[35869]: pgmap v3157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:16:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:13 smithi097 ceph-mon[32231]: pgmap v3157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:16:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:15 smithi115 ceph-mon[35364]: pgmap v3158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:16:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:15 smithi181 ceph-mon[35869]: pgmap v3158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:16:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:15 smithi097 ceph-mon[32231]: pgmap v3158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:16:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4612.722287 secs 2024-07-26T09:16:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4612.722287 secs 2024-07-26T09:16:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4612.722287 secs 2024-07-26T09:16:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:17 smithi115 ceph-mon[35364]: pgmap v3159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:17 smithi181 ceph-mon[35869]: pgmap v3159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:17 smithi097 ceph-mon[32231]: pgmap v3159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:19 smithi115 ceph-mon[35364]: pgmap v3160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:16:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:19 smithi181 ceph-mon[35869]: pgmap v3160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:16:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:19 smithi097 ceph-mon[32231]: pgmap v3160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:16:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:19.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:16:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:19.524+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:16:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:19.524+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:16:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:19.524+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:16:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:19.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4617.722426 secs 2024-07-26T09:16:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:21 smithi115 ceph-mon[35364]: pgmap v3161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4617.722426 secs 2024-07-26T09:16:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:21 smithi181 ceph-mon[35869]: pgmap v3161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4617.722426 secs 2024-07-26T09:16:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:21 smithi097 ceph-mon[32231]: pgmap v3161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:22 smithi181 ceph-mon[35869]: pgmap v3162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:22 smithi097 ceph-mon[32231]: pgmap v3162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:22 smithi115 ceph-mon[35364]: pgmap v3162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:24 smithi097 ceph-mon[32231]: pgmap v3163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:16:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:24 smithi115 ceph-mon[35364]: pgmap v3163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:16:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:24 smithi181 ceph-mon[35869]: pgmap v3163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:16:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4622.722600 secs 2024-07-26T09:16:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4622.722600 secs 2024-07-26T09:16:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4622.722600 secs 2024-07-26T09:16:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:27 smithi115 ceph-mon[35364]: pgmap v3164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:27 smithi181 ceph-mon[35869]: pgmap v3164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:27 smithi097 ceph-mon[32231]: pgmap v3164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:29 smithi115 ceph-mon[35364]: pgmap v3165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:16:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:29 smithi181 ceph-mon[35869]: pgmap v3165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:16:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:29 smithi097 ceph-mon[32231]: pgmap v3165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:16:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4627.722739 secs 2024-07-26T09:16:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:31 smithi115 ceph-mon[35364]: pgmap v3166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4627.722739 secs 2024-07-26T09:16:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:31 smithi181 ceph-mon[35869]: pgmap v3166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4627.722739 secs 2024-07-26T09:16:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:31 smithi097 ceph-mon[32231]: pgmap v3166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:33 smithi115 ceph-mon[35364]: pgmap v3167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:33 smithi181 ceph-mon[35869]: pgmap v3167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:33 smithi097 ceph-mon[32231]: pgmap v3167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:34.530+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:34.531+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:34.531+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:34.531+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:16:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:34.531+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:35 smithi115 ceph-mon[35364]: pgmap v3168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:35 smithi181 ceph-mon[35869]: pgmap v3168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:35 smithi097 ceph-mon[32231]: pgmap v3168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:36.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4632.722881 secs 2024-07-26T09:16:36.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4632.722881 secs 2024-07-26T09:16:36.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4632.722881 secs 2024-07-26T09:16:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:37 smithi115 ceph-mon[35364]: pgmap v3169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:37 smithi181 ceph-mon[35869]: pgmap v3169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:37 smithi097 ceph-mon[32231]: pgmap v3169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:39 smithi115 ceph-mon[35364]: pgmap v3170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:16:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:39 smithi181 ceph-mon[35869]: pgmap v3170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:16:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:39 smithi097 ceph-mon[32231]: pgmap v3170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:16:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4637.723022 secs 2024-07-26T09:16:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:41 smithi115 ceph-mon[35364]: pgmap v3171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:16:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4637.723022 secs 2024-07-26T09:16:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:41 smithi181 ceph-mon[35869]: pgmap v3171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:16:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4637.723022 secs 2024-07-26T09:16:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:41 smithi097 ceph-mon[32231]: pgmap v3171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:16:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:43 smithi115 ceph-mon[35364]: pgmap v3172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:16:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:43 smithi181 ceph-mon[35869]: pgmap v3172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:16:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:43 smithi097 ceph-mon[32231]: pgmap v3172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:16:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:45 smithi115 ceph-mon[35364]: pgmap v3173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:45 smithi181 ceph-mon[35869]: pgmap v3173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:45 smithi097 ceph-mon[32231]: pgmap v3173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4642.723159 secs 2024-07-26T09:16:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4642.723159 secs 2024-07-26T09:16:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4642.723159 secs 2024-07-26T09:16:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:47 smithi115 ceph-mon[35364]: pgmap v3174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:47 smithi181 ceph-mon[35869]: pgmap v3174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:47 smithi097 ceph-mon[32231]: pgmap v3174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:49 smithi115 ceph-mon[35364]: pgmap v3175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:16:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:49 smithi181 ceph-mon[35869]: pgmap v3175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:16:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:49 smithi097 ceph-mon[32231]: pgmap v3175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:16:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:49.541+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:16:49.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:49.543+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:16:49.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:49.543+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:16:49.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:49.543+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:16:49.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:16:49.543+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4647.723289 secs 2024-07-26T09:16:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:51 smithi115 ceph-mon[35364]: pgmap v3176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:16:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4647.723289 secs 2024-07-26T09:16:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:51 smithi181 ceph-mon[35869]: pgmap v3176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:16:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4647.723289 secs 2024-07-26T09:16:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:51 smithi097 ceph-mon[32231]: pgmap v3176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:16:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:53 smithi115 ceph-mon[35364]: pgmap v3177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:16:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:53 smithi181 ceph-mon[35869]: pgmap v3177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:16:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:53 smithi097 ceph-mon[32231]: pgmap v3177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:16:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:55 smithi115 ceph-mon[35364]: pgmap v3178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:55 smithi181 ceph-mon[35869]: pgmap v3178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:55 smithi097 ceph-mon[32231]: pgmap v3178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4652.723423 secs 2024-07-26T09:16:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4652.723423 secs 2024-07-26T09:16:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4652.723423 secs 2024-07-26T09:16:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:57 smithi115 ceph-mon[35364]: pgmap v3179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:57 smithi181 ceph-mon[35869]: pgmap v3179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:57 smithi097 ceph-mon[32231]: pgmap v3179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:16:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:16:59 smithi115 ceph-mon[35364]: pgmap v3180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:16:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:16:59 smithi181 ceph-mon[35869]: pgmap v3180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:16:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:16:59 smithi097 ceph-mon[32231]: pgmap v3180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:17:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4657.723558 secs 2024-07-26T09:17:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:01 smithi115 ceph-mon[35364]: pgmap v3181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4657.723558 secs 2024-07-26T09:17:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:01 smithi181 ceph-mon[35869]: pgmap v3181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4657.723558 secs 2024-07-26T09:17:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:01 smithi097 ceph-mon[32231]: pgmap v3181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:03 smithi115 ceph-mon[35364]: pgmap v3182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:03 smithi181 ceph-mon[35869]: pgmap v3182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:03 smithi097 ceph-mon[32231]: pgmap v3182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:04.548+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:17:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:04.549+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:17:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:04.549+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:17:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:04.549+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:17:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:04.550+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:05 smithi115 ceph-mon[35364]: pgmap v3183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:05 smithi181 ceph-mon[35869]: pgmap v3183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:05 smithi097 ceph-mon[32231]: pgmap v3183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4662.723706 secs 2024-07-26T09:17:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4662.723706 secs 2024-07-26T09:17:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4662.723706 secs 2024-07-26T09:17:07.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:07 smithi115 ceph-mon[35364]: pgmap v3184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:07.473 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:07 smithi181 ceph-mon[35869]: pgmap v3184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:07 smithi097 ceph-mon[32231]: pgmap v3184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: pgmap v3185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: pgmap v3186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: pgmap v3187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:09.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: pgmap v3185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: pgmap v3186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: pgmap v3187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: pgmap v3185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:09.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:09.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:09.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: pgmap v3186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:17:09.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: pgmap v3187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:09.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:17:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4667.723862 secs 2024-07-26T09:17:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:11 smithi115 ceph-mon[35364]: pgmap v3188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:17:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4667.723862 secs 2024-07-26T09:17:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:11 smithi181 ceph-mon[35869]: pgmap v3188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:17:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4667.723862 secs 2024-07-26T09:17:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:11 smithi097 ceph-mon[32231]: pgmap v3188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:17:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:13 smithi115 ceph-mon[35364]: pgmap v3189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T09:17:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:13 smithi181 ceph-mon[35869]: pgmap v3189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T09:17:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:13 smithi097 ceph-mon[32231]: pgmap v3189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T09:17:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:15 smithi115 ceph-mon[35364]: pgmap v3190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:17:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:15 smithi181 ceph-mon[35869]: pgmap v3190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:17:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:15 smithi097 ceph-mon[32231]: pgmap v3190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:17:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4672.724035 secs 2024-07-26T09:17:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4672.724035 secs 2024-07-26T09:17:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4672.724035 secs 2024-07-26T09:17:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:17 smithi115 ceph-mon[35364]: pgmap v3191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:17:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:17 smithi181 ceph-mon[35869]: pgmap v3191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:17:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:17 smithi097 ceph-mon[32231]: pgmap v3191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:17:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:19 smithi115 ceph-mon[35364]: pgmap v3192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:17:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:19 smithi181 ceph-mon[35869]: pgmap v3192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:17:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:19 smithi097 ceph-mon[32231]: pgmap v3192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:17:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:19.559+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:17:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:19.560+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:17:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:19.560+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:17:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:19.560+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:17:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:19.560+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4677.724162 secs 2024-07-26T09:17:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:21 smithi115 ceph-mon[35364]: pgmap v3193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4677.724162 secs 2024-07-26T09:17:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:21 smithi181 ceph-mon[35869]: pgmap v3193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4677.724162 secs 2024-07-26T09:17:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:21 smithi097 ceph-mon[32231]: pgmap v3193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:22 smithi097 ceph-mon[32231]: pgmap v3194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:22 smithi115 ceph-mon[35364]: pgmap v3194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:22 smithi181 ceph-mon[35869]: pgmap v3194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:25 smithi115 ceph-mon[35364]: pgmap v3195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:25 smithi181 ceph-mon[35869]: pgmap v3195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:25 smithi097 ceph-mon[32231]: pgmap v3195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4682.724304 secs 2024-07-26T09:17:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4682.724304 secs 2024-07-26T09:17:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4682.724304 secs 2024-07-26T09:17:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:27 smithi115 ceph-mon[35364]: pgmap v3196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:27 smithi181 ceph-mon[35869]: pgmap v3196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:27 smithi097 ceph-mon[32231]: pgmap v3196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:29 smithi115 ceph-mon[35364]: pgmap v3197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:29 smithi181 ceph-mon[35869]: pgmap v3197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:29 smithi097 ceph-mon[32231]: pgmap v3197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4687.724438 secs 2024-07-26T09:17:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:31 smithi115 ceph-mon[35364]: pgmap v3198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4687.724438 secs 2024-07-26T09:17:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:31 smithi181 ceph-mon[35869]: pgmap v3198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4687.724438 secs 2024-07-26T09:17:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:31 smithi097 ceph-mon[32231]: pgmap v3198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:33 smithi115 ceph-mon[35364]: pgmap v3199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:33 smithi181 ceph-mon[35869]: pgmap v3199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:33 smithi097 ceph-mon[32231]: pgmap v3199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:34.565+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:17:34.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:34.567+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:17:34.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:34.567+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:17:34.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:34.567+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:17:34.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:34.567+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:34 smithi115 ceph-mon[35364]: pgmap v3200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:34 smithi181 ceph-mon[35869]: pgmap v3200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:34 smithi097 ceph-mon[32231]: pgmap v3200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4692.724578 secs 2024-07-26T09:17:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4692.724578 secs 2024-07-26T09:17:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4692.724578 secs 2024-07-26T09:17:36.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:36 smithi115 ceph-mon[35364]: pgmap v3201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:36.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:36 smithi181 ceph-mon[35869]: pgmap v3201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:36.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:36 smithi097 ceph-mon[32231]: pgmap v3201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:39 smithi115 ceph-mon[35364]: pgmap v3202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:39 smithi181 ceph-mon[35869]: pgmap v3202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:39 smithi097 ceph-mon[32231]: pgmap v3202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4697.724720 secs 2024-07-26T09:17:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:41 smithi115 ceph-mon[35364]: pgmap v3203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4697.724720 secs 2024-07-26T09:17:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:41 smithi181 ceph-mon[35869]: pgmap v3203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4697.724720 secs 2024-07-26T09:17:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:41 smithi097 ceph-mon[32231]: pgmap v3203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:42 smithi115 ceph-mon[35364]: pgmap v3204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:42 smithi181 ceph-mon[35869]: pgmap v3204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:42 smithi097 ceph-mon[32231]: pgmap v3204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:45 smithi115 ceph-mon[35364]: pgmap v3205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:45 smithi181 ceph-mon[35869]: pgmap v3205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:45 smithi097 ceph-mon[32231]: pgmap v3205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4702.724853 secs 2024-07-26T09:17:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4702.724853 secs 2024-07-26T09:17:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4702.724853 secs 2024-07-26T09:17:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:47 smithi115 ceph-mon[35364]: pgmap v3206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:47 smithi181 ceph-mon[35869]: pgmap v3206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:47 smithi097 ceph-mon[32231]: pgmap v3206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:49 smithi115 ceph-mon[35364]: pgmap v3207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:49 smithi181 ceph-mon[35869]: pgmap v3207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:49 smithi097 ceph-mon[32231]: pgmap v3207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:49.572 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:49.570+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:17:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:49.571+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:17:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:49.571+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:17:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:49.571+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:17:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:17:49.571+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4707.724997 secs 2024-07-26T09:17:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:51 smithi181 ceph-mon[35869]: pgmap v3208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4707.724997 secs 2024-07-26T09:17:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:51 smithi097 ceph-mon[32231]: pgmap v3208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4707.724997 secs 2024-07-26T09:17:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:51 smithi115 ceph-mon[35364]: pgmap v3208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:52 smithi115 ceph-mon[35364]: pgmap v3209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:52 smithi181 ceph-mon[35869]: pgmap v3209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:52 smithi097 ceph-mon[32231]: pgmap v3209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:55 smithi115 ceph-mon[35364]: pgmap v3210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:55 smithi181 ceph-mon[35869]: pgmap v3210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:55 smithi097 ceph-mon[32231]: pgmap v3210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:17:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4712.725133 secs 2024-07-26T09:17:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4712.725133 secs 2024-07-26T09:17:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4712.725133 secs 2024-07-26T09:17:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:57 smithi115 ceph-mon[35364]: pgmap v3211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:57 smithi181 ceph-mon[35869]: pgmap v3211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:57 smithi097 ceph-mon[32231]: pgmap v3211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:17:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:17:59 smithi181 ceph-mon[35869]: pgmap v3212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:59.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:17:59 smithi097 ceph-mon[32231]: pgmap v3212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:17:59 smithi115 ceph-mon[35364]: pgmap v3212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4717.725292 secs 2024-07-26T09:18:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:01 smithi097 ceph-mon[32231]: pgmap v3213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4717.725292 secs 2024-07-26T09:18:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:01 smithi115 ceph-mon[35364]: pgmap v3213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4717.725292 secs 2024-07-26T09:18:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:01 smithi181 ceph-mon[35869]: pgmap v3213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:02 smithi097 ceph-mon[32231]: pgmap v3214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:02 smithi115 ceph-mon[35364]: pgmap v3214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:02 smithi181 ceph-mon[35869]: pgmap v3214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:04.575+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:18:04.579 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:04.576+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:18:04.579 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:04.576+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:18:04.579 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:04.576+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:18:04.579 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:04.576+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:05 smithi115 ceph-mon[35364]: pgmap v3215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:18:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:05 smithi181 ceph-mon[35869]: pgmap v3215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:18:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:05 smithi097 ceph-mon[32231]: pgmap v3215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:18:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4722.725440 secs 2024-07-26T09:18:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4722.725440 secs 2024-07-26T09:18:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4722.725440 secs 2024-07-26T09:18:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:07 smithi181 ceph-mon[35869]: pgmap v3216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:18:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:07 smithi097 ceph-mon[32231]: pgmap v3216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:18:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:07 smithi115 ceph-mon[35364]: pgmap v3216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:18:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:08 smithi097 ceph-mon[32231]: pgmap v3217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:08 smithi115 ceph-mon[35364]: pgmap v3217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:08 smithi181 ceph-mon[35869]: pgmap v3217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:10 smithi115 ceph-mon[35364]: pgmap v3218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:18:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:10 smithi115 ceph-mon[35364]: pgmap v3219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:18:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:10.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:10.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:10 smithi181 ceph-mon[35869]: pgmap v3218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:18:10.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:10 smithi181 ceph-mon[35869]: pgmap v3219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:10.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:18:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:10 smithi097 ceph-mon[32231]: pgmap v3218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:18:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:10 smithi097 ceph-mon[32231]: pgmap v3219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:18:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4727.725532 secs 2024-07-26T09:18:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4727.725532 secs 2024-07-26T09:18:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4727.725532 secs 2024-07-26T09:18:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:12 smithi115 ceph-mon[35364]: pgmap v3220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:12 smithi181 ceph-mon[35869]: pgmap v3220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:12 smithi097 ceph-mon[32231]: pgmap v3220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:14 smithi115 ceph-mon[35364]: pgmap v3221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:18:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:14 smithi181 ceph-mon[35869]: pgmap v3221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:18:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:14 smithi097 ceph-mon[32231]: pgmap v3221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:18:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4732.725681 secs 2024-07-26T09:18:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4732.725681 secs 2024-07-26T09:18:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4732.725681 secs 2024-07-26T09:18:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:16 smithi115 ceph-mon[35364]: pgmap v3222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:16 smithi181 ceph-mon[35869]: pgmap v3222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:16 smithi097 ceph-mon[32231]: pgmap v3222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:18 smithi115 ceph-mon[35364]: pgmap v3223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:18 smithi181 ceph-mon[35869]: pgmap v3223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:18 smithi097 ceph-mon[32231]: pgmap v3223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:19.587+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:18:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:19.588+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:18:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:19.588+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:18:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:19.588+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:18:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:19.588+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:20 smithi115 ceph-mon[35364]: pgmap v3224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:18:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4737.725807 secs 2024-07-26T09:18:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:20 smithi181 ceph-mon[35869]: pgmap v3224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:18:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4737.725807 secs 2024-07-26T09:18:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:20 smithi097 ceph-mon[32231]: pgmap v3224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:18:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4737.725807 secs 2024-07-26T09:18:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:22 smithi115 ceph-mon[35364]: pgmap v3225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:22 smithi181 ceph-mon[35869]: pgmap v3225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:22 smithi097 ceph-mon[32231]: pgmap v3225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:24 smithi115 ceph-mon[35364]: pgmap v3226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:24 smithi181 ceph-mon[35869]: pgmap v3226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:24 smithi097 ceph-mon[32231]: pgmap v3226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4742.725950 secs 2024-07-26T09:18:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4742.725950 secs 2024-07-26T09:18:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4742.725950 secs 2024-07-26T09:18:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:26 smithi115 ceph-mon[35364]: pgmap v3227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:26 smithi181 ceph-mon[35869]: pgmap v3227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:26 smithi097 ceph-mon[32231]: pgmap v3227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:28 smithi115 ceph-mon[35364]: pgmap v3228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:28 smithi181 ceph-mon[35869]: pgmap v3228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:28 smithi097 ceph-mon[32231]: pgmap v3228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:30 smithi115 ceph-mon[35364]: pgmap v3229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4747.726096 secs 2024-07-26T09:18:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:30 smithi181 ceph-mon[35869]: pgmap v3229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4747.726096 secs 2024-07-26T09:18:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:30 smithi097 ceph-mon[32231]: pgmap v3229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4747.726096 secs 2024-07-26T09:18:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:32 smithi115 ceph-mon[35364]: pgmap v3230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:32 smithi181 ceph-mon[35869]: pgmap v3230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:32 smithi097 ceph-mon[32231]: pgmap v3230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:34.594+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:34.595+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:34.595+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:34.595+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:34.595+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:34 smithi115 ceph-mon[35364]: pgmap v3231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:34 smithi181 ceph-mon[35869]: pgmap v3231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:34 smithi097 ceph-mon[32231]: pgmap v3231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4752.726273 secs 2024-07-26T09:18:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4752.726273 secs 2024-07-26T09:18:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4752.726273 secs 2024-07-26T09:18:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:36 smithi115 ceph-mon[35364]: pgmap v3232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:36 smithi181 ceph-mon[35869]: pgmap v3232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:36 smithi097 ceph-mon[32231]: pgmap v3232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:38 smithi115 ceph-mon[35364]: pgmap v3233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:38 smithi181 ceph-mon[35869]: pgmap v3233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:38 smithi097 ceph-mon[32231]: pgmap v3233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:40 smithi115 ceph-mon[35364]: pgmap v3234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:18:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4757.726405 secs 2024-07-26T09:18:40.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:40 smithi181 ceph-mon[35869]: pgmap v3234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:18:40.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4757.726405 secs 2024-07-26T09:18:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:40 smithi097 ceph-mon[32231]: pgmap v3234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:18:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4757.726405 secs 2024-07-26T09:18:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:42 smithi115 ceph-mon[35364]: pgmap v3235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:18:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:42 smithi181 ceph-mon[35869]: pgmap v3235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:18:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:42 smithi097 ceph-mon[32231]: pgmap v3235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:18:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:44 smithi115 ceph-mon[35364]: pgmap v3236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:18:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:44 smithi181 ceph-mon[35869]: pgmap v3236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:18:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:44 smithi097 ceph-mon[32231]: pgmap v3236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:18:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4762.726539 secs 2024-07-26T09:18:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4762.726539 secs 2024-07-26T09:18:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4762.726539 secs 2024-07-26T09:18:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:46 smithi181 ceph-mon[35869]: pgmap v3237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:46 smithi097 ceph-mon[32231]: pgmap v3237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:46 smithi115 ceph-mon[35364]: pgmap v3237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:18:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:48 smithi097 ceph-mon[32231]: pgmap v3238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:48 smithi115 ceph-mon[35364]: pgmap v3238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:48 smithi181 ceph-mon[35869]: pgmap v3238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:49.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:49.605+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:18:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:49.607+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:18:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:49.607+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:18:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:49.607+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:18:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:18:49.607+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:50 smithi097 ceph-mon[32231]: pgmap v3239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4767.726616 secs 2024-07-26T09:18:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:50 smithi115 ceph-mon[35364]: pgmap v3239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4767.726616 secs 2024-07-26T09:18:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:50 smithi181 ceph-mon[35869]: pgmap v3239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4767.726616 secs 2024-07-26T09:18:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:52 smithi097 ceph-mon[32231]: pgmap v3240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:52 smithi115 ceph-mon[35364]: pgmap v3240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:52 smithi181 ceph-mon[35869]: pgmap v3240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:54 smithi115 ceph-mon[35364]: pgmap v3241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:54 smithi181 ceph-mon[35869]: pgmap v3241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:54 smithi097 ceph-mon[32231]: pgmap v3241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4772.726757 secs 2024-07-26T09:18:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4772.726757 secs 2024-07-26T09:18:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4772.726757 secs 2024-07-26T09:18:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:56 smithi115 ceph-mon[35364]: pgmap v3242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:56 smithi181 ceph-mon[35869]: pgmap v3242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:56 smithi097 ceph-mon[32231]: pgmap v3242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:18:58 smithi115 ceph-mon[35364]: pgmap v3243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:18:58 smithi181 ceph-mon[35869]: pgmap v3243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:18:58 smithi097 ceph-mon[32231]: pgmap v3243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:00 smithi115 ceph-mon[35364]: pgmap v3244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4777.726891 secs 2024-07-26T09:19:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:00 smithi181 ceph-mon[35869]: pgmap v3244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4777.726891 secs 2024-07-26T09:19:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:00 smithi097 ceph-mon[32231]: pgmap v3244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4777.726891 secs 2024-07-26T09:19:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:02 smithi115 ceph-mon[35364]: pgmap v3245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:02 smithi181 ceph-mon[35869]: pgmap v3245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:02 smithi097 ceph-mon[32231]: pgmap v3245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:04.611+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:19:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:04.613+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:19:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:04.613+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:19:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:04.613+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:19:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:04.613+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:04 smithi115 ceph-mon[35364]: pgmap v3246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:04 smithi181 ceph-mon[35869]: pgmap v3246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:04 smithi097 ceph-mon[32231]: pgmap v3246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:06.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4782.727029 secs 2024-07-26T09:19:06.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4782.727029 secs 2024-07-26T09:19:06.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4782.727029 secs 2024-07-26T09:19:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:06 smithi115 ceph-mon[35364]: pgmap v3247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:06 smithi181 ceph-mon[35869]: pgmap v3247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:06 smithi097 ceph-mon[32231]: pgmap v3247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:07.717+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:19:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:07.717+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:19:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:07.718+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f070 con 0x7ff3f01061a0 2024-07-26T09:19:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:07.718+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:19:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:08 smithi115 ceph-mon[35364]: pgmap v3248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:09.342 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:08 smithi097 ceph-mon[32231]: pgmap v3248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:09.342 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:08 smithi181 ceph-mon[35869]: pgmap v3248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:10 smithi115 ceph-mon[35364]: pgmap v3249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4787.727174 secs 2024-07-26T09:19:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:10 smithi181 ceph-mon[35869]: pgmap v3249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4787.727174 secs 2024-07-26T09:19:11.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:10 smithi097 ceph-mon[32231]: pgmap v3249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4787.727174 secs 2024-07-26T09:19:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:19:12.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:19:12.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:19:12.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:19:12.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:12.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:12.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: pgmap v3250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:19:12.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: pgmap v3251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:12.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:19:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:19:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:19:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:19:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: pgmap v3250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:19:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: pgmap v3251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:19:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:12.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: pgmap v3250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:19:12.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: pgmap v3251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:12.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:19:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:14 smithi115 ceph-mon[35364]: pgmap v3252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:14 smithi181 ceph-mon[35869]: pgmap v3252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:14 smithi097 ceph-mon[32231]: pgmap v3252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4792.727317 secs 2024-07-26T09:19:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:16 smithi115 ceph-mon[35364]: pgmap v3253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4792.727317 secs 2024-07-26T09:19:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:16 smithi181 ceph-mon[35869]: pgmap v3253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4792.727317 secs 2024-07-26T09:19:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:16 smithi097 ceph-mon[32231]: pgmap v3253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:18 smithi115 ceph-mon[35364]: pgmap v3254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:19:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:18 smithi181 ceph-mon[35869]: pgmap v3254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:19:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:18 smithi097 ceph-mon[32231]: pgmap v3254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:19:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:19.624+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:19:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:19.625+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:19:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:19.625+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:19:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:19.625+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:19:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:19.625+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:20 smithi115 ceph-mon[35364]: pgmap v3255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:20 smithi181 ceph-mon[35869]: pgmap v3255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:20 smithi097 ceph-mon[32231]: pgmap v3255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4797.727410 secs 2024-07-26T09:19:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4797.727410 secs 2024-07-26T09:19:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4797.727410 secs 2024-07-26T09:19:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:22 smithi115 ceph-mon[35364]: pgmap v3256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:19:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:22 smithi181 ceph-mon[35869]: pgmap v3256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:19:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:22 smithi097 ceph-mon[32231]: pgmap v3256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:19:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:24 smithi115 ceph-mon[35364]: pgmap v3257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:24 smithi181 ceph-mon[35869]: pgmap v3257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:24 smithi097 ceph-mon[32231]: pgmap v3257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4802.727504 secs 2024-07-26T09:19:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:26 smithi115 ceph-mon[35364]: pgmap v3258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4802.727504 secs 2024-07-26T09:19:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:26 smithi181 ceph-mon[35869]: pgmap v3258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4802.727504 secs 2024-07-26T09:19:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:26 smithi097 ceph-mon[32231]: pgmap v3258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:28 smithi115 ceph-mon[35364]: pgmap v3259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:28 smithi181 ceph-mon[35869]: pgmap v3259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:28 smithi097 ceph-mon[32231]: pgmap v3259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:30 smithi115 ceph-mon[35364]: pgmap v3260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:30 smithi181 ceph-mon[35869]: pgmap v3260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:30 smithi097 ceph-mon[32231]: pgmap v3260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4807.727674 secs 2024-07-26T09:19:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4807.727674 secs 2024-07-26T09:19:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4807.727674 secs 2024-07-26T09:19:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:32 smithi115 ceph-mon[35364]: pgmap v3261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:32 smithi181 ceph-mon[35869]: pgmap v3261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:32 smithi097 ceph-mon[32231]: pgmap v3261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:34 smithi115 ceph-mon[35364]: pgmap v3262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:34 smithi181 ceph-mon[35869]: pgmap v3262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:34 smithi097 ceph-mon[32231]: pgmap v3262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:34.631+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:19:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:34.632+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:19:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:34.632+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:19:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:34.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:19:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:34.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4812.727812 secs 2024-07-26T09:19:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:36 smithi115 ceph-mon[35364]: pgmap v3263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4812.727812 secs 2024-07-26T09:19:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:36 smithi181 ceph-mon[35869]: pgmap v3263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4812.727812 secs 2024-07-26T09:19:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:36 smithi097 ceph-mon[32231]: pgmap v3263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:38 smithi115 ceph-mon[35364]: pgmap v3264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:38 smithi181 ceph-mon[35869]: pgmap v3264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:38 smithi097 ceph-mon[32231]: pgmap v3264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:40 smithi115 ceph-mon[35364]: pgmap v3265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:40 smithi181 ceph-mon[35869]: pgmap v3265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:40 smithi097 ceph-mon[32231]: pgmap v3265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4817.727961 secs 2024-07-26T09:19:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4817.727961 secs 2024-07-26T09:19:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4817.727961 secs 2024-07-26T09:19:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:42 smithi115 ceph-mon[35364]: pgmap v3266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:42 smithi181 ceph-mon[35869]: pgmap v3266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:42 smithi097 ceph-mon[32231]: pgmap v3266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:44 smithi115 ceph-mon[35364]: pgmap v3267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:44 smithi181 ceph-mon[35869]: pgmap v3267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:44 smithi097 ceph-mon[32231]: pgmap v3267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4822.728134 secs 2024-07-26T09:19:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:46 smithi115 ceph-mon[35364]: pgmap v3268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4822.728134 secs 2024-07-26T09:19:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:46 smithi181 ceph-mon[35869]: pgmap v3268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4822.728134 secs 2024-07-26T09:19:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:46 smithi097 ceph-mon[32231]: pgmap v3268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:48 smithi115 ceph-mon[35364]: pgmap v3269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:48 smithi181 ceph-mon[35869]: pgmap v3269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:48 smithi097 ceph-mon[32231]: pgmap v3269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:49.642+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:19:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:49.644+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:19:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:49.644+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:19:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:49.644+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:19:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:19:49.644+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:50 smithi115 ceph-mon[35364]: pgmap v3270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:50 smithi181 ceph-mon[35869]: pgmap v3270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:50 smithi097 ceph-mon[32231]: pgmap v3270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4827.728279 secs 2024-07-26T09:19:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4827.728279 secs 2024-07-26T09:19:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4827.728279 secs 2024-07-26T09:19:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:52 smithi097 ceph-mon[32231]: pgmap v3271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:52 smithi115 ceph-mon[35364]: pgmap v3271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:52 smithi181 ceph-mon[35869]: pgmap v3271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:54 smithi097 ceph-mon[32231]: pgmap v3272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:54 smithi115 ceph-mon[35364]: pgmap v3272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:54 smithi181 ceph-mon[35869]: pgmap v3272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4832.728415 secs 2024-07-26T09:19:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:56 smithi097 ceph-mon[32231]: pgmap v3273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4832.728415 secs 2024-07-26T09:19:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:56 smithi115 ceph-mon[35364]: pgmap v3273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4832.728415 secs 2024-07-26T09:19:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:56 smithi181 ceph-mon[35869]: pgmap v3273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:58 smithi115 ceph-mon[35364]: pgmap v3274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:58 smithi181 ceph-mon[35869]: pgmap v3274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:58 smithi097 ceph-mon[32231]: pgmap v3274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:19:59 smithi115 ceph-mon[35364]: pgmap v3275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:19:59 smithi181 ceph-mon[35869]: pgmap v3275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:19:59 smithi097 ceph-mon[32231]: pgmap v3275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:20:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:20:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:20:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:20:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4837.728548 secs 2024-07-26T09:20:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:01 smithi115 ceph-mon[35364]: pgmap v3276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4837.728548 secs 2024-07-26T09:20:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:01 smithi181 ceph-mon[35869]: pgmap v3276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4837.728548 secs 2024-07-26T09:20:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:01 smithi097 ceph-mon[32231]: pgmap v3276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:04 smithi115 ceph-mon[35364]: pgmap v3277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:04 smithi181 ceph-mon[35869]: pgmap v3277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:04 smithi097 ceph-mon[32231]: pgmap v3277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:04.648+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:20:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:04.649+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:20:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:04.649+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:20:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:04.649+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:20:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:04.649+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4842.728684 secs 2024-07-26T09:20:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:06 smithi115 ceph-mon[35364]: pgmap v3278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4842.728684 secs 2024-07-26T09:20:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:06 smithi181 ceph-mon[35869]: pgmap v3278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4842.728684 secs 2024-07-26T09:20:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:06 smithi097 ceph-mon[32231]: pgmap v3278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:08 smithi115 ceph-mon[35364]: pgmap v3279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:08 smithi181 ceph-mon[35869]: pgmap v3279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:08 smithi097 ceph-mon[32231]: pgmap v3279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:10 smithi115 ceph-mon[35364]: pgmap v3280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:20:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:10 smithi181 ceph-mon[35869]: pgmap v3280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:20:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:10 smithi097 ceph-mon[32231]: pgmap v3280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:20:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4847.728841 secs 2024-07-26T09:20:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4847.728841 secs 2024-07-26T09:20:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4847.728841 secs 2024-07-26T09:20:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:12 smithi115 ceph-mon[35364]: pgmap v3281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:12 smithi181 ceph-mon[35869]: pgmap v3281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:12 smithi097 ceph-mon[32231]: pgmap v3281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:13.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:13.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:20:13.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:13.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:13.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:20:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:20:13.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:13.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:13.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:20:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:14 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:20:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:14 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:20:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:14 smithi115 ceph-mon[35364]: pgmap v3282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T09:20:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:14 smithi115 ceph-mon[35364]: pgmap v3283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:14 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:20:14.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:14 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:20:14.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:14 smithi181 ceph-mon[35869]: pgmap v3282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T09:20:14.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:14 smithi181 ceph-mon[35869]: pgmap v3283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:14 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:20:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:14 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:20:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:14 smithi097 ceph-mon[32231]: pgmap v3282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T09:20:14.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:14 smithi097 ceph-mon[32231]: pgmap v3283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:16 smithi115 ceph-mon[35364]: pgmap v3284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4852.728981 secs 2024-07-26T09:20:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:16 smithi181 ceph-mon[35869]: pgmap v3284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4852.728981 secs 2024-07-26T09:20:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:16 smithi097 ceph-mon[32231]: pgmap v3284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4852.728981 secs 2024-07-26T09:20:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:17 smithi115 ceph-mon[35364]: pgmap v3285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:17 smithi181 ceph-mon[35869]: pgmap v3285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:17 smithi097 ceph-mon[32231]: pgmap v3285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:19.659+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:20:19.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:19.661+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:20:19.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:19.661+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:20:19.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:19.661+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:20:19.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:19.661+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:19 smithi115 ceph-mon[35364]: pgmap v3286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:19 smithi181 ceph-mon[35869]: pgmap v3286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:19 smithi097 ceph-mon[32231]: pgmap v3286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:20:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4857.729123 secs 2024-07-26T09:20:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:21 smithi115 ceph-mon[35364]: pgmap v3287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4857.729123 secs 2024-07-26T09:20:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:21 smithi181 ceph-mon[35869]: pgmap v3287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4857.729123 secs 2024-07-26T09:20:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:21 smithi097 ceph-mon[32231]: pgmap v3287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:23 smithi115 ceph-mon[35364]: pgmap v3288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:20:23.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:23 smithi181 ceph-mon[35869]: pgmap v3288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:20:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:23 smithi097 ceph-mon[32231]: pgmap v3288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:20:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:25 smithi115 ceph-mon[35364]: pgmap v3289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:25 smithi181 ceph-mon[35869]: pgmap v3289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:25 smithi097 ceph-mon[32231]: pgmap v3289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4862.729291 secs 2024-07-26T09:20:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4862.729291 secs 2024-07-26T09:20:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4862.729291 secs 2024-07-26T09:20:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:27 smithi115 ceph-mon[35364]: pgmap v3290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:27 smithi181 ceph-mon[35869]: pgmap v3290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:27 smithi097 ceph-mon[32231]: pgmap v3290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:29 smithi115 ceph-mon[35364]: pgmap v3291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:29 smithi181 ceph-mon[35869]: pgmap v3291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:29 smithi097 ceph-mon[32231]: pgmap v3291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4867.729437 secs 2024-07-26T09:20:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4867.729437 secs 2024-07-26T09:20:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4867.729437 secs 2024-07-26T09:20:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:31 smithi115 ceph-mon[35364]: pgmap v3292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:31 smithi181 ceph-mon[35869]: pgmap v3292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:31 smithi097 ceph-mon[32231]: pgmap v3292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:33 smithi115 ceph-mon[35364]: pgmap v3293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:20:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:33 smithi181 ceph-mon[35869]: pgmap v3293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:20:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:33 smithi097 ceph-mon[32231]: pgmap v3293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:20:34.668 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:34.667+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:20:34.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:34.667+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:20:34.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:34.667+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:20:34.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:34.667+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:20:34.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:34.668+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:35 smithi115 ceph-mon[35364]: pgmap v3294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:20:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4872.729577 secs 2024-07-26T09:20:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:35 smithi181 ceph-mon[35869]: pgmap v3294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:20:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4872.729577 secs 2024-07-26T09:20:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:35 smithi097 ceph-mon[32231]: pgmap v3294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:20:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4872.729577 secs 2024-07-26T09:20:37.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:37.733+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:20:37.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:37.733+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:20:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:37.734+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:20:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:37.734+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T09:20:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:37 smithi115 ceph-mon[35364]: pgmap v3295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:37 smithi181 ceph-mon[35869]: pgmap v3295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:37 smithi097 ceph-mon[32231]: pgmap v3295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:39 smithi115 ceph-mon[35364]: pgmap v3296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:39 smithi181 ceph-mon[35869]: pgmap v3296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:39 smithi097 ceph-mon[32231]: pgmap v3296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4877.729716 secs 2024-07-26T09:20:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4877.729716 secs 2024-07-26T09:20:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4877.729716 secs 2024-07-26T09:20:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:41 smithi115 ceph-mon[35364]: pgmap v3297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:41 smithi181 ceph-mon[35869]: pgmap v3297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:41 smithi097 ceph-mon[32231]: pgmap v3297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:43 smithi115 ceph-mon[35364]: pgmap v3298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:43 smithi181 ceph-mon[35869]: pgmap v3298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:43 smithi097 ceph-mon[32231]: pgmap v3298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:45 smithi115 ceph-mon[35364]: pgmap v3299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4882.730535 secs 2024-07-26T09:20:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:45 smithi181 ceph-mon[35869]: pgmap v3299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4882.730535 secs 2024-07-26T09:20:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:45 smithi097 ceph-mon[32231]: pgmap v3299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4882.730535 secs 2024-07-26T09:20:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:47 smithi115 ceph-mon[35364]: pgmap v3300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:47 smithi181 ceph-mon[35869]: pgmap v3300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:47 smithi097 ceph-mon[32231]: pgmap v3300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:49.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:49.678+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:20:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:49.679+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:20:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:49.679+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:20:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:49.679+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:20:49.679+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:49 smithi115 ceph-mon[35364]: pgmap v3301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:49 smithi181 ceph-mon[35869]: pgmap v3301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:49 smithi097 ceph-mon[32231]: pgmap v3301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4887.730613 secs 2024-07-26T09:20:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4887.730613 secs 2024-07-26T09:20:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4887.730613 secs 2024-07-26T09:20:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:51 smithi115 ceph-mon[35364]: pgmap v3302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:51 smithi181 ceph-mon[35869]: pgmap v3302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:51 smithi097 ceph-mon[32231]: pgmap v3302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:53 smithi115 ceph-mon[35364]: pgmap v3303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:53 smithi181 ceph-mon[35869]: pgmap v3303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:53 smithi097 ceph-mon[32231]: pgmap v3303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:55 smithi115 ceph-mon[35364]: pgmap v3304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4892.730696 secs 2024-07-26T09:20:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:55 smithi181 ceph-mon[35869]: pgmap v3304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4892.730696 secs 2024-07-26T09:20:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:55 smithi097 ceph-mon[32231]: pgmap v3304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4892.730696 secs 2024-07-26T09:20:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:57 smithi115 ceph-mon[35364]: pgmap v3305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:57 smithi181 ceph-mon[35869]: pgmap v3305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:57 smithi097 ceph-mon[32231]: pgmap v3305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:20:59 smithi115 ceph-mon[35364]: pgmap v3306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:20:59 smithi181 ceph-mon[35869]: pgmap v3306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:20:59 smithi097 ceph-mon[32231]: pgmap v3306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4897.730776 secs 2024-07-26T09:21:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4897.730776 secs 2024-07-26T09:21:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4897.730776 secs 2024-07-26T09:21:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:01 smithi115 ceph-mon[35364]: pgmap v3307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:01 smithi181 ceph-mon[35869]: pgmap v3307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:01 smithi097 ceph-mon[32231]: pgmap v3307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:03 smithi181 ceph-mon[35869]: pgmap v3308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:03 smithi097 ceph-mon[32231]: pgmap v3308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:03 smithi115 ceph-mon[35364]: pgmap v3308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:04.684 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:04.681+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:21:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:04.683+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:21:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:04.683+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:21:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:04.683+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:21:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:04.683+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:05 smithi097 ceph-mon[32231]: pgmap v3309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4902.730871 secs 2024-07-26T09:21:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:05 smithi115 ceph-mon[35364]: pgmap v3309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4902.730871 secs 2024-07-26T09:21:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:05 smithi181 ceph-mon[35869]: pgmap v3309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4902.730871 secs 2024-07-26T09:21:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:07 smithi097 ceph-mon[32231]: pgmap v3310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:08.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:07 smithi115 ceph-mon[35364]: pgmap v3310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:08.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:07 smithi181 ceph-mon[35869]: pgmap v3310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:09 smithi097 ceph-mon[32231]: pgmap v3311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:09 smithi115 ceph-mon[35364]: pgmap v3311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:09 smithi181 ceph-mon[35869]: pgmap v3311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4907.730947 secs 2024-07-26T09:21:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4907.730947 secs 2024-07-26T09:21:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4907.730947 secs 2024-07-26T09:21:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:11 smithi115 ceph-mon[35364]: pgmap v3312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:11 smithi181 ceph-mon[35869]: pgmap v3312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:11 smithi097 ceph-mon[32231]: pgmap v3312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:13 smithi115 ceph-mon[35364]: pgmap v3313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:13 smithi181 ceph-mon[35869]: pgmap v3313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:13 smithi097 ceph-mon[32231]: pgmap v3313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:21:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:15 smithi115 ceph-mon[35364]: pgmap v3314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:21:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:15 smithi115 ceph-mon[35364]: pgmap v3315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:21:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4912.731022 secs 2024-07-26T09:21:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:15 smithi181 ceph-mon[35869]: pgmap v3314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:21:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:15 smithi181 ceph-mon[35869]: pgmap v3315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:21:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4912.731022 secs 2024-07-26T09:21:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:15 smithi097 ceph-mon[32231]: pgmap v3314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:21:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:15 smithi097 ceph-mon[32231]: pgmap v3315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:21:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4912.731022 secs 2024-07-26T09:21:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:17 smithi115 ceph-mon[35364]: pgmap v3316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:21:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:17 smithi181 ceph-mon[35869]: pgmap v3316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:21:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:17 smithi097 ceph-mon[32231]: pgmap v3316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:21:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:19.688+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:21:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:19.690+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:21:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:19.690+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:21:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:19.690+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:21:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:19.690+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:19 smithi115 ceph-mon[35364]: pgmap v3317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:19 smithi181 ceph-mon[35869]: pgmap v3317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:19 smithi097 ceph-mon[32231]: pgmap v3317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4917.731134 secs 2024-07-26T09:21:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4917.731134 secs 2024-07-26T09:21:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4917.731134 secs 2024-07-26T09:21:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:21 smithi115 ceph-mon[35364]: pgmap v3318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:21 smithi181 ceph-mon[35869]: pgmap v3318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:21 smithi097 ceph-mon[32231]: pgmap v3318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:23 smithi115 ceph-mon[35364]: pgmap v3319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:23 smithi181 ceph-mon[35869]: pgmap v3319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:23 smithi097 ceph-mon[32231]: pgmap v3319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:25 smithi115 ceph-mon[35364]: pgmap v3320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:21:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4922.731271 secs 2024-07-26T09:21:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:25 smithi181 ceph-mon[35869]: pgmap v3320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:21:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4922.731271 secs 2024-07-26T09:21:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:25 smithi097 ceph-mon[32231]: pgmap v3320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:21:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4922.731271 secs 2024-07-26T09:21:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:27 smithi115 ceph-mon[35364]: pgmap v3321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:27 smithi181 ceph-mon[35869]: pgmap v3321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:27 smithi097 ceph-mon[32231]: pgmap v3321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:29 smithi115 ceph-mon[35364]: pgmap v3322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:29 smithi181 ceph-mon[35869]: pgmap v3322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:29 smithi097 ceph-mon[32231]: pgmap v3322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4927.731410 secs 2024-07-26T09:21:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4927.731410 secs 2024-07-26T09:21:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4927.731410 secs 2024-07-26T09:21:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:31 smithi115 ceph-mon[35364]: pgmap v3323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:31 smithi181 ceph-mon[35869]: pgmap v3323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:31 smithi097 ceph-mon[32231]: pgmap v3323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:33 smithi115 ceph-mon[35364]: pgmap v3324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:33 smithi181 ceph-mon[35869]: pgmap v3324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:33 smithi097 ceph-mon[32231]: pgmap v3324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:34.695+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:21:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:34.696+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:21:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:34.696+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:21:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:34.696+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:21:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:34.696+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:35 smithi115 ceph-mon[35364]: pgmap v3325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4932.731552 secs 2024-07-26T09:21:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:35 smithi181 ceph-mon[35869]: pgmap v3325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4932.731552 secs 2024-07-26T09:21:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:35 smithi097 ceph-mon[32231]: pgmap v3325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4932.731552 secs 2024-07-26T09:21:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:37 smithi115 ceph-mon[35364]: pgmap v3326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:37 smithi181 ceph-mon[35869]: pgmap v3326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:37 smithi097 ceph-mon[32231]: pgmap v3326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:39 smithi115 ceph-mon[35364]: pgmap v3327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:39 smithi181 ceph-mon[35869]: pgmap v3327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:39 smithi097 ceph-mon[32231]: pgmap v3327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4937.731644 secs 2024-07-26T09:21:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4937.731644 secs 2024-07-26T09:21:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4937.731644 secs 2024-07-26T09:21:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:41 smithi115 ceph-mon[35364]: pgmap v3328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:41 smithi181 ceph-mon[35869]: pgmap v3328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:41 smithi097 ceph-mon[32231]: pgmap v3328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:43 smithi115 ceph-mon[35364]: pgmap v3329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:43 smithi181 ceph-mon[35869]: pgmap v3329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:43 smithi097 ceph-mon[32231]: pgmap v3329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:45 smithi115 ceph-mon[35364]: pgmap v3330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4942.731783 secs 2024-07-26T09:21:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:45 smithi181 ceph-mon[35869]: pgmap v3330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4942.731783 secs 2024-07-26T09:21:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:45 smithi097 ceph-mon[32231]: pgmap v3330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4942.731783 secs 2024-07-26T09:21:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:47 smithi115 ceph-mon[35364]: pgmap v3331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:47 smithi181 ceph-mon[35869]: pgmap v3331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:47 smithi097 ceph-mon[32231]: pgmap v3331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:49.706+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:21:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:49.707+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:21:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:49.707+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:21:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:49.707+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:21:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:21:49.707+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:49 smithi115 ceph-mon[35364]: pgmap v3332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:49 smithi181 ceph-mon[35869]: pgmap v3332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:49 smithi097 ceph-mon[32231]: pgmap v3332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4947.731923 secs 2024-07-26T09:21:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4947.731923 secs 2024-07-26T09:21:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4947.731923 secs 2024-07-26T09:21:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:51 smithi115 ceph-mon[35364]: pgmap v3333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:51 smithi181 ceph-mon[35869]: pgmap v3333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:51 smithi097 ceph-mon[32231]: pgmap v3333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:53 smithi181 ceph-mon[35869]: pgmap v3334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:53 smithi097 ceph-mon[32231]: pgmap v3334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:53 smithi115 ceph-mon[35364]: pgmap v3334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:55 smithi097 ceph-mon[32231]: pgmap v3335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4952.732064 secs 2024-07-26T09:21:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:55 smithi115 ceph-mon[35364]: pgmap v3335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4952.732064 secs 2024-07-26T09:21:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:55 smithi181 ceph-mon[35869]: pgmap v3335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4952.732064 secs 2024-07-26T09:21:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:21:57 smithi097 ceph-mon[32231]: pgmap v3336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:21:57 smithi115 ceph-mon[35364]: pgmap v3336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:21:57 smithi181 ceph-mon[35869]: pgmap v3336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:00 smithi115 ceph-mon[35364]: pgmap v3337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:00 smithi181 ceph-mon[35869]: pgmap v3337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:00 smithi097 ceph-mon[32231]: pgmap v3337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4957.732197 secs 2024-07-26T09:22:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4957.732197 secs 2024-07-26T09:22:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4957.732197 secs 2024-07-26T09:22:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:02 smithi115 ceph-mon[35364]: pgmap v3338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:22:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:02 smithi181 ceph-mon[35869]: pgmap v3338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:22:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:02 smithi097 ceph-mon[32231]: pgmap v3338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:22:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:04 smithi115 ceph-mon[35364]: pgmap v3339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:22:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:04 smithi181 ceph-mon[35869]: pgmap v3339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:22:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:04 smithi097 ceph-mon[32231]: pgmap v3339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:22:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:04.712+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:22:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:04.713+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:22:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:04.713+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:22:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:04.713+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:22:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:04.713+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:06 smithi115 ceph-mon[35364]: pgmap v3340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:22:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4962.732340 secs 2024-07-26T09:22:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:06 smithi181 ceph-mon[35869]: pgmap v3340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:22:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4962.732340 secs 2024-07-26T09:22:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:06 smithi097 ceph-mon[32231]: pgmap v3340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:22:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4962.732340 secs 2024-07-26T09:22:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:07.749+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:22:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:07.749+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:22:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:07.750+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:22:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:07.750+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T09:22:08.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:08 smithi097 ceph-mon[32231]: pgmap v3341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:22:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:08 smithi115 ceph-mon[35364]: pgmap v3341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:22:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:08 smithi181 ceph-mon[35869]: pgmap v3341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:22:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:10 smithi115 ceph-mon[35364]: pgmap v3342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:10 smithi181 ceph-mon[35869]: pgmap v3342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:10 smithi097 ceph-mon[32231]: pgmap v3342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4967.732475 secs 2024-07-26T09:22:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4967.732475 secs 2024-07-26T09:22:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4967.732475 secs 2024-07-26T09:22:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:12 smithi115 ceph-mon[35364]: pgmap v3343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:12 smithi181 ceph-mon[35869]: pgmap v3343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:12 smithi097 ceph-mon[32231]: pgmap v3343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:14 smithi115 ceph-mon[35364]: pgmap v3344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:14 smithi181 ceph-mon[35869]: pgmap v3344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:14 smithi097 ceph-mon[32231]: pgmap v3344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:16 smithi115 ceph-mon[35364]: pgmap v3345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:16.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4972.732609 secs 2024-07-26T09:22:16.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:16 smithi181 ceph-mon[35869]: pgmap v3345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4972.732609 secs 2024-07-26T09:22:16.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:16 smithi097 ceph-mon[32231]: pgmap v3345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:16.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4972.732609 secs 2024-07-26T09:22:16.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:16.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:17.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:17.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:22:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:18 smithi115 ceph-mon[35364]: pgmap v3346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:22:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:18 smithi115 ceph-mon[35364]: pgmap v3347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:18 smithi181 ceph-mon[35869]: pgmap v3346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:22:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:18 smithi181 ceph-mon[35869]: pgmap v3347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:18 smithi097 ceph-mon[32231]: pgmap v3346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:22:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:18 smithi097 ceph-mon[32231]: pgmap v3347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:19.726 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:19.724+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:22:19.728 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:19.725+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:22:19.728 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:19.725+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:22:19.728 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:19.725+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:22:19.728 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:19.725+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:20 smithi115 ceph-mon[35364]: pgmap v3348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-26T09:22:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:20 smithi181 ceph-mon[35869]: pgmap v3348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-26T09:22:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:20 smithi097 ceph-mon[32231]: pgmap v3348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-26T09:22:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4977.732741 secs 2024-07-26T09:22:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4977.732741 secs 2024-07-26T09:22:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4977.732741 secs 2024-07-26T09:22:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:22 smithi115 ceph-mon[35364]: pgmap v3349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:22 smithi181 ceph-mon[35869]: pgmap v3349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:22 smithi097 ceph-mon[32231]: pgmap v3349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:24 smithi115 ceph-mon[35364]: pgmap v3350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:24 smithi181 ceph-mon[35869]: pgmap v3350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:24 smithi097 ceph-mon[32231]: pgmap v3350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:26 smithi115 ceph-mon[35364]: pgmap v3351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:22:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4982.732883 secs 2024-07-26T09:22:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:26 smithi181 ceph-mon[35869]: pgmap v3351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:22:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4982.732883 secs 2024-07-26T09:22:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:26 smithi097 ceph-mon[32231]: pgmap v3351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:22:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4982.732883 secs 2024-07-26T09:22:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:28 smithi115 ceph-mon[35364]: pgmap v3352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:22:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:28 smithi181 ceph-mon[35869]: pgmap v3352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:22:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:28 smithi097 ceph-mon[32231]: pgmap v3352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:22:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:30 smithi115 ceph-mon[35364]: pgmap v3353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:22:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:30 smithi181 ceph-mon[35869]: pgmap v3353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:22:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:30 smithi097 ceph-mon[32231]: pgmap v3353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:22:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4987.733008 secs 2024-07-26T09:22:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4987.733008 secs 2024-07-26T09:22:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4987.733008 secs 2024-07-26T09:22:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:32 smithi115 ceph-mon[35364]: pgmap v3354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:32 smithi181 ceph-mon[35869]: pgmap v3354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:32 smithi097 ceph-mon[32231]: pgmap v3354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:22:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:34 smithi115 ceph-mon[35364]: pgmap v3355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:22:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:34 smithi181 ceph-mon[35869]: pgmap v3355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:22:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:34 smithi097 ceph-mon[32231]: pgmap v3355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:22:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:34.731+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:22:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:34.732+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:22:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:34.732+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:22:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:34.732+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:22:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:34.732+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:36 smithi115 ceph-mon[35364]: pgmap v3356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:22:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4992.733149 secs 2024-07-26T09:22:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:36 smithi181 ceph-mon[35869]: pgmap v3356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:22:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4992.733149 secs 2024-07-26T09:22:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:36 smithi097 ceph-mon[32231]: pgmap v3356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:22:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4992.733149 secs 2024-07-26T09:22:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:38 smithi115 ceph-mon[35364]: pgmap v3357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:38 smithi181 ceph-mon[35869]: pgmap v3357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:38 smithi097 ceph-mon[32231]: pgmap v3357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:40 smithi115 ceph-mon[35364]: pgmap v3358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:40 smithi181 ceph-mon[35869]: pgmap v3358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:40 smithi097 ceph-mon[32231]: pgmap v3358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 4997.733290 secs 2024-07-26T09:22:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 4997.733290 secs 2024-07-26T09:22:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 4997.733290 secs 2024-07-26T09:22:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:42 smithi115 ceph-mon[35364]: pgmap v3359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:42 smithi181 ceph-mon[35869]: pgmap v3359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:42 smithi097 ceph-mon[32231]: pgmap v3359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:44 smithi115 ceph-mon[35364]: pgmap v3360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:44 smithi181 ceph-mon[35869]: pgmap v3360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:44 smithi097 ceph-mon[32231]: pgmap v3360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:46 smithi181 ceph-mon[35869]: pgmap v3361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:46.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5002.733392 secs 2024-07-26T09:22:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:46 smithi097 ceph-mon[32231]: pgmap v3361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5002.733392 secs 2024-07-26T09:22:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:46 smithi115 ceph-mon[35364]: pgmap v3361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5002.733392 secs 2024-07-26T09:22:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:48 smithi181 ceph-mon[35869]: pgmap v3362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:48 smithi097 ceph-mon[32231]: pgmap v3362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:48 smithi115 ceph-mon[35364]: pgmap v3362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:49.742+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:22:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:49.743+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:22:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:49.743+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:22:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:49.743+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:22:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:22:49.743+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:50 smithi097 ceph-mon[32231]: pgmap v3363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:50 smithi115 ceph-mon[35364]: pgmap v3363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:50 smithi181 ceph-mon[35869]: pgmap v3363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5007.733524 secs 2024-07-26T09:22:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5007.733524 secs 2024-07-26T09:22:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5007.733524 secs 2024-07-26T09:22:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:52 smithi115 ceph-mon[35364]: pgmap v3364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:52 smithi181 ceph-mon[35869]: pgmap v3364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:52 smithi097 ceph-mon[32231]: pgmap v3364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:54 smithi115 ceph-mon[35364]: pgmap v3365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:54 smithi181 ceph-mon[35869]: pgmap v3365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:54 smithi097 ceph-mon[32231]: pgmap v3365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:56 smithi115 ceph-mon[35364]: pgmap v3366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5012.733662 secs 2024-07-26T09:22:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:56 smithi181 ceph-mon[35869]: pgmap v3366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5012.733662 secs 2024-07-26T09:22:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:56 smithi097 ceph-mon[32231]: pgmap v3366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5012.733662 secs 2024-07-26T09:22:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:22:58 smithi115 ceph-mon[35364]: pgmap v3367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:22:58 smithi181 ceph-mon[35869]: pgmap v3367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:22:58 smithi097 ceph-mon[32231]: pgmap v3367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:00 smithi115 ceph-mon[35364]: pgmap v3368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:00 smithi181 ceph-mon[35869]: pgmap v3368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:00 smithi097 ceph-mon[32231]: pgmap v3368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5017.733815 secs 2024-07-26T09:23:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5017.733815 secs 2024-07-26T09:23:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5017.733815 secs 2024-07-26T09:23:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:02 smithi115 ceph-mon[35364]: pgmap v3369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:02 smithi181 ceph-mon[35869]: pgmap v3369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:02 smithi097 ceph-mon[32231]: pgmap v3369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:03 smithi115 ceph-mon[35364]: pgmap v3370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:03 smithi181 ceph-mon[35869]: pgmap v3370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:03 smithi097 ceph-mon[32231]: pgmap v3370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:04.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:04.748+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:23:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:04.749+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:23:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:04.749+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:23:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:04.749+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:23:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:04.749+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:05 smithi115 ceph-mon[35364]: pgmap v3371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:05 smithi181 ceph-mon[35869]: pgmap v3371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:05 smithi097 ceph-mon[32231]: pgmap v3371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5022.733957 secs 2024-07-26T09:23:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5022.733957 secs 2024-07-26T09:23:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5022.733957 secs 2024-07-26T09:23:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:07 smithi115 ceph-mon[35364]: pgmap v3372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:07 smithi181 ceph-mon[35869]: pgmap v3372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:07 smithi097 ceph-mon[32231]: pgmap v3372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:09 smithi181 ceph-mon[35869]: pgmap v3373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:09 smithi097 ceph-mon[32231]: pgmap v3373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:09 smithi115 ceph-mon[35364]: pgmap v3373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5027.734114 secs 2024-07-26T09:23:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:11 smithi097 ceph-mon[32231]: pgmap v3374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5027.734114 secs 2024-07-26T09:23:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:11 smithi115 ceph-mon[35364]: pgmap v3374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5027.734114 secs 2024-07-26T09:23:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:11 smithi181 ceph-mon[35869]: pgmap v3374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:13 smithi097 ceph-mon[32231]: pgmap v3375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:13 smithi115 ceph-mon[35364]: pgmap v3375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:13 smithi181 ceph-mon[35869]: pgmap v3375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:15 smithi097 ceph-mon[32231]: pgmap v3376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:15 smithi115 ceph-mon[35364]: pgmap v3376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:15 smithi181 ceph-mon[35869]: pgmap v3376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5032.734284 secs 2024-07-26T09:23:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5032.734284 secs 2024-07-26T09:23:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5032.734284 secs 2024-07-26T09:23:17.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:17 smithi097 ceph-mon[32231]: pgmap v3377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:17.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:17.869 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:17 smithi181 ceph-mon[35869]: pgmap v3377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:17.869 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:17.880 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:17 smithi115 ceph-mon[35364]: pgmap v3377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:17.880 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:19.759+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:23:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:19.760+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:23:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:19.760+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:23:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:19.760+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:23:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:19.760+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:19 smithi115 ceph-mon[35364]: pgmap v3378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:19 smithi181 ceph-mon[35869]: pgmap v3378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:19 smithi097 ceph-mon[32231]: pgmap v3378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5037.734379 secs 2024-07-26T09:23:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5037.734379 secs 2024-07-26T09:23:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5037.734379 secs 2024-07-26T09:23:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:21 smithi115 ceph-mon[35364]: pgmap v3379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:21 smithi115 ceph-mon[35364]: pgmap v3380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:22.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:21 smithi115 ceph-mon[35364]: pgmap v3381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T09:23:22.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:21 smithi181 ceph-mon[35869]: pgmap v3379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:22.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:22.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:22.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:21 smithi181 ceph-mon[35869]: pgmap v3380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:22.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:21 smithi181 ceph-mon[35869]: pgmap v3381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T09:23:22.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:21 smithi097 ceph-mon[32231]: pgmap v3379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:21 smithi097 ceph-mon[32231]: pgmap v3380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:21 smithi097 ceph-mon[32231]: pgmap v3381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T09:23:22.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:23:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:24 smithi115 ceph-mon[35364]: pgmap v3382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:24 smithi181 ceph-mon[35869]: pgmap v3382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:24 smithi097 ceph-mon[32231]: pgmap v3382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5042.734543 secs 2024-07-26T09:23:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:26 smithi115 ceph-mon[35364]: pgmap v3383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5042.734543 secs 2024-07-26T09:23:26.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:26 smithi181 ceph-mon[35869]: pgmap v3383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5042.734543 secs 2024-07-26T09:23:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:26 smithi097 ceph-mon[32231]: pgmap v3383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:28 smithi115 ceph-mon[35364]: pgmap v3384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T09:23:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:28 smithi181 ceph-mon[35869]: pgmap v3384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T09:23:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:28 smithi097 ceph-mon[32231]: pgmap v3384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T09:23:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:30 smithi115 ceph-mon[35364]: pgmap v3385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:30 smithi181 ceph-mon[35869]: pgmap v3385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:30 smithi097 ceph-mon[32231]: pgmap v3385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5047.734680 secs 2024-07-26T09:23:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5047.734680 secs 2024-07-26T09:23:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5047.734680 secs 2024-07-26T09:23:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:32 smithi115 ceph-mon[35364]: pgmap v3386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:32 smithi181 ceph-mon[35869]: pgmap v3386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:32 smithi097 ceph-mon[32231]: pgmap v3386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:34 smithi115 ceph-mon[35364]: pgmap v3387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:34 smithi181 ceph-mon[35869]: pgmap v3387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:34 smithi097 ceph-mon[32231]: pgmap v3387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:34.764+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:23:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:34.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:23:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:34.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:23:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:34.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:23:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:34.766+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5052.734767 secs 2024-07-26T09:23:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:36 smithi115 ceph-mon[35364]: pgmap v3388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5052.734767 secs 2024-07-26T09:23:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:36 smithi181 ceph-mon[35869]: pgmap v3388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5052.734767 secs 2024-07-26T09:23:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:36 smithi097 ceph-mon[32231]: pgmap v3388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:38 smithi115 ceph-mon[35364]: pgmap v3389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:38 smithi181 ceph-mon[35869]: pgmap v3389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:38 smithi097 ceph-mon[32231]: pgmap v3389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:40 smithi181 ceph-mon[35869]: pgmap v3390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:40 smithi097 ceph-mon[32231]: pgmap v3390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:40.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:40 smithi115 ceph-mon[35364]: pgmap v3390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5057.734905 secs 2024-07-26T09:23:41.650 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5057.734905 secs 2024-07-26T09:23:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5057.734905 secs 2024-07-26T09:23:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:42 smithi097 ceph-mon[32231]: pgmap v3391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:42 smithi115 ceph-mon[35364]: pgmap v3391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:42 smithi181 ceph-mon[35869]: pgmap v3391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:44 smithi097 ceph-mon[32231]: pgmap v3392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:44 smithi115 ceph-mon[35364]: pgmap v3392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:44 smithi181 ceph-mon[35869]: pgmap v3392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5062.735065 secs 2024-07-26T09:23:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:46 smithi115 ceph-mon[35364]: pgmap v3393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5062.735065 secs 2024-07-26T09:23:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:46 smithi181 ceph-mon[35869]: pgmap v3393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5062.735065 secs 2024-07-26T09:23:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:46 smithi097 ceph-mon[32231]: pgmap v3393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:48 smithi115 ceph-mon[35364]: pgmap v3394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:23:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:48 smithi181 ceph-mon[35869]: pgmap v3394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:23:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:48 smithi097 ceph-mon[32231]: pgmap v3394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:23:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:49.776+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:23:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:49.777+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:23:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:49.777+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:23:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:49.777+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:23:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:23:49.777+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:50 smithi115 ceph-mon[35364]: pgmap v3395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:23:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:50 smithi181 ceph-mon[35869]: pgmap v3395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:23:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:50 smithi097 ceph-mon[32231]: pgmap v3395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:23:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5067.735202 secs 2024-07-26T09:23:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5067.735202 secs 2024-07-26T09:23:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5067.735202 secs 2024-07-26T09:23:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:52 smithi115 ceph-mon[35364]: pgmap v3396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:52 smithi181 ceph-mon[35869]: pgmap v3396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:52 smithi097 ceph-mon[32231]: pgmap v3396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:54 smithi115 ceph-mon[35364]: pgmap v3397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:23:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:54 smithi181 ceph-mon[35869]: pgmap v3397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:23:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:54 smithi097 ceph-mon[32231]: pgmap v3397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:23:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5072.735349 secs 2024-07-26T09:23:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:56 smithi115 ceph-mon[35364]: pgmap v3398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5072.735349 secs 2024-07-26T09:23:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:56 smithi181 ceph-mon[35869]: pgmap v3398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5072.735349 secs 2024-07-26T09:23:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:56 smithi097 ceph-mon[32231]: pgmap v3398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:58 smithi115 ceph-mon[35364]: pgmap v3399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:58 smithi181 ceph-mon[35869]: pgmap v3399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:58 smithi097 ceph-mon[32231]: pgmap v3399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:23:59 smithi115 ceph-mon[35364]: pgmap v3400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:23:59 smithi181 ceph-mon[35869]: pgmap v3400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:23:59 smithi097 ceph-mon[32231]: pgmap v3400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:24:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5077.735459 secs 2024-07-26T09:24:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5077.735459 secs 2024-07-26T09:24:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5077.735459 secs 2024-07-26T09:24:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:01 smithi115 ceph-mon[35364]: pgmap v3401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:01 smithi181 ceph-mon[35869]: pgmap v3401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:01 smithi097 ceph-mon[32231]: pgmap v3401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:04 smithi115 ceph-mon[35364]: pgmap v3402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:04 smithi181 ceph-mon[35869]: pgmap v3402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:04 smithi097 ceph-mon[32231]: pgmap v3402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:04.781+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:24:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:04.783+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:24:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:04.783+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:24:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:04.783+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:24:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:04.783+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5082.735543 secs 2024-07-26T09:24:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:06 smithi115 ceph-mon[35364]: pgmap v3403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5082.735543 secs 2024-07-26T09:24:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:06 smithi181 ceph-mon[35869]: pgmap v3403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5082.735543 secs 2024-07-26T09:24:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:06 smithi097 ceph-mon[32231]: pgmap v3403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:08 smithi115 ceph-mon[35364]: pgmap v3404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:08 smithi181 ceph-mon[35869]: pgmap v3404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:08 smithi097 ceph-mon[32231]: pgmap v3404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:10 smithi115 ceph-mon[35364]: pgmap v3405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:24:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:10 smithi181 ceph-mon[35869]: pgmap v3405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:24:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:10 smithi097 ceph-mon[32231]: pgmap v3405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:24:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5087.735665 secs 2024-07-26T09:24:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5087.735665 secs 2024-07-26T09:24:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5087.735665 secs 2024-07-26T09:24:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:12 smithi115 ceph-mon[35364]: pgmap v3406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:12 smithi181 ceph-mon[35869]: pgmap v3406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:12 smithi097 ceph-mon[32231]: pgmap v3406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:14.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:14 smithi181 ceph-mon[35869]: pgmap v3407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:14 smithi097 ceph-mon[32231]: pgmap v3407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:14 smithi115 ceph-mon[35364]: pgmap v3407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5092.735799 secs 2024-07-26T09:24:16.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:16 smithi181 ceph-mon[35869]: pgmap v3408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5092.735799 secs 2024-07-26T09:24:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:16 smithi097 ceph-mon[32231]: pgmap v3408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5092.735799 secs 2024-07-26T09:24:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:16 smithi115 ceph-mon[35364]: pgmap v3408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:17 smithi115 ceph-mon[35364]: pgmap v3409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:17 smithi181 ceph-mon[35869]: pgmap v3409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:17 smithi097 ceph-mon[32231]: pgmap v3409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:19.793+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:24:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:19.794+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:24:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:19.794+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:24:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:19.794+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:24:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:19.794+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:20 smithi115 ceph-mon[35364]: pgmap v3410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:24:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:20 smithi181 ceph-mon[35869]: pgmap v3410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:24:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:20 smithi097 ceph-mon[32231]: pgmap v3410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:24:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5097.735940 secs 2024-07-26T09:24:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5097.735940 secs 2024-07-26T09:24:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5097.735940 secs 2024-07-26T09:24:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:22 smithi115 ceph-mon[35364]: pgmap v3411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:22 smithi181 ceph-mon[35869]: pgmap v3411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:22 smithi097 ceph-mon[32231]: pgmap v3411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:24:23.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:23.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:23.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:24:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:24:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:24 smithi115 ceph-mon[35364]: pgmap v3412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:24:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:24 smithi115 ceph-mon[35364]: pgmap v3413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:24 smithi181 ceph-mon[35869]: pgmap v3412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:24:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:24 smithi181 ceph-mon[35869]: pgmap v3413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:24 smithi097 ceph-mon[32231]: pgmap v3412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:24:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:24 smithi097 ceph-mon[32231]: pgmap v3413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:26 smithi181 ceph-mon[35869]: pgmap v3414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:26.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5102.736085 secs 2024-07-26T09:24:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:26 smithi097 ceph-mon[32231]: pgmap v3414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5102.736085 secs 2024-07-26T09:24:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:26 smithi115 ceph-mon[35364]: pgmap v3414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5102.736085 secs 2024-07-26T09:24:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:28 smithi097 ceph-mon[32231]: pgmap v3415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:28 smithi115 ceph-mon[35364]: pgmap v3415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:28 smithi181 ceph-mon[35869]: pgmap v3415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:30 smithi097 ceph-mon[32231]: pgmap v3416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:30 smithi115 ceph-mon[35364]: pgmap v3416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:30 smithi181 ceph-mon[35869]: pgmap v3416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:24:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5107.736215 secs 2024-07-26T09:24:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5107.736215 secs 2024-07-26T09:24:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5107.736215 secs 2024-07-26T09:24:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:32 smithi115 ceph-mon[35364]: pgmap v3417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:32 smithi181 ceph-mon[35869]: pgmap v3417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:32 smithi097 ceph-mon[32231]: pgmap v3417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:34 smithi115 ceph-mon[35364]: pgmap v3418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:24:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:34 smithi181 ceph-mon[35869]: pgmap v3418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:24:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:34 smithi097 ceph-mon[32231]: pgmap v3418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:24:34.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:34.798+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:24:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:34.799+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:24:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:34.799+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:24:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:34.799+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:24:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:34.799+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:36 smithi115 ceph-mon[35364]: pgmap v3419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:36.742 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5112.736363 secs 2024-07-26T09:24:36.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:36 smithi181 ceph-mon[35869]: pgmap v3419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:36.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5112.736363 secs 2024-07-26T09:24:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:36 smithi097 ceph-mon[32231]: pgmap v3419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5112.736363 secs 2024-07-26T09:24:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:38 smithi115 ceph-mon[35364]: pgmap v3420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:38 smithi181 ceph-mon[35869]: pgmap v3420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:38 smithi097 ceph-mon[32231]: pgmap v3420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:40.538 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:40 smithi181 ceph-mon[35869]: pgmap v3421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:40.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:40 smithi097 ceph-mon[32231]: pgmap v3421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:40 smithi115 ceph-mon[35364]: pgmap v3421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5117.736497 secs 2024-07-26T09:24:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5117.736497 secs 2024-07-26T09:24:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5117.736497 secs 2024-07-26T09:24:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:42 smithi115 ceph-mon[35364]: pgmap v3422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:42 smithi181 ceph-mon[35869]: pgmap v3422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:42.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:42 smithi097 ceph-mon[32231]: pgmap v3422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:44 smithi115 ceph-mon[35364]: pgmap v3423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:44 smithi181 ceph-mon[35869]: pgmap v3423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:44 smithi097 ceph-mon[32231]: pgmap v3423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:46 smithi115 ceph-mon[35364]: pgmap v3424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5122.736642 secs 2024-07-26T09:24:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:46 smithi181 ceph-mon[35869]: pgmap v3424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5122.736642 secs 2024-07-26T09:24:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:46 smithi097 ceph-mon[32231]: pgmap v3424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5122.736642 secs 2024-07-26T09:24:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:47 smithi115 ceph-mon[35364]: pgmap v3425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:47 smithi181 ceph-mon[35869]: pgmap v3425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:47 smithi097 ceph-mon[32231]: pgmap v3425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:49.810+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:24:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:49.811+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:24:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:49.811+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:24:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:49.811+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:24:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:24:49.811+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:49 smithi115 ceph-mon[35364]: pgmap v3426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:49 smithi181 ceph-mon[35869]: pgmap v3426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:49 smithi097 ceph-mon[32231]: pgmap v3426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5127.736783 secs 2024-07-26T09:24:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5127.736783 secs 2024-07-26T09:24:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5127.736783 secs 2024-07-26T09:24:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:51 smithi115 ceph-mon[35364]: pgmap v3427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:51 smithi181 ceph-mon[35869]: pgmap v3427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:51 smithi097 ceph-mon[32231]: pgmap v3427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:53 smithi115 ceph-mon[35364]: pgmap v3428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:53 smithi181 ceph-mon[35869]: pgmap v3428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:53 smithi097 ceph-mon[32231]: pgmap v3428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:55 smithi115 ceph-mon[35364]: pgmap v3429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5132.736920 secs 2024-07-26T09:24:55.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:55 smithi181 ceph-mon[35869]: pgmap v3429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:55.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5132.736920 secs 2024-07-26T09:24:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:55 smithi097 ceph-mon[32231]: pgmap v3429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5132.736920 secs 2024-07-26T09:24:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:57 smithi115 ceph-mon[35364]: pgmap v3430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:57 smithi181 ceph-mon[35869]: pgmap v3430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:57 smithi097 ceph-mon[32231]: pgmap v3430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:24:59 smithi115 ceph-mon[35364]: pgmap v3431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:24:59 smithi181 ceph-mon[35869]: pgmap v3431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:24:59 smithi097 ceph-mon[32231]: pgmap v3431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5137.737075 secs 2024-07-26T09:25:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5137.737075 secs 2024-07-26T09:25:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5137.737075 secs 2024-07-26T09:25:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:01 smithi115 ceph-mon[35364]: pgmap v3432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:01 smithi181 ceph-mon[35869]: pgmap v3432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:01 smithi097 ceph-mon[32231]: pgmap v3432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:03 smithi115 ceph-mon[35364]: pgmap v3433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:03 smithi181 ceph-mon[35869]: pgmap v3433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:03 smithi097 ceph-mon[32231]: pgmap v3433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:04.817+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:25:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:04.818+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:25:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:04.818+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:25:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:04.818+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:25:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:04.818+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:05 smithi181 ceph-mon[35869]: pgmap v3434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:05.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5142.737280 secs 2024-07-26T09:25:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:05 smithi097 ceph-mon[32231]: pgmap v3434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5142.737280 secs 2024-07-26T09:25:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:05 smithi115 ceph-mon[35364]: pgmap v3434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5142.737280 secs 2024-07-26T09:25:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:07.783+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:25:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:07.783+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:25:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:07.783+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T09:25:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:07.784+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:25:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:07 smithi097 ceph-mon[32231]: pgmap v3435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:07 smithi115 ceph-mon[35364]: pgmap v3435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:07 smithi181 ceph-mon[35869]: pgmap v3435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:09 smithi097 ceph-mon[32231]: pgmap v3436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:10.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:09 smithi115 ceph-mon[35364]: pgmap v3436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:10.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:09 smithi181 ceph-mon[35869]: pgmap v3436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5147.737369 secs 2024-07-26T09:25:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5147.737369 secs 2024-07-26T09:25:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5147.737369 secs 2024-07-26T09:25:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:11 smithi115 ceph-mon[35364]: pgmap v3437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:11 smithi181 ceph-mon[35869]: pgmap v3437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:11 smithi097 ceph-mon[32231]: pgmap v3437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:13 smithi115 ceph-mon[35364]: pgmap v3438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:13 smithi181 ceph-mon[35869]: pgmap v3438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:13 smithi097 ceph-mon[32231]: pgmap v3438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:15 smithi115 ceph-mon[35364]: pgmap v3439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5152.737517 secs 2024-07-26T09:25:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:15 smithi181 ceph-mon[35869]: pgmap v3439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5152.737517 secs 2024-07-26T09:25:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:15 smithi097 ceph-mon[32231]: pgmap v3439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5152.737517 secs 2024-07-26T09:25:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:17 smithi115 ceph-mon[35364]: pgmap v3440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:18.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:17 smithi181 ceph-mon[35869]: pgmap v3440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:17 smithi097 ceph-mon[32231]: pgmap v3440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:19.829+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:19.830+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:19.830+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:19.830+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:19.830+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:19 smithi115 ceph-mon[35364]: pgmap v3441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:19 smithi181 ceph-mon[35869]: pgmap v3441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:19 smithi097 ceph-mon[32231]: pgmap v3441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5157.737655 secs 2024-07-26T09:25:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5157.737655 secs 2024-07-26T09:25:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5157.737655 secs 2024-07-26T09:25:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:21 smithi115 ceph-mon[35364]: pgmap v3442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:21 smithi181 ceph-mon[35869]: pgmap v3442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:21 smithi097 ceph-mon[32231]: pgmap v3442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:23 smithi115 ceph-mon[35364]: pgmap v3443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:23 smithi181 ceph-mon[35869]: pgmap v3443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:24.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:23 smithi097 ceph-mon[32231]: pgmap v3443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:24 smithi115 ceph-mon[35364]: pgmap v3444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:25:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:24 smithi115 ceph-mon[35364]: pgmap v3445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:25:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:24 smithi181 ceph-mon[35869]: pgmap v3444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:25:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:24 smithi181 ceph-mon[35869]: pgmap v3445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:25:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:24 smithi097 ceph-mon[32231]: pgmap v3444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:25:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:24 smithi097 ceph-mon[32231]: pgmap v3445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:25:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5162.737796 secs 2024-07-26T09:25:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5162.737796 secs 2024-07-26T09:25:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5162.737796 secs 2024-07-26T09:25:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:26 smithi115 ceph-mon[35364]: pgmap v3446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:26 smithi181 ceph-mon[35869]: pgmap v3446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:26 smithi097 ceph-mon[32231]: pgmap v3446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:29 smithi115 ceph-mon[35364]: pgmap v3447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:29 smithi181 ceph-mon[35869]: pgmap v3447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:29 smithi097 ceph-mon[32231]: pgmap v3447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5167.737940 secs 2024-07-26T09:25:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:31 smithi115 ceph-mon[35364]: pgmap v3448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5167.737940 secs 2024-07-26T09:25:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:31 smithi181 ceph-mon[35869]: pgmap v3448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5167.737940 secs 2024-07-26T09:25:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:31 smithi097 ceph-mon[32231]: pgmap v3448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:33 smithi115 ceph-mon[35364]: pgmap v3449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:25:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:33 smithi181 ceph-mon[35869]: pgmap v3449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:25:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:33 smithi097 ceph-mon[32231]: pgmap v3449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:25:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:34.835+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:25:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:34.836+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:25:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:34.836+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:25:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:34.836+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:25:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:34.837+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:35 smithi115 ceph-mon[35364]: pgmap v3450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:25:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:35 smithi181 ceph-mon[35869]: pgmap v3450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:25:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:35 smithi097 ceph-mon[32231]: pgmap v3450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:25:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5172.738081 secs 2024-07-26T09:25:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5172.738081 secs 2024-07-26T09:25:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5172.738081 secs 2024-07-26T09:25:37.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:37 smithi115 ceph-mon[35364]: pgmap v3451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:37 smithi181 ceph-mon[35869]: pgmap v3451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:37 smithi097 ceph-mon[32231]: pgmap v3451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:39 smithi115 ceph-mon[35364]: pgmap v3452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:25:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:39 smithi181 ceph-mon[35869]: pgmap v3452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:25:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:39 smithi097 ceph-mon[32231]: pgmap v3452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:25:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5177.738221 secs 2024-07-26T09:25:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:41 smithi115 ceph-mon[35364]: pgmap v3453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:25:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5177.738221 secs 2024-07-26T09:25:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:41 smithi181 ceph-mon[35869]: pgmap v3453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:25:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5177.738221 secs 2024-07-26T09:25:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:41 smithi097 ceph-mon[32231]: pgmap v3453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:25:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:43 smithi115 ceph-mon[35364]: pgmap v3454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:25:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:43 smithi181 ceph-mon[35869]: pgmap v3454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:25:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:43 smithi097 ceph-mon[32231]: pgmap v3454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:25:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:45 smithi115 ceph-mon[35364]: pgmap v3455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:25:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:45 smithi181 ceph-mon[35869]: pgmap v3455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:25:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:45 smithi097 ceph-mon[32231]: pgmap v3455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:25:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5182.738382 secs 2024-07-26T09:25:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5182.738382 secs 2024-07-26T09:25:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5182.738382 secs 2024-07-26T09:25:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:47 smithi115 ceph-mon[35364]: pgmap v3456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:47 smithi181 ceph-mon[35869]: pgmap v3456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:47 smithi097 ceph-mon[32231]: pgmap v3456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:49 smithi115 ceph-mon[35364]: pgmap v3457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:49 smithi181 ceph-mon[35869]: pgmap v3457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:49 smithi097 ceph-mon[32231]: pgmap v3457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:49.842+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:25:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:49.843+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:25:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:49.843+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:25:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:49.843+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:25:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:25:49.843+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5187.738555 secs 2024-07-26T09:25:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:51 smithi115 ceph-mon[35364]: pgmap v3458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5187.738555 secs 2024-07-26T09:25:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:51 smithi181 ceph-mon[35869]: pgmap v3458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5187.738555 secs 2024-07-26T09:25:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:51 smithi097 ceph-mon[32231]: pgmap v3458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:53 smithi115 ceph-mon[35364]: pgmap v3459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:53 smithi181 ceph-mon[35869]: pgmap v3459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:53 smithi097 ceph-mon[32231]: pgmap v3459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:55 smithi115 ceph-mon[35364]: pgmap v3460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:25:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:55 smithi181 ceph-mon[35869]: pgmap v3460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:25:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:55 smithi097 ceph-mon[32231]: pgmap v3460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:25:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5192.738694 secs 2024-07-26T09:25:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5192.738694 secs 2024-07-26T09:25:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5192.738694 secs 2024-07-26T09:25:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:57 smithi097 ceph-mon[32231]: pgmap v3461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:57 smithi115 ceph-mon[35364]: pgmap v3461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:57 smithi181 ceph-mon[35869]: pgmap v3461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:25:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:25:59 smithi097 ceph-mon[32231]: pgmap v3462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:25:59 smithi115 ceph-mon[35364]: pgmap v3462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:25:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:25:59 smithi181 ceph-mon[35869]: pgmap v3462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5197.738833 secs 2024-07-26T09:26:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:01 smithi115 ceph-mon[35364]: pgmap v3463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5197.738833 secs 2024-07-26T09:26:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:01 smithi181 ceph-mon[35869]: pgmap v3463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5197.738833 secs 2024-07-26T09:26:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:01 smithi097 ceph-mon[32231]: pgmap v3463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:02 smithi097 ceph-mon[32231]: pgmap v3464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:02 smithi115 ceph-mon[35364]: pgmap v3464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:02 smithi181 ceph-mon[35869]: pgmap v3464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:04.845+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:26:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:04.846+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:26:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:04.846+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:26:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:04.846+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:26:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:04.846+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:05 smithi115 ceph-mon[35364]: pgmap v3465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:05 smithi181 ceph-mon[35869]: pgmap v3465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:05 smithi097 ceph-mon[32231]: pgmap v3465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5202.738972 secs 2024-07-26T09:26:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5202.738972 secs 2024-07-26T09:26:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5202.738972 secs 2024-07-26T09:26:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:07 smithi115 ceph-mon[35364]: pgmap v3466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:07 smithi181 ceph-mon[35869]: pgmap v3466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:07 smithi097 ceph-mon[32231]: pgmap v3466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:09 smithi115 ceph-mon[35364]: pgmap v3467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:09 smithi181 ceph-mon[35869]: pgmap v3467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:09 smithi097 ceph-mon[32231]: pgmap v3467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5207.739114 secs 2024-07-26T09:26:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:11 smithi115 ceph-mon[35364]: pgmap v3468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5207.739114 secs 2024-07-26T09:26:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:11 smithi181 ceph-mon[35869]: pgmap v3468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5207.739114 secs 2024-07-26T09:26:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:11 smithi097 ceph-mon[32231]: pgmap v3468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:13 smithi115 ceph-mon[35364]: pgmap v3469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:13 smithi181 ceph-mon[35869]: pgmap v3469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:13 smithi097 ceph-mon[32231]: pgmap v3469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:15 smithi115 ceph-mon[35364]: pgmap v3470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:15 smithi181 ceph-mon[35869]: pgmap v3470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:15 smithi097 ceph-mon[32231]: pgmap v3470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5212.739279 secs 2024-07-26T09:26:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5212.739279 secs 2024-07-26T09:26:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5212.739279 secs 2024-07-26T09:26:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:17 smithi115 ceph-mon[35364]: pgmap v3471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:17 smithi181 ceph-mon[35869]: pgmap v3471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:17 smithi097 ceph-mon[32231]: pgmap v3471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:19 smithi115 ceph-mon[35364]: pgmap v3472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:19 smithi181 ceph-mon[35869]: pgmap v3472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:19 smithi097 ceph-mon[32231]: pgmap v3472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:19.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:19.856+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:26:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:19.857+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:26:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:19.857+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:26:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:19.857+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:26:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:19.857+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5217.739416 secs 2024-07-26T09:26:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:21 smithi115 ceph-mon[35364]: pgmap v3473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5217.739416 secs 2024-07-26T09:26:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:21 smithi181 ceph-mon[35869]: pgmap v3473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5217.739416 secs 2024-07-26T09:26:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:21 smithi097 ceph-mon[32231]: pgmap v3473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:23 smithi115 ceph-mon[35364]: pgmap v3474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:23 smithi181 ceph-mon[35869]: pgmap v3474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:23 smithi097 ceph-mon[32231]: pgmap v3474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:25 smithi115 ceph-mon[35364]: pgmap v3475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:25.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:25 smithi181 ceph-mon[35869]: pgmap v3475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:25 smithi097 ceph-mon[32231]: pgmap v3475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:26:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5222.739552 secs 2024-07-26T09:26:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:26.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:26:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5222.739552 secs 2024-07-26T09:26:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:26:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5222.739552 secs 2024-07-26T09:26:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:26:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:27 smithi097 ceph-mon[32231]: pgmap v3476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:26:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:27 smithi115 ceph-mon[35364]: pgmap v3476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:26:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:27 smithi181 ceph-mon[35869]: pgmap v3476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:26:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:29 smithi097 ceph-mon[32231]: pgmap v3477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:26:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:29 smithi115 ceph-mon[35364]: pgmap v3477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:26:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:29 smithi181 ceph-mon[35869]: pgmap v3477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:26:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:31 smithi115 ceph-mon[35364]: pgmap v3478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:26:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5227.739692 secs 2024-07-26T09:26:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:31 smithi181 ceph-mon[35869]: pgmap v3478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:26:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5227.739692 secs 2024-07-26T09:26:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:31 smithi097 ceph-mon[32231]: pgmap v3478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:26:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5227.739692 secs 2024-07-26T09:26:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:33 smithi115 ceph-mon[35364]: pgmap v3479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:26:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:33 smithi181 ceph-mon[35869]: pgmap v3479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:26:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:33 smithi097 ceph-mon[32231]: pgmap v3479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:34.862+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:26:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:34.864+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:26:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:34.864+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:26:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:34.864+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:26:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:34.864+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:35 smithi115 ceph-mon[35364]: pgmap v3480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:26:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:35 smithi181 ceph-mon[35869]: pgmap v3480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:26:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:35 smithi097 ceph-mon[32231]: pgmap v3480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:26:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5232.739858 secs 2024-07-26T09:26:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5232.739858 secs 2024-07-26T09:26:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5232.739858 secs 2024-07-26T09:26:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:37 smithi115 ceph-mon[35364]: pgmap v3481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:26:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:37 smithi181 ceph-mon[35869]: pgmap v3481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:26:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:37 smithi097 ceph-mon[32231]: pgmap v3481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:26:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:39 smithi115 ceph-mon[35364]: pgmap v3482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:39 smithi181 ceph-mon[35869]: pgmap v3482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:39 smithi097 ceph-mon[32231]: pgmap v3482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:41.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:41 smithi097 ceph-mon[32231]: pgmap v3483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:41.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5237.739995 secs 2024-07-26T09:26:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:41 smithi115 ceph-mon[35364]: pgmap v3483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5237.739995 secs 2024-07-26T09:26:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:41 smithi181 ceph-mon[35869]: pgmap v3483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5237.739995 secs 2024-07-26T09:26:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:43 smithi115 ceph-mon[35364]: pgmap v3484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:43 smithi181 ceph-mon[35869]: pgmap v3484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:43 smithi097 ceph-mon[32231]: pgmap v3484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:45 smithi115 ceph-mon[35364]: pgmap v3485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:45 smithi181 ceph-mon[35869]: pgmap v3485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:45 smithi097 ceph-mon[32231]: pgmap v3485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5242.740128 secs 2024-07-26T09:26:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5242.740128 secs 2024-07-26T09:26:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5242.740128 secs 2024-07-26T09:26:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:47 smithi115 ceph-mon[35364]: pgmap v3486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:47 smithi181 ceph-mon[35869]: pgmap v3486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:47 smithi097 ceph-mon[32231]: pgmap v3486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:49 smithi115 ceph-mon[35364]: pgmap v3487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:49 smithi181 ceph-mon[35869]: pgmap v3487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:49 smithi097 ceph-mon[32231]: pgmap v3487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:49.874+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:26:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:49.876+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:26:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:49.876+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:26:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:49.876+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:26:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:26:49.876+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:51 smithi115 ceph-mon[35364]: pgmap v3488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5247.740316 secs 2024-07-26T09:26:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:51 smithi181 ceph-mon[35869]: pgmap v3488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5247.740316 secs 2024-07-26T09:26:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:51 smithi097 ceph-mon[32231]: pgmap v3488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5247.740316 secs 2024-07-26T09:26:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:53 smithi115 ceph-mon[35364]: pgmap v3489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:53 smithi181 ceph-mon[35869]: pgmap v3489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:53 smithi097 ceph-mon[32231]: pgmap v3489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:54 smithi115 ceph-mon[35364]: pgmap v3490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:54 smithi181 ceph-mon[35869]: pgmap v3490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:54 smithi097 ceph-mon[32231]: pgmap v3490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5252.740461 secs 2024-07-26T09:26:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:56 smithi115 ceph-mon[35364]: pgmap v3491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5252.740461 secs 2024-07-26T09:26:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:56 smithi181 ceph-mon[35869]: pgmap v3491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5252.740461 secs 2024-07-26T09:26:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:56 smithi097 ceph-mon[32231]: pgmap v3491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:26:58 smithi115 ceph-mon[35364]: pgmap v3492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:26:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:26:58 smithi181 ceph-mon[35869]: pgmap v3492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:26:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:26:58 smithi097 ceph-mon[32231]: pgmap v3492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:27:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:00 smithi115 ceph-mon[35364]: pgmap v3493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:00 smithi181 ceph-mon[35869]: pgmap v3493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:00 smithi097 ceph-mon[32231]: pgmap v3493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5257.740590 secs 2024-07-26T09:27:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5257.740590 secs 2024-07-26T09:27:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5257.740590 secs 2024-07-26T09:27:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:02 smithi115 ceph-mon[35364]: pgmap v3494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:02 smithi181 ceph-mon[35869]: pgmap v3494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:02 smithi097 ceph-mon[32231]: pgmap v3494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:04.882+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:27:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:04.882+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:27:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:04.882+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:27:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:04.882+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:27:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:04.882+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:04 smithi115 ceph-mon[35364]: pgmap v3495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:04 smithi181 ceph-mon[35869]: pgmap v3495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:04 smithi097 ceph-mon[32231]: pgmap v3495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5262.740724 secs 2024-07-26T09:27:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:06 smithi115 ceph-mon[35364]: pgmap v3496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5262.740724 secs 2024-07-26T09:27:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:06 smithi181 ceph-mon[35869]: pgmap v3496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5262.740724 secs 2024-07-26T09:27:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:06 smithi097 ceph-mon[32231]: pgmap v3496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:08 smithi115 ceph-mon[35364]: pgmap v3497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:08 smithi181 ceph-mon[35869]: pgmap v3497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:08 smithi097 ceph-mon[32231]: pgmap v3497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:10 smithi115 ceph-mon[35364]: pgmap v3498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:10 smithi181 ceph-mon[35869]: pgmap v3498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:10 smithi097 ceph-mon[32231]: pgmap v3498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:11.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5267.740858 secs 2024-07-26T09:27:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5267.740858 secs 2024-07-26T09:27:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5267.740858 secs 2024-07-26T09:27:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:12 smithi115 ceph-mon[35364]: pgmap v3499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:12 smithi181 ceph-mon[35869]: pgmap v3499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:12 smithi097 ceph-mon[32231]: pgmap v3499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:14 smithi115 ceph-mon[35364]: pgmap v3500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:14 smithi181 ceph-mon[35869]: pgmap v3500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:14 smithi097 ceph-mon[32231]: pgmap v3500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5272.741000 secs 2024-07-26T09:27:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5272.741000 secs 2024-07-26T09:27:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5272.741000 secs 2024-07-26T09:27:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:16 smithi115 ceph-mon[35364]: pgmap v3501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:16 smithi181 ceph-mon[35869]: pgmap v3501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:16 smithi097 ceph-mon[32231]: pgmap v3501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:18 smithi115 ceph-mon[35364]: pgmap v3502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:18 smithi181 ceph-mon[35869]: pgmap v3502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:18 smithi097 ceph-mon[32231]: pgmap v3502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:19.889+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:27:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:19.890+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:27:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:19.890+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:27:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:19.890+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:27:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:19.890+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:20 smithi115 ceph-mon[35364]: pgmap v3503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5277.741138 secs 2024-07-26T09:27:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:20 smithi181 ceph-mon[35869]: pgmap v3503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5277.741138 secs 2024-07-26T09:27:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:20 smithi097 ceph-mon[32231]: pgmap v3503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:27:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5277.741138 secs 2024-07-26T09:27:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:22 smithi115 ceph-mon[35364]: pgmap v3504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:22 smithi181 ceph-mon[35869]: pgmap v3504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:22 smithi097 ceph-mon[32231]: pgmap v3504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:24 smithi115 ceph-mon[35364]: pgmap v3505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:24 smithi181 ceph-mon[35869]: pgmap v3505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:24 smithi097 ceph-mon[32231]: pgmap v3505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5282.741297 secs 2024-07-26T09:27:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5282.741297 secs 2024-07-26T09:27:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5282.741297 secs 2024-07-26T09:27:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:26 smithi115 ceph-mon[35364]: pgmap v3506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:26 smithi181 ceph-mon[35869]: pgmap v3506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:26 smithi097 ceph-mon[32231]: pgmap v3506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:27 smithi115 ceph-mon[35364]: pgmap v3507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:27:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:27 smithi115 ceph-mon[35364]: pgmap v3508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:27.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:27:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:27 smithi181 ceph-mon[35869]: pgmap v3507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:27:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:27 smithi181 ceph-mon[35869]: pgmap v3508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:27:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:27 smithi097 ceph-mon[32231]: pgmap v3507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:27:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:27 smithi097 ceph-mon[32231]: pgmap v3508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:27:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:30 smithi115 ceph-mon[35364]: pgmap v3509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:27:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:30 smithi181 ceph-mon[35869]: pgmap v3509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:27:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:30 smithi097 ceph-mon[32231]: pgmap v3509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:27:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5287.741434 secs 2024-07-26T09:27:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5287.741434 secs 2024-07-26T09:27:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5287.741434 secs 2024-07-26T09:27:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:32 smithi115 ceph-mon[35364]: pgmap v3510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:27:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:32 smithi181 ceph-mon[35869]: pgmap v3510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:27:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:32 smithi097 ceph-mon[32231]: pgmap v3510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:27:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:34 smithi115 ceph-mon[35364]: pgmap v3511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:34 smithi181 ceph-mon[35869]: pgmap v3511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:34 smithi097 ceph-mon[32231]: pgmap v3511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:34.896+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:27:34.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:34.897+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:27:34.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:34.897+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:27:34.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:34.897+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:27:34.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:34.897+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5292.741608 secs 2024-07-26T09:27:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:36 smithi115 ceph-mon[35364]: pgmap v3512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5292.741608 secs 2024-07-26T09:27:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:36 smithi181 ceph-mon[35869]: pgmap v3512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5292.741608 secs 2024-07-26T09:27:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:36 smithi097 ceph-mon[32231]: pgmap v3512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:38 smithi115 ceph-mon[35364]: pgmap v3513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:27:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:38 smithi181 ceph-mon[35869]: pgmap v3513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:27:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:38 smithi097 ceph-mon[32231]: pgmap v3513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:27:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:40 smithi115 ceph-mon[35364]: pgmap v3514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:27:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:40 smithi181 ceph-mon[35869]: pgmap v3514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:27:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:40 smithi097 ceph-mon[32231]: pgmap v3514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:27:41.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5297.741746 secs 2024-07-26T09:27:41.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5297.741746 secs 2024-07-26T09:27:41.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5297.741746 secs 2024-07-26T09:27:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:42 smithi115 ceph-mon[35364]: pgmap v3515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:42 smithi181 ceph-mon[35869]: pgmap v3515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:42 smithi097 ceph-mon[32231]: pgmap v3515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:44 smithi115 ceph-mon[35364]: pgmap v3516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:44 smithi181 ceph-mon[35869]: pgmap v3516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:44 smithi097 ceph-mon[32231]: pgmap v3516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5302.741919 secs 2024-07-26T09:27:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:46 smithi115 ceph-mon[35364]: pgmap v3517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5302.741919 secs 2024-07-26T09:27:46.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:46 smithi181 ceph-mon[35869]: pgmap v3517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5302.741919 secs 2024-07-26T09:27:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:46 smithi097 ceph-mon[32231]: pgmap v3517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:48 smithi115 ceph-mon[35364]: pgmap v3518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:48 smithi181 ceph-mon[35869]: pgmap v3518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:48 smithi097 ceph-mon[32231]: pgmap v3518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:49.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:49.906+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:27:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:49.907+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:27:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:49.907+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:27:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:49.907+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:27:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:27:49.907+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:50 smithi115 ceph-mon[35364]: pgmap v3519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:50 smithi181 ceph-mon[35869]: pgmap v3519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:50 smithi097 ceph-mon[32231]: pgmap v3519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5307.742052 secs 2024-07-26T09:27:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5307.742052 secs 2024-07-26T09:27:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5307.742052 secs 2024-07-26T09:27:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:52 smithi115 ceph-mon[35364]: pgmap v3520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:52 smithi181 ceph-mon[35869]: pgmap v3520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:52 smithi097 ceph-mon[32231]: pgmap v3520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:54 smithi115 ceph-mon[35364]: pgmap v3521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:54 smithi181 ceph-mon[35869]: pgmap v3521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:54 smithi097 ceph-mon[32231]: pgmap v3521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5312.742192 secs 2024-07-26T09:27:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:56 smithi115 ceph-mon[35364]: pgmap v3522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5312.742192 secs 2024-07-26T09:27:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:56 smithi181 ceph-mon[35869]: pgmap v3522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5312.742192 secs 2024-07-26T09:27:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:56 smithi097 ceph-mon[32231]: pgmap v3522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:27:58 smithi115 ceph-mon[35364]: pgmap v3523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:27:58 smithi181 ceph-mon[35869]: pgmap v3523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:27:58 smithi097 ceph-mon[32231]: pgmap v3523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:00 smithi115 ceph-mon[35364]: pgmap v3524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:00 smithi181 ceph-mon[35869]: pgmap v3524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:00 smithi097 ceph-mon[32231]: pgmap v3524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5317.742345 secs 2024-07-26T09:28:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5317.742345 secs 2024-07-26T09:28:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5317.742345 secs 2024-07-26T09:28:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:02 smithi115 ceph-mon[35364]: pgmap v3525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:02 smithi181 ceph-mon[35869]: pgmap v3525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:02 smithi097 ceph-mon[32231]: pgmap v3525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:04 smithi115 ceph-mon[35364]: pgmap v3526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:28:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:04 smithi181 ceph-mon[35869]: pgmap v3526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:28:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:04 smithi097 ceph-mon[32231]: pgmap v3526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:28:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:04.912+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:28:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:04.914+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:28:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:04.914+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:28:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:04.914+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:28:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:04.914+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5322.742476 secs 2024-07-26T09:28:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:06 smithi115 ceph-mon[35364]: pgmap v3527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:28:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5322.742476 secs 2024-07-26T09:28:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:06 smithi181 ceph-mon[35869]: pgmap v3527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:28:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5322.742476 secs 2024-07-26T09:28:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:06 smithi097 ceph-mon[32231]: pgmap v3527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:28:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.893+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 msgr2=0x7ff3d40c0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T09:28:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.893+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 msgr2=0x7ff3d40c0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:28:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.893+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0830 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e006a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:28:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.893+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0830 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e006a000 comp rx=0 tx=0).stop 2024-07-26T09:28:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.893+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 msgr2=0x7ff3d40c0830 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T09:28:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.893+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 msgr2=0x7ff3d40c0830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:28:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.893+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40bdff0 0x7ff3d40c0830 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:28:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.893+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:28:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.894+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:28:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:06.895+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e008f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:28:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:07.815+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:28:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:07.816+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:28:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:07.817+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:28:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:07.817+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T09:28:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:08 smithi181 ceph-mon[35869]: pgmap v3528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:08 smithi097 ceph-mon[32231]: pgmap v3528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:08 smithi115 ceph-mon[35364]: pgmap v3528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:10 smithi097 ceph-mon[32231]: pgmap v3529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:10 smithi115 ceph-mon[35364]: pgmap v3529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:10 smithi181 ceph-mon[35869]: pgmap v3529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5327.742603 secs 2024-07-26T09:28:11.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5327.742603 secs 2024-07-26T09:28:11.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5327.742603 secs 2024-07-26T09:28:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:12 smithi097 ceph-mon[32231]: pgmap v3530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:12 smithi115 ceph-mon[35364]: pgmap v3530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:12 smithi181 ceph-mon[35869]: pgmap v3530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:14 smithi115 ceph-mon[35364]: pgmap v3531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:14 smithi181 ceph-mon[35869]: pgmap v3531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:14 smithi097 ceph-mon[32231]: pgmap v3531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5332.742779 secs 2024-07-26T09:28:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:16 smithi115 ceph-mon[35364]: pgmap v3532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5332.742779 secs 2024-07-26T09:28:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:16 smithi181 ceph-mon[35869]: pgmap v3532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5332.742779 secs 2024-07-26T09:28:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:16 smithi097 ceph-mon[32231]: pgmap v3532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:17 smithi115 ceph-mon[35364]: pgmap v3533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:17.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:17 smithi181 ceph-mon[35869]: pgmap v3533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:17 smithi097 ceph-mon[32231]: pgmap v3533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:19.924+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:28:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:19.925+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:28:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:19.925+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:28:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:19.925+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:28:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:19.925+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:20 smithi115 ceph-mon[35364]: pgmap v3534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:28:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:20 smithi181 ceph-mon[35869]: pgmap v3534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:28:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:20 smithi097 ceph-mon[32231]: pgmap v3534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:28:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5337.742913 secs 2024-07-26T09:28:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5337.742913 secs 2024-07-26T09:28:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5337.742913 secs 2024-07-26T09:28:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:22 smithi115 ceph-mon[35364]: pgmap v3535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:22 smithi181 ceph-mon[35869]: pgmap v3535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:22 smithi097 ceph-mon[32231]: pgmap v3535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:24 smithi115 ceph-mon[35364]: pgmap v3536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:24 smithi181 ceph-mon[35869]: pgmap v3536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:24 smithi097 ceph-mon[32231]: pgmap v3536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5342.743054 secs 2024-07-26T09:28:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:26 smithi115 ceph-mon[35364]: pgmap v3537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5342.743054 secs 2024-07-26T09:28:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:26 smithi181 ceph-mon[35869]: pgmap v3537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5342.743054 secs 2024-07-26T09:28:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:26 smithi097 ceph-mon[32231]: pgmap v3537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:28 smithi115 ceph-mon[35364]: pgmap v3538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:28 smithi181 ceph-mon[35869]: pgmap v3538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:28 smithi097 ceph-mon[32231]: pgmap v3538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:28:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:28:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:28:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:30 smithi115 ceph-mon[35364]: pgmap v3539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:28:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:30 smithi115 ceph-mon[35364]: pgmap v3540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:28:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:30 smithi181 ceph-mon[35869]: pgmap v3539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:28:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:30 smithi181 ceph-mon[35869]: pgmap v3540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:28:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:30 smithi097 ceph-mon[32231]: pgmap v3539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:28:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:30 smithi097 ceph-mon[32231]: pgmap v3540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:28:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5347.743210 secs 2024-07-26T09:28:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5347.743210 secs 2024-07-26T09:28:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5347.743210 secs 2024-07-26T09:28:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:32 smithi115 ceph-mon[35364]: pgmap v3541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:28:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:32 smithi181 ceph-mon[35869]: pgmap v3541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:28:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:32 smithi097 ceph-mon[32231]: pgmap v3541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:28:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:34 smithi115 ceph-mon[35364]: pgmap v3542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:34 smithi181 ceph-mon[35869]: pgmap v3542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:34 smithi097 ceph-mon[32231]: pgmap v3542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:34.930+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:28:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:34.932+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:28:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:34.932+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:28:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:34.932+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:28:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:34.932+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:36 smithi115 ceph-mon[35364]: pgmap v3543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5352.743351 secs 2024-07-26T09:28:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:36 smithi181 ceph-mon[35869]: pgmap v3543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5352.743351 secs 2024-07-26T09:28:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:36 smithi097 ceph-mon[32231]: pgmap v3543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5352.743351 secs 2024-07-26T09:28:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:38 smithi115 ceph-mon[35364]: pgmap v3544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:38 smithi181 ceph-mon[35869]: pgmap v3544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:38 smithi097 ceph-mon[32231]: pgmap v3544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:40 smithi115 ceph-mon[35364]: pgmap v3545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:28:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:40 smithi181 ceph-mon[35869]: pgmap v3545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:28:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:40 smithi097 ceph-mon[32231]: pgmap v3545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:28:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5357.743437 secs 2024-07-26T09:28:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5357.743437 secs 2024-07-26T09:28:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5357.743437 secs 2024-07-26T09:28:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:42 smithi115 ceph-mon[35364]: pgmap v3546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:42 smithi181 ceph-mon[35869]: pgmap v3546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:42 smithi097 ceph-mon[32231]: pgmap v3546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:44 smithi115 ceph-mon[35364]: pgmap v3547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:44 smithi181 ceph-mon[35869]: pgmap v3547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:44 smithi097 ceph-mon[32231]: pgmap v3547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:46 smithi115 ceph-mon[35364]: pgmap v3548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:28:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5362.743574 secs 2024-07-26T09:28:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:46 smithi181 ceph-mon[35869]: pgmap v3548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:28:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5362.743574 secs 2024-07-26T09:28:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:46 smithi097 ceph-mon[32231]: pgmap v3548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:28:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5362.743574 secs 2024-07-26T09:28:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:48 smithi115 ceph-mon[35364]: pgmap v3549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:48 smithi181 ceph-mon[35869]: pgmap v3549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:48 smithi097 ceph-mon[32231]: pgmap v3549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:49.938+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:28:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:49.939+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:28:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:49.939+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:28:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:49.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:28:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:49.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:50 smithi181 ceph-mon[35869]: pgmap v3550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:50 smithi097 ceph-mon[32231]: pgmap v3550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:50 smithi115 ceph-mon[35364]: pgmap v3550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5367.743713 secs 2024-07-26T09:28:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5367.743713 secs 2024-07-26T09:28:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5367.743713 secs 2024-07-26T09:28:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:52 smithi115 ceph-mon[35364]: pgmap v3551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:28:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:52 smithi181 ceph-mon[35869]: pgmap v3551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:28:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:52 smithi097 ceph-mon[32231]: pgmap v3551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:28:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:52.825+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:28:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:52.825+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:28:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:52.825+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:28:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:28:52.825+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:28:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:54 smithi115 ceph-mon[35364]: pgmap v3552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:28:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:54 smithi181 ceph-mon[35869]: pgmap v3552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:28:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:54 smithi097 ceph-mon[32231]: pgmap v3552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:28:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:56 smithi115 ceph-mon[35364]: pgmap v3553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5372.743858 secs 2024-07-26T09:28:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:56 smithi181 ceph-mon[35869]: pgmap v3553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5372.743858 secs 2024-07-26T09:28:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:56 smithi097 ceph-mon[32231]: pgmap v3553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:28:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5372.743858 secs 2024-07-26T09:28:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:28:58 smithi115 ceph-mon[35364]: pgmap v3554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:28:58 smithi181 ceph-mon[35869]: pgmap v3554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:28:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:28:58 smithi097 ceph-mon[32231]: pgmap v3554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:00 smithi115 ceph-mon[35364]: pgmap v3555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:29:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:00 smithi181 ceph-mon[35869]: pgmap v3555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:29:00.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:00 smithi097 ceph-mon[32231]: pgmap v3555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:29:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5377.744010 secs 2024-07-26T09:29:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5377.744010 secs 2024-07-26T09:29:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5377.744010 secs 2024-07-26T09:29:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:02 smithi115 ceph-mon[35364]: pgmap v3556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:29:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:02 smithi181 ceph-mon[35869]: pgmap v3556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:29:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:02 smithi097 ceph-mon[32231]: pgmap v3556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:29:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:04 smithi115 ceph-mon[35364]: pgmap v3557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:29:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:04 smithi181 ceph-mon[35869]: pgmap v3557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:29:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:04 smithi097 ceph-mon[32231]: pgmap v3557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:29:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:04.943+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:29:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:04.945+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:29:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:04.945+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:29:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:04.945+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:29:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:04.945+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:06 smithi115 ceph-mon[35364]: pgmap v3558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5382.744143 secs 2024-07-26T09:29:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:06 smithi181 ceph-mon[35869]: pgmap v3558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5382.744143 secs 2024-07-26T09:29:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:06 smithi097 ceph-mon[32231]: pgmap v3558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5382.744143 secs 2024-07-26T09:29:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:08 smithi115 ceph-mon[35364]: pgmap v3559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:08 smithi181 ceph-mon[35869]: pgmap v3559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:08 smithi097 ceph-mon[32231]: pgmap v3559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:10 smithi115 ceph-mon[35364]: pgmap v3560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:10 smithi181 ceph-mon[35869]: pgmap v3560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:10 smithi097 ceph-mon[32231]: pgmap v3560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:11.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5387.744311 secs 2024-07-26T09:29:11.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5387.744311 secs 2024-07-26T09:29:11.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5387.744311 secs 2024-07-26T09:29:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:12 smithi115 ceph-mon[35364]: pgmap v3561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:12 smithi181 ceph-mon[35869]: pgmap v3561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:12 smithi097 ceph-mon[32231]: pgmap v3561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:14 smithi115 ceph-mon[35364]: pgmap v3562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:14 smithi181 ceph-mon[35869]: pgmap v3562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:14 smithi097 ceph-mon[32231]: pgmap v3562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:16 smithi115 ceph-mon[35364]: pgmap v3563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5392.744455 secs 2024-07-26T09:29:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:16 smithi181 ceph-mon[35869]: pgmap v3563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:16.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5392.744455 secs 2024-07-26T09:29:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:16 smithi097 ceph-mon[32231]: pgmap v3563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5392.744455 secs 2024-07-26T09:29:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:18 smithi115 ceph-mon[35364]: pgmap v3564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:18 smithi181 ceph-mon[35869]: pgmap v3564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:18 smithi097 ceph-mon[32231]: pgmap v3564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:19.955+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:29:19.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:19.956+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:29:19.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:19.956+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:29:19.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:19.956+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:29:19.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:19.957+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:20 smithi115 ceph-mon[35364]: pgmap v3565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:20 smithi181 ceph-mon[35869]: pgmap v3565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:20 smithi097 ceph-mon[32231]: pgmap v3565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5397.744547 secs 2024-07-26T09:29:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5397.744547 secs 2024-07-26T09:29:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5397.744547 secs 2024-07-26T09:29:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:22 smithi115 ceph-mon[35364]: pgmap v3566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:22 smithi181 ceph-mon[35869]: pgmap v3566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:22 smithi097 ceph-mon[32231]: pgmap v3566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:23 smithi115 ceph-mon[35364]: pgmap v3567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:23 smithi181 ceph-mon[35869]: pgmap v3567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:23 smithi097 ceph-mon[32231]: pgmap v3567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:25 smithi115 ceph-mon[35364]: pgmap v3568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:25 smithi181 ceph-mon[35869]: pgmap v3568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:25 smithi097 ceph-mon[32231]: pgmap v3568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5402.744688 secs 2024-07-26T09:29:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5402.744688 secs 2024-07-26T09:29:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5402.744688 secs 2024-07-26T09:29:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:27 smithi115 ceph-mon[35364]: pgmap v3569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:27 smithi181 ceph-mon[35869]: pgmap v3569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:27 smithi097 ceph-mon[32231]: pgmap v3569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:29 smithi115 ceph-mon[35364]: pgmap v3570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:29 smithi181 ceph-mon[35869]: pgmap v3570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:29 smithi097 ceph-mon[32231]: pgmap v3570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:29:30.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:29:30.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:29:30.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:29:30.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:30.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:30.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: pgmap v3571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:29:30.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: pgmap v3572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:30.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:29:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: pgmap v3571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: pgmap v3572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:30.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:29:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:31.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: pgmap v3571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:29:31.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: pgmap v3572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:31.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:29:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5407.744848 secs 2024-07-26T09:29:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5407.744848 secs 2024-07-26T09:29:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5407.744848 secs 2024-07-26T09:29:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:32 smithi115 ceph-mon[35364]: pgmap v3573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:32 smithi181 ceph-mon[35869]: pgmap v3573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:32 smithi097 ceph-mon[32231]: pgmap v3573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:34.963+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:29:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:34.964+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:29:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:34.964+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:29:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:34.964+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:34.964+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:34 smithi115 ceph-mon[35364]: pgmap v3574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:34 smithi181 ceph-mon[35869]: pgmap v3574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:34 smithi097 ceph-mon[32231]: pgmap v3574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5412.745016 secs 2024-07-26T09:29:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5412.745016 secs 2024-07-26T09:29:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5412.745016 secs 2024-07-26T09:29:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:36 smithi115 ceph-mon[35364]: pgmap v3575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:36 smithi181 ceph-mon[35869]: pgmap v3575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:36 smithi097 ceph-mon[32231]: pgmap v3575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:38 smithi115 ceph-mon[35364]: pgmap v3576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:38 smithi181 ceph-mon[35869]: pgmap v3576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:38 smithi097 ceph-mon[32231]: pgmap v3576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5417.745182 secs 2024-07-26T09:29:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:40 smithi181 ceph-mon[35869]: pgmap v3577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:29:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5417.745182 secs 2024-07-26T09:29:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:40 smithi097 ceph-mon[32231]: pgmap v3577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:29:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5417.745182 secs 2024-07-26T09:29:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:40 smithi115 ceph-mon[35364]: pgmap v3577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:29:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:42 smithi181 ceph-mon[35869]: pgmap v3578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:42 smithi097 ceph-mon[32231]: pgmap v3578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:42 smithi115 ceph-mon[35364]: pgmap v3578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:44 smithi097 ceph-mon[32231]: pgmap v3579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:44 smithi115 ceph-mon[35364]: pgmap v3579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:44 smithi181 ceph-mon[35869]: pgmap v3579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5422.745341 secs 2024-07-26T09:29:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5422.745341 secs 2024-07-26T09:29:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5422.745341 secs 2024-07-26T09:29:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:47 smithi115 ceph-mon[35364]: pgmap v3580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:47 smithi181 ceph-mon[35869]: pgmap v3580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:47 smithi097 ceph-mon[32231]: pgmap v3580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:49 smithi115 ceph-mon[35364]: pgmap v3581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:49 smithi181 ceph-mon[35869]: pgmap v3581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:49 smithi097 ceph-mon[32231]: pgmap v3581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:49.972+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:29:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:49.973+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:29:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:49.973+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:29:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:49.973+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:29:49.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:29:49.973+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5427.745481 secs 2024-07-26T09:29:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:51 smithi115 ceph-mon[35364]: pgmap v3582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5427.745481 secs 2024-07-26T09:29:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:51 smithi181 ceph-mon[35869]: pgmap v3582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5427.745481 secs 2024-07-26T09:29:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:51 smithi097 ceph-mon[32231]: pgmap v3582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:52 smithi115 ceph-mon[35364]: pgmap v3583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:52 smithi181 ceph-mon[35869]: pgmap v3583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:52 smithi097 ceph-mon[32231]: pgmap v3583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:54 smithi115 ceph-mon[35364]: pgmap v3584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:54 smithi181 ceph-mon[35869]: pgmap v3584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:54 smithi097 ceph-mon[32231]: pgmap v3584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:29:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5432.745626 secs 2024-07-26T09:29:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5432.745626 secs 2024-07-26T09:29:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5432.745626 secs 2024-07-26T09:29:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:56 smithi115 ceph-mon[35364]: pgmap v3585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:56 smithi181 ceph-mon[35869]: pgmap v3585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:56 smithi097 ceph-mon[32231]: pgmap v3585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:29:58 smithi181 ceph-mon[35869]: pgmap v3586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:29:58 smithi097 ceph-mon[32231]: pgmap v3586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:29:58 smithi115 ceph-mon[35364]: pgmap v3586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5437.745708 secs 2024-07-26T09:30:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:00 smithi181 ceph-mon[35869]: pgmap v3587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:30:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5437.745708 secs 2024-07-26T09:30:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:00 smithi097 ceph-mon[32231]: pgmap v3587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:30:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5437.745708 secs 2024-07-26T09:30:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:00 smithi115 ceph-mon[35364]: pgmap v3587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:30:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:02 smithi097 ceph-mon[32231]: pgmap v3588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:02 smithi115 ceph-mon[35364]: pgmap v3588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:02 smithi181 ceph-mon[35869]: pgmap v3588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:04.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:04.978+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:30:04.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:04.979+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:30:04.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:04.979+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:30:04.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:04.979+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:30:04.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:04.979+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:04 smithi097 ceph-mon[32231]: pgmap v3589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:04 smithi115 ceph-mon[35364]: pgmap v3589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:04 smithi181 ceph-mon[35869]: pgmap v3589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5442.745849 secs 2024-07-26T09:30:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5442.745849 secs 2024-07-26T09:30:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5442.745849 secs 2024-07-26T09:30:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:07 smithi115 ceph-mon[35364]: pgmap v3590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:07 smithi181 ceph-mon[35869]: pgmap v3590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:07 smithi097 ceph-mon[32231]: pgmap v3590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:09 smithi115 ceph-mon[35364]: pgmap v3591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:09 smithi181 ceph-mon[35869]: pgmap v3591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:09 smithi097 ceph-mon[32231]: pgmap v3591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:11.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5447.745992 secs 2024-07-26T09:30:11.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:11 smithi115 ceph-mon[35364]: pgmap v3592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:11.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5447.745992 secs 2024-07-26T09:30:11.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:11 smithi181 ceph-mon[35869]: pgmap v3592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:11.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5447.745992 secs 2024-07-26T09:30:11.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:11 smithi097 ceph-mon[32231]: pgmap v3592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:13 smithi097 ceph-mon[32231]: pgmap v3593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:13 smithi115 ceph-mon[35364]: pgmap v3593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:13 smithi181 ceph-mon[35869]: pgmap v3593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:15 smithi115 ceph-mon[35364]: pgmap v3594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:15 smithi181 ceph-mon[35869]: pgmap v3594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:15 smithi097 ceph-mon[32231]: pgmap v3594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5452.746126 secs 2024-07-26T09:30:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5452.746126 secs 2024-07-26T09:30:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5452.746126 secs 2024-07-26T09:30:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:17 smithi115 ceph-mon[35364]: pgmap v3595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:17 smithi181 ceph-mon[35869]: pgmap v3595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:17 smithi097 ceph-mon[32231]: pgmap v3595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:19 smithi115 ceph-mon[35364]: pgmap v3596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:19 smithi181 ceph-mon[35869]: pgmap v3596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:19 smithi097 ceph-mon[32231]: pgmap v3596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:19.989+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:30:19.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:19.990+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:30:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:19.990+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:30:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:19.990+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:30:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:19.990+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5457.746224 secs 2024-07-26T09:30:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:21 smithi115 ceph-mon[35364]: pgmap v3597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5457.746224 secs 2024-07-26T09:30:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:21 smithi181 ceph-mon[35869]: pgmap v3597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5457.746224 secs 2024-07-26T09:30:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:21 smithi097 ceph-mon[32231]: pgmap v3597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:23 smithi097 ceph-mon[32231]: pgmap v3598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:23 smithi115 ceph-mon[35364]: pgmap v3598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:23 smithi181 ceph-mon[35869]: pgmap v3598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:25 smithi115 ceph-mon[35364]: pgmap v3599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:25 smithi181 ceph-mon[35869]: pgmap v3599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:25 smithi097 ceph-mon[32231]: pgmap v3599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5462.746366 secs 2024-07-26T09:30:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5462.746366 secs 2024-07-26T09:30:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5462.746366 secs 2024-07-26T09:30:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:27 smithi115 ceph-mon[35364]: pgmap v3600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:27 smithi181 ceph-mon[35869]: pgmap v3600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:27 smithi097 ceph-mon[32231]: pgmap v3600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:29 smithi115 ceph-mon[35364]: pgmap v3601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:29 smithi181 ceph-mon[35869]: pgmap v3601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:29 smithi097 ceph-mon[32231]: pgmap v3601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5467.746495 secs 2024-07-26T09:30:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:31 smithi115 ceph-mon[35364]: pgmap v3602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5467.746495 secs 2024-07-26T09:30:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:31 smithi181 ceph-mon[35869]: pgmap v3602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:31.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5467.746495 secs 2024-07-26T09:30:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:31 smithi097 ceph-mon[32231]: pgmap v3602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: pgmap v3603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:30:32.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: pgmap v3604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:32.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: pgmap v3603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:30:32.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: pgmap v3604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:32.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:30:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:30:32.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:30:32.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:30:32.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:32.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:32.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: pgmap v3603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:30:32.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: pgmap v3604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:32.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:30:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:33 smithi115 ceph-mon[35364]: pgmap v3605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:30:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:33 smithi181 ceph-mon[35869]: pgmap v3605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:30:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:33 smithi097 ceph-mon[32231]: pgmap v3605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:30:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:34.994+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:30:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:34.995+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:30:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:34.995+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:30:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:34.995+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:30:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:34.995+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:35 smithi115 ceph-mon[35364]: pgmap v3606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:35 smithi181 ceph-mon[35869]: pgmap v3606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:35 smithi097 ceph-mon[32231]: pgmap v3606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5472.746634 secs 2024-07-26T09:30:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5472.746634 secs 2024-07-26T09:30:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5472.746634 secs 2024-07-26T09:30:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:37 smithi115 ceph-mon[35364]: pgmap v3607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:37 smithi181 ceph-mon[35869]: pgmap v3607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:37 smithi097 ceph-mon[32231]: pgmap v3607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:39 smithi115 ceph-mon[35364]: pgmap v3608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:39 smithi181 ceph-mon[35869]: pgmap v3608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:39 smithi097 ceph-mon[32231]: pgmap v3608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5477.746721 secs 2024-07-26T09:30:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5477.746721 secs 2024-07-26T09:30:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5477.746721 secs 2024-07-26T09:30:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:41 smithi097 ceph-mon[32231]: pgmap v3609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:30:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:41 smithi115 ceph-mon[35364]: pgmap v3609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:30:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:41 smithi181 ceph-mon[35869]: pgmap v3609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:30:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:43 smithi115 ceph-mon[35364]: pgmap v3610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:43 smithi181 ceph-mon[35869]: pgmap v3610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:43 smithi097 ceph-mon[32231]: pgmap v3610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:45 smithi115 ceph-mon[35364]: pgmap v3611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:45.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:45 smithi181 ceph-mon[35869]: pgmap v3611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:45 smithi097 ceph-mon[32231]: pgmap v3611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5482.746882 secs 2024-07-26T09:30:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5482.746882 secs 2024-07-26T09:30:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5482.746882 secs 2024-07-26T09:30:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:47 smithi115 ceph-mon[35364]: pgmap v3612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:30:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:47 smithi181 ceph-mon[35869]: pgmap v3612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:30:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:47 smithi097 ceph-mon[32231]: pgmap v3612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:30:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:49 smithi115 ceph-mon[35364]: pgmap v3613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:30:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:49 smithi181 ceph-mon[35869]: pgmap v3613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:30:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:49 smithi097 ceph-mon[32231]: pgmap v3613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:30:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:50.005+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:30:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:50.006+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:30:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:50.007+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:30:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:50.007+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:30:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:30:50.007+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5487.746960 secs 2024-07-26T09:30:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5487.746960 secs 2024-07-26T09:30:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5487.746960 secs 2024-07-26T09:30:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:51 smithi097 ceph-mon[32231]: pgmap v3614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:51 smithi115 ceph-mon[35364]: pgmap v3614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:51 smithi181 ceph-mon[35869]: pgmap v3614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:53 smithi115 ceph-mon[35364]: pgmap v3615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:30:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:53 smithi181 ceph-mon[35869]: pgmap v3615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:30:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:53 smithi097 ceph-mon[32231]: pgmap v3615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:30:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:55 smithi115 ceph-mon[35364]: pgmap v3616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:55 smithi181 ceph-mon[35869]: pgmap v3616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:55 smithi097 ceph-mon[32231]: pgmap v3616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5492.747098 secs 2024-07-26T09:30:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5492.747098 secs 2024-07-26T09:30:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5492.747098 secs 2024-07-26T09:30:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:57 smithi115 ceph-mon[35364]: pgmap v3617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:57 smithi181 ceph-mon[35869]: pgmap v3617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:57 smithi097 ceph-mon[32231]: pgmap v3617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:30:59 smithi115 ceph-mon[35364]: pgmap v3618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:30:59 smithi181 ceph-mon[35869]: pgmap v3618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:30:59 smithi097 ceph-mon[32231]: pgmap v3618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:31:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5497.747267 secs 2024-07-26T09:31:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5497.747267 secs 2024-07-26T09:31:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5497.747267 secs 2024-07-26T09:31:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:01 smithi097 ceph-mon[32231]: pgmap v3619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:01 smithi115 ceph-mon[35364]: pgmap v3619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:01 smithi181 ceph-mon[35869]: pgmap v3619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:03 smithi115 ceph-mon[35364]: pgmap v3620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:03 smithi181 ceph-mon[35869]: pgmap v3620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:03.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:03 smithi097 ceph-mon[32231]: pgmap v3620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:05.010+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:31:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:05.011+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:31:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:05.011+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:31:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:05.011+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:31:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:05.011+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:05 smithi115 ceph-mon[35364]: pgmap v3621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:31:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:05 smithi181 ceph-mon[35869]: pgmap v3621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:31:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:05 smithi097 ceph-mon[32231]: pgmap v3621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:31:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5502.747397 secs 2024-07-26T09:31:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5502.747397 secs 2024-07-26T09:31:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5502.747397 secs 2024-07-26T09:31:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:07 smithi115 ceph-mon[35364]: pgmap v3622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:07 smithi181 ceph-mon[35869]: pgmap v3622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:07 smithi097 ceph-mon[32231]: pgmap v3622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:07.849+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:31:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:07.849+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:31:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:07.850+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T09:31:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:07.850+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:31:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:09 smithi115 ceph-mon[35364]: pgmap v3623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:09 smithi181 ceph-mon[35869]: pgmap v3623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:09 smithi097 ceph-mon[32231]: pgmap v3623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5507.747493 secs 2024-07-26T09:31:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5507.747493 secs 2024-07-26T09:31:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5507.747493 secs 2024-07-26T09:31:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:11 smithi097 ceph-mon[32231]: pgmap v3624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:11 smithi115 ceph-mon[35364]: pgmap v3624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:11 smithi181 ceph-mon[35869]: pgmap v3624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:13 smithi115 ceph-mon[35364]: pgmap v3625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:13 smithi181 ceph-mon[35869]: pgmap v3625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:13 smithi097 ceph-mon[32231]: pgmap v3625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:15 smithi115 ceph-mon[35364]: pgmap v3626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:15 smithi181 ceph-mon[35869]: pgmap v3626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:15 smithi097 ceph-mon[32231]: pgmap v3626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5512.747631 secs 2024-07-26T09:31:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5512.747631 secs 2024-07-26T09:31:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5512.747631 secs 2024-07-26T09:31:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:17 smithi115 ceph-mon[35364]: pgmap v3627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:17 smithi181 ceph-mon[35869]: pgmap v3627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:17 smithi097 ceph-mon[32231]: pgmap v3627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:19 smithi115 ceph-mon[35364]: pgmap v3628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:19 smithi181 ceph-mon[35869]: pgmap v3628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:19 smithi097 ceph-mon[32231]: pgmap v3628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:20.021+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:31:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:20.023+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:31:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:20.023+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:31:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:20.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:31:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:20.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5517.747767 secs 2024-07-26T09:31:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5517.747767 secs 2024-07-26T09:31:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5517.747767 secs 2024-07-26T09:31:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:21 smithi097 ceph-mon[32231]: pgmap v3629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:21 smithi115 ceph-mon[35364]: pgmap v3629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:22.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:21 smithi181 ceph-mon[35869]: pgmap v3629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:23 smithi115 ceph-mon[35364]: pgmap v3630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:23 smithi181 ceph-mon[35869]: pgmap v3630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:23 smithi097 ceph-mon[32231]: pgmap v3630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:25 smithi115 ceph-mon[35364]: pgmap v3631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:25 smithi181 ceph-mon[35869]: pgmap v3631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:25 smithi097 ceph-mon[32231]: pgmap v3631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5522.747910 secs 2024-07-26T09:31:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5522.747910 secs 2024-07-26T09:31:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5522.747910 secs 2024-07-26T09:31:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:27 smithi181 ceph-mon[35869]: pgmap v3632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:27 smithi097 ceph-mon[32231]: pgmap v3632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:27 smithi115 ceph-mon[35364]: pgmap v3632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:29 smithi115 ceph-mon[35364]: pgmap v3633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:29 smithi181 ceph-mon[35869]: pgmap v3633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:29 smithi097 ceph-mon[32231]: pgmap v3633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5527.748068 secs 2024-07-26T09:31:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5527.748068 secs 2024-07-26T09:31:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5527.748068 secs 2024-07-26T09:31:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:31 smithi097 ceph-mon[32231]: pgmap v3634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:31 smithi115 ceph-mon[35364]: pgmap v3634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:31 smithi181 ceph-mon[35869]: pgmap v3634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:33.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:34 smithi115 ceph-mon[35364]: pgmap v3635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:34 smithi115 ceph-mon[35364]: pgmap v3636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T09:31:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:34 smithi181 ceph-mon[35869]: pgmap v3635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:34 smithi181 ceph-mon[35869]: pgmap v3636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T09:31:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:34 smithi097 ceph-mon[32231]: pgmap v3635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:34 smithi097 ceph-mon[32231]: pgmap v3636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T09:31:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:31:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:35.027+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:31:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:35.029+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:31:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:35.029+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:31:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:35.029+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:31:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:35.029+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5532.748266 secs 2024-07-26T09:31:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:36 smithi115 ceph-mon[35364]: pgmap v3637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T09:31:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5532.748266 secs 2024-07-26T09:31:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:36 smithi181 ceph-mon[35869]: pgmap v3637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T09:31:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5532.748266 secs 2024-07-26T09:31:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:36 smithi097 ceph-mon[32231]: pgmap v3637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T09:31:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:38 smithi115 ceph-mon[35364]: pgmap v3638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:38 smithi181 ceph-mon[35869]: pgmap v3638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:38 smithi097 ceph-mon[32231]: pgmap v3638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:40 smithi115 ceph-mon[35364]: pgmap v3639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:40 smithi181 ceph-mon[35869]: pgmap v3639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:40 smithi097 ceph-mon[32231]: pgmap v3639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5537.748293 secs 2024-07-26T09:31:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5537.748293 secs 2024-07-26T09:31:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5537.748293 secs 2024-07-26T09:31:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:42 smithi115 ceph-mon[35364]: pgmap v3640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:42 smithi181 ceph-mon[35869]: pgmap v3640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:42 smithi097 ceph-mon[32231]: pgmap v3640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:43 smithi115 ceph-mon[35364]: pgmap v3641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:43 smithi181 ceph-mon[35869]: pgmap v3641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:43 smithi097 ceph-mon[32231]: pgmap v3641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:31:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:45 smithi115 ceph-mon[35364]: pgmap v3642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:31:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:45 smithi181 ceph-mon[35869]: pgmap v3642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:31:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:45 smithi097 ceph-mon[32231]: pgmap v3642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:31:46.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5542.748432 secs 2024-07-26T09:31:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5542.748432 secs 2024-07-26T09:31:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5542.748432 secs 2024-07-26T09:31:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:47 smithi181 ceph-mon[35869]: pgmap v3643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:47.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:47 smithi097 ceph-mon[32231]: pgmap v3643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:47 smithi115 ceph-mon[35364]: pgmap v3643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:31:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:49 smithi115 ceph-mon[35364]: pgmap v3644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:49 smithi181 ceph-mon[35869]: pgmap v3644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:49 smithi097 ceph-mon[32231]: pgmap v3644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:50.033 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:50.031+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:31:50.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:50.032+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:31:50.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:50.032+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:31:50.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:50.032+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:31:50.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:31:50.032+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5547.748573 secs 2024-07-26T09:31:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5547.748573 secs 2024-07-26T09:31:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5547.748573 secs 2024-07-26T09:31:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:51 smithi097 ceph-mon[32231]: pgmap v3645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:51 smithi115 ceph-mon[35364]: pgmap v3645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:51 smithi181 ceph-mon[35869]: pgmap v3645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:53 smithi115 ceph-mon[35364]: pgmap v3646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:53 smithi181 ceph-mon[35869]: pgmap v3646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:53 smithi097 ceph-mon[32231]: pgmap v3646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:55 smithi115 ceph-mon[35364]: pgmap v3647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:55 smithi181 ceph-mon[35869]: pgmap v3647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:55 smithi097 ceph-mon[32231]: pgmap v3647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5552.748707 secs 2024-07-26T09:31:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5552.748707 secs 2024-07-26T09:31:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5552.748707 secs 2024-07-26T09:31:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:57 smithi181 ceph-mon[35869]: pgmap v3648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:57 smithi097 ceph-mon[32231]: pgmap v3648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:57 smithi115 ceph-mon[35364]: pgmap v3648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:31:59 smithi115 ceph-mon[35364]: pgmap v3649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:31:59 smithi181 ceph-mon[35869]: pgmap v3649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:31:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:31:59 smithi097 ceph-mon[32231]: pgmap v3649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5557.748847 secs 2024-07-26T09:32:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5557.748847 secs 2024-07-26T09:32:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5557.748847 secs 2024-07-26T09:32:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:01 smithi097 ceph-mon[32231]: pgmap v3650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:01 smithi115 ceph-mon[35364]: pgmap v3650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:01 smithi181 ceph-mon[35869]: pgmap v3650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:03 smithi115 ceph-mon[35364]: pgmap v3651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:03 smithi181 ceph-mon[35869]: pgmap v3651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:03 smithi097 ceph-mon[32231]: pgmap v3651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:05.037+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:32:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:05.038+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:32:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:05.038+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:32:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:05.038+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:32:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:05.039+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:05 smithi115 ceph-mon[35364]: pgmap v3652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:05 smithi181 ceph-mon[35869]: pgmap v3652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:05 smithi097 ceph-mon[32231]: pgmap v3652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5562.748983 secs 2024-07-26T09:32:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5562.748983 secs 2024-07-26T09:32:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5562.748983 secs 2024-07-26T09:32:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:07 smithi097 ceph-mon[32231]: pgmap v3653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:07 smithi115 ceph-mon[35364]: pgmap v3653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:07 smithi181 ceph-mon[35869]: pgmap v3653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:09 smithi115 ceph-mon[35364]: pgmap v3654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:09 smithi181 ceph-mon[35869]: pgmap v3654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:09 smithi097 ceph-mon[32231]: pgmap v3654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5567.749121 secs 2024-07-26T09:32:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5567.749121 secs 2024-07-26T09:32:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5567.749121 secs 2024-07-26T09:32:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:11 smithi097 ceph-mon[32231]: pgmap v3655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:11 smithi115 ceph-mon[35364]: pgmap v3655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:11 smithi181 ceph-mon[35869]: pgmap v3655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:13 smithi115 ceph-mon[35364]: pgmap v3656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:13 smithi181 ceph-mon[35869]: pgmap v3656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:13 smithi097 ceph-mon[32231]: pgmap v3656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:15 smithi115 ceph-mon[35364]: pgmap v3657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:15 smithi181 ceph-mon[35869]: pgmap v3657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:15 smithi097 ceph-mon[32231]: pgmap v3657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5572.749220 secs 2024-07-26T09:32:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5572.749220 secs 2024-07-26T09:32:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5572.749220 secs 2024-07-26T09:32:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:17 smithi115 ceph-mon[35364]: pgmap v3658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:17 smithi181 ceph-mon[35869]: pgmap v3658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:17 smithi097 ceph-mon[32231]: pgmap v3658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:19 smithi115 ceph-mon[35364]: pgmap v3659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:19 smithi181 ceph-mon[35869]: pgmap v3659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:19 smithi097 ceph-mon[32231]: pgmap v3659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:20.050+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:32:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:20.051+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:32:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:20.051+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:32:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:20.051+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:32:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:20.051+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5577.749291 secs 2024-07-26T09:32:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5577.749291 secs 2024-07-26T09:32:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5577.749291 secs 2024-07-26T09:32:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:21 smithi115 ceph-mon[35364]: pgmap v3660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:21 smithi181 ceph-mon[35869]: pgmap v3660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:21 smithi097 ceph-mon[32231]: pgmap v3660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:23 smithi115 ceph-mon[35364]: pgmap v3661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:23 smithi181 ceph-mon[35869]: pgmap v3661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:23 smithi097 ceph-mon[32231]: pgmap v3661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:25 smithi115 ceph-mon[35364]: pgmap v3662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:25 smithi181 ceph-mon[35869]: pgmap v3662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:25 smithi097 ceph-mon[32231]: pgmap v3662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5582.749458 secs 2024-07-26T09:32:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5582.749458 secs 2024-07-26T09:32:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5582.749458 secs 2024-07-26T09:32:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:27 smithi097 ceph-mon[32231]: pgmap v3663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:27 smithi115 ceph-mon[35364]: pgmap v3663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:27 smithi181 ceph-mon[35869]: pgmap v3663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:29 smithi115 ceph-mon[35364]: pgmap v3664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:29 smithi181 ceph-mon[35869]: pgmap v3664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:29 smithi097 ceph-mon[32231]: pgmap v3664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5587.749616 secs 2024-07-26T09:32:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5587.749616 secs 2024-07-26T09:32:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5587.749616 secs 2024-07-26T09:32:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:31 smithi115 ceph-mon[35364]: pgmap v3665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:31 smithi181 ceph-mon[35869]: pgmap v3665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:31 smithi097 ceph-mon[32231]: pgmap v3665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:33 smithi115 ceph-mon[35364]: pgmap v3666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:33 smithi181 ceph-mon[35869]: pgmap v3666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:33 smithi097 ceph-mon[32231]: pgmap v3666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:35.055+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:32:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:35.057+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:32:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:35.057+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:32:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:35.057+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:32:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:35.057+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:35 smithi097 ceph-mon[32231]: pgmap v3667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:35 smithi097 ceph-mon[32231]: pgmap v3668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:35 smithi115 ceph-mon[35364]: pgmap v3667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:35 smithi115 ceph-mon[35364]: pgmap v3668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:35 smithi181 ceph-mon[35869]: pgmap v3667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:35 smithi181 ceph-mon[35869]: pgmap v3668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:32:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5592.749706 secs 2024-07-26T09:32:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5592.749706 secs 2024-07-26T09:32:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5592.749706 secs 2024-07-26T09:32:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:37 smithi115 ceph-mon[35364]: pgmap v3669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:37 smithi181 ceph-mon[35869]: pgmap v3669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:37 smithi097 ceph-mon[32231]: pgmap v3669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:40 smithi115 ceph-mon[35364]: pgmap v3670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:40 smithi181 ceph-mon[35869]: pgmap v3670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:40 smithi097 ceph-mon[32231]: pgmap v3670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5597.749874 secs 2024-07-26T09:32:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5597.749874 secs 2024-07-26T09:32:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5597.749874 secs 2024-07-26T09:32:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:42 smithi115 ceph-mon[35364]: pgmap v3671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:32:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:42 smithi181 ceph-mon[35869]: pgmap v3671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:32:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:42 smithi097 ceph-mon[32231]: pgmap v3671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:32:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:43 smithi115 ceph-mon[35364]: pgmap v3672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:32:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:43 smithi181 ceph-mon[35869]: pgmap v3672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:32:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:43 smithi097 ceph-mon[32231]: pgmap v3672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:32:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5602.750037 secs 2024-07-26T09:32:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:46 smithi115 ceph-mon[35364]: pgmap v3673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5602.750037 secs 2024-07-26T09:32:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:46 smithi181 ceph-mon[35869]: pgmap v3673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5602.750037 secs 2024-07-26T09:32:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:46 smithi097 ceph-mon[32231]: pgmap v3673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:32:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:48 smithi115 ceph-mon[35364]: pgmap v3674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:48 smithi181 ceph-mon[35869]: pgmap v3674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:48 smithi097 ceph-mon[32231]: pgmap v3674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:49 smithi181 ceph-mon[35869]: pgmap v3675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:49 smithi097 ceph-mon[32231]: pgmap v3675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:49 smithi115 ceph-mon[35364]: pgmap v3675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:50.067+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:32:50.071 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:50.069+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:32:50.071 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:50.069+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:32:50.071 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:50.069+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:32:50.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:32:50.069+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5607.750176 secs 2024-07-26T09:32:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5607.750176 secs 2024-07-26T09:32:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5607.750176 secs 2024-07-26T09:32:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:51 smithi115 ceph-mon[35364]: pgmap v3676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:52.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:51 smithi181 ceph-mon[35869]: pgmap v3676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:51 smithi097 ceph-mon[32231]: pgmap v3676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:54 smithi115 ceph-mon[35364]: pgmap v3677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:54 smithi181 ceph-mon[35869]: pgmap v3677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:54 smithi097 ceph-mon[32231]: pgmap v3677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5612.750319 secs 2024-07-26T09:32:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:56 smithi115 ceph-mon[35364]: pgmap v3678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5612.750319 secs 2024-07-26T09:32:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:56 smithi181 ceph-mon[35869]: pgmap v3678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5612.750319 secs 2024-07-26T09:32:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:56 smithi097 ceph-mon[32231]: pgmap v3678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:32:57 smithi181 ceph-mon[35869]: pgmap v3679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:32:57 smithi097 ceph-mon[32231]: pgmap v3679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:32:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:32:57 smithi115 ceph-mon[35364]: pgmap v3679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:00 smithi115 ceph-mon[35364]: pgmap v3680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:00 smithi181 ceph-mon[35869]: pgmap v3680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:00 smithi097 ceph-mon[32231]: pgmap v3680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5617.750404 secs 2024-07-26T09:33:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5617.750404 secs 2024-07-26T09:33:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5617.750404 secs 2024-07-26T09:33:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:02 smithi115 ceph-mon[35364]: pgmap v3681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:02 smithi181 ceph-mon[35869]: pgmap v3681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:02 smithi097 ceph-mon[32231]: pgmap v3681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:03 smithi115 ceph-mon[35364]: pgmap v3682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:03 smithi181 ceph-mon[35869]: pgmap v3682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:03 smithi097 ceph-mon[32231]: pgmap v3682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:05.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:05.075+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:33:05.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:05.076+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:33:05.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:05.076+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:33:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:05.076+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:33:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:05.076+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5622.750532 secs 2024-07-26T09:33:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:06 smithi115 ceph-mon[35364]: pgmap v3683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5622.750532 secs 2024-07-26T09:33:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:06 smithi181 ceph-mon[35869]: pgmap v3683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:06.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5622.750532 secs 2024-07-26T09:33:06.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:06 smithi097 ceph-mon[32231]: pgmap v3683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:07 smithi181 ceph-mon[35869]: pgmap v3684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:07 smithi097 ceph-mon[32231]: pgmap v3684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:07 smithi115 ceph-mon[35364]: pgmap v3684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:10 smithi115 ceph-mon[35364]: pgmap v3685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:10 smithi181 ceph-mon[35869]: pgmap v3685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:10 smithi097 ceph-mon[32231]: pgmap v3685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5627.750670 secs 2024-07-26T09:33:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5627.750670 secs 2024-07-26T09:33:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5627.750670 secs 2024-07-26T09:33:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:12 smithi115 ceph-mon[35364]: pgmap v3686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:12 smithi181 ceph-mon[35869]: pgmap v3686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:12 smithi097 ceph-mon[32231]: pgmap v3686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:13 smithi115 ceph-mon[35364]: pgmap v3687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:13 smithi181 ceph-mon[35869]: pgmap v3687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:13 smithi097 ceph-mon[32231]: pgmap v3687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5632.750811 secs 2024-07-26T09:33:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:16 smithi115 ceph-mon[35364]: pgmap v3688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5632.750811 secs 2024-07-26T09:33:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:16 smithi181 ceph-mon[35869]: pgmap v3688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5632.750811 secs 2024-07-26T09:33:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:16 smithi097 ceph-mon[32231]: pgmap v3688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:17 smithi115 ceph-mon[35364]: pgmap v3689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:17 smithi181 ceph-mon[35869]: pgmap v3689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:17 smithi097 ceph-mon[32231]: pgmap v3689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:20.087+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:33:20.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:20.088+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:33:20.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:20.088+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:33:20.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:20.088+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:33:20.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:20.088+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:20 smithi115 ceph-mon[35364]: pgmap v3690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:20 smithi181 ceph-mon[35869]: pgmap v3690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:20 smithi097 ceph-mon[32231]: pgmap v3690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5637.750964 secs 2024-07-26T09:33:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:21 smithi115 ceph-mon[35364]: pgmap v3691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5637.750964 secs 2024-07-26T09:33:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:21 smithi181 ceph-mon[35869]: pgmap v3691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5637.750964 secs 2024-07-26T09:33:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:21 smithi097 ceph-mon[32231]: pgmap v3691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:23 smithi097 ceph-mon[32231]: pgmap v3692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:23 smithi115 ceph-mon[35364]: pgmap v3692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:23 smithi181 ceph-mon[35869]: pgmap v3692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:25 smithi097 ceph-mon[32231]: pgmap v3693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:25 smithi115 ceph-mon[35364]: pgmap v3693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:25 smithi181 ceph-mon[35869]: pgmap v3693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5642.751096 secs 2024-07-26T09:33:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5642.751096 secs 2024-07-26T09:33:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5642.751096 secs 2024-07-26T09:33:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:27 smithi097 ceph-mon[32231]: pgmap v3694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:27 smithi115 ceph-mon[35364]: pgmap v3694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:27 smithi181 ceph-mon[35869]: pgmap v3694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:29 smithi097 ceph-mon[32231]: pgmap v3695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:29 smithi115 ceph-mon[35364]: pgmap v3695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:29 smithi181 ceph-mon[35869]: pgmap v3695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5647.751277 secs 2024-07-26T09:33:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5647.751277 secs 2024-07-26T09:33:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5647.751277 secs 2024-07-26T09:33:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:31 smithi115 ceph-mon[35364]: pgmap v3696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:31 smithi181 ceph-mon[35869]: pgmap v3696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:31 smithi097 ceph-mon[32231]: pgmap v3696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:33 smithi097 ceph-mon[32231]: pgmap v3697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:33 smithi115 ceph-mon[35364]: pgmap v3697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:33 smithi181 ceph-mon[35869]: pgmap v3697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:35.092+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:33:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:35.094+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:33:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:35.094+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:33:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:35.094+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:33:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:35.094+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:35 smithi097 ceph-mon[32231]: pgmap v3698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:35 smithi115 ceph-mon[35364]: pgmap v3698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:35 smithi181 ceph-mon[35869]: pgmap v3698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5652.751410 secs 2024-07-26T09:33:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:36.815 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5652.751410 secs 2024-07-26T09:33:36.815 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5652.751410 secs 2024-07-26T09:33:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:37 smithi097 ceph-mon[32231]: pgmap v3699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:37 smithi115 ceph-mon[35364]: pgmap v3699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:37 smithi181 ceph-mon[35869]: pgmap v3699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:39 smithi115 ceph-mon[35364]: pgmap v3700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:39 smithi181 ceph-mon[35869]: pgmap v3700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:40.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:39 smithi097 ceph-mon[32231]: pgmap v3700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5657.751547 secs 2024-07-26T09:33:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:40 smithi097 ceph-mon[32231]: pgmap v3701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5657.751547 secs 2024-07-26T09:33:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:40 smithi115 ceph-mon[35364]: pgmap v3701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5657.751547 secs 2024-07-26T09:33:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:41.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:41.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:40 smithi181 ceph-mon[35869]: pgmap v3701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:41.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:33:43.168 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:43 smithi115 ceph-mon[35364]: pgmap v3702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:33:43.170 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:43 smithi181 ceph-mon[35869]: pgmap v3702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:33:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:43 smithi097 ceph-mon[32231]: pgmap v3702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:33:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:45 smithi115 ceph-mon[35364]: pgmap v3703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:45 smithi181 ceph-mon[35869]: pgmap v3703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:45 smithi097 ceph-mon[32231]: pgmap v3703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5662.751695 secs 2024-07-26T09:33:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5662.751695 secs 2024-07-26T09:33:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5662.751695 secs 2024-07-26T09:33:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:47 smithi115 ceph-mon[35364]: pgmap v3704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:33:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:47 smithi181 ceph-mon[35869]: pgmap v3704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:33:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:47 smithi097 ceph-mon[32231]: pgmap v3704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:33:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:49 smithi115 ceph-mon[35364]: pgmap v3705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:33:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:49 smithi181 ceph-mon[35869]: pgmap v3705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:33:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:49 smithi097 ceph-mon[32231]: pgmap v3705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:33:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:50.105+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:33:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:50.106+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:33:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:50.106+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:33:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:50.106+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:33:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:33:50.106+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5667.751833 secs 2024-07-26T09:33:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:51 smithi115 ceph-mon[35364]: pgmap v3706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5667.751833 secs 2024-07-26T09:33:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:51 smithi181 ceph-mon[35869]: pgmap v3706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5667.751833 secs 2024-07-26T09:33:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:51 smithi097 ceph-mon[32231]: pgmap v3706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:53 smithi115 ceph-mon[35364]: pgmap v3707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:53 smithi181 ceph-mon[35869]: pgmap v3707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:53 smithi097 ceph-mon[32231]: pgmap v3707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:55 smithi115 ceph-mon[35364]: pgmap v3708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:33:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:55 smithi181 ceph-mon[35869]: pgmap v3708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:33:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:55 smithi097 ceph-mon[32231]: pgmap v3708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:33:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5672.752008 secs 2024-07-26T09:33:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5672.752008 secs 2024-07-26T09:33:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5672.752008 secs 2024-07-26T09:33:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:57 smithi115 ceph-mon[35364]: pgmap v3709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:57 smithi181 ceph-mon[35869]: pgmap v3709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:57 smithi097 ceph-mon[32231]: pgmap v3709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:33:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:33:59 smithi115 ceph-mon[35364]: pgmap v3710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:33:59 smithi181 ceph-mon[35869]: pgmap v3710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:33:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:33:59 smithi097 ceph-mon[32231]: pgmap v3710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5677.752175 secs 2024-07-26T09:34:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:01 smithi115 ceph-mon[35364]: pgmap v3711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:34:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5677.752175 secs 2024-07-26T09:34:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:01 smithi181 ceph-mon[35869]: pgmap v3711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:34:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5677.752175 secs 2024-07-26T09:34:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:01 smithi097 ceph-mon[32231]: pgmap v3711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:34:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:03 smithi115 ceph-mon[35364]: pgmap v3712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:34:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:03 smithi181 ceph-mon[35869]: pgmap v3712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:34:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:03 smithi097 ceph-mon[32231]: pgmap v3712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:34:05.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:05.111+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:34:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:05.112+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:34:05.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:05.112+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:34:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:05.112+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:34:05.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:05.112+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:05 smithi115 ceph-mon[35364]: pgmap v3713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:34:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:05 smithi181 ceph-mon[35869]: pgmap v3713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:34:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:05 smithi097 ceph-mon[32231]: pgmap v3713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:34:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5682.752319 secs 2024-07-26T09:34:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5682.752319 secs 2024-07-26T09:34:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5682.752319 secs 2024-07-26T09:34:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:07 smithi115 ceph-mon[35364]: pgmap v3714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:07 smithi181 ceph-mon[35869]: pgmap v3714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:07 smithi097 ceph-mon[32231]: pgmap v3714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:07.881+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:34:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:07.881+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:34:07.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:07.882+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:34:07.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:07.882+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f070 con 0x7ff3f01061a0 2024-07-26T09:34:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:09 smithi115 ceph-mon[35364]: pgmap v3715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:34:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:09 smithi181 ceph-mon[35869]: pgmap v3715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:34:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:09 smithi097 ceph-mon[32231]: pgmap v3715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:34:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5687.752460 secs 2024-07-26T09:34:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:11 smithi115 ceph-mon[35364]: pgmap v3716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:34:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5687.752460 secs 2024-07-26T09:34:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:11 smithi181 ceph-mon[35869]: pgmap v3716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:34:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5687.752460 secs 2024-07-26T09:34:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:11 smithi097 ceph-mon[32231]: pgmap v3716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:34:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:13 smithi115 ceph-mon[35364]: pgmap v3717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:13 smithi181 ceph-mon[35869]: pgmap v3717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:13 smithi097 ceph-mon[32231]: pgmap v3717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:15 smithi115 ceph-mon[35364]: pgmap v3718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:15 smithi181 ceph-mon[35869]: pgmap v3718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:15 smithi097 ceph-mon[32231]: pgmap v3718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5692.752599 secs 2024-07-26T09:34:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5692.752599 secs 2024-07-26T09:34:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5692.752599 secs 2024-07-26T09:34:17.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:17 smithi115 ceph-mon[35364]: pgmap v3719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:17 smithi181 ceph-mon[35869]: pgmap v3719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:17 smithi097 ceph-mon[32231]: pgmap v3719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:19 smithi115 ceph-mon[35364]: pgmap v3720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:19 smithi181 ceph-mon[35869]: pgmap v3720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:19 smithi097 ceph-mon[32231]: pgmap v3720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:20.122+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:34:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:20.124+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:34:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:20.124+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:34:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:20.124+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:34:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:20.124+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5697.752754 secs 2024-07-26T09:34:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:21 smithi115 ceph-mon[35364]: pgmap v3721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5697.752754 secs 2024-07-26T09:34:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:21 smithi181 ceph-mon[35869]: pgmap v3721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5697.752754 secs 2024-07-26T09:34:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:21 smithi097 ceph-mon[32231]: pgmap v3721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:23 smithi181 ceph-mon[35869]: pgmap v3722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:23 smithi097 ceph-mon[32231]: pgmap v3722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:23 smithi115 ceph-mon[35364]: pgmap v3722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:25 smithi097 ceph-mon[32231]: pgmap v3723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:25 smithi115 ceph-mon[35364]: pgmap v3723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:25 smithi181 ceph-mon[35869]: pgmap v3723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5702.752914 secs 2024-07-26T09:34:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5702.752914 secs 2024-07-26T09:34:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5702.752914 secs 2024-07-26T09:34:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:27 smithi115 ceph-mon[35364]: pgmap v3724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:27 smithi181 ceph-mon[35869]: pgmap v3724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:27 smithi097 ceph-mon[32231]: pgmap v3724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:29 smithi115 ceph-mon[35364]: pgmap v3725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:29 smithi181 ceph-mon[35869]: pgmap v3725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:29 smithi097 ceph-mon[32231]: pgmap v3725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5707.753055 secs 2024-07-26T09:34:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:31 smithi115 ceph-mon[35364]: pgmap v3726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5707.753055 secs 2024-07-26T09:34:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:31 smithi181 ceph-mon[35869]: pgmap v3726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5707.753055 secs 2024-07-26T09:34:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:31 smithi097 ceph-mon[32231]: pgmap v3726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:33 smithi115 ceph-mon[35364]: pgmap v3727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:33 smithi181 ceph-mon[35869]: pgmap v3727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:33 smithi097 ceph-mon[32231]: pgmap v3727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:35.131 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:35.129+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:34:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:35.130+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:34:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:35.130+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:34:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:35.130+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:34:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:35.130+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:35 smithi115 ceph-mon[35364]: pgmap v3728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:35 smithi181 ceph-mon[35869]: pgmap v3728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:35 smithi097 ceph-mon[32231]: pgmap v3728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:34:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5712.753194 secs 2024-07-26T09:34:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5712.753194 secs 2024-07-26T09:34:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5712.753194 secs 2024-07-26T09:34:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:37 smithi115 ceph-mon[35364]: pgmap v3729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:37 smithi181 ceph-mon[35869]: pgmap v3729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:37 smithi097 ceph-mon[32231]: pgmap v3729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:39 smithi115 ceph-mon[35364]: pgmap v3730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:39 smithi181 ceph-mon[35869]: pgmap v3730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:39 smithi097 ceph-mon[32231]: pgmap v3730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5717.753328 secs 2024-07-26T09:34:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:41 smithi115 ceph-mon[35364]: pgmap v3731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5717.753328 secs 2024-07-26T09:34:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:41 smithi181 ceph-mon[35869]: pgmap v3731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5717.753328 secs 2024-07-26T09:34:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:41 smithi097 ceph-mon[32231]: pgmap v3731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:42 smithi115 ceph-mon[35364]: pgmap v3732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:34:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:42 smithi115 ceph-mon[35364]: pgmap v3733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:34:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:42 smithi181 ceph-mon[35869]: pgmap v3732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:34:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:42 smithi181 ceph-mon[35869]: pgmap v3733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:34:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:42 smithi097 ceph-mon[32231]: pgmap v3732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:34:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:42 smithi097 ceph-mon[32231]: pgmap v3733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:34:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:43 smithi115 ceph-mon[35364]: pgmap v3734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:34:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:43 smithi181 ceph-mon[35869]: pgmap v3734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:34:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:43 smithi097 ceph-mon[32231]: pgmap v3734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:34:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:45 smithi115 ceph-mon[35364]: pgmap v3735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:45 smithi181 ceph-mon[35869]: pgmap v3735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:45 smithi097 ceph-mon[32231]: pgmap v3735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5722.753415 secs 2024-07-26T09:34:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5722.753415 secs 2024-07-26T09:34:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5722.753415 secs 2024-07-26T09:34:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:47 smithi115 ceph-mon[35364]: pgmap v3736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:47 smithi181 ceph-mon[35869]: pgmap v3736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:47 smithi097 ceph-mon[32231]: pgmap v3736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:50.140+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:34:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:50.142+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:34:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:50.142+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:34:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:50.142+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:34:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:34:50.142+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:50 smithi115 ceph-mon[35364]: pgmap v3737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:50 smithi181 ceph-mon[35869]: pgmap v3737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:50 smithi097 ceph-mon[32231]: pgmap v3737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5727.753541 secs 2024-07-26T09:34:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5727.753541 secs 2024-07-26T09:34:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5727.753541 secs 2024-07-26T09:34:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:52 smithi115 ceph-mon[35364]: pgmap v3738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:34:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:52 smithi181 ceph-mon[35869]: pgmap v3738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:34:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:52 smithi097 ceph-mon[32231]: pgmap v3738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:34:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:54 smithi115 ceph-mon[35364]: pgmap v3739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:54 smithi181 ceph-mon[35869]: pgmap v3739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:54 smithi097 ceph-mon[32231]: pgmap v3739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5732.753678 secs 2024-07-26T09:34:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:56 smithi115 ceph-mon[35364]: pgmap v3740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5732.753678 secs 2024-07-26T09:34:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:56 smithi181 ceph-mon[35869]: pgmap v3740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5732.753678 secs 2024-07-26T09:34:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:56 smithi097 ceph-mon[32231]: pgmap v3740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:34:58 smithi115 ceph-mon[35364]: pgmap v3741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:34:58 smithi181 ceph-mon[35869]: pgmap v3741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:34:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:34:58 smithi097 ceph-mon[32231]: pgmap v3741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:00 smithi115 ceph-mon[35364]: pgmap v3742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:00 smithi181 ceph-mon[35869]: pgmap v3742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:00 smithi097 ceph-mon[32231]: pgmap v3742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5737.753829 secs 2024-07-26T09:35:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5737.753829 secs 2024-07-26T09:35:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5737.753829 secs 2024-07-26T09:35:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:02 smithi115 ceph-mon[35364]: pgmap v3743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:02 smithi181 ceph-mon[35869]: pgmap v3743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:02 smithi097 ceph-mon[32231]: pgmap v3743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:04 smithi115 ceph-mon[35364]: pgmap v3744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:04 smithi181 ceph-mon[35869]: pgmap v3744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:04 smithi097 ceph-mon[32231]: pgmap v3744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:05.149 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:05.147+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:35:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:05.148+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:35:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:05.148+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:35:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:05.148+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:35:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:05.148+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5742.753960 secs 2024-07-26T09:35:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:06 smithi115 ceph-mon[35364]: pgmap v3745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5742.753960 secs 2024-07-26T09:35:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:06 smithi181 ceph-mon[35869]: pgmap v3745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5742.753960 secs 2024-07-26T09:35:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:06 smithi097 ceph-mon[32231]: pgmap v3745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:08 smithi115 ceph-mon[35364]: pgmap v3746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:08 smithi181 ceph-mon[35869]: pgmap v3746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:08 smithi097 ceph-mon[32231]: pgmap v3746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:10 smithi115 ceph-mon[35364]: pgmap v3747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:10 smithi181 ceph-mon[35869]: pgmap v3747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:10 smithi097 ceph-mon[32231]: pgmap v3747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:11.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5747.754115 secs 2024-07-26T09:35:11.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5747.754115 secs 2024-07-26T09:35:11.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5747.754115 secs 2024-07-26T09:35:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:12 smithi115 ceph-mon[35364]: pgmap v3748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:12 smithi181 ceph-mon[35869]: pgmap v3748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:12 smithi097 ceph-mon[32231]: pgmap v3748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:14 smithi115 ceph-mon[35364]: pgmap v3749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:14 smithi181 ceph-mon[35869]: pgmap v3749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:14 smithi097 ceph-mon[32231]: pgmap v3749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5752.754290 secs 2024-07-26T09:35:16.991 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:16 smithi115 ceph-mon[35364]: pgmap v3750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:16.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5752.754290 secs 2024-07-26T09:35:16.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:16 smithi181 ceph-mon[35869]: pgmap v3750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5752.754290 secs 2024-07-26T09:35:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:16 smithi097 ceph-mon[32231]: pgmap v3750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:18 smithi115 ceph-mon[35364]: pgmap v3751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:18 smithi181 ceph-mon[35869]: pgmap v3751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:18 smithi097 ceph-mon[32231]: pgmap v3751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:20.154+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:35:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:20.155+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:35:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:20.155+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:35:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:20.155+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:35:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:20.155+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:20 smithi115 ceph-mon[35364]: pgmap v3752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:20 smithi181 ceph-mon[35869]: pgmap v3752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:20 smithi097 ceph-mon[32231]: pgmap v3752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5757.754447 secs 2024-07-26T09:35:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5757.754447 secs 2024-07-26T09:35:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5757.754447 secs 2024-07-26T09:35:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:22 smithi115 ceph-mon[35364]: pgmap v3753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:22 smithi181 ceph-mon[35869]: pgmap v3753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:22 smithi097 ceph-mon[32231]: pgmap v3753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:24 smithi115 ceph-mon[35364]: pgmap v3754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:24 smithi181 ceph-mon[35869]: pgmap v3754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:24 smithi097 ceph-mon[32231]: pgmap v3754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:26 smithi115 ceph-mon[35364]: 1 slow requests, 1 included below; oldest blocked for > 5762.754594 secs 2024-07-26T09:35:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:26 smithi115 ceph-mon[35364]: slow request 5762.754593 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T09:35:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:26 smithi115 ceph-mon[35364]: pgmap v3755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:26 smithi181 ceph-mon[35869]: 1 slow requests, 1 included below; oldest blocked for > 5762.754594 secs 2024-07-26T09:35:26.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:26 smithi181 ceph-mon[35869]: slow request 5762.754593 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T09:35:26.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:26 smithi181 ceph-mon[35869]: pgmap v3755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:26 smithi097 ceph-mon[32231]: 1 slow requests, 1 included below; oldest blocked for > 5762.754594 secs 2024-07-26T09:35:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:26 smithi097 ceph-mon[32231]: slow request 5762.754593 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T09:35:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:26 smithi097 ceph-mon[32231]: pgmap v3755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:28 smithi115 ceph-mon[35364]: pgmap v3756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:28 smithi181 ceph-mon[35869]: pgmap v3756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:28 smithi097 ceph-mon[32231]: pgmap v3756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:30 smithi115 ceph-mon[35364]: pgmap v3757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:30 smithi181 ceph-mon[35869]: pgmap v3757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:30 smithi097 ceph-mon[32231]: pgmap v3757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5767.754731 secs 2024-07-26T09:35:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5767.754731 secs 2024-07-26T09:35:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5767.754731 secs 2024-07-26T09:35:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:32 smithi115 ceph-mon[35364]: pgmap v3758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:32 smithi181 ceph-mon[35869]: pgmap v3758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:32 smithi097 ceph-mon[32231]: pgmap v3758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:34 smithi115 ceph-mon[35364]: pgmap v3759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:34 smithi181 ceph-mon[35869]: pgmap v3759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:34 smithi097 ceph-mon[32231]: pgmap v3759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:35.160+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:35:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:35.162+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:35:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:35.162+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:35:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:35.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:35:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:35.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5772.754867 secs 2024-07-26T09:35:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:36 smithi115 ceph-mon[35364]: pgmap v3760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5772.754867 secs 2024-07-26T09:35:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:36 smithi181 ceph-mon[35869]: pgmap v3760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5772.754867 secs 2024-07-26T09:35:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:36 smithi097 ceph-mon[32231]: pgmap v3760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:38 smithi115 ceph-mon[35364]: pgmap v3761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:38 smithi181 ceph-mon[35869]: pgmap v3761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:38 smithi097 ceph-mon[32231]: pgmap v3761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:40 smithi115 ceph-mon[35364]: pgmap v3762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:40 smithi181 ceph-mon[35869]: pgmap v3762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:40 smithi097 ceph-mon[32231]: pgmap v3762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5777.755003 secs 2024-07-26T09:35:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5777.755003 secs 2024-07-26T09:35:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5777.755003 secs 2024-07-26T09:35:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:42 smithi097 ceph-mon[32231]: pgmap v3763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:42 smithi115 ceph-mon[35364]: pgmap v3763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:42 smithi181 ceph-mon[35869]: pgmap v3763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:43 smithi097 ceph-mon[32231]: pgmap v3764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:35:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:43 smithi097 ceph-mon[32231]: pgmap v3765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:44.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:35:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:43 smithi115 ceph-mon[35364]: pgmap v3764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:35:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:43 smithi115 ceph-mon[35364]: pgmap v3765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:35:44.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:44.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:44.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:43 smithi181 ceph-mon[35869]: pgmap v3764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:35:44.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:43 smithi181 ceph-mon[35869]: pgmap v3765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:44.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:35:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5782.755159 secs 2024-07-26T09:35:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:45 smithi115 ceph-mon[35364]: pgmap v3766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:35:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5782.755159 secs 2024-07-26T09:35:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:45 smithi181 ceph-mon[35869]: pgmap v3766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:35:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5782.755159 secs 2024-07-26T09:35:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:45 smithi097 ceph-mon[32231]: pgmap v3766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:35:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:47 smithi115 ceph-mon[35364]: pgmap v3767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:35:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:47 smithi181 ceph-mon[35869]: pgmap v3767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:35:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:47 smithi097 ceph-mon[32231]: pgmap v3767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:35:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:50.172+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:35:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:50.174+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:35:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:50.174+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:35:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:50.174+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:35:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:35:50.174+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:49 smithi115 ceph-mon[35364]: pgmap v3768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:49 smithi181 ceph-mon[35869]: pgmap v3768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:49 smithi097 ceph-mon[32231]: pgmap v3768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:35:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5787.755300 secs 2024-07-26T09:35:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5787.755300 secs 2024-07-26T09:35:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5787.755300 secs 2024-07-26T09:35:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:51 smithi115 ceph-mon[35364]: pgmap v3769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:51 smithi181 ceph-mon[35869]: pgmap v3769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:51 smithi097 ceph-mon[32231]: pgmap v3769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:53 smithi115 ceph-mon[35364]: pgmap v3770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:35:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:53 smithi181 ceph-mon[35869]: pgmap v3770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:35:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:53 smithi097 ceph-mon[32231]: pgmap v3770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:35:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5792.755437 secs 2024-07-26T09:35:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:55 smithi115 ceph-mon[35364]: pgmap v3771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:35:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5792.755437 secs 2024-07-26T09:35:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:55 smithi181 ceph-mon[35869]: pgmap v3771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:35:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5792.755437 secs 2024-07-26T09:35:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:55 smithi097 ceph-mon[32231]: pgmap v3771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:35:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:57 smithi115 ceph-mon[35364]: pgmap v3772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:35:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:57 smithi181 ceph-mon[35869]: pgmap v3772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:35:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:57 smithi097 ceph-mon[32231]: pgmap v3772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:36:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:35:59 smithi181 ceph-mon[35869]: pgmap v3773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:35:59 smithi097 ceph-mon[32231]: pgmap v3773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:35:59 smithi115 ceph-mon[35364]: pgmap v3773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5797.755580 secs 2024-07-26T09:36:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5797.755580 secs 2024-07-26T09:36:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5797.755580 secs 2024-07-26T09:36:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:01 smithi097 ceph-mon[32231]: pgmap v3774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:01 smithi115 ceph-mon[35364]: pgmap v3774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:01 smithi181 ceph-mon[35869]: pgmap v3774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:03 smithi097 ceph-mon[32231]: pgmap v3775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:03 smithi115 ceph-mon[35364]: pgmap v3775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:03 smithi181 ceph-mon[35869]: pgmap v3775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:05.179+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:36:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:05.180+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:36:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:05.180+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:36:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:05.180+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:36:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:05.180+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5802.755748 secs 2024-07-26T09:36:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:06 smithi115 ceph-mon[35364]: pgmap v3776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5802.755748 secs 2024-07-26T09:36:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:06 smithi181 ceph-mon[35869]: pgmap v3776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5802.755748 secs 2024-07-26T09:36:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:06 smithi097 ceph-mon[32231]: pgmap v3776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:08 smithi115 ceph-mon[35364]: pgmap v3777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:08 smithi181 ceph-mon[35869]: pgmap v3777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:08 smithi097 ceph-mon[32231]: pgmap v3777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:10 smithi115 ceph-mon[35364]: pgmap v3778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:10 smithi181 ceph-mon[35869]: pgmap v3778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:10 smithi097 ceph-mon[32231]: pgmap v3778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:11.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5807.755886 secs 2024-07-26T09:36:11.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5807.755886 secs 2024-07-26T09:36:11.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5807.755886 secs 2024-07-26T09:36:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:12 smithi115 ceph-mon[35364]: pgmap v3779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:12 smithi181 ceph-mon[35869]: pgmap v3779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:12 smithi097 ceph-mon[32231]: pgmap v3779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:14 smithi115 ceph-mon[35364]: pgmap v3780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:14 smithi181 ceph-mon[35869]: pgmap v3780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:14 smithi097 ceph-mon[32231]: pgmap v3780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5812.756029 secs 2024-07-26T09:36:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:16 smithi115 ceph-mon[35364]: pgmap v3781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5812.756029 secs 2024-07-26T09:36:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:16 smithi181 ceph-mon[35869]: pgmap v3781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5812.756029 secs 2024-07-26T09:36:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:16 smithi097 ceph-mon[32231]: pgmap v3781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:17 smithi115 ceph-mon[35364]: pgmap v3782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:17 smithi181 ceph-mon[35869]: pgmap v3782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:17 smithi097 ceph-mon[32231]: pgmap v3782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:20.190+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:36:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:20.191+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:36:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:20.191+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:36:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:20.191+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:36:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:20.191+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:19 smithi115 ceph-mon[35364]: pgmap v3783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:19 smithi181 ceph-mon[35869]: pgmap v3783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:19 smithi097 ceph-mon[32231]: pgmap v3783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5817.756108 secs 2024-07-26T09:36:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5817.756108 secs 2024-07-26T09:36:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5817.756108 secs 2024-07-26T09:36:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:21 smithi115 ceph-mon[35364]: pgmap v3784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:21 smithi181 ceph-mon[35869]: pgmap v3784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:21 smithi097 ceph-mon[32231]: pgmap v3784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:23 smithi115 ceph-mon[35364]: pgmap v3785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:23 smithi181 ceph-mon[35869]: pgmap v3785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:23 smithi097 ceph-mon[32231]: pgmap v3785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5822.756281 secs 2024-07-26T09:36:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:25 smithi115 ceph-mon[35364]: pgmap v3786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5822.756281 secs 2024-07-26T09:36:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:25 smithi181 ceph-mon[35869]: pgmap v3786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:26.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5822.756281 secs 2024-07-26T09:36:26.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:25 smithi097 ceph-mon[32231]: pgmap v3786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:27 smithi115 ceph-mon[35364]: pgmap v3787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:27 smithi181 ceph-mon[35869]: pgmap v3787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:27 smithi097 ceph-mon[32231]: pgmap v3787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:29 smithi115 ceph-mon[35364]: pgmap v3788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:29 smithi181 ceph-mon[35869]: pgmap v3788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:29 smithi097 ceph-mon[32231]: pgmap v3788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5827.756414 secs 2024-07-26T09:36:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5827.756414 secs 2024-07-26T09:36:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5827.756414 secs 2024-07-26T09:36:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:31 smithi097 ceph-mon[32231]: pgmap v3789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:31 smithi115 ceph-mon[35364]: pgmap v3789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:31 smithi181 ceph-mon[35869]: pgmap v3789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:33 smithi097 ceph-mon[32231]: pgmap v3790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:33 smithi115 ceph-mon[35364]: pgmap v3790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:33 smithi181 ceph-mon[35869]: pgmap v3790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:35.196+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:36:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:35.198+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:36:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:35.198+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:36:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:35.198+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:36:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:35.198+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5832.756555 secs 2024-07-26T09:36:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:35 smithi097 ceph-mon[32231]: pgmap v3791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5832.756555 secs 2024-07-26T09:36:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:35 smithi115 ceph-mon[35364]: pgmap v3791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5832.756555 secs 2024-07-26T09:36:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:35 smithi181 ceph-mon[35869]: pgmap v3791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:38 smithi115 ceph-mon[35364]: pgmap v3792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:38 smithi181 ceph-mon[35869]: pgmap v3792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:38 smithi097 ceph-mon[32231]: pgmap v3792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:40 smithi115 ceph-mon[35364]: pgmap v3793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:40 smithi181 ceph-mon[35869]: pgmap v3793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:40 smithi097 ceph-mon[32231]: pgmap v3793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:41.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5837.756692 secs 2024-07-26T09:36:41.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5837.756692 secs 2024-07-26T09:36:41.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5837.756692 secs 2024-07-26T09:36:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:42 smithi115 ceph-mon[35364]: pgmap v3794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:42 smithi181 ceph-mon[35869]: pgmap v3794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:42 smithi097 ceph-mon[32231]: pgmap v3794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:44 smithi115 ceph-mon[35364]: pgmap v3795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:44 smithi181 ceph-mon[35869]: pgmap v3795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:44 smithi097 ceph-mon[32231]: pgmap v3795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:36:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:36:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:45 smithi115 ceph-mon[35364]: pgmap v3796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:36:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:45 smithi115 ceph-mon[35364]: pgmap v3797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:36:45.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:36:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:36:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:36:45.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:45 smithi181 ceph-mon[35869]: pgmap v3796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:36:45.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:45 smithi181 ceph-mon[35869]: pgmap v3797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:36:45.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:36:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:36:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:36:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:45 smithi097 ceph-mon[32231]: pgmap v3796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:36:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:45 smithi097 ceph-mon[32231]: pgmap v3797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:36:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:36:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5842.756853 secs 2024-07-26T09:36:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5842.756853 secs 2024-07-26T09:36:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5842.756853 secs 2024-07-26T09:36:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:47 smithi115 ceph-mon[35364]: pgmap v3798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:47 smithi181 ceph-mon[35869]: pgmap v3798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:47 smithi097 ceph-mon[32231]: pgmap v3798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:49 smithi115 ceph-mon[35364]: pgmap v3799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:49 smithi181 ceph-mon[35869]: pgmap v3799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:49 smithi097 ceph-mon[32231]: pgmap v3799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:50.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:50.208+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:36:50.212 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:50.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:36:50.212 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:50.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:36:50.212 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:50.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:36:50.212 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:36:50.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5847.756990 secs 2024-07-26T09:36:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:51 smithi115 ceph-mon[35364]: pgmap v3800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5847.756990 secs 2024-07-26T09:36:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:51 smithi181 ceph-mon[35869]: pgmap v3800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5847.756990 secs 2024-07-26T09:36:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:51 smithi097 ceph-mon[32231]: pgmap v3800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:53 smithi115 ceph-mon[35364]: pgmap v3801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:36:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:53 smithi181 ceph-mon[35869]: pgmap v3801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:36:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:53 smithi097 ceph-mon[32231]: pgmap v3801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:36:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:55 smithi115 ceph-mon[35364]: pgmap v3802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:36:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:55 smithi181 ceph-mon[35869]: pgmap v3802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:36:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:55 smithi097 ceph-mon[32231]: pgmap v3802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:36:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5852.757137 secs 2024-07-26T09:36:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5852.757137 secs 2024-07-26T09:36:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5852.757137 secs 2024-07-26T09:36:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:57 smithi115 ceph-mon[35364]: pgmap v3803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:36:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:57 smithi181 ceph-mon[35869]: pgmap v3803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:36:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:57 smithi097 ceph-mon[32231]: pgmap v3803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:36:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:36:59 smithi115 ceph-mon[35364]: pgmap v3804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:36:59 smithi181 ceph-mon[35869]: pgmap v3804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:36:59 smithi097 ceph-mon[32231]: pgmap v3804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5857.757277 secs 2024-07-26T09:37:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:01 smithi115 ceph-mon[35364]: pgmap v3805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5857.757277 secs 2024-07-26T09:37:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:01 smithi181 ceph-mon[35869]: pgmap v3805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5857.757277 secs 2024-07-26T09:37:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:01 smithi097 ceph-mon[32231]: pgmap v3805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:03 smithi115 ceph-mon[35364]: pgmap v3806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:03 smithi181 ceph-mon[35869]: pgmap v3806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:03 smithi097 ceph-mon[32231]: pgmap v3806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:05.213+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:37:05.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:05.214+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:37:05.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:05.214+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:37:05.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:05.214+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:37:05.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:05.214+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:05 smithi115 ceph-mon[35364]: pgmap v3807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:05 smithi181 ceph-mon[35869]: pgmap v3807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:05 smithi097 ceph-mon[32231]: pgmap v3807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5862.757418 secs 2024-07-26T09:37:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5862.757418 secs 2024-07-26T09:37:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5862.757418 secs 2024-07-26T09:37:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:07 smithi115 ceph-mon[35364]: pgmap v3808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:07 smithi181 ceph-mon[35869]: pgmap v3808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:07 smithi097 ceph-mon[32231]: pgmap v3808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:07.913+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00063a0 con 0x7ff3f01061a0 2024-07-26T09:37:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:07.913+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:37:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:07.914+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T09:37:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:07.914+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:37:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:07.914+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T09:37:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:07.914+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T09:37:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:07.915+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:37:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:07.915+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061d0 con 0x7ff3f01061a0 2024-07-26T09:37:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:09 smithi115 ceph-mon[35364]: pgmap v3809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:37:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:09 smithi181 ceph-mon[35869]: pgmap v3809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:37:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:09 smithi097 ceph-mon[32231]: pgmap v3809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:37:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5867.757549 secs 2024-07-26T09:37:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:11 smithi115 ceph-mon[35364]: pgmap v3810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:37:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5867.757549 secs 2024-07-26T09:37:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:11 smithi181 ceph-mon[35869]: pgmap v3810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:37:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5867.757549 secs 2024-07-26T09:37:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:11 smithi097 ceph-mon[32231]: pgmap v3810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:37:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:13 smithi181 ceph-mon[35869]: pgmap v3811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:37:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:13 smithi097 ceph-mon[32231]: pgmap v3811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:37:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:13 smithi115 ceph-mon[35364]: pgmap v3811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:37:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:14 smithi097 ceph-mon[32231]: pgmap v3812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:14 smithi115 ceph-mon[35364]: pgmap v3812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:14 smithi181 ceph-mon[35869]: pgmap v3812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5872.757714 secs 2024-07-26T09:37:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5872.757714 secs 2024-07-26T09:37:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5872.757714 secs 2024-07-26T09:37:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:16 smithi115 ceph-mon[35364]: pgmap v3813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:16 smithi181 ceph-mon[35869]: pgmap v3813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:16 smithi097 ceph-mon[32231]: pgmap v3813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:19 smithi115 ceph-mon[35364]: pgmap v3814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:37:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:19 smithi181 ceph-mon[35869]: pgmap v3814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:37:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:19 smithi097 ceph-mon[32231]: pgmap v3814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:37:20.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:20.225+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:37:20.228 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:20.226+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:37:20.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:20.226+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:37:20.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:20.226+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:37:20.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:20.226+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5877.757876 secs 2024-07-26T09:37:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:21 smithi115 ceph-mon[35364]: pgmap v3815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5877.757876 secs 2024-07-26T09:37:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:21 smithi181 ceph-mon[35869]: pgmap v3815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5877.757876 secs 2024-07-26T09:37:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:21 smithi097 ceph-mon[32231]: pgmap v3815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:23 smithi115 ceph-mon[35364]: pgmap v3816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:23 smithi181 ceph-mon[35869]: pgmap v3816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:23 smithi097 ceph-mon[32231]: pgmap v3816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:25 smithi115 ceph-mon[35364]: pgmap v3817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:25 smithi181 ceph-mon[35869]: pgmap v3817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:25 smithi097 ceph-mon[32231]: pgmap v3817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5882.757977 secs 2024-07-26T09:37:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5882.757977 secs 2024-07-26T09:37:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5882.757977 secs 2024-07-26T09:37:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:27 smithi115 ceph-mon[35364]: pgmap v3818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:27 smithi181 ceph-mon[35869]: pgmap v3818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:27 smithi097 ceph-mon[32231]: pgmap v3818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:29 smithi115 ceph-mon[35364]: pgmap v3819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:29 smithi181 ceph-mon[35869]: pgmap v3819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:29 smithi097 ceph-mon[32231]: pgmap v3819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5887.758134 secs 2024-07-26T09:37:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:31 smithi115 ceph-mon[35364]: pgmap v3820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5887.758134 secs 2024-07-26T09:37:31.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:31 smithi181 ceph-mon[35869]: pgmap v3820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5887.758134 secs 2024-07-26T09:37:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:31 smithi097 ceph-mon[32231]: pgmap v3820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:33 smithi115 ceph-mon[35364]: pgmap v3821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:33 smithi181 ceph-mon[35869]: pgmap v3821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:33 smithi097 ceph-mon[32231]: pgmap v3821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:35.231+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:37:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:35.233+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:37:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:35.233+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:37:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:35.233+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:37:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:35.233+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:35 smithi115 ceph-mon[35364]: pgmap v3822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:35 smithi181 ceph-mon[35869]: pgmap v3822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:35 smithi097 ceph-mon[32231]: pgmap v3822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:36.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5892.758292 secs 2024-07-26T09:37:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5892.758292 secs 2024-07-26T09:37:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5892.758292 secs 2024-07-26T09:37:37.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:37 smithi181 ceph-mon[35869]: pgmap v3823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:37 smithi097 ceph-mon[32231]: pgmap v3823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:37 smithi115 ceph-mon[35364]: pgmap v3823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:37:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:39 smithi097 ceph-mon[32231]: pgmap v3824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:39 smithi115 ceph-mon[35364]: pgmap v3824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:39 smithi181 ceph-mon[35869]: pgmap v3824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:40 smithi115 ceph-mon[35364]: pgmap v3825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:40 smithi181 ceph-mon[35869]: pgmap v3825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:40 smithi097 ceph-mon[32231]: pgmap v3825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:41.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5897.758442 secs 2024-07-26T09:37:41.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5897.758442 secs 2024-07-26T09:37:41.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5897.758442 secs 2024-07-26T09:37:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:42 smithi115 ceph-mon[35364]: pgmap v3826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:37:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:42 smithi181 ceph-mon[35869]: pgmap v3826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:37:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:42 smithi097 ceph-mon[32231]: pgmap v3826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:37:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:45 smithi115 ceph-mon[35364]: pgmap v3827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:37:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:45 smithi181 ceph-mon[35869]: pgmap v3827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:37:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:45 smithi097 ceph-mon[32231]: pgmap v3827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:37:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5902.758536 secs 2024-07-26T09:37:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:46.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5902.758536 secs 2024-07-26T09:37:46.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:46.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:46.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:46.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:46.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5902.758536 secs 2024-07-26T09:37:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:37:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:47 smithi115 ceph-mon[35364]: pgmap v3828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:37:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:47 smithi115 ceph-mon[35364]: pgmap v3829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:47 smithi181 ceph-mon[35869]: pgmap v3828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:37:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:47 smithi181 ceph-mon[35869]: pgmap v3829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:47 smithi097 ceph-mon[32231]: pgmap v3828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:37:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:47 smithi097 ceph-mon[32231]: pgmap v3829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:49 smithi115 ceph-mon[35364]: pgmap v3830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:49 smithi181 ceph-mon[35869]: pgmap v3830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:49 smithi097 ceph-mon[32231]: pgmap v3830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:50.243+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:37:50.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:50.244+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:37:50.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:50.244+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:37:50.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:50.244+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:37:50.247 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:37:50.245+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5907.758688 secs 2024-07-26T09:37:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:51 smithi115 ceph-mon[35364]: pgmap v3831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5907.758688 secs 2024-07-26T09:37:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:51 smithi181 ceph-mon[35869]: pgmap v3831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5907.758688 secs 2024-07-26T09:37:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:51 smithi097 ceph-mon[32231]: pgmap v3831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:53 smithi115 ceph-mon[35364]: pgmap v3832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:53 smithi181 ceph-mon[35869]: pgmap v3832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:53 smithi097 ceph-mon[32231]: pgmap v3832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:55 smithi115 ceph-mon[35364]: pgmap v3833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:55 smithi181 ceph-mon[35869]: pgmap v3833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:55 smithi097 ceph-mon[32231]: pgmap v3833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:37:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5912.758842 secs 2024-07-26T09:37:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5912.758842 secs 2024-07-26T09:37:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5912.758842 secs 2024-07-26T09:37:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:57 smithi115 ceph-mon[35364]: pgmap v3834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:57 smithi181 ceph-mon[35869]: pgmap v3834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:57 smithi097 ceph-mon[32231]: pgmap v3834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:37:59 smithi115 ceph-mon[35364]: pgmap v3835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:37:59 smithi181 ceph-mon[35869]: pgmap v3835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:37:59 smithi097 ceph-mon[32231]: pgmap v3835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5917.759011 secs 2024-07-26T09:38:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:01 smithi115 ceph-mon[35364]: pgmap v3836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5917.759011 secs 2024-07-26T09:38:01.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:01 smithi181 ceph-mon[35869]: pgmap v3836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5917.759011 secs 2024-07-26T09:38:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:01 smithi097 ceph-mon[32231]: pgmap v3836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:03 smithi181 ceph-mon[35869]: pgmap v3837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:03 smithi097 ceph-mon[32231]: pgmap v3837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:03 smithi115 ceph-mon[35364]: pgmap v3837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:05.250+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:38:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:05.251+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:38:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:05.251+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:38:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:05.251+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:38:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:05.251+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:05 smithi097 ceph-mon[32231]: pgmap v3838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:05 smithi115 ceph-mon[35364]: pgmap v3838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:05 smithi181 ceph-mon[35869]: pgmap v3838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5922.759156 secs 2024-07-26T09:38:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5922.759156 secs 2024-07-26T09:38:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5922.759156 secs 2024-07-26T09:38:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:07 smithi115 ceph-mon[35364]: pgmap v3839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:07 smithi181 ceph-mon[35869]: pgmap v3839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:07 smithi097 ceph-mon[32231]: pgmap v3839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:08 smithi115 ceph-mon[35364]: pgmap v3840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:08 smithi181 ceph-mon[35869]: pgmap v3840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:08 smithi097 ceph-mon[32231]: pgmap v3840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:11.679 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5927.759303 secs 2024-07-26T09:38:11.679 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:11 smithi115 ceph-mon[35364]: pgmap v3841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:11.687 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5927.759303 secs 2024-07-26T09:38:11.687 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:11 smithi181 ceph-mon[35869]: pgmap v3841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:11.705 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5927.759303 secs 2024-07-26T09:38:11.706 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:11 smithi097 ceph-mon[32231]: pgmap v3841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:13 smithi115 ceph-mon[35364]: pgmap v3842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:13 smithi181 ceph-mon[35869]: pgmap v3842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:13 smithi097 ceph-mon[32231]: pgmap v3842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:15 smithi115 ceph-mon[35364]: pgmap v3843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:15 smithi181 ceph-mon[35869]: pgmap v3843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:15 smithi097 ceph-mon[32231]: pgmap v3843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5932.759406 secs 2024-07-26T09:38:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5932.759406 secs 2024-07-26T09:38:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5932.759406 secs 2024-07-26T09:38:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:17 smithi115 ceph-mon[35364]: pgmap v3844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:17 smithi181 ceph-mon[35869]: pgmap v3844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:17 smithi097 ceph-mon[32231]: pgmap v3844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:19 smithi115 ceph-mon[35364]: pgmap v3845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:19 smithi181 ceph-mon[35869]: pgmap v3845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:19 smithi097 ceph-mon[32231]: pgmap v3845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:20.262+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:20.263+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:20.263+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:20.263+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:20.263+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5937.759575 secs 2024-07-26T09:38:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:21 smithi115 ceph-mon[35364]: pgmap v3846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5937.759575 secs 2024-07-26T09:38:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:21 smithi181 ceph-mon[35869]: pgmap v3846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5937.759575 secs 2024-07-26T09:38:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:21 smithi097 ceph-mon[32231]: pgmap v3846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:23 smithi115 ceph-mon[35364]: pgmap v3847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:23 smithi181 ceph-mon[35869]: pgmap v3847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:23 smithi097 ceph-mon[32231]: pgmap v3847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:25 smithi115 ceph-mon[35364]: pgmap v3848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:25 smithi181 ceph-mon[35869]: pgmap v3848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:25 smithi097 ceph-mon[32231]: pgmap v3848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5942.759734 secs 2024-07-26T09:38:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5942.759734 secs 2024-07-26T09:38:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5942.759734 secs 2024-07-26T09:38:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:27 smithi115 ceph-mon[35364]: pgmap v3849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:27 smithi181 ceph-mon[35869]: pgmap v3849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:27 smithi097 ceph-mon[32231]: pgmap v3849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:29 smithi115 ceph-mon[35364]: pgmap v3850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:29 smithi181 ceph-mon[35869]: pgmap v3850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:29 smithi097 ceph-mon[32231]: pgmap v3850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5947.759830 secs 2024-07-26T09:38:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:31 smithi115 ceph-mon[35364]: pgmap v3851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5947.759830 secs 2024-07-26T09:38:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:31 smithi181 ceph-mon[35869]: pgmap v3851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5947.759830 secs 2024-07-26T09:38:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:31 smithi097 ceph-mon[32231]: pgmap v3851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:33 smithi115 ceph-mon[35364]: pgmap v3852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:33 smithi181 ceph-mon[35869]: pgmap v3852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:33 smithi097 ceph-mon[32231]: pgmap v3852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:35.268+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:38:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:35.269+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:38:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:35.269+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:38:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:35.269+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:38:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:35.270+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:35 smithi115 ceph-mon[35364]: pgmap v3853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:35 smithi181 ceph-mon[35869]: pgmap v3853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:35 smithi097 ceph-mon[32231]: pgmap v3853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5952.760009 secs 2024-07-26T09:38:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5952.760009 secs 2024-07-26T09:38:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5952.760009 secs 2024-07-26T09:38:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:37 smithi115 ceph-mon[35364]: pgmap v3854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:37 smithi181 ceph-mon[35869]: pgmap v3854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:37 smithi097 ceph-mon[32231]: pgmap v3854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:39 smithi115 ceph-mon[35364]: pgmap v3855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:39 smithi181 ceph-mon[35869]: pgmap v3855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:39.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:39 smithi097 ceph-mon[32231]: pgmap v3855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5957.760088 secs 2024-07-26T09:38:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:41 smithi115 ceph-mon[35364]: pgmap v3856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5957.760088 secs 2024-07-26T09:38:41.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:41 smithi181 ceph-mon[35869]: pgmap v3856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5957.760088 secs 2024-07-26T09:38:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:41 smithi097 ceph-mon[32231]: pgmap v3856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:43 smithi115 ceph-mon[35364]: pgmap v3857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:43 smithi181 ceph-mon[35869]: pgmap v3857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:43 smithi097 ceph-mon[32231]: pgmap v3857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:45 smithi115 ceph-mon[35364]: pgmap v3858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:45 smithi181 ceph-mon[35869]: pgmap v3858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:45 smithi097 ceph-mon[32231]: pgmap v3858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5962.760306 secs 2024-07-26T09:38:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5962.760306 secs 2024-07-26T09:38:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5962.760306 secs 2024-07-26T09:38:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:47 smithi097 ceph-mon[32231]: pgmap v3859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:47 smithi115 ceph-mon[35364]: pgmap v3859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:47 smithi181 ceph-mon[35869]: pgmap v3859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:49 smithi115 ceph-mon[35364]: pgmap v3860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:49 smithi181 ceph-mon[35869]: pgmap v3860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:49.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:49.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:49.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:49 smithi097 ceph-mon[32231]: pgmap v3860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:49.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:38:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:50.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:38:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:50.274+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:38:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:50.274+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:38:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:50.274+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:38:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:38:50.274+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5967.760377 secs 2024-07-26T09:38:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:51 smithi115 ceph-mon[35364]: pgmap v3861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5967.760377 secs 2024-07-26T09:38:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:51 smithi181 ceph-mon[35869]: pgmap v3861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5967.760377 secs 2024-07-26T09:38:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:51 smithi097 ceph-mon[32231]: pgmap v3861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:53 smithi115 ceph-mon[35364]: pgmap v3862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:53 smithi181 ceph-mon[35869]: pgmap v3862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:53 smithi097 ceph-mon[32231]: pgmap v3862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:55 smithi115 ceph-mon[35364]: pgmap v3863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:55 smithi181 ceph-mon[35869]: pgmap v3863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:55 smithi097 ceph-mon[32231]: pgmap v3863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5972.760520 secs 2024-07-26T09:38:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5972.760520 secs 2024-07-26T09:38:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5972.760520 secs 2024-07-26T09:38:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:57 smithi115 ceph-mon[35364]: pgmap v3864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:38:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:57 smithi181 ceph-mon[35869]: pgmap v3864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:38:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:57 smithi097 ceph-mon[32231]: pgmap v3864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:38:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:38:59 smithi115 ceph-mon[35364]: pgmap v3865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:38:59 smithi181 ceph-mon[35869]: pgmap v3865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:38:59 smithi097 ceph-mon[32231]: pgmap v3865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5977.760673 secs 2024-07-26T09:39:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:01 smithi115 ceph-mon[35364]: pgmap v3866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5977.760673 secs 2024-07-26T09:39:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:01 smithi181 ceph-mon[35869]: pgmap v3866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5977.760673 secs 2024-07-26T09:39:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:01 smithi097 ceph-mon[32231]: pgmap v3866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:03 smithi115 ceph-mon[35364]: pgmap v3867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:03 smithi181 ceph-mon[35869]: pgmap v3867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:03 smithi097 ceph-mon[32231]: pgmap v3867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:05.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:05.278+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:39:05.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:05.280+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:39:05.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:05.280+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:39:05.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:05.280+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:39:05.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:05.280+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:05 smithi115 ceph-mon[35364]: pgmap v3868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:05 smithi181 ceph-mon[35869]: pgmap v3868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:05 smithi097 ceph-mon[32231]: pgmap v3868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5982.760852 secs 2024-07-26T09:39:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5982.760852 secs 2024-07-26T09:39:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5982.760852 secs 2024-07-26T09:39:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:07 smithi115 ceph-mon[35364]: pgmap v3869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:07 smithi181 ceph-mon[35869]: pgmap v3869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:07 smithi097 ceph-mon[32231]: pgmap v3869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:09 smithi115 ceph-mon[35364]: pgmap v3870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:39:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:09 smithi181 ceph-mon[35869]: pgmap v3870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:39:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:09 smithi097 ceph-mon[32231]: pgmap v3870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:39:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5987.761007 secs 2024-07-26T09:39:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:11 smithi115 ceph-mon[35364]: pgmap v3871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:39:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5987.761007 secs 2024-07-26T09:39:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:11 smithi181 ceph-mon[35869]: pgmap v3871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:39:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5987.761007 secs 2024-07-26T09:39:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:11 smithi097 ceph-mon[32231]: pgmap v3871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:39:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:13 smithi115 ceph-mon[35364]: pgmap v3872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:39:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:13 smithi181 ceph-mon[35869]: pgmap v3872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:39:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:13 smithi097 ceph-mon[32231]: pgmap v3872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:39:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:15 smithi115 ceph-mon[35364]: pgmap v3873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:39:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:15 smithi181 ceph-mon[35869]: pgmap v3873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:39:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:15 smithi097 ceph-mon[32231]: pgmap v3873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:39:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5992.761182 secs 2024-07-26T09:39:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5992.761182 secs 2024-07-26T09:39:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5992.761182 secs 2024-07-26T09:39:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:17 smithi115 ceph-mon[35364]: pgmap v3874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:17 smithi181 ceph-mon[35869]: pgmap v3874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:17 smithi097 ceph-mon[32231]: pgmap v3874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:19 smithi115 ceph-mon[35364]: pgmap v3875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:19 smithi181 ceph-mon[35869]: pgmap v3875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:19 smithi097 ceph-mon[32231]: pgmap v3875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:20.286+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:39:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:20.288+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:39:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:20.288+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:39:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:20.288+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:39:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:20.288+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 5997.761345 secs 2024-07-26T09:39:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:21 smithi115 ceph-mon[35364]: pgmap v3876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 5997.761345 secs 2024-07-26T09:39:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:21 smithi181 ceph-mon[35869]: pgmap v3876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 5997.761345 secs 2024-07-26T09:39:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:21 smithi097 ceph-mon[32231]: pgmap v3876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:23 smithi115 ceph-mon[35364]: pgmap v3877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:23 smithi181 ceph-mon[35869]: pgmap v3877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:23 smithi097 ceph-mon[32231]: pgmap v3877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:25 smithi097 ceph-mon[32231]: pgmap v3878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:25 smithi115 ceph-mon[35364]: pgmap v3878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:25 smithi181 ceph-mon[35869]: pgmap v3878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6002.761520 secs 2024-07-26T09:39:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6002.761520 secs 2024-07-26T09:39:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6002.761520 secs 2024-07-26T09:39:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:27 smithi115 ceph-mon[35364]: pgmap v3879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:27 smithi181 ceph-mon[35869]: pgmap v3879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:27 smithi097 ceph-mon[32231]: pgmap v3879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:28 smithi115 ceph-mon[35364]: pgmap v3880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:28 smithi181 ceph-mon[35869]: pgmap v3880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:28 smithi097 ceph-mon[32231]: pgmap v3880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6007.761699 secs 2024-07-26T09:39:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:31 smithi115 ceph-mon[35364]: pgmap v3881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6007.761699 secs 2024-07-26T09:39:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:31 smithi181 ceph-mon[35869]: pgmap v3881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6007.761699 secs 2024-07-26T09:39:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:31 smithi097 ceph-mon[32231]: pgmap v3881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:33 smithi115 ceph-mon[35364]: pgmap v3882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:33 smithi181 ceph-mon[35869]: pgmap v3882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:33 smithi097 ceph-mon[32231]: pgmap v3882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:35.294+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:39:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:35.295+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:39:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:35.295+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:39:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:35.295+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:39:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:35.295+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:35 smithi115 ceph-mon[35364]: pgmap v3883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:35 smithi181 ceph-mon[35869]: pgmap v3883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:35 smithi097 ceph-mon[32231]: pgmap v3883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6012.761844 secs 2024-07-26T09:39:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6012.761844 secs 2024-07-26T09:39:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6012.761844 secs 2024-07-26T09:39:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:37 smithi115 ceph-mon[35364]: pgmap v3884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:37 smithi181 ceph-mon[35869]: pgmap v3884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:37 smithi097 ceph-mon[32231]: pgmap v3884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:39 smithi115 ceph-mon[35364]: pgmap v3885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:39 smithi181 ceph-mon[35869]: pgmap v3885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:39 smithi097 ceph-mon[32231]: pgmap v3885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6017.761999 secs 2024-07-26T09:39:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:41 smithi115 ceph-mon[35364]: pgmap v3886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6017.761999 secs 2024-07-26T09:39:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:41 smithi181 ceph-mon[35869]: pgmap v3886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6017.761999 secs 2024-07-26T09:39:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:41 smithi097 ceph-mon[32231]: pgmap v3886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:43 smithi115 ceph-mon[35364]: pgmap v3887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:43 smithi181 ceph-mon[35869]: pgmap v3887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:43 smithi097 ceph-mon[32231]: pgmap v3887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:45 smithi115 ceph-mon[35364]: pgmap v3888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:45 smithi181 ceph-mon[35869]: pgmap v3888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:45 smithi097 ceph-mon[32231]: pgmap v3888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6022.762155 secs 2024-07-26T09:39:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6022.762155 secs 2024-07-26T09:39:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6022.762155 secs 2024-07-26T09:39:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:47 smithi115 ceph-mon[35364]: pgmap v3889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:47 smithi181 ceph-mon[35869]: pgmap v3889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:47 smithi097 ceph-mon[32231]: pgmap v3889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:49 smithi115 ceph-mon[35364]: pgmap v3890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:49 smithi181 ceph-mon[35869]: pgmap v3890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:49 smithi097 ceph-mon[32231]: pgmap v3890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:50.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:50.306+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:39:50.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:50.307+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:39:50.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:50.307+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:39:50.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:50.307+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:39:50.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:39:50.307+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:50.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:50.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:50.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:39:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:39:50.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:50.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:50.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:39:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:51 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:39:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:51 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:39:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:51 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:39:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:51 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:39:51.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:51 smithi115 ceph-mon[35364]: pgmap v3891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:39:51.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:51 smithi115 ceph-mon[35364]: pgmap v3892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:51.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6027.762308 secs 2024-07-26T09:39:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:51 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:39:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:51 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:39:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:51 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:39:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:51 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:39:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:51 smithi181 ceph-mon[35869]: pgmap v3891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:39:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:51 smithi181 ceph-mon[35869]: pgmap v3892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6027.762308 secs 2024-07-26T09:39:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:51 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:39:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:51 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:39:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:51 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:39:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:51 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:39:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:51 smithi097 ceph-mon[32231]: pgmap v3891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:39:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:51 smithi097 ceph-mon[32231]: pgmap v3892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6027.762308 secs 2024-07-26T09:39:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:53 smithi115 ceph-mon[35364]: pgmap v3893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:53 smithi181 ceph-mon[35869]: pgmap v3893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:53 smithi097 ceph-mon[32231]: pgmap v3893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:55 smithi115 ceph-mon[35364]: pgmap v3894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:55 smithi181 ceph-mon[35869]: pgmap v3894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:55 smithi097 ceph-mon[32231]: pgmap v3894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6032.762394 secs 2024-07-26T09:39:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6032.762394 secs 2024-07-26T09:39:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6032.762394 secs 2024-07-26T09:39:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:57 smithi181 ceph-mon[35869]: pgmap v3895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:57 smithi097 ceph-mon[32231]: pgmap v3895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:57 smithi115 ceph-mon[35364]: pgmap v3895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:39:59 smithi097 ceph-mon[32231]: pgmap v3896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:39:59 smithi115 ceph-mon[35364]: pgmap v3896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:39:59 smithi181 ceph-mon[35869]: pgmap v3896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6037.762566 secs 2024-07-26T09:40:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:40:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6037.762566 secs 2024-07-26T09:40:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:40:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6037.762566 secs 2024-07-26T09:40:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:40:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:01 smithi115 ceph-mon[35364]: pgmap v3897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:40:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:01 smithi181 ceph-mon[35869]: pgmap v3897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:40:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:01 smithi097 ceph-mon[32231]: pgmap v3897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:40:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:03 smithi115 ceph-mon[35364]: pgmap v3898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:03 smithi181 ceph-mon[35869]: pgmap v3898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:03 smithi097 ceph-mon[32231]: pgmap v3898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:05.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:05.312+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:40:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:05.313+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:40:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:05.313+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:40:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:05.313+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:40:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:05.314+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:05 smithi115 ceph-mon[35364]: pgmap v3899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6042.762655 secs 2024-07-26T09:40:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:05 smithi181 ceph-mon[35869]: pgmap v3899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6042.762655 secs 2024-07-26T09:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:05 smithi097 ceph-mon[32231]: pgmap v3899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6042.762655 secs 2024-07-26T09:40:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:06 smithi115 ceph-mon[35364]: pgmap v3900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:06 smithi181 ceph-mon[35869]: pgmap v3900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:06 smithi097 ceph-mon[32231]: pgmap v3900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:07.945+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006640 con 0x7ff3f01061a0 2024-07-26T09:40:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:07.945+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:40:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:07.946+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:40:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:07.947+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006640 con 0x7ff3f01061a0 2024-07-26T09:40:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:08 smithi115 ceph-mon[35364]: pgmap v3901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:08 smithi181 ceph-mon[35869]: pgmap v3901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:08 smithi097 ceph-mon[32231]: pgmap v3901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:09 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6047.762845 secs 2024-07-26T09:40:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:09 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6047.762845 secs 2024-07-26T09:40:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:09 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6047.762845 secs 2024-07-26T09:40:11.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:10 smithi115 ceph-mon[35364]: pgmap v3902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:11.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:10 smithi181 ceph-mon[35869]: pgmap v3902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:11.155 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:10 smithi097 ceph-mon[32231]: pgmap v3902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:12 smithi115 ceph-mon[35364]: pgmap v3903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:12 smithi181 ceph-mon[35869]: pgmap v3903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:12 smithi097 ceph-mon[32231]: pgmap v3903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:14 smithi115 ceph-mon[35364]: pgmap v3904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:15.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:14 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6052.762994 secs 2024-07-26T09:40:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:14 smithi181 ceph-mon[35869]: pgmap v3904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:14 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6052.762994 secs 2024-07-26T09:40:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:14 smithi097 ceph-mon[32231]: pgmap v3904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:14 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6052.762994 secs 2024-07-26T09:40:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:16 smithi115 ceph-mon[35364]: pgmap v3905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:16 smithi181 ceph-mon[35869]: pgmap v3905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:16 smithi097 ceph-mon[32231]: pgmap v3905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:18 smithi115 ceph-mon[35364]: pgmap v3906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:18 smithi181 ceph-mon[35869]: pgmap v3906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:18 smithi097 ceph-mon[32231]: pgmap v3906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:19 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6057.763149 secs 2024-07-26T09:40:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:19 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6057.763149 secs 2024-07-26T09:40:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:19 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6057.763149 secs 2024-07-26T09:40:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:20.325+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:20.326+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:20.326+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:40:20.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:20.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:40:20.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:20.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:20 smithi115 ceph-mon[35364]: pgmap v3907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:20 smithi181 ceph-mon[35869]: pgmap v3907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:20 smithi097 ceph-mon[32231]: pgmap v3907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:22 smithi115 ceph-mon[35364]: pgmap v3908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:22 smithi181 ceph-mon[35869]: pgmap v3908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:22 smithi097 ceph-mon[32231]: pgmap v3908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:24 smithi115 ceph-mon[35364]: pgmap v3909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:24 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6062.763345 secs 2024-07-26T09:40:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:24 smithi181 ceph-mon[35869]: pgmap v3909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:24 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6062.763345 secs 2024-07-26T09:40:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:24 smithi097 ceph-mon[32231]: pgmap v3909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:24 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6062.763345 secs 2024-07-26T09:40:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:26 smithi115 ceph-mon[35364]: pgmap v3910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:26 smithi181 ceph-mon[35869]: pgmap v3910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:26 smithi097 ceph-mon[32231]: pgmap v3910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:28 smithi115 ceph-mon[35364]: pgmap v3911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:28 smithi181 ceph-mon[35869]: pgmap v3911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:28 smithi097 ceph-mon[32231]: pgmap v3911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:29 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6067.763456 secs 2024-07-26T09:40:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:29 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6067.763456 secs 2024-07-26T09:40:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:29 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6067.763456 secs 2024-07-26T09:40:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:30 smithi115 ceph-mon[35364]: pgmap v3912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:30 smithi181 ceph-mon[35869]: pgmap v3912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:30 smithi097 ceph-mon[32231]: pgmap v3912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:32 smithi115 ceph-mon[35364]: pgmap v3913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:32 smithi181 ceph-mon[35869]: pgmap v3913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:32 smithi097 ceph-mon[32231]: pgmap v3913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:34 smithi115 ceph-mon[35364]: pgmap v3914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:34 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6072.763554 secs 2024-07-26T09:40:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:34 smithi181 ceph-mon[35869]: pgmap v3914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:34 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6072.763554 secs 2024-07-26T09:40:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:34 smithi097 ceph-mon[32231]: pgmap v3914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:34 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6072.763554 secs 2024-07-26T09:40:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:35.332+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:40:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:35.333+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:40:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:35.333+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:40:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:35.333+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:40:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:35.333+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:36 smithi115 ceph-mon[35364]: pgmap v3915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:36 smithi181 ceph-mon[35869]: pgmap v3915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:36 smithi097 ceph-mon[32231]: pgmap v3915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:38 smithi115 ceph-mon[35364]: pgmap v3916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:38 smithi181 ceph-mon[35869]: pgmap v3916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:38 smithi097 ceph-mon[32231]: pgmap v3916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:39 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6077.763658 secs 2024-07-26T09:40:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:39 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6077.763658 secs 2024-07-26T09:40:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:39 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6077.763658 secs 2024-07-26T09:40:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:40 smithi097 ceph-mon[32231]: pgmap v3917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:41.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:40 smithi115 ceph-mon[35364]: pgmap v3917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:40 smithi181 ceph-mon[35869]: pgmap v3917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:42 smithi097 ceph-mon[32231]: pgmap v3918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:42 smithi115 ceph-mon[35364]: pgmap v3918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:42 smithi181 ceph-mon[35869]: pgmap v3918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:44 smithi097 ceph-mon[32231]: pgmap v3919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:44 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6082.763796 secs 2024-07-26T09:40:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:44 smithi115 ceph-mon[35364]: pgmap v3919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:44 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6082.763796 secs 2024-07-26T09:40:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:44 smithi181 ceph-mon[35869]: pgmap v3919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:44 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6082.763796 secs 2024-07-26T09:40:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:47 smithi115 ceph-mon[35364]: pgmap v3920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:47 smithi181 ceph-mon[35869]: pgmap v3920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:47 smithi097 ceph-mon[32231]: pgmap v3920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:49 smithi115 ceph-mon[35364]: pgmap v3921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:49 smithi181 ceph-mon[35869]: pgmap v3921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:49 smithi097 ceph-mon[32231]: pgmap v3921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:50.342+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:40:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:50.343+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:40:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:50.343+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:40:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:50.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:40:50.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:40:50.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6087.763952 secs 2024-07-26T09:40:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6087.763952 secs 2024-07-26T09:40:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6087.763952 secs 2024-07-26T09:40:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:51 smithi115 ceph-mon[35364]: pgmap v3922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:51 smithi181 ceph-mon[35869]: pgmap v3922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:51 smithi097 ceph-mon[32231]: pgmap v3922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: pgmap v3923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: pgmap v3924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: pgmap v3923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:40:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: pgmap v3924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:52.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:40:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: pgmap v3923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: pgmap v3924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:52.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:40:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:54 smithi115 ceph-mon[35364]: pgmap v3925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:40:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:54 smithi181 ceph-mon[35869]: pgmap v3925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:40:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:54 smithi097 ceph-mon[32231]: pgmap v3925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:40:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6092.764128 secs 2024-07-26T09:40:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6092.764128 secs 2024-07-26T09:40:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6092.764128 secs 2024-07-26T09:40:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:56 smithi097 ceph-mon[32231]: pgmap v3926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:40:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:56 smithi115 ceph-mon[35364]: pgmap v3926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:40:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:56 smithi181 ceph-mon[35869]: pgmap v3926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:40:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:40:58 smithi097 ceph-mon[32231]: pgmap v3927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:40:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:40:58 smithi115 ceph-mon[35364]: pgmap v3927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:40:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:40:58 smithi181 ceph-mon[35869]: pgmap v3927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:41:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:00 smithi097 ceph-mon[32231]: pgmap v3928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:41:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6097.764282 secs 2024-07-26T09:41:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:00 smithi115 ceph-mon[35364]: pgmap v3928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:41:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6097.764282 secs 2024-07-26T09:41:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:00 smithi181 ceph-mon[35869]: pgmap v3928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:41:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6097.764282 secs 2024-07-26T09:41:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:02 smithi115 ceph-mon[35364]: pgmap v3929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:41:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:02 smithi181 ceph-mon[35869]: pgmap v3929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:41:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:02 smithi097 ceph-mon[32231]: pgmap v3929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:41:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:04 smithi115 ceph-mon[35364]: pgmap v3930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:41:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:04 smithi181 ceph-mon[35869]: pgmap v3930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:41:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:04 smithi097 ceph-mon[32231]: pgmap v3930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:41:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:05.349+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:41:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:05.350+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:41:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:05.350+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:41:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:05.350+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:41:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:05.350+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6102.764444 secs 2024-07-26T09:41:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6102.764444 secs 2024-07-26T09:41:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6102.764444 secs 2024-07-26T09:41:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:06 smithi115 ceph-mon[35364]: pgmap v3931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:41:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:06 smithi181 ceph-mon[35869]: pgmap v3931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:41:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:06 smithi097 ceph-mon[32231]: pgmap v3931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:41:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:08 smithi115 ceph-mon[35364]: pgmap v3932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:41:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:08 smithi181 ceph-mon[35869]: pgmap v3932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:41:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:08 smithi097 ceph-mon[32231]: pgmap v3932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:41:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:10 smithi115 ceph-mon[35364]: pgmap v3933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:41:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6107.764589 secs 2024-07-26T09:41:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:10 smithi181 ceph-mon[35869]: pgmap v3933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:41:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6107.764589 secs 2024-07-26T09:41:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:10 smithi097 ceph-mon[32231]: pgmap v3933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:41:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6107.764589 secs 2024-07-26T09:41:12.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:12 smithi097 ceph-mon[32231]: pgmap v3934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:12 smithi115 ceph-mon[35364]: pgmap v3934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:12 smithi181 ceph-mon[35869]: pgmap v3934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:14 smithi115 ceph-mon[35364]: pgmap v3935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:14 smithi181 ceph-mon[35869]: pgmap v3935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:14 smithi097 ceph-mon[32231]: pgmap v3935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6112.764765 secs 2024-07-26T09:41:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6112.764765 secs 2024-07-26T09:41:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6112.764765 secs 2024-07-26T09:41:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:16 smithi115 ceph-mon[35364]: pgmap v3936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:16 smithi181 ceph-mon[35869]: pgmap v3936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:16 smithi097 ceph-mon[32231]: pgmap v3936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:18 smithi115 ceph-mon[35364]: pgmap v3937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:18 smithi181 ceph-mon[35869]: pgmap v3937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:18 smithi097 ceph-mon[32231]: pgmap v3937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:20.361+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:41:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:20.362+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:41:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:20.362+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:41:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:20.362+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:41:20.365 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:20.362+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:20 smithi115 ceph-mon[35364]: pgmap v3938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6117.764927 secs 2024-07-26T09:41:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:20 smithi181 ceph-mon[35869]: pgmap v3938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6117.764927 secs 2024-07-26T09:41:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:20 smithi097 ceph-mon[32231]: pgmap v3938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6117.764927 secs 2024-07-26T09:41:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:22 smithi115 ceph-mon[35364]: pgmap v3939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:22 smithi181 ceph-mon[35869]: pgmap v3939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:22 smithi097 ceph-mon[32231]: pgmap v3939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:24 smithi115 ceph-mon[35364]: pgmap v3940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:24 smithi181 ceph-mon[35869]: pgmap v3940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:24 smithi097 ceph-mon[32231]: pgmap v3940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6122.765083 secs 2024-07-26T09:41:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6122.765083 secs 2024-07-26T09:41:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6122.765083 secs 2024-07-26T09:41:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:26 smithi115 ceph-mon[35364]: pgmap v3941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:26 smithi181 ceph-mon[35869]: pgmap v3941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:26 smithi097 ceph-mon[32231]: pgmap v3941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:28 smithi115 ceph-mon[35364]: pgmap v3942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:28 smithi181 ceph-mon[35869]: pgmap v3942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:28 smithi097 ceph-mon[32231]: pgmap v3942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:30 smithi115 ceph-mon[35364]: pgmap v3943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6127.765289 secs 2024-07-26T09:41:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:30 smithi181 ceph-mon[35869]: pgmap v3943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6127.765289 secs 2024-07-26T09:41:30.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:30 smithi097 ceph-mon[32231]: pgmap v3943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:30.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6127.765289 secs 2024-07-26T09:41:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:32 smithi115 ceph-mon[35364]: pgmap v3944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:32 smithi181 ceph-mon[35869]: pgmap v3944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:32 smithi097 ceph-mon[32231]: pgmap v3944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:34 smithi115 ceph-mon[35364]: pgmap v3945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:34 smithi181 ceph-mon[35869]: pgmap v3945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:34 smithi097 ceph-mon[32231]: pgmap v3945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:35.367+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:41:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:35.367+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:41:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:35.367+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:41:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:35.367+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:41:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:35.367+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6132.765444 secs 2024-07-26T09:41:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6132.765444 secs 2024-07-26T09:41:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6132.765444 secs 2024-07-26T09:41:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:36 smithi115 ceph-mon[35364]: pgmap v3946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:36 smithi181 ceph-mon[35869]: pgmap v3946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:36 smithi097 ceph-mon[32231]: pgmap v3946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:38 smithi115 ceph-mon[35364]: pgmap v3947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:38 smithi181 ceph-mon[35869]: pgmap v3947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:38 smithi097 ceph-mon[32231]: pgmap v3947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:40 smithi115 ceph-mon[35364]: pgmap v3948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6137.765552 secs 2024-07-26T09:41:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:40 smithi181 ceph-mon[35869]: pgmap v3948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:40.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6137.765552 secs 2024-07-26T09:41:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:40 smithi097 ceph-mon[32231]: pgmap v3948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6137.765552 secs 2024-07-26T09:41:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:42 smithi115 ceph-mon[35364]: pgmap v3949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:42 smithi181 ceph-mon[35869]: pgmap v3949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:42 smithi097 ceph-mon[32231]: pgmap v3949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:44 smithi115 ceph-mon[35364]: pgmap v3950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:44 smithi181 ceph-mon[35869]: pgmap v3950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:44 smithi097 ceph-mon[32231]: pgmap v3950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6142.765662 secs 2024-07-26T09:41:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6142.765662 secs 2024-07-26T09:41:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6142.765662 secs 2024-07-26T09:41:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:46 smithi115 ceph-mon[35364]: pgmap v3951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:46 smithi181 ceph-mon[35869]: pgmap v3951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:46 smithi097 ceph-mon[32231]: pgmap v3951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:48 smithi115 ceph-mon[35364]: pgmap v3952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:48 smithi181 ceph-mon[35869]: pgmap v3952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:48 smithi097 ceph-mon[32231]: pgmap v3952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:50.377+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:41:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:50.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:41:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:50.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:41:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:50.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:41:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:41:50.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:50 smithi115 ceph-mon[35364]: pgmap v3953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6147.765815 secs 2024-07-26T09:41:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:50 smithi181 ceph-mon[35869]: pgmap v3953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6147.765815 secs 2024-07-26T09:41:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:50 smithi097 ceph-mon[32231]: pgmap v3953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:41:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6147.765815 secs 2024-07-26T09:41:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:52 smithi181 ceph-mon[35869]: pgmap v3954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:52 smithi097 ceph-mon[32231]: pgmap v3954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:52 smithi115 ceph-mon[35364]: pgmap v3954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:41:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:41:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:41:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:41:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:41:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:41:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:41:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:41:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:54 smithi097 ceph-mon[32231]: pgmap v3955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:54 smithi097 ceph-mon[32231]: pgmap v3956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:41:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:54 smithi115 ceph-mon[35364]: pgmap v3955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:54 smithi115 ceph-mon[35364]: pgmap v3956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:41:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:54 smithi181 ceph-mon[35869]: pgmap v3955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:54 smithi181 ceph-mon[35869]: pgmap v3956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:41:55.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6152.765976 secs 2024-07-26T09:41:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6152.765976 secs 2024-07-26T09:41:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6152.765976 secs 2024-07-26T09:41:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:56 smithi115 ceph-mon[35364]: pgmap v3957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:41:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:56 smithi181 ceph-mon[35869]: pgmap v3957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:41:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:56 smithi097 ceph-mon[32231]: pgmap v3957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:41:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:41:58 smithi115 ceph-mon[35364]: pgmap v3958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:41:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:41:58 smithi181 ceph-mon[35869]: pgmap v3958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:41:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:41:58 smithi097 ceph-mon[32231]: pgmap v3958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:00 smithi115 ceph-mon[35364]: pgmap v3959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6157.766149 secs 2024-07-26T09:42:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:00 smithi181 ceph-mon[35869]: pgmap v3959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6157.766149 secs 2024-07-26T09:42:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:00 smithi097 ceph-mon[32231]: pgmap v3959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6157.766149 secs 2024-07-26T09:42:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:02 smithi115 ceph-mon[35364]: pgmap v3960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:02 smithi181 ceph-mon[35869]: pgmap v3960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:02 smithi097 ceph-mon[32231]: pgmap v3960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:04 smithi115 ceph-mon[35364]: pgmap v3961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:42:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:04 smithi181 ceph-mon[35869]: pgmap v3961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:42:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:04 smithi097 ceph-mon[32231]: pgmap v3961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:42:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:05.385+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:42:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:05.386+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:42:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:05.386+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:42:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:05.386+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:42:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:05.386+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6162.766303 secs 2024-07-26T09:42:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:05 smithi115 ceph-mon[35364]: pgmap v3962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6162.766303 secs 2024-07-26T09:42:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:05 smithi181 ceph-mon[35869]: pgmap v3962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6162.766303 secs 2024-07-26T09:42:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:05 smithi097 ceph-mon[32231]: pgmap v3962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:08 smithi115 ceph-mon[35364]: pgmap v3963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:08 smithi181 ceph-mon[35869]: pgmap v3963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:08 smithi097 ceph-mon[32231]: pgmap v3963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:10 smithi115 ceph-mon[35364]: pgmap v3964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6167.766412 secs 2024-07-26T09:42:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:10 smithi181 ceph-mon[35869]: pgmap v3964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6167.766412 secs 2024-07-26T09:42:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:10 smithi097 ceph-mon[32231]: pgmap v3964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6167.766412 secs 2024-07-26T09:42:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:12 smithi115 ceph-mon[35364]: pgmap v3965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:12 smithi181 ceph-mon[35869]: pgmap v3965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:12 smithi097 ceph-mon[32231]: pgmap v3965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:14 smithi115 ceph-mon[35364]: pgmap v3966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:14 smithi181 ceph-mon[35869]: pgmap v3966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:14 smithi097 ceph-mon[32231]: pgmap v3966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6172.766565 secs 2024-07-26T09:42:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6172.766565 secs 2024-07-26T09:42:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6172.766565 secs 2024-07-26T09:42:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:16 smithi115 ceph-mon[35364]: pgmap v3967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:42:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:16 smithi181 ceph-mon[35869]: pgmap v3967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:42:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:16 smithi097 ceph-mon[32231]: pgmap v3967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:42:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:18 smithi115 ceph-mon[35364]: pgmap v3968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:42:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:18 smithi181 ceph-mon[35869]: pgmap v3968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:42:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:18 smithi097 ceph-mon[32231]: pgmap v3968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:42:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:20.394+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:42:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:20.395+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:42:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:20.395+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:42:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:20.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:42:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:20.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:20 smithi181 ceph-mon[35869]: pgmap v3969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:20.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6177.766671 secs 2024-07-26T09:42:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:20 smithi097 ceph-mon[32231]: pgmap v3969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6177.766671 secs 2024-07-26T09:42:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:20 smithi115 ceph-mon[35364]: pgmap v3969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6177.766671 secs 2024-07-26T09:42:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:22 smithi097 ceph-mon[32231]: pgmap v3970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:22 smithi115 ceph-mon[35364]: pgmap v3970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:22 smithi181 ceph-mon[35869]: pgmap v3970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:24 smithi097 ceph-mon[32231]: pgmap v3971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:24 smithi115 ceph-mon[35364]: pgmap v3971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:24 smithi181 ceph-mon[35869]: pgmap v3971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6182.766844 secs 2024-07-26T09:42:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6182.766844 secs 2024-07-26T09:42:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6182.766844 secs 2024-07-26T09:42:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:26 smithi115 ceph-mon[35364]: pgmap v3972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:26 smithi181 ceph-mon[35869]: pgmap v3972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:26 smithi097 ceph-mon[32231]: pgmap v3972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:28 smithi115 ceph-mon[35364]: pgmap v3973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:28 smithi181 ceph-mon[35869]: pgmap v3973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:28 smithi097 ceph-mon[32231]: pgmap v3973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:30 smithi115 ceph-mon[35364]: pgmap v3974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6187.766997 secs 2024-07-26T09:42:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:30 smithi181 ceph-mon[35869]: pgmap v3974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:30.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6187.766997 secs 2024-07-26T09:42:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:30 smithi097 ceph-mon[32231]: pgmap v3974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6187.766997 secs 2024-07-26T09:42:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:32 smithi115 ceph-mon[35364]: pgmap v3975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:32 smithi181 ceph-mon[35869]: pgmap v3975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:32 smithi097 ceph-mon[32231]: pgmap v3975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:34 smithi115 ceph-mon[35364]: pgmap v3976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:34 smithi181 ceph-mon[35869]: pgmap v3976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:34 smithi097 ceph-mon[32231]: pgmap v3976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:35.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:35.400+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:42:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:35.402+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:42:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:35.402+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:42:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:35.402+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:42:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:35.402+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6192.767147 secs 2024-07-26T09:42:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6192.767147 secs 2024-07-26T09:42:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6192.767147 secs 2024-07-26T09:42:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:36 smithi115 ceph-mon[35364]: pgmap v3977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:36.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:36 smithi181 ceph-mon[35869]: pgmap v3977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:36 smithi097 ceph-mon[32231]: pgmap v3977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:37 smithi115 ceph-mon[35364]: pgmap v3978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:37 smithi181 ceph-mon[35869]: pgmap v3978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:37 smithi097 ceph-mon[32231]: pgmap v3978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:40 smithi115 ceph-mon[35364]: pgmap v3979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6197.767305 secs 2024-07-26T09:42:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:40 smithi181 ceph-mon[35869]: pgmap v3979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6197.767305 secs 2024-07-26T09:42:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:40 smithi097 ceph-mon[32231]: pgmap v3979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6197.767305 secs 2024-07-26T09:42:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:42 smithi115 ceph-mon[35364]: pgmap v3980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:42 smithi181 ceph-mon[35869]: pgmap v3980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:42 smithi097 ceph-mon[32231]: pgmap v3980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:44 smithi115 ceph-mon[35364]: pgmap v3981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:44 smithi181 ceph-mon[35869]: pgmap v3981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:44 smithi097 ceph-mon[32231]: pgmap v3981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6202.767420 secs 2024-07-26T09:42:45.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6202.767420 secs 2024-07-26T09:42:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6202.767420 secs 2024-07-26T09:42:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:46 smithi115 ceph-mon[35364]: pgmap v3982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:42:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:46 smithi181 ceph-mon[35869]: pgmap v3982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:42:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:46 smithi097 ceph-mon[32231]: pgmap v3982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:42:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:48 smithi181 ceph-mon[35869]: pgmap v3983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:42:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:48 smithi097 ceph-mon[32231]: pgmap v3983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:42:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:48 smithi115 ceph-mon[35364]: pgmap v3983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:42:50.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:50.413+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:42:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:50.414+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:42:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:50.414+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:42:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:50.414+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:42:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:42:50.414+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:50 smithi097 ceph-mon[32231]: pgmap v3984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6207.767558 secs 2024-07-26T09:42:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:50 smithi115 ceph-mon[35364]: pgmap v3984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6207.767558 secs 2024-07-26T09:42:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:50 smithi181 ceph-mon[35869]: pgmap v3984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6207.767558 secs 2024-07-26T09:42:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:52 smithi097 ceph-mon[32231]: pgmap v3985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:52 smithi115 ceph-mon[35364]: pgmap v3985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:52 smithi181 ceph-mon[35869]: pgmap v3985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:54 smithi097 ceph-mon[32231]: pgmap v3986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:42:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:54 smithi115 ceph-mon[35364]: pgmap v3986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:42:54.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:54 smithi181 ceph-mon[35869]: pgmap v3986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:42:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:54.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6212.767637 secs 2024-07-26T09:42:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6212.767637 secs 2024-07-26T09:42:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6212.767637 secs 2024-07-26T09:42:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:55.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:55.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:42:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:56 smithi115 ceph-mon[35364]: pgmap v3987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:56 smithi115 ceph-mon[35364]: pgmap v3988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:56 smithi181 ceph-mon[35869]: pgmap v3987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:56 smithi181 ceph-mon[35869]: pgmap v3988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:56 smithi097 ceph-mon[32231]: pgmap v3987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:56 smithi097 ceph-mon[32231]: pgmap v3988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:42:58 smithi115 ceph-mon[35364]: pgmap v3989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:42:58 smithi181 ceph-mon[35869]: pgmap v3989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:42:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:42:58 smithi097 ceph-mon[32231]: pgmap v3989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:00 smithi115 ceph-mon[35364]: pgmap v3990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6217.767773 secs 2024-07-26T09:43:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:00 smithi181 ceph-mon[35869]: pgmap v3990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6217.767773 secs 2024-07-26T09:43:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:00 smithi097 ceph-mon[32231]: pgmap v3990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6217.767773 secs 2024-07-26T09:43:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:02 smithi115 ceph-mon[35364]: pgmap v3991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:02 smithi181 ceph-mon[35869]: pgmap v3991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:02 smithi097 ceph-mon[32231]: pgmap v3991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:04 smithi115 ceph-mon[35364]: pgmap v3992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:04 smithi181 ceph-mon[35869]: pgmap v3992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:04 smithi097 ceph-mon[32231]: pgmap v3992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:05.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:05.417+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:43:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:05.418+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:43:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:05.418+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:43:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:05.418+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:43:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:05.418+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6222.767913 secs 2024-07-26T09:43:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6222.767913 secs 2024-07-26T09:43:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6222.767913 secs 2024-07-26T09:43:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:06 smithi115 ceph-mon[35364]: pgmap v3993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:43:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:06 smithi181 ceph-mon[35869]: pgmap v3993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:43:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:06 smithi097 ceph-mon[32231]: pgmap v3993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:43:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.896+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T09:43:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.896+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:43:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.896+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e008f000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:43:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.896+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e008f000 comp rx=0 tx=0).stop 2024-07-26T09:43:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.896+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T09:43:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.896+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:43:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.896+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:43:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.896+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 0x7ff3d40b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:43:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.897+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 0x7ff3d40b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:43:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:06.898+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 0x7ff3d40b4fa0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e0071000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:43:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:07.978+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:43:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:07.978+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006640 con 0x7ff3f01061a0 2024-07-26T09:43:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:07.979+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:43:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:07.979+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006640 con 0x7ff3f01061a0 2024-07-26T09:43:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:08 smithi115 ceph-mon[35364]: pgmap v3994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:08 smithi181 ceph-mon[35869]: pgmap v3994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:08 smithi097 ceph-mon[32231]: pgmap v3994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:10 smithi115 ceph-mon[35364]: pgmap v3995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6227.768054 secs 2024-07-26T09:43:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:10 smithi181 ceph-mon[35869]: pgmap v3995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6227.768054 secs 2024-07-26T09:43:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:10 smithi097 ceph-mon[32231]: pgmap v3995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6227.768054 secs 2024-07-26T09:43:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:12 smithi115 ceph-mon[35364]: pgmap v3996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:12 smithi181 ceph-mon[35869]: pgmap v3996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:12 smithi097 ceph-mon[32231]: pgmap v3996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:14 smithi115 ceph-mon[35364]: pgmap v3997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:14 smithi181 ceph-mon[35869]: pgmap v3997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:14 smithi097 ceph-mon[32231]: pgmap v3997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6232.768198 secs 2024-07-26T09:43:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6232.768198 secs 2024-07-26T09:43:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6232.768198 secs 2024-07-26T09:43:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:16 smithi115 ceph-mon[35364]: pgmap v3998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:16 smithi181 ceph-mon[35869]: pgmap v3998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:16 smithi097 ceph-mon[32231]: pgmap v3998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:18 smithi115 ceph-mon[35364]: pgmap v3999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:18 smithi181 ceph-mon[35869]: pgmap v3999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:18 smithi097 ceph-mon[32231]: pgmap v3999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:20.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:20.430+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:20.430+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:20.430+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:43:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:20.430+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:20.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:20 smithi115 ceph-mon[35364]: pgmap v4000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:20.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6237.768342 secs 2024-07-26T09:43:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:20 smithi181 ceph-mon[35869]: pgmap v4000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6237.768342 secs 2024-07-26T09:43:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:20 smithi097 ceph-mon[32231]: pgmap v4000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6237.768342 secs 2024-07-26T09:43:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:22 smithi115 ceph-mon[35364]: pgmap v4001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:22 smithi181 ceph-mon[35869]: pgmap v4001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:22 smithi097 ceph-mon[32231]: pgmap v4001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:24 smithi115 ceph-mon[35364]: pgmap v4002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:24 smithi181 ceph-mon[35869]: pgmap v4002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:24 smithi097 ceph-mon[32231]: pgmap v4002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6242.768491 secs 2024-07-26T09:43:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6242.768491 secs 2024-07-26T09:43:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6242.768491 secs 2024-07-26T09:43:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:26 smithi115 ceph-mon[35364]: pgmap v4003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:26.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:26 smithi181 ceph-mon[35869]: pgmap v4003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:26 smithi097 ceph-mon[32231]: pgmap v4003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:28 smithi115 ceph-mon[35364]: pgmap v4004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:28.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:28 smithi181 ceph-mon[35869]: pgmap v4004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:28 smithi097 ceph-mon[32231]: pgmap v4004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:30 smithi115 ceph-mon[35364]: pgmap v4005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6247.768658 secs 2024-07-26T09:43:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:30 smithi181 ceph-mon[35869]: pgmap v4005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6247.768658 secs 2024-07-26T09:43:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:30 smithi097 ceph-mon[32231]: pgmap v4005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6247.768658 secs 2024-07-26T09:43:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:32 smithi115 ceph-mon[35364]: pgmap v4006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:32 smithi181 ceph-mon[35869]: pgmap v4006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:32 smithi097 ceph-mon[32231]: pgmap v4006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:34 smithi115 ceph-mon[35364]: pgmap v4007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:34 smithi181 ceph-mon[35869]: pgmap v4007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:34 smithi097 ceph-mon[32231]: pgmap v4007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:35.435+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:43:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:35.436+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:43:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:35.436+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:43:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:35.436+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:43:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:35.436+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6252.768765 secs 2024-07-26T09:43:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6252.768765 secs 2024-07-26T09:43:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6252.768765 secs 2024-07-26T09:43:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:36 smithi115 ceph-mon[35364]: pgmap v4008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:43:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:36 smithi181 ceph-mon[35869]: pgmap v4008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:43:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:36 smithi097 ceph-mon[32231]: pgmap v4008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:43:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:38 smithi115 ceph-mon[35364]: pgmap v4009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:43:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:38 smithi181 ceph-mon[35869]: pgmap v4009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:43:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:38 smithi097 ceph-mon[32231]: pgmap v4009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:43:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:40 smithi115 ceph-mon[35364]: pgmap v4010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:43:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6257.768907 secs 2024-07-26T09:43:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:40 smithi181 ceph-mon[35869]: pgmap v4010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:43:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6257.768907 secs 2024-07-26T09:43:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:40 smithi097 ceph-mon[32231]: pgmap v4010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:43:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6257.768907 secs 2024-07-26T09:43:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:42 smithi181 ceph-mon[35869]: pgmap v4011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:42 smithi097 ceph-mon[32231]: pgmap v4011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:42 smithi115 ceph-mon[35364]: pgmap v4011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:44.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:44 smithi181 ceph-mon[35869]: pgmap v4012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:43:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:44 smithi097 ceph-mon[32231]: pgmap v4012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:43:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:44 smithi115 ceph-mon[35364]: pgmap v4012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:43:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6262.769054 secs 2024-07-26T09:43:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6262.769054 secs 2024-07-26T09:43:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6262.769054 secs 2024-07-26T09:43:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:46 smithi097 ceph-mon[32231]: pgmap v4013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:43:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:46 smithi115 ceph-mon[35364]: pgmap v4013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:43:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:46 smithi181 ceph-mon[35869]: pgmap v4013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:43:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:48 smithi115 ceph-mon[35364]: pgmap v4014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:48 smithi181 ceph-mon[35869]: pgmap v4014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:48 smithi097 ceph-mon[32231]: pgmap v4014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:50.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:43:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:50.442+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:43:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:50.442+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:43:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:50.442+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:43:50.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:43:50.442+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:50 smithi115 ceph-mon[35364]: pgmap v4015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6267.769189 secs 2024-07-26T09:43:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:50 smithi181 ceph-mon[35869]: pgmap v4015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:50.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6267.769189 secs 2024-07-26T09:43:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:50 smithi097 ceph-mon[32231]: pgmap v4015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6267.769189 secs 2024-07-26T09:43:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:52 smithi115 ceph-mon[35364]: pgmap v4016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:52 smithi181 ceph-mon[35869]: pgmap v4016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:52 smithi097 ceph-mon[32231]: pgmap v4016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:54 smithi115 ceph-mon[35364]: pgmap v4017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:54 smithi181 ceph-mon[35869]: pgmap v4017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:54 smithi097 ceph-mon[32231]: pgmap v4017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6272.769331 secs 2024-07-26T09:43:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6272.769331 secs 2024-07-26T09:43:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6272.769331 secs 2024-07-26T09:43:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:56 smithi115 ceph-mon[35364]: pgmap v4018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:56 smithi181 ceph-mon[35869]: pgmap v4018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:56 smithi097 ceph-mon[32231]: pgmap v4018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:58 smithi115 ceph-mon[35364]: pgmap v4019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:58 smithi181 ceph-mon[35869]: pgmap v4019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:58.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:58.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:58 smithi097 ceph-mon[32231]: pgmap v4019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:43:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:43:59.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:43:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:44:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:44:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:44:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:43:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:44:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:00 smithi115 ceph-mon[35364]: pgmap v4020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:44:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6277.769468 secs 2024-07-26T09:44:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:00.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:44:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:00 smithi181 ceph-mon[35869]: pgmap v4020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:44:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6277.769468 secs 2024-07-26T09:44:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:44:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:00 smithi097 ceph-mon[32231]: pgmap v4020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:44:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6277.769468 secs 2024-07-26T09:44:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:01.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:01.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:44:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:01 smithi115 ceph-mon[35364]: pgmap v4021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:44:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:01 smithi115 ceph-mon[35364]: pgmap v4022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T09:44:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:01 smithi181 ceph-mon[35869]: pgmap v4021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:44:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:01 smithi181 ceph-mon[35869]: pgmap v4022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T09:44:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:01 smithi097 ceph-mon[32231]: pgmap v4021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:44:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:01 smithi097 ceph-mon[32231]: pgmap v4022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T09:44:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:03 smithi115 ceph-mon[35364]: pgmap v4023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:44:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:03 smithi181 ceph-mon[35869]: pgmap v4023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:44:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:03 smithi097 ceph-mon[32231]: pgmap v4023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:44:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:04 smithi115 ceph-mon[35364]: pgmap v4024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:44:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:04 smithi181 ceph-mon[35869]: pgmap v4024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:44:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:04 smithi097 ceph-mon[32231]: pgmap v4024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:44:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:05.447+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:44:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:05.448+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:44:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:05.448+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:44:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:05.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:44:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:05.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6282.769561 secs 2024-07-26T09:44:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6282.769561 secs 2024-07-26T09:44:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6282.769561 secs 2024-07-26T09:44:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:06 smithi115 ceph-mon[35364]: pgmap v4025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:44:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:06 smithi181 ceph-mon[35869]: pgmap v4025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:44:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:06 smithi097 ceph-mon[32231]: pgmap v4025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:44:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:09 smithi115 ceph-mon[35364]: pgmap v4026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:44:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:09 smithi181 ceph-mon[35869]: pgmap v4026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:44:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:09 smithi097 ceph-mon[32231]: pgmap v4026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:44:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6287.769714 secs 2024-07-26T09:44:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6287.769714 secs 2024-07-26T09:44:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6287.769714 secs 2024-07-26T09:44:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:11 smithi115 ceph-mon[35364]: pgmap v4027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:44:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:11 smithi181 ceph-mon[35869]: pgmap v4027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:44:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:11 smithi097 ceph-mon[32231]: pgmap v4027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:44:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:13 smithi115 ceph-mon[35364]: pgmap v4028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:13 smithi181 ceph-mon[35869]: pgmap v4028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:13 smithi097 ceph-mon[32231]: pgmap v4028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:15 smithi181 ceph-mon[35869]: pgmap v4029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6292.769845 secs 2024-07-26T09:44:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:15 smithi115 ceph-mon[35364]: pgmap v4029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6292.769845 secs 2024-07-26T09:44:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:15 smithi097 ceph-mon[32231]: pgmap v4029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6292.769845 secs 2024-07-26T09:44:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:17 smithi115 ceph-mon[35364]: pgmap v4030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:17 smithi181 ceph-mon[35869]: pgmap v4030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:17 smithi097 ceph-mon[32231]: pgmap v4030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:19 smithi115 ceph-mon[35364]: pgmap v4031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:19 smithi181 ceph-mon[35869]: pgmap v4031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:19 smithi097 ceph-mon[32231]: pgmap v4031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:20.458+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:44:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:20.458+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:44:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:20.458+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:44:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:20.458+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:44:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:20.458+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6297.770003 secs 2024-07-26T09:44:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6297.770003 secs 2024-07-26T09:44:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6297.770003 secs 2024-07-26T09:44:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:21 smithi115 ceph-mon[35364]: pgmap v4032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:21 smithi181 ceph-mon[35869]: pgmap v4032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:21 smithi097 ceph-mon[32231]: pgmap v4032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:23 smithi115 ceph-mon[35364]: pgmap v4033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:23 smithi181 ceph-mon[35869]: pgmap v4033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:23 smithi097 ceph-mon[32231]: pgmap v4033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:25 smithi115 ceph-mon[35364]: pgmap v4034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6302.770130 secs 2024-07-26T09:44:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:25 smithi181 ceph-mon[35869]: pgmap v4034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6302.770130 secs 2024-07-26T09:44:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:25 smithi097 ceph-mon[32231]: pgmap v4034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6302.770130 secs 2024-07-26T09:44:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:27 smithi115 ceph-mon[35364]: pgmap v4035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:27 smithi181 ceph-mon[35869]: pgmap v4035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:27 smithi097 ceph-mon[32231]: pgmap v4035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:29 smithi115 ceph-mon[35364]: pgmap v4036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:29 smithi181 ceph-mon[35869]: pgmap v4036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:29 smithi097 ceph-mon[32231]: pgmap v4036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6307.770224 secs 2024-07-26T09:44:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6307.770224 secs 2024-07-26T09:44:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6307.770224 secs 2024-07-26T09:44:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:31 smithi115 ceph-mon[35364]: pgmap v4037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:31 smithi181 ceph-mon[35869]: pgmap v4037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:31 smithi097 ceph-mon[32231]: pgmap v4037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:33 smithi115 ceph-mon[35364]: pgmap v4038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:33 smithi181 ceph-mon[35869]: pgmap v4038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:33 smithi097 ceph-mon[32231]: pgmap v4038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:35.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:35.464+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:44:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:35.465+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:44:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:35.465+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:44:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:35.465+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:44:35.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:35.465+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:35 smithi115 ceph-mon[35364]: pgmap v4039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6312.770372 secs 2024-07-26T09:44:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:35 smithi181 ceph-mon[35869]: pgmap v4039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6312.770372 secs 2024-07-26T09:44:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:35 smithi097 ceph-mon[32231]: pgmap v4039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6312.770372 secs 2024-07-26T09:44:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:37 smithi115 ceph-mon[35364]: pgmap v4040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:37 smithi181 ceph-mon[35869]: pgmap v4040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:37 smithi097 ceph-mon[32231]: pgmap v4040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:39 smithi115 ceph-mon[35364]: pgmap v4041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:39 smithi181 ceph-mon[35869]: pgmap v4041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:39 smithi097 ceph-mon[32231]: pgmap v4041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6317.770440 secs 2024-07-26T09:44:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6317.770440 secs 2024-07-26T09:44:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6317.770440 secs 2024-07-26T09:44:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:41 smithi097 ceph-mon[32231]: pgmap v4042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:41 smithi115 ceph-mon[35364]: pgmap v4042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:41 smithi181 ceph-mon[35869]: pgmap v4042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:43 smithi097 ceph-mon[32231]: pgmap v4043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:43.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:43 smithi115 ceph-mon[35364]: pgmap v4043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:43.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:43 smithi181 ceph-mon[35869]: pgmap v4043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:45 smithi097 ceph-mon[32231]: pgmap v4044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6322.770583 secs 2024-07-26T09:44:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:45 smithi115 ceph-mon[35364]: pgmap v4044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6322.770583 secs 2024-07-26T09:44:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:45 smithi181 ceph-mon[35869]: pgmap v4044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:45.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6322.770583 secs 2024-07-26T09:44:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:47 smithi115 ceph-mon[35364]: pgmap v4045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:47 smithi181 ceph-mon[35869]: pgmap v4045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:47 smithi097 ceph-mon[32231]: pgmap v4045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:49 smithi115 ceph-mon[35364]: pgmap v4046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:49 smithi181 ceph-mon[35869]: pgmap v4046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:49 smithi097 ceph-mon[32231]: pgmap v4046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:50.475+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:44:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:50.477+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:44:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:50.477+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:44:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:50.477+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:44:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:44:50.477+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6327.770731 secs 2024-07-26T09:44:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6327.770731 secs 2024-07-26T09:44:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6327.770731 secs 2024-07-26T09:44:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:51 smithi115 ceph-mon[35364]: pgmap v4047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:51 smithi181 ceph-mon[35869]: pgmap v4047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:51 smithi097 ceph-mon[32231]: pgmap v4047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:53 smithi115 ceph-mon[35364]: pgmap v4048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:53 smithi181 ceph-mon[35869]: pgmap v4048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:53 smithi097 ceph-mon[32231]: pgmap v4048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:55 smithi115 ceph-mon[35364]: pgmap v4049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:44:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6332.770869 secs 2024-07-26T09:44:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:55 smithi181 ceph-mon[35869]: pgmap v4049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:44:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6332.770869 secs 2024-07-26T09:44:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:55 smithi097 ceph-mon[32231]: pgmap v4049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:44:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6332.770869 secs 2024-07-26T09:44:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:57 smithi115 ceph-mon[35364]: pgmap v4050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:57 smithi181 ceph-mon[35869]: pgmap v4050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:57 smithi097 ceph-mon[32231]: pgmap v4050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:44:59 smithi115 ceph-mon[35364]: pgmap v4051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:44:59 smithi181 ceph-mon[35869]: pgmap v4051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:44:59 smithi097 ceph-mon[32231]: pgmap v4051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6337.771003 secs 2024-07-26T09:45:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6337.771003 secs 2024-07-26T09:45:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6337.771003 secs 2024-07-26T09:45:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:01 smithi115 ceph-mon[35364]: pgmap v4052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:01 smithi181 ceph-mon[35869]: pgmap v4052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:01.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:01 smithi097 ceph-mon[32231]: pgmap v4052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:02 smithi115 ceph-mon[35364]: pgmap v4053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:45:02.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:02 smithi115 ceph-mon[35364]: pgmap v4054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:02.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:45:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:02.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:02 smithi181 ceph-mon[35869]: pgmap v4053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:45:02.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:02 smithi181 ceph-mon[35869]: pgmap v4054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:02.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:45:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:02.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:02 smithi097 ceph-mon[32231]: pgmap v4053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:45:02.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:02 smithi097 ceph-mon[32231]: pgmap v4054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:02.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:45:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:04 smithi115 ceph-mon[35364]: pgmap v4055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:45:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:04 smithi181 ceph-mon[35869]: pgmap v4055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:45:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:04 smithi097 ceph-mon[32231]: pgmap v4055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:45:05.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:05.483+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:45:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:05.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:45:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:05.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:45:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:05.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:45:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:05.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6342.771142 secs 2024-07-26T09:45:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6342.771142 secs 2024-07-26T09:45:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6342.771142 secs 2024-07-26T09:45:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:06 smithi115 ceph-mon[35364]: pgmap v4056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:06 smithi181 ceph-mon[35869]: pgmap v4056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:06 smithi097 ceph-mon[32231]: pgmap v4056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:07 smithi115 ceph-mon[35364]: pgmap v4057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:07 smithi181 ceph-mon[35869]: pgmap v4057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:07 smithi097 ceph-mon[32231]: pgmap v4057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:10 smithi115 ceph-mon[35364]: pgmap v4058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6347.771282 secs 2024-07-26T09:45:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:10 smithi181 ceph-mon[35869]: pgmap v4058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6347.771282 secs 2024-07-26T09:45:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:10 smithi097 ceph-mon[32231]: pgmap v4058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6347.771282 secs 2024-07-26T09:45:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:11 smithi115 ceph-mon[35364]: pgmap v4059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:45:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:11 smithi181 ceph-mon[35869]: pgmap v4059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:45:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:11 smithi097 ceph-mon[32231]: pgmap v4059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:45:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:14 smithi115 ceph-mon[35364]: pgmap v4060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:14 smithi181 ceph-mon[35869]: pgmap v4060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:14 smithi097 ceph-mon[32231]: pgmap v4060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6352.771427 secs 2024-07-26T09:45:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6352.771427 secs 2024-07-26T09:45:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6352.771427 secs 2024-07-26T09:45:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:16 smithi115 ceph-mon[35364]: pgmap v4061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:16 smithi181 ceph-mon[35869]: pgmap v4061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:16 smithi097 ceph-mon[32231]: pgmap v4061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:17 smithi115 ceph-mon[35364]: pgmap v4062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:17 smithi181 ceph-mon[35869]: pgmap v4062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:17 smithi097 ceph-mon[32231]: pgmap v4062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:20.495+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:45:20.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:20.496+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:45:20.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:20.496+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:45:20.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:20.496+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:45:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:20.496+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:20 smithi115 ceph-mon[35364]: pgmap v4063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6357.771558 secs 2024-07-26T09:45:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:20 smithi181 ceph-mon[35869]: pgmap v4063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6357.771558 secs 2024-07-26T09:45:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:20 smithi097 ceph-mon[32231]: pgmap v4063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6357.771558 secs 2024-07-26T09:45:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:21 smithi115 ceph-mon[35364]: pgmap v4064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:21 smithi181 ceph-mon[35869]: pgmap v4064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:21 smithi097 ceph-mon[32231]: pgmap v4064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:23.001+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0006640 con 0x7ff3f01061a0 2024-07-26T09:45:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:23.001+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:45:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:23.002+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:45:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:23.002+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c0006640 con 0x7ff3f01061a0 2024-07-26T09:45:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:24 smithi115 ceph-mon[35364]: pgmap v4065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:24 smithi181 ceph-mon[35869]: pgmap v4065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:24 smithi097 ceph-mon[32231]: pgmap v4065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6362.771703 secs 2024-07-26T09:45:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:25 smithi115 ceph-mon[35364]: pgmap v4066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:45:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6362.771703 secs 2024-07-26T09:45:25.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:25 smithi181 ceph-mon[35869]: pgmap v4066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:45:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6362.771703 secs 2024-07-26T09:45:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:25 smithi097 ceph-mon[32231]: pgmap v4066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:45:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:27 smithi115 ceph-mon[35364]: pgmap v4067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:45:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:27 smithi181 ceph-mon[35869]: pgmap v4067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:45:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:27 smithi097 ceph-mon[32231]: pgmap v4067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:45:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:29 smithi115 ceph-mon[35364]: pgmap v4068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:29 smithi181 ceph-mon[35869]: pgmap v4068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:29 smithi097 ceph-mon[32231]: pgmap v4068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:45:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6367.771838 secs 2024-07-26T09:45:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6367.771838 secs 2024-07-26T09:45:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6367.771838 secs 2024-07-26T09:45:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:31 smithi115 ceph-mon[35364]: pgmap v4069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:31 smithi181 ceph-mon[35869]: pgmap v4069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:31 smithi097 ceph-mon[32231]: pgmap v4069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:33 smithi115 ceph-mon[35364]: pgmap v4070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:33 smithi181 ceph-mon[35869]: pgmap v4070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:33 smithi097 ceph-mon[32231]: pgmap v4070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:34 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6372.771980 secs 2024-07-26T09:45:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:34 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6372.771980 secs 2024-07-26T09:45:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:34 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6372.771980 secs 2024-07-26T09:45:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:35.502+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:45:35.505 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:35.503+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:45:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:35.503+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:45:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:35.503+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:45:35.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:35.503+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:35 smithi115 ceph-mon[35364]: pgmap v4071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:35 smithi181 ceph-mon[35869]: pgmap v4071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:35 smithi097 ceph-mon[32231]: pgmap v4071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:37 smithi115 ceph-mon[35364]: pgmap v4072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:37 smithi181 ceph-mon[35869]: pgmap v4072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:37 smithi097 ceph-mon[32231]: pgmap v4072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:39 smithi115 ceph-mon[35364]: pgmap v4073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:39 smithi181 ceph-mon[35869]: pgmap v4073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:39 smithi097 ceph-mon[32231]: pgmap v4073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6377.772151 secs 2024-07-26T09:45:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6377.772151 secs 2024-07-26T09:45:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6377.772151 secs 2024-07-26T09:45:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:41 smithi115 ceph-mon[35364]: pgmap v4074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:41 smithi181 ceph-mon[35869]: pgmap v4074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:41 smithi097 ceph-mon[32231]: pgmap v4074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:43 smithi115 ceph-mon[35364]: pgmap v4075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:43 smithi181 ceph-mon[35869]: pgmap v4075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:43 smithi097 ceph-mon[32231]: pgmap v4075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:44 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6382.772310 secs 2024-07-26T09:45:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:44 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6382.772310 secs 2024-07-26T09:45:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:44 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6382.772310 secs 2024-07-26T09:45:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:45 smithi115 ceph-mon[35364]: pgmap v4076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:45 smithi181 ceph-mon[35869]: pgmap v4076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:45 smithi097 ceph-mon[32231]: pgmap v4076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:47 smithi115 ceph-mon[35364]: pgmap v4077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:47 smithi181 ceph-mon[35869]: pgmap v4077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:47 smithi097 ceph-mon[32231]: pgmap v4077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:49 smithi115 ceph-mon[35364]: pgmap v4078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:49 smithi181 ceph-mon[35869]: pgmap v4078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:49 smithi097 ceph-mon[32231]: pgmap v4078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:50.505+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:45:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:50.506+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:45:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:50.506+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:45:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:50.506+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:45:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:45:50.507+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6387.772445 secs 2024-07-26T09:45:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6387.772445 secs 2024-07-26T09:45:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6387.772445 secs 2024-07-26T09:45:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:51 smithi115 ceph-mon[35364]: pgmap v4079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:51 smithi181 ceph-mon[35869]: pgmap v4079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:51 smithi097 ceph-mon[32231]: pgmap v4079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:53 smithi115 ceph-mon[35364]: pgmap v4080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:53 smithi181 ceph-mon[35869]: pgmap v4080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:53 smithi097 ceph-mon[32231]: pgmap v4080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:54 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6392.772586 secs 2024-07-26T09:45:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:54 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6392.772586 secs 2024-07-26T09:45:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:54 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6392.772586 secs 2024-07-26T09:45:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:55 smithi115 ceph-mon[35364]: pgmap v4081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:55 smithi181 ceph-mon[35869]: pgmap v4081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:55 smithi097 ceph-mon[32231]: pgmap v4081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:57 smithi115 ceph-mon[35364]: pgmap v4082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:57 smithi181 ceph-mon[35869]: pgmap v4082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:57 smithi097 ceph-mon[32231]: pgmap v4082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:45:59 smithi115 ceph-mon[35364]: pgmap v4083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:45:59 smithi181 ceph-mon[35869]: pgmap v4083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:45:59 smithi097 ceph-mon[32231]: pgmap v4083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6397.772730 secs 2024-07-26T09:46:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6397.772730 secs 2024-07-26T09:46:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6397.772730 secs 2024-07-26T09:46:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:01 smithi115 ceph-mon[35364]: pgmap v4084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:01 smithi181 ceph-mon[35869]: pgmap v4084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:01 smithi097 ceph-mon[32231]: pgmap v4084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:03 smithi115 ceph-mon[35364]: pgmap v4085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:03 smithi181 ceph-mon[35869]: pgmap v4085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:03.998 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:03 smithi097 ceph-mon[32231]: pgmap v4085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:04 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6402.772884 secs 2024-07-26T09:46:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:04 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6402.772884 secs 2024-07-26T09:46:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:04 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6402.772884 secs 2024-07-26T09:46:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:05.512+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:46:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:05.513+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:46:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:05.513+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:46:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:05.513+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:46:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:05.513+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:05 smithi115 ceph-mon[35364]: pgmap v4086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:05 smithi181 ceph-mon[35869]: pgmap v4086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:05 smithi097 ceph-mon[32231]: pgmap v4086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:07.896 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:07 smithi097 ceph-mon[32231]: pgmap v4087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:07 smithi115 ceph-mon[35364]: pgmap v4087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:07 smithi181 ceph-mon[35869]: pgmap v4087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:08.009+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:46:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:08.009+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:46:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:08.009+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:46:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:08.010+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:46:08.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:08.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:08.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:08.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:08.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:08 smithi181 ceph-mon[35869]: pgmap v4088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:46:08.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:08 smithi181 ceph-mon[35869]: pgmap v4089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:08.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:08 smithi097 ceph-mon[32231]: pgmap v4088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:46:09.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:08 smithi097 ceph-mon[32231]: pgmap v4089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:09.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:09.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:08 smithi115 ceph-mon[35364]: pgmap v4088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:46:09.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:08 smithi115 ceph-mon[35364]: pgmap v4089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:09.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:46:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:10 smithi115 ceph-mon[35364]: pgmap v4090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:46:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6407.773027 secs 2024-07-26T09:46:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:10 smithi181 ceph-mon[35869]: pgmap v4090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:46:11.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6407.773027 secs 2024-07-26T09:46:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:10 smithi097 ceph-mon[32231]: pgmap v4090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:46:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6407.773027 secs 2024-07-26T09:46:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:12 smithi115 ceph-mon[35364]: pgmap v4091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:12 smithi181 ceph-mon[35869]: pgmap v4091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:12 smithi097 ceph-mon[32231]: pgmap v4091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:15.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:14 smithi097 ceph-mon[32231]: pgmap v4092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:46:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:14 smithi115 ceph-mon[35364]: pgmap v4092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:46:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:14 smithi181 ceph-mon[35869]: pgmap v4092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:46:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6412.773168 secs 2024-07-26T09:46:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6412.773168 secs 2024-07-26T09:46:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6412.773168 secs 2024-07-26T09:46:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:16 smithi115 ceph-mon[35364]: pgmap v4093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:46:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:16 smithi181 ceph-mon[35869]: pgmap v4093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:46:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:16 smithi097 ceph-mon[32231]: pgmap v4093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:46:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:18 smithi115 ceph-mon[35364]: pgmap v4094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:46:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:18 smithi181 ceph-mon[35869]: pgmap v4094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:46:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:18 smithi097 ceph-mon[32231]: pgmap v4094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:46:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:19 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6417.773322 secs 2024-07-26T09:46:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:19 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6417.773322 secs 2024-07-26T09:46:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:19 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6417.773322 secs 2024-07-26T09:46:20.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:20.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:46:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:20.524+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:46:20.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:20.524+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:46:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:20.524+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:46:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:20.524+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:20 smithi115 ceph-mon[35364]: pgmap v4095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:20 smithi181 ceph-mon[35869]: pgmap v4095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:20 smithi097 ceph-mon[32231]: pgmap v4095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:22 smithi115 ceph-mon[35364]: pgmap v4096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:22 smithi181 ceph-mon[35869]: pgmap v4096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:22 smithi097 ceph-mon[32231]: pgmap v4096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:25.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:24 smithi097 ceph-mon[32231]: pgmap v4097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:25.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:24 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6422.773479 secs 2024-07-26T09:46:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:24 smithi115 ceph-mon[35364]: pgmap v4097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:24 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6422.773479 secs 2024-07-26T09:46:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:24 smithi181 ceph-mon[35869]: pgmap v4097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:25.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:24 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6422.773479 secs 2024-07-26T09:46:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:26 smithi115 ceph-mon[35364]: pgmap v4098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:26 smithi181 ceph-mon[35869]: pgmap v4098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:26 smithi097 ceph-mon[32231]: pgmap v4098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:28 smithi115 ceph-mon[35364]: pgmap v4099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:28 smithi181 ceph-mon[35869]: pgmap v4099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:28 smithi097 ceph-mon[32231]: pgmap v4099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:29 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6427.773615 secs 2024-07-26T09:46:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:29 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6427.773615 secs 2024-07-26T09:46:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:29 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6427.773615 secs 2024-07-26T09:46:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:30 smithi115 ceph-mon[35364]: pgmap v4100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:30 smithi181 ceph-mon[35869]: pgmap v4100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:30 smithi097 ceph-mon[32231]: pgmap v4100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:32 smithi115 ceph-mon[35364]: pgmap v4101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:32 smithi181 ceph-mon[35869]: pgmap v4101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:32 smithi097 ceph-mon[32231]: pgmap v4101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:34 smithi115 ceph-mon[35364]: pgmap v4102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:34 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6432.773759 secs 2024-07-26T09:46:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:34 smithi181 ceph-mon[35869]: pgmap v4102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:34 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6432.773759 secs 2024-07-26T09:46:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:34 smithi097 ceph-mon[32231]: pgmap v4102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:34 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6432.773759 secs 2024-07-26T09:46:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:35.530+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:46:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:35.531+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:46:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:35.531+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:46:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:35.531+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:46:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:35.531+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:36 smithi115 ceph-mon[35364]: pgmap v4103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:36 smithi181 ceph-mon[35869]: pgmap v4103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:36 smithi097 ceph-mon[32231]: pgmap v4103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:38 smithi115 ceph-mon[35364]: pgmap v4104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:38 smithi181 ceph-mon[35869]: pgmap v4104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:38 smithi097 ceph-mon[32231]: pgmap v4104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:39 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6437.773912 secs 2024-07-26T09:46:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:39 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6437.773912 secs 2024-07-26T09:46:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:39 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6437.773912 secs 2024-07-26T09:46:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:40 smithi115 ceph-mon[35364]: pgmap v4105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:40 smithi181 ceph-mon[35869]: pgmap v4105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:40 smithi097 ceph-mon[32231]: pgmap v4105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:42 smithi181 ceph-mon[35869]: pgmap v4106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:42 smithi097 ceph-mon[32231]: pgmap v4106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:42 smithi115 ceph-mon[35364]: pgmap v4106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:45.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:44 smithi181 ceph-mon[35869]: pgmap v4107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:45.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:44 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6442.774059 secs 2024-07-26T09:46:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:44 smithi097 ceph-mon[32231]: pgmap v4107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:44 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6442.774059 secs 2024-07-26T09:46:45.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:44 smithi115 ceph-mon[35364]: pgmap v4107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:45.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:44 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6442.774059 secs 2024-07-26T09:46:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:46 smithi097 ceph-mon[32231]: pgmap v4108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:46 smithi115 ceph-mon[35364]: pgmap v4108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:46 smithi181 ceph-mon[35869]: pgmap v4108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:48 smithi097 ceph-mon[32231]: pgmap v4109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:48 smithi115 ceph-mon[35364]: pgmap v4109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:48 smithi181 ceph-mon[35869]: pgmap v4109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:49 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6447.774223 secs 2024-07-26T09:46:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:49 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6447.774223 secs 2024-07-26T09:46:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:49 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6447.774223 secs 2024-07-26T09:46:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:50.540+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:46:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:50.542+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:46:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:50.542+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:46:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:50.542+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:46:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:46:50.542+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:51 smithi115 ceph-mon[35364]: pgmap v4110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:51 smithi181 ceph-mon[35869]: pgmap v4110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:51 smithi097 ceph-mon[32231]: pgmap v4110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:53 smithi115 ceph-mon[35364]: pgmap v4111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:53 smithi181 ceph-mon[35869]: pgmap v4111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:53 smithi097 ceph-mon[32231]: pgmap v4111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:55 smithi115 ceph-mon[35364]: pgmap v4112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6452.774362 secs 2024-07-26T09:46:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:55 smithi181 ceph-mon[35869]: pgmap v4112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:55.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6452.774362 secs 2024-07-26T09:46:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:55 smithi097 ceph-mon[32231]: pgmap v4112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6452.774362 secs 2024-07-26T09:46:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:57 smithi115 ceph-mon[35364]: pgmap v4113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:57 smithi181 ceph-mon[35869]: pgmap v4113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:57 smithi097 ceph-mon[32231]: pgmap v4113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:46:59 smithi115 ceph-mon[35364]: pgmap v4114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:46:59 smithi181 ceph-mon[35869]: pgmap v4114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:46:59 smithi097 ceph-mon[32231]: pgmap v4114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6457.774504 secs 2024-07-26T09:47:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6457.774504 secs 2024-07-26T09:47:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6457.774504 secs 2024-07-26T09:47:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:01 smithi115 ceph-mon[35364]: pgmap v4115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:01 smithi181 ceph-mon[35869]: pgmap v4115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:01 smithi097 ceph-mon[32231]: pgmap v4115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:03 smithi115 ceph-mon[35364]: pgmap v4116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:03 smithi181 ceph-mon[35869]: pgmap v4116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:03 smithi097 ceph-mon[32231]: pgmap v4116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:05 smithi115 ceph-mon[35364]: pgmap v4117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:05.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6462.774673 secs 2024-07-26T09:47:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:05 smithi181 ceph-mon[35869]: pgmap v4117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:05.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6462.774673 secs 2024-07-26T09:47:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:05 smithi097 ceph-mon[32231]: pgmap v4117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6462.774673 secs 2024-07-26T09:47:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:05.548+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:05.549+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:05.549+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:05.549+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:05.549+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:07 smithi115 ceph-mon[35364]: pgmap v4118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:07 smithi181 ceph-mon[35869]: pgmap v4118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:07 smithi097 ceph-mon[32231]: pgmap v4118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:09 smithi115 ceph-mon[35364]: pgmap v4119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:09 smithi181 ceph-mon[35869]: pgmap v4119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:09 smithi097 ceph-mon[32231]: pgmap v4119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:10.201 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6467.774801 secs 2024-07-26T09:47:10.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6467.774801 secs 2024-07-26T09:47:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6467.774801 secs 2024-07-26T09:47:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:11 smithi115 ceph-mon[35364]: pgmap v4120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:11.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:11 smithi181 ceph-mon[35869]: pgmap v4120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:11 smithi097 ceph-mon[32231]: pgmap v4120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:13.398 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:13 smithi181 ceph-mon[35869]: pgmap v4121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:13 smithi115 ceph-mon[35364]: pgmap v4121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:13 smithi097 ceph-mon[32231]: pgmap v4121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: pgmap v4122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: pgmap v4123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: pgmap v4124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:14 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6472.774962 secs 2024-07-26T09:47:15.390 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:14 smithi181 ceph-mon[35869]: pgmap v4122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:15.390 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.390 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.390 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.390 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.390 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:15.390 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:15.390 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: pgmap v4123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:47:15.391 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: pgmap v4124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:15.391 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.391 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6472.774962 secs 2024-07-26T09:47:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: pgmap v4122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: pgmap v4123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: pgmap v4124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:47:15.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6472.774962 secs 2024-07-26T09:47:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:17 smithi115 ceph-mon[35364]: pgmap v4125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T09:47:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:17 smithi181 ceph-mon[35869]: pgmap v4125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T09:47:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:17 smithi097 ceph-mon[32231]: pgmap v4125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T09:47:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:19 smithi115 ceph-mon[35364]: pgmap v4126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:47:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:19 smithi181 ceph-mon[35869]: pgmap v4126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:47:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:19 smithi097 ceph-mon[32231]: pgmap v4126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:47:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:20.561+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:47:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:20.562+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:47:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:20.562+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:47:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:20.562+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:47:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:20.562+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6477.775104 secs 2024-07-26T09:47:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6477.775104 secs 2024-07-26T09:47:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6477.775104 secs 2024-07-26T09:47:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:21 smithi115 ceph-mon[35364]: pgmap v4127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T09:47:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:21 smithi181 ceph-mon[35869]: pgmap v4127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T09:47:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:21 smithi097 ceph-mon[32231]: pgmap v4127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T09:47:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:23 smithi115 ceph-mon[35364]: pgmap v4128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T09:47:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:23 smithi181 ceph-mon[35869]: pgmap v4128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T09:47:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:23 smithi097 ceph-mon[32231]: pgmap v4128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T09:47:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:25 smithi115 ceph-mon[35364]: pgmap v4129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:47:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6482.775310 secs 2024-07-26T09:47:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:25 smithi181 ceph-mon[35869]: pgmap v4129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:47:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6482.775310 secs 2024-07-26T09:47:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:25 smithi097 ceph-mon[32231]: pgmap v4129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:47:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6482.775310 secs 2024-07-26T09:47:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:27 smithi115 ceph-mon[35364]: pgmap v4130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:27 smithi181 ceph-mon[35869]: pgmap v4130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:27 smithi097 ceph-mon[32231]: pgmap v4130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:29.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:29 smithi115 ceph-mon[35364]: pgmap v4131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:29 smithi181 ceph-mon[35869]: pgmap v4131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:29 smithi097 ceph-mon[32231]: pgmap v4131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6487.775442 secs 2024-07-26T09:47:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6487.775442 secs 2024-07-26T09:47:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6487.775442 secs 2024-07-26T09:47:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:31 smithi115 ceph-mon[35364]: pgmap v4132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:31 smithi181 ceph-mon[35869]: pgmap v4132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:31 smithi097 ceph-mon[32231]: pgmap v4132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:33 smithi115 ceph-mon[35364]: pgmap v4133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:33 smithi181 ceph-mon[35869]: pgmap v4133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:33 smithi097 ceph-mon[32231]: pgmap v4133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:35.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:35.567+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:47:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:35.569+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:47:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:35.569+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:47:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:35.569+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:47:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:35.569+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:35 smithi115 ceph-mon[35364]: pgmap v4134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6492.775593 secs 2024-07-26T09:47:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:35 smithi181 ceph-mon[35869]: pgmap v4134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6492.775593 secs 2024-07-26T09:47:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:35 smithi097 ceph-mon[32231]: pgmap v4134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6492.775593 secs 2024-07-26T09:47:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:37 smithi115 ceph-mon[35364]: pgmap v4135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:37 smithi181 ceph-mon[35869]: pgmap v4135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:37 smithi097 ceph-mon[32231]: pgmap v4135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:39 smithi115 ceph-mon[35364]: pgmap v4136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:39 smithi181 ceph-mon[35869]: pgmap v4136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:39 smithi097 ceph-mon[32231]: pgmap v4136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6497.775756 secs 2024-07-26T09:47:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6497.775756 secs 2024-07-26T09:47:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6497.775756 secs 2024-07-26T09:47:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:41 smithi115 ceph-mon[35364]: pgmap v4137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:41 smithi181 ceph-mon[35869]: pgmap v4137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:41 smithi097 ceph-mon[32231]: pgmap v4137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:43 smithi115 ceph-mon[35364]: pgmap v4138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:43 smithi181 ceph-mon[35869]: pgmap v4138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:43 smithi097 ceph-mon[32231]: pgmap v4138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:45 smithi115 ceph-mon[35364]: pgmap v4139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:47:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6502.775889 secs 2024-07-26T09:47:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:45 smithi181 ceph-mon[35869]: pgmap v4139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:47:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6502.775889 secs 2024-07-26T09:47:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:45 smithi097 ceph-mon[32231]: pgmap v4139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:47:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6502.775889 secs 2024-07-26T09:47:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:47 smithi115 ceph-mon[35364]: pgmap v4140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:47 smithi181 ceph-mon[35869]: pgmap v4140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:47 smithi097 ceph-mon[32231]: pgmap v4140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:49 smithi115 ceph-mon[35364]: pgmap v4141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:49 smithi181 ceph-mon[35869]: pgmap v4141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:49 smithi097 ceph-mon[32231]: pgmap v4141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:50.578+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:47:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:50.579+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:47:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:50.579+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:47:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:50.579+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:47:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:47:50.579+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6507.776029 secs 2024-07-26T09:47:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6507.776029 secs 2024-07-26T09:47:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6507.776029 secs 2024-07-26T09:47:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:51 smithi115 ceph-mon[35364]: pgmap v4142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:51 smithi181 ceph-mon[35869]: pgmap v4142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:51 smithi097 ceph-mon[32231]: pgmap v4142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:53 smithi115 ceph-mon[35364]: pgmap v4143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:53 smithi181 ceph-mon[35869]: pgmap v4143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:53 smithi097 ceph-mon[32231]: pgmap v4143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:55 smithi115 ceph-mon[35364]: pgmap v4144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:47:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6512.776184 secs 2024-07-26T09:47:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:55 smithi181 ceph-mon[35869]: pgmap v4144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:47:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6512.776184 secs 2024-07-26T09:47:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:55 smithi097 ceph-mon[32231]: pgmap v4144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:47:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6512.776184 secs 2024-07-26T09:47:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:57 smithi115 ceph-mon[35364]: pgmap v4145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:57 smithi181 ceph-mon[35869]: pgmap v4145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:57 smithi097 ceph-mon[32231]: pgmap v4145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:47:59 smithi181 ceph-mon[35869]: pgmap v4146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:47:59 smithi097 ceph-mon[32231]: pgmap v4146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:47:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:47:59 smithi115 ceph-mon[35364]: pgmap v4146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:48:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6517.776331 secs 2024-07-26T09:48:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6517.776331 secs 2024-07-26T09:48:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6517.776331 secs 2024-07-26T09:48:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:01 smithi097 ceph-mon[32231]: pgmap v4147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:01 smithi115 ceph-mon[35364]: pgmap v4147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:01 smithi181 ceph-mon[35869]: pgmap v4147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:03 smithi115 ceph-mon[35364]: pgmap v4148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:03 smithi181 ceph-mon[35869]: pgmap v4148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:03 smithi097 ceph-mon[32231]: pgmap v4148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:05.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:05.583+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:48:05.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:05.585+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:48:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:05.585+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:48:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:05.585+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:48:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:05.585+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:05 smithi115 ceph-mon[35364]: pgmap v4149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:48:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6522.776478 secs 2024-07-26T09:48:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:05 smithi181 ceph-mon[35869]: pgmap v4149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:48:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6522.776478 secs 2024-07-26T09:48:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:05 smithi097 ceph-mon[32231]: pgmap v4149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:48:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6522.776478 secs 2024-07-26T09:48:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:07 smithi115 ceph-mon[35364]: pgmap v4150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:48:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:07 smithi181 ceph-mon[35869]: pgmap v4150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:48:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:07 smithi097 ceph-mon[32231]: pgmap v4150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:48:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:09 smithi115 ceph-mon[35364]: pgmap v4151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:09 smithi181 ceph-mon[35869]: pgmap v4151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:09 smithi097 ceph-mon[32231]: pgmap v4151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6527.776624 secs 2024-07-26T09:48:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6527.776624 secs 2024-07-26T09:48:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6527.776624 secs 2024-07-26T09:48:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:11 smithi115 ceph-mon[35364]: pgmap v4152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:11 smithi181 ceph-mon[35869]: pgmap v4152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:12.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:11 smithi097 ceph-mon[32231]: pgmap v4152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:13 smithi115 ceph-mon[35364]: pgmap v4153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:13 smithi181 ceph-mon[35869]: pgmap v4153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:13 smithi097 ceph-mon[32231]: pgmap v4153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:15.804 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:15 smithi097 ceph-mon[32231]: pgmap v4154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:48:15.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:15.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6532.776711 secs 2024-07-26T09:48:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:15 smithi115 ceph-mon[35364]: pgmap v4154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:48:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6532.776711 secs 2024-07-26T09:48:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:15 smithi181 ceph-mon[35869]: pgmap v4154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:48:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6532.776711 secs 2024-07-26T09:48:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:48:16.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:16.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:16.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:48:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:48:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:17 smithi115 ceph-mon[35364]: pgmap v4155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:48:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:17 smithi115 ceph-mon[35364]: pgmap v4156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:17.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:17 smithi181 ceph-mon[35869]: pgmap v4155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:48:17.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:17 smithi181 ceph-mon[35869]: pgmap v4156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:17 smithi097 ceph-mon[32231]: pgmap v4155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:48:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:17 smithi097 ceph-mon[32231]: pgmap v4156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:19 smithi115 ceph-mon[35364]: pgmap v4157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:48:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:19 smithi181 ceph-mon[35869]: pgmap v4157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:48:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:19 smithi097 ceph-mon[32231]: pgmap v4157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:48:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:20.595+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:48:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:20.596+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:48:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:20.596+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:48:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:20.596+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:48:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:20.596+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6537.776797 secs 2024-07-26T09:48:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6537.776797 secs 2024-07-26T09:48:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6537.776797 secs 2024-07-26T09:48:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:21 smithi115 ceph-mon[35364]: pgmap v4158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:48:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:21 smithi181 ceph-mon[35869]: pgmap v4158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:48:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:21 smithi097 ceph-mon[32231]: pgmap v4158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:48:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:23 smithi115 ceph-mon[35364]: pgmap v4159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:48:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:23 smithi181 ceph-mon[35869]: pgmap v4159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:48:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:23 smithi097 ceph-mon[32231]: pgmap v4159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:48:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:25 smithi115 ceph-mon[35364]: pgmap v4160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:48:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6542.776961 secs 2024-07-26T09:48:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:25 smithi181 ceph-mon[35869]: pgmap v4160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:48:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6542.776961 secs 2024-07-26T09:48:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:25 smithi097 ceph-mon[32231]: pgmap v4160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:48:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6542.776961 secs 2024-07-26T09:48:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:27 smithi115 ceph-mon[35364]: pgmap v4161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:48:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:27 smithi181 ceph-mon[35869]: pgmap v4161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:48:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:27 smithi097 ceph-mon[32231]: pgmap v4161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:48:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:29 smithi115 ceph-mon[35364]: pgmap v4162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:48:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:29 smithi181 ceph-mon[35869]: pgmap v4162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:48:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:29 smithi097 ceph-mon[32231]: pgmap v4162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:48:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6547.777134 secs 2024-07-26T09:48:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6547.777134 secs 2024-07-26T09:48:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6547.777134 secs 2024-07-26T09:48:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:31 smithi115 ceph-mon[35364]: pgmap v4163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:31 smithi181 ceph-mon[35869]: pgmap v4163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:31 smithi097 ceph-mon[32231]: pgmap v4163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:32 smithi115 ceph-mon[35364]: pgmap v4164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:32 smithi181 ceph-mon[35869]: pgmap v4164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:32 smithi097 ceph-mon[32231]: pgmap v4164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:34 smithi115 ceph-mon[35364]: pgmap v4165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:34 smithi181 ceph-mon[35869]: pgmap v4165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:34 smithi097 ceph-mon[32231]: pgmap v4165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:35.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:35.601+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:48:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:35.603+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:48:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:35.603+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:48:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:35.603+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:48:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:35.603+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6552.777288 secs 2024-07-26T09:48:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6552.777288 secs 2024-07-26T09:48:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6552.777288 secs 2024-07-26T09:48:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:36 smithi115 ceph-mon[35364]: pgmap v4166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:36 smithi181 ceph-mon[35869]: pgmap v4166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:36 smithi097 ceph-mon[32231]: pgmap v4166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:38 smithi115 ceph-mon[35364]: pgmap v4167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:38 smithi181 ceph-mon[35869]: pgmap v4167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:38 smithi097 ceph-mon[32231]: pgmap v4167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:40 smithi115 ceph-mon[35364]: pgmap v4168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6557.777431 secs 2024-07-26T09:48:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:40 smithi181 ceph-mon[35869]: pgmap v4168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6557.777431 secs 2024-07-26T09:48:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:40 smithi097 ceph-mon[32231]: pgmap v4168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6557.777431 secs 2024-07-26T09:48:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:42 smithi115 ceph-mon[35364]: pgmap v4169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:42 smithi181 ceph-mon[35869]: pgmap v4169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:42 smithi097 ceph-mon[32231]: pgmap v4169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:44 smithi115 ceph-mon[35364]: pgmap v4170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:44 smithi181 ceph-mon[35869]: pgmap v4170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:44 smithi097 ceph-mon[32231]: pgmap v4170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6562.777565 secs 2024-07-26T09:48:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6562.777565 secs 2024-07-26T09:48:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6562.777565 secs 2024-07-26T09:48:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:46 smithi115 ceph-mon[35364]: pgmap v4171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:46 smithi181 ceph-mon[35869]: pgmap v4171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:46 smithi097 ceph-mon[32231]: pgmap v4171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:48 smithi115 ceph-mon[35364]: pgmap v4172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:48 smithi181 ceph-mon[35869]: pgmap v4172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:48 smithi097 ceph-mon[32231]: pgmap v4172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:50.613+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:48:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:50.614+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:48:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:50.614+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:48:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:50.614+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:48:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:48:50.614+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:50 smithi115 ceph-mon[35364]: pgmap v4173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6567.777713 secs 2024-07-26T09:48:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:50 smithi181 ceph-mon[35869]: pgmap v4173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6567.777713 secs 2024-07-26T09:48:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:50 smithi097 ceph-mon[32231]: pgmap v4173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6567.777713 secs 2024-07-26T09:48:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:52 smithi115 ceph-mon[35364]: pgmap v4174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:52 smithi181 ceph-mon[35869]: pgmap v4174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:52 smithi097 ceph-mon[32231]: pgmap v4174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:54 smithi115 ceph-mon[35364]: pgmap v4175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:54 smithi181 ceph-mon[35869]: pgmap v4175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:54 smithi097 ceph-mon[32231]: pgmap v4175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6572.777866 secs 2024-07-26T09:48:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6572.777866 secs 2024-07-26T09:48:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6572.777866 secs 2024-07-26T09:48:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:56 smithi115 ceph-mon[35364]: pgmap v4176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:56 smithi181 ceph-mon[35869]: pgmap v4176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:56 smithi097 ceph-mon[32231]: pgmap v4176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:58 smithi115 ceph-mon[35364]: pgmap v4177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:58 smithi181 ceph-mon[35869]: pgmap v4177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:58 smithi097 ceph-mon[32231]: pgmap v4177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:48:59 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6577.777982 secs 2024-07-26T09:49:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:48:59 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6577.777982 secs 2024-07-26T09:49:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:48:59 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6577.777982 secs 2024-07-26T09:49:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:00 smithi115 ceph-mon[35364]: pgmap v4178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:00 smithi181 ceph-mon[35869]: pgmap v4178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:00 smithi097 ceph-mon[32231]: pgmap v4178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:02 smithi115 ceph-mon[35364]: pgmap v4179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:02 smithi181 ceph-mon[35869]: pgmap v4179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:02 smithi097 ceph-mon[32231]: pgmap v4179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:04 smithi115 ceph-mon[35364]: pgmap v4180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:04 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6582.778117 secs 2024-07-26T09:49:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:04 smithi181 ceph-mon[35869]: pgmap v4180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:04 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6582.778117 secs 2024-07-26T09:49:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:04 smithi097 ceph-mon[32231]: pgmap v4180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:04 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6582.778117 secs 2024-07-26T09:49:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:05.619+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:49:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:05.620+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:49:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:05.620+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:49:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:05.620+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:49:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:05.620+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:06 smithi115 ceph-mon[35364]: pgmap v4181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:06 smithi181 ceph-mon[35869]: pgmap v4181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:06 smithi097 ceph-mon[32231]: pgmap v4181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:08.042+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:49:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:08.042+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:49:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:08.043+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:49:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:08.043+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:49:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:08 smithi115 ceph-mon[35364]: pgmap v4182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:08 smithi181 ceph-mon[35869]: pgmap v4182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:08 smithi097 ceph-mon[32231]: pgmap v4182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:09 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6587.778269 secs 2024-07-26T09:49:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:09 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6587.778269 secs 2024-07-26T09:49:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:09 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6587.778269 secs 2024-07-26T09:49:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:10 smithi115 ceph-mon[35364]: pgmap v4183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:10 smithi181 ceph-mon[35869]: pgmap v4183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:10 smithi097 ceph-mon[32231]: pgmap v4183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:12 smithi115 ceph-mon[35364]: pgmap v4184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:12 smithi181 ceph-mon[35869]: pgmap v4184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:13.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:12 smithi097 ceph-mon[32231]: pgmap v4184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:14 smithi115 ceph-mon[35364]: pgmap v4185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:14 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6592.778413 secs 2024-07-26T09:49:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:14 smithi181 ceph-mon[35869]: pgmap v4185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:14 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6592.778413 secs 2024-07-26T09:49:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:14 smithi097 ceph-mon[32231]: pgmap v4185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:14 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6592.778413 secs 2024-07-26T09:49:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:16 smithi115 ceph-mon[35364]: pgmap v4186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:16 smithi181 ceph-mon[35869]: pgmap v4186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:16 smithi097 ceph-mon[32231]: pgmap v4186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:18.277 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:18.277 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:17 smithi097 ceph-mon[32231]: pgmap v4187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:49:18.277 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:17 smithi097 ceph-mon[32231]: pgmap v4188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:49:18.277 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:49:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:17 smithi115 ceph-mon[35364]: pgmap v4187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:49:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:17 smithi115 ceph-mon[35364]: pgmap v4188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:49:18.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:49:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:18.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:18.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:17 smithi181 ceph-mon[35869]: pgmap v4187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:49:18.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:17 smithi181 ceph-mon[35869]: pgmap v4188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:49:18.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:49:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:20 smithi115 ceph-mon[35364]: pgmap v4189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6597.778585 secs 2024-07-26T09:49:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:20 smithi181 ceph-mon[35869]: pgmap v4189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6597.778585 secs 2024-07-26T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:20 smithi097 ceph-mon[32231]: pgmap v4189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6597.778585 secs 2024-07-26T09:49:20.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:20.631+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:49:20.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:20.632+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:49:20.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:20.632+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:49:20.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:20.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:49:20.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:20.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:22 smithi115 ceph-mon[35364]: pgmap v4190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:22 smithi181 ceph-mon[35869]: pgmap v4190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:22 smithi097 ceph-mon[32231]: pgmap v4190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:24 smithi115 ceph-mon[35364]: pgmap v4191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:49:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:24 smithi181 ceph-mon[35869]: pgmap v4191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:49:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:24 smithi097 ceph-mon[32231]: pgmap v4191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:49:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6602.778724 secs 2024-07-26T09:49:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6602.778724 secs 2024-07-26T09:49:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6602.778724 secs 2024-07-26T09:49:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:26 smithi115 ceph-mon[35364]: pgmap v4192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:26 smithi181 ceph-mon[35869]: pgmap v4192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:26 smithi097 ceph-mon[32231]: pgmap v4192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:28 smithi181 ceph-mon[35869]: pgmap v4193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:28 smithi097 ceph-mon[32231]: pgmap v4193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:28 smithi115 ceph-mon[35364]: pgmap v4193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:30 smithi097 ceph-mon[32231]: pgmap v4194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:49:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6607.778880 secs 2024-07-26T09:49:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:30 smithi115 ceph-mon[35364]: pgmap v4194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:49:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6607.778880 secs 2024-07-26T09:49:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:30 smithi181 ceph-mon[35869]: pgmap v4194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:49:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6607.778880 secs 2024-07-26T09:49:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:32 smithi097 ceph-mon[32231]: pgmap v4195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:32 smithi115 ceph-mon[35364]: pgmap v4195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:32 smithi181 ceph-mon[35869]: pgmap v4195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:34 smithi097 ceph-mon[32231]: pgmap v4196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:49:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:34 smithi115 ceph-mon[35364]: pgmap v4196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:49:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:34 smithi181 ceph-mon[35869]: pgmap v4196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:49:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:35.637+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:49:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:35.639+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:49:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:35.639+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:49:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:35.639+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:49:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:35.639+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6612.778993 secs 2024-07-26T09:49:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6612.778993 secs 2024-07-26T09:49:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6612.778993 secs 2024-07-26T09:49:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:36 smithi115 ceph-mon[35364]: pgmap v4197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:49:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:36 smithi181 ceph-mon[35869]: pgmap v4197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:49:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:36 smithi097 ceph-mon[32231]: pgmap v4197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:49:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:38 smithi115 ceph-mon[35364]: pgmap v4198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:49:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:38 smithi181 ceph-mon[35869]: pgmap v4198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:49:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:38 smithi097 ceph-mon[32231]: pgmap v4198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:49:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:40 smithi115 ceph-mon[35364]: pgmap v4199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:49:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6617.779132 secs 2024-07-26T09:49:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:40 smithi181 ceph-mon[35869]: pgmap v4199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:49:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6617.779132 secs 2024-07-26T09:49:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:40 smithi097 ceph-mon[32231]: pgmap v4199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:49:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6617.779132 secs 2024-07-26T09:49:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:42 smithi115 ceph-mon[35364]: pgmap v4200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:42 smithi181 ceph-mon[35869]: pgmap v4200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:42 smithi097 ceph-mon[32231]: pgmap v4200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:44 smithi115 ceph-mon[35364]: pgmap v4201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:44 smithi181 ceph-mon[35869]: pgmap v4201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:44 smithi097 ceph-mon[32231]: pgmap v4201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:45.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6622.779332 secs 2024-07-26T09:49:45.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6622.779332 secs 2024-07-26T09:49:45.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6622.779332 secs 2024-07-26T09:49:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:46 smithi115 ceph-mon[35364]: pgmap v4202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:46 smithi181 ceph-mon[35869]: pgmap v4202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:46 smithi097 ceph-mon[32231]: pgmap v4202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:48 smithi115 ceph-mon[35364]: pgmap v4203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:49:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:48 smithi181 ceph-mon[35869]: pgmap v4203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:49:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:48 smithi097 ceph-mon[32231]: pgmap v4203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:49:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:50.649+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:49:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:50.650+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:49:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:50.650+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:49:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:50.650+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:49:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:49:50.651+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:50 smithi115 ceph-mon[35364]: pgmap v4204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6627.779477 secs 2024-07-26T09:49:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:50 smithi181 ceph-mon[35869]: pgmap v4204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6627.779477 secs 2024-07-26T09:49:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:50 smithi097 ceph-mon[32231]: pgmap v4204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6627.779477 secs 2024-07-26T09:49:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:52 smithi115 ceph-mon[35364]: pgmap v4205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:52 smithi181 ceph-mon[35869]: pgmap v4205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:52 smithi097 ceph-mon[32231]: pgmap v4205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:54 smithi115 ceph-mon[35364]: pgmap v4206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:54 smithi181 ceph-mon[35869]: pgmap v4206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:54.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:54 smithi097 ceph-mon[32231]: pgmap v4206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6632.779574 secs 2024-07-26T09:49:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6632.779574 secs 2024-07-26T09:49:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6632.779574 secs 2024-07-26T09:49:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:56 smithi115 ceph-mon[35364]: pgmap v4207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:56 smithi181 ceph-mon[35869]: pgmap v4207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:56 smithi097 ceph-mon[32231]: pgmap v4207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:49:58 smithi115 ceph-mon[35364]: pgmap v4208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:49:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:49:58 smithi181 ceph-mon[35869]: pgmap v4208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:49:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:49:58 smithi097 ceph-mon[32231]: pgmap v4208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:50:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:00 smithi115 ceph-mon[35364]: pgmap v4209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6637.779719 secs 2024-07-26T09:50:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:50:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:00 smithi181 ceph-mon[35869]: pgmap v4209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6637.779719 secs 2024-07-26T09:50:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:50:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:00 smithi097 ceph-mon[32231]: pgmap v4209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6637.779719 secs 2024-07-26T09:50:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:50:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:02 smithi115 ceph-mon[35364]: pgmap v4210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:02 smithi181 ceph-mon[35869]: pgmap v4210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:02 smithi097 ceph-mon[32231]: pgmap v4210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:04 smithi115 ceph-mon[35364]: pgmap v4211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:04 smithi181 ceph-mon[35869]: pgmap v4211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:04 smithi097 ceph-mon[32231]: pgmap v4211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:05.657+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:50:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:05.658+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:50:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:05.658+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:50:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:05.658+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:50:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:05.658+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6642.779866 secs 2024-07-26T09:50:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6642.779866 secs 2024-07-26T09:50:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6642.779866 secs 2024-07-26T09:50:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:06 smithi115 ceph-mon[35364]: pgmap v4212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:06 smithi181 ceph-mon[35869]: pgmap v4212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:06 smithi097 ceph-mon[32231]: pgmap v4212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:08 smithi115 ceph-mon[35364]: pgmap v4213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:08 smithi181 ceph-mon[35869]: pgmap v4213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:08 smithi097 ceph-mon[32231]: pgmap v4213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:10 smithi115 ceph-mon[35364]: pgmap v4214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6647.780024 secs 2024-07-26T09:50:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:10 smithi181 ceph-mon[35869]: pgmap v4214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6647.780024 secs 2024-07-26T09:50:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:10 smithi097 ceph-mon[32231]: pgmap v4214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6647.780024 secs 2024-07-26T09:50:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:12 smithi115 ceph-mon[35364]: pgmap v4215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:12 smithi181 ceph-mon[35869]: pgmap v4215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:12 smithi097 ceph-mon[32231]: pgmap v4215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:14.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:14 smithi115 ceph-mon[35364]: pgmap v4216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:14 smithi181 ceph-mon[35869]: pgmap v4216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:14 smithi097 ceph-mon[32231]: pgmap v4216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:15.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6652.780165 secs 2024-07-26T09:50:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6652.780165 secs 2024-07-26T09:50:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6652.780165 secs 2024-07-26T09:50:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:16 smithi115 ceph-mon[35364]: pgmap v4217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:16 smithi181 ceph-mon[35869]: pgmap v4217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:16 smithi097 ceph-mon[32231]: pgmap v4217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:18 smithi115 ceph-mon[35364]: pgmap v4218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:18 smithi181 ceph-mon[35869]: pgmap v4218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:18 smithi097 ceph-mon[32231]: pgmap v4218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:50:19.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:50:19.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:50:19.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:50:19.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:19.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:19.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: pgmap v4219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:50:19.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: pgmap v4220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:19.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:50:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: pgmap v4219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: pgmap v4220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:19.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T09:50:19.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T09:50:19.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T09:50:19.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T09:50:19.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:19.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:19.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: pgmap v4219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:50:19.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: pgmap v4220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:19.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:50:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:20.666+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:50:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:20.667+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:50:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:20.667+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:50:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:20.667+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:50:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:20.667+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6657.780341 secs 2024-07-26T09:50:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6657.780341 secs 2024-07-26T09:50:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6657.780341 secs 2024-07-26T09:50:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:21 smithi097 ceph-mon[32231]: pgmap v4221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:21 smithi115 ceph-mon[35364]: pgmap v4221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:21 smithi181 ceph-mon[35869]: pgmap v4221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:22 smithi115 ceph-mon[35364]: pgmap v4222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:22 smithi181 ceph-mon[35869]: pgmap v4222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:22 smithi097 ceph-mon[32231]: pgmap v4222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:24 smithi115 ceph-mon[35364]: pgmap v4223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:24 smithi181 ceph-mon[35869]: pgmap v4223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:24 smithi097 ceph-mon[32231]: pgmap v4223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6662.780503 secs 2024-07-26T09:50:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6662.780503 secs 2024-07-26T09:50:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6662.780503 secs 2024-07-26T09:50:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:26 smithi115 ceph-mon[35364]: pgmap v4224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:26 smithi181 ceph-mon[35869]: pgmap v4224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:26 smithi097 ceph-mon[32231]: pgmap v4224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:28 smithi115 ceph-mon[35364]: pgmap v4225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:50:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:28 smithi181 ceph-mon[35869]: pgmap v4225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:50:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:28 smithi097 ceph-mon[32231]: pgmap v4225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:50:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:29 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6667.780654 secs 2024-07-26T09:50:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:29 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6667.780654 secs 2024-07-26T09:50:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:29 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6667.780654 secs 2024-07-26T09:50:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:30 smithi115 ceph-mon[35364]: pgmap v4226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:30 smithi181 ceph-mon[35869]: pgmap v4226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:30 smithi097 ceph-mon[32231]: pgmap v4226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:32 smithi115 ceph-mon[35364]: pgmap v4227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:32 smithi181 ceph-mon[35869]: pgmap v4227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:32 smithi097 ceph-mon[32231]: pgmap v4227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:34 smithi115 ceph-mon[35364]: pgmap v4228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:34 smithi181 ceph-mon[35869]: pgmap v4228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:34 smithi097 ceph-mon[32231]: pgmap v4228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:35.674+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:50:35.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:35.674+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:50:35.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:35.674+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:50:35.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:35.674+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:50:35.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:35.674+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6672.780818 secs 2024-07-26T09:50:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6672.780818 secs 2024-07-26T09:50:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6672.780818 secs 2024-07-26T09:50:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:36 smithi115 ceph-mon[35364]: pgmap v4229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:36 smithi181 ceph-mon[35869]: pgmap v4229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:36 smithi097 ceph-mon[32231]: pgmap v4229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:38 smithi115 ceph-mon[35364]: pgmap v4230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:38 smithi181 ceph-mon[35869]: pgmap v4230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:38 smithi097 ceph-mon[32231]: pgmap v4230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:39 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6677.780971 secs 2024-07-26T09:50:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:39 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6677.780971 secs 2024-07-26T09:50:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:39 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6677.780971 secs 2024-07-26T09:50:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:40 smithi115 ceph-mon[35364]: pgmap v4231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:40 smithi181 ceph-mon[35869]: pgmap v4231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:40 smithi097 ceph-mon[32231]: pgmap v4231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:42 smithi115 ceph-mon[35364]: pgmap v4232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:42 smithi181 ceph-mon[35869]: pgmap v4232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:42 smithi097 ceph-mon[32231]: pgmap v4232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:44 smithi115 ceph-mon[35364]: pgmap v4233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:44 smithi181 ceph-mon[35869]: pgmap v4233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:44 smithi097 ceph-mon[32231]: pgmap v4233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6682.781127 secs 2024-07-26T09:50:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6682.781127 secs 2024-07-26T09:50:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6682.781127 secs 2024-07-26T09:50:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:46 smithi115 ceph-mon[35364]: pgmap v4234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:46 smithi181 ceph-mon[35869]: pgmap v4234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:46 smithi097 ceph-mon[32231]: pgmap v4234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:48 smithi115 ceph-mon[35364]: pgmap v4235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:48 smithi181 ceph-mon[35869]: pgmap v4235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:48 smithi097 ceph-mon[32231]: pgmap v4235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:49 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6687.781273 secs 2024-07-26T09:50:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:49 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6687.781273 secs 2024-07-26T09:50:50.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:49 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6687.781273 secs 2024-07-26T09:50:50.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:50.685+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:50:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:50.687+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:50:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:50.687+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:50:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:50.687+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:50:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:50:50.687+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:50 smithi115 ceph-mon[35364]: pgmap v4236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:50 smithi181 ceph-mon[35869]: pgmap v4236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:50 smithi097 ceph-mon[32231]: pgmap v4236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:53 smithi115 ceph-mon[35364]: pgmap v4237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:53 smithi181 ceph-mon[35869]: pgmap v4237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:53 smithi097 ceph-mon[32231]: pgmap v4237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:54 smithi115 ceph-mon[35364]: pgmap v4238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:54 smithi181 ceph-mon[35869]: pgmap v4238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:54 smithi097 ceph-mon[32231]: pgmap v4238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6692.781424 secs 2024-07-26T09:50:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6692.781424 secs 2024-07-26T09:50:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6692.781424 secs 2024-07-26T09:50:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:56 smithi115 ceph-mon[35364]: pgmap v4239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:56 smithi181 ceph-mon[35869]: pgmap v4239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:56 smithi097 ceph-mon[32231]: pgmap v4239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:58 smithi115 ceph-mon[35364]: pgmap v4240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:58 smithi181 ceph-mon[35869]: pgmap v4240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:58 smithi097 ceph-mon[32231]: pgmap v4240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:50:59 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6697.781574 secs 2024-07-26T09:51:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:50:59 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6697.781574 secs 2024-07-26T09:51:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:50:59 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6697.781574 secs 2024-07-26T09:51:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:00 smithi115 ceph-mon[35364]: pgmap v4241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:00 smithi181 ceph-mon[35869]: pgmap v4241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:00 smithi097 ceph-mon[32231]: pgmap v4241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:02 smithi115 ceph-mon[35364]: pgmap v4242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:02 smithi181 ceph-mon[35869]: pgmap v4242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:02 smithi097 ceph-mon[32231]: pgmap v4242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:04 smithi115 ceph-mon[35364]: pgmap v4243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:04 smithi181 ceph-mon[35869]: pgmap v4243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:04 smithi097 ceph-mon[32231]: pgmap v4243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:05.693+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:51:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:05.694+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:51:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:05.694+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:51:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:05.694+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:51:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:05.694+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:05.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6702.781727 secs 2024-07-26T09:51:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6702.781727 secs 2024-07-26T09:51:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6702.781727 secs 2024-07-26T09:51:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:06 smithi181 ceph-mon[35869]: pgmap v4244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:06 smithi097 ceph-mon[32231]: pgmap v4244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:06 smithi115 ceph-mon[35364]: pgmap v4244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:08 smithi115 ceph-mon[35364]: pgmap v4245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:08 smithi181 ceph-mon[35869]: pgmap v4245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:08 smithi097 ceph-mon[32231]: pgmap v4245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:09 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6707.781890 secs 2024-07-26T09:51:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:09 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6707.781890 secs 2024-07-26T09:51:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:09 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6707.781890 secs 2024-07-26T09:51:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:10 smithi115 ceph-mon[35364]: pgmap v4246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:10 smithi181 ceph-mon[35869]: pgmap v4246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:10 smithi097 ceph-mon[32231]: pgmap v4246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:12 smithi115 ceph-mon[35364]: pgmap v4247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:12 smithi181 ceph-mon[35869]: pgmap v4247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:12 smithi097 ceph-mon[32231]: pgmap v4247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:14 smithi115 ceph-mon[35364]: pgmap v4248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:14 smithi181 ceph-mon[35869]: pgmap v4248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:14 smithi097 ceph-mon[32231]: pgmap v4248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6712.782060 secs 2024-07-26T09:51:15.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6712.782060 secs 2024-07-26T09:51:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6712.782060 secs 2024-07-26T09:51:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:16 smithi181 ceph-mon[35869]: pgmap v4249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:16 smithi097 ceph-mon[32231]: pgmap v4249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:16 smithi115 ceph-mon[35364]: pgmap v4249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:18 smithi115 ceph-mon[35364]: pgmap v4250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:18 smithi181 ceph-mon[35869]: pgmap v4250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:18 smithi097 ceph-mon[32231]: pgmap v4250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:20.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:51:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:20.698+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:51:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:20.698+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:51:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:20.698+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:51:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:20.698+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6717.782167 secs 2024-07-26T09:51:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:20.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:51:20.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:51:20.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:20.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:20.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: pgmap v4251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:51:20.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: pgmap v4252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:20.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:51:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6717.782167 secs 2024-07-26T09:51:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: pgmap v4251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: pgmap v4252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:21.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:51:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6717.782167 secs 2024-07-26T09:51:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: pgmap v4251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: pgmap v4252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:21.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:51:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:23 smithi115 ceph-mon[35364]: pgmap v4253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:23 smithi181 ceph-mon[35869]: pgmap v4253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:23 smithi097 ceph-mon[32231]: pgmap v4253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:25.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:25 smithi115 ceph-mon[35364]: pgmap v4254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:25.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6722.782310 secs 2024-07-26T09:51:25.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:25 smithi181 ceph-mon[35869]: pgmap v4254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:25.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6722.782310 secs 2024-07-26T09:51:25.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:25 smithi097 ceph-mon[32231]: pgmap v4254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:25.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6722.782310 secs 2024-07-26T09:51:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:27 smithi115 ceph-mon[35364]: pgmap v4255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:27 smithi181 ceph-mon[35869]: pgmap v4255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:27 smithi097 ceph-mon[32231]: pgmap v4255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:29 smithi115 ceph-mon[35364]: pgmap v4256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:29 smithi181 ceph-mon[35869]: pgmap v4256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:29 smithi097 ceph-mon[32231]: pgmap v4256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6727.782453 secs 2024-07-26T09:51:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6727.782453 secs 2024-07-26T09:51:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6727.782453 secs 2024-07-26T09:51:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:31 smithi115 ceph-mon[35364]: pgmap v4257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:51:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:31 smithi181 ceph-mon[35869]: pgmap v4257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:51:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:31 smithi097 ceph-mon[32231]: pgmap v4257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:51:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:33 smithi115 ceph-mon[35364]: pgmap v4258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:33 smithi181 ceph-mon[35869]: pgmap v4258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:33 smithi097 ceph-mon[32231]: pgmap v4258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:35 smithi115 ceph-mon[35364]: pgmap v4259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6732.782596 secs 2024-07-26T09:51:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:35 smithi181 ceph-mon[35869]: pgmap v4259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6732.782596 secs 2024-07-26T09:51:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:35 smithi097 ceph-mon[32231]: pgmap v4259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6732.782596 secs 2024-07-26T09:51:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:35.703+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:51:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:35.704+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:51:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:35.704+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:51:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:35.704+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:51:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:35.704+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:37 smithi115 ceph-mon[35364]: pgmap v4260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:37 smithi181 ceph-mon[35869]: pgmap v4260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:37 smithi097 ceph-mon[32231]: pgmap v4260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:39 smithi115 ceph-mon[35364]: pgmap v4261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:39 smithi181 ceph-mon[35869]: pgmap v4261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:39 smithi097 ceph-mon[32231]: pgmap v4261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6737.782731 secs 2024-07-26T09:51:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6737.782731 secs 2024-07-26T09:51:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6737.782731 secs 2024-07-26T09:51:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:41 smithi115 ceph-mon[35364]: pgmap v4262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:41 smithi181 ceph-mon[35869]: pgmap v4262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:41 smithi097 ceph-mon[32231]: pgmap v4262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:43 smithi115 ceph-mon[35364]: pgmap v4263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:43 smithi181 ceph-mon[35869]: pgmap v4263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:43 smithi097 ceph-mon[32231]: pgmap v4263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:45 smithi181 ceph-mon[35869]: pgmap v4264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:45.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6742.782818 secs 2024-07-26T09:51:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:45 smithi097 ceph-mon[32231]: pgmap v4264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6742.782818 secs 2024-07-26T09:51:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:45 smithi115 ceph-mon[35364]: pgmap v4264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6742.782818 secs 2024-07-26T09:51:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:47 smithi097 ceph-mon[32231]: pgmap v4265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:47 smithi115 ceph-mon[35364]: pgmap v4265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:47.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:47 smithi181 ceph-mon[35869]: pgmap v4265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:49 smithi097 ceph-mon[32231]: pgmap v4266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:49 smithi115 ceph-mon[35364]: pgmap v4266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:49 smithi181 ceph-mon[35869]: pgmap v4266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:50.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:50.710+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:51:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:50.711+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:51:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:50.712+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:51:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:50.712+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:51:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:51:50.712+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6747.782919 secs 2024-07-26T09:51:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6747.782919 secs 2024-07-26T09:51:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6747.782919 secs 2024-07-26T09:51:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:51 smithi115 ceph-mon[35364]: pgmap v4267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:51 smithi181 ceph-mon[35869]: pgmap v4267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:51 smithi097 ceph-mon[32231]: pgmap v4267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:53 smithi115 ceph-mon[35364]: pgmap v4268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:53 smithi181 ceph-mon[35869]: pgmap v4268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:53 smithi097 ceph-mon[32231]: pgmap v4268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:55 smithi115 ceph-mon[35364]: pgmap v4269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6752.783032 secs 2024-07-26T09:51:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:55 smithi181 ceph-mon[35869]: pgmap v4269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6752.783032 secs 2024-07-26T09:51:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:55 smithi097 ceph-mon[32231]: pgmap v4269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:51:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6752.783032 secs 2024-07-26T09:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:57 smithi115 ceph-mon[35364]: pgmap v4270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:57 smithi181 ceph-mon[35869]: pgmap v4270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:57 smithi097 ceph-mon[32231]: pgmap v4270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:51:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:51:59 smithi115 ceph-mon[35364]: pgmap v4271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:51:59 smithi181 ceph-mon[35869]: pgmap v4271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:51:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:51:59 smithi097 ceph-mon[32231]: pgmap v4271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6757.783172 secs 2024-07-26T09:52:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6757.783172 secs 2024-07-26T09:52:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6757.783172 secs 2024-07-26T09:52:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:01 smithi115 ceph-mon[35364]: pgmap v4272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:01 smithi181 ceph-mon[35869]: pgmap v4272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:01 smithi097 ceph-mon[32231]: pgmap v4272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:03 smithi115 ceph-mon[35364]: pgmap v4273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:03 smithi181 ceph-mon[35869]: pgmap v4273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:03 smithi097 ceph-mon[32231]: pgmap v4273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:05.718+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:52:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:05.719+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:52:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:05.719+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:52:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:05.719+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:52:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:05.719+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:05 smithi115 ceph-mon[35364]: pgmap v4274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6762.783363 secs 2024-07-26T09:52:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:05 smithi181 ceph-mon[35869]: pgmap v4274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6762.783363 secs 2024-07-26T09:52:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:05 smithi097 ceph-mon[32231]: pgmap v4274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6762.783363 secs 2024-07-26T09:52:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:07 smithi115 ceph-mon[35364]: pgmap v4275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:07 smithi181 ceph-mon[35869]: pgmap v4275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:07 smithi097 ceph-mon[32231]: pgmap v4275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:08.074+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:52:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:08.074+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:52:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:08.074+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:52:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:08.075+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:52:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:09 smithi115 ceph-mon[35364]: pgmap v4276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:09 smithi181 ceph-mon[35869]: pgmap v4276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:09 smithi097 ceph-mon[32231]: pgmap v4276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6767.783508 secs 2024-07-26T09:52:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6767.783508 secs 2024-07-26T09:52:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6767.783508 secs 2024-07-26T09:52:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:11 smithi115 ceph-mon[35364]: pgmap v4277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:11 smithi181 ceph-mon[35869]: pgmap v4277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:11 smithi097 ceph-mon[32231]: pgmap v4277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:13 smithi115 ceph-mon[35364]: pgmap v4278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:13 smithi181 ceph-mon[35869]: pgmap v4278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:13 smithi097 ceph-mon[32231]: pgmap v4278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:15 smithi115 ceph-mon[35364]: pgmap v4279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6772.783650 secs 2024-07-26T09:52:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:15 smithi181 ceph-mon[35869]: pgmap v4279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6772.783650 secs 2024-07-26T09:52:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:15 smithi097 ceph-mon[32231]: pgmap v4279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6772.783650 secs 2024-07-26T09:52:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:17 smithi115 ceph-mon[35364]: pgmap v4280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:17 smithi181 ceph-mon[35869]: pgmap v4280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:17 smithi097 ceph-mon[32231]: pgmap v4280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:19 smithi115 ceph-mon[35364]: pgmap v4281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:19 smithi181 ceph-mon[35869]: pgmap v4281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:19 smithi097 ceph-mon[32231]: pgmap v4281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6777.783789 secs 2024-07-26T09:52:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:20.728+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:52:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:20.729+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:52:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:20.729+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:52:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:20.729+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:52:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:20.729+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6777.783789 secs 2024-07-26T09:52:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6777.783789 secs 2024-07-26T09:52:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:21 smithi115 ceph-mon[35364]: pgmap v4282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:21 smithi181 ceph-mon[35869]: pgmap v4282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:21 smithi097 ceph-mon[32231]: pgmap v4282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:22.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:52:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:23 smithi115 ceph-mon[35364]: pgmap v4283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:23 smithi115 ceph-mon[35364]: pgmap v4284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:23 smithi181 ceph-mon[35869]: pgmap v4283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:23 smithi181 ceph-mon[35869]: pgmap v4284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:23 smithi097 ceph-mon[32231]: pgmap v4283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:23 smithi097 ceph-mon[32231]: pgmap v4284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:25 smithi115 ceph-mon[35364]: pgmap v4285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6782.783932 secs 2024-07-26T09:52:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:25 smithi181 ceph-mon[35869]: pgmap v4285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6782.783932 secs 2024-07-26T09:52:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:25 smithi097 ceph-mon[32231]: pgmap v4285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6782.783932 secs 2024-07-26T09:52:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:27 smithi115 ceph-mon[35364]: pgmap v4286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:27 smithi181 ceph-mon[35869]: pgmap v4286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:27 smithi097 ceph-mon[32231]: pgmap v4286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:29 smithi115 ceph-mon[35364]: pgmap v4287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:29 smithi181 ceph-mon[35869]: pgmap v4287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:29 smithi097 ceph-mon[32231]: pgmap v4287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6787.784070 secs 2024-07-26T09:52:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6787.784070 secs 2024-07-26T09:52:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6787.784070 secs 2024-07-26T09:52:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:31 smithi115 ceph-mon[35364]: pgmap v4288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:31 smithi181 ceph-mon[35869]: pgmap v4288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:31 smithi097 ceph-mon[32231]: pgmap v4288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:52:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:33 smithi115 ceph-mon[35364]: pgmap v4289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:52:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:33 smithi181 ceph-mon[35869]: pgmap v4289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:52:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:33 smithi097 ceph-mon[32231]: pgmap v4289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:52:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:35 smithi115 ceph-mon[35364]: pgmap v4290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6792.784268 secs 2024-07-26T09:52:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:35 smithi181 ceph-mon[35869]: pgmap v4290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6792.784268 secs 2024-07-26T09:52:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:35.735+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:35.736+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:35.736+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:35.736+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:35.736+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:35 smithi097 ceph-mon[32231]: pgmap v4290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6792.784268 secs 2024-07-26T09:52:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:37 smithi115 ceph-mon[35364]: pgmap v4291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:37 smithi181 ceph-mon[35869]: pgmap v4291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:37 smithi097 ceph-mon[32231]: pgmap v4291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:39 smithi115 ceph-mon[35364]: pgmap v4292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:39 smithi181 ceph-mon[35869]: pgmap v4292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:39 smithi097 ceph-mon[32231]: pgmap v4292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6797.784359 secs 2024-07-26T09:52:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6797.784359 secs 2024-07-26T09:52:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6797.784359 secs 2024-07-26T09:52:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:41 smithi097 ceph-mon[32231]: pgmap v4293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:41 smithi115 ceph-mon[35364]: pgmap v4293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:41 smithi181 ceph-mon[35869]: pgmap v4293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:43.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:43 smithi097 ceph-mon[32231]: pgmap v4294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:43 smithi115 ceph-mon[35364]: pgmap v4294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:43 smithi181 ceph-mon[35869]: pgmap v4294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:45 smithi097 ceph-mon[32231]: pgmap v4295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6802.784501 secs 2024-07-26T09:52:45.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:45 smithi115 ceph-mon[35364]: pgmap v4295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:45.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6802.784501 secs 2024-07-26T09:52:45.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:45 smithi181 ceph-mon[35869]: pgmap v4295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:52:45.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6802.784501 secs 2024-07-26T09:52:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:47 smithi115 ceph-mon[35364]: pgmap v4296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:47.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:47 smithi181 ceph-mon[35869]: pgmap v4296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:47 smithi097 ceph-mon[32231]: pgmap v4296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:49 smithi115 ceph-mon[35364]: pgmap v4297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:52:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:49 smithi181 ceph-mon[35869]: pgmap v4297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:52:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:49 smithi097 ceph-mon[32231]: pgmap v4297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:52:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:50.747+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:52:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:50.748+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:52:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:50.748+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:52:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:50.748+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:52:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:52:50.748+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6807.784647 secs 2024-07-26T09:52:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6807.784647 secs 2024-07-26T09:52:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6807.784647 secs 2024-07-26T09:52:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:51 smithi115 ceph-mon[35364]: pgmap v4298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:52:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:51 smithi181 ceph-mon[35869]: pgmap v4298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:52:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:51 smithi097 ceph-mon[32231]: pgmap v4298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:52:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:53 smithi115 ceph-mon[35364]: pgmap v4299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:53 smithi181 ceph-mon[35869]: pgmap v4299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:53 smithi097 ceph-mon[32231]: pgmap v4299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:55 smithi115 ceph-mon[35364]: pgmap v4300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:52:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6812.784789 secs 2024-07-26T09:52:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:55 smithi181 ceph-mon[35869]: pgmap v4300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:52:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6812.784789 secs 2024-07-26T09:52:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:55 smithi097 ceph-mon[32231]: pgmap v4300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:52:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6812.784789 secs 2024-07-26T09:52:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:57 smithi115 ceph-mon[35364]: pgmap v4301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:57 smithi181 ceph-mon[35869]: pgmap v4301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:57 smithi097 ceph-mon[32231]: pgmap v4301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:52:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:52:59 smithi115 ceph-mon[35364]: pgmap v4302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:52:59 smithi181 ceph-mon[35869]: pgmap v4302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:52:59 smithi097 ceph-mon[32231]: pgmap v4302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6817.784881 secs 2024-07-26T09:53:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6817.784881 secs 2024-07-26T09:53:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6817.784881 secs 2024-07-26T09:53:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:01 smithi115 ceph-mon[35364]: pgmap v4303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:01 smithi181 ceph-mon[35869]: pgmap v4303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:01 smithi097 ceph-mon[32231]: pgmap v4303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:03 smithi115 ceph-mon[35364]: pgmap v4304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:03.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:03 smithi181 ceph-mon[35869]: pgmap v4304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:03 smithi097 ceph-mon[32231]: pgmap v4304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:05.754+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:53:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:05.755+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:53:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:05.755+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:53:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:05.755+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:53:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:05.755+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:05 smithi115 ceph-mon[35364]: pgmap v4305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:53:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6822.785041 secs 2024-07-26T09:53:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:05 smithi181 ceph-mon[35869]: pgmap v4305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:53:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6822.785041 secs 2024-07-26T09:53:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:05 smithi097 ceph-mon[32231]: pgmap v4305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:53:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6822.785041 secs 2024-07-26T09:53:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:07 smithi115 ceph-mon[35364]: pgmap v4306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:07 smithi181 ceph-mon[35869]: pgmap v4306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:07 smithi097 ceph-mon[32231]: pgmap v4306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:09 smithi115 ceph-mon[35364]: pgmap v4307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:09 smithi181 ceph-mon[35869]: pgmap v4307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:09 smithi097 ceph-mon[32231]: pgmap v4307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6827.785197 secs 2024-07-26T09:53:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6827.785197 secs 2024-07-26T09:53:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6827.785197 secs 2024-07-26T09:53:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:11 smithi115 ceph-mon[35364]: pgmap v4308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:11 smithi181 ceph-mon[35869]: pgmap v4308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:11 smithi097 ceph-mon[32231]: pgmap v4308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:13 smithi115 ceph-mon[35364]: pgmap v4309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:13 smithi181 ceph-mon[35869]: pgmap v4309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:13 smithi097 ceph-mon[32231]: pgmap v4309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:15 smithi115 ceph-mon[35364]: pgmap v4310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:53:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6832.785344 secs 2024-07-26T09:53:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:15 smithi181 ceph-mon[35869]: pgmap v4310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:53:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6832.785344 secs 2024-07-26T09:53:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:15 smithi097 ceph-mon[32231]: pgmap v4310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:53:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6832.785344 secs 2024-07-26T09:53:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:17 smithi115 ceph-mon[35364]: pgmap v4311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:17 smithi181 ceph-mon[35869]: pgmap v4311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:17 smithi097 ceph-mon[32231]: pgmap v4311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:19 smithi115 ceph-mon[35364]: pgmap v4312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:53:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:19 smithi181 ceph-mon[35869]: pgmap v4312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:53:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:19 smithi097 ceph-mon[32231]: pgmap v4312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:53:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:20.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:53:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:20.767+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:53:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:20.767+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:53:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:20.767+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:53:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:20.767+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6837.785440 secs 2024-07-26T09:53:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6837.785440 secs 2024-07-26T09:53:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6837.785440 secs 2024-07-26T09:53:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:21 smithi115 ceph-mon[35364]: pgmap v4313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:21 smithi181 ceph-mon[35869]: pgmap v4313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:21 smithi097 ceph-mon[32231]: pgmap v4313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:23 smithi115 ceph-mon[35364]: pgmap v4314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:53:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:23 smithi181 ceph-mon[35869]: pgmap v4314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:53:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:23.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:23.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:23.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:23 smithi097 ceph-mon[32231]: pgmap v4314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:53:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:24.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:24.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:53:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:25 smithi115 ceph-mon[35364]: pgmap v4315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:25 smithi115 ceph-mon[35364]: pgmap v4316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6842.785590 secs 2024-07-26T09:53:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:25 smithi181 ceph-mon[35869]: pgmap v4315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:25 smithi181 ceph-mon[35869]: pgmap v4316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:25.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6842.785590 secs 2024-07-26T09:53:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:25 smithi097 ceph-mon[32231]: pgmap v4315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:25 smithi097 ceph-mon[32231]: pgmap v4316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6842.785590 secs 2024-07-26T09:53:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:26 smithi115 ceph-mon[35364]: pgmap v4317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:26.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:26 smithi181 ceph-mon[35869]: pgmap v4317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:26 smithi097 ceph-mon[32231]: pgmap v4317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:29 smithi115 ceph-mon[35364]: pgmap v4318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:29 smithi181 ceph-mon[35869]: pgmap v4318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:29 smithi097 ceph-mon[32231]: pgmap v4318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6847.785732 secs 2024-07-26T09:53:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6847.785732 secs 2024-07-26T09:53:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6847.785732 secs 2024-07-26T09:53:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:31 smithi115 ceph-mon[35364]: pgmap v4319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:31 smithi181 ceph-mon[35869]: pgmap v4319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:31 smithi097 ceph-mon[32231]: pgmap v4319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:33 smithi115 ceph-mon[35364]: pgmap v4320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:33 smithi181 ceph-mon[35869]: pgmap v4320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:33 smithi097 ceph-mon[32231]: pgmap v4320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:53:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:35 smithi115 ceph-mon[35364]: pgmap v4321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:53:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6852.785882 secs 2024-07-26T09:53:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:35 smithi181 ceph-mon[35869]: pgmap v4321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:53:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6852.785882 secs 2024-07-26T09:53:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:35 smithi097 ceph-mon[32231]: pgmap v4321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:53:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6852.785882 secs 2024-07-26T09:53:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:35.772+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:53:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:35.773+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:53:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:35.773+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:53:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:35.773+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:53:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:35.773+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:37 smithi115 ceph-mon[35364]: pgmap v4322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:37 smithi181 ceph-mon[35869]: pgmap v4322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:37 smithi097 ceph-mon[32231]: pgmap v4322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:38.090+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:53:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:38.090+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:53:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:38.091+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:53:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:38.091+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:53:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:39 smithi115 ceph-mon[35364]: pgmap v4323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:39 smithi181 ceph-mon[35869]: pgmap v4323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:39 smithi097 ceph-mon[32231]: pgmap v4323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6857.786048 secs 2024-07-26T09:53:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6857.786048 secs 2024-07-26T09:53:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6857.786048 secs 2024-07-26T09:53:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:41 smithi181 ceph-mon[35869]: pgmap v4324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:41 smithi097 ceph-mon[32231]: pgmap v4324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:41 smithi115 ceph-mon[35364]: pgmap v4324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:43 smithi097 ceph-mon[32231]: pgmap v4325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:43 smithi115 ceph-mon[35364]: pgmap v4325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:43 smithi181 ceph-mon[35869]: pgmap v4325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:45 smithi097 ceph-mon[32231]: pgmap v4326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6862.786158 secs 2024-07-26T09:53:45.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:45 smithi115 ceph-mon[35364]: pgmap v4326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:45.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6862.786158 secs 2024-07-26T09:53:45.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:45 smithi181 ceph-mon[35869]: pgmap v4326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:45.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6862.786158 secs 2024-07-26T09:53:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:47 smithi097 ceph-mon[32231]: pgmap v4327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:47 smithi115 ceph-mon[35364]: pgmap v4327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:47 smithi181 ceph-mon[35869]: pgmap v4327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:49 smithi115 ceph-mon[35364]: pgmap v4328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:53:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:49 smithi181 ceph-mon[35869]: pgmap v4328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:53:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:49 smithi097 ceph-mon[32231]: pgmap v4328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:53:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6867.786288 secs 2024-07-26T09:53:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6867.786288 secs 2024-07-26T09:53:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6867.786288 secs 2024-07-26T09:53:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:50.784+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:53:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:50.785+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:53:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:50.785+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:53:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:50.785+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:53:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:53:50.785+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:51 smithi115 ceph-mon[35364]: pgmap v4329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:51 smithi181 ceph-mon[35869]: pgmap v4329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:51 smithi097 ceph-mon[32231]: pgmap v4329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:53 smithi115 ceph-mon[35364]: pgmap v4330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:53:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:53 smithi181 ceph-mon[35869]: pgmap v4330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:53:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:53 smithi097 ceph-mon[32231]: pgmap v4330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:53:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:55 smithi115 ceph-mon[35364]: pgmap v4331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6872.786433 secs 2024-07-26T09:53:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:55 smithi181 ceph-mon[35869]: pgmap v4331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6872.786433 secs 2024-07-26T09:53:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:55 smithi097 ceph-mon[32231]: pgmap v4331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:53:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6872.786433 secs 2024-07-26T09:53:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:57 smithi115 ceph-mon[35364]: pgmap v4332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:57 smithi181 ceph-mon[35869]: pgmap v4332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:57 smithi097 ceph-mon[32231]: pgmap v4332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:53:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:53:59 smithi115 ceph-mon[35364]: pgmap v4333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:53:59 smithi181 ceph-mon[35869]: pgmap v4333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:53:59 smithi097 ceph-mon[32231]: pgmap v4333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6877.786578 secs 2024-07-26T09:54:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6877.786578 secs 2024-07-26T09:54:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6877.786578 secs 2024-07-26T09:54:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:01 smithi115 ceph-mon[35364]: pgmap v4334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:01 smithi181 ceph-mon[35869]: pgmap v4334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:01 smithi097 ceph-mon[32231]: pgmap v4334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:03 smithi115 ceph-mon[35364]: pgmap v4335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:03 smithi181 ceph-mon[35869]: pgmap v4335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:03 smithi097 ceph-mon[32231]: pgmap v4335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:05 smithi115 ceph-mon[35364]: pgmap v4336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6882.786742 secs 2024-07-26T09:54:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:05 smithi181 ceph-mon[35869]: pgmap v4336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:05.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6882.786742 secs 2024-07-26T09:54:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:05 smithi097 ceph-mon[32231]: pgmap v4336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6882.786742 secs 2024-07-26T09:54:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:05.791+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:05.792+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:05.792+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:05.792+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:05.792+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:07 smithi115 ceph-mon[35364]: pgmap v4337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:07 smithi181 ceph-mon[35869]: pgmap v4337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:07 smithi097 ceph-mon[32231]: pgmap v4337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:09 smithi115 ceph-mon[35364]: pgmap v4338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:09 smithi181 ceph-mon[35869]: pgmap v4338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:09 smithi097 ceph-mon[32231]: pgmap v4338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6887.786899 secs 2024-07-26T09:54:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6887.786899 secs 2024-07-26T09:54:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6887.786899 secs 2024-07-26T09:54:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:11 smithi115 ceph-mon[35364]: pgmap v4339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:11 smithi181 ceph-mon[35869]: pgmap v4339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:11 smithi097 ceph-mon[32231]: pgmap v4339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:13 smithi115 ceph-mon[35364]: pgmap v4340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:13 smithi181 ceph-mon[35869]: pgmap v4340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:13 smithi097 ceph-mon[32231]: pgmap v4340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:15 smithi115 ceph-mon[35364]: pgmap v4341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6892.787035 secs 2024-07-26T09:54:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:15 smithi181 ceph-mon[35869]: pgmap v4341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6892.787035 secs 2024-07-26T09:54:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:15 smithi097 ceph-mon[32231]: pgmap v4341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6892.787035 secs 2024-07-26T09:54:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:17 smithi115 ceph-mon[35364]: pgmap v4342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:17 smithi181 ceph-mon[35869]: pgmap v4342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:17 smithi097 ceph-mon[32231]: pgmap v4342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:19 smithi115 ceph-mon[35364]: pgmap v4343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:19 smithi181 ceph-mon[35869]: pgmap v4343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:19 smithi097 ceph-mon[32231]: pgmap v4343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6897.787181 secs 2024-07-26T09:54:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6897.787181 secs 2024-07-26T09:54:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6897.787181 secs 2024-07-26T09:54:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:20.804+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:54:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:20.806+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:54:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:20.806+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:54:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:20.806+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:54:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:20.806+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:21 smithi115 ceph-mon[35364]: pgmap v4344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:21 smithi181 ceph-mon[35869]: pgmap v4344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:21 smithi097 ceph-mon[32231]: pgmap v4344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:23 smithi115 ceph-mon[35364]: pgmap v4345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:23 smithi181 ceph-mon[35869]: pgmap v4345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:23 smithi097 ceph-mon[32231]: pgmap v4345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:25.721 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:25 smithi115 ceph-mon[35364]: pgmap v4346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:25.721 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:25.721 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6902.787302 secs 2024-07-26T09:54:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:25 smithi181 ceph-mon[35869]: pgmap v4346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6902.787302 secs 2024-07-26T09:54:25.747 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:25 smithi097 ceph-mon[32231]: pgmap v4346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:54:25.747 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:25.747 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6902.787302 secs 2024-07-26T09:54:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:27 smithi115 ceph-mon[35364]: pgmap v4347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:27 smithi181 ceph-mon[35869]: pgmap v4347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:27 smithi097 ceph-mon[32231]: pgmap v4347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:29 smithi115 ceph-mon[35364]: pgmap v4348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:29 smithi181 ceph-mon[35869]: pgmap v4348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:29.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:29.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:29 smithi097 ceph-mon[32231]: pgmap v4348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:54:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:30 smithi115 ceph-mon[35364]: pgmap v4349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:30 smithi115 ceph-mon[35364]: pgmap v4350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6907.787443 secs 2024-07-26T09:54:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:30 smithi181 ceph-mon[35869]: pgmap v4349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:30 smithi181 ceph-mon[35869]: pgmap v4350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6907.787443 secs 2024-07-26T09:54:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:30 smithi097 ceph-mon[32231]: pgmap v4349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:30 smithi097 ceph-mon[32231]: pgmap v4350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6907.787443 secs 2024-07-26T09:54:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:32 smithi115 ceph-mon[35364]: pgmap v4351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T09:54:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:32 smithi181 ceph-mon[35869]: pgmap v4351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T09:54:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:32 smithi097 ceph-mon[32231]: pgmap v4351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T09:54:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:34 smithi115 ceph-mon[35364]: pgmap v4352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:34 smithi181 ceph-mon[35869]: pgmap v4352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:34 smithi097 ceph-mon[32231]: pgmap v4352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6912.787547 secs 2024-07-26T09:54:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:35.812+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:35.813+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:35.813+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:35.813+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:54:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:35.813+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6912.787547 secs 2024-07-26T09:54:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6912.787547 secs 2024-07-26T09:54:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:36 smithi097 ceph-mon[32231]: pgmap v4353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T09:54:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:36 smithi115 ceph-mon[35364]: pgmap v4353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T09:54:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:36 smithi181 ceph-mon[35869]: pgmap v4353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T09:54:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:38 smithi097 ceph-mon[32231]: pgmap v4354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:54:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:38 smithi115 ceph-mon[35364]: pgmap v4354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:54:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:38 smithi181 ceph-mon[35869]: pgmap v4354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:54:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:40 smithi097 ceph-mon[32231]: pgmap v4355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:54:40.784 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6917.787680 secs 2024-07-26T09:54:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:40 smithi115 ceph-mon[35364]: pgmap v4355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:54:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6917.787680 secs 2024-07-26T09:54:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:40 smithi181 ceph-mon[35869]: pgmap v4355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:54:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6917.787680 secs 2024-07-26T09:54:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:42 smithi115 ceph-mon[35364]: pgmap v4356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:42 smithi181 ceph-mon[35869]: pgmap v4356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:42 smithi097 ceph-mon[32231]: pgmap v4356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:44 smithi115 ceph-mon[35364]: pgmap v4357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:54:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:44 smithi181 ceph-mon[35869]: pgmap v4357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:54:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:44 smithi097 ceph-mon[32231]: pgmap v4357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:54:45.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6922.787831 secs 2024-07-26T09:54:45.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6922.787831 secs 2024-07-26T09:54:45.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6922.787831 secs 2024-07-26T09:54:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:46 smithi115 ceph-mon[35364]: pgmap v4358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:46.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:46 smithi181 ceph-mon[35869]: pgmap v4358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:46 smithi097 ceph-mon[32231]: pgmap v4358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:48 smithi115 ceph-mon[35364]: pgmap v4359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:54:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:48 smithi181 ceph-mon[35869]: pgmap v4359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:54:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:48 smithi097 ceph-mon[32231]: pgmap v4359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:54:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:50.822+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:54:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:50.823+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:54:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:50.823+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:54:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:50.823+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:54:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:54:50.823+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:50 smithi115 ceph-mon[35364]: pgmap v4360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:54:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6927.787974 secs 2024-07-26T09:54:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:50 smithi181 ceph-mon[35869]: pgmap v4360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:54:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6927.787974 secs 2024-07-26T09:54:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:50 smithi097 ceph-mon[32231]: pgmap v4360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:54:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6927.787974 secs 2024-07-26T09:54:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:52 smithi115 ceph-mon[35364]: pgmap v4361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:52 smithi181 ceph-mon[35869]: pgmap v4361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:52 smithi097 ceph-mon[32231]: pgmap v4361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:54:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:54 smithi115 ceph-mon[35364]: pgmap v4362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:54 smithi181 ceph-mon[35869]: pgmap v4362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:54 smithi097 ceph-mon[32231]: pgmap v4362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6932.788116 secs 2024-07-26T09:54:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6932.788116 secs 2024-07-26T09:54:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6932.788116 secs 2024-07-26T09:54:56.962 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:56 smithi181 ceph-mon[35869]: pgmap v4363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:56 smithi115 ceph-mon[35364]: pgmap v4363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:56 smithi097 ceph-mon[32231]: pgmap v4363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:54:58 smithi115 ceph-mon[35364]: pgmap v4364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:54:58 smithi181 ceph-mon[35869]: pgmap v4364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:54:58 smithi097 ceph-mon[32231]: pgmap v4364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:00 smithi115 ceph-mon[35364]: pgmap v4365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6937.788285 secs 2024-07-26T09:55:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:00 smithi181 ceph-mon[35869]: pgmap v4365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6937.788285 secs 2024-07-26T09:55:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:00 smithi097 ceph-mon[32231]: pgmap v4365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6937.788285 secs 2024-07-26T09:55:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:02 smithi115 ceph-mon[35364]: pgmap v4366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:02 smithi181 ceph-mon[35869]: pgmap v4366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:02 smithi097 ceph-mon[32231]: pgmap v4366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:04 smithi115 ceph-mon[35364]: pgmap v4367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:04 smithi181 ceph-mon[35869]: pgmap v4367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:04 smithi097 ceph-mon[32231]: pgmap v4367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:05.829+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:55:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:05.830+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:55:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:05.830+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:55:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:05.830+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:55:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:05.830+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6942.788426 secs 2024-07-26T09:55:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6942.788426 secs 2024-07-26T09:55:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6942.788426 secs 2024-07-26T09:55:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:06 smithi115 ceph-mon[35364]: pgmap v4368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:06 smithi181 ceph-mon[35869]: pgmap v4368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:06 smithi097 ceph-mon[32231]: pgmap v4368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:08.107+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:55:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:08.107+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:55:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:08.107+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:55:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:08.107+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:55:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:08 smithi115 ceph-mon[35364]: pgmap v4369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:08 smithi181 ceph-mon[35869]: pgmap v4369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:08 smithi097 ceph-mon[32231]: pgmap v4369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:10 smithi115 ceph-mon[35364]: pgmap v4370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6947.788570 secs 2024-07-26T09:55:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:10 smithi181 ceph-mon[35869]: pgmap v4370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6947.788570 secs 2024-07-26T09:55:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:10 smithi097 ceph-mon[32231]: pgmap v4370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6947.788570 secs 2024-07-26T09:55:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:12 smithi115 ceph-mon[35364]: pgmap v4371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:12 smithi181 ceph-mon[35869]: pgmap v4371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:12 smithi097 ceph-mon[32231]: pgmap v4371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:14 smithi115 ceph-mon[35364]: pgmap v4372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:14 smithi181 ceph-mon[35869]: pgmap v4372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:14 smithi097 ceph-mon[32231]: pgmap v4372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6952.788713 secs 2024-07-26T09:55:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6952.788713 secs 2024-07-26T09:55:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6952.788713 secs 2024-07-26T09:55:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:16 smithi115 ceph-mon[35364]: pgmap v4373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:16 smithi181 ceph-mon[35869]: pgmap v4373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:16 smithi097 ceph-mon[32231]: pgmap v4373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:18 smithi115 ceph-mon[35364]: pgmap v4374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:18 smithi181 ceph-mon[35869]: pgmap v4374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:18 smithi097 ceph-mon[32231]: pgmap v4374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:20.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:20.841+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:55:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:20.842+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:55:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:20.842+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:55:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:20.842+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:55:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:20.842+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:20 smithi115 ceph-mon[35364]: pgmap v4375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6957.788856 secs 2024-07-26T09:55:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:20 smithi181 ceph-mon[35869]: pgmap v4375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6957.788856 secs 2024-07-26T09:55:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:20 smithi097 ceph-mon[32231]: pgmap v4375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6957.788856 secs 2024-07-26T09:55:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:22 smithi115 ceph-mon[35364]: pgmap v4376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:22 smithi181 ceph-mon[35869]: pgmap v4376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:22 smithi097 ceph-mon[32231]: pgmap v4376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:24 smithi115 ceph-mon[35364]: pgmap v4377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:24 smithi181 ceph-mon[35869]: pgmap v4377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:24 smithi097 ceph-mon[32231]: pgmap v4377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6962.788997 secs 2024-07-26T09:55:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6962.788997 secs 2024-07-26T09:55:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6962.788997 secs 2024-07-26T09:55:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:26 smithi115 ceph-mon[35364]: pgmap v4378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:26 smithi181 ceph-mon[35869]: pgmap v4378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:26 smithi097 ceph-mon[32231]: pgmap v4378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:28 smithi115 ceph-mon[35364]: pgmap v4379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:28 smithi181 ceph-mon[35869]: pgmap v4379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:28 smithi097 ceph-mon[32231]: pgmap v4379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:30 smithi097 ceph-mon[32231]: pgmap v4380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6967.789098 secs 2024-07-26T09:55:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:30 smithi097 ceph-mon[32231]: pgmap v4381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:55:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:30 smithi097 ceph-mon[32231]: pgmap v4382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:31.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:55:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:30 smithi115 ceph-mon[35364]: pgmap v4380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6967.789098 secs 2024-07-26T09:55:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:31.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:30 smithi115 ceph-mon[35364]: pgmap v4381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:55:31.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:30 smithi115 ceph-mon[35364]: pgmap v4382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:31.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:55:31.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:30 smithi181 ceph-mon[35869]: pgmap v4380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:31.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6967.789098 secs 2024-07-26T09:55:31.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:31.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:31.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:30 smithi181 ceph-mon[35869]: pgmap v4381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:55:31.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:30 smithi181 ceph-mon[35869]: pgmap v4382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:31.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:55:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:33 smithi181 ceph-mon[35869]: pgmap v4383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:55:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:33 smithi097 ceph-mon[32231]: pgmap v4383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:55:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:33 smithi115 ceph-mon[35364]: pgmap v4383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:55:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:35 smithi097 ceph-mon[32231]: pgmap v4384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6972.789284 secs 2024-07-26T09:55:35.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:35.848+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:55:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:35.849+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:55:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:35.849+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:55:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:35.849+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:55:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:35.849+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:35 smithi115 ceph-mon[35364]: pgmap v4384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6972.789284 secs 2024-07-26T09:55:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:35 smithi181 ceph-mon[35869]: pgmap v4384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6972.789284 secs 2024-07-26T09:55:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:37 smithi097 ceph-mon[32231]: pgmap v4385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:37 smithi115 ceph-mon[35364]: pgmap v4385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:37 smithi181 ceph-mon[35869]: pgmap v4385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:39 smithi097 ceph-mon[32231]: pgmap v4386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:55:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:39 smithi115 ceph-mon[35364]: pgmap v4386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:55:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:39 smithi181 ceph-mon[35869]: pgmap v4386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:55:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6977.789385 secs 2024-07-26T09:55:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6977.789385 secs 2024-07-26T09:55:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6977.789385 secs 2024-07-26T09:55:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:41 smithi115 ceph-mon[35364]: pgmap v4387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:55:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:41 smithi181 ceph-mon[35869]: pgmap v4387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:55:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:41 smithi097 ceph-mon[32231]: pgmap v4387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:55:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:43 smithi115 ceph-mon[35364]: pgmap v4388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:43 smithi181 ceph-mon[35869]: pgmap v4388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:43 smithi097 ceph-mon[32231]: pgmap v4388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:45.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:45 smithi115 ceph-mon[35364]: pgmap v4389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:45.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6982.789489 secs 2024-07-26T09:55:45.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:45 smithi181 ceph-mon[35869]: pgmap v4389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:45.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6982.789489 secs 2024-07-26T09:55:45.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:45 smithi097 ceph-mon[32231]: pgmap v4389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:45.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6982.789489 secs 2024-07-26T09:55:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:47 smithi115 ceph-mon[35364]: pgmap v4390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:47 smithi181 ceph-mon[35869]: pgmap v4390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:47 smithi097 ceph-mon[32231]: pgmap v4390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:49 smithi115 ceph-mon[35364]: pgmap v4391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:49 smithi181 ceph-mon[35869]: pgmap v4391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:49 smithi097 ceph-mon[32231]: pgmap v4391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:50.860+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:55:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:50.862+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:55:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:50.862+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:55:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:50.862+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:55:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:55:50.862+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6987.789645 secs 2024-07-26T09:55:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6987.789645 secs 2024-07-26T09:55:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6987.789645 secs 2024-07-26T09:55:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:51 smithi115 ceph-mon[35364]: pgmap v4392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:51 smithi181 ceph-mon[35869]: pgmap v4392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:51 smithi097 ceph-mon[32231]: pgmap v4392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:53 smithi115 ceph-mon[35364]: pgmap v4393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:53 smithi181 ceph-mon[35869]: pgmap v4393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:53 smithi097 ceph-mon[32231]: pgmap v4393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:55 smithi115 ceph-mon[35364]: pgmap v4394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6992.789760 secs 2024-07-26T09:55:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:55 smithi181 ceph-mon[35869]: pgmap v4394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6992.789760 secs 2024-07-26T09:55:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:55 smithi097 ceph-mon[32231]: pgmap v4394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:55:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6992.789760 secs 2024-07-26T09:55:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:57 smithi115 ceph-mon[35364]: pgmap v4395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:57 smithi181 ceph-mon[35869]: pgmap v4395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:57 smithi097 ceph-mon[32231]: pgmap v4395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:55:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:55:59 smithi115 ceph-mon[35364]: pgmap v4396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:55:59 smithi181 ceph-mon[35869]: pgmap v4396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:55:59 smithi097 ceph-mon[32231]: pgmap v4396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 6997.789917 secs 2024-07-26T09:56:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 6997.789917 secs 2024-07-26T09:56:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 6997.789917 secs 2024-07-26T09:56:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:01 smithi115 ceph-mon[35364]: pgmap v4397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:01 smithi181 ceph-mon[35869]: pgmap v4397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:01 smithi097 ceph-mon[32231]: pgmap v4397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:03 smithi115 ceph-mon[35364]: pgmap v4398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:03 smithi181 ceph-mon[35869]: pgmap v4398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:03 smithi097 ceph-mon[32231]: pgmap v4398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:05.869+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:56:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:05.869+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:56:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:05.869+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:56:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:05.869+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:56:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:05.869+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:05 smithi115 ceph-mon[35364]: pgmap v4399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7002.790051 secs 2024-07-26T09:56:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:05 smithi181 ceph-mon[35869]: pgmap v4399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7002.790051 secs 2024-07-26T09:56:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:05 smithi097 ceph-mon[32231]: pgmap v4399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7002.790051 secs 2024-07-26T09:56:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:07 smithi115 ceph-mon[35364]: pgmap v4400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:07 smithi181 ceph-mon[35869]: pgmap v4400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:07 smithi097 ceph-mon[32231]: pgmap v4400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:09 smithi115 ceph-mon[35364]: pgmap v4401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:09 smithi181 ceph-mon[35869]: pgmap v4401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:09 smithi097 ceph-mon[32231]: pgmap v4401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7007.790207 secs 2024-07-26T09:56:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7007.790207 secs 2024-07-26T09:56:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7007.790207 secs 2024-07-26T09:56:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:11 smithi115 ceph-mon[35364]: pgmap v4402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:11 smithi181 ceph-mon[35869]: pgmap v4402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:11 smithi097 ceph-mon[32231]: pgmap v4402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:13 smithi115 ceph-mon[35364]: pgmap v4403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:13 smithi181 ceph-mon[35869]: pgmap v4403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:13 smithi097 ceph-mon[32231]: pgmap v4403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:15 smithi115 ceph-mon[35364]: pgmap v4404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7012.790355 secs 2024-07-26T09:56:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:15 smithi181 ceph-mon[35869]: pgmap v4404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7012.790355 secs 2024-07-26T09:56:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:15 smithi097 ceph-mon[32231]: pgmap v4404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7012.790355 secs 2024-07-26T09:56:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:17 smithi115 ceph-mon[35364]: pgmap v4405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:17 smithi181 ceph-mon[35869]: pgmap v4405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:17 smithi097 ceph-mon[32231]: pgmap v4405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:19 smithi115 ceph-mon[35364]: pgmap v4406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:19 smithi181 ceph-mon[35869]: pgmap v4406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:19 smithi097 ceph-mon[32231]: pgmap v4406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:20.879+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:56:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:20.880+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:56:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:20.880+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:56:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:20.880+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:56:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:20.880+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7017.790445 secs 2024-07-26T09:56:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7017.790445 secs 2024-07-26T09:56:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7017.790445 secs 2024-07-26T09:56:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:21 smithi115 ceph-mon[35364]: pgmap v4407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:21 smithi181 ceph-mon[35869]: pgmap v4407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:21 smithi097 ceph-mon[32231]: pgmap v4407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:23 smithi115 ceph-mon[35364]: pgmap v4408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:23 smithi181 ceph-mon[35869]: pgmap v4408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:23 smithi097 ceph-mon[32231]: pgmap v4408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:25.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:25 smithi115 ceph-mon[35364]: pgmap v4409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:25.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7022.790586 secs 2024-07-26T09:56:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:25 smithi181 ceph-mon[35869]: pgmap v4409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7022.790586 secs 2024-07-26T09:56:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:25 smithi097 ceph-mon[32231]: pgmap v4409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7022.790586 secs 2024-07-26T09:56:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:27 smithi115 ceph-mon[35364]: pgmap v4410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:27 smithi181 ceph-mon[35869]: pgmap v4410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:27 smithi097 ceph-mon[32231]: pgmap v4410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:29 smithi115 ceph-mon[35364]: pgmap v4411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:29 smithi181 ceph-mon[35869]: pgmap v4411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:29 smithi097 ceph-mon[32231]: pgmap v4411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7027.790727 secs 2024-07-26T09:56:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7027.790727 secs 2024-07-26T09:56:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7027.790727 secs 2024-07-26T09:56:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:31 smithi097 ceph-mon[32231]: pgmap v4412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:31 smithi115 ceph-mon[35364]: pgmap v4412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:31 smithi181 ceph-mon[35869]: pgmap v4412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:32 smithi097 ceph-mon[32231]: pgmap v4413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:56:33.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:32 smithi097 ceph-mon[32231]: pgmap v4414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:33.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:56:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:32 smithi115 ceph-mon[35364]: pgmap v4413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:56:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:32 smithi115 ceph-mon[35364]: pgmap v4414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:56:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:33.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:33.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:32 smithi181 ceph-mon[35869]: pgmap v4413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:56:33.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:32 smithi181 ceph-mon[35869]: pgmap v4414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:33.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:56:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:34 smithi115 ceph-mon[35364]: pgmap v4415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:34 smithi181 ceph-mon[35869]: pgmap v4415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:34 smithi097 ceph-mon[32231]: pgmap v4415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:35.886+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:56:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:35.887+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:56:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:35.887+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:56:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:35.887+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:56:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:35.888+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7032.790864 secs 2024-07-26T09:56:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7032.790864 secs 2024-07-26T09:56:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7032.790864 secs 2024-07-26T09:56:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:36 smithi115 ceph-mon[35364]: pgmap v4416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:36 smithi181 ceph-mon[35869]: pgmap v4416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:36 smithi097 ceph-mon[32231]: pgmap v4416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:38 smithi115 ceph-mon[35364]: pgmap v4417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:38 smithi181 ceph-mon[35869]: pgmap v4417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:38 smithi097 ceph-mon[32231]: pgmap v4417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7037.791008 secs 2024-07-26T09:56:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:40 smithi115 ceph-mon[35364]: pgmap v4418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7037.791008 secs 2024-07-26T09:56:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:40 smithi181 ceph-mon[35869]: pgmap v4418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7037.791008 secs 2024-07-26T09:56:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:40 smithi097 ceph-mon[32231]: pgmap v4418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:42 smithi115 ceph-mon[35364]: pgmap v4419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:56:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:42 smithi181 ceph-mon[35869]: pgmap v4419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:56:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:42 smithi097 ceph-mon[32231]: pgmap v4419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:56:45.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:44 smithi181 ceph-mon[35869]: pgmap v4420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:45.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:44 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7042.791138 secs 2024-07-26T09:56:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:44 smithi097 ceph-mon[32231]: pgmap v4420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:44 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7042.791138 secs 2024-07-26T09:56:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:44 smithi115 ceph-mon[35364]: pgmap v4420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:44 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7042.791138 secs 2024-07-26T09:56:47.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:46 smithi181 ceph-mon[35869]: pgmap v4421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:46 smithi097 ceph-mon[32231]: pgmap v4421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:46 smithi115 ceph-mon[35364]: pgmap v4421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:48 smithi097 ceph-mon[32231]: pgmap v4422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:48 smithi115 ceph-mon[35364]: pgmap v4422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:48 smithi181 ceph-mon[35869]: pgmap v4422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:49 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7047.791277 secs 2024-07-26T09:56:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7047.791277 secs 2024-07-26T09:56:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7047.791277 secs 2024-07-26T09:56:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:50.891+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:56:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:50.892+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:56:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:50.892+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:56:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:50.892+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:56:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:56:50.892+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:51 smithi115 ceph-mon[35364]: pgmap v4423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:51 smithi181 ceph-mon[35869]: pgmap v4423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:51 smithi097 ceph-mon[32231]: pgmap v4423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:53 smithi115 ceph-mon[35364]: pgmap v4424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:53 smithi181 ceph-mon[35869]: pgmap v4424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:53 smithi097 ceph-mon[32231]: pgmap v4424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:55 smithi115 ceph-mon[35364]: pgmap v4425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7052.791440 secs 2024-07-26T09:56:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:55 smithi181 ceph-mon[35869]: pgmap v4425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:55.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7052.791440 secs 2024-07-26T09:56:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:55 smithi097 ceph-mon[32231]: pgmap v4425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:56:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7052.791440 secs 2024-07-26T09:56:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:57 smithi115 ceph-mon[35364]: pgmap v4426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:57 smithi181 ceph-mon[35869]: pgmap v4426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:57 smithi097 ceph-mon[32231]: pgmap v4426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:56:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:56:59 smithi115 ceph-mon[35364]: pgmap v4427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:56:59 smithi181 ceph-mon[35869]: pgmap v4427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:56:59 smithi097 ceph-mon[32231]: pgmap v4427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7057.791525 secs 2024-07-26T09:57:00.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7057.791525 secs 2024-07-26T09:57:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7057.791525 secs 2024-07-26T09:57:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:01 smithi115 ceph-mon[35364]: pgmap v4428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:01 smithi181 ceph-mon[35869]: pgmap v4428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:01 smithi097 ceph-mon[32231]: pgmap v4428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:03 smithi115 ceph-mon[35364]: pgmap v4429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:03 smithi181 ceph-mon[35869]: pgmap v4429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:03 smithi097 ceph-mon[32231]: pgmap v4429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:05 smithi115 ceph-mon[35364]: pgmap v4430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7062.791685 secs 2024-07-26T09:57:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:05 smithi181 ceph-mon[35869]: pgmap v4430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7062.791685 secs 2024-07-26T09:57:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:05 smithi097 ceph-mon[32231]: pgmap v4430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7062.791685 secs 2024-07-26T09:57:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:05.898+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:57:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:05.899+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:57:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:05.899+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:57:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:05.899+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:57:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:05.899+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:07 smithi115 ceph-mon[35364]: pgmap v4431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:07 smithi181 ceph-mon[35869]: pgmap v4431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:07 smithi097 ceph-mon[32231]: pgmap v4431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:09 smithi115 ceph-mon[35364]: pgmap v4432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:09 smithi181 ceph-mon[35869]: pgmap v4432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:09 smithi097 ceph-mon[32231]: pgmap v4432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7067.791820 secs 2024-07-26T09:57:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7067.791820 secs 2024-07-26T09:57:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7067.791820 secs 2024-07-26T09:57:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:11 smithi115 ceph-mon[35364]: pgmap v4433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:11 smithi181 ceph-mon[35869]: pgmap v4433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:11 smithi097 ceph-mon[32231]: pgmap v4433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:13 smithi115 ceph-mon[35364]: pgmap v4434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:13 smithi181 ceph-mon[35869]: pgmap v4434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:13 smithi097 ceph-mon[32231]: pgmap v4434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:15 smithi115 ceph-mon[35364]: pgmap v4435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7072.791963 secs 2024-07-26T09:57:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:15 smithi181 ceph-mon[35869]: pgmap v4435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:15.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7072.791963 secs 2024-07-26T09:57:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:15 smithi097 ceph-mon[32231]: pgmap v4435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7072.791963 secs 2024-07-26T09:57:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:17 smithi115 ceph-mon[35364]: pgmap v4436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:17 smithi181 ceph-mon[35869]: pgmap v4436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:17 smithi097 ceph-mon[32231]: pgmap v4436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:19 smithi115 ceph-mon[35364]: pgmap v4437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:19 smithi181 ceph-mon[35869]: pgmap v4437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:19 smithi097 ceph-mon[32231]: pgmap v4437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7077.792105 secs 2024-07-26T09:57:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7077.792105 secs 2024-07-26T09:57:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7077.792105 secs 2024-07-26T09:57:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:20.910+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:57:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:20.911+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:57:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:20.911+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:57:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:20.911+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:57:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:20.911+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:21 smithi115 ceph-mon[35364]: pgmap v4438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:21 smithi181 ceph-mon[35869]: pgmap v4438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:21 smithi097 ceph-mon[32231]: pgmap v4438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:23 smithi115 ceph-mon[35364]: pgmap v4439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:23 smithi181 ceph-mon[35869]: pgmap v4439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:23 smithi097 ceph-mon[32231]: pgmap v4439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:25 smithi115 ceph-mon[35364]: pgmap v4440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7082.792277 secs 2024-07-26T09:57:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:25 smithi181 ceph-mon[35869]: pgmap v4440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:25.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7082.792277 secs 2024-07-26T09:57:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:25 smithi097 ceph-mon[32231]: pgmap v4440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7082.792277 secs 2024-07-26T09:57:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:27 smithi115 ceph-mon[35364]: pgmap v4441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:27 smithi181 ceph-mon[35869]: pgmap v4441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:27 smithi097 ceph-mon[32231]: pgmap v4441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:29 smithi115 ceph-mon[35364]: pgmap v4442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:29 smithi181 ceph-mon[35869]: pgmap v4442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:29 smithi097 ceph-mon[32231]: pgmap v4442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7087.792421 secs 2024-07-26T09:57:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7087.792421 secs 2024-07-26T09:57:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7087.792421 secs 2024-07-26T09:57:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:31 smithi115 ceph-mon[35364]: pgmap v4443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:31 smithi181 ceph-mon[35869]: pgmap v4443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:31 smithi097 ceph-mon[32231]: pgmap v4443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:33 smithi115 ceph-mon[35364]: pgmap v4444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:33.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:33 smithi181 ceph-mon[35869]: pgmap v4444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:33.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:33 smithi097 ceph-mon[32231]: pgmap v4444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:34 smithi115 ceph-mon[35364]: pgmap v4445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:57:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:34 smithi115 ceph-mon[35364]: pgmap v4446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:57:34.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:57:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:34 smithi181 ceph-mon[35869]: pgmap v4445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:57:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:34 smithi181 ceph-mon[35869]: pgmap v4446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:57:34.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:57:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:34 smithi097 ceph-mon[32231]: pgmap v4445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T09:57:34.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:34 smithi097 ceph-mon[32231]: pgmap v4446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T09:57:34.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:57:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7092.792563 secs 2024-07-26T09:57:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7092.792563 secs 2024-07-26T09:57:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7092.792563 secs 2024-07-26T09:57:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:35.917+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:57:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:35.918+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:57:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:35.918+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:57:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:35.918+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:57:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:35.918+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:36 smithi181 ceph-mon[35869]: pgmap v4447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:57:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:36 smithi097 ceph-mon[32231]: pgmap v4447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:57:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:36 smithi115 ceph-mon[35364]: pgmap v4447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:57:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:38 smithi115 ceph-mon[35364]: pgmap v4448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:57:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:38 smithi181 ceph-mon[35869]: pgmap v4448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:57:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:38 smithi097 ceph-mon[32231]: pgmap v4448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:57:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:40 smithi115 ceph-mon[35364]: pgmap v4449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7097.792707 secs 2024-07-26T09:57:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:40 smithi181 ceph-mon[35869]: pgmap v4449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7097.792707 secs 2024-07-26T09:57:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:40 smithi097 ceph-mon[32231]: pgmap v4449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7097.792707 secs 2024-07-26T09:57:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:42 smithi115 ceph-mon[35364]: pgmap v4450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:57:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:42 smithi181 ceph-mon[35869]: pgmap v4450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:57:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:42 smithi097 ceph-mon[32231]: pgmap v4450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:57:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:44 smithi115 ceph-mon[35364]: pgmap v4451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:57:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:44 smithi181 ceph-mon[35869]: pgmap v4451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:57:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:44 smithi097 ceph-mon[32231]: pgmap v4451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:57:45.655 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7102.792865 secs 2024-07-26T09:57:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7102.792865 secs 2024-07-26T09:57:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7102.792865 secs 2024-07-26T09:57:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:46 smithi115 ceph-mon[35364]: pgmap v4452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:46 smithi181 ceph-mon[35869]: pgmap v4452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:46 smithi097 ceph-mon[32231]: pgmap v4452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:48 smithi115 ceph-mon[35364]: pgmap v4453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:48 smithi181 ceph-mon[35869]: pgmap v4453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:48 smithi097 ceph-mon[32231]: pgmap v4453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:50 smithi115 ceph-mon[35364]: pgmap v4454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7107.793011 secs 2024-07-26T09:57:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:50 smithi181 ceph-mon[35869]: pgmap v4454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7107.793011 secs 2024-07-26T09:57:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:50 smithi097 ceph-mon[32231]: pgmap v4454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7107.793011 secs 2024-07-26T09:57:50.931 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:50.929+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:57:50.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:50.930+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:57:50.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:50.930+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:57:50.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:50.930+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:57:50.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:57:50.930+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:52 smithi115 ceph-mon[35364]: pgmap v4455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:52 smithi181 ceph-mon[35869]: pgmap v4455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:52 smithi097 ceph-mon[32231]: pgmap v4455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:54 smithi115 ceph-mon[35364]: pgmap v4456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:54 smithi181 ceph-mon[35869]: pgmap v4456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:54 smithi097 ceph-mon[32231]: pgmap v4456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7112.793160 secs 2024-07-26T09:57:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7112.793160 secs 2024-07-26T09:57:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7112.793160 secs 2024-07-26T09:57:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:56 smithi115 ceph-mon[35364]: pgmap v4457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:57:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:56 smithi181 ceph-mon[35869]: pgmap v4457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:57:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:56 smithi097 ceph-mon[32231]: pgmap v4457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:57:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:57:58 smithi115 ceph-mon[35364]: pgmap v4458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:57:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:57:58 smithi181 ceph-mon[35869]: pgmap v4458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:57:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:57:58 smithi097 ceph-mon[32231]: pgmap v4458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:58:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:00 smithi115 ceph-mon[35364]: pgmap v4459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:58:00.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7117.793304 secs 2024-07-26T09:58:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:00 smithi181 ceph-mon[35869]: pgmap v4459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:58:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7117.793304 secs 2024-07-26T09:58:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:00 smithi097 ceph-mon[32231]: pgmap v4459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:58:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7117.793304 secs 2024-07-26T09:58:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:02 smithi115 ceph-mon[35364]: pgmap v4460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:02 smithi181 ceph-mon[35869]: pgmap v4460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:02 smithi097 ceph-mon[32231]: pgmap v4460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:04 smithi115 ceph-mon[35364]: pgmap v4461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:04 smithi181 ceph-mon[35869]: pgmap v4461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:04 smithi097 ceph-mon[32231]: pgmap v4461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7122.793404 secs 2024-07-26T09:58:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:05.935+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:58:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:05.937+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:58:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:05.937+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:58:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:05.937+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:58:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:05.937+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7122.793404 secs 2024-07-26T09:58:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7122.793404 secs 2024-07-26T09:58:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:06 smithi097 ceph-mon[32231]: pgmap v4462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.899+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 msgr2=0x7ff3d40b4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T09:58:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.899+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 msgr2=0x7ff3d40b4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:58:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.899+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 0x7ff3d40b4fa0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e0071000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:58:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.899+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 0x7ff3d40b4fa0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e0071000 comp rx=0 tx=0).stop 2024-07-26T09:58:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.899+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 msgr2=0x7ff3d40b4fa0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T09:58:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.899+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 msgr2=0x7ff3d40b4fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:58:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.899+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4b90 0x7ff3d40b4fa0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:58:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.899+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:58:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.900+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:58:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:06.901+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006300 tx=0x7ff3e009e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:58:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:06 smithi115 ceph-mon[35364]: pgmap v4462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:06 smithi181 ceph-mon[35869]: pgmap v4462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:08.138+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:58:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:08.138+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:58:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:08.139+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T09:58:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:08.139+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T09:58:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:08 smithi097 ceph-mon[32231]: pgmap v4463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:58:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:08 smithi115 ceph-mon[35364]: pgmap v4463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:58:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:08 smithi181 ceph-mon[35869]: pgmap v4463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:58:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:10 smithi115 ceph-mon[35364]: pgmap v4464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7127.793546 secs 2024-07-26T09:58:10.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:10 smithi181 ceph-mon[35869]: pgmap v4464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:10.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7127.793546 secs 2024-07-26T09:58:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:10 smithi097 ceph-mon[32231]: pgmap v4464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7127.793546 secs 2024-07-26T09:58:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:12 smithi115 ceph-mon[35364]: pgmap v4465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:12 smithi181 ceph-mon[35869]: pgmap v4465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:12 smithi097 ceph-mon[32231]: pgmap v4465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:14 smithi115 ceph-mon[35364]: pgmap v4466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:14 smithi181 ceph-mon[35869]: pgmap v4466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:14 smithi097 ceph-mon[32231]: pgmap v4466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:15.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7132.793692 secs 2024-07-26T09:58:15.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7132.793692 secs 2024-07-26T09:58:15.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7132.793692 secs 2024-07-26T09:58:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:16 smithi115 ceph-mon[35364]: pgmap v4467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:16 smithi181 ceph-mon[35869]: pgmap v4467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:16 smithi097 ceph-mon[32231]: pgmap v4467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:18 smithi115 ceph-mon[35364]: pgmap v4468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:58:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:18 smithi181 ceph-mon[35869]: pgmap v4468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:58:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:18 smithi097 ceph-mon[32231]: pgmap v4468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:58:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:20.948+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:58:20.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:20.950+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:58:20.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:20.950+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:58:20.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:20.950+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:58:20.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:20.950+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:20 smithi115 ceph-mon[35364]: pgmap v4469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7137.793835 secs 2024-07-26T09:58:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:20 smithi181 ceph-mon[35869]: pgmap v4469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7137.793835 secs 2024-07-26T09:58:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:20 smithi097 ceph-mon[32231]: pgmap v4469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7137.793835 secs 2024-07-26T09:58:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:22 smithi115 ceph-mon[35364]: pgmap v4470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:22 smithi181 ceph-mon[35869]: pgmap v4470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:22 smithi097 ceph-mon[32231]: pgmap v4470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:24 smithi115 ceph-mon[35364]: pgmap v4471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:24 smithi181 ceph-mon[35869]: pgmap v4471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:24 smithi097 ceph-mon[32231]: pgmap v4471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7142.793982 secs 2024-07-26T09:58:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7142.793982 secs 2024-07-26T09:58:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7142.793982 secs 2024-07-26T09:58:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:26 smithi115 ceph-mon[35364]: pgmap v4472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:26.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:26 smithi181 ceph-mon[35869]: pgmap v4472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:26 smithi097 ceph-mon[32231]: pgmap v4472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:28 smithi115 ceph-mon[35364]: pgmap v4473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:28 smithi181 ceph-mon[35869]: pgmap v4473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:28 smithi097 ceph-mon[32231]: pgmap v4473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:30 smithi115 ceph-mon[35364]: pgmap v4474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7147.794114 secs 2024-07-26T09:58:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:30 smithi181 ceph-mon[35869]: pgmap v4474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7147.794114 secs 2024-07-26T09:58:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:30 smithi097 ceph-mon[32231]: pgmap v4474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7147.794114 secs 2024-07-26T09:58:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:32 smithi115 ceph-mon[35364]: pgmap v4475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:32 smithi181 ceph-mon[35869]: pgmap v4475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:32 smithi097 ceph-mon[32231]: pgmap v4475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:34 smithi115 ceph-mon[35364]: pgmap v4476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:34 smithi181 ceph-mon[35869]: pgmap v4476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:34 smithi097 ceph-mon[32231]: pgmap v4476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:35.957+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:58:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:35.957+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:58:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:35.957+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:58:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:35.957+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:58:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:35.957+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:35 smithi115 ceph-mon[35364]: pgmap v4477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:58:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:35 smithi115 ceph-mon[35364]: pgmap v4478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:58:35.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7152.794276 secs 2024-07-26T09:58:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:35 smithi181 ceph-mon[35869]: pgmap v4477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:58:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:35 smithi181 ceph-mon[35869]: pgmap v4478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:58:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7152.794276 secs 2024-07-26T09:58:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:35 smithi097 ceph-mon[32231]: pgmap v4477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:58:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:35 smithi097 ceph-mon[32231]: pgmap v4478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:58:36.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7152.794276 secs 2024-07-26T09:58:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:36 smithi115 ceph-mon[35364]: pgmap v4479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:36 smithi181 ceph-mon[35869]: pgmap v4479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:36 smithi097 ceph-mon[32231]: pgmap v4479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:38 smithi115 ceph-mon[35364]: pgmap v4480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:38 smithi181 ceph-mon[35869]: pgmap v4480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:38 smithi097 ceph-mon[32231]: pgmap v4480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7157.794429 secs 2024-07-26T09:58:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7157.794429 secs 2024-07-26T09:58:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7157.794429 secs 2024-07-26T09:58:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:41 smithi115 ceph-mon[35364]: pgmap v4481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:41 smithi181 ceph-mon[35869]: pgmap v4481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:41 smithi097 ceph-mon[32231]: pgmap v4481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:42 smithi115 ceph-mon[35364]: pgmap v4482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:42 smithi181 ceph-mon[35869]: pgmap v4482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:42 smithi097 ceph-mon[32231]: pgmap v4482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:45.056 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:44 smithi097 ceph-mon[32231]: pgmap v4483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:58:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:44 smithi115 ceph-mon[35364]: pgmap v4483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:58:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:44 smithi181 ceph-mon[35869]: pgmap v4483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:58:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7162.794574 secs 2024-07-26T09:58:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7162.794574 secs 2024-07-26T09:58:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7162.794574 secs 2024-07-26T09:58:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:46 smithi115 ceph-mon[35364]: pgmap v4484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:46 smithi181 ceph-mon[35869]: pgmap v4484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:46 smithi097 ceph-mon[32231]: pgmap v4484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:48 smithi115 ceph-mon[35364]: pgmap v4485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:48 smithi181 ceph-mon[35869]: pgmap v4485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:48 smithi097 ceph-mon[32231]: pgmap v4485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7167.794670 secs 2024-07-26T09:58:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7167.794670 secs 2024-07-26T09:58:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7167.794670 secs 2024-07-26T09:58:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:50.960+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:58:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:50.961+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:58:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:50.961+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:58:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:50.961+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:58:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:58:50.961+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:51 smithi115 ceph-mon[35364]: pgmap v4486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:51 smithi181 ceph-mon[35869]: pgmap v4486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:51 smithi097 ceph-mon[32231]: pgmap v4486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:52 smithi115 ceph-mon[35364]: pgmap v4487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:52 smithi181 ceph-mon[35869]: pgmap v4487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:52 smithi097 ceph-mon[32231]: pgmap v4487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:58:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:54 smithi115 ceph-mon[35364]: pgmap v4488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:54 smithi181 ceph-mon[35869]: pgmap v4488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:54 smithi097 ceph-mon[32231]: pgmap v4488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7172.794765 secs 2024-07-26T09:58:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7172.794765 secs 2024-07-26T09:58:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7172.794765 secs 2024-07-26T09:58:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:56 smithi115 ceph-mon[35364]: pgmap v4489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:58:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:56 smithi181 ceph-mon[35869]: pgmap v4489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:58:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:56 smithi097 ceph-mon[32231]: pgmap v4489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:58:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:58:58 smithi115 ceph-mon[35364]: pgmap v4490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:58:58 smithi181 ceph-mon[35869]: pgmap v4490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:58:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:58:58 smithi097 ceph-mon[32231]: pgmap v4490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7177.794853 secs 2024-07-26T09:59:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7177.794853 secs 2024-07-26T09:59:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7177.794853 secs 2024-07-26T09:59:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:01 smithi115 ceph-mon[35364]: pgmap v4491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:01 smithi181 ceph-mon[35869]: pgmap v4491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:01 smithi097 ceph-mon[32231]: pgmap v4491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:02 smithi115 ceph-mon[35364]: pgmap v4492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:02 smithi181 ceph-mon[35869]: pgmap v4492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:02 smithi097 ceph-mon[32231]: pgmap v4492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:04 smithi115 ceph-mon[35364]: pgmap v4493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:04 smithi181 ceph-mon[35869]: pgmap v4493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:04 smithi097 ceph-mon[32231]: pgmap v4493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:05.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:05.968+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:59:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:05.969+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:59:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:05.969+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:59:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:05.969+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:59:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:05.969+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7182.794958 secs 2024-07-26T09:59:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7182.794958 secs 2024-07-26T09:59:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7182.794958 secs 2024-07-26T09:59:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:06 smithi115 ceph-mon[35364]: pgmap v4494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:06 smithi181 ceph-mon[35869]: pgmap v4494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:07.266 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:06 smithi097 ceph-mon[32231]: pgmap v4494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:08 smithi115 ceph-mon[35364]: pgmap v4495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:08 smithi181 ceph-mon[35869]: pgmap v4495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:08 smithi097 ceph-mon[32231]: pgmap v4495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7187.795099 secs 2024-07-26T09:59:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7187.795099 secs 2024-07-26T09:59:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7187.795099 secs 2024-07-26T09:59:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:11 smithi115 ceph-mon[35364]: pgmap v4496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:11 smithi181 ceph-mon[35869]: pgmap v4496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:11 smithi097 ceph-mon[32231]: pgmap v4496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:12 smithi115 ceph-mon[35364]: pgmap v4497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:12 smithi181 ceph-mon[35869]: pgmap v4497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:12 smithi097 ceph-mon[32231]: pgmap v4497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:14 smithi115 ceph-mon[35364]: pgmap v4498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:14 smithi181 ceph-mon[35869]: pgmap v4498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:14 smithi097 ceph-mon[32231]: pgmap v4498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7192.795259 secs 2024-07-26T09:59:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7192.795259 secs 2024-07-26T09:59:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7192.795259 secs 2024-07-26T09:59:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:16 smithi115 ceph-mon[35364]: pgmap v4499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:16 smithi181 ceph-mon[35869]: pgmap v4499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:16 smithi097 ceph-mon[32231]: pgmap v4499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:18 smithi115 ceph-mon[35364]: pgmap v4500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:18 smithi181 ceph-mon[35869]: pgmap v4500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:18 smithi097 ceph-mon[32231]: pgmap v4500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7197.795342 secs 2024-07-26T09:59:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7197.795342 secs 2024-07-26T09:59:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7197.795342 secs 2024-07-26T09:59:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:20.980+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:59:20.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:20.982+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:59:20.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:20.982+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:59:20.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:20.982+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:59:20.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:20.982+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:21 smithi115 ceph-mon[35364]: pgmap v4501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:21 smithi181 ceph-mon[35869]: pgmap v4501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:21 smithi097 ceph-mon[32231]: pgmap v4501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:22 smithi115 ceph-mon[35364]: pgmap v4502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:22 smithi181 ceph-mon[35869]: pgmap v4502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:22 smithi097 ceph-mon[32231]: pgmap v4502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:24 smithi115 ceph-mon[35364]: pgmap v4503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:24 smithi181 ceph-mon[35869]: pgmap v4503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:24 smithi097 ceph-mon[32231]: pgmap v4503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7202.795473 secs 2024-07-26T09:59:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7202.795473 secs 2024-07-26T09:59:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7202.795473 secs 2024-07-26T09:59:27.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:26 smithi115 ceph-mon[35364]: pgmap v4504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:26 smithi181 ceph-mon[35869]: pgmap v4504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:26 smithi097 ceph-mon[32231]: pgmap v4504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:28 smithi115 ceph-mon[35364]: pgmap v4505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:28 smithi181 ceph-mon[35869]: pgmap v4505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:28 smithi097 ceph-mon[32231]: pgmap v4505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7207.795614 secs 2024-07-26T09:59:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7207.795614 secs 2024-07-26T09:59:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7207.795614 secs 2024-07-26T09:59:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:31 smithi115 ceph-mon[35364]: pgmap v4506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:31 smithi181 ceph-mon[35869]: pgmap v4506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:31 smithi097 ceph-mon[32231]: pgmap v4506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:32 smithi115 ceph-mon[35364]: pgmap v4507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:32 smithi181 ceph-mon[35869]: pgmap v4507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:32 smithi097 ceph-mon[32231]: pgmap v4507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:34 smithi115 ceph-mon[35364]: pgmap v4508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:34 smithi181 ceph-mon[35869]: pgmap v4508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:34 smithi097 ceph-mon[32231]: pgmap v4508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:35.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:35.987+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:59:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:35.989+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:59:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:35.989+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:59:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:35.989+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:59:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:35.989+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7212.795749 secs 2024-07-26T09:59:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7212.795749 secs 2024-07-26T09:59:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7212.795749 secs 2024-07-26T09:59:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:36 smithi115 ceph-mon[35364]: pgmap v4509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:59:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:36 smithi115 ceph-mon[35364]: pgmap v4510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:59:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:37.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:37.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:36 smithi181 ceph-mon[35869]: pgmap v4509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:59:37.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:36 smithi181 ceph-mon[35869]: pgmap v4510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:37.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:59:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:36 smithi097 ceph-mon[32231]: pgmap v4509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:59:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:36 smithi097 ceph-mon[32231]: pgmap v4510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T09:59:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:39 smithi097 ceph-mon[32231]: pgmap v4511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:39 smithi115 ceph-mon[35364]: pgmap v4511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:39 smithi181 ceph-mon[35869]: pgmap v4511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7217.795890 secs 2024-07-26T09:59:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7217.795890 secs 2024-07-26T09:59:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7217.795890 secs 2024-07-26T09:59:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:41 smithi115 ceph-mon[35364]: pgmap v4512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:41 smithi181 ceph-mon[35869]: pgmap v4512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:41 smithi097 ceph-mon[32231]: pgmap v4512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:43 smithi115 ceph-mon[35364]: pgmap v4513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:43.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:43 smithi181 ceph-mon[35869]: pgmap v4513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:43 smithi097 ceph-mon[32231]: pgmap v4513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:45.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:45 smithi115 ceph-mon[35364]: pgmap v4514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:59:45.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7222.796029 secs 2024-07-26T09:59:45.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:45 smithi181 ceph-mon[35869]: pgmap v4514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:59:45.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7222.796029 secs 2024-07-26T09:59:45.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:45 smithi097 ceph-mon[32231]: pgmap v4514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:59:45.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7222.796029 secs 2024-07-26T09:59:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:47 smithi115 ceph-mon[35364]: pgmap v4515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:59:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:47 smithi181 ceph-mon[35869]: pgmap v4515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:59:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:47 smithi097 ceph-mon[32231]: pgmap v4515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:59:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:49 smithi115 ceph-mon[35364]: pgmap v4516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:49 smithi181 ceph-mon[35869]: pgmap v4516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:49 smithi097 ceph-mon[32231]: pgmap v4516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7227.796202 secs 2024-07-26T09:59:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7227.796202 secs 2024-07-26T09:59:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7227.796202 secs 2024-07-26T09:59:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:50.996+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:59:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:50.997+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T09:59:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:50.997+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T09:59:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:50.997+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:59:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T09:59:50.998+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:51 smithi115 ceph-mon[35364]: pgmap v4517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:51 smithi181 ceph-mon[35869]: pgmap v4517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:51 smithi097 ceph-mon[32231]: pgmap v4517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:53 smithi115 ceph-mon[35364]: pgmap v4518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:53 smithi181 ceph-mon[35869]: pgmap v4518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:53 smithi097 ceph-mon[32231]: pgmap v4518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:55 smithi115 ceph-mon[35364]: pgmap v4519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:59:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7232.796346 secs 2024-07-26T09:59:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:55 smithi181 ceph-mon[35869]: pgmap v4519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:59:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7232.796346 secs 2024-07-26T09:59:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:55 smithi097 ceph-mon[32231]: pgmap v4519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:59:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7232.796346 secs 2024-07-26T09:59:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:57 smithi115 ceph-mon[35364]: pgmap v4520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:59:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:57 smithi181 ceph-mon[35869]: pgmap v4520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:59:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:57 smithi097 ceph-mon[32231]: pgmap v4520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:59:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 09:59:59 smithi115 ceph-mon[35364]: pgmap v4521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 09:59:59 smithi181 ceph-mon[35869]: pgmap v4521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 09:59:59 smithi097 ceph-mon[32231]: pgmap v4521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:00:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7237.796492 secs 2024-07-26T10:00:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:00:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7237.796492 secs 2024-07-26T10:00:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:00:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7237.796492 secs 2024-07-26T10:00:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:00:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:01 smithi115 ceph-mon[35364]: pgmap v4522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:00:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:01 smithi181 ceph-mon[35869]: pgmap v4522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:00:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:01 smithi097 ceph-mon[32231]: pgmap v4522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:00:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:03 smithi115 ceph-mon[35364]: pgmap v4523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:03 smithi181 ceph-mon[35869]: pgmap v4523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:03 smithi097 ceph-mon[32231]: pgmap v4523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:05 smithi115 ceph-mon[35364]: pgmap v4524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7242.796627 secs 2024-07-26T10:00:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:05 smithi181 ceph-mon[35869]: pgmap v4524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7242.796627 secs 2024-07-26T10:00:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:05 smithi097 ceph-mon[32231]: pgmap v4524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7242.796627 secs 2024-07-26T10:00:06.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:06.000+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:00:06.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:06.001+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:00:06.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:06.001+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:00:06.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:06.001+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:00:06.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:06.001+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:07 smithi115 ceph-mon[35364]: pgmap v4525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:07 smithi181 ceph-mon[35869]: pgmap v4525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:07 smithi097 ceph-mon[32231]: pgmap v4525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:09 smithi115 ceph-mon[35364]: pgmap v4526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:09.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:09 smithi181 ceph-mon[35869]: pgmap v4526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:09 smithi097 ceph-mon[32231]: pgmap v4526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7247.796773 secs 2024-07-26T10:00:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7247.796773 secs 2024-07-26T10:00:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7247.796773 secs 2024-07-26T10:00:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:11 smithi115 ceph-mon[35364]: pgmap v4527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:11 smithi181 ceph-mon[35869]: pgmap v4527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:11 smithi097 ceph-mon[32231]: pgmap v4527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:13 smithi115 ceph-mon[35364]: pgmap v4528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:13 smithi181 ceph-mon[35869]: pgmap v4528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:13 smithi097 ceph-mon[32231]: pgmap v4528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:15 smithi115 ceph-mon[35364]: pgmap v4529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7252.796917 secs 2024-07-26T10:00:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:15 smithi181 ceph-mon[35869]: pgmap v4529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7252.796917 secs 2024-07-26T10:00:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:15 smithi097 ceph-mon[32231]: pgmap v4529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7252.796917 secs 2024-07-26T10:00:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:17 smithi115 ceph-mon[35364]: pgmap v4530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:17 smithi181 ceph-mon[35869]: pgmap v4530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:17 smithi097 ceph-mon[32231]: pgmap v4530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:19 smithi115 ceph-mon[35364]: pgmap v4531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:19 smithi181 ceph-mon[35869]: pgmap v4531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:19 smithi097 ceph-mon[32231]: pgmap v4531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7257.797060 secs 2024-07-26T10:00:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7257.797060 secs 2024-07-26T10:00:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7257.797060 secs 2024-07-26T10:00:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:21.006+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:21.007+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:21.007+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:21.007+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:00:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:21.007+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:21 smithi115 ceph-mon[35364]: pgmap v4532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:21 smithi181 ceph-mon[35869]: pgmap v4532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:21 smithi097 ceph-mon[32231]: pgmap v4532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:23 smithi115 ceph-mon[35364]: pgmap v4533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:23 smithi181 ceph-mon[35869]: pgmap v4533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:23 smithi097 ceph-mon[32231]: pgmap v4533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:25 smithi115 ceph-mon[35364]: pgmap v4534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7262.797202 secs 2024-07-26T10:00:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:25 smithi181 ceph-mon[35869]: pgmap v4534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7262.797202 secs 2024-07-26T10:00:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:25 smithi097 ceph-mon[32231]: pgmap v4534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7262.797202 secs 2024-07-26T10:00:27.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:27 smithi181 ceph-mon[35869]: pgmap v4535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:27 smithi097 ceph-mon[32231]: pgmap v4535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:27 smithi115 ceph-mon[35364]: pgmap v4535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:29 smithi097 ceph-mon[32231]: pgmap v4536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:29 smithi115 ceph-mon[35364]: pgmap v4536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:29 smithi181 ceph-mon[35869]: pgmap v4536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7267.797350 secs 2024-07-26T10:00:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7267.797350 secs 2024-07-26T10:00:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7267.797350 secs 2024-07-26T10:00:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:31 smithi097 ceph-mon[32231]: pgmap v4537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:31 smithi115 ceph-mon[35364]: pgmap v4537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:31 smithi181 ceph-mon[35869]: pgmap v4537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:33 smithi115 ceph-mon[35364]: pgmap v4538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:33 smithi181 ceph-mon[35869]: pgmap v4538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:33 smithi097 ceph-mon[32231]: pgmap v4538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:35 smithi115 ceph-mon[35364]: pgmap v4539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7272.797486 secs 2024-07-26T10:00:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:35 smithi181 ceph-mon[35869]: pgmap v4539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7272.797486 secs 2024-07-26T10:00:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:35 smithi097 ceph-mon[32231]: pgmap v4539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:00:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7272.797486 secs 2024-07-26T10:00:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:36.013+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:00:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:36.015+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:00:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:36.015+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:00:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:36.015+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:00:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:36.015+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:37 smithi115 ceph-mon[35364]: pgmap v4540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:37 smithi181 ceph-mon[35869]: pgmap v4540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:37 smithi097 ceph-mon[32231]: pgmap v4540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:00:38.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:00:38.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:00:38.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:00:38.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:38.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:38.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: pgmap v4541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:00:38.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: pgmap v4542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:00:38.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:00:38.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: pgmap v4541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: pgmap v4542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:00:38.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:00:39.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:39.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:39.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:39.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:39.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:39.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:39.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: pgmap v4541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: pgmap v4542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:00:39.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:00:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:40 smithi115 ceph-mon[35364]: pgmap v4543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7277.797580 secs 2024-07-26T10:00:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:40 smithi181 ceph-mon[35869]: pgmap v4543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7277.797580 secs 2024-07-26T10:00:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:40 smithi097 ceph-mon[32231]: pgmap v4543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7277.797580 secs 2024-07-26T10:00:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:42 smithi115 ceph-mon[35364]: pgmap v4544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:42 smithi181 ceph-mon[35869]: pgmap v4544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:42 smithi097 ceph-mon[32231]: pgmap v4544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:44 smithi115 ceph-mon[35364]: pgmap v4545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:44 smithi181 ceph-mon[35869]: pgmap v4545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:44 smithi097 ceph-mon[32231]: pgmap v4545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:00:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7282.797752 secs 2024-07-26T10:00:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7282.797752 secs 2024-07-26T10:00:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7282.797752 secs 2024-07-26T10:00:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:46 smithi115 ceph-mon[35364]: pgmap v4546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:46 smithi181 ceph-mon[35869]: pgmap v4546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:46 smithi097 ceph-mon[32231]: pgmap v4546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:48 smithi115 ceph-mon[35364]: pgmap v4547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:00:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:48 smithi181 ceph-mon[35869]: pgmap v4547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:00:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:48 smithi097 ceph-mon[32231]: pgmap v4547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:00:50.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:50 smithi181 ceph-mon[35869]: pgmap v4548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:50.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7287.797850 secs 2024-07-26T10:00:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:50 smithi097 ceph-mon[32231]: pgmap v4548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7287.797850 secs 2024-07-26T10:00:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:51.018+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:00:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:51.019+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:00:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:51.019+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:00:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:51.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:00:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:00:51.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:50 smithi115 ceph-mon[35364]: pgmap v4548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7287.797850 secs 2024-07-26T10:00:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:52 smithi097 ceph-mon[32231]: pgmap v4549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:52 smithi115 ceph-mon[35364]: pgmap v4549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:52 smithi181 ceph-mon[35869]: pgmap v4549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:54 smithi097 ceph-mon[32231]: pgmap v4550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:54 smithi115 ceph-mon[35364]: pgmap v4550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:54 smithi181 ceph-mon[35869]: pgmap v4550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7292.797992 secs 2024-07-26T10:00:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7292.797992 secs 2024-07-26T10:00:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7292.797992 secs 2024-07-26T10:00:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:56 smithi115 ceph-mon[35364]: pgmap v4551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:56 smithi181 ceph-mon[35869]: pgmap v4551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:56 smithi097 ceph-mon[32231]: pgmap v4551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:00:58 smithi097 ceph-mon[32231]: pgmap v4552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:00:58 smithi115 ceph-mon[35364]: pgmap v4552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:00:58 smithi181 ceph-mon[35869]: pgmap v4552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:01.060 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:00 smithi115 ceph-mon[35364]: pgmap v4553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:01:01.060 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7297.798139 secs 2024-07-26T10:01:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:00 smithi181 ceph-mon[35869]: pgmap v4553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:01:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7297.798139 secs 2024-07-26T10:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:00 smithi097 ceph-mon[32231]: pgmap v4553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:01:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7297.798139 secs 2024-07-26T10:01:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:02 smithi115 ceph-mon[35364]: pgmap v4554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:02 smithi181 ceph-mon[35869]: pgmap v4554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:02 smithi097 ceph-mon[32231]: pgmap v4554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:04 smithi115 ceph-mon[35364]: pgmap v4555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:04 smithi181 ceph-mon[35869]: pgmap v4555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:04 smithi097 ceph-mon[32231]: pgmap v4555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:06.025+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:01:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:06.026+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:01:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:06.026+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:01:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:06.026+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:01:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:06.026+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7302.798284 secs 2024-07-26T10:01:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:05 smithi115 ceph-mon[35364]: pgmap v4556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7302.798284 secs 2024-07-26T10:01:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:05 smithi181 ceph-mon[35869]: pgmap v4556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7302.798284 secs 2024-07-26T10:01:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:05 smithi097 ceph-mon[32231]: pgmap v4556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:08.171+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:01:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:08.171+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T10:01:08.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:08.172+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T10:01:08.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:08.172+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:01:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:08 smithi115 ceph-mon[35364]: pgmap v4557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:08 smithi181 ceph-mon[35869]: pgmap v4557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:08 smithi097 ceph-mon[32231]: pgmap v4557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:10 smithi115 ceph-mon[35364]: pgmap v4558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7307.798426 secs 2024-07-26T10:01:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:10 smithi181 ceph-mon[35869]: pgmap v4558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7307.798426 secs 2024-07-26T10:01:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:10 smithi097 ceph-mon[32231]: pgmap v4558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7307.798426 secs 2024-07-26T10:01:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:12 smithi115 ceph-mon[35364]: pgmap v4559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:12 smithi181 ceph-mon[35869]: pgmap v4559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:12 smithi097 ceph-mon[32231]: pgmap v4559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:14 smithi115 ceph-mon[35364]: pgmap v4560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:14 smithi181 ceph-mon[35869]: pgmap v4560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:14 smithi097 ceph-mon[32231]: pgmap v4560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7312.798562 secs 2024-07-26T10:01:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7312.798562 secs 2024-07-26T10:01:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7312.798562 secs 2024-07-26T10:01:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:16 smithi097 ceph-mon[32231]: pgmap v4561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:16 smithi115 ceph-mon[35364]: pgmap v4561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:17.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:16 smithi181 ceph-mon[35869]: pgmap v4561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:18 smithi097 ceph-mon[32231]: pgmap v4562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:18 smithi115 ceph-mon[35364]: pgmap v4562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:18 smithi181 ceph-mon[35869]: pgmap v4562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:21.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:21.037+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:01:21.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:21.038+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:01:21.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:21.038+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:01:21.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:21.038+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:01:21.041 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:21.038+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:20 smithi115 ceph-mon[35364]: pgmap v4563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7317.798707 secs 2024-07-26T10:01:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:20 smithi181 ceph-mon[35869]: pgmap v4563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7317.798707 secs 2024-07-26T10:01:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:20 smithi097 ceph-mon[32231]: pgmap v4563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7317.798707 secs 2024-07-26T10:01:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:22 smithi115 ceph-mon[35364]: pgmap v4564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:22 smithi181 ceph-mon[35869]: pgmap v4564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:22 smithi097 ceph-mon[32231]: pgmap v4564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:24 smithi115 ceph-mon[35364]: pgmap v4565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:24 smithi181 ceph-mon[35869]: pgmap v4565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:24 smithi097 ceph-mon[32231]: pgmap v4565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7322.798856 secs 2024-07-26T10:01:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7322.798856 secs 2024-07-26T10:01:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7322.798856 secs 2024-07-26T10:01:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:26 smithi115 ceph-mon[35364]: pgmap v4566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:26 smithi181 ceph-mon[35869]: pgmap v4566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:26 smithi097 ceph-mon[32231]: pgmap v4566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:28 smithi115 ceph-mon[35364]: pgmap v4567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:28 smithi181 ceph-mon[35869]: pgmap v4567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:28 smithi097 ceph-mon[32231]: pgmap v4567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:29 smithi115 ceph-mon[35364]: pgmap v4568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:29 smithi181 ceph-mon[35869]: pgmap v4568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:29 smithi097 ceph-mon[32231]: pgmap v4568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7327.798953 secs 2024-07-26T10:01:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7327.798953 secs 2024-07-26T10:01:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7327.798953 secs 2024-07-26T10:01:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:31 smithi115 ceph-mon[35364]: pgmap v4569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:31 smithi181 ceph-mon[35869]: pgmap v4569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:31 smithi097 ceph-mon[32231]: pgmap v4569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:34 smithi115 ceph-mon[35364]: pgmap v4570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:34 smithi181 ceph-mon[35869]: pgmap v4570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:34 smithi097 ceph-mon[32231]: pgmap v4570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7332.799107 secs 2024-07-26T10:01:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7332.799107 secs 2024-07-26T10:01:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:36.043+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:01:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:36.044+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:01:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:36.044+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:01:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:36.045+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:01:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:36.045+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7332.799107 secs 2024-07-26T10:01:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:36 smithi181 ceph-mon[35869]: pgmap v4571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:36 smithi097 ceph-mon[32231]: pgmap v4571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:36 smithi115 ceph-mon[35364]: pgmap v4571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:38 smithi097 ceph-mon[32231]: pgmap v4572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:38 smithi115 ceph-mon[35364]: pgmap v4572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:38 smithi181 ceph-mon[35869]: pgmap v4572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:39.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:40.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:40.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:01:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:40.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:40.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:01:40.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:01:40.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:40.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:40.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:01:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:40 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:01:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:40 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:01:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:40 smithi115 ceph-mon[35364]: pgmap v4573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:01:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:40 smithi115 ceph-mon[35364]: pgmap v4574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:01:41.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7337.799290 secs 2024-07-26T10:01:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:40 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:01:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:40 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:01:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:40 smithi181 ceph-mon[35869]: pgmap v4573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:01:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:40 smithi181 ceph-mon[35869]: pgmap v4574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:01:41.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7337.799290 secs 2024-07-26T10:01:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:40 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:01:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:40 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:01:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:40 smithi097 ceph-mon[32231]: pgmap v4573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:01:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:40 smithi097 ceph-mon[32231]: pgmap v4574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:01:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7337.799290 secs 2024-07-26T10:01:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:42 smithi115 ceph-mon[35364]: pgmap v4575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:42 smithi181 ceph-mon[35869]: pgmap v4575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:42 smithi097 ceph-mon[32231]: pgmap v4575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:45.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:44 smithi115 ceph-mon[35364]: pgmap v4576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:45.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:44 smithi181 ceph-mon[35869]: pgmap v4576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:45.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:44 smithi097 ceph-mon[32231]: pgmap v4576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7342.799389 secs 2024-07-26T10:01:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7342.799389 secs 2024-07-26T10:01:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7342.799389 secs 2024-07-26T10:01:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:46 smithi115 ceph-mon[35364]: pgmap v4577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:46 smithi181 ceph-mon[35869]: pgmap v4577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:46 smithi097 ceph-mon[32231]: pgmap v4577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:48 smithi115 ceph-mon[35364]: pgmap v4578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:01:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:48 smithi181 ceph-mon[35869]: pgmap v4578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:01:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:48 smithi097 ceph-mon[32231]: pgmap v4578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:01:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:49 smithi115 ceph-mon[35364]: pgmap v4579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:01:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:49 smithi181 ceph-mon[35869]: pgmap v4579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:01:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:49 smithi097 ceph-mon[32231]: pgmap v4579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:01:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:51.056+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:01:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:51.058+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:01:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:51.058+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:01:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:51.058+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:01:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:51.058+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7347.799525 secs 2024-07-26T10:01:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7347.799525 secs 2024-07-26T10:01:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7347.799525 secs 2024-07-26T10:01:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:51 smithi115 ceph-mon[35364]: pgmap v4580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:01:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:51 smithi181 ceph-mon[35869]: pgmap v4580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:01:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:51 smithi097 ceph-mon[32231]: pgmap v4580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:01:53.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:53.179+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:01:53.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:53.179+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:01:53.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:53.180+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:01:53.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:01:53.180+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:01:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:54 smithi115 ceph-mon[35364]: pgmap v4581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:54 smithi181 ceph-mon[35869]: pgmap v4581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:54 smithi097 ceph-mon[32231]: pgmap v4581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7352.799664 secs 2024-07-26T10:01:55.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7352.799664 secs 2024-07-26T10:01:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7352.799664 secs 2024-07-26T10:01:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:56 smithi115 ceph-mon[35364]: pgmap v4582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:56 smithi181 ceph-mon[35869]: pgmap v4582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:56 smithi097 ceph-mon[32231]: pgmap v4582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:01:58 smithi115 ceph-mon[35364]: pgmap v4583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:01:58 smithi181 ceph-mon[35869]: pgmap v4583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:01:58 smithi097 ceph-mon[32231]: pgmap v4583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:00 smithi115 ceph-mon[35364]: pgmap v4584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:00.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7357.799800 secs 2024-07-26T10:02:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:00 smithi181 ceph-mon[35869]: pgmap v4584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7357.799800 secs 2024-07-26T10:02:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:00 smithi097 ceph-mon[32231]: pgmap v4584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7357.799800 secs 2024-07-26T10:02:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:02 smithi115 ceph-mon[35364]: pgmap v4585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:02 smithi181 ceph-mon[35869]: pgmap v4585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:02 smithi097 ceph-mon[32231]: pgmap v4585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:04 smithi115 ceph-mon[35364]: pgmap v4586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:04 smithi181 ceph-mon[35869]: pgmap v4586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:04 smithi097 ceph-mon[32231]: pgmap v4586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7362.799937 secs 2024-07-26T10:02:05.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7362.799937 secs 2024-07-26T10:02:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7362.799937 secs 2024-07-26T10:02:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:06.059+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:02:06.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:06.061+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:02:06.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:06.061+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:02:06.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:06.061+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:02:06.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:06.061+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:06 smithi115 ceph-mon[35364]: pgmap v4587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:06.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:06 smithi181 ceph-mon[35869]: pgmap v4587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:06 smithi097 ceph-mon[32231]: pgmap v4587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:08 smithi115 ceph-mon[35364]: pgmap v4588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:08 smithi181 ceph-mon[35869]: pgmap v4588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:08 smithi097 ceph-mon[32231]: pgmap v4588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:10 smithi115 ceph-mon[35364]: pgmap v4589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7367.800079 secs 2024-07-26T10:02:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:10 smithi181 ceph-mon[35869]: pgmap v4589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7367.800079 secs 2024-07-26T10:02:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:10 smithi097 ceph-mon[32231]: pgmap v4589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7367.800079 secs 2024-07-26T10:02:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:12 smithi115 ceph-mon[35364]: pgmap v4590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:12 smithi181 ceph-mon[35869]: pgmap v4590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:12 smithi097 ceph-mon[32231]: pgmap v4590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:14 smithi181 ceph-mon[35869]: pgmap v4591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:14 smithi097 ceph-mon[32231]: pgmap v4591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:14 smithi115 ceph-mon[35364]: pgmap v4591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7372.800221 secs 2024-07-26T10:02:15.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7372.800221 secs 2024-07-26T10:02:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7372.800221 secs 2024-07-26T10:02:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:16 smithi097 ceph-mon[32231]: pgmap v4592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:16.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:16 smithi115 ceph-mon[35364]: pgmap v4592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:16 smithi181 ceph-mon[35869]: pgmap v4592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:18 smithi097 ceph-mon[32231]: pgmap v4593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:18 smithi115 ceph-mon[35364]: pgmap v4593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:18 smithi181 ceph-mon[35869]: pgmap v4593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:20 smithi115 ceph-mon[35364]: pgmap v4594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7377.800355 secs 2024-07-26T10:02:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:20 smithi181 ceph-mon[35869]: pgmap v4594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7377.800355 secs 2024-07-26T10:02:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:20 smithi097 ceph-mon[32231]: pgmap v4594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:02:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7377.800355 secs 2024-07-26T10:02:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:21.073+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:02:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:21.074+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:02:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:21.074+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:02:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:21.074+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:02:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:21.075+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:22 smithi115 ceph-mon[35364]: pgmap v4595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:22 smithi181 ceph-mon[35869]: pgmap v4595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:22 smithi097 ceph-mon[32231]: pgmap v4595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:24 smithi115 ceph-mon[35364]: pgmap v4596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:24 smithi181 ceph-mon[35869]: pgmap v4596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:24 smithi097 ceph-mon[32231]: pgmap v4596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7382.800466 secs 2024-07-26T10:02:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7382.800466 secs 2024-07-26T10:02:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7382.800466 secs 2024-07-26T10:02:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:26 smithi115 ceph-mon[35364]: pgmap v4597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:02:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:26 smithi181 ceph-mon[35869]: pgmap v4597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:02:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:26 smithi097 ceph-mon[32231]: pgmap v4597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:02:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:28 smithi115 ceph-mon[35364]: pgmap v4598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:02:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:28 smithi181 ceph-mon[35869]: pgmap v4598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:02:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:28 smithi097 ceph-mon[32231]: pgmap v4598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:02:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:30 smithi115 ceph-mon[35364]: pgmap v4599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:02:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7387.800606 secs 2024-07-26T10:02:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:30 smithi181 ceph-mon[35869]: pgmap v4599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:02:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7387.800606 secs 2024-07-26T10:02:30.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:30 smithi097 ceph-mon[32231]: pgmap v4599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:02:30.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7387.800606 secs 2024-07-26T10:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:32 smithi115 ceph-mon[35364]: pgmap v4600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:32 smithi181 ceph-mon[35869]: pgmap v4600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:32 smithi097 ceph-mon[32231]: pgmap v4600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:34 smithi115 ceph-mon[35364]: pgmap v4601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:34 smithi181 ceph-mon[35869]: pgmap v4601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:34 smithi097 ceph-mon[32231]: pgmap v4601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7392.800754 secs 2024-07-26T10:02:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7392.800754 secs 2024-07-26T10:02:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7392.800754 secs 2024-07-26T10:02:36.082 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:36.080+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:02:36.083 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:36.081+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:02:36.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:36.081+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:02:36.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:36.081+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:02:36.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:36.081+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:36 smithi115 ceph-mon[35364]: pgmap v4602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:36 smithi181 ceph-mon[35869]: pgmap v4602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:36 smithi097 ceph-mon[32231]: pgmap v4602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:38 smithi115 ceph-mon[35364]: pgmap v4603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:38 smithi181 ceph-mon[35869]: pgmap v4603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:38 smithi097 ceph-mon[32231]: pgmap v4603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:40 smithi115 ceph-mon[35364]: pgmap v4604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:02:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7397.800882 secs 2024-07-26T10:02:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:40.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:40 smithi181 ceph-mon[35869]: pgmap v4604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:02:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:40.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7397.800882 secs 2024-07-26T10:02:40.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:40.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:40 smithi097 ceph-mon[32231]: pgmap v4604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:02:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7397.800882 secs 2024-07-26T10:02:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:41.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:41.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:02:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:42 smithi115 ceph-mon[35364]: pgmap v4605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:42 smithi115 ceph-mon[35364]: pgmap v4606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:42 smithi181 ceph-mon[35869]: pgmap v4605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:42 smithi181 ceph-mon[35869]: pgmap v4606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:42 smithi097 ceph-mon[32231]: pgmap v4605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:02:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:42 smithi097 ceph-mon[32231]: pgmap v4606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:44 smithi115 ceph-mon[35364]: pgmap v4607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:44 smithi181 ceph-mon[35869]: pgmap v4607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:44 smithi097 ceph-mon[32231]: pgmap v4607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:45.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7402.801016 secs 2024-07-26T10:02:45.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7402.801016 secs 2024-07-26T10:02:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7402.801016 secs 2024-07-26T10:02:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:46 smithi115 ceph-mon[35364]: pgmap v4608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:46 smithi181 ceph-mon[35869]: pgmap v4608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:46 smithi097 ceph-mon[32231]: pgmap v4608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:48 smithi115 ceph-mon[35364]: pgmap v4609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:48 smithi181 ceph-mon[35869]: pgmap v4609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:48 smithi097 ceph-mon[32231]: pgmap v4609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:02:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:50 smithi115 ceph-mon[35364]: pgmap v4610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:02:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7407.801176 secs 2024-07-26T10:02:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:50 smithi181 ceph-mon[35869]: pgmap v4610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:02:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7407.801176 secs 2024-07-26T10:02:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:50 smithi097 ceph-mon[32231]: pgmap v4610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:02:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7407.801176 secs 2024-07-26T10:02:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:51.092+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:02:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:51.093+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:02:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:51.093+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:02:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:51.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:02:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:02:51.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:52 smithi115 ceph-mon[35364]: pgmap v4611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:02:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:52 smithi181 ceph-mon[35869]: pgmap v4611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:02:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:52 smithi097 ceph-mon[32231]: pgmap v4611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:02:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:54 smithi115 ceph-mon[35364]: pgmap v4612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:54 smithi181 ceph-mon[35869]: pgmap v4612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:54 smithi097 ceph-mon[32231]: pgmap v4612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7412.801294 secs 2024-07-26T10:02:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7412.801294 secs 2024-07-26T10:02:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7412.801294 secs 2024-07-26T10:02:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:56 smithi115 ceph-mon[35364]: pgmap v4613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:56 smithi181 ceph-mon[35869]: pgmap v4613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:56 smithi097 ceph-mon[32231]: pgmap v4613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:02:58 smithi115 ceph-mon[35364]: pgmap v4614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:02:58 smithi181 ceph-mon[35869]: pgmap v4614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:02:58 smithi097 ceph-mon[32231]: pgmap v4614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:00 smithi115 ceph-mon[35364]: pgmap v4615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7417.801425 secs 2024-07-26T10:03:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:00 smithi181 ceph-mon[35869]: pgmap v4615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7417.801425 secs 2024-07-26T10:03:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:00 smithi097 ceph-mon[32231]: pgmap v4615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7417.801425 secs 2024-07-26T10:03:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:02 smithi115 ceph-mon[35364]: pgmap v4616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:02 smithi181 ceph-mon[35869]: pgmap v4616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:02 smithi097 ceph-mon[32231]: pgmap v4616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:04 smithi115 ceph-mon[35364]: pgmap v4617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:04 smithi181 ceph-mon[35869]: pgmap v4617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:04 smithi097 ceph-mon[32231]: pgmap v4617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7422.801635 secs 2024-07-26T10:03:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7422.801635 secs 2024-07-26T10:03:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7422.801635 secs 2024-07-26T10:03:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:06.099+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:03:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:06.101+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:03:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:06.101+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:03:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:06.101+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:03:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:06.101+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:06 smithi115 ceph-mon[35364]: pgmap v4618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:06 smithi181 ceph-mon[35869]: pgmap v4618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:06 smithi097 ceph-mon[32231]: pgmap v4618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:08 smithi181 ceph-mon[35869]: pgmap v4619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:03:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:08 smithi097 ceph-mon[32231]: pgmap v4619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:03:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:08 smithi115 ceph-mon[35364]: pgmap v4619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:03:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:10 smithi097 ceph-mon[32231]: pgmap v4620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:03:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7427.801777 secs 2024-07-26T10:03:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:10 smithi115 ceph-mon[35364]: pgmap v4620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:03:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7427.801777 secs 2024-07-26T10:03:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:10 smithi181 ceph-mon[35869]: pgmap v4620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:03:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7427.801777 secs 2024-07-26T10:03:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:12 smithi097 ceph-mon[32231]: pgmap v4621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:12 smithi115 ceph-mon[35364]: pgmap v4621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:12 smithi181 ceph-mon[35869]: pgmap v4621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:14 smithi115 ceph-mon[35364]: pgmap v4622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:03:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:14 smithi181 ceph-mon[35869]: pgmap v4622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:03:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:14 smithi097 ceph-mon[32231]: pgmap v4622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:03:15.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7432.801923 secs 2024-07-26T10:03:15.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7432.801923 secs 2024-07-26T10:03:15.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7432.801923 secs 2024-07-26T10:03:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:16 smithi115 ceph-mon[35364]: pgmap v4623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:16 smithi181 ceph-mon[35869]: pgmap v4623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:16 smithi097 ceph-mon[32231]: pgmap v4623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:18 smithi115 ceph-mon[35364]: pgmap v4624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:18 smithi181 ceph-mon[35869]: pgmap v4624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:18 smithi097 ceph-mon[32231]: pgmap v4624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:20 smithi115 ceph-mon[35364]: pgmap v4625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7437.802066 secs 2024-07-26T10:03:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:20 smithi181 ceph-mon[35869]: pgmap v4625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7437.802066 secs 2024-07-26T10:03:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:20 smithi097 ceph-mon[32231]: pgmap v4625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7437.802066 secs 2024-07-26T10:03:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:21.113+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:03:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:21.113+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:03:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:21.113+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:03:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:21.113+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:03:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:21.113+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:22 smithi115 ceph-mon[35364]: pgmap v4626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:22 smithi181 ceph-mon[35869]: pgmap v4626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:22 smithi097 ceph-mon[32231]: pgmap v4626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:24 smithi115 ceph-mon[35364]: pgmap v4627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:24 smithi181 ceph-mon[35869]: pgmap v4627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:24 smithi097 ceph-mon[32231]: pgmap v4627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7442.802213 secs 2024-07-26T10:03:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7442.802213 secs 2024-07-26T10:03:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7442.802213 secs 2024-07-26T10:03:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:26 smithi115 ceph-mon[35364]: pgmap v4628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:26 smithi181 ceph-mon[35869]: pgmap v4628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:26 smithi097 ceph-mon[32231]: pgmap v4628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:28 smithi115 ceph-mon[35364]: pgmap v4629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:28 smithi181 ceph-mon[35869]: pgmap v4629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:28 smithi097 ceph-mon[32231]: pgmap v4629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:30 smithi115 ceph-mon[35364]: pgmap v4630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7447.802356 secs 2024-07-26T10:03:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:30 smithi181 ceph-mon[35869]: pgmap v4630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7447.802356 secs 2024-07-26T10:03:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:30 smithi097 ceph-mon[32231]: pgmap v4630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7447.802356 secs 2024-07-26T10:03:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:32 smithi115 ceph-mon[35364]: pgmap v4631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:32 smithi181 ceph-mon[35869]: pgmap v4631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:32 smithi097 ceph-mon[32231]: pgmap v4631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:34 smithi115 ceph-mon[35364]: pgmap v4632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:34 smithi181 ceph-mon[35869]: pgmap v4632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:34 smithi097 ceph-mon[32231]: pgmap v4632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7452.802498 secs 2024-07-26T10:03:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7452.802498 secs 2024-07-26T10:03:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7452.802498 secs 2024-07-26T10:03:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:36.119+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:03:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:36.120+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:03:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:36.120+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:03:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:36.120+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:03:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:36.120+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:36 smithi115 ceph-mon[35364]: pgmap v4633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:36 smithi181 ceph-mon[35869]: pgmap v4633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:36 smithi097 ceph-mon[32231]: pgmap v4633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:38 smithi115 ceph-mon[35364]: pgmap v4634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:38 smithi181 ceph-mon[35869]: pgmap v4634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:38 smithi097 ceph-mon[32231]: pgmap v4634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:40 smithi115 ceph-mon[35364]: pgmap v4635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7457.802640 secs 2024-07-26T10:03:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:40 smithi181 ceph-mon[35869]: pgmap v4635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:40.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7457.802640 secs 2024-07-26T10:03:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:40 smithi097 ceph-mon[32231]: pgmap v4635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:03:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7457.802640 secs 2024-07-26T10:03:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:42 smithi115 ceph-mon[35364]: pgmap v4636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:42 smithi181 ceph-mon[35869]: pgmap v4636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:42.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:42.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:42.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:42 smithi097 ceph-mon[32231]: pgmap v4636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:03:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:03:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:44 smithi115 ceph-mon[35364]: pgmap v4637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:44 smithi115 ceph-mon[35364]: pgmap v4638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:03:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:44 smithi181 ceph-mon[35869]: pgmap v4637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:44.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:44 smithi181 ceph-mon[35869]: pgmap v4638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:03:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:44 smithi097 ceph-mon[32231]: pgmap v4637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:44 smithi097 ceph-mon[32231]: pgmap v4638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:03:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7462.802783 secs 2024-07-26T10:03:45.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7462.802783 secs 2024-07-26T10:03:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7462.802783 secs 2024-07-26T10:03:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:46 smithi115 ceph-mon[35364]: pgmap v4639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:46 smithi181 ceph-mon[35869]: pgmap v4639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:46 smithi097 ceph-mon[32231]: pgmap v4639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:48 smithi115 ceph-mon[35364]: pgmap v4640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:48 smithi181 ceph-mon[35869]: pgmap v4640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:48 smithi097 ceph-mon[32231]: pgmap v4640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:50 smithi115 ceph-mon[35364]: pgmap v4641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7467.802928 secs 2024-07-26T10:03:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:50 smithi181 ceph-mon[35869]: pgmap v4641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7467.802928 secs 2024-07-26T10:03:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:50 smithi097 ceph-mon[32231]: pgmap v4641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7467.802928 secs 2024-07-26T10:03:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:51.131+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:03:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:51.133+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:03:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:51.133+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:03:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:51.133+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:03:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:03:51.133+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:52 smithi115 ceph-mon[35364]: pgmap v4642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:52 smithi181 ceph-mon[35869]: pgmap v4642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:52 smithi097 ceph-mon[32231]: pgmap v4642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:54 smithi115 ceph-mon[35364]: pgmap v4643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:54 smithi181 ceph-mon[35869]: pgmap v4643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:54 smithi097 ceph-mon[32231]: pgmap v4643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:03:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7472.803072 secs 2024-07-26T10:03:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7472.803072 secs 2024-07-26T10:03:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7472.803072 secs 2024-07-26T10:03:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:56 smithi181 ceph-mon[35869]: pgmap v4644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:56 smithi097 ceph-mon[32231]: pgmap v4644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:56 smithi115 ceph-mon[35364]: pgmap v4644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:03:58 smithi181 ceph-mon[35869]: pgmap v4645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:03:58 smithi097 ceph-mon[32231]: pgmap v4645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:03:58 smithi115 ceph-mon[35364]: pgmap v4645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:00 smithi097 ceph-mon[32231]: pgmap v4646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7477.803156 secs 2024-07-26T10:04:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:00 smithi115 ceph-mon[35364]: pgmap v4646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7477.803156 secs 2024-07-26T10:04:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:00 smithi181 ceph-mon[35869]: pgmap v4646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7477.803156 secs 2024-07-26T10:04:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:02 smithi097 ceph-mon[32231]: pgmap v4647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:02 smithi115 ceph-mon[35364]: pgmap v4647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:02 smithi181 ceph-mon[35869]: pgmap v4647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:04 smithi097 ceph-mon[32231]: pgmap v4648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:04 smithi115 ceph-mon[35364]: pgmap v4648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:04 smithi181 ceph-mon[35869]: pgmap v4648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:06.141 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:06.139+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:04:06.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:06.140+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:04:06.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:06.140+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:04:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:06.140+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:04:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:06.140+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7482.803294 secs 2024-07-26T10:04:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7482.803294 secs 2024-07-26T10:04:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7482.803294 secs 2024-07-26T10:04:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:06 smithi115 ceph-mon[35364]: pgmap v4649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:04:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:06 smithi181 ceph-mon[35869]: pgmap v4649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:04:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:06 smithi097 ceph-mon[32231]: pgmap v4649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:04:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:08.203+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:04:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:08.203+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:04:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:08.204+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:04:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:08.204+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:04:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:08 smithi115 ceph-mon[35364]: pgmap v4650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:04:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:08 smithi181 ceph-mon[35869]: pgmap v4650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:04:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:08 smithi097 ceph-mon[32231]: pgmap v4650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:04:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:10 smithi115 ceph-mon[35364]: pgmap v4651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:04:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7487.803394 secs 2024-07-26T10:04:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:10 smithi181 ceph-mon[35869]: pgmap v4651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:04:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7487.803394 secs 2024-07-26T10:04:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:10 smithi097 ceph-mon[32231]: pgmap v4651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:04:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7487.803394 secs 2024-07-26T10:04:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:12 smithi115 ceph-mon[35364]: pgmap v4652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:12 smithi181 ceph-mon[35869]: pgmap v4652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:12 smithi097 ceph-mon[32231]: pgmap v4652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:15.056 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:14 smithi097 ceph-mon[32231]: pgmap v4653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:04:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:14 smithi115 ceph-mon[35364]: pgmap v4653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:04:15.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:14 smithi181 ceph-mon[35869]: pgmap v4653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:04:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7492.803491 secs 2024-07-26T10:04:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7492.803491 secs 2024-07-26T10:04:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7492.803491 secs 2024-07-26T10:04:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:16 smithi115 ceph-mon[35364]: pgmap v4654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:04:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:16 smithi181 ceph-mon[35869]: pgmap v4654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:04:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:16 smithi097 ceph-mon[32231]: pgmap v4654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:04:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:18 smithi115 ceph-mon[35364]: pgmap v4655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:18 smithi181 ceph-mon[35869]: pgmap v4655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:18 smithi097 ceph-mon[32231]: pgmap v4655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:20 smithi115 ceph-mon[35364]: pgmap v4656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:20.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7497.803622 secs 2024-07-26T10:04:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:20 smithi181 ceph-mon[35869]: pgmap v4656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7497.803622 secs 2024-07-26T10:04:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:20 smithi097 ceph-mon[32231]: pgmap v4656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7497.803622 secs 2024-07-26T10:04:21.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:21.151+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:04:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:21.152+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:04:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:21.152+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:04:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:21.152+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:04:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:21.152+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:22 smithi115 ceph-mon[35364]: pgmap v4657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:22 smithi181 ceph-mon[35869]: pgmap v4657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:22 smithi097 ceph-mon[32231]: pgmap v4657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:24 smithi115 ceph-mon[35364]: pgmap v4658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:24 smithi181 ceph-mon[35869]: pgmap v4658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:24 smithi097 ceph-mon[32231]: pgmap v4658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7502.803770 secs 2024-07-26T10:04:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7502.803770 secs 2024-07-26T10:04:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7502.803770 secs 2024-07-26T10:04:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:26 smithi115 ceph-mon[35364]: pgmap v4659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:26 smithi181 ceph-mon[35869]: pgmap v4659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:26 smithi097 ceph-mon[32231]: pgmap v4659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:28 smithi115 ceph-mon[35364]: pgmap v4660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:28 smithi181 ceph-mon[35869]: pgmap v4660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:28 smithi097 ceph-mon[32231]: pgmap v4660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:30 smithi115 ceph-mon[35364]: pgmap v4661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7507.803911 secs 2024-07-26T10:04:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:30 smithi181 ceph-mon[35869]: pgmap v4661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7507.803911 secs 2024-07-26T10:04:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:30 smithi097 ceph-mon[32231]: pgmap v4661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7507.803911 secs 2024-07-26T10:04:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:32 smithi115 ceph-mon[35364]: pgmap v4662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:32 smithi181 ceph-mon[35869]: pgmap v4662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:32 smithi097 ceph-mon[32231]: pgmap v4662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:34 smithi115 ceph-mon[35364]: pgmap v4663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:34 smithi181 ceph-mon[35869]: pgmap v4663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:34 smithi097 ceph-mon[32231]: pgmap v4663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7512.804094 secs 2024-07-26T10:04:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7512.804094 secs 2024-07-26T10:04:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7512.804094 secs 2024-07-26T10:04:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:36.158+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:04:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:36.159+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:04:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:36.159+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:04:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:36.159+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:04:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:36.159+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:36 smithi115 ceph-mon[35364]: pgmap v4664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:36 smithi181 ceph-mon[35869]: pgmap v4664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:36 smithi097 ceph-mon[32231]: pgmap v4664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:38 smithi115 ceph-mon[35364]: pgmap v4665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:38 smithi181 ceph-mon[35869]: pgmap v4665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:38 smithi097 ceph-mon[32231]: pgmap v4665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:40 smithi115 ceph-mon[35364]: pgmap v4666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7517.804182 secs 2024-07-26T10:04:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:40 smithi181 ceph-mon[35869]: pgmap v4666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7517.804182 secs 2024-07-26T10:04:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:40 smithi097 ceph-mon[32231]: pgmap v4666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7517.804182 secs 2024-07-26T10:04:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:42 smithi115 ceph-mon[35364]: pgmap v4667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:42 smithi181 ceph-mon[35869]: pgmap v4667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:42 smithi097 ceph-mon[32231]: pgmap v4667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:44.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:44 smithi181 ceph-mon[35869]: pgmap v4668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:44 smithi097 ceph-mon[32231]: pgmap v4668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:44 smithi115 ceph-mon[35364]: pgmap v4668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7522.804368 secs 2024-07-26T10:04:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7522.804368 secs 2024-07-26T10:04:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7522.804368 secs 2024-07-26T10:04:46.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:46 smithi115 ceph-mon[35364]: pgmap v4669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:46.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:46.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:46.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:46.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:46 smithi181 ceph-mon[35869]: pgmap v4669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:46 smithi097 ceph-mon[32231]: pgmap v4669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:47 smithi115 ceph-mon[35364]: pgmap v4670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:47 smithi181 ceph-mon[35869]: pgmap v4670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:47 smithi097 ceph-mon[32231]: pgmap v4670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:48 smithi115 ceph-mon[35364]: pgmap v4671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:48 smithi115 ceph-mon[35364]: pgmap v4672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:04:49.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:48 smithi181 ceph-mon[35869]: pgmap v4671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:48 smithi181 ceph-mon[35869]: pgmap v4672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:04:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:48 smithi097 ceph-mon[32231]: pgmap v4671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:49.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:48 smithi097 ceph-mon[32231]: pgmap v4672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:04:49.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:04:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:51.163+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:04:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:51.164+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:04:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:51.164+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:04:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:51.164+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:04:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:04:51.164+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7527.804455 secs 2024-07-26T10:04:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:50 smithi115 ceph-mon[35364]: pgmap v4673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T10:04:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7527.804455 secs 2024-07-26T10:04:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:50 smithi181 ceph-mon[35869]: pgmap v4673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T10:04:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7527.804455 secs 2024-07-26T10:04:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:50 smithi097 ceph-mon[32231]: pgmap v4673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T10:04:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:52 smithi115 ceph-mon[35364]: pgmap v4674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:04:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:52 smithi181 ceph-mon[35869]: pgmap v4674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:04:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:52 smithi097 ceph-mon[32231]: pgmap v4674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:04:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:54 smithi115 ceph-mon[35364]: pgmap v4675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:04:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:54 smithi181 ceph-mon[35869]: pgmap v4675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:04:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:54 smithi097 ceph-mon[32231]: pgmap v4675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:04:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7532.804590 secs 2024-07-26T10:04:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7532.804590 secs 2024-07-26T10:04:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7532.804590 secs 2024-07-26T10:04:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:56 smithi115 ceph-mon[35364]: pgmap v4676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:04:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:56 smithi181 ceph-mon[35869]: pgmap v4676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:04:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:56 smithi097 ceph-mon[32231]: pgmap v4676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:04:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:04:58 smithi115 ceph-mon[35364]: pgmap v4677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:04:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:04:58 smithi181 ceph-mon[35869]: pgmap v4677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:04:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:04:58 smithi097 ceph-mon[32231]: pgmap v4677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:05:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7537.804729 secs 2024-07-26T10:05:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:00 smithi115 ceph-mon[35364]: pgmap v4678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7537.804729 secs 2024-07-26T10:05:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:00 smithi181 ceph-mon[35869]: pgmap v4678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7537.804729 secs 2024-07-26T10:05:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:00 smithi097 ceph-mon[32231]: pgmap v4678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:02 smithi115 ceph-mon[35364]: pgmap v4679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:02 smithi181 ceph-mon[35869]: pgmap v4679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:02 smithi097 ceph-mon[32231]: pgmap v4679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:04 smithi115 ceph-mon[35364]: pgmap v4680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:04 smithi181 ceph-mon[35869]: pgmap v4680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:04 smithi097 ceph-mon[32231]: pgmap v4680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:06.170+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:05:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:06.171+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:05:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:06.171+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:05:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:06.171+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:05:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:06.171+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7542.804886 secs 2024-07-26T10:05:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7542.804886 secs 2024-07-26T10:05:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7542.804886 secs 2024-07-26T10:05:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:06 smithi115 ceph-mon[35364]: pgmap v4681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:06 smithi181 ceph-mon[35869]: pgmap v4681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:06 smithi097 ceph-mon[32231]: pgmap v4681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:08 smithi115 ceph-mon[35364]: pgmap v4682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:08 smithi181 ceph-mon[35869]: pgmap v4682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:08 smithi097 ceph-mon[32231]: pgmap v4682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:11.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7547.805043 secs 2024-07-26T10:05:11.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:10 smithi181 ceph-mon[35869]: pgmap v4683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7547.805043 secs 2024-07-26T10:05:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:10 smithi097 ceph-mon[32231]: pgmap v4683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7547.805043 secs 2024-07-26T10:05:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:10 smithi115 ceph-mon[35364]: pgmap v4683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:13.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:12 smithi181 ceph-mon[35869]: pgmap v4684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:12 smithi097 ceph-mon[32231]: pgmap v4684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:12 smithi115 ceph-mon[35364]: pgmap v4684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:15.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:14 smithi097 ceph-mon[32231]: pgmap v4685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:15.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:14 smithi115 ceph-mon[35364]: pgmap v4685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:15.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:14 smithi181 ceph-mon[35869]: pgmap v4685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7552.805184 secs 2024-07-26T10:05:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7552.805184 secs 2024-07-26T10:05:16.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7552.805184 secs 2024-07-26T10:05:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:17 smithi115 ceph-mon[35364]: pgmap v4686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:17 smithi181 ceph-mon[35869]: pgmap v4686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:17 smithi097 ceph-mon[32231]: pgmap v4686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:19 smithi115 ceph-mon[35364]: pgmap v4687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:19 smithi181 ceph-mon[35869]: pgmap v4687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:19 smithi097 ceph-mon[32231]: pgmap v4687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:21.181+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:05:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:21.182+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:05:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:21.182+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:05:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:21.182+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:05:21.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:21.182+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7557.805329 secs 2024-07-26T10:05:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:21 smithi115 ceph-mon[35364]: pgmap v4688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7557.805329 secs 2024-07-26T10:05:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:21 smithi181 ceph-mon[35869]: pgmap v4688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7557.805329 secs 2024-07-26T10:05:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:21 smithi097 ceph-mon[32231]: pgmap v4688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:23 smithi115 ceph-mon[35364]: pgmap v4689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:23 smithi181 ceph-mon[35869]: pgmap v4689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:23 smithi097 ceph-mon[32231]: pgmap v4689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:25 smithi115 ceph-mon[35364]: pgmap v4690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:25 smithi181 ceph-mon[35869]: pgmap v4690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:25 smithi097 ceph-mon[32231]: pgmap v4690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7562.805473 secs 2024-07-26T10:05:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7562.805473 secs 2024-07-26T10:05:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7562.805473 secs 2024-07-26T10:05:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:27 smithi115 ceph-mon[35364]: pgmap v4691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:27 smithi181 ceph-mon[35869]: pgmap v4691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:27 smithi097 ceph-mon[32231]: pgmap v4691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:29 smithi115 ceph-mon[35364]: pgmap v4692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:29 smithi181 ceph-mon[35869]: pgmap v4692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:29 smithi097 ceph-mon[32231]: pgmap v4692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7567.805620 secs 2024-07-26T10:05:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:31 smithi115 ceph-mon[35364]: pgmap v4693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7567.805620 secs 2024-07-26T10:05:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:31 smithi181 ceph-mon[35869]: pgmap v4693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7567.805620 secs 2024-07-26T10:05:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:31 smithi097 ceph-mon[32231]: pgmap v4693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:33 smithi115 ceph-mon[35364]: pgmap v4694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:33 smithi181 ceph-mon[35869]: pgmap v4694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:33 smithi097 ceph-mon[32231]: pgmap v4694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:35 smithi115 ceph-mon[35364]: pgmap v4695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7572.805789 secs 2024-07-26T10:05:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:35 smithi181 ceph-mon[35869]: pgmap v4695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7572.805789 secs 2024-07-26T10:05:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:35 smithi097 ceph-mon[32231]: pgmap v4695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7572.805789 secs 2024-07-26T10:05:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:36.189+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:05:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:36.190+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:05:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:36.190+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:05:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:36.190+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:05:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:36.191+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:37 smithi115 ceph-mon[35364]: pgmap v4696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:37 smithi181 ceph-mon[35869]: pgmap v4696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:37 smithi097 ceph-mon[32231]: pgmap v4696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:39 smithi115 ceph-mon[35364]: pgmap v4697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:39 smithi181 ceph-mon[35869]: pgmap v4697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:39 smithi097 ceph-mon[32231]: pgmap v4697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:40.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7577.805952 secs 2024-07-26T10:05:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7577.805952 secs 2024-07-26T10:05:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7577.805952 secs 2024-07-26T10:05:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:41 smithi115 ceph-mon[35364]: pgmap v4698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:41.535 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:41 smithi097 ceph-mon[32231]: pgmap v4698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:41.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:41 smithi181 ceph-mon[35869]: pgmap v4698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:43 smithi115 ceph-mon[35364]: pgmap v4699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:43 smithi181 ceph-mon[35869]: pgmap v4699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:43 smithi097 ceph-mon[32231]: pgmap v4699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:45 smithi115 ceph-mon[35364]: pgmap v4700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7582.806054 secs 2024-07-26T10:05:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:45 smithi181 ceph-mon[35869]: pgmap v4700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7582.806054 secs 2024-07-26T10:05:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:45 smithi097 ceph-mon[32231]: pgmap v4700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:05:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7582.806054 secs 2024-07-26T10:05:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:47 smithi115 ceph-mon[35364]: pgmap v4701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:47 smithi181 ceph-mon[35869]: pgmap v4701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:47 smithi097 ceph-mon[32231]: pgmap v4701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:49 smithi115 ceph-mon[35364]: pgmap v4702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:49 smithi181 ceph-mon[35869]: pgmap v4702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:49 smithi097 ceph-mon[32231]: pgmap v4702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:50 smithi115 ceph-mon[35364]: pgmap v4703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:05:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:50 smithi115 ceph-mon[35364]: pgmap v4704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:50.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:05:50.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7587.806219 secs 2024-07-26T10:05:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:50 smithi181 ceph-mon[35869]: pgmap v4703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:05:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:50 smithi181 ceph-mon[35869]: pgmap v4704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:05:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7587.806219 secs 2024-07-26T10:05:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:50 smithi097 ceph-mon[32231]: pgmap v4703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:05:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:50 smithi097 ceph-mon[32231]: pgmap v4704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:05:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7587.806219 secs 2024-07-26T10:05:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:51.203+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:05:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:51.203+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:05:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:51.203+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:05:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:51.203+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:05:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:05:51.203+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:52 smithi115 ceph-mon[35364]: pgmap v4705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:52 smithi181 ceph-mon[35869]: pgmap v4705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:52 smithi097 ceph-mon[32231]: pgmap v4705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:54 smithi115 ceph-mon[35364]: pgmap v4706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:05:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:54 smithi181 ceph-mon[35869]: pgmap v4706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:05:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:54 smithi097 ceph-mon[32231]: pgmap v4706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:05:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7592.806341 secs 2024-07-26T10:05:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7592.806341 secs 2024-07-26T10:05:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7592.806341 secs 2024-07-26T10:05:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:56 smithi115 ceph-mon[35364]: pgmap v4707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:56 smithi181 ceph-mon[35869]: pgmap v4707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:56 smithi097 ceph-mon[32231]: pgmap v4707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:05:58 smithi115 ceph-mon[35364]: pgmap v4708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:05:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:05:58 smithi181 ceph-mon[35869]: pgmap v4708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:05:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:05:58 smithi097 ceph-mon[32231]: pgmap v4708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:06:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:00 smithi115 ceph-mon[35364]: pgmap v4709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7597.806490 secs 2024-07-26T10:06:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:00 smithi181 ceph-mon[35869]: pgmap v4709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7597.806490 secs 2024-07-26T10:06:00.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:00 smithi097 ceph-mon[32231]: pgmap v4709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:00.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7597.806490 secs 2024-07-26T10:06:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:02 smithi115 ceph-mon[35364]: pgmap v4710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:02 smithi181 ceph-mon[35869]: pgmap v4710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:02 smithi097 ceph-mon[32231]: pgmap v4710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:04 smithi115 ceph-mon[35364]: pgmap v4711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:04 smithi181 ceph-mon[35869]: pgmap v4711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:04 smithi097 ceph-mon[32231]: pgmap v4711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7602.806636 secs 2024-07-26T10:06:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7602.806636 secs 2024-07-26T10:06:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7602.806636 secs 2024-07-26T10:06:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:06.208+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:06:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:06.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:06:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:06.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:06:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:06.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:06:06.212 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:06.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:06 smithi115 ceph-mon[35364]: pgmap v4712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:06 smithi181 ceph-mon[35869]: pgmap v4712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:06 smithi097 ceph-mon[32231]: pgmap v4712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:08 smithi115 ceph-mon[35364]: pgmap v4713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:06:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:08 smithi181 ceph-mon[35869]: pgmap v4713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:06:08.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:08 smithi097 ceph-mon[32231]: pgmap v4713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:06:10.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:10 smithi181 ceph-mon[35869]: pgmap v4714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:10.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7607.806779 secs 2024-07-26T10:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:10 smithi097 ceph-mon[32231]: pgmap v4714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7607.806779 secs 2024-07-26T10:06:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:10 smithi115 ceph-mon[35364]: pgmap v4714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7607.806779 secs 2024-07-26T10:06:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:12 smithi181 ceph-mon[35869]: pgmap v4715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:12 smithi097 ceph-mon[32231]: pgmap v4715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:12 smithi115 ceph-mon[35364]: pgmap v4715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:14 smithi097 ceph-mon[32231]: pgmap v4716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:14 smithi115 ceph-mon[35364]: pgmap v4716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:14 smithi181 ceph-mon[35869]: pgmap v4716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7612.806922 secs 2024-07-26T10:06:15.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7612.806922 secs 2024-07-26T10:06:15.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7612.806922 secs 2024-07-26T10:06:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:16 smithi115 ceph-mon[35364]: pgmap v4717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:16 smithi181 ceph-mon[35869]: pgmap v4717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:16 smithi097 ceph-mon[32231]: pgmap v4717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:18.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:18 smithi097 ceph-mon[32231]: pgmap v4718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:06:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:18 smithi115 ceph-mon[35364]: pgmap v4718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:06:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:18 smithi181 ceph-mon[35869]: pgmap v4718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:06:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:20 smithi097 ceph-mon[32231]: pgmap v4719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7617.807007 secs 2024-07-26T10:06:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:20 smithi115 ceph-mon[35364]: pgmap v4719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7617.807007 secs 2024-07-26T10:06:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:20 smithi181 ceph-mon[35869]: pgmap v4719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7617.807007 secs 2024-07-26T10:06:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:21.212+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:06:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:21.213+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:06:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:21.213+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:06:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:21.213+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:06:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:21.213+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:22 smithi115 ceph-mon[35364]: pgmap v4720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:22 smithi181 ceph-mon[35869]: pgmap v4720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:22 smithi097 ceph-mon[32231]: pgmap v4720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:24 smithi115 ceph-mon[35364]: pgmap v4721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:24 smithi181 ceph-mon[35869]: pgmap v4721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:24 smithi097 ceph-mon[32231]: pgmap v4721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7622.807152 secs 2024-07-26T10:06:25.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7622.807152 secs 2024-07-26T10:06:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7622.807152 secs 2024-07-26T10:06:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:26 smithi115 ceph-mon[35364]: pgmap v4722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:26.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:26 smithi181 ceph-mon[35869]: pgmap v4722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:26 smithi097 ceph-mon[32231]: pgmap v4722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:28 smithi115 ceph-mon[35364]: pgmap v4723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:28 smithi181 ceph-mon[35869]: pgmap v4723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:28 smithi097 ceph-mon[32231]: pgmap v4723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:30 smithi097 ceph-mon[32231]: pgmap v4724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7627.807288 secs 2024-07-26T10:06:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:30 smithi115 ceph-mon[35364]: pgmap v4724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7627.807288 secs 2024-07-26T10:06:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:30 smithi181 ceph-mon[35869]: pgmap v4724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:06:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7627.807288 secs 2024-07-26T10:06:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:32 smithi115 ceph-mon[35364]: pgmap v4725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:32 smithi181 ceph-mon[35869]: pgmap v4725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:32 smithi097 ceph-mon[32231]: pgmap v4725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:34 smithi115 ceph-mon[35364]: pgmap v4726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:34 smithi181 ceph-mon[35869]: pgmap v4726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:34 smithi097 ceph-mon[32231]: pgmap v4726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7632.807406 secs 2024-07-26T10:06:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7632.807406 secs 2024-07-26T10:06:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7632.807406 secs 2024-07-26T10:06:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:36.219+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:06:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:36.220+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:06:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:36.220+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:06:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:36.220+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:06:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:36.220+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:36 smithi115 ceph-mon[35364]: pgmap v4727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:36 smithi181 ceph-mon[35869]: pgmap v4727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:36 smithi097 ceph-mon[32231]: pgmap v4727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:38 smithi115 ceph-mon[35364]: pgmap v4728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:38 smithi181 ceph-mon[35869]: pgmap v4728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:38 smithi097 ceph-mon[32231]: pgmap v4728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:40 smithi115 ceph-mon[35364]: pgmap v4729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7637.807510 secs 2024-07-26T10:06:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:40 smithi181 ceph-mon[35869]: pgmap v4729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7637.807510 secs 2024-07-26T10:06:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:40 smithi097 ceph-mon[32231]: pgmap v4729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7637.807510 secs 2024-07-26T10:06:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:42 smithi115 ceph-mon[35364]: pgmap v4730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:42 smithi181 ceph-mon[35869]: pgmap v4730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:42 smithi097 ceph-mon[32231]: pgmap v4730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:44 smithi115 ceph-mon[35364]: pgmap v4731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:44 smithi181 ceph-mon[35869]: pgmap v4731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:44 smithi097 ceph-mon[32231]: pgmap v4731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:45.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7642.807650 secs 2024-07-26T10:06:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7642.807650 secs 2024-07-26T10:06:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7642.807650 secs 2024-07-26T10:06:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:46 smithi115 ceph-mon[35364]: pgmap v4732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:46.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:46 smithi181 ceph-mon[35869]: pgmap v4732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:46 smithi097 ceph-mon[32231]: pgmap v4732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:48 smithi115 ceph-mon[35364]: pgmap v4733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:48 smithi181 ceph-mon[35869]: pgmap v4733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:48 smithi097 ceph-mon[32231]: pgmap v4733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:06:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:50 smithi115 ceph-mon[35364]: pgmap v4734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7647.807745 secs 2024-07-26T10:06:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:50 smithi181 ceph-mon[35869]: pgmap v4734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7647.807745 secs 2024-07-26T10:06:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:50 smithi097 ceph-mon[32231]: pgmap v4734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:06:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7647.807745 secs 2024-07-26T10:06:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:51.231+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:06:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:51.233+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:06:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:51.233+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:06:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:51.233+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:06:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:06:51.233+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:51 smithi115 ceph-mon[35364]: pgmap v4735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:51.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:06:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:51.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:51 smithi181 ceph-mon[35869]: pgmap v4735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:51.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:06:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:51 smithi097 ceph-mon[32231]: pgmap v4735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:52.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:06:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:53 smithi115 ceph-mon[35364]: pgmap v4736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:06:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:53 smithi181 ceph-mon[35869]: pgmap v4736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:06:54.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:53 smithi097 ceph-mon[32231]: pgmap v4736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:06:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:55 smithi115 ceph-mon[35364]: pgmap v4737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:06:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7652.807887 secs 2024-07-26T10:06:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:55 smithi181 ceph-mon[35869]: pgmap v4737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:06:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7652.807887 secs 2024-07-26T10:06:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:55 smithi097 ceph-mon[32231]: pgmap v4737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:06:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7652.807887 secs 2024-07-26T10:06:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:57 smithi115 ceph-mon[35364]: pgmap v4738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:06:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:57 smithi181 ceph-mon[35869]: pgmap v4738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:06:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:57 smithi097 ceph-mon[32231]: pgmap v4738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:07:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:06:59 smithi115 ceph-mon[35364]: pgmap v4739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:07:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:06:59 smithi181 ceph-mon[35869]: pgmap v4739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:07:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:06:59 smithi097 ceph-mon[32231]: pgmap v4739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:07:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7657.808020 secs 2024-07-26T10:07:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7657.808020 secs 2024-07-26T10:07:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7657.808020 secs 2024-07-26T10:07:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:01 smithi115 ceph-mon[35364]: pgmap v4740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:07:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:01 smithi181 ceph-mon[35869]: pgmap v4740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:07:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:01 smithi097 ceph-mon[32231]: pgmap v4740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:07:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:03 smithi115 ceph-mon[35364]: pgmap v4741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:03 smithi181 ceph-mon[35869]: pgmap v4741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:03 smithi097 ceph-mon[32231]: pgmap v4741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:05 smithi115 ceph-mon[35364]: pgmap v4742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7662.808169 secs 2024-07-26T10:07:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:05 smithi181 ceph-mon[35869]: pgmap v4742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7662.808169 secs 2024-07-26T10:07:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:06.237+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:07:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:06.238+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:07:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:06.238+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:07:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:06.238+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:07:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:06.238+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:05 smithi097 ceph-mon[32231]: pgmap v4742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7662.808169 secs 2024-07-26T10:07:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:07 smithi115 ceph-mon[35364]: pgmap v4743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:07 smithi181 ceph-mon[35869]: pgmap v4743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:08.237 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:08.235+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:07:08.237 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:08.235+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:07:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:08.236+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:07:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:08.236+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:07:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:07 smithi097 ceph-mon[32231]: pgmap v4743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:09 smithi115 ceph-mon[35364]: pgmap v4744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:09 smithi181 ceph-mon[35869]: pgmap v4744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:09 smithi097 ceph-mon[32231]: pgmap v4744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7667.808311 secs 2024-07-26T10:07:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7667.808311 secs 2024-07-26T10:07:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7667.808311 secs 2024-07-26T10:07:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:11 smithi115 ceph-mon[35364]: pgmap v4745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:11 smithi181 ceph-mon[35869]: pgmap v4745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:11 smithi097 ceph-mon[32231]: pgmap v4745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:12 smithi115 ceph-mon[35364]: pgmap v4746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:12 smithi181 ceph-mon[35869]: pgmap v4746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:12 smithi097 ceph-mon[32231]: pgmap v4746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:15 smithi115 ceph-mon[35364]: pgmap v4747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7672.808462 secs 2024-07-26T10:07:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:15 smithi181 ceph-mon[35869]: pgmap v4747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7672.808462 secs 2024-07-26T10:07:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:15 smithi097 ceph-mon[32231]: pgmap v4747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7672.808462 secs 2024-07-26T10:07:18.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:17 smithi097 ceph-mon[32231]: pgmap v4748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:17 smithi115 ceph-mon[35364]: pgmap v4748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:17 smithi181 ceph-mon[35869]: pgmap v4748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:19 smithi115 ceph-mon[35364]: pgmap v4749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:19 smithi181 ceph-mon[35869]: pgmap v4749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:19 smithi097 ceph-mon[32231]: pgmap v4749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7677.808597 secs 2024-07-26T10:07:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7677.808597 secs 2024-07-26T10:07:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:21.240+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:07:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:21.241+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:07:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:21.241+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:07:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:21.241+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:07:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:21.241+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7677.808597 secs 2024-07-26T10:07:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:21 smithi115 ceph-mon[35364]: pgmap v4750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:21 smithi181 ceph-mon[35869]: pgmap v4750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:21 smithi097 ceph-mon[32231]: pgmap v4750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:23 smithi115 ceph-mon[35364]: pgmap v4751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:23 smithi181 ceph-mon[35869]: pgmap v4751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:23 smithi097 ceph-mon[32231]: pgmap v4751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:25 smithi115 ceph-mon[35364]: pgmap v4752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7682.808740 secs 2024-07-26T10:07:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:25 smithi181 ceph-mon[35869]: pgmap v4752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7682.808740 secs 2024-07-26T10:07:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:25 smithi097 ceph-mon[32231]: pgmap v4752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7682.808740 secs 2024-07-26T10:07:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:27 smithi115 ceph-mon[35364]: pgmap v4753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:27 smithi181 ceph-mon[35869]: pgmap v4753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:27 smithi097 ceph-mon[32231]: pgmap v4753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:29 smithi115 ceph-mon[35364]: pgmap v4754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:29 smithi181 ceph-mon[35869]: pgmap v4754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:29 smithi097 ceph-mon[32231]: pgmap v4754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7687.808875 secs 2024-07-26T10:07:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7687.808875 secs 2024-07-26T10:07:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7687.808875 secs 2024-07-26T10:07:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:31 smithi115 ceph-mon[35364]: pgmap v4755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:31 smithi181 ceph-mon[35869]: pgmap v4755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:31 smithi097 ceph-mon[32231]: pgmap v4755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:32 smithi115 ceph-mon[35364]: pgmap v4756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:32 smithi181 ceph-mon[35869]: pgmap v4756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:32 smithi097 ceph-mon[32231]: pgmap v4756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:35 smithi115 ceph-mon[35364]: pgmap v4757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7692.808965 secs 2024-07-26T10:07:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:35 smithi181 ceph-mon[35869]: pgmap v4757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7692.808965 secs 2024-07-26T10:07:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:36.248+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:07:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:36.249+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:07:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:36.249+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:07:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:36.249+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:07:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:36.249+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:35 smithi097 ceph-mon[32231]: pgmap v4757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7692.808965 secs 2024-07-26T10:07:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:37 smithi115 ceph-mon[35364]: pgmap v4758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:37 smithi181 ceph-mon[35869]: pgmap v4758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:37 smithi097 ceph-mon[32231]: pgmap v4758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:39 smithi115 ceph-mon[35364]: pgmap v4759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:39 smithi181 ceph-mon[35869]: pgmap v4759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:39 smithi097 ceph-mon[32231]: pgmap v4759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7697.809113 secs 2024-07-26T10:07:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7697.809113 secs 2024-07-26T10:07:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7697.809113 secs 2024-07-26T10:07:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:41 smithi115 ceph-mon[35364]: pgmap v4760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:41 smithi181 ceph-mon[35869]: pgmap v4760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:41 smithi097 ceph-mon[32231]: pgmap v4760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:43 smithi115 ceph-mon[35364]: pgmap v4761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:43 smithi181 ceph-mon[35869]: pgmap v4761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:43 smithi097 ceph-mon[32231]: pgmap v4761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:45 smithi115 ceph-mon[35364]: pgmap v4762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7702.809287 secs 2024-07-26T10:07:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:45 smithi181 ceph-mon[35869]: pgmap v4762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7702.809287 secs 2024-07-26T10:07:46.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:45 smithi097 ceph-mon[32231]: pgmap v4762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:46.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7702.809287 secs 2024-07-26T10:07:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:47 smithi115 ceph-mon[35364]: pgmap v4763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:47 smithi181 ceph-mon[35869]: pgmap v4763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:47 smithi097 ceph-mon[32231]: pgmap v4763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:48 smithi115 ceph-mon[35364]: pgmap v4764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:48 smithi181 ceph-mon[35869]: pgmap v4764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:48 smithi097 ceph-mon[32231]: pgmap v4764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7707.809389 secs 2024-07-26T10:07:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7707.809389 secs 2024-07-26T10:07:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7707.809389 secs 2024-07-26T10:07:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:51.251+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:07:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:51.252+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:07:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:51.252+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:07:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:51.252+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:07:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:07:51.252+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:51 smithi115 ceph-mon[35364]: pgmap v4765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:51 smithi181 ceph-mon[35869]: pgmap v4765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:51 smithi097 ceph-mon[32231]: pgmap v4765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:53.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:07:53.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:07:53.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:53 smithi115 ceph-mon[35364]: pgmap v4766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:53.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:53 smithi115 ceph-mon[35364]: pgmap v4767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:07:53.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:07:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:07:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:53 smithi181 ceph-mon[35869]: pgmap v4766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:53 smithi181 ceph-mon[35869]: pgmap v4767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:07:53.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:54.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:07:54.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:07:54.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:53 smithi097 ceph-mon[32231]: pgmap v4766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:07:54.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:53 smithi097 ceph-mon[32231]: pgmap v4767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:07:54.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:07:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:55 smithi097 ceph-mon[32231]: pgmap v4768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:07:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7712.809533 secs 2024-07-26T10:07:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:55 smithi115 ceph-mon[35364]: pgmap v4768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:07:56.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7712.809533 secs 2024-07-26T10:07:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:55 smithi181 ceph-mon[35869]: pgmap v4768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:07:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7712.809533 secs 2024-07-26T10:07:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:57 smithi097 ceph-mon[32231]: pgmap v4769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:07:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:57 smithi115 ceph-mon[35364]: pgmap v4769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:07:58.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:57 smithi181 ceph-mon[35869]: pgmap v4769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:07:59 smithi097 ceph-mon[32231]: pgmap v4770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:07:59 smithi115 ceph-mon[35364]: pgmap v4770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:07:59 smithi181 ceph-mon[35869]: pgmap v4770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7717.809680 secs 2024-07-26T10:08:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7717.809680 secs 2024-07-26T10:08:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7717.809680 secs 2024-07-26T10:08:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:01 smithi115 ceph-mon[35364]: pgmap v4771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:02.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:01 smithi181 ceph-mon[35869]: pgmap v4771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:01 smithi097 ceph-mon[32231]: pgmap v4771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:03 smithi115 ceph-mon[35364]: pgmap v4772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:03 smithi181 ceph-mon[35869]: pgmap v4772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:03 smithi097 ceph-mon[32231]: pgmap v4772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:08:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:05 smithi115 ceph-mon[35364]: pgmap v4773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:08:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7722.809858 secs 2024-07-26T10:08:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:05 smithi181 ceph-mon[35869]: pgmap v4773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:08:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7722.809858 secs 2024-07-26T10:08:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:05 smithi097 ceph-mon[32231]: pgmap v4773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:08:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7722.809858 secs 2024-07-26T10:08:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:06.257+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:08:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:06.258+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:08:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:06.258+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:08:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:06.258+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:08:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:06.258+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:07 smithi115 ceph-mon[35364]: pgmap v4774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:08:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:07 smithi181 ceph-mon[35869]: pgmap v4774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:08:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:07 smithi097 ceph-mon[32231]: pgmap v4774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:08:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:09 smithi115 ceph-mon[35364]: pgmap v4775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:09 smithi181 ceph-mon[35869]: pgmap v4775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:09 smithi097 ceph-mon[32231]: pgmap v4775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7727.809996 secs 2024-07-26T10:08:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7727.809996 secs 2024-07-26T10:08:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7727.809996 secs 2024-07-26T10:08:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:11 smithi115 ceph-mon[35364]: pgmap v4776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:11 smithi181 ceph-mon[35869]: pgmap v4776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:11 smithi097 ceph-mon[32231]: pgmap v4776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:13 smithi115 ceph-mon[35364]: pgmap v4777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:13 smithi181 ceph-mon[35869]: pgmap v4777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:13 smithi097 ceph-mon[32231]: pgmap v4777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:15 smithi115 ceph-mon[35364]: pgmap v4778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7732.810175 secs 2024-07-26T10:08:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:15 smithi181 ceph-mon[35869]: pgmap v4778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7732.810175 secs 2024-07-26T10:08:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:15 smithi097 ceph-mon[32231]: pgmap v4778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7732.810175 secs 2024-07-26T10:08:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:17 smithi115 ceph-mon[35364]: pgmap v4779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:08:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:17 smithi181 ceph-mon[35869]: pgmap v4779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:08:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:17 smithi097 ceph-mon[32231]: pgmap v4779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:08:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:19 smithi115 ceph-mon[35364]: pgmap v4780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:19 smithi181 ceph-mon[35869]: pgmap v4780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:19 smithi097 ceph-mon[32231]: pgmap v4780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:08:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7737.810317 secs 2024-07-26T10:08:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7737.810317 secs 2024-07-26T10:08:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7737.810317 secs 2024-07-26T10:08:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:21.261+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:08:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:21.262+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:08:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:21.262+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:08:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:21.262+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:08:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:21.262+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:21 smithi115 ceph-mon[35364]: pgmap v4781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:21 smithi181 ceph-mon[35869]: pgmap v4781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:21 smithi097 ceph-mon[32231]: pgmap v4781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:23 smithi115 ceph-mon[35364]: pgmap v4782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:23 smithi181 ceph-mon[35869]: pgmap v4782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:23 smithi097 ceph-mon[32231]: pgmap v4782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:08:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:25 smithi115 ceph-mon[35364]: pgmap v4783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:08:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7742.810482 secs 2024-07-26T10:08:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:25 smithi181 ceph-mon[35869]: pgmap v4783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:08:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7742.810482 secs 2024-07-26T10:08:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:25 smithi097 ceph-mon[32231]: pgmap v4783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:08:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7742.810482 secs 2024-07-26T10:08:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:27 smithi115 ceph-mon[35364]: pgmap v4784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:27 smithi181 ceph-mon[35869]: pgmap v4784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:27 smithi097 ceph-mon[32231]: pgmap v4784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:28 smithi115 ceph-mon[35364]: pgmap v4785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:28 smithi181 ceph-mon[35869]: pgmap v4785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:28 smithi097 ceph-mon[32231]: pgmap v4785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7747.810628 secs 2024-07-26T10:08:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7747.810628 secs 2024-07-26T10:08:30.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7747.810628 secs 2024-07-26T10:08:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:31 smithi115 ceph-mon[35364]: pgmap v4786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:31 smithi181 ceph-mon[35869]: pgmap v4786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:31 smithi097 ceph-mon[32231]: pgmap v4786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:33 smithi115 ceph-mon[35364]: pgmap v4787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:33 smithi181 ceph-mon[35869]: pgmap v4787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:33 smithi097 ceph-mon[32231]: pgmap v4787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:35 smithi115 ceph-mon[35364]: pgmap v4788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7752.810795 secs 2024-07-26T10:08:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:35 smithi181 ceph-mon[35869]: pgmap v4788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7752.810795 secs 2024-07-26T10:08:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:35 smithi097 ceph-mon[32231]: pgmap v4788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7752.810795 secs 2024-07-26T10:08:36.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:36.268+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:08:36.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:36.269+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:08:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:36.269+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:08:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:36.269+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:08:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:36.269+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:37 smithi115 ceph-mon[35364]: pgmap v4789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:37 smithi181 ceph-mon[35869]: pgmap v4789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:37 smithi097 ceph-mon[32231]: pgmap v4789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:39 smithi115 ceph-mon[35364]: pgmap v4790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:39 smithi181 ceph-mon[35869]: pgmap v4790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:39 smithi097 ceph-mon[32231]: pgmap v4790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7757.810943 secs 2024-07-26T10:08:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7757.810943 secs 2024-07-26T10:08:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7757.810943 secs 2024-07-26T10:08:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:41 smithi115 ceph-mon[35364]: pgmap v4791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:41 smithi181 ceph-mon[35869]: pgmap v4791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:41 smithi097 ceph-mon[32231]: pgmap v4791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:43 smithi115 ceph-mon[35364]: pgmap v4792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:43 smithi181 ceph-mon[35869]: pgmap v4792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:43 smithi097 ceph-mon[32231]: pgmap v4792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:45 smithi115 ceph-mon[35364]: pgmap v4793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7762.811079 secs 2024-07-26T10:08:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:45 smithi181 ceph-mon[35869]: pgmap v4793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7762.811079 secs 2024-07-26T10:08:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:45 smithi097 ceph-mon[32231]: pgmap v4793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7762.811079 secs 2024-07-26T10:08:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:47 smithi115 ceph-mon[35364]: pgmap v4794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:47 smithi181 ceph-mon[35869]: pgmap v4794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:47 smithi097 ceph-mon[32231]: pgmap v4794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:49 smithi115 ceph-mon[35364]: pgmap v4795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:49 smithi181 ceph-mon[35869]: pgmap v4795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:49 smithi097 ceph-mon[32231]: pgmap v4795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7767.811221 secs 2024-07-26T10:08:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7767.811221 secs 2024-07-26T10:08:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7767.811221 secs 2024-07-26T10:08:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:51.271+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:08:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:51.272+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:08:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:51.272+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:08:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:51.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:08:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:08:51.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:51 smithi115 ceph-mon[35364]: pgmap v4796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:51 smithi181 ceph-mon[35869]: pgmap v4796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:51 smithi097 ceph-mon[32231]: pgmap v4796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:53 smithi115 ceph-mon[35364]: pgmap v4797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:08:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:53 smithi181 ceph-mon[35869]: pgmap v4797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:08:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:53 smithi097 ceph-mon[32231]: pgmap v4797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:08:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:08:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: pgmap v4798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: pgmap v4799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: pgmap v4800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7772.811373 secs 2024-07-26T10:08:55.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: pgmap v4798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:55.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:55.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:55.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: pgmap v4799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:08:55.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: pgmap v4800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:08:55.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:55.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7772.811373 secs 2024-07-26T10:08:56.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:56.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:56.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: pgmap v4798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:56.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:56.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:56.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:56.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:56.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: pgmap v4799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:08:56.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: pgmap v4800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:08:56.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:08:56.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7772.811373 secs 2024-07-26T10:08:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:57 smithi115 ceph-mon[35364]: pgmap v4801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:08:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:57 smithi181 ceph-mon[35869]: pgmap v4801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:08:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:57 smithi097 ceph-mon[32231]: pgmap v4801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:09:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:08:59 smithi115 ceph-mon[35364]: pgmap v4802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:09:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:08:59 smithi181 ceph-mon[35869]: pgmap v4802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:09:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:08:59 smithi097 ceph-mon[32231]: pgmap v4802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:09:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7777.811511 secs 2024-07-26T10:09:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7777.811511 secs 2024-07-26T10:09:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7777.811511 secs 2024-07-26T10:09:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:01 smithi115 ceph-mon[35364]: pgmap v4803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:09:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:01 smithi181 ceph-mon[35869]: pgmap v4803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:09:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:01 smithi097 ceph-mon[32231]: pgmap v4803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:09:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:03 smithi115 ceph-mon[35364]: pgmap v4804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:09:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:03 smithi181 ceph-mon[35869]: pgmap v4804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:09:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:03 smithi097 ceph-mon[32231]: pgmap v4804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:09:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:05 smithi115 ceph-mon[35364]: pgmap v4805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:09:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7782.811645 secs 2024-07-26T10:09:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:05 smithi181 ceph-mon[35869]: pgmap v4805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:09:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7782.811645 secs 2024-07-26T10:09:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:05 smithi097 ceph-mon[32231]: pgmap v4805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:09:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7782.811645 secs 2024-07-26T10:09:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:06.278+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:09:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:06.279+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:09:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:06.279+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:09:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:06.279+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:09:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:06.279+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:07 smithi115 ceph-mon[35364]: pgmap v4806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:07 smithi181 ceph-mon[35869]: pgmap v4806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:07 smithi097 ceph-mon[32231]: pgmap v4806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:09 smithi115 ceph-mon[35364]: pgmap v4807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:09 smithi181 ceph-mon[35869]: pgmap v4807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:09 smithi097 ceph-mon[32231]: pgmap v4807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:11.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7787.811791 secs 2024-07-26T10:09:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7787.811791 secs 2024-07-26T10:09:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7787.811791 secs 2024-07-26T10:09:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:11 smithi115 ceph-mon[35364]: pgmap v4808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:11 smithi181 ceph-mon[35869]: pgmap v4808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:11 smithi097 ceph-mon[32231]: pgmap v4808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:13 smithi115 ceph-mon[35364]: pgmap v4809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:13 smithi181 ceph-mon[35869]: pgmap v4809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:13 smithi097 ceph-mon[32231]: pgmap v4809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:15 smithi115 ceph-mon[35364]: pgmap v4810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7792.811932 secs 2024-07-26T10:09:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:15 smithi181 ceph-mon[35869]: pgmap v4810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7792.811932 secs 2024-07-26T10:09:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:15 smithi097 ceph-mon[32231]: pgmap v4810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7792.811932 secs 2024-07-26T10:09:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:17 smithi115 ceph-mon[35364]: pgmap v4811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:17 smithi181 ceph-mon[35869]: pgmap v4811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:17 smithi097 ceph-mon[32231]: pgmap v4811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:18 smithi115 ceph-mon[35364]: pgmap v4812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:18 smithi181 ceph-mon[35869]: pgmap v4812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:18 smithi097 ceph-mon[32231]: pgmap v4812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7797.812093 secs 2024-07-26T10:09:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7797.812093 secs 2024-07-26T10:09:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7797.812093 secs 2024-07-26T10:09:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:21.281+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:09:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:21.282+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:09:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:21.282+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:09:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:21.282+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:09:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:21.282+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:21 smithi115 ceph-mon[35364]: pgmap v4813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:09:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:21 smithi181 ceph-mon[35869]: pgmap v4813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:09:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:21 smithi097 ceph-mon[32231]: pgmap v4813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:09:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:23 smithi115 ceph-mon[35364]: pgmap v4814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:09:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:23 smithi181 ceph-mon[35869]: pgmap v4814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:09:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:23 smithi097 ceph-mon[32231]: pgmap v4814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:09:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:25 smithi115 ceph-mon[35364]: pgmap v4815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7802.812200 secs 2024-07-26T10:09:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:25 smithi181 ceph-mon[35869]: pgmap v4815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7802.812200 secs 2024-07-26T10:09:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:25 smithi097 ceph-mon[32231]: pgmap v4815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7802.812200 secs 2024-07-26T10:09:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:27 smithi115 ceph-mon[35364]: pgmap v4816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:27 smithi181 ceph-mon[35869]: pgmap v4816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:27 smithi097 ceph-mon[32231]: pgmap v4816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:29 smithi115 ceph-mon[35364]: pgmap v4817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:29 smithi181 ceph-mon[35869]: pgmap v4817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:29 smithi097 ceph-mon[32231]: pgmap v4817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7807.812357 secs 2024-07-26T10:09:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7807.812357 secs 2024-07-26T10:09:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7807.812357 secs 2024-07-26T10:09:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:31 smithi115 ceph-mon[35364]: pgmap v4818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:31 smithi181 ceph-mon[35869]: pgmap v4818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:31 smithi097 ceph-mon[32231]: pgmap v4818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:33 smithi115 ceph-mon[35364]: pgmap v4819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:33 smithi181 ceph-mon[35869]: pgmap v4819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:33 smithi097 ceph-mon[32231]: pgmap v4819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:35 smithi115 ceph-mon[35364]: pgmap v4820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7812.812491 secs 2024-07-26T10:09:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:35 smithi181 ceph-mon[35869]: pgmap v4820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7812.812491 secs 2024-07-26T10:09:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:35 smithi097 ceph-mon[32231]: pgmap v4820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7812.812491 secs 2024-07-26T10:09:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:36.289+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:09:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:36.290+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:09:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:36.290+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:09:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:36.290+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:09:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:36.290+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:37 smithi115 ceph-mon[35364]: pgmap v4821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:37 smithi181 ceph-mon[35869]: pgmap v4821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:37 smithi097 ceph-mon[32231]: pgmap v4821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:39 smithi115 ceph-mon[35364]: pgmap v4822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:39 smithi181 ceph-mon[35869]: pgmap v4822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:39 smithi097 ceph-mon[32231]: pgmap v4822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7817.812637 secs 2024-07-26T10:09:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7817.812637 secs 2024-07-26T10:09:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7817.812637 secs 2024-07-26T10:09:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:41 smithi115 ceph-mon[35364]: pgmap v4823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:41 smithi181 ceph-mon[35869]: pgmap v4823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:41 smithi097 ceph-mon[32231]: pgmap v4823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:42 smithi115 ceph-mon[35364]: pgmap v4824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:42 smithi181 ceph-mon[35869]: pgmap v4824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:42 smithi097 ceph-mon[32231]: pgmap v4824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:45 smithi115 ceph-mon[35364]: pgmap v4825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7822.812780 secs 2024-07-26T10:09:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:45 smithi181 ceph-mon[35869]: pgmap v4825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7822.812780 secs 2024-07-26T10:09:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:45 smithi097 ceph-mon[32231]: pgmap v4825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7822.812780 secs 2024-07-26T10:09:48.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:47 smithi115 ceph-mon[35364]: pgmap v4826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:47 smithi181 ceph-mon[35869]: pgmap v4826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:47 smithi097 ceph-mon[32231]: pgmap v4826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:49 smithi115 ceph-mon[35364]: pgmap v4827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:49 smithi181 ceph-mon[35869]: pgmap v4827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:49 smithi097 ceph-mon[32231]: pgmap v4827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7827.812871 secs 2024-07-26T10:09:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7827.812871 secs 2024-07-26T10:09:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7827.812871 secs 2024-07-26T10:09:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:51.291+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:09:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:51.292+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:09:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:51.292+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:09:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:51.292+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:09:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:09:51.292+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:51 smithi115 ceph-mon[35364]: pgmap v4828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:51 smithi181 ceph-mon[35869]: pgmap v4828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:51 smithi097 ceph-mon[32231]: pgmap v4828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:53 smithi115 ceph-mon[35364]: pgmap v4829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:53 smithi181 ceph-mon[35869]: pgmap v4829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:53 smithi097 ceph-mon[32231]: pgmap v4829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:55 smithi115 ceph-mon[35364]: pgmap v4830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7832.813015 secs 2024-07-26T10:09:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:55 smithi181 ceph-mon[35869]: pgmap v4830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7832.813015 secs 2024-07-26T10:09:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:55 smithi097 ceph-mon[32231]: pgmap v4830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7832.813015 secs 2024-07-26T10:09:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:09:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:09:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:09:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:57 smithi115 ceph-mon[35364]: pgmap v4831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:09:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:57 smithi115 ceph-mon[35364]: pgmap v4832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:57 smithi181 ceph-mon[35869]: pgmap v4831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:09:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:57 smithi181 ceph-mon[35869]: pgmap v4832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:58.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:57 smithi097 ceph-mon[32231]: pgmap v4831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:09:58.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:57 smithi097 ceph-mon[32231]: pgmap v4832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:09:59 smithi115 ceph-mon[35364]: pgmap v4833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:09:59 smithi181 ceph-mon[35869]: pgmap v4833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:09:59 smithi097 ceph-mon[32231]: pgmap v4833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7837.813160 secs 2024-07-26T10:10:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:10:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7837.813160 secs 2024-07-26T10:10:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:10:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7837.813160 secs 2024-07-26T10:10:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:10:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:01 smithi115 ceph-mon[35364]: pgmap v4834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:01 smithi181 ceph-mon[35869]: pgmap v4834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:01 smithi097 ceph-mon[32231]: pgmap v4834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:03 smithi115 ceph-mon[35364]: pgmap v4835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:10:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:03 smithi181 ceph-mon[35869]: pgmap v4835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:10:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:03 smithi097 ceph-mon[32231]: pgmap v4835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:10:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:05 smithi115 ceph-mon[35364]: pgmap v4836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7842.813319 secs 2024-07-26T10:10:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:05 smithi181 ceph-mon[35869]: pgmap v4836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7842.813319 secs 2024-07-26T10:10:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:05 smithi097 ceph-mon[32231]: pgmap v4836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7842.813319 secs 2024-07-26T10:10:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:06.298+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:10:06.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:06.299+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:10:06.302 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:06.299+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:10:06.302 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:06.299+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:10:06.302 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:06.299+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:07.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:07 smithi181 ceph-mon[35869]: pgmap v4837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:10:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:07 smithi097 ceph-mon[32231]: pgmap v4837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:10:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:07 smithi115 ceph-mon[35364]: pgmap v4837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:10:08.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:08.267+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:10:08.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:08.267+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:10:08.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:08.268+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:10:08.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:08.268+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:10:08.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:08.268+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:10:08.270 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:08.268+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:10:08.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:08.268+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:10:08.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:08.269+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:10:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:09 smithi181 ceph-mon[35869]: pgmap v4838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:09 smithi097 ceph-mon[32231]: pgmap v4838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:09 smithi115 ceph-mon[35364]: pgmap v4838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7847.813461 secs 2024-07-26T10:10:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7847.813461 secs 2024-07-26T10:10:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7847.813461 secs 2024-07-26T10:10:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:11 smithi097 ceph-mon[32231]: pgmap v4839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:11 smithi115 ceph-mon[35364]: pgmap v4839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:11 smithi181 ceph-mon[35869]: pgmap v4839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:13 smithi115 ceph-mon[35364]: pgmap v4840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:13 smithi181 ceph-mon[35869]: pgmap v4840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:13 smithi097 ceph-mon[32231]: pgmap v4840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:15 smithi115 ceph-mon[35364]: pgmap v4841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7852.813608 secs 2024-07-26T10:10:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:15 smithi181 ceph-mon[35869]: pgmap v4841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7852.813608 secs 2024-07-26T10:10:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:15 smithi097 ceph-mon[32231]: pgmap v4841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7852.813608 secs 2024-07-26T10:10:18.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:17 smithi115 ceph-mon[35364]: pgmap v4842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:18.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:17 smithi181 ceph-mon[35869]: pgmap v4842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:18.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:17 smithi097 ceph-mon[32231]: pgmap v4842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:19 smithi115 ceph-mon[35364]: pgmap v4843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:19 smithi181 ceph-mon[35869]: pgmap v4843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:19 smithi097 ceph-mon[32231]: pgmap v4843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7857.813717 secs 2024-07-26T10:10:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7857.813717 secs 2024-07-26T10:10:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7857.813717 secs 2024-07-26T10:10:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:21.301+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:10:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:21.302+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:10:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:21.302+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:10:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:21.302+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:10:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:21.302+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:21 smithi115 ceph-mon[35364]: pgmap v4844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:21 smithi181 ceph-mon[35869]: pgmap v4844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:21 smithi097 ceph-mon[32231]: pgmap v4844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:23 smithi115 ceph-mon[35364]: pgmap v4845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:23 smithi181 ceph-mon[35869]: pgmap v4845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:23 smithi097 ceph-mon[32231]: pgmap v4845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:25 smithi115 ceph-mon[35364]: pgmap v4846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7862.813803 secs 2024-07-26T10:10:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:25 smithi181 ceph-mon[35869]: pgmap v4846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7862.813803 secs 2024-07-26T10:10:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:25 smithi097 ceph-mon[32231]: pgmap v4846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7862.813803 secs 2024-07-26T10:10:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:27 smithi115 ceph-mon[35364]: pgmap v4847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:27 smithi181 ceph-mon[35869]: pgmap v4847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:27 smithi097 ceph-mon[32231]: pgmap v4847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:29 smithi115 ceph-mon[35364]: pgmap v4848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:29 smithi181 ceph-mon[35869]: pgmap v4848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:29 smithi097 ceph-mon[32231]: pgmap v4848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7867.813894 secs 2024-07-26T10:10:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7867.813894 secs 2024-07-26T10:10:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7867.813894 secs 2024-07-26T10:10:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:31 smithi115 ceph-mon[35364]: pgmap v4849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:31 smithi181 ceph-mon[35869]: pgmap v4849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:31 smithi097 ceph-mon[32231]: pgmap v4849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:33 smithi115 ceph-mon[35364]: pgmap v4850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:33 smithi181 ceph-mon[35869]: pgmap v4850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:33 smithi097 ceph-mon[32231]: pgmap v4850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:35 smithi115 ceph-mon[35364]: pgmap v4851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:36.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7872.814056 secs 2024-07-26T10:10:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:35 smithi181 ceph-mon[35869]: pgmap v4851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7872.814056 secs 2024-07-26T10:10:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:35 smithi097 ceph-mon[32231]: pgmap v4851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7872.814056 secs 2024-07-26T10:10:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:36.308+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:10:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:36.309+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:10:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:36.309+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:10:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:36.309+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:10:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:36.309+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:37 smithi115 ceph-mon[35364]: pgmap v4852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:37 smithi181 ceph-mon[35869]: pgmap v4852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:37 smithi097 ceph-mon[32231]: pgmap v4852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:39 smithi115 ceph-mon[35364]: pgmap v4853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:39 smithi181 ceph-mon[35869]: pgmap v4853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:39 smithi097 ceph-mon[32231]: pgmap v4853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7877.814193 secs 2024-07-26T10:10:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7877.814193 secs 2024-07-26T10:10:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7877.814193 secs 2024-07-26T10:10:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:41 smithi115 ceph-mon[35364]: pgmap v4854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:41 smithi181 ceph-mon[35869]: pgmap v4854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:41 smithi097 ceph-mon[32231]: pgmap v4854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:43 smithi115 ceph-mon[35364]: pgmap v4855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:43 smithi181 ceph-mon[35869]: pgmap v4855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:43 smithi097 ceph-mon[32231]: pgmap v4855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:45 smithi115 ceph-mon[35364]: pgmap v4856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7882.814353 secs 2024-07-26T10:10:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:45 smithi181 ceph-mon[35869]: pgmap v4856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7882.814353 secs 2024-07-26T10:10:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:45 smithi097 ceph-mon[32231]: pgmap v4856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7882.814353 secs 2024-07-26T10:10:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:47 smithi115 ceph-mon[35364]: pgmap v4857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:47 smithi181 ceph-mon[35869]: pgmap v4857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:47 smithi097 ceph-mon[32231]: pgmap v4857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:49 smithi115 ceph-mon[35364]: pgmap v4858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:49 smithi181 ceph-mon[35869]: pgmap v4858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:49 smithi097 ceph-mon[32231]: pgmap v4858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7887.814493 secs 2024-07-26T10:10:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7887.814493 secs 2024-07-26T10:10:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7887.814493 secs 2024-07-26T10:10:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:51.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:10:51.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:51.312+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:10:51.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:51.312+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:10:51.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:51.312+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:10:51.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:10:51.312+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:51 smithi115 ceph-mon[35364]: pgmap v4859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:51 smithi181 ceph-mon[35869]: pgmap v4859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:51 smithi097 ceph-mon[32231]: pgmap v4859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:52 smithi115 ceph-mon[35364]: pgmap v4860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:52 smithi181 ceph-mon[35869]: pgmap v4860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:52 smithi097 ceph-mon[32231]: pgmap v4860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:55 smithi115 ceph-mon[35364]: pgmap v4861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7892.814634 secs 2024-07-26T10:10:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:55 smithi181 ceph-mon[35869]: pgmap v4861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7892.814634 secs 2024-07-26T10:10:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:55 smithi097 ceph-mon[32231]: pgmap v4861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7892.814634 secs 2024-07-26T10:10:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:57 smithi115 ceph-mon[35364]: pgmap v4862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:57 smithi181 ceph-mon[35869]: pgmap v4862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:57 smithi097 ceph-mon[32231]: pgmap v4862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:10:58.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:10:58.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:10:58.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:10:58.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:58.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:58.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: pgmap v4863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:10:58.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: pgmap v4864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:58.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:10:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:10:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: pgmap v4863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: pgmap v4864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:10:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:10:58.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:58.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:58.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: pgmap v4863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:10:58.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: pgmap v4864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:58.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:10:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:11:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:00 smithi115 ceph-mon[35364]: pgmap v4865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7897.814796 secs 2024-07-26T10:11:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:00 smithi181 ceph-mon[35869]: pgmap v4865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7897.814796 secs 2024-07-26T10:11:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:00 smithi097 ceph-mon[32231]: pgmap v4865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7897.814796 secs 2024-07-26T10:11:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:02 smithi115 ceph-mon[35364]: pgmap v4866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:02 smithi181 ceph-mon[35869]: pgmap v4866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:02 smithi097 ceph-mon[32231]: pgmap v4866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:04 smithi181 ceph-mon[35869]: pgmap v4867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:11:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:04 smithi097 ceph-mon[32231]: pgmap v4867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:11:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:04 smithi115 ceph-mon[35364]: pgmap v4867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:11:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7902.814939 secs 2024-07-26T10:11:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7902.814939 secs 2024-07-26T10:11:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7902.814939 secs 2024-07-26T10:11:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:06.319+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:11:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:06.320+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:11:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:06.320+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:11:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:06.320+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:11:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:06.320+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:06 smithi097 ceph-mon[32231]: pgmap v4868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:06 smithi115 ceph-mon[35364]: pgmap v4868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:06 smithi181 ceph-mon[35869]: pgmap v4868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:08 smithi097 ceph-mon[32231]: pgmap v4869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:11:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:08 smithi115 ceph-mon[35364]: pgmap v4869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:11:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:08 smithi181 ceph-mon[35869]: pgmap v4869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:11:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:10 smithi115 ceph-mon[35364]: pgmap v4870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7907.815077 secs 2024-07-26T10:11:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:10 smithi181 ceph-mon[35869]: pgmap v4870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7907.815077 secs 2024-07-26T10:11:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:10 smithi097 ceph-mon[32231]: pgmap v4870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7907.815077 secs 2024-07-26T10:11:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:12 smithi115 ceph-mon[35364]: pgmap v4871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:12 smithi181 ceph-mon[35869]: pgmap v4871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:12 smithi097 ceph-mon[32231]: pgmap v4871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:14 smithi115 ceph-mon[35364]: pgmap v4872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:14 smithi181 ceph-mon[35869]: pgmap v4872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:14 smithi097 ceph-mon[32231]: pgmap v4872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7912.815216 secs 2024-07-26T10:11:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7912.815216 secs 2024-07-26T10:11:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7912.815216 secs 2024-07-26T10:11:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:16 smithi115 ceph-mon[35364]: pgmap v4873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:16 smithi181 ceph-mon[35869]: pgmap v4873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:16 smithi097 ceph-mon[32231]: pgmap v4873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:19.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:18 smithi097 ceph-mon[32231]: pgmap v4874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:19.066 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:18 smithi181 ceph-mon[35869]: pgmap v4874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:18 smithi115 ceph-mon[35364]: pgmap v4874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:20 smithi181 ceph-mon[35869]: pgmap v4875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:20.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7917.815307 secs 2024-07-26T10:11:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:20 smithi115 ceph-mon[35364]: pgmap v4875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7917.815307 secs 2024-07-26T10:11:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:20 smithi097 ceph-mon[32231]: pgmap v4875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7917.815307 secs 2024-07-26T10:11:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:21.321+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:11:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:21.322+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:11:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:21.322+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:11:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:21.322+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:11:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:21.323+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:22 smithi115 ceph-mon[35364]: pgmap v4876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:22 smithi181 ceph-mon[35869]: pgmap v4876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:22 smithi097 ceph-mon[32231]: pgmap v4876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:24 smithi115 ceph-mon[35364]: pgmap v4877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:24 smithi181 ceph-mon[35869]: pgmap v4877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:24 smithi097 ceph-mon[32231]: pgmap v4877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7922.815474 secs 2024-07-26T10:11:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7922.815474 secs 2024-07-26T10:11:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7922.815474 secs 2024-07-26T10:11:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:26 smithi115 ceph-mon[35364]: pgmap v4878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:26 smithi181 ceph-mon[35869]: pgmap v4878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:26 smithi097 ceph-mon[32231]: pgmap v4878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:28 smithi115 ceph-mon[35364]: pgmap v4879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:28 smithi181 ceph-mon[35869]: pgmap v4879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:28 smithi097 ceph-mon[32231]: pgmap v4879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:31.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:30 smithi115 ceph-mon[35364]: pgmap v4880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:31.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7927.815614 secs 2024-07-26T10:11:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:30 smithi181 ceph-mon[35869]: pgmap v4880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7927.815614 secs 2024-07-26T10:11:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:30 smithi097 ceph-mon[32231]: pgmap v4880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7927.815614 secs 2024-07-26T10:11:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:32 smithi115 ceph-mon[35364]: pgmap v4881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:32 smithi181 ceph-mon[35869]: pgmap v4881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:32 smithi097 ceph-mon[32231]: pgmap v4881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:34 smithi115 ceph-mon[35364]: pgmap v4882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:34 smithi181 ceph-mon[35869]: pgmap v4882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:34 smithi097 ceph-mon[32231]: pgmap v4882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7932.815766 secs 2024-07-26T10:11:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7932.815766 secs 2024-07-26T10:11:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7932.815766 secs 2024-07-26T10:11:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:36.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:11:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:36.327+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:11:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:36.327+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:11:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:36.327+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:11:36.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:36.327+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:36 smithi115 ceph-mon[35364]: pgmap v4883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:36 smithi181 ceph-mon[35869]: pgmap v4883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:36 smithi097 ceph-mon[32231]: pgmap v4883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:38 smithi115 ceph-mon[35364]: pgmap v4884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:38 smithi181 ceph-mon[35869]: pgmap v4884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:38 smithi097 ceph-mon[32231]: pgmap v4884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:40 smithi115 ceph-mon[35364]: pgmap v4885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7937.815976 secs 2024-07-26T10:11:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:40 smithi181 ceph-mon[35869]: pgmap v4885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7937.815976 secs 2024-07-26T10:11:41.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:40 smithi097 ceph-mon[32231]: pgmap v4885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:41.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7937.815976 secs 2024-07-26T10:11:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:42 smithi115 ceph-mon[35364]: pgmap v4886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:42 smithi181 ceph-mon[35869]: pgmap v4886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:42 smithi097 ceph-mon[32231]: pgmap v4886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:44 smithi115 ceph-mon[35364]: pgmap v4887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:44 smithi181 ceph-mon[35869]: pgmap v4887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:44 smithi097 ceph-mon[32231]: pgmap v4887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7942.816137 secs 2024-07-26T10:11:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7942.816137 secs 2024-07-26T10:11:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7942.816137 secs 2024-07-26T10:11:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:46 smithi115 ceph-mon[35364]: pgmap v4888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:46 smithi181 ceph-mon[35869]: pgmap v4888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:46 smithi097 ceph-mon[32231]: pgmap v4888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:47 smithi115 ceph-mon[35364]: pgmap v4889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:47 smithi181 ceph-mon[35869]: pgmap v4889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:47 smithi097 ceph-mon[32231]: pgmap v4889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:50 smithi097 ceph-mon[32231]: pgmap v4890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7947.816280 secs 2024-07-26T10:11:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:50 smithi115 ceph-mon[35364]: pgmap v4890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7947.816280 secs 2024-07-26T10:11:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:50 smithi181 ceph-mon[35869]: pgmap v4890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7947.816280 secs 2024-07-26T10:11:51.331 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:51.329+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:11:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:51.330+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:11:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:51.330+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:11:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:51.330+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:11:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:11:51.330+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:52 smithi097 ceph-mon[32231]: pgmap v4891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:52 smithi115 ceph-mon[35364]: pgmap v4891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:52 smithi181 ceph-mon[35869]: pgmap v4891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:54 smithi115 ceph-mon[35364]: pgmap v4892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:54 smithi181 ceph-mon[35869]: pgmap v4892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:55.283 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:54 smithi097 ceph-mon[32231]: pgmap v4892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7952.816377 secs 2024-07-26T10:11:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7952.816377 secs 2024-07-26T10:11:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7952.816377 secs 2024-07-26T10:11:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:56 smithi115 ceph-mon[35364]: pgmap v4893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:56 smithi181 ceph-mon[35869]: pgmap v4893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:56 smithi097 ceph-mon[32231]: pgmap v4893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:58 smithi115 ceph-mon[35364]: pgmap v4894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:11:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:58 smithi181 ceph-mon[35869]: pgmap v4894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:11:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:58 smithi097 ceph-mon[32231]: pgmap v4894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:12:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:00.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:00.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:11:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:12:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:00.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:11:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:12:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:12:00.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:00.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:00.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:11:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:12:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:00 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:12:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:00 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:12:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:00 smithi115 ceph-mon[35364]: pgmap v4895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:12:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:00 smithi115 ceph-mon[35364]: pgmap v4896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7957.816522 secs 2024-07-26T10:12:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:00 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:12:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:00 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:12:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:00 smithi181 ceph-mon[35869]: pgmap v4895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:12:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:00 smithi181 ceph-mon[35869]: pgmap v4896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7957.816522 secs 2024-07-26T10:12:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:00 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:12:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:00 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:12:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:00 smithi097 ceph-mon[32231]: pgmap v4895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:12:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:00 smithi097 ceph-mon[32231]: pgmap v4896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7957.816522 secs 2024-07-26T10:12:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:02 smithi115 ceph-mon[35364]: pgmap v4897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:02 smithi181 ceph-mon[35869]: pgmap v4897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:02 smithi097 ceph-mon[32231]: pgmap v4897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:04 smithi115 ceph-mon[35364]: pgmap v4898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:04 smithi181 ceph-mon[35869]: pgmap v4898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:04 smithi097 ceph-mon[32231]: pgmap v4898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7962.816663 secs 2024-07-26T10:12:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7962.816663 secs 2024-07-26T10:12:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7962.816663 secs 2024-07-26T10:12:06.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:06.334+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:12:06.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:06.335+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:12:06.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:06.335+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:12:06.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:06.335+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:12:06.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:06.335+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:06 smithi115 ceph-mon[35364]: pgmap v4899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:06 smithi181 ceph-mon[35869]: pgmap v4899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:06 smithi097 ceph-mon[32231]: pgmap v4899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:08 smithi115 ceph-mon[35364]: pgmap v4900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:12:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:08 smithi181 ceph-mon[35869]: pgmap v4900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:12:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:08 smithi097 ceph-mon[32231]: pgmap v4900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:12:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:10 smithi115 ceph-mon[35364]: pgmap v4901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:12:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7967.816798 secs 2024-07-26T10:12:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:10 smithi181 ceph-mon[35869]: pgmap v4901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:12:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7967.816798 secs 2024-07-26T10:12:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:10 smithi097 ceph-mon[32231]: pgmap v4901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:12:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7967.816798 secs 2024-07-26T10:12:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:12 smithi115 ceph-mon[35364]: pgmap v4902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:12 smithi181 ceph-mon[35869]: pgmap v4902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:12 smithi097 ceph-mon[32231]: pgmap v4902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:14 smithi115 ceph-mon[35364]: pgmap v4903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:14 smithi181 ceph-mon[35869]: pgmap v4903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:14 smithi097 ceph-mon[32231]: pgmap v4903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7972.816947 secs 2024-07-26T10:12:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7972.816947 secs 2024-07-26T10:12:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7972.816947 secs 2024-07-26T10:12:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:16 smithi115 ceph-mon[35364]: pgmap v4904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:16 smithi181 ceph-mon[35869]: pgmap v4904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:16 smithi097 ceph-mon[32231]: pgmap v4904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:18 smithi115 ceph-mon[35364]: pgmap v4905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:18 smithi181 ceph-mon[35869]: pgmap v4905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:18 smithi097 ceph-mon[32231]: pgmap v4905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:20 smithi115 ceph-mon[35364]: pgmap v4906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7977.817089 secs 2024-07-26T10:12:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:20 smithi181 ceph-mon[35869]: pgmap v4906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7977.817089 secs 2024-07-26T10:12:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:20 smithi097 ceph-mon[32231]: pgmap v4906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7977.817089 secs 2024-07-26T10:12:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:21.336+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:12:21.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:21.337+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:12:21.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:21.337+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:12:21.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:21.337+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:12:21.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:21.337+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:22 smithi115 ceph-mon[35364]: pgmap v4907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:22 smithi181 ceph-mon[35869]: pgmap v4907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:22 smithi097 ceph-mon[32231]: pgmap v4907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:24 smithi115 ceph-mon[35364]: pgmap v4908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:24 smithi181 ceph-mon[35869]: pgmap v4908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:24 smithi097 ceph-mon[32231]: pgmap v4908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7982.817270 secs 2024-07-26T10:12:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7982.817270 secs 2024-07-26T10:12:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7982.817270 secs 2024-07-26T10:12:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:26 smithi115 ceph-mon[35364]: pgmap v4909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:26 smithi181 ceph-mon[35869]: pgmap v4909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:26 smithi097 ceph-mon[32231]: pgmap v4909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:28 smithi115 ceph-mon[35364]: pgmap v4910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:28 smithi181 ceph-mon[35869]: pgmap v4910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:28 smithi097 ceph-mon[32231]: pgmap v4910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:30 smithi115 ceph-mon[35364]: pgmap v4911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7987.817348 secs 2024-07-26T10:12:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:30 smithi181 ceph-mon[35869]: pgmap v4911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7987.817348 secs 2024-07-26T10:12:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:30 smithi097 ceph-mon[32231]: pgmap v4911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7987.817348 secs 2024-07-26T10:12:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:32 smithi115 ceph-mon[35364]: pgmap v4912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:32 smithi181 ceph-mon[35869]: pgmap v4912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:32 smithi097 ceph-mon[32231]: pgmap v4912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:33 smithi115 ceph-mon[35364]: pgmap v4913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:33 smithi181 ceph-mon[35869]: pgmap v4913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:33 smithi097 ceph-mon[32231]: pgmap v4913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7992.817438 secs 2024-07-26T10:12:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7992.817438 secs 2024-07-26T10:12:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7992.817438 secs 2024-07-26T10:12:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:36.342+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:12:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:36.344+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:12:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:36.344+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:12:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:36.344+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:12:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:36.344+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:36 smithi115 ceph-mon[35364]: pgmap v4914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:36 smithi181 ceph-mon[35869]: pgmap v4914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:36 smithi097 ceph-mon[32231]: pgmap v4914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:38 smithi115 ceph-mon[35364]: pgmap v4915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:38 smithi181 ceph-mon[35869]: pgmap v4915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:38 smithi097 ceph-mon[32231]: pgmap v4915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:40 smithi115 ceph-mon[35364]: pgmap v4916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 7997.817539 secs 2024-07-26T10:12:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:40 smithi181 ceph-mon[35869]: pgmap v4916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 7997.817539 secs 2024-07-26T10:12:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:40 smithi097 ceph-mon[32231]: pgmap v4916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 7997.817539 secs 2024-07-26T10:12:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:42 smithi115 ceph-mon[35364]: pgmap v4917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:42 smithi181 ceph-mon[35869]: pgmap v4917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:42 smithi097 ceph-mon[32231]: pgmap v4917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:44 smithi181 ceph-mon[35869]: pgmap v4918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:44 smithi097 ceph-mon[32231]: pgmap v4918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:44 smithi115 ceph-mon[35364]: pgmap v4918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8002.817684 secs 2024-07-26T10:12:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8002.817684 secs 2024-07-26T10:12:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8002.817684 secs 2024-07-26T10:12:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:46 smithi097 ceph-mon[32231]: pgmap v4919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:46 smithi115 ceph-mon[35364]: pgmap v4919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:46 smithi181 ceph-mon[35869]: pgmap v4919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:48 smithi097 ceph-mon[32231]: pgmap v4920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:48 smithi115 ceph-mon[35364]: pgmap v4920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:48 smithi181 ceph-mon[35869]: pgmap v4920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:50 smithi115 ceph-mon[35364]: pgmap v4921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8007.817823 secs 2024-07-26T10:12:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:50 smithi181 ceph-mon[35869]: pgmap v4921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8007.817823 secs 2024-07-26T10:12:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:50 smithi097 ceph-mon[32231]: pgmap v4921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:12:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8007.817823 secs 2024-07-26T10:12:51.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:51.346+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:12:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:51.347+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:12:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:51.347+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:12:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:51.347+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:12:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:12:51.347+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:52.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:52 smithi115 ceph-mon[35364]: pgmap v4922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:52.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:52 smithi181 ceph-mon[35869]: pgmap v4922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:52.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:52 smithi097 ceph-mon[32231]: pgmap v4922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:12:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:54 smithi115 ceph-mon[35364]: pgmap v4923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:54 smithi181 ceph-mon[35869]: pgmap v4923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:54 smithi097 ceph-mon[32231]: pgmap v4923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8012.817982 secs 2024-07-26T10:12:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8012.817982 secs 2024-07-26T10:12:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8012.817982 secs 2024-07-26T10:12:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:56 smithi115 ceph-mon[35364]: pgmap v4924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:56.773 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:56 smithi097 ceph-mon[32231]: pgmap v4924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:56.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:56 smithi181 ceph-mon[35869]: pgmap v4924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:12:58 smithi115 ceph-mon[35364]: pgmap v4925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:12:58 smithi181 ceph-mon[35869]: pgmap v4925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:12:58 smithi097 ceph-mon[32231]: pgmap v4925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:00 smithi115 ceph-mon[35364]: pgmap v4926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:00.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:00.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8017.818077 secs 2024-07-26T10:13:00.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:00.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:00 smithi181 ceph-mon[35869]: pgmap v4926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8017.818077 secs 2024-07-26T10:13:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:00 smithi097 ceph-mon[32231]: pgmap v4926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8017.818077 secs 2024-07-26T10:13:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:01.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:13:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:02 smithi115 ceph-mon[35364]: pgmap v4927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:13:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:02 smithi115 ceph-mon[35364]: pgmap v4928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:02 smithi181 ceph-mon[35869]: pgmap v4927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:13:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:02 smithi181 ceph-mon[35869]: pgmap v4928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:02 smithi097 ceph-mon[32231]: pgmap v4927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:13:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:02 smithi097 ceph-mon[32231]: pgmap v4928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:04 smithi115 ceph-mon[35364]: pgmap v4929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:04 smithi181 ceph-mon[35869]: pgmap v4929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:04 smithi097 ceph-mon[32231]: pgmap v4929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8022.818210 secs 2024-07-26T10:13:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8022.818210 secs 2024-07-26T10:13:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8022.818210 secs 2024-07-26T10:13:06.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.353+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:13:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.354+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:13:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.354+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:13:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:13:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:06 smithi115 ceph-mon[35364]: pgmap v4930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:06 smithi181 ceph-mon[35869]: pgmap v4930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:06 smithi097 ceph-mon[32231]: pgmap v4930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.902+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T10:13:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.902+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:13:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.902+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006300 tx=0x7ff3e009e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:13:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.902+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006300 tx=0x7ff3e009e000 comp rx=0 tx=0).stop 2024-07-26T10:13:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.902+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T10:13:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.902+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:13:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.902+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b3c50 0x7ff3d40b4650 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:13:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.902+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:13:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.903+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:13:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:06.904+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000ed10 tx=0x7ff3e004a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:13:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:08.300+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:13:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:08.300+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006480 con 0x7ff3f01061a0 2024-07-26T10:13:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:08.300+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:13:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:08.301+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:13:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:08 smithi115 ceph-mon[35364]: pgmap v4931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:08 smithi181 ceph-mon[35869]: pgmap v4931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:08 smithi097 ceph-mon[32231]: pgmap v4931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:10 smithi115 ceph-mon[35364]: pgmap v4932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8027.818368 secs 2024-07-26T10:13:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:10 smithi181 ceph-mon[35869]: pgmap v4932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8027.818368 secs 2024-07-26T10:13:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:10 smithi097 ceph-mon[32231]: pgmap v4932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8027.818368 secs 2024-07-26T10:13:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:12 smithi115 ceph-mon[35364]: pgmap v4933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:12 smithi181 ceph-mon[35869]: pgmap v4933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:12 smithi097 ceph-mon[32231]: pgmap v4933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:14 smithi115 ceph-mon[35364]: pgmap v4934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:14 smithi181 ceph-mon[35869]: pgmap v4934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:14 smithi097 ceph-mon[32231]: pgmap v4934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8032.818510 secs 2024-07-26T10:13:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8032.818510 secs 2024-07-26T10:13:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8032.818510 secs 2024-07-26T10:13:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:16 smithi115 ceph-mon[35364]: pgmap v4935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:16 smithi181 ceph-mon[35869]: pgmap v4935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:16 smithi097 ceph-mon[32231]: pgmap v4935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:18 smithi115 ceph-mon[35364]: pgmap v4936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:18 smithi181 ceph-mon[35869]: pgmap v4936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:18 smithi097 ceph-mon[32231]: pgmap v4936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:20 smithi115 ceph-mon[35364]: pgmap v4937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8037.818652 secs 2024-07-26T10:13:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:20 smithi181 ceph-mon[35869]: pgmap v4937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:20.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8037.818652 secs 2024-07-26T10:13:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:20 smithi097 ceph-mon[32231]: pgmap v4937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8037.818652 secs 2024-07-26T10:13:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:21.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:13:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:21.355+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:13:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:21.355+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:13:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:21.355+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:13:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:21.355+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:22 smithi115 ceph-mon[35364]: pgmap v4938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:22 smithi181 ceph-mon[35869]: pgmap v4938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:22 smithi097 ceph-mon[32231]: pgmap v4938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:24 smithi115 ceph-mon[35364]: pgmap v4939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:13:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:24 smithi181 ceph-mon[35869]: pgmap v4939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:13:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:24 smithi097 ceph-mon[32231]: pgmap v4939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:13:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8042.818796 secs 2024-07-26T10:13:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8042.818796 secs 2024-07-26T10:13:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8042.818796 secs 2024-07-26T10:13:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:26 smithi115 ceph-mon[35364]: pgmap v4940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:13:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:26 smithi181 ceph-mon[35869]: pgmap v4940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:13:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:26 smithi097 ceph-mon[32231]: pgmap v4940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:13:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:28 smithi115 ceph-mon[35364]: pgmap v4941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:28 smithi181 ceph-mon[35869]: pgmap v4941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:28 smithi097 ceph-mon[32231]: pgmap v4941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:30 smithi115 ceph-mon[35364]: pgmap v4942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:13:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8047.818947 secs 2024-07-26T10:13:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:30 smithi181 ceph-mon[35869]: pgmap v4942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:13:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8047.818947 secs 2024-07-26T10:13:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:30 smithi097 ceph-mon[32231]: pgmap v4942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:13:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8047.818947 secs 2024-07-26T10:13:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:32 smithi115 ceph-mon[35364]: pgmap v4943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:13:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:32 smithi181 ceph-mon[35869]: pgmap v4943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:13:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:32 smithi097 ceph-mon[32231]: pgmap v4943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:13:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:34 smithi115 ceph-mon[35364]: pgmap v4944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:34 smithi181 ceph-mon[35869]: pgmap v4944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:34 smithi097 ceph-mon[32231]: pgmap v4944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8052.819094 secs 2024-07-26T10:13:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8052.819094 secs 2024-07-26T10:13:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8052.819094 secs 2024-07-26T10:13:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:36.361+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:13:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:36.362+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:13:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:36.362+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:13:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:36.362+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:13:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:36.362+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:36 smithi115 ceph-mon[35364]: pgmap v4945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:36 smithi181 ceph-mon[35869]: pgmap v4945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:36 smithi097 ceph-mon[32231]: pgmap v4945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:38.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:38 smithi181 ceph-mon[35869]: pgmap v4946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:38.500 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:38 smithi115 ceph-mon[35364]: pgmap v4946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:38 smithi097 ceph-mon[32231]: pgmap v4946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:40 smithi097 ceph-mon[32231]: pgmap v4947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:13:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8057.819304 secs 2024-07-26T10:13:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:40 smithi115 ceph-mon[35364]: pgmap v4947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:13:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8057.819304 secs 2024-07-26T10:13:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:40 smithi181 ceph-mon[35869]: pgmap v4947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:13:40.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8057.819304 secs 2024-07-26T10:13:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:42 smithi097 ceph-mon[32231]: pgmap v4948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:42 smithi115 ceph-mon[35364]: pgmap v4948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:42 smithi181 ceph-mon[35869]: pgmap v4948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:44 smithi097 ceph-mon[32231]: pgmap v4949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:13:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:44 smithi115 ceph-mon[35364]: pgmap v4949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:13:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:44 smithi181 ceph-mon[35869]: pgmap v4949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:13:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8062.819459 secs 2024-07-26T10:13:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8062.819459 secs 2024-07-26T10:13:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8062.819459 secs 2024-07-26T10:13:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:46 smithi115 ceph-mon[35364]: pgmap v4950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:46 smithi181 ceph-mon[35869]: pgmap v4950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:46 smithi097 ceph-mon[32231]: pgmap v4950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:48 smithi115 ceph-mon[35364]: pgmap v4951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:48 smithi181 ceph-mon[35869]: pgmap v4951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:48 smithi097 ceph-mon[32231]: pgmap v4951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:50 smithi115 ceph-mon[35364]: pgmap v4952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8067.819620 secs 2024-07-26T10:13:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:50 smithi181 ceph-mon[35869]: pgmap v4952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8067.819620 secs 2024-07-26T10:13:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:50 smithi097 ceph-mon[32231]: pgmap v4952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:13:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8067.819620 secs 2024-07-26T10:13:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:51.364+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:13:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:51.365+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:13:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:51.365+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:13:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:51.365+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:13:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:13:51.365+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:52.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:52 smithi181 ceph-mon[35869]: pgmap v4953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:52.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:52 smithi097 ceph-mon[32231]: pgmap v4953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:52 smithi115 ceph-mon[35364]: pgmap v4953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:13:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:54 smithi115 ceph-mon[35364]: pgmap v4954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:54 smithi181 ceph-mon[35869]: pgmap v4954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:54 smithi097 ceph-mon[32231]: pgmap v4954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8072.819777 secs 2024-07-26T10:13:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8072.819777 secs 2024-07-26T10:13:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8072.819777 secs 2024-07-26T10:13:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:56 smithi115 ceph-mon[35364]: pgmap v4955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:56 smithi181 ceph-mon[35869]: pgmap v4955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:56 smithi097 ceph-mon[32231]: pgmap v4955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:13:58 smithi115 ceph-mon[35364]: pgmap v4956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:13:58 smithi181 ceph-mon[35869]: pgmap v4956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:13:58 smithi097 ceph-mon[32231]: pgmap v4956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:00 smithi115 ceph-mon[35364]: pgmap v4957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8077.820877 secs 2024-07-26T10:14:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:00 smithi181 ceph-mon[35869]: pgmap v4957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8077.820877 secs 2024-07-26T10:14:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:00 smithi097 ceph-mon[32231]: pgmap v4957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8077.820877 secs 2024-07-26T10:14:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:02 smithi115 ceph-mon[35364]: pgmap v4958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:02.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:02.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:02 smithi181 ceph-mon[35869]: pgmap v4958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:02 smithi097 ceph-mon[32231]: pgmap v4958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:03.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:03.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:03.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:03.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:14:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:04 smithi115 ceph-mon[35364]: pgmap v4959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:04 smithi115 ceph-mon[35364]: pgmap v4960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:04 smithi181 ceph-mon[35869]: pgmap v4959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:04 smithi181 ceph-mon[35869]: pgmap v4960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:04 smithi097 ceph-mon[32231]: pgmap v4959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:04 smithi097 ceph-mon[32231]: pgmap v4960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8082.821011 secs 2024-07-26T10:14:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8082.821011 secs 2024-07-26T10:14:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8082.821011 secs 2024-07-26T10:14:06.373 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:06.371+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:14:06.374 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:06.372+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:14:06.374 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:06.372+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:14:06.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:06.372+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:14:06.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:06.372+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:06 smithi115 ceph-mon[35364]: pgmap v4961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:06 smithi181 ceph-mon[35869]: pgmap v4961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:06 smithi097 ceph-mon[32231]: pgmap v4961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:08 smithi115 ceph-mon[35364]: pgmap v4962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:08 smithi181 ceph-mon[35869]: pgmap v4962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:08 smithi097 ceph-mon[32231]: pgmap v4962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:10 smithi115 ceph-mon[35364]: pgmap v4963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8087.821174 secs 2024-07-26T10:14:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:10 smithi181 ceph-mon[35869]: pgmap v4963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8087.821174 secs 2024-07-26T10:14:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:10 smithi097 ceph-mon[32231]: pgmap v4963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8087.821174 secs 2024-07-26T10:14:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:12 smithi115 ceph-mon[35364]: pgmap v4964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:12 smithi181 ceph-mon[35869]: pgmap v4964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:12 smithi097 ceph-mon[32231]: pgmap v4964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:14 smithi115 ceph-mon[35364]: pgmap v4965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:14 smithi181 ceph-mon[35869]: pgmap v4965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:14 smithi097 ceph-mon[32231]: pgmap v4965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8092.821340 secs 2024-07-26T10:14:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8092.821340 secs 2024-07-26T10:14:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8092.821340 secs 2024-07-26T10:14:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:16 smithi115 ceph-mon[35364]: pgmap v4966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:16 smithi181 ceph-mon[35869]: pgmap v4966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:16 smithi097 ceph-mon[32231]: pgmap v4966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:18 smithi115 ceph-mon[35364]: pgmap v4967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:18 smithi181 ceph-mon[35869]: pgmap v4967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:18 smithi097 ceph-mon[32231]: pgmap v4967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:20 smithi115 ceph-mon[35364]: pgmap v4968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:14:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8097.821504 secs 2024-07-26T10:14:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:20 smithi181 ceph-mon[35869]: pgmap v4968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:14:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8097.821504 secs 2024-07-26T10:14:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:20 smithi097 ceph-mon[32231]: pgmap v4968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:14:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8097.821504 secs 2024-07-26T10:14:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:21.374+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:14:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:21.375+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:14:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:21.375+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:14:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:21.375+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:14:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:21.375+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:22 smithi115 ceph-mon[35364]: pgmap v4969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:22 smithi181 ceph-mon[35869]: pgmap v4969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:22 smithi097 ceph-mon[32231]: pgmap v4969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:24.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:24 smithi115 ceph-mon[35364]: pgmap v4970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:24 smithi181 ceph-mon[35869]: pgmap v4970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:24 smithi097 ceph-mon[32231]: pgmap v4970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8102.821602 secs 2024-07-26T10:14:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8102.821602 secs 2024-07-26T10:14:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8102.821602 secs 2024-07-26T10:14:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:26 smithi115 ceph-mon[35364]: pgmap v4971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:26 smithi181 ceph-mon[35869]: pgmap v4971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:26 smithi097 ceph-mon[32231]: pgmap v4971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:28 smithi115 ceph-mon[35364]: pgmap v4972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:14:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:28 smithi181 ceph-mon[35869]: pgmap v4972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:14:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:28 smithi097 ceph-mon[32231]: pgmap v4972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:14:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:30 smithi181 ceph-mon[35869]: pgmap v4973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:14:30.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8107.821694 secs 2024-07-26T10:14:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:30 smithi097 ceph-mon[32231]: pgmap v4973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:14:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8107.821694 secs 2024-07-26T10:14:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:30 smithi115 ceph-mon[35364]: pgmap v4973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:14:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8107.821694 secs 2024-07-26T10:14:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:32 smithi181 ceph-mon[35869]: pgmap v4974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:32 smithi097 ceph-mon[32231]: pgmap v4974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:32 smithi115 ceph-mon[35364]: pgmap v4974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:34 smithi097 ceph-mon[32231]: pgmap v4975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:34 smithi115 ceph-mon[35364]: pgmap v4975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:34 smithi181 ceph-mon[35869]: pgmap v4975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8112.821850 secs 2024-07-26T10:14:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8112.821850 secs 2024-07-26T10:14:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8112.821850 secs 2024-07-26T10:14:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:36.376+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:14:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:36.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:14:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:36.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:14:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:36.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:14:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:36.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:36 smithi115 ceph-mon[35364]: pgmap v4976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:36 smithi181 ceph-mon[35869]: pgmap v4976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:36 smithi097 ceph-mon[32231]: pgmap v4976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:38 smithi115 ceph-mon[35364]: pgmap v4977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:38 smithi181 ceph-mon[35869]: pgmap v4977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:38 smithi097 ceph-mon[32231]: pgmap v4977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:40 smithi115 ceph-mon[35364]: pgmap v4978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8117.821989 secs 2024-07-26T10:14:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:40 smithi181 ceph-mon[35869]: pgmap v4978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8117.821989 secs 2024-07-26T10:14:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:40 smithi097 ceph-mon[32231]: pgmap v4978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8117.821989 secs 2024-07-26T10:14:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:42 smithi115 ceph-mon[35364]: pgmap v4979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:42 smithi181 ceph-mon[35869]: pgmap v4979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:42 smithi097 ceph-mon[32231]: pgmap v4979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:44 smithi115 ceph-mon[35364]: pgmap v4980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:44 smithi181 ceph-mon[35869]: pgmap v4980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:44 smithi097 ceph-mon[32231]: pgmap v4980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8122.822148 secs 2024-07-26T10:14:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8122.822148 secs 2024-07-26T10:14:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8122.822148 secs 2024-07-26T10:14:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:46 smithi115 ceph-mon[35364]: pgmap v4981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:46 smithi181 ceph-mon[35869]: pgmap v4981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:46 smithi097 ceph-mon[32231]: pgmap v4981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:48 smithi115 ceph-mon[35364]: pgmap v4982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:48 smithi181 ceph-mon[35869]: pgmap v4982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:48 smithi097 ceph-mon[32231]: pgmap v4982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:50 smithi115 ceph-mon[35364]: pgmap v4983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8127.822314 secs 2024-07-26T10:14:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:50 smithi181 ceph-mon[35869]: pgmap v4983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8127.822314 secs 2024-07-26T10:14:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:50 smithi097 ceph-mon[32231]: pgmap v4983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:14:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8127.822314 secs 2024-07-26T10:14:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:51.381+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:14:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:51.382+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:14:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:51.382+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:14:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:51.382+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:14:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:14:51.382+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:52 smithi115 ceph-mon[35364]: pgmap v4984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:52 smithi181 ceph-mon[35869]: pgmap v4984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:52 smithi097 ceph-mon[32231]: pgmap v4984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:14:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:54 smithi115 ceph-mon[35364]: pgmap v4985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:54 smithi181 ceph-mon[35869]: pgmap v4985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:54 smithi097 ceph-mon[32231]: pgmap v4985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8132.822407 secs 2024-07-26T10:14:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8132.822407 secs 2024-07-26T10:14:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8132.822407 secs 2024-07-26T10:14:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:56 smithi115 ceph-mon[35364]: pgmap v4986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:56 smithi181 ceph-mon[35869]: pgmap v4986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:56 smithi097 ceph-mon[32231]: pgmap v4986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:14:58 smithi115 ceph-mon[35364]: pgmap v4987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:14:58 smithi181 ceph-mon[35869]: pgmap v4987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:14:58 smithi097 ceph-mon[32231]: pgmap v4987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:00 smithi115 ceph-mon[35364]: pgmap v4988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8137.822573 secs 2024-07-26T10:15:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:00 smithi181 ceph-mon[35869]: pgmap v4988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8137.822573 secs 2024-07-26T10:15:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:00 smithi097 ceph-mon[32231]: pgmap v4988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8137.822573 secs 2024-07-26T10:15:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:02 smithi115 ceph-mon[35364]: pgmap v4989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:02 smithi181 ceph-mon[35869]: pgmap v4989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:02 smithi097 ceph-mon[32231]: pgmap v4989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:04 smithi115 ceph-mon[35364]: pgmap v4990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:04 smithi181 ceph-mon[35869]: pgmap v4990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:04 smithi097 ceph-mon[32231]: pgmap v4990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8142.822680 secs 2024-07-26T10:15:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8142.822680 secs 2024-07-26T10:15:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8142.822680 secs 2024-07-26T10:15:06.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:06.387+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:15:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:06.388+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:15:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:06.388+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:15:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:06.388+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:15:06.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:06.388+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:06 smithi115 ceph-mon[35364]: pgmap v4991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:06 smithi181 ceph-mon[35869]: pgmap v4991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:06 smithi097 ceph-mon[32231]: pgmap v4991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:07.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:15:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:08 smithi115 ceph-mon[35364]: pgmap v4992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:08 smithi115 ceph-mon[35364]: pgmap v4993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T10:15:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:08 smithi115 ceph-mon[35364]: pgmap v4994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:08 smithi181 ceph-mon[35869]: pgmap v4992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:09.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:08 smithi181 ceph-mon[35869]: pgmap v4993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T10:15:09.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:08 smithi181 ceph-mon[35869]: pgmap v4994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:08 smithi097 ceph-mon[32231]: pgmap v4992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:08 smithi097 ceph-mon[32231]: pgmap v4993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T10:15:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:08 smithi097 ceph-mon[32231]: pgmap v4994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:10 smithi115 ceph-mon[35364]: pgmap v4995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8147.822840 secs 2024-07-26T10:15:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:10 smithi181 ceph-mon[35869]: pgmap v4995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8147.822840 secs 2024-07-26T10:15:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:10 smithi097 ceph-mon[32231]: pgmap v4995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8147.822840 secs 2024-07-26T10:15:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:12 smithi115 ceph-mon[35364]: pgmap v4996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T10:15:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:12 smithi181 ceph-mon[35869]: pgmap v4996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T10:15:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:12 smithi097 ceph-mon[32231]: pgmap v4996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T10:15:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:14 smithi115 ceph-mon[35364]: pgmap v4997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T10:15:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:14 smithi181 ceph-mon[35869]: pgmap v4997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T10:15:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:14 smithi097 ceph-mon[32231]: pgmap v4997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T10:15:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8152.822992 secs 2024-07-26T10:15:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8152.822992 secs 2024-07-26T10:15:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8152.822992 secs 2024-07-26T10:15:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:16 smithi115 ceph-mon[35364]: pgmap v4998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T10:15:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:16 smithi181 ceph-mon[35869]: pgmap v4998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T10:15:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:16 smithi097 ceph-mon[32231]: pgmap v4998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T10:15:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:18 smithi115 ceph-mon[35364]: pgmap v4999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:18 smithi181 ceph-mon[35869]: pgmap v4999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:18 smithi097 ceph-mon[32231]: pgmap v4999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:20 smithi115 ceph-mon[35364]: pgmap v5000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:20 smithi181 ceph-mon[35869]: pgmap v5000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:20 smithi097 ceph-mon[32231]: pgmap v5000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:21.389+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:15:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:21.390+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:15:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:21.390+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:15:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:21.390+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:15:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:21.391+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8157.823161 secs 2024-07-26T10:15:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8157.823161 secs 2024-07-26T10:15:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8157.823161 secs 2024-07-26T10:15:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:22 smithi115 ceph-mon[35364]: pgmap v5001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:22 smithi181 ceph-mon[35869]: pgmap v5001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:22 smithi097 ceph-mon[32231]: pgmap v5001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:25.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:24 smithi115 ceph-mon[35364]: pgmap v5002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:25.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:24 smithi181 ceph-mon[35869]: pgmap v5002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:25.206 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:24 smithi097 ceph-mon[32231]: pgmap v5002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8162.823321 secs 2024-07-26T10:15:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8162.823321 secs 2024-07-26T10:15:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8162.823321 secs 2024-07-26T10:15:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:26 smithi115 ceph-mon[35364]: pgmap v5003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:26 smithi181 ceph-mon[35869]: pgmap v5003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:26 smithi097 ceph-mon[32231]: pgmap v5003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:28 smithi115 ceph-mon[35364]: pgmap v5004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:28 smithi181 ceph-mon[35869]: pgmap v5004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:28 smithi097 ceph-mon[32231]: pgmap v5004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:30 smithi115 ceph-mon[35364]: pgmap v5005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8167.823482 secs 2024-07-26T10:15:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:30 smithi181 ceph-mon[35869]: pgmap v5005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8167.823482 secs 2024-07-26T10:15:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:30 smithi097 ceph-mon[32231]: pgmap v5005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8167.823482 secs 2024-07-26T10:15:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:32 smithi115 ceph-mon[35364]: pgmap v5006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:32 smithi181 ceph-mon[35869]: pgmap v5006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:32 smithi097 ceph-mon[32231]: pgmap v5006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:34 smithi115 ceph-mon[35364]: pgmap v5007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:34 smithi181 ceph-mon[35869]: pgmap v5007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:34 smithi097 ceph-mon[32231]: pgmap v5007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8172.823570 secs 2024-07-26T10:15:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8172.823570 secs 2024-07-26T10:15:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8172.823570 secs 2024-07-26T10:15:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:36.397+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:15:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:36.398+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:15:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:36.398+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:15:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:36.398+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:15:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:36.398+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:36 smithi115 ceph-mon[35364]: pgmap v5008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:36 smithi181 ceph-mon[35869]: pgmap v5008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:36 smithi097 ceph-mon[32231]: pgmap v5008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:38 smithi115 ceph-mon[35364]: pgmap v5009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:38 smithi181 ceph-mon[35869]: pgmap v5009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:38 smithi097 ceph-mon[32231]: pgmap v5009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:40 smithi115 ceph-mon[35364]: pgmap v5010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8177.823718 secs 2024-07-26T10:15:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:40 smithi181 ceph-mon[35869]: pgmap v5010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8177.823718 secs 2024-07-26T10:15:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:40 smithi097 ceph-mon[32231]: pgmap v5010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8177.823718 secs 2024-07-26T10:15:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:42 smithi115 ceph-mon[35364]: pgmap v5011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:42 smithi181 ceph-mon[35869]: pgmap v5011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:42 smithi097 ceph-mon[32231]: pgmap v5011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:44 smithi115 ceph-mon[35364]: pgmap v5012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:44 smithi181 ceph-mon[35869]: pgmap v5012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:44 smithi097 ceph-mon[32231]: pgmap v5012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8182.823891 secs 2024-07-26T10:15:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8182.823891 secs 2024-07-26T10:15:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8182.823891 secs 2024-07-26T10:15:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:46 smithi115 ceph-mon[35364]: pgmap v5013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:46 smithi181 ceph-mon[35869]: pgmap v5013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:46 smithi097 ceph-mon[32231]: pgmap v5013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:48 smithi115 ceph-mon[35364]: pgmap v5014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:48 smithi181 ceph-mon[35869]: pgmap v5014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:48 smithi097 ceph-mon[32231]: pgmap v5014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:50 smithi115 ceph-mon[35364]: pgmap v5015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8187.824050 secs 2024-07-26T10:15:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:50 smithi181 ceph-mon[35869]: pgmap v5015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8187.824050 secs 2024-07-26T10:15:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:50 smithi097 ceph-mon[32231]: pgmap v5015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8187.824050 secs 2024-07-26T10:15:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:51.399+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:15:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:51.401+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:15:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:51.401+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:15:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:51.401+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:15:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:15:51.401+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:52 smithi115 ceph-mon[35364]: pgmap v5016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:52 smithi181 ceph-mon[35869]: pgmap v5016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:52 smithi097 ceph-mon[32231]: pgmap v5016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:54 smithi115 ceph-mon[35364]: pgmap v5017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:54 smithi181 ceph-mon[35869]: pgmap v5017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:54 smithi097 ceph-mon[32231]: pgmap v5017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:15:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8192.824204 secs 2024-07-26T10:15:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8192.824204 secs 2024-07-26T10:15:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8192.824204 secs 2024-07-26T10:15:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:56 smithi097 ceph-mon[32231]: pgmap v5018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:56 smithi115 ceph-mon[35364]: pgmap v5018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:56 smithi181 ceph-mon[35869]: pgmap v5018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:15:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:15:57 smithi097 ceph-mon[32231]: pgmap v5019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:15:57 smithi115 ceph-mon[35364]: pgmap v5019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:15:57 smithi181 ceph-mon[35869]: pgmap v5019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:00 smithi115 ceph-mon[35364]: pgmap v5020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8197.824363 secs 2024-07-26T10:16:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:00 smithi181 ceph-mon[35869]: pgmap v5020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8197.824363 secs 2024-07-26T10:16:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:00 smithi097 ceph-mon[32231]: pgmap v5020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8197.824363 secs 2024-07-26T10:16:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:02 smithi115 ceph-mon[35364]: pgmap v5021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:02 smithi181 ceph-mon[35869]: pgmap v5021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:02 smithi097 ceph-mon[32231]: pgmap v5021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:04 smithi115 ceph-mon[35364]: pgmap v5022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:04 smithi181 ceph-mon[35869]: pgmap v5022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:04 smithi097 ceph-mon[32231]: pgmap v5022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8202.824537 secs 2024-07-26T10:16:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8202.824537 secs 2024-07-26T10:16:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8202.824537 secs 2024-07-26T10:16:06.410 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:06.407+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:16:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:06.408+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:16:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:06.408+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:16:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:06.408+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:16:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:06.409+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:06 smithi115 ceph-mon[35364]: pgmap v5023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:06 smithi181 ceph-mon[35869]: pgmap v5023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:06 smithi097 ceph-mon[32231]: pgmap v5023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:08.334+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00066a0 con 0x7ff3f01061a0 2024-07-26T10:16:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:08.334+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:16:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:08.334+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:16:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:08.335+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:16:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:08 smithi115 ceph-mon[35364]: pgmap v5024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:08 smithi181 ceph-mon[35869]: pgmap v5024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:08 smithi097 ceph-mon[32231]: pgmap v5024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:10 smithi115 ceph-mon[35364]: pgmap v5025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8207.824698 secs 2024-07-26T10:16:11.231 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:10 smithi097 ceph-mon[32231]: pgmap v5025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:11.231 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8207.824698 secs 2024-07-26T10:16:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:10 smithi181 ceph-mon[35869]: pgmap v5025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8207.824698 secs 2024-07-26T10:16:13.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:12 smithi097 ceph-mon[32231]: pgmap v5026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:12 smithi115 ceph-mon[35364]: pgmap v5026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:12 smithi181 ceph-mon[35869]: pgmap v5026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:14 smithi115 ceph-mon[35364]: pgmap v5027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:15.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:15.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:15.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:14 smithi181 ceph-mon[35869]: pgmap v5027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:15.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:14 smithi097 ceph-mon[32231]: pgmap v5027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:16:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:15 smithi115 ceph-mon[35364]: pgmap v5028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:16:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:15 smithi115 ceph-mon[35364]: pgmap v5029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8212.824831 secs 2024-07-26T10:16:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:15 smithi181 ceph-mon[35869]: pgmap v5028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:16:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:15 smithi181 ceph-mon[35869]: pgmap v5029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8212.824831 secs 2024-07-26T10:16:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:15 smithi097 ceph-mon[32231]: pgmap v5028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:16:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:15 smithi097 ceph-mon[32231]: pgmap v5029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8212.824831 secs 2024-07-26T10:16:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:17 smithi115 ceph-mon[35364]: pgmap v5030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:16:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:17 smithi181 ceph-mon[35869]: pgmap v5030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:16:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:17 smithi097 ceph-mon[32231]: pgmap v5030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:16:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:19 smithi115 ceph-mon[35364]: pgmap v5031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:16:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:19 smithi181 ceph-mon[35869]: pgmap v5031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:16:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:19 smithi097 ceph-mon[32231]: pgmap v5031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:16:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8217.825006 secs 2024-07-26T10:16:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8217.825006 secs 2024-07-26T10:16:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8217.825006 secs 2024-07-26T10:16:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:21.411+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:16:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:21.412+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:16:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:21.412+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:16:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:21.412+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:16:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:21.412+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:21 smithi115 ceph-mon[35364]: pgmap v5032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:16:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:21 smithi181 ceph-mon[35869]: pgmap v5032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:16:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:21 smithi097 ceph-mon[32231]: pgmap v5032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:16:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:23 smithi115 ceph-mon[35364]: pgmap v5033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T10:16:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:23 smithi181 ceph-mon[35869]: pgmap v5033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T10:16:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:23 smithi097 ceph-mon[32231]: pgmap v5033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T10:16:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:25 smithi115 ceph-mon[35364]: pgmap v5034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:16:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8222.825144 secs 2024-07-26T10:16:26.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:25 smithi181 ceph-mon[35869]: pgmap v5034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:16:26.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8222.825144 secs 2024-07-26T10:16:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:25 smithi097 ceph-mon[32231]: pgmap v5034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:16:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8222.825144 secs 2024-07-26T10:16:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:27 smithi115 ceph-mon[35364]: pgmap v5035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:27 smithi181 ceph-mon[35869]: pgmap v5035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:27 smithi097 ceph-mon[32231]: pgmap v5035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:29 smithi115 ceph-mon[35364]: pgmap v5036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:29 smithi181 ceph-mon[35869]: pgmap v5036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:29 smithi097 ceph-mon[32231]: pgmap v5036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8227.825285 secs 2024-07-26T10:16:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8227.825285 secs 2024-07-26T10:16:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8227.825285 secs 2024-07-26T10:16:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:31 smithi097 ceph-mon[32231]: pgmap v5037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:31 smithi115 ceph-mon[35364]: pgmap v5037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:31 smithi181 ceph-mon[35869]: pgmap v5037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:33 smithi115 ceph-mon[35364]: pgmap v5038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:33 smithi181 ceph-mon[35869]: pgmap v5038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:33 smithi097 ceph-mon[32231]: pgmap v5038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:35 smithi115 ceph-mon[35364]: pgmap v5039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8232.825384 secs 2024-07-26T10:16:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:35 smithi181 ceph-mon[35869]: pgmap v5039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8232.825384 secs 2024-07-26T10:16:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:35 smithi097 ceph-mon[32231]: pgmap v5039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8232.825384 secs 2024-07-26T10:16:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:36.419+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:16:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:36.420+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:16:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:36.420+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:16:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:36.420+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:16:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:36.420+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:37 smithi097 ceph-mon[32231]: pgmap v5040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:37 smithi115 ceph-mon[35364]: pgmap v5040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:37 smithi181 ceph-mon[35869]: pgmap v5040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:39 smithi115 ceph-mon[35364]: pgmap v5041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:39 smithi181 ceph-mon[35869]: pgmap v5041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:39 smithi097 ceph-mon[32231]: pgmap v5041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8237.825486 secs 2024-07-26T10:16:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8237.825486 secs 2024-07-26T10:16:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8237.825486 secs 2024-07-26T10:16:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:41 smithi115 ceph-mon[35364]: pgmap v5042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:16:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:41 smithi181 ceph-mon[35869]: pgmap v5042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:16:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:41 smithi097 ceph-mon[32231]: pgmap v5042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:16:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:43 smithi115 ceph-mon[35364]: pgmap v5043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:16:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:43 smithi181 ceph-mon[35869]: pgmap v5043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:16:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:43 smithi097 ceph-mon[32231]: pgmap v5043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:16:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:45 smithi115 ceph-mon[35364]: pgmap v5044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:16:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8242.825646 secs 2024-07-26T10:16:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:45 smithi181 ceph-mon[35869]: pgmap v5044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:16:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8242.825646 secs 2024-07-26T10:16:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:45 smithi097 ceph-mon[32231]: pgmap v5044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:16:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8242.825646 secs 2024-07-26T10:16:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:47 smithi115 ceph-mon[35364]: pgmap v5045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:47 smithi181 ceph-mon[35869]: pgmap v5045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:47 smithi097 ceph-mon[32231]: pgmap v5045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:49 smithi115 ceph-mon[35364]: pgmap v5046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:16:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:49 smithi181 ceph-mon[35869]: pgmap v5046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:16:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:49 smithi097 ceph-mon[32231]: pgmap v5046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:16:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8247.825785 secs 2024-07-26T10:16:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8247.825785 secs 2024-07-26T10:16:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8247.825785 secs 2024-07-26T10:16:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:51.422+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:16:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:51.423+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:16:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:51.423+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:16:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:51.423+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:16:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:16:51.423+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:51 smithi115 ceph-mon[35364]: pgmap v5047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:16:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:51 smithi181 ceph-mon[35869]: pgmap v5047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:16:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:51 smithi097 ceph-mon[32231]: pgmap v5047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:16:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:53 smithi115 ceph-mon[35364]: pgmap v5048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:16:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:53 smithi181 ceph-mon[35869]: pgmap v5048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:16:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:53 smithi097 ceph-mon[32231]: pgmap v5048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:16:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:55 smithi115 ceph-mon[35364]: pgmap v5049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8252.825943 secs 2024-07-26T10:16:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:55 smithi181 ceph-mon[35869]: pgmap v5049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8252.825943 secs 2024-07-26T10:16:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:55 smithi097 ceph-mon[32231]: pgmap v5049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:16:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8252.825943 secs 2024-07-26T10:16:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:57 smithi115 ceph-mon[35364]: pgmap v5050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:57 smithi181 ceph-mon[35869]: pgmap v5050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:16:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:57 smithi097 ceph-mon[32231]: pgmap v5050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:17:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:16:59 smithi115 ceph-mon[35364]: pgmap v5051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:16:59 smithi181 ceph-mon[35869]: pgmap v5051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:16:59 smithi097 ceph-mon[32231]: pgmap v5051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8257.826103 secs 2024-07-26T10:17:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8257.826103 secs 2024-07-26T10:17:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8257.826103 secs 2024-07-26T10:17:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:01 smithi115 ceph-mon[35364]: pgmap v5052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:01 smithi181 ceph-mon[35869]: pgmap v5052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:01 smithi097 ceph-mon[32231]: pgmap v5052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:03 smithi181 ceph-mon[35869]: pgmap v5053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:03 smithi097 ceph-mon[32231]: pgmap v5053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:03 smithi115 ceph-mon[35364]: pgmap v5053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:05 smithi181 ceph-mon[35869]: pgmap v5054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:17:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8262.826296 secs 2024-07-26T10:17:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:05 smithi097 ceph-mon[32231]: pgmap v5054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:17:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8262.826296 secs 2024-07-26T10:17:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:06.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:17:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:06.429+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:17:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:06.429+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:17:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:06.429+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:17:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:06.430+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:05 smithi115 ceph-mon[35364]: pgmap v5054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:17:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8262.826296 secs 2024-07-26T10:17:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:07 smithi097 ceph-mon[32231]: pgmap v5055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:17:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:07 smithi115 ceph-mon[35364]: pgmap v5055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:17:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:07 smithi181 ceph-mon[35869]: pgmap v5055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:17:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:09 smithi115 ceph-mon[35364]: pgmap v5056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:09 smithi181 ceph-mon[35869]: pgmap v5056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:09 smithi097 ceph-mon[32231]: pgmap v5056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8267.826474 secs 2024-07-26T10:17:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8267.826474 secs 2024-07-26T10:17:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8267.826474 secs 2024-07-26T10:17:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:11 smithi115 ceph-mon[35364]: pgmap v5057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:11 smithi181 ceph-mon[35869]: pgmap v5057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:11 smithi097 ceph-mon[32231]: pgmap v5057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:13 smithi115 ceph-mon[35364]: pgmap v5058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:13 smithi181 ceph-mon[35869]: pgmap v5058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:13 smithi097 ceph-mon[32231]: pgmap v5058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:15 smithi115 ceph-mon[35364]: pgmap v5059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:17:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8272.826560 secs 2024-07-26T10:17:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:15 smithi181 ceph-mon[35869]: pgmap v5059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:17:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8272.826560 secs 2024-07-26T10:17:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:15 smithi097 ceph-mon[32231]: pgmap v5059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:17:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8272.826560 secs 2024-07-26T10:17:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:17 smithi115 ceph-mon[35364]: pgmap v5060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:17:17.675 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:17 smithi181 ceph-mon[35869]: pgmap v5060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:17:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:17 smithi097 ceph-mon[32231]: pgmap v5060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:17:19.599 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:19 smithi181 ceph-mon[35869]: pgmap v5061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:19 smithi115 ceph-mon[35364]: pgmap v5061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:19 smithi097 ceph-mon[32231]: pgmap v5061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8277.826719 secs 2024-07-26T10:17:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8277.826719 secs 2024-07-26T10:17:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8277.826719 secs 2024-07-26T10:17:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:21.431+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:17:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:21.433+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:17:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:21.433+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:17:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:21.433+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:17:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:21.433+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: pgmap v5062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: pgmap v5063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: pgmap v5064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:17:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: pgmap v5062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: pgmap v5063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: pgmap v5064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:17:21.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: pgmap v5062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: pgmap v5063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:17:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: pgmap v5064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:17:21.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:17:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:23 smithi115 ceph-mon[35364]: pgmap v5065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:23 smithi181 ceph-mon[35869]: pgmap v5065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:23 smithi097 ceph-mon[32231]: pgmap v5065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:25 smithi115 ceph-mon[35364]: pgmap v5066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:17:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8282.826875 secs 2024-07-26T10:17:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:25 smithi181 ceph-mon[35869]: pgmap v5066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:17:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8282.826875 secs 2024-07-26T10:17:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:25 smithi097 ceph-mon[32231]: pgmap v5066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:17:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8282.826875 secs 2024-07-26T10:17:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:27 smithi115 ceph-mon[35364]: pgmap v5067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:17:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:27 smithi181 ceph-mon[35869]: pgmap v5067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:17:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:27 smithi097 ceph-mon[32231]: pgmap v5067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:17:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:29 smithi115 ceph-mon[35364]: pgmap v5068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:17:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:29 smithi181 ceph-mon[35869]: pgmap v5068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:17:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:29 smithi097 ceph-mon[32231]: pgmap v5068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:17:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8287.827038 secs 2024-07-26T10:17:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8287.827038 secs 2024-07-26T10:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8287.827038 secs 2024-07-26T10:17:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:31 smithi097 ceph-mon[32231]: pgmap v5069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:17:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:31 smithi115 ceph-mon[35364]: pgmap v5069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:17:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:31 smithi181 ceph-mon[35869]: pgmap v5069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:17:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:33 smithi097 ceph-mon[32231]: pgmap v5070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:33 smithi115 ceph-mon[35364]: pgmap v5070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:33 smithi181 ceph-mon[35869]: pgmap v5070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:35 smithi097 ceph-mon[32231]: pgmap v5071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8292.827177 secs 2024-07-26T10:17:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:35 smithi115 ceph-mon[35364]: pgmap v5071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8292.827177 secs 2024-07-26T10:17:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:35 smithi181 ceph-mon[35869]: pgmap v5071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8292.827177 secs 2024-07-26T10:17:36.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:36.439+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:17:36.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:36.441+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:17:36.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:36.441+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:17:36.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:36.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:17:36.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:36.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:37 smithi115 ceph-mon[35364]: pgmap v5072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:37 smithi181 ceph-mon[35869]: pgmap v5072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:37 smithi097 ceph-mon[32231]: pgmap v5072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:39 smithi115 ceph-mon[35364]: pgmap v5073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:39 smithi181 ceph-mon[35869]: pgmap v5073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:39 smithi097 ceph-mon[32231]: pgmap v5073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8297.827349 secs 2024-07-26T10:17:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8297.827349 secs 2024-07-26T10:17:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8297.827349 secs 2024-07-26T10:17:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:41 smithi115 ceph-mon[35364]: pgmap v5074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:41 smithi181 ceph-mon[35869]: pgmap v5074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:41 smithi097 ceph-mon[32231]: pgmap v5074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:43 smithi115 ceph-mon[35364]: pgmap v5075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:43 smithi181 ceph-mon[35869]: pgmap v5075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:43 smithi097 ceph-mon[32231]: pgmap v5075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:45 smithi115 ceph-mon[35364]: pgmap v5076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8302.827507 secs 2024-07-26T10:17:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:45 smithi181 ceph-mon[35869]: pgmap v5076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8302.827507 secs 2024-07-26T10:17:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:45 smithi097 ceph-mon[32231]: pgmap v5076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8302.827507 secs 2024-07-26T10:17:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:47 smithi115 ceph-mon[35364]: pgmap v5077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:47 smithi181 ceph-mon[35869]: pgmap v5077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:47 smithi097 ceph-mon[32231]: pgmap v5077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:49 smithi115 ceph-mon[35364]: pgmap v5078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:49 smithi181 ceph-mon[35869]: pgmap v5078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:49 smithi097 ceph-mon[32231]: pgmap v5078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8307.827662 secs 2024-07-26T10:17:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8307.827662 secs 2024-07-26T10:17:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8307.827662 secs 2024-07-26T10:17:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:51.443+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:17:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:51.443+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:17:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:51.443+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:17:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:51.443+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:17:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:17:51.443+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:51 smithi115 ceph-mon[35364]: pgmap v5079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:51 smithi181 ceph-mon[35869]: pgmap v5079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:51 smithi097 ceph-mon[32231]: pgmap v5079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:53 smithi115 ceph-mon[35364]: pgmap v5080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:53 smithi181 ceph-mon[35869]: pgmap v5080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:53 smithi097 ceph-mon[32231]: pgmap v5080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:55 smithi115 ceph-mon[35364]: pgmap v5081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8312.827759 secs 2024-07-26T10:17:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:55 smithi181 ceph-mon[35869]: pgmap v5081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8312.827759 secs 2024-07-26T10:17:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:55 smithi097 ceph-mon[32231]: pgmap v5081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8312.827759 secs 2024-07-26T10:17:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:57 smithi115 ceph-mon[35364]: pgmap v5082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:57 smithi181 ceph-mon[35869]: pgmap v5082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:57 smithi097 ceph-mon[32231]: pgmap v5082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:17:59 smithi115 ceph-mon[35364]: pgmap v5083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:17:59 smithi181 ceph-mon[35869]: pgmap v5083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:17:59 smithi097 ceph-mon[32231]: pgmap v5083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8317.827920 secs 2024-07-26T10:18:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8317.827920 secs 2024-07-26T10:18:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8317.827920 secs 2024-07-26T10:18:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:01 smithi115 ceph-mon[35364]: pgmap v5084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:01 smithi181 ceph-mon[35869]: pgmap v5084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:01 smithi097 ceph-mon[32231]: pgmap v5084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:03 smithi115 ceph-mon[35364]: pgmap v5085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:03 smithi181 ceph-mon[35869]: pgmap v5085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:03 smithi097 ceph-mon[32231]: pgmap v5085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:05 smithi115 ceph-mon[35364]: pgmap v5086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8322.828073 secs 2024-07-26T10:18:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:05 smithi181 ceph-mon[35869]: pgmap v5086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8322.828073 secs 2024-07-26T10:18:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:05 smithi097 ceph-mon[32231]: pgmap v5086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8322.828073 secs 2024-07-26T10:18:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:06.449+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:18:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:06.450+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:18:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:06.450+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:18:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:06.450+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:18:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:06.450+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:07 smithi115 ceph-mon[35364]: pgmap v5087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:07 smithi181 ceph-mon[35869]: pgmap v5087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:07 smithi097 ceph-mon[32231]: pgmap v5087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:09 smithi115 ceph-mon[35364]: pgmap v5088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:09 smithi181 ceph-mon[35869]: pgmap v5088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:09 smithi097 ceph-mon[32231]: pgmap v5088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8327.828275 secs 2024-07-26T10:18:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8327.828275 secs 2024-07-26T10:18:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8327.828275 secs 2024-07-26T10:18:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:11 smithi115 ceph-mon[35364]: pgmap v5089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:11 smithi181 ceph-mon[35869]: pgmap v5089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:11 smithi097 ceph-mon[32231]: pgmap v5089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:13 smithi115 ceph-mon[35364]: pgmap v5090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:13 smithi181 ceph-mon[35869]: pgmap v5090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:13 smithi097 ceph-mon[32231]: pgmap v5090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:15 smithi115 ceph-mon[35364]: pgmap v5091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8332.828432 secs 2024-07-26T10:18:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:15 smithi181 ceph-mon[35869]: pgmap v5091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8332.828432 secs 2024-07-26T10:18:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:15 smithi097 ceph-mon[32231]: pgmap v5091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8332.828432 secs 2024-07-26T10:18:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:17 smithi115 ceph-mon[35364]: pgmap v5092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:17 smithi181 ceph-mon[35869]: pgmap v5092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:17 smithi097 ceph-mon[32231]: pgmap v5092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:19 smithi115 ceph-mon[35364]: pgmap v5093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:19 smithi181 ceph-mon[35869]: pgmap v5093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:19 smithi097 ceph-mon[32231]: pgmap v5093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8337.828567 secs 2024-07-26T10:18:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8337.828567 secs 2024-07-26T10:18:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8337.828567 secs 2024-07-26T10:18:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:21.452+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:18:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:21.453+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:18:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:21.453+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:18:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:21.453+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:18:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:21.453+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:21 smithi115 ceph-mon[35364]: pgmap v5094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:22.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:21 smithi181 ceph-mon[35869]: pgmap v5094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:21 smithi097 ceph-mon[32231]: pgmap v5094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:18:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:18:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:18:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:23.357+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:18:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:23.357+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00064c0 con 0x7ff3f01061a0 2024-07-26T10:18:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:23.358+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:18:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:23.358+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:18:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:23 smithi115 ceph-mon[35364]: pgmap v5095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:23 smithi181 ceph-mon[35869]: pgmap v5095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:23 smithi097 ceph-mon[32231]: pgmap v5095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:26.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:25 smithi181 ceph-mon[35869]: pgmap v5096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:18:26.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8342.828707 secs 2024-07-26T10:18:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:25 smithi097 ceph-mon[32231]: pgmap v5096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:18:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8342.828707 secs 2024-07-26T10:18:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:25 smithi115 ceph-mon[35364]: pgmap v5096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:18:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8342.828707 secs 2024-07-26T10:18:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:27 smithi097 ceph-mon[32231]: pgmap v5097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:28.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:27 smithi115 ceph-mon[35364]: pgmap v5097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:27 smithi181 ceph-mon[35869]: pgmap v5097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:29 smithi097 ceph-mon[32231]: pgmap v5098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:18:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:29 smithi115 ceph-mon[35364]: pgmap v5098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:18:30.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:29 smithi181 ceph-mon[35869]: pgmap v5098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:18:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8347.828865 secs 2024-07-26T10:18:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8347.828865 secs 2024-07-26T10:18:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8347.828865 secs 2024-07-26T10:18:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:32 smithi115 ceph-mon[35364]: pgmap v5099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:18:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:32 smithi181 ceph-mon[35869]: pgmap v5099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:18:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:32 smithi097 ceph-mon[32231]: pgmap v5099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:18:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:33 smithi115 ceph-mon[35364]: pgmap v5100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:33 smithi181 ceph-mon[35869]: pgmap v5100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:33 smithi097 ceph-mon[32231]: pgmap v5100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:18:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:35 smithi115 ceph-mon[35364]: pgmap v5101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:18:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:35 smithi181 ceph-mon[35869]: pgmap v5101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:18:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:35 smithi097 ceph-mon[32231]: pgmap v5101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:18:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:36.459+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:18:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:36.460+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:18:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:36.460+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:18:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:36.460+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:18:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:36.460+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8352.829021 secs 2024-07-26T10:18:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8352.829021 secs 2024-07-26T10:18:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8352.829021 secs 2024-07-26T10:18:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:37 smithi115 ceph-mon[35364]: pgmap v5102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:37 smithi181 ceph-mon[35869]: pgmap v5102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:37 smithi097 ceph-mon[32231]: pgmap v5102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:39 smithi115 ceph-mon[35364]: pgmap v5103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:39 smithi181 ceph-mon[35869]: pgmap v5103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:39 smithi097 ceph-mon[32231]: pgmap v5103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8357.829187 secs 2024-07-26T10:18:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:41 smithi115 ceph-mon[35364]: pgmap v5104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:18:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8357.829187 secs 2024-07-26T10:18:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:41 smithi181 ceph-mon[35869]: pgmap v5104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:18:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8357.829187 secs 2024-07-26T10:18:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:41 smithi097 ceph-mon[32231]: pgmap v5104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:18:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:43 smithi115 ceph-mon[35364]: pgmap v5105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:18:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:43 smithi181 ceph-mon[35869]: pgmap v5105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:18:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:43 smithi097 ceph-mon[32231]: pgmap v5105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:18:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:45 smithi115 ceph-mon[35364]: pgmap v5106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8362.829346 secs 2024-07-26T10:18:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:45 smithi181 ceph-mon[35869]: pgmap v5106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8362.829346 secs 2024-07-26T10:18:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:45 smithi097 ceph-mon[32231]: pgmap v5106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8362.829346 secs 2024-07-26T10:18:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:47 smithi115 ceph-mon[35364]: pgmap v5107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:47 smithi181 ceph-mon[35869]: pgmap v5107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:47 smithi097 ceph-mon[32231]: pgmap v5107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:49 smithi181 ceph-mon[35869]: pgmap v5108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:18:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:49 smithi097 ceph-mon[32231]: pgmap v5108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:18:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:49 smithi115 ceph-mon[35364]: pgmap v5108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:18:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8367.829472 secs 2024-07-26T10:18:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8367.829472 secs 2024-07-26T10:18:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8367.829472 secs 2024-07-26T10:18:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:51.461+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:18:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:51.462+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:18:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:51.462+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:18:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:51.462+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:18:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:18:51.462+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:51 smithi097 ceph-mon[32231]: pgmap v5109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:18:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:51 smithi115 ceph-mon[35364]: pgmap v5109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:18:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:51 smithi181 ceph-mon[35869]: pgmap v5109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:18:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:53 smithi097 ceph-mon[32231]: pgmap v5110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:53 smithi115 ceph-mon[35364]: pgmap v5110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:53 smithi181 ceph-mon[35869]: pgmap v5110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:55 smithi115 ceph-mon[35364]: pgmap v5111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:18:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8372.829615 secs 2024-07-26T10:18:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:55 smithi181 ceph-mon[35869]: pgmap v5111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:18:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8372.829615 secs 2024-07-26T10:18:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:55 smithi097 ceph-mon[32231]: pgmap v5111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:18:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8372.829615 secs 2024-07-26T10:18:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:57 smithi115 ceph-mon[35364]: pgmap v5112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:57 smithi181 ceph-mon[35869]: pgmap v5112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:57 smithi097 ceph-mon[32231]: pgmap v5112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:59.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:18:59 smithi115 ceph-mon[35364]: pgmap v5113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:59.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:18:59 smithi181 ceph-mon[35869]: pgmap v5113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:59.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:18:59 smithi097 ceph-mon[32231]: pgmap v5113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8377.829778 secs 2024-07-26T10:19:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8377.829778 secs 2024-07-26T10:19:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8377.829778 secs 2024-07-26T10:19:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:01 smithi115 ceph-mon[35364]: pgmap v5114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:01 smithi181 ceph-mon[35869]: pgmap v5114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:01 smithi097 ceph-mon[32231]: pgmap v5114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:03 smithi115 ceph-mon[35364]: pgmap v5115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:03 smithi181 ceph-mon[35869]: pgmap v5115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:03 smithi097 ceph-mon[32231]: pgmap v5115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:05 smithi115 ceph-mon[35364]: pgmap v5116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8382.829936 secs 2024-07-26T10:19:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:05 smithi181 ceph-mon[35869]: pgmap v5116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8382.829936 secs 2024-07-26T10:19:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:05 smithi097 ceph-mon[32231]: pgmap v5116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8382.829936 secs 2024-07-26T10:19:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:06.469+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:19:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:06.470+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:19:06.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:06.470+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:19:06.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:06.470+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:19:06.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:06.470+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:07 smithi115 ceph-mon[35364]: pgmap v5117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:19:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:07 smithi181 ceph-mon[35869]: pgmap v5117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:19:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:07 smithi097 ceph-mon[32231]: pgmap v5117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:19:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:08.365+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:19:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:08.365+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T10:19:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:08.366+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T10:19:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:08.366+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:19:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:09 smithi115 ceph-mon[35364]: pgmap v5118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:09 smithi181 ceph-mon[35869]: pgmap v5118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:09 smithi097 ceph-mon[32231]: pgmap v5118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8387.830090 secs 2024-07-26T10:19:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8387.830090 secs 2024-07-26T10:19:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8387.830090 secs 2024-07-26T10:19:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:11 smithi115 ceph-mon[35364]: pgmap v5119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:19:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:11 smithi181 ceph-mon[35869]: pgmap v5119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:19:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:11 smithi097 ceph-mon[32231]: pgmap v5119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:19:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:13 smithi115 ceph-mon[35364]: pgmap v5120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:19:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:13 smithi181 ceph-mon[35869]: pgmap v5120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:19:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:13 smithi097 ceph-mon[32231]: pgmap v5120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:19:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:15 smithi115 ceph-mon[35364]: pgmap v5121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8392.830214 secs 2024-07-26T10:19:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:15 smithi181 ceph-mon[35869]: pgmap v5121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8392.830214 secs 2024-07-26T10:19:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:15 smithi097 ceph-mon[32231]: pgmap v5121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8392.830214 secs 2024-07-26T10:19:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:17 smithi115 ceph-mon[35364]: pgmap v5122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:17 smithi181 ceph-mon[35869]: pgmap v5122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:17 smithi097 ceph-mon[32231]: pgmap v5122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:19 smithi115 ceph-mon[35364]: pgmap v5123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:19 smithi181 ceph-mon[35869]: pgmap v5123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:19 smithi097 ceph-mon[32231]: pgmap v5123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8397.830363 secs 2024-07-26T10:19:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8397.830363 secs 2024-07-26T10:19:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8397.830363 secs 2024-07-26T10:19:21.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:21.473+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:19:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:21.474+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:19:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:21.474+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:19:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:21.474+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:19:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:21.474+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:21 smithi115 ceph-mon[35364]: pgmap v5124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:21 smithi181 ceph-mon[35869]: pgmap v5124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:21 smithi097 ceph-mon[32231]: pgmap v5124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:23 smithi115 ceph-mon[35364]: pgmap v5125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:23 smithi181 ceph-mon[35869]: pgmap v5125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:23 smithi097 ceph-mon[32231]: pgmap v5125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:24.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:24.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:24 smithi115 ceph-mon[35364]: pgmap v5126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:24.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:24 smithi115 ceph-mon[35364]: pgmap v5127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:24.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:19:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:24 smithi181 ceph-mon[35869]: pgmap v5126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:24 smithi181 ceph-mon[35869]: pgmap v5127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:19:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:24 smithi097 ceph-mon[32231]: pgmap v5126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:19:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:24 smithi097 ceph-mon[32231]: pgmap v5127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:19:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8402.830502 secs 2024-07-26T10:19:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8402.830502 secs 2024-07-26T10:19:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8402.830502 secs 2024-07-26T10:19:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:26 smithi115 ceph-mon[35364]: pgmap v5128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:26 smithi181 ceph-mon[35869]: pgmap v5128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:26 smithi097 ceph-mon[32231]: pgmap v5128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:28.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:28 smithi115 ceph-mon[35364]: pgmap v5129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:28 smithi181 ceph-mon[35869]: pgmap v5129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:28 smithi097 ceph-mon[32231]: pgmap v5129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:30 smithi115 ceph-mon[35364]: pgmap v5130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8407.830662 secs 2024-07-26T10:19:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:30 smithi181 ceph-mon[35869]: pgmap v5130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8407.830662 secs 2024-07-26T10:19:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:30 smithi097 ceph-mon[32231]: pgmap v5130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8407.830662 secs 2024-07-26T10:19:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:32 smithi115 ceph-mon[35364]: pgmap v5131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:32 smithi181 ceph-mon[35869]: pgmap v5131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:32 smithi097 ceph-mon[32231]: pgmap v5131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:34 smithi115 ceph-mon[35364]: pgmap v5132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:34 smithi181 ceph-mon[35869]: pgmap v5132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:34 smithi097 ceph-mon[32231]: pgmap v5132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8412.830752 secs 2024-07-26T10:19:35.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8412.830752 secs 2024-07-26T10:19:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8412.830752 secs 2024-07-26T10:19:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:36.480+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:19:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:36.482+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:19:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:36.482+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:19:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:36.482+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:19:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:36.482+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:36 smithi115 ceph-mon[35364]: pgmap v5133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:36 smithi181 ceph-mon[35869]: pgmap v5133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:36 smithi097 ceph-mon[32231]: pgmap v5133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:38 smithi115 ceph-mon[35364]: pgmap v5134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:38 smithi181 ceph-mon[35869]: pgmap v5134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:38 smithi097 ceph-mon[32231]: pgmap v5134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:40 smithi115 ceph-mon[35364]: pgmap v5135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8417.830908 secs 2024-07-26T10:19:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:40 smithi181 ceph-mon[35869]: pgmap v5135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8417.830908 secs 2024-07-26T10:19:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:40 smithi097 ceph-mon[32231]: pgmap v5135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8417.830908 secs 2024-07-26T10:19:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:42 smithi115 ceph-mon[35364]: pgmap v5136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:42 smithi181 ceph-mon[35869]: pgmap v5136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:42 smithi097 ceph-mon[32231]: pgmap v5136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:44 smithi115 ceph-mon[35364]: pgmap v5137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:44 smithi181 ceph-mon[35869]: pgmap v5137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:44 smithi097 ceph-mon[32231]: pgmap v5137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8422.831068 secs 2024-07-26T10:19:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8422.831068 secs 2024-07-26T10:19:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8422.831068 secs 2024-07-26T10:19:46.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:46 smithi115 ceph-mon[35364]: pgmap v5138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:46 smithi181 ceph-mon[35869]: pgmap v5138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:46 smithi097 ceph-mon[32231]: pgmap v5138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:48 smithi115 ceph-mon[35364]: pgmap v5139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:48 smithi181 ceph-mon[35869]: pgmap v5139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:48 smithi097 ceph-mon[32231]: pgmap v5139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:50 smithi115 ceph-mon[35364]: pgmap v5140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8427.831267 secs 2024-07-26T10:19:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:50 smithi181 ceph-mon[35869]: pgmap v5140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8427.831267 secs 2024-07-26T10:19:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:50 smithi097 ceph-mon[32231]: pgmap v5140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8427.831267 secs 2024-07-26T10:19:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:51.483+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:19:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:51.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:19:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:51.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:19:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:51.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:19:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:19:51.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:52 smithi115 ceph-mon[35364]: pgmap v5141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:52.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:52 smithi181 ceph-mon[35869]: pgmap v5141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:52 smithi097 ceph-mon[32231]: pgmap v5141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:54 smithi115 ceph-mon[35364]: pgmap v5142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:54 smithi181 ceph-mon[35869]: pgmap v5142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:54 smithi097 ceph-mon[32231]: pgmap v5142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8432.831396 secs 2024-07-26T10:19:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8432.831396 secs 2024-07-26T10:19:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8432.831396 secs 2024-07-26T10:19:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:56 smithi181 ceph-mon[35869]: pgmap v5143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:56 smithi097 ceph-mon[32231]: pgmap v5143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:56 smithi115 ceph-mon[35364]: pgmap v5143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:19:58 smithi097 ceph-mon[32231]: pgmap v5144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:19:58 smithi115 ceph-mon[35364]: pgmap v5144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:19:58 smithi181 ceph-mon[35869]: pgmap v5144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:00 smithi097 ceph-mon[32231]: pgmap v5145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8437.831529 secs 2024-07-26T10:20:01.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:20:01.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:00 smithi115 ceph-mon[35364]: pgmap v5145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:01.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8437.831529 secs 2024-07-26T10:20:01.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:20:01.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:00 smithi181 ceph-mon[35869]: pgmap v5145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:01.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8437.831529 secs 2024-07-26T10:20:01.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:20:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:02 smithi097 ceph-mon[32231]: pgmap v5146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:02 smithi115 ceph-mon[35364]: pgmap v5146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:02 smithi181 ceph-mon[35869]: pgmap v5146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:04 smithi115 ceph-mon[35364]: pgmap v5147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:04 smithi181 ceph-mon[35869]: pgmap v5147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:04 smithi097 ceph-mon[32231]: pgmap v5147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8442.831708 secs 2024-07-26T10:20:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8442.831708 secs 2024-07-26T10:20:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8442.831708 secs 2024-07-26T10:20:06.493 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:06.490+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:20:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:06.492+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:20:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:06.492+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:20:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:06.492+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:20:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:06.492+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:06 smithi115 ceph-mon[35364]: pgmap v5148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:06 smithi181 ceph-mon[35869]: pgmap v5148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:06 smithi097 ceph-mon[32231]: pgmap v5148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:08 smithi115 ceph-mon[35364]: pgmap v5149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:08 smithi181 ceph-mon[35869]: pgmap v5149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:08 smithi097 ceph-mon[32231]: pgmap v5149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:10 smithi115 ceph-mon[35364]: pgmap v5150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8447.831792 secs 2024-07-26T10:20:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:10 smithi181 ceph-mon[35869]: pgmap v5150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8447.831792 secs 2024-07-26T10:20:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:10 smithi097 ceph-mon[32231]: pgmap v5150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8447.831792 secs 2024-07-26T10:20:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:12 smithi115 ceph-mon[35364]: pgmap v5151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:12 smithi181 ceph-mon[35869]: pgmap v5151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:12 smithi097 ceph-mon[32231]: pgmap v5151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:14 smithi115 ceph-mon[35364]: pgmap v5152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:14 smithi181 ceph-mon[35869]: pgmap v5152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:14 smithi097 ceph-mon[32231]: pgmap v5152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8452.831953 secs 2024-07-26T10:20:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8452.831953 secs 2024-07-26T10:20:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8452.831953 secs 2024-07-26T10:20:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:16 smithi115 ceph-mon[35364]: pgmap v5153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:16 smithi181 ceph-mon[35869]: pgmap v5153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:16 smithi097 ceph-mon[32231]: pgmap v5153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:18 smithi097 ceph-mon[32231]: pgmap v5154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:18 smithi115 ceph-mon[35364]: pgmap v5154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:18 smithi181 ceph-mon[35869]: pgmap v5154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:21 smithi115 ceph-mon[35364]: pgmap v5155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8457.832080 secs 2024-07-26T10:20:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:21 smithi181 ceph-mon[35869]: pgmap v5155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8457.832080 secs 2024-07-26T10:20:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:21.493+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:20:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:21.495+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:20:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:21.495+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:20:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:21.495+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:20:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:21.495+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:21 smithi097 ceph-mon[32231]: pgmap v5155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8457.832080 secs 2024-07-26T10:20:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:23 smithi115 ceph-mon[35364]: pgmap v5156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:23 smithi181 ceph-mon[35869]: pgmap v5156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:23 smithi097 ceph-mon[32231]: pgmap v5156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:25 smithi115 ceph-mon[35364]: pgmap v5157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:20:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:25 smithi181 ceph-mon[35869]: pgmap v5157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:20:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:25 smithi097 ceph-mon[32231]: pgmap v5157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:20:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8462.832210 secs 2024-07-26T10:20:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:26 smithi115 ceph-mon[35364]: pgmap v5158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:20:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:26 smithi115 ceph-mon[35364]: pgmap v5159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8462.832210 secs 2024-07-26T10:20:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:26 smithi181 ceph-mon[35869]: pgmap v5158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:20:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:26 smithi181 ceph-mon[35869]: pgmap v5159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8462.832210 secs 2024-07-26T10:20:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:26 smithi097 ceph-mon[32231]: pgmap v5158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:20:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:26 smithi097 ceph-mon[32231]: pgmap v5159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:27 smithi115 ceph-mon[35364]: pgmap v5160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:20:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:27 smithi181 ceph-mon[35869]: pgmap v5160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:20:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:27 smithi097 ceph-mon[32231]: pgmap v5160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:20:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:30 smithi115 ceph-mon[35364]: pgmap v5161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:30 smithi181 ceph-mon[35869]: pgmap v5161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:30 smithi097 ceph-mon[32231]: pgmap v5161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:31.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8467.832370 secs 2024-07-26T10:20:31.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8467.832370 secs 2024-07-26T10:20:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8467.832370 secs 2024-07-26T10:20:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:32 smithi115 ceph-mon[35364]: pgmap v5162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:32 smithi181 ceph-mon[35869]: pgmap v5162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:32.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:32 smithi097 ceph-mon[32231]: pgmap v5162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:34 smithi115 ceph-mon[35364]: pgmap v5163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:20:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:34 smithi181 ceph-mon[35869]: pgmap v5163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:20:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:34 smithi097 ceph-mon[32231]: pgmap v5163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:20:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:35 smithi115 ceph-mon[35364]: pgmap v5164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:20:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:35 smithi181 ceph-mon[35869]: pgmap v5164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:20:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:35 smithi097 ceph-mon[32231]: pgmap v5164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:20:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8472.832530 secs 2024-07-26T10:20:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8472.832530 secs 2024-07-26T10:20:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8472.832530 secs 2024-07-26T10:20:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:36.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:20:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:36.502+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:20:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:36.502+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:20:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:36.502+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:20:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:36.502+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:37 smithi115 ceph-mon[35364]: pgmap v5165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:37 smithi181 ceph-mon[35869]: pgmap v5165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:37 smithi097 ceph-mon[32231]: pgmap v5165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:40 smithi115 ceph-mon[35364]: pgmap v5166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:40 smithi181 ceph-mon[35869]: pgmap v5166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:40 smithi097 ceph-mon[32231]: pgmap v5166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8477.832701 secs 2024-07-26T10:20:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8477.832701 secs 2024-07-26T10:20:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8477.832701 secs 2024-07-26T10:20:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:42 smithi115 ceph-mon[35364]: pgmap v5167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:42 smithi181 ceph-mon[35869]: pgmap v5167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:42 smithi097 ceph-mon[32231]: pgmap v5167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:44 smithi115 ceph-mon[35364]: pgmap v5168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:44 smithi181 ceph-mon[35869]: pgmap v5168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:44 smithi097 ceph-mon[32231]: pgmap v5168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:45 smithi115 ceph-mon[35364]: pgmap v5169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:45 smithi181 ceph-mon[35869]: pgmap v5169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:45 smithi097 ceph-mon[32231]: pgmap v5169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8482.832797 secs 2024-07-26T10:20:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8482.832797 secs 2024-07-26T10:20:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8482.832797 secs 2024-07-26T10:20:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:47 smithi115 ceph-mon[35364]: pgmap v5170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:47 smithi181 ceph-mon[35869]: pgmap v5170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:47 smithi097 ceph-mon[32231]: pgmap v5170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:50 smithi115 ceph-mon[35364]: pgmap v5171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:50 smithi181 ceph-mon[35869]: pgmap v5171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:50 smithi097 ceph-mon[32231]: pgmap v5171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8487.832970 secs 2024-07-26T10:20:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8487.832970 secs 2024-07-26T10:20:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8487.832970 secs 2024-07-26T10:20:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:51.503+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:20:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:51.504+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:20:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:51.504+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:20:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:51.504+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:20:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:20:51.504+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:52 smithi115 ceph-mon[35364]: pgmap v5172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:52 smithi181 ceph-mon[35869]: pgmap v5172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:52 smithi097 ceph-mon[32231]: pgmap v5172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:54 smithi115 ceph-mon[35364]: pgmap v5173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:54 smithi181 ceph-mon[35869]: pgmap v5173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:54 smithi097 ceph-mon[32231]: pgmap v5173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:55 smithi115 ceph-mon[35364]: pgmap v5174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:55 smithi181 ceph-mon[35869]: pgmap v5174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:55 smithi097 ceph-mon[32231]: pgmap v5174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8492.833130 secs 2024-07-26T10:20:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8492.833130 secs 2024-07-26T10:20:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8492.833130 secs 2024-07-26T10:20:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:20:57 smithi115 ceph-mon[35364]: pgmap v5175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:20:57 smithi181 ceph-mon[35869]: pgmap v5175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:20:57 smithi097 ceph-mon[32231]: pgmap v5175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:00 smithi115 ceph-mon[35364]: pgmap v5176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:00.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:00 smithi181 ceph-mon[35869]: pgmap v5176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:00 smithi097 ceph-mon[32231]: pgmap v5176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:01.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8497.833288 secs 2024-07-26T10:21:01.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8497.833288 secs 2024-07-26T10:21:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8497.833288 secs 2024-07-26T10:21:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:02 smithi115 ceph-mon[35364]: pgmap v5177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:02 smithi181 ceph-mon[35869]: pgmap v5177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:02 smithi097 ceph-mon[32231]: pgmap v5177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:03 smithi115 ceph-mon[35364]: pgmap v5178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:03 smithi181 ceph-mon[35869]: pgmap v5178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:03 smithi097 ceph-mon[32231]: pgmap v5178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8502.833451 secs 2024-07-26T10:21:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:06 smithi115 ceph-mon[35364]: pgmap v5179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8502.833451 secs 2024-07-26T10:21:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:06 smithi181 ceph-mon[35869]: pgmap v5179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8502.833451 secs 2024-07-26T10:21:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:06 smithi097 ceph-mon[32231]: pgmap v5179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:06.513 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:06.511+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:21:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:06.512+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:21:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:06.512+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:21:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:06.512+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:21:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:06.512+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:08 smithi115 ceph-mon[35364]: pgmap v5180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:08 smithi181 ceph-mon[35869]: pgmap v5180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:08 smithi097 ceph-mon[32231]: pgmap v5180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:10 smithi115 ceph-mon[35364]: pgmap v5181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:10 smithi181 ceph-mon[35869]: pgmap v5181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:10 smithi097 ceph-mon[32231]: pgmap v5181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8507.833586 secs 2024-07-26T10:21:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8507.833586 secs 2024-07-26T10:21:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8507.833586 secs 2024-07-26T10:21:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:12 smithi115 ceph-mon[35364]: pgmap v5182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:12 smithi181 ceph-mon[35869]: pgmap v5182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:12 smithi097 ceph-mon[32231]: pgmap v5182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:13 smithi115 ceph-mon[35364]: pgmap v5183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:13 smithi181 ceph-mon[35869]: pgmap v5183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:13 smithi097 ceph-mon[32231]: pgmap v5183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8512.833726 secs 2024-07-26T10:21:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:16 smithi115 ceph-mon[35364]: pgmap v5184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8512.833726 secs 2024-07-26T10:21:16.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:16 smithi181 ceph-mon[35869]: pgmap v5184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8512.833726 secs 2024-07-26T10:21:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:16 smithi097 ceph-mon[32231]: pgmap v5184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:18 smithi115 ceph-mon[35364]: pgmap v5185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:18 smithi181 ceph-mon[35869]: pgmap v5185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:18 smithi097 ceph-mon[32231]: pgmap v5185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:20 smithi115 ceph-mon[35364]: pgmap v5186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:20 smithi181 ceph-mon[35869]: pgmap v5186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:20 smithi097 ceph-mon[32231]: pgmap v5186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8517.833807 secs 2024-07-26T10:21:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8517.833807 secs 2024-07-26T10:21:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8517.833807 secs 2024-07-26T10:21:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:21.513+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:21:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:21.515+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:21:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:21.515+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:21:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:21.515+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:21:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:21.515+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:22 smithi115 ceph-mon[35364]: pgmap v5187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:22 smithi181 ceph-mon[35869]: pgmap v5187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:22 smithi097 ceph-mon[32231]: pgmap v5187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:23 smithi115 ceph-mon[35364]: pgmap v5188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:23 smithi181 ceph-mon[35869]: pgmap v5188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:23 smithi097 ceph-mon[32231]: pgmap v5188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8522.833901 secs 2024-07-26T10:21:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:26 smithi115 ceph-mon[35364]: pgmap v5189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8522.833901 secs 2024-07-26T10:21:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:26 smithi181 ceph-mon[35869]: pgmap v5189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8522.833901 secs 2024-07-26T10:21:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:26 smithi097 ceph-mon[32231]: pgmap v5189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:27.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:27.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:27.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:21:27.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:21:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:28 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:21:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:28 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:21:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:28 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:21:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:28 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:21:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:28 smithi115 ceph-mon[35364]: pgmap v5190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:21:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:28 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:21:28.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:28 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:21:28.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:28 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:21:28.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:28 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:21:28.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:28 smithi181 ceph-mon[35869]: pgmap v5190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:21:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:28 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:21:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:28 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:21:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:28 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:21:28.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:28 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:21:28.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:28 smithi097 ceph-mon[32231]: pgmap v5190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:21:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:29 smithi115 ceph-mon[35364]: pgmap v5191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:21:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:29 smithi181 ceph-mon[35869]: pgmap v5191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:21:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:29 smithi097 ceph-mon[32231]: pgmap v5191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:21:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8527.833999 secs 2024-07-26T10:21:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8527.833999 secs 2024-07-26T10:21:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8527.833999 secs 2024-07-26T10:21:31.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:31 smithi115 ceph-mon[35364]: pgmap v5192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:21:31.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:31 smithi181 ceph-mon[35869]: pgmap v5192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:21:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:31 smithi097 ceph-mon[32231]: pgmap v5192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:21:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:33 smithi097 ceph-mon[32231]: pgmap v5193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:21:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:33 smithi115 ceph-mon[35364]: pgmap v5193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:21:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:33 smithi181 ceph-mon[35869]: pgmap v5193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:21:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:35 smithi097 ceph-mon[32231]: pgmap v5194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:21:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8532.834096 secs 2024-07-26T10:21:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:35 smithi115 ceph-mon[35364]: pgmap v5194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:21:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8532.834096 secs 2024-07-26T10:21:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:35 smithi181 ceph-mon[35869]: pgmap v5194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:21:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8532.834096 secs 2024-07-26T10:21:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:36.521+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:21:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:36.522+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:21:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:36.522+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:21:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:36.522+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:21:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:36.522+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:37 smithi097 ceph-mon[32231]: pgmap v5195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:21:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:37 smithi115 ceph-mon[35364]: pgmap v5195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:21:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:37 smithi181 ceph-mon[35869]: pgmap v5195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:21:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:39 smithi115 ceph-mon[35364]: pgmap v5196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:39 smithi181 ceph-mon[35869]: pgmap v5196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:39 smithi097 ceph-mon[32231]: pgmap v5196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8537.834195 secs 2024-07-26T10:21:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8537.834195 secs 2024-07-26T10:21:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8537.834195 secs 2024-07-26T10:21:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:41 smithi115 ceph-mon[35364]: pgmap v5197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:41 smithi181 ceph-mon[35869]: pgmap v5197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:41 smithi097 ceph-mon[32231]: pgmap v5197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:43 smithi115 ceph-mon[35364]: pgmap v5198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:43 smithi181 ceph-mon[35869]: pgmap v5198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:43 smithi097 ceph-mon[32231]: pgmap v5198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:45 smithi115 ceph-mon[35364]: pgmap v5199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8542.834304 secs 2024-07-26T10:21:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:45 smithi181 ceph-mon[35869]: pgmap v5199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8542.834304 secs 2024-07-26T10:21:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:45 smithi097 ceph-mon[32231]: pgmap v5199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8542.834304 secs 2024-07-26T10:21:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:47 smithi115 ceph-mon[35364]: pgmap v5200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:47 smithi181 ceph-mon[35869]: pgmap v5200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:47 smithi097 ceph-mon[32231]: pgmap v5200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:49 smithi115 ceph-mon[35364]: pgmap v5201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:49 smithi181 ceph-mon[35869]: pgmap v5201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:49 smithi097 ceph-mon[32231]: pgmap v5201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8547.834414 secs 2024-07-26T10:21:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8547.834414 secs 2024-07-26T10:21:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8547.834414 secs 2024-07-26T10:21:51.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:51.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:21:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:51.525+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:21:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:51.525+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:21:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:51.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:21:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:21:51.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:51 smithi115 ceph-mon[35364]: pgmap v5202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:21:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:51 smithi181 ceph-mon[35869]: pgmap v5202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:21:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:51 smithi097 ceph-mon[32231]: pgmap v5202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:21:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:53 smithi115 ceph-mon[35364]: pgmap v5203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:21:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:53 smithi181 ceph-mon[35869]: pgmap v5203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:21:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:53 smithi097 ceph-mon[32231]: pgmap v5203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:21:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:55 smithi115 ceph-mon[35364]: pgmap v5204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:21:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8552.834547 secs 2024-07-26T10:21:55.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:55 smithi181 ceph-mon[35869]: pgmap v5204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:21:55.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8552.834547 secs 2024-07-26T10:21:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:55 smithi097 ceph-mon[32231]: pgmap v5204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:21:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8552.834547 secs 2024-07-26T10:21:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:57 smithi115 ceph-mon[35364]: pgmap v5205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:21:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:57 smithi181 ceph-mon[35869]: pgmap v5205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:21:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:57 smithi097 ceph-mon[32231]: pgmap v5205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:21:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:21:59 smithi115 ceph-mon[35364]: pgmap v5206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:21:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:21:59 smithi181 ceph-mon[35869]: pgmap v5206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:22:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:21:59 smithi097 ceph-mon[32231]: pgmap v5206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:22:00.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8557.834709 secs 2024-07-26T10:22:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8557.834709 secs 2024-07-26T10:22:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8557.834709 secs 2024-07-26T10:22:01.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:01 smithi115 ceph-mon[35364]: pgmap v5207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:22:01.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:01 smithi181 ceph-mon[35869]: pgmap v5207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:22:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:01 smithi097 ceph-mon[32231]: pgmap v5207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:22:03.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:03 smithi115 ceph-mon[35364]: pgmap v5208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:22:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:03 smithi181 ceph-mon[35869]: pgmap v5208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:22:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:03 smithi097 ceph-mon[32231]: pgmap v5208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:22:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:05 smithi115 ceph-mon[35364]: pgmap v5209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8562.834863 secs 2024-07-26T10:22:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:05 smithi181 ceph-mon[35869]: pgmap v5209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8562.834863 secs 2024-07-26T10:22:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:05 smithi097 ceph-mon[32231]: pgmap v5209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8562.834863 secs 2024-07-26T10:22:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:06.531+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:22:06.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:06.532+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:22:06.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:06.533+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:22:06.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:06.533+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:22:06.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:06.533+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:07 smithi115 ceph-mon[35364]: pgmap v5210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:07 smithi181 ceph-mon[35869]: pgmap v5210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:07 smithi097 ceph-mon[32231]: pgmap v5210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:08.398+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:22:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:08.398+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T10:22:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:08.399+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:22:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:08.399+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:22:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:09 smithi115 ceph-mon[35364]: pgmap v5211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:09 smithi181 ceph-mon[35869]: pgmap v5211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:09 smithi097 ceph-mon[32231]: pgmap v5211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8567.835025 secs 2024-07-26T10:22:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8567.835025 secs 2024-07-26T10:22:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8567.835025 secs 2024-07-26T10:22:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:11 smithi115 ceph-mon[35364]: pgmap v5212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:11 smithi181 ceph-mon[35869]: pgmap v5212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:11 smithi097 ceph-mon[32231]: pgmap v5212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:13 smithi115 ceph-mon[35364]: pgmap v5213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:13 smithi181 ceph-mon[35869]: pgmap v5213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:13 smithi097 ceph-mon[32231]: pgmap v5213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:15 smithi115 ceph-mon[35364]: pgmap v5214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8572.835195 secs 2024-07-26T10:22:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:15 smithi181 ceph-mon[35869]: pgmap v5214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8572.835195 secs 2024-07-26T10:22:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:15 smithi097 ceph-mon[32231]: pgmap v5214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8572.835195 secs 2024-07-26T10:22:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:17 smithi115 ceph-mon[35364]: pgmap v5215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:17 smithi181 ceph-mon[35869]: pgmap v5215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:17 smithi097 ceph-mon[32231]: pgmap v5215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:19 smithi115 ceph-mon[35364]: pgmap v5216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:19 smithi181 ceph-mon[35869]: pgmap v5216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:19 smithi097 ceph-mon[32231]: pgmap v5216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8577.835374 secs 2024-07-26T10:22:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8577.835374 secs 2024-07-26T10:22:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8577.835374 secs 2024-07-26T10:22:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:21.534+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:22:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:21.536+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:22:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:21.536+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:22:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:21.536+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:22:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:21.536+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:21 smithi115 ceph-mon[35364]: pgmap v5217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:21 smithi181 ceph-mon[35869]: pgmap v5217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:21 smithi097 ceph-mon[32231]: pgmap v5217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:23 smithi115 ceph-mon[35364]: pgmap v5218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:23 smithi181 ceph-mon[35869]: pgmap v5218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:23 smithi097 ceph-mon[32231]: pgmap v5218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:25 smithi115 ceph-mon[35364]: pgmap v5219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8582.835471 secs 2024-07-26T10:22:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:25 smithi181 ceph-mon[35869]: pgmap v5219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8582.835471 secs 2024-07-26T10:22:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:25 smithi097 ceph-mon[32231]: pgmap v5219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:26.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8582.835471 secs 2024-07-26T10:22:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:27 smithi181 ceph-mon[35869]: pgmap v5220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:27.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:27 smithi097 ceph-mon[32231]: pgmap v5220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:27 smithi115 ceph-mon[35364]: pgmap v5220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: pgmap v5221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:29.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:22:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: pgmap v5221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:22:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: pgmap v5221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:22:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8587.835561 secs 2024-07-26T10:22:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:30 smithi115 ceph-mon[35364]: pgmap v5222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8587.835561 secs 2024-07-26T10:22:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:30 smithi181 ceph-mon[35869]: pgmap v5222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8587.835561 secs 2024-07-26T10:22:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:30 smithi097 ceph-mon[32231]: pgmap v5222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:32 smithi115 ceph-mon[35364]: pgmap v5223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:22:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:32 smithi181 ceph-mon[35869]: pgmap v5223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:22:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:32 smithi097 ceph-mon[32231]: pgmap v5223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:22:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:34 smithi115 ceph-mon[35364]: pgmap v5224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T10:22:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:34 smithi181 ceph-mon[35869]: pgmap v5224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T10:22:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:34 smithi097 ceph-mon[32231]: pgmap v5224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T10:22:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8592.835645 secs 2024-07-26T10:22:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8592.835645 secs 2024-07-26T10:22:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8592.835645 secs 2024-07-26T10:22:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:36.540+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:22:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:36.541+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:22:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:36.541+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:22:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:36.541+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:22:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:36.541+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:36 smithi115 ceph-mon[35364]: pgmap v5225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:22:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:36 smithi181 ceph-mon[35869]: pgmap v5225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:22:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:36 smithi097 ceph-mon[32231]: pgmap v5225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:22:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:38 smithi181 ceph-mon[35869]: pgmap v5226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:38 smithi097 ceph-mon[32231]: pgmap v5226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:38 smithi115 ceph-mon[35364]: pgmap v5226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:22:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8597.835789 secs 2024-07-26T10:22:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:40 smithi097 ceph-mon[32231]: pgmap v5227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8597.835789 secs 2024-07-26T10:22:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:40 smithi115 ceph-mon[35364]: pgmap v5227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8597.835789 secs 2024-07-26T10:22:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:40 smithi181 ceph-mon[35869]: pgmap v5227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:42 smithi097 ceph-mon[32231]: pgmap v5228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:42 smithi115 ceph-mon[35364]: pgmap v5228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:42 smithi181 ceph-mon[35869]: pgmap v5228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:44 smithi097 ceph-mon[32231]: pgmap v5229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:22:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:44 smithi115 ceph-mon[35364]: pgmap v5229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:22:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:44 smithi181 ceph-mon[35869]: pgmap v5229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:22:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8602.835947 secs 2024-07-26T10:22:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8602.835947 secs 2024-07-26T10:22:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8602.835947 secs 2024-07-26T10:22:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:47 smithi115 ceph-mon[35364]: pgmap v5230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:22:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:47 smithi181 ceph-mon[35869]: pgmap v5230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:22:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:47 smithi097 ceph-mon[32231]: pgmap v5230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:22:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:49 smithi115 ceph-mon[35364]: pgmap v5231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:49 smithi181 ceph-mon[35869]: pgmap v5231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:49 smithi097 ceph-mon[32231]: pgmap v5231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8607.836044 secs 2024-07-26T10:22:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:51 smithi115 ceph-mon[35364]: pgmap v5232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8607.836044 secs 2024-07-26T10:22:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:51 smithi181 ceph-mon[35869]: pgmap v5232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8607.836044 secs 2024-07-26T10:22:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:51 smithi097 ceph-mon[32231]: pgmap v5232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:51.543+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:22:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:51.544+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:22:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:51.544+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:22:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:51.544+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:22:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:22:51.544+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:53 smithi115 ceph-mon[35364]: pgmap v5233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:53 smithi181 ceph-mon[35869]: pgmap v5233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:53 smithi097 ceph-mon[32231]: pgmap v5233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:55 smithi115 ceph-mon[35364]: pgmap v5234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:22:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:55 smithi181 ceph-mon[35869]: pgmap v5234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:22:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:55 smithi097 ceph-mon[32231]: pgmap v5234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:22:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8612.836225 secs 2024-07-26T10:22:56.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8612.836225 secs 2024-07-26T10:22:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8612.836225 secs 2024-07-26T10:22:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:57 smithi115 ceph-mon[35364]: pgmap v5235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:22:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:57 smithi181 ceph-mon[35869]: pgmap v5235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:22:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:57 smithi097 ceph-mon[32231]: pgmap v5235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:22:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:22:59 smithi115 ceph-mon[35364]: pgmap v5236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:22:59 smithi181 ceph-mon[35869]: pgmap v5236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:22:59 smithi097 ceph-mon[32231]: pgmap v5236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:00 smithi115 ceph-mon[35364]: pgmap v5237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:00 smithi181 ceph-mon[35869]: pgmap v5237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:00 smithi097 ceph-mon[32231]: pgmap v5237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:01.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8617.836372 secs 2024-07-26T10:23:01.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8617.836372 secs 2024-07-26T10:23:01.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8617.836372 secs 2024-07-26T10:23:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:02 smithi115 ceph-mon[35364]: pgmap v5238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:02 smithi181 ceph-mon[35869]: pgmap v5238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:02 smithi097 ceph-mon[32231]: pgmap v5238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:04 smithi115 ceph-mon[35364]: pgmap v5239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:04 smithi181 ceph-mon[35869]: pgmap v5239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:04 smithi097 ceph-mon[32231]: pgmap v5239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8622.836475 secs 2024-07-26T10:23:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8622.836475 secs 2024-07-26T10:23:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8622.836475 secs 2024-07-26T10:23:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:06.551+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:23:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:06.553+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:23:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:06.553+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:23:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:06.553+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:23:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:06.553+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:06 smithi181 ceph-mon[35869]: pgmap v5240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:06 smithi097 ceph-mon[32231]: pgmap v5240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:06 smithi115 ceph-mon[35364]: pgmap v5240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:08 smithi181 ceph-mon[35869]: pgmap v5241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:08 smithi097 ceph-mon[32231]: pgmap v5241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:08 smithi115 ceph-mon[35364]: pgmap v5241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8627.836560 secs 2024-07-26T10:23:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:10 smithi097 ceph-mon[32231]: pgmap v5242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8627.836560 secs 2024-07-26T10:23:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:10 smithi115 ceph-mon[35364]: pgmap v5242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8627.836560 secs 2024-07-26T10:23:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:10 smithi181 ceph-mon[35869]: pgmap v5242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:12 smithi097 ceph-mon[32231]: pgmap v5243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:12 smithi115 ceph-mon[35364]: pgmap v5243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:12 smithi181 ceph-mon[35869]: pgmap v5243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:15 smithi115 ceph-mon[35364]: pgmap v5244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:15 smithi181 ceph-mon[35869]: pgmap v5244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:15 smithi097 ceph-mon[32231]: pgmap v5244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8632.836694 secs 2024-07-26T10:23:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8632.836694 secs 2024-07-26T10:23:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8632.836694 secs 2024-07-26T10:23:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:17 smithi115 ceph-mon[35364]: pgmap v5245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:17 smithi181 ceph-mon[35869]: pgmap v5245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:17 smithi097 ceph-mon[32231]: pgmap v5245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:19 smithi115 ceph-mon[35364]: pgmap v5246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:19 smithi181 ceph-mon[35869]: pgmap v5246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:19 smithi097 ceph-mon[32231]: pgmap v5246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8637.836836 secs 2024-07-26T10:23:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:21 smithi115 ceph-mon[35364]: pgmap v5247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8637.836836 secs 2024-07-26T10:23:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:21 smithi181 ceph-mon[35869]: pgmap v5247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8637.836836 secs 2024-07-26T10:23:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:21 smithi097 ceph-mon[32231]: pgmap v5247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:21.554+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:23:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:21.555+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:23:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:21.556+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:23:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:21.556+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:23:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:21.556+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:23 smithi115 ceph-mon[35364]: pgmap v5248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:23 smithi181 ceph-mon[35869]: pgmap v5248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:23 smithi097 ceph-mon[32231]: pgmap v5248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:25 smithi115 ceph-mon[35364]: pgmap v5249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:25 smithi181 ceph-mon[35869]: pgmap v5249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:25 smithi097 ceph-mon[32231]: pgmap v5249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8642.836974 secs 2024-07-26T10:23:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8642.836974 secs 2024-07-26T10:23:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8642.836974 secs 2024-07-26T10:23:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:27 smithi115 ceph-mon[35364]: pgmap v5250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:27 smithi181 ceph-mon[35869]: pgmap v5250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:27 smithi097 ceph-mon[32231]: pgmap v5250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:29 smithi115 ceph-mon[35364]: pgmap v5251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:29.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:29.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:29.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:29 smithi181 ceph-mon[35869]: pgmap v5251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:29.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:29 smithi097 ceph-mon[32231]: pgmap v5251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:23:31.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:31 smithi115 ceph-mon[35364]: pgmap v5252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:31.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:31 smithi115 ceph-mon[35364]: pgmap v5253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:31.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8647.837110 secs 2024-07-26T10:23:31.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:31 smithi181 ceph-mon[35869]: pgmap v5252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:31.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:31 smithi181 ceph-mon[35869]: pgmap v5253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:31.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8647.837110 secs 2024-07-26T10:23:31.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:31 smithi097 ceph-mon[32231]: pgmap v5252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:31.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:31 smithi097 ceph-mon[32231]: pgmap v5253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:31.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8647.837110 secs 2024-07-26T10:23:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:32 smithi115 ceph-mon[35364]: pgmap v5254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:32 smithi181 ceph-mon[35869]: pgmap v5254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:32 smithi097 ceph-mon[32231]: pgmap v5254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:34 smithi115 ceph-mon[35364]: pgmap v5255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:34 smithi181 ceph-mon[35869]: pgmap v5255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:34 smithi097 ceph-mon[32231]: pgmap v5255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8652.837283 secs 2024-07-26T10:23:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8652.837283 secs 2024-07-26T10:23:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8652.837283 secs 2024-07-26T10:23:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:36.562+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:23:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:36.563+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:23:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:36.563+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:23:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:36.563+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:23:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:36.563+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:36 smithi115 ceph-mon[35364]: pgmap v5256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:36 smithi181 ceph-mon[35869]: pgmap v5256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:36 smithi097 ceph-mon[32231]: pgmap v5256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:38 smithi115 ceph-mon[35364]: pgmap v5257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:38 smithi181 ceph-mon[35869]: pgmap v5257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:38 smithi097 ceph-mon[32231]: pgmap v5257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:40 smithi115 ceph-mon[35364]: pgmap v5258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:23:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8657.837392 secs 2024-07-26T10:23:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:40 smithi181 ceph-mon[35869]: pgmap v5258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:23:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8657.837392 secs 2024-07-26T10:23:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:40 smithi097 ceph-mon[32231]: pgmap v5258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:23:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8657.837392 secs 2024-07-26T10:23:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:42 smithi115 ceph-mon[35364]: pgmap v5259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:42 smithi181 ceph-mon[35869]: pgmap v5259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:42 smithi097 ceph-mon[32231]: pgmap v5259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:44 smithi115 ceph-mon[35364]: pgmap v5260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:23:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:44 smithi181 ceph-mon[35869]: pgmap v5260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:23:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:44 smithi097 ceph-mon[32231]: pgmap v5260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:23:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8662.837538 secs 2024-07-26T10:23:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8662.837538 secs 2024-07-26T10:23:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8662.837538 secs 2024-07-26T10:23:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:46 smithi115 ceph-mon[35364]: pgmap v5261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:46 smithi181 ceph-mon[35869]: pgmap v5261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:46 smithi097 ceph-mon[32231]: pgmap v5261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:48 smithi115 ceph-mon[35364]: pgmap v5262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:23:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:48 smithi181 ceph-mon[35869]: pgmap v5262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:23:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:48 smithi097 ceph-mon[32231]: pgmap v5262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:23:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:50 smithi115 ceph-mon[35364]: pgmap v5263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:23:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8667.837635 secs 2024-07-26T10:23:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:50 smithi181 ceph-mon[35869]: pgmap v5263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:23:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8667.837635 secs 2024-07-26T10:23:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:50 smithi097 ceph-mon[32231]: pgmap v5263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:23:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8667.837635 secs 2024-07-26T10:23:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:51.565+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:23:51.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:51.567+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:23:51.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:51.567+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:23:51.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:51.567+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:23:51.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:23:51.567+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:52 smithi115 ceph-mon[35364]: pgmap v5264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:23:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:52 smithi181 ceph-mon[35869]: pgmap v5264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:23:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:52 smithi097 ceph-mon[32231]: pgmap v5264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:23:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:54 smithi115 ceph-mon[35364]: pgmap v5265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:23:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:54 smithi181 ceph-mon[35869]: pgmap v5265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:23:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:54 smithi097 ceph-mon[32231]: pgmap v5265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:23:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8672.837778 secs 2024-07-26T10:23:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8672.837778 secs 2024-07-26T10:23:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8672.837778 secs 2024-07-26T10:23:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:56 smithi115 ceph-mon[35364]: pgmap v5266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:56 smithi181 ceph-mon[35869]: pgmap v5266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:56 smithi097 ceph-mon[32231]: pgmap v5266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:23:58 smithi181 ceph-mon[35869]: pgmap v5267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:23:58 smithi097 ceph-mon[32231]: pgmap v5267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:23:58 smithi115 ceph-mon[35364]: pgmap v5267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:00 smithi181 ceph-mon[35869]: pgmap v5268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8677.837926 secs 2024-07-26T10:24:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:00 smithi097 ceph-mon[32231]: pgmap v5268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8677.837926 secs 2024-07-26T10:24:01.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:00 smithi115 ceph-mon[35364]: pgmap v5268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:01.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8677.837926 secs 2024-07-26T10:24:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:02 smithi097 ceph-mon[32231]: pgmap v5269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:02 smithi115 ceph-mon[35364]: pgmap v5269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:02 smithi181 ceph-mon[35869]: pgmap v5269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:04 smithi115 ceph-mon[35364]: pgmap v5270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:04.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:04 smithi181 ceph-mon[35869]: pgmap v5270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:04 smithi097 ceph-mon[32231]: pgmap v5270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8682.838066 secs 2024-07-26T10:24:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8682.838066 secs 2024-07-26T10:24:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8682.838066 secs 2024-07-26T10:24:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:06.571+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:24:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:06.573+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:24:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:06.573+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:24:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:06.573+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:24:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:06.573+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:06 smithi115 ceph-mon[35364]: pgmap v5271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:06 smithi181 ceph-mon[35869]: pgmap v5271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:06 smithi097 ceph-mon[32231]: pgmap v5271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:08 smithi115 ceph-mon[35364]: pgmap v5272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:08 smithi181 ceph-mon[35869]: pgmap v5272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:08 smithi097 ceph-mon[32231]: pgmap v5272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8687.838207 secs 2024-07-26T10:24:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:10 smithi115 ceph-mon[35364]: pgmap v5273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8687.838207 secs 2024-07-26T10:24:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:10 smithi181 ceph-mon[35869]: pgmap v5273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8687.838207 secs 2024-07-26T10:24:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:10 smithi097 ceph-mon[32231]: pgmap v5273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:12 smithi115 ceph-mon[35364]: pgmap v5274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:12 smithi181 ceph-mon[35869]: pgmap v5274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:12 smithi097 ceph-mon[32231]: pgmap v5274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:14 smithi115 ceph-mon[35364]: pgmap v5275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:14 smithi181 ceph-mon[35869]: pgmap v5275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:14 smithi097 ceph-mon[32231]: pgmap v5275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8692.838348 secs 2024-07-26T10:24:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8692.838348 secs 2024-07-26T10:24:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8692.838348 secs 2024-07-26T10:24:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:16 smithi115 ceph-mon[35364]: pgmap v5276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:16 smithi181 ceph-mon[35869]: pgmap v5276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:16 smithi097 ceph-mon[32231]: pgmap v5276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:18 smithi115 ceph-mon[35364]: pgmap v5277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:18 smithi181 ceph-mon[35869]: pgmap v5277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:18 smithi097 ceph-mon[32231]: pgmap v5277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8697.838434 secs 2024-07-26T10:24:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:20 smithi115 ceph-mon[35364]: pgmap v5278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8697.838434 secs 2024-07-26T10:24:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:20 smithi181 ceph-mon[35869]: pgmap v5278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8697.838434 secs 2024-07-26T10:24:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:20 smithi097 ceph-mon[32231]: pgmap v5278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:21.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:21.574+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:24:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:21.575+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:24:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:21.576+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:24:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:21.576+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:24:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:21.576+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:22 smithi115 ceph-mon[35364]: pgmap v5279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:22 smithi181 ceph-mon[35869]: pgmap v5279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:22 smithi097 ceph-mon[32231]: pgmap v5279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:24 smithi115 ceph-mon[35364]: pgmap v5280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:24 smithi181 ceph-mon[35869]: pgmap v5280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:24 smithi097 ceph-mon[32231]: pgmap v5280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8702.838566 secs 2024-07-26T10:24:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8702.838566 secs 2024-07-26T10:24:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8702.838566 secs 2024-07-26T10:24:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:26 smithi097 ceph-mon[32231]: pgmap v5281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:26 smithi115 ceph-mon[35364]: pgmap v5281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:26 smithi181 ceph-mon[35869]: pgmap v5281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:28 smithi097 ceph-mon[32231]: pgmap v5282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:28 smithi115 ceph-mon[35364]: pgmap v5282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:28 smithi181 ceph-mon[35869]: pgmap v5282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:31.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8707.838712 secs 2024-07-26T10:24:31.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:31 smithi115 ceph-mon[35364]: pgmap v5283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:31.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:31.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.287 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8707.838712 secs 2024-07-26T10:24:31.287 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:31 smithi181 ceph-mon[35869]: pgmap v5283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:31.287 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:31.287 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.287 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.287 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.287 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8707.838712 secs 2024-07-26T10:24:31.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:31 smithi097 ceph-mon[32231]: pgmap v5283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:31.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:31.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:31.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:32.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:24:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:33 smithi115 ceph-mon[35364]: pgmap v5284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:24:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:33 smithi115 ceph-mon[35364]: pgmap v5285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:33 smithi181 ceph-mon[35869]: pgmap v5284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:24:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:33 smithi181 ceph-mon[35869]: pgmap v5285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:33 smithi097 ceph-mon[32231]: pgmap v5284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:24:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:33 smithi097 ceph-mon[32231]: pgmap v5285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:35 smithi115 ceph-mon[35364]: pgmap v5286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:35 smithi181 ceph-mon[35869]: pgmap v5286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:35 smithi097 ceph-mon[32231]: pgmap v5286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8712.838855 secs 2024-07-26T10:24:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8712.838855 secs 2024-07-26T10:24:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8712.838855 secs 2024-07-26T10:24:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:36.582+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:24:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:36.583+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:24:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:36.583+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:24:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:36.583+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:24:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:36.583+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:37 smithi115 ceph-mon[35364]: pgmap v5287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:37 smithi181 ceph-mon[35869]: pgmap v5287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:37 smithi097 ceph-mon[32231]: pgmap v5287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:39 smithi115 ceph-mon[35364]: pgmap v5288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:39 smithi181 ceph-mon[35869]: pgmap v5288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:39 smithi097 ceph-mon[32231]: pgmap v5288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:41 smithi115 ceph-mon[35364]: pgmap v5289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8717.838995 secs 2024-07-26T10:24:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:41 smithi181 ceph-mon[35869]: pgmap v5289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8717.838995 secs 2024-07-26T10:24:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:41 smithi097 ceph-mon[32231]: pgmap v5289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8717.838995 secs 2024-07-26T10:24:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:43 smithi115 ceph-mon[35364]: pgmap v5290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:43 smithi181 ceph-mon[35869]: pgmap v5290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:43 smithi097 ceph-mon[32231]: pgmap v5290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:45 smithi115 ceph-mon[35364]: pgmap v5291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:45 smithi181 ceph-mon[35869]: pgmap v5291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:45 smithi097 ceph-mon[32231]: pgmap v5291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8722.839137 secs 2024-07-26T10:24:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8722.839137 secs 2024-07-26T10:24:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8722.839137 secs 2024-07-26T10:24:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:47 smithi115 ceph-mon[35364]: pgmap v5292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:47 smithi181 ceph-mon[35869]: pgmap v5292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:47 smithi097 ceph-mon[32231]: pgmap v5292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:48 smithi115 ceph-mon[35364]: pgmap v5293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:48 smithi181 ceph-mon[35869]: pgmap v5293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:48 smithi097 ceph-mon[32231]: pgmap v5293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:50 smithi115 ceph-mon[35364]: pgmap v5294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8727.839278 secs 2024-07-26T10:24:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:50 smithi181 ceph-mon[35869]: pgmap v5294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8727.839278 secs 2024-07-26T10:24:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:50 smithi097 ceph-mon[32231]: pgmap v5294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8727.839278 secs 2024-07-26T10:24:51.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:51.584+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:24:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:51.586+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:24:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:51.586+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:24:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:51.586+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:24:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:24:51.586+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:52 smithi115 ceph-mon[35364]: pgmap v5295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:52 smithi181 ceph-mon[35869]: pgmap v5295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:52 smithi097 ceph-mon[32231]: pgmap v5295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:54 smithi115 ceph-mon[35364]: pgmap v5296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:54 smithi181 ceph-mon[35869]: pgmap v5296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:54 smithi097 ceph-mon[32231]: pgmap v5296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8732.839380 secs 2024-07-26T10:24:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8732.839380 secs 2024-07-26T10:24:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8732.839380 secs 2024-07-26T10:24:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:56 smithi115 ceph-mon[35364]: pgmap v5297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:56 smithi181 ceph-mon[35869]: pgmap v5297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:56 smithi097 ceph-mon[32231]: pgmap v5297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:24:58 smithi115 ceph-mon[35364]: pgmap v5298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:24:58 smithi181 ceph-mon[35869]: pgmap v5298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:24:58 smithi097 ceph-mon[32231]: pgmap v5298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:00 smithi115 ceph-mon[35364]: pgmap v5299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8737.839515 secs 2024-07-26T10:25:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:00 smithi181 ceph-mon[35869]: pgmap v5299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8737.839515 secs 2024-07-26T10:25:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:00 smithi097 ceph-mon[32231]: pgmap v5299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8737.839515 secs 2024-07-26T10:25:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:02 smithi115 ceph-mon[35364]: pgmap v5300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:02 smithi181 ceph-mon[35869]: pgmap v5300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:02 smithi097 ceph-mon[32231]: pgmap v5300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:04 smithi115 ceph-mon[35364]: pgmap v5301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:04 smithi181 ceph-mon[35869]: pgmap v5301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:04 smithi097 ceph-mon[32231]: pgmap v5301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8742.839645 secs 2024-07-26T10:25:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8742.839645 secs 2024-07-26T10:25:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8742.839645 secs 2024-07-26T10:25:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:06.593+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:25:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:06.594+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:25:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:06.594+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:25:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:06.594+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:25:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:06.594+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:06 smithi115 ceph-mon[35364]: pgmap v5302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:06 smithi181 ceph-mon[35869]: pgmap v5302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:06 smithi097 ceph-mon[32231]: pgmap v5302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:08.430+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:25:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:08.430+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:25:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:08.431+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:25:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:08.431+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:25:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:08 smithi115 ceph-mon[35364]: pgmap v5303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:08 smithi181 ceph-mon[35869]: pgmap v5303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:08 smithi097 ceph-mon[32231]: pgmap v5303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:10 smithi115 ceph-mon[35364]: pgmap v5304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8747.839788 secs 2024-07-26T10:25:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:10 smithi181 ceph-mon[35869]: pgmap v5304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8747.839788 secs 2024-07-26T10:25:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:10 smithi097 ceph-mon[32231]: pgmap v5304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8747.839788 secs 2024-07-26T10:25:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:12 smithi115 ceph-mon[35364]: pgmap v5305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:12 smithi181 ceph-mon[35869]: pgmap v5305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:12 smithi097 ceph-mon[32231]: pgmap v5305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:14 smithi115 ceph-mon[35364]: pgmap v5306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:14 smithi181 ceph-mon[35869]: pgmap v5306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:14 smithi097 ceph-mon[32231]: pgmap v5306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8752.839947 secs 2024-07-26T10:25:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8752.839947 secs 2024-07-26T10:25:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8752.839947 secs 2024-07-26T10:25:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:16 smithi097 ceph-mon[32231]: pgmap v5307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:16 smithi115 ceph-mon[35364]: pgmap v5307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:16 smithi181 ceph-mon[35869]: pgmap v5307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:18 smithi097 ceph-mon[32231]: pgmap v5308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:18 smithi115 ceph-mon[35364]: pgmap v5308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:18 smithi181 ceph-mon[35869]: pgmap v5308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:20 smithi097 ceph-mon[32231]: pgmap v5309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8757.840109 secs 2024-07-26T10:25:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:20 smithi115 ceph-mon[35364]: pgmap v5309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8757.840109 secs 2024-07-26T10:25:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:20 smithi181 ceph-mon[35869]: pgmap v5309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8757.840109 secs 2024-07-26T10:25:21.597 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:21.595+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:25:21.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:21.596+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:25:21.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:21.596+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:25:21.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:21.596+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:25:21.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:21.596+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:23 smithi115 ceph-mon[35364]: pgmap v5310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:23 smithi181 ceph-mon[35869]: pgmap v5310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:23 smithi097 ceph-mon[32231]: pgmap v5310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:25 smithi115 ceph-mon[35364]: pgmap v5311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:25 smithi181 ceph-mon[35869]: pgmap v5311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:25 smithi097 ceph-mon[32231]: pgmap v5311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8762.840306 secs 2024-07-26T10:25:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8762.840306 secs 2024-07-26T10:25:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8762.840306 secs 2024-07-26T10:25:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:27 smithi115 ceph-mon[35364]: pgmap v5312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:27 smithi181 ceph-mon[35869]: pgmap v5312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:27 smithi097 ceph-mon[32231]: pgmap v5312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:29 smithi115 ceph-mon[35364]: pgmap v5313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:29 smithi181 ceph-mon[35869]: pgmap v5313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:29 smithi097 ceph-mon[32231]: pgmap v5313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:31.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:31 smithi097 ceph-mon[32231]: pgmap v5314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:31.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8767.840442 secs 2024-07-26T10:25:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:31 smithi115 ceph-mon[35364]: pgmap v5314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8767.840442 secs 2024-07-26T10:25:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:31 smithi181 ceph-mon[35869]: pgmap v5314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8767.840442 secs 2024-07-26T10:25:33.361 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:33 smithi097 ceph-mon[32231]: pgmap v5315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:33.361 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:33.384 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:33 smithi115 ceph-mon[35364]: pgmap v5315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:33.384 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:33.413 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:33 smithi181 ceph-mon[35869]: pgmap v5315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:33.414 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:35 smithi115 ceph-mon[35364]: pgmap v5316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:35 smithi181 ceph-mon[35869]: pgmap v5316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:35 smithi097 ceph-mon[32231]: pgmap v5316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8772.840547 secs 2024-07-26T10:25:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:36 smithi115 ceph-mon[35364]: pgmap v5317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8772.840547 secs 2024-07-26T10:25:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:36 smithi181 ceph-mon[35869]: pgmap v5317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8772.840547 secs 2024-07-26T10:25:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:36 smithi097 ceph-mon[32231]: pgmap v5317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:36.602+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:25:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:36.603+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:25:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:36.603+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:25:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:36.603+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:25:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:36.603+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:37 smithi115 ceph-mon[35364]: pgmap v5318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:37 smithi115 ceph-mon[35364]: pgmap v5319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:25:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:37 smithi181 ceph-mon[35869]: pgmap v5318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:37.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:37 smithi181 ceph-mon[35869]: pgmap v5319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:25:37.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:37 smithi097 ceph-mon[32231]: pgmap v5318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:37 smithi097 ceph-mon[32231]: pgmap v5319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:25:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:25:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:39 smithi115 ceph-mon[35364]: pgmap v5320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:25:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:39 smithi181 ceph-mon[35869]: pgmap v5320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:25:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:39 smithi097 ceph-mon[32231]: pgmap v5320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:25:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8777.840674 secs 2024-07-26T10:25:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8777.840674 secs 2024-07-26T10:25:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8777.840674 secs 2024-07-26T10:25:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:41 smithi115 ceph-mon[35364]: pgmap v5321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:41 smithi181 ceph-mon[35869]: pgmap v5321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:41 smithi097 ceph-mon[32231]: pgmap v5321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:43 smithi115 ceph-mon[35364]: pgmap v5322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:25:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:43 smithi181 ceph-mon[35869]: pgmap v5322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:25:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:43 smithi097 ceph-mon[32231]: pgmap v5322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:25:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:45 smithi115 ceph-mon[35364]: pgmap v5323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8782.840814 secs 2024-07-26T10:25:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:45 smithi181 ceph-mon[35869]: pgmap v5323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8782.840814 secs 2024-07-26T10:25:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:45 smithi097 ceph-mon[32231]: pgmap v5323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8782.840814 secs 2024-07-26T10:25:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:47 smithi115 ceph-mon[35364]: pgmap v5324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:25:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:47 smithi181 ceph-mon[35869]: pgmap v5324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:25:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:47 smithi097 ceph-mon[32231]: pgmap v5324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:25:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:49 smithi115 ceph-mon[35364]: pgmap v5325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:49 smithi181 ceph-mon[35869]: pgmap v5325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:49 smithi097 ceph-mon[32231]: pgmap v5325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8787.840907 secs 2024-07-26T10:25:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8787.840907 secs 2024-07-26T10:25:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8787.840907 secs 2024-07-26T10:25:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:51.605+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:25:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:51.606+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:25:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:51.606+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:25:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:51.606+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:25:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:25:51.606+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:51 smithi115 ceph-mon[35364]: pgmap v5326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:51 smithi181 ceph-mon[35869]: pgmap v5326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:51 smithi097 ceph-mon[32231]: pgmap v5326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:53 smithi115 ceph-mon[35364]: pgmap v5327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:53 smithi181 ceph-mon[35869]: pgmap v5327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:53 smithi097 ceph-mon[32231]: pgmap v5327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:55 smithi115 ceph-mon[35364]: pgmap v5328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8792.841049 secs 2024-07-26T10:25:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:55 smithi181 ceph-mon[35869]: pgmap v5328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8792.841049 secs 2024-07-26T10:25:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:55 smithi097 ceph-mon[32231]: pgmap v5328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8792.841049 secs 2024-07-26T10:25:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:57 smithi115 ceph-mon[35364]: pgmap v5329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:57 smithi181 ceph-mon[35869]: pgmap v5329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:57 smithi097 ceph-mon[32231]: pgmap v5329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:25:59 smithi115 ceph-mon[35364]: pgmap v5330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:25:59 smithi181 ceph-mon[35869]: pgmap v5330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:25:59 smithi097 ceph-mon[32231]: pgmap v5330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8797.841197 secs 2024-07-26T10:26:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8797.841197 secs 2024-07-26T10:26:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8797.841197 secs 2024-07-26T10:26:01.934 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:01 smithi097 ceph-mon[32231]: pgmap v5331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:01 smithi115 ceph-mon[35364]: pgmap v5331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:01 smithi181 ceph-mon[35869]: pgmap v5331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:03 smithi115 ceph-mon[35364]: pgmap v5332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:26:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:03 smithi181 ceph-mon[35869]: pgmap v5332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:26:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:03 smithi097 ceph-mon[32231]: pgmap v5332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:26:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:05 smithi115 ceph-mon[35364]: pgmap v5333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8802.841335 secs 2024-07-26T10:26:05.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:05 smithi181 ceph-mon[35869]: pgmap v5333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:05.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8802.841335 secs 2024-07-26T10:26:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:05 smithi097 ceph-mon[32231]: pgmap v5333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8802.841335 secs 2024-07-26T10:26:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:06.612+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:26:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:06.614+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:26:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:06.614+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:26:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:06.614+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:26:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:06.614+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:07 smithi115 ceph-mon[35364]: pgmap v5334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:07 smithi181 ceph-mon[35869]: pgmap v5334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:07 smithi097 ceph-mon[32231]: pgmap v5334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:09 smithi115 ceph-mon[35364]: pgmap v5335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:09 smithi181 ceph-mon[35869]: pgmap v5335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:09 smithi097 ceph-mon[32231]: pgmap v5335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8807.841512 secs 2024-07-26T10:26:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8807.841512 secs 2024-07-26T10:26:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8807.841512 secs 2024-07-26T10:26:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:11 smithi115 ceph-mon[35364]: pgmap v5336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:11 smithi181 ceph-mon[35869]: pgmap v5336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:11 smithi097 ceph-mon[32231]: pgmap v5336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:13 smithi181 ceph-mon[35869]: pgmap v5337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:13 smithi097 ceph-mon[32231]: pgmap v5337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:13 smithi115 ceph-mon[35364]: pgmap v5337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:15 smithi097 ceph-mon[32231]: pgmap v5338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8812.841635 secs 2024-07-26T10:26:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:15 smithi115 ceph-mon[35364]: pgmap v5338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8812.841635 secs 2024-07-26T10:26:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:15 smithi181 ceph-mon[35869]: pgmap v5338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8812.841635 secs 2024-07-26T10:26:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:17 smithi097 ceph-mon[32231]: pgmap v5339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:17 smithi115 ceph-mon[35364]: pgmap v5339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:17 smithi181 ceph-mon[35869]: pgmap v5339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:19 smithi097 ceph-mon[32231]: pgmap v5340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:19 smithi115 ceph-mon[35364]: pgmap v5340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:19 smithi181 ceph-mon[35869]: pgmap v5340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8817.841798 secs 2024-07-26T10:26:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8817.841798 secs 2024-07-26T10:26:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8817.841798 secs 2024-07-26T10:26:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:21.616+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:26:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:21.617+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:26:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:21.617+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:26:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:21.617+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:26:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:21.617+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:21 smithi115 ceph-mon[35364]: pgmap v5341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:21 smithi181 ceph-mon[35869]: pgmap v5341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:21 smithi097 ceph-mon[32231]: pgmap v5341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:23 smithi115 ceph-mon[35364]: pgmap v5342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:23 smithi181 ceph-mon[35869]: pgmap v5342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:23 smithi097 ceph-mon[32231]: pgmap v5342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:25 smithi115 ceph-mon[35364]: pgmap v5343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8822.841945 secs 2024-07-26T10:26:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:25 smithi181 ceph-mon[35869]: pgmap v5343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8822.841945 secs 2024-07-26T10:26:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:25 smithi097 ceph-mon[32231]: pgmap v5343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8822.841945 secs 2024-07-26T10:26:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:27 smithi115 ceph-mon[35364]: pgmap v5344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:27 smithi181 ceph-mon[35869]: pgmap v5344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:27 smithi097 ceph-mon[32231]: pgmap v5344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:29 smithi115 ceph-mon[35364]: pgmap v5345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:29 smithi181 ceph-mon[35869]: pgmap v5345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:29 smithi097 ceph-mon[32231]: pgmap v5345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:31.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8827.842082 secs 2024-07-26T10:26:31.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8827.842082 secs 2024-07-26T10:26:31.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8827.842082 secs 2024-07-26T10:26:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:31 smithi115 ceph-mon[35364]: pgmap v5346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:31 smithi181 ceph-mon[35869]: pgmap v5346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:31 smithi097 ceph-mon[32231]: pgmap v5346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:33 smithi115 ceph-mon[35364]: pgmap v5347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:33 smithi181 ceph-mon[35869]: pgmap v5347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:33 smithi097 ceph-mon[32231]: pgmap v5347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:35 smithi115 ceph-mon[35364]: pgmap v5348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:36.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8832.842295 secs 2024-07-26T10:26:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:35 smithi181 ceph-mon[35869]: pgmap v5348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8832.842295 secs 2024-07-26T10:26:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:35 smithi097 ceph-mon[32231]: pgmap v5348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8832.842295 secs 2024-07-26T10:26:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:36.623+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:26:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:36.625+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:26:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:36.625+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:26:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:36.625+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:26:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:36.625+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:37 smithi115 ceph-mon[35364]: pgmap v5349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:37 smithi181 ceph-mon[35869]: pgmap v5349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:37 smithi097 ceph-mon[32231]: pgmap v5349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:38.445+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T10:26:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:38.445+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:26:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:38.446+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:26:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:38.446+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f070 con 0x7ff3f01061a0 2024-07-26T10:26:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:38 smithi115 ceph-mon[35364]: pgmap v5350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:26:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:38 smithi115 ceph-mon[35364]: pgmap v5351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:26:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:38 smithi181 ceph-mon[35869]: pgmap v5350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:26:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:38 smithi181 ceph-mon[35869]: pgmap v5351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:26:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:38 smithi097 ceph-mon[32231]: pgmap v5350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:26:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:38 smithi097 ceph-mon[32231]: pgmap v5351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:26:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8837.842468 secs 2024-07-26T10:26:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:41 smithi115 ceph-mon[35364]: pgmap v5352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8837.842468 secs 2024-07-26T10:26:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:41 smithi181 ceph-mon[35869]: pgmap v5352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8837.842468 secs 2024-07-26T10:26:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:41 smithi097 ceph-mon[32231]: pgmap v5352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:43 smithi115 ceph-mon[35364]: pgmap v5353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:43 smithi181 ceph-mon[35869]: pgmap v5353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:43 smithi097 ceph-mon[32231]: pgmap v5353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:45 smithi115 ceph-mon[35364]: pgmap v5354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:26:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:45 smithi181 ceph-mon[35869]: pgmap v5354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:26:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:45 smithi097 ceph-mon[32231]: pgmap v5354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:26:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8842.842617 secs 2024-07-26T10:26:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8842.842617 secs 2024-07-26T10:26:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8842.842617 secs 2024-07-26T10:26:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:47 smithi115 ceph-mon[35364]: pgmap v5355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:47 smithi181 ceph-mon[35869]: pgmap v5355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:47 smithi097 ceph-mon[32231]: pgmap v5355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:49 smithi115 ceph-mon[35364]: pgmap v5356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:26:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:49 smithi181 ceph-mon[35869]: pgmap v5356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:26:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:49 smithi097 ceph-mon[32231]: pgmap v5356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:26:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8847.842779 secs 2024-07-26T10:26:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:51 smithi115 ceph-mon[35364]: pgmap v5357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8847.842779 secs 2024-07-26T10:26:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:51 smithi181 ceph-mon[35869]: pgmap v5357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8847.842779 secs 2024-07-26T10:26:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:51 smithi097 ceph-mon[32231]: pgmap v5357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:51.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:51.626+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:26:51.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:51.627+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:26:51.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:51.627+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:26:51.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:51.627+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:26:51.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:26:51.627+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:53 smithi115 ceph-mon[35364]: pgmap v5358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:53 smithi181 ceph-mon[35869]: pgmap v5358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:53 smithi097 ceph-mon[32231]: pgmap v5358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:55 smithi115 ceph-mon[35364]: pgmap v5359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:26:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:55 smithi181 ceph-mon[35869]: pgmap v5359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:26:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:55 smithi097 ceph-mon[32231]: pgmap v5359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:26:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8852.842939 secs 2024-07-26T10:26:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8852.842939 secs 2024-07-26T10:26:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8852.842939 secs 2024-07-26T10:26:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:57 smithi115 ceph-mon[35364]: pgmap v5360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:26:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:57 smithi181 ceph-mon[35869]: pgmap v5360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:26:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:57 smithi097 ceph-mon[32231]: pgmap v5360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:26:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:26:59 smithi115 ceph-mon[35364]: pgmap v5361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:26:59 smithi181 ceph-mon[35869]: pgmap v5361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:26:59 smithi097 ceph-mon[32231]: pgmap v5361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8857.843087 secs 2024-07-26T10:27:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:01 smithi115 ceph-mon[35364]: pgmap v5362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:27:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8857.843087 secs 2024-07-26T10:27:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:01 smithi181 ceph-mon[35869]: pgmap v5362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:27:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8857.843087 secs 2024-07-26T10:27:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:01 smithi097 ceph-mon[32231]: pgmap v5362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:27:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:02 smithi115 ceph-mon[35364]: pgmap v5363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:27:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:02 smithi181 ceph-mon[35869]: pgmap v5363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:27:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:02 smithi097 ceph-mon[32231]: pgmap v5363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:27:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:05 smithi115 ceph-mon[35364]: pgmap v5364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:27:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:05 smithi181 ceph-mon[35869]: pgmap v5364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:27:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:05 smithi097 ceph-mon[32231]: pgmap v5364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:27:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8862.843258 secs 2024-07-26T10:27:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8862.843258 secs 2024-07-26T10:27:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8862.843258 secs 2024-07-26T10:27:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:06.634+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:27:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:06.635+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:27:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:06.635+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:27:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:06.635+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:27:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:06.635+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:07 smithi115 ceph-mon[35364]: pgmap v5365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:07 smithi181 ceph-mon[35869]: pgmap v5365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:07 smithi097 ceph-mon[32231]: pgmap v5365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:09 smithi115 ceph-mon[35364]: pgmap v5366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:27:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:09 smithi181 ceph-mon[35869]: pgmap v5366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:27:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:09 smithi097 ceph-mon[32231]: pgmap v5366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:27:11.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8867.843370 secs 2024-07-26T10:27:11.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:11 smithi115 ceph-mon[35364]: pgmap v5367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:27:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8867.843370 secs 2024-07-26T10:27:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:11 smithi181 ceph-mon[35869]: pgmap v5367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:27:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8867.843370 secs 2024-07-26T10:27:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:11 smithi097 ceph-mon[32231]: pgmap v5367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:27:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:13 smithi115 ceph-mon[35364]: pgmap v5368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:13 smithi181 ceph-mon[35869]: pgmap v5368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:13 smithi097 ceph-mon[32231]: pgmap v5368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:15 smithi115 ceph-mon[35364]: pgmap v5369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:15 smithi181 ceph-mon[35869]: pgmap v5369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:15 smithi097 ceph-mon[32231]: pgmap v5369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8872.843542 secs 2024-07-26T10:27:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8872.843542 secs 2024-07-26T10:27:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8872.843542 secs 2024-07-26T10:27:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:17 smithi115 ceph-mon[35364]: pgmap v5370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:17 smithi181 ceph-mon[35869]: pgmap v5370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:17 smithi097 ceph-mon[32231]: pgmap v5370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:19 smithi115 ceph-mon[35364]: pgmap v5371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:19 smithi181 ceph-mon[35869]: pgmap v5371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:19 smithi097 ceph-mon[32231]: pgmap v5371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8877.843698 secs 2024-07-26T10:27:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:21 smithi115 ceph-mon[35364]: pgmap v5372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8877.843698 secs 2024-07-26T10:27:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:21 smithi181 ceph-mon[35869]: pgmap v5372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8877.843698 secs 2024-07-26T10:27:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:21 smithi097 ceph-mon[32231]: pgmap v5372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:21.638+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:27:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:21.639+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:27:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:21.639+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:27:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:21.639+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:27:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:21.639+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:23 smithi115 ceph-mon[35364]: pgmap v5373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:23 smithi181 ceph-mon[35869]: pgmap v5373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:23 smithi097 ceph-mon[32231]: pgmap v5373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:24 smithi115 ceph-mon[35364]: pgmap v5374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:24 smithi181 ceph-mon[35869]: pgmap v5374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:24 smithi097 ceph-mon[32231]: pgmap v5374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8882.843852 secs 2024-07-26T10:27:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8882.843852 secs 2024-07-26T10:27:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8882.843852 secs 2024-07-26T10:27:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:26 smithi115 ceph-mon[35364]: pgmap v5375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:26 smithi181 ceph-mon[35869]: pgmap v5375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:26 smithi097 ceph-mon[32231]: pgmap v5375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:29 smithi115 ceph-mon[35364]: pgmap v5376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:29 smithi181 ceph-mon[35869]: pgmap v5376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:29 smithi097 ceph-mon[32231]: pgmap v5376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8887.843992 secs 2024-07-26T10:27:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:31 smithi115 ceph-mon[35364]: pgmap v5377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8887.843992 secs 2024-07-26T10:27:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:31 smithi181 ceph-mon[35869]: pgmap v5377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8887.843992 secs 2024-07-26T10:27:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:31 smithi097 ceph-mon[32231]: pgmap v5377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:33 smithi115 ceph-mon[35364]: pgmap v5378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:33 smithi181 ceph-mon[35869]: pgmap v5378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:33 smithi097 ceph-mon[32231]: pgmap v5378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:35 smithi115 ceph-mon[35364]: pgmap v5379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:35 smithi181 ceph-mon[35869]: pgmap v5379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:35 smithi097 ceph-mon[32231]: pgmap v5379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:27:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8892.844128 secs 2024-07-26T10:27:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8892.844128 secs 2024-07-26T10:27:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8892.844128 secs 2024-07-26T10:27:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:36.645+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:27:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:36.647+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:27:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:36.647+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:27:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:36.647+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:27:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:36.647+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:37 smithi115 ceph-mon[35364]: pgmap v5380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:37 smithi181 ceph-mon[35869]: pgmap v5380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:37 smithi097 ceph-mon[32231]: pgmap v5380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:39 smithi115 ceph-mon[35364]: pgmap v5381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:39 smithi181 ceph-mon[35869]: pgmap v5381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:39 smithi097 ceph-mon[32231]: pgmap v5381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:27:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:27:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:27:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:41 smithi115 ceph-mon[35364]: pgmap v5382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:27:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:41 smithi115 ceph-mon[35364]: pgmap v5383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8897.844309 secs 2024-07-26T10:27:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:41 smithi181 ceph-mon[35869]: pgmap v5382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:27:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:41 smithi181 ceph-mon[35869]: pgmap v5383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8897.844309 secs 2024-07-26T10:27:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:41 smithi097 ceph-mon[32231]: pgmap v5382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:27:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:41 smithi097 ceph-mon[32231]: pgmap v5383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8897.844309 secs 2024-07-26T10:27:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:43 smithi115 ceph-mon[35364]: pgmap v5384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:43 smithi181 ceph-mon[35869]: pgmap v5384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:43 smithi097 ceph-mon[32231]: pgmap v5384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:44 smithi115 ceph-mon[35364]: pgmap v5385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:27:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:44 smithi181 ceph-mon[35869]: pgmap v5385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:27:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:44 smithi097 ceph-mon[32231]: pgmap v5385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:27:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8902.844463 secs 2024-07-26T10:27:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8902.844463 secs 2024-07-26T10:27:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8902.844463 secs 2024-07-26T10:27:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:46 smithi115 ceph-mon[35364]: pgmap v5386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:46 smithi181 ceph-mon[35869]: pgmap v5386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:46 smithi097 ceph-mon[32231]: pgmap v5386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:48 smithi115 ceph-mon[35364]: pgmap v5387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:27:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:48 smithi181 ceph-mon[35869]: pgmap v5387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:27:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:48 smithi097 ceph-mon[32231]: pgmap v5387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:27:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:50 smithi115 ceph-mon[35364]: pgmap v5388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:27:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8907.844587 secs 2024-07-26T10:27:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:50 smithi181 ceph-mon[35869]: pgmap v5388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:27:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8907.844587 secs 2024-07-26T10:27:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:50 smithi097 ceph-mon[32231]: pgmap v5388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:27:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8907.844587 secs 2024-07-26T10:27:51.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:51.649+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:27:51.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:51.650+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:27:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:51.650+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:27:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:51.650+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:27:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:27:51.650+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:52 smithi181 ceph-mon[35869]: pgmap v5389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:52 smithi097 ceph-mon[32231]: pgmap v5389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:52.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:52 smithi115 ceph-mon[35364]: pgmap v5389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:27:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:54 smithi097 ceph-mon[32231]: pgmap v5390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:54 smithi115 ceph-mon[35364]: pgmap v5390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:54 smithi181 ceph-mon[35869]: pgmap v5390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8912.844728 secs 2024-07-26T10:27:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8912.844728 secs 2024-07-26T10:27:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8912.844728 secs 2024-07-26T10:27:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:56 smithi115 ceph-mon[35364]: pgmap v5391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:56 smithi181 ceph-mon[35869]: pgmap v5391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:56 smithi097 ceph-mon[32231]: pgmap v5391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:27:58 smithi115 ceph-mon[35364]: pgmap v5392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:27:58 smithi181 ceph-mon[35869]: pgmap v5392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:27:58 smithi097 ceph-mon[32231]: pgmap v5392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:00 smithi115 ceph-mon[35364]: pgmap v5393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8917.844897 secs 2024-07-26T10:28:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:00 smithi181 ceph-mon[35869]: pgmap v5393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8917.844897 secs 2024-07-26T10:28:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:00 smithi097 ceph-mon[32231]: pgmap v5393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8917.844897 secs 2024-07-26T10:28:02.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:02 smithi115 ceph-mon[35364]: pgmap v5394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:02.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:02 smithi181 ceph-mon[35869]: pgmap v5394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:02.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:02 smithi097 ceph-mon[32231]: pgmap v5394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:04 smithi115 ceph-mon[35364]: pgmap v5395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:04 smithi181 ceph-mon[35869]: pgmap v5395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:04 smithi097 ceph-mon[32231]: pgmap v5395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8922.845068 secs 2024-07-26T10:28:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8922.845068 secs 2024-07-26T10:28:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8922.845068 secs 2024-07-26T10:28:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.654+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:28:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.656+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:28:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.656+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:28:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.656+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:28:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.656+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.904+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T10:28:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.904+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:28:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.904+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000ed10 tx=0x7ff3e004a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:28:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.905+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000ed10 tx=0x7ff3e004a000 comp rx=0 tx=0).stop 2024-07-26T10:28:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.905+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b53b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T10:28:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.905+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b53b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:28:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.905+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:28:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.905+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:28:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.906+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:28:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:06.907+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e0059040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:28:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:06 smithi115 ceph-mon[35364]: pgmap v5396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:06 smithi181 ceph-mon[35869]: pgmap v5396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:06 smithi097 ceph-mon[32231]: pgmap v5396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:08.461+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:28:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:08.461+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T10:28:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:08.462+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:28:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:08.462+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:28:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:08 smithi115 ceph-mon[35364]: pgmap v5397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:08 smithi181 ceph-mon[35869]: pgmap v5397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:08 smithi097 ceph-mon[32231]: pgmap v5397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:10 smithi115 ceph-mon[35364]: pgmap v5398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8927.845276 secs 2024-07-26T10:28:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:10 smithi181 ceph-mon[35869]: pgmap v5398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8927.845276 secs 2024-07-26T10:28:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:10 smithi097 ceph-mon[32231]: pgmap v5398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8927.845276 secs 2024-07-26T10:28:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:12 smithi115 ceph-mon[35364]: pgmap v5399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:12 smithi181 ceph-mon[35869]: pgmap v5399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:12 smithi097 ceph-mon[32231]: pgmap v5399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:14 smithi115 ceph-mon[35364]: pgmap v5400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:14 smithi181 ceph-mon[35869]: pgmap v5400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:14 smithi097 ceph-mon[32231]: pgmap v5400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8932.845364 secs 2024-07-26T10:28:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8932.845364 secs 2024-07-26T10:28:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8932.845364 secs 2024-07-26T10:28:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:16 smithi115 ceph-mon[35364]: pgmap v5401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:16 smithi181 ceph-mon[35869]: pgmap v5401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:16 smithi097 ceph-mon[32231]: pgmap v5401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:18 smithi115 ceph-mon[35364]: pgmap v5402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:18 smithi181 ceph-mon[35869]: pgmap v5402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:18 smithi097 ceph-mon[32231]: pgmap v5402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:20 smithi115 ceph-mon[35364]: pgmap v5403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8937.845525 secs 2024-07-26T10:28:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:20 smithi181 ceph-mon[35869]: pgmap v5403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8937.845525 secs 2024-07-26T10:28:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:20 smithi097 ceph-mon[32231]: pgmap v5403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8937.845525 secs 2024-07-26T10:28:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:21.658+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:28:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:21.660+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:28:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:21.660+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:28:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:21.660+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:28:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:21.660+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:22 smithi115 ceph-mon[35364]: pgmap v5404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:22 smithi181 ceph-mon[35869]: pgmap v5404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:22 smithi097 ceph-mon[32231]: pgmap v5404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:24 smithi115 ceph-mon[35364]: pgmap v5405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:24 smithi181 ceph-mon[35869]: pgmap v5405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:24 smithi097 ceph-mon[32231]: pgmap v5405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8942.845698 secs 2024-07-26T10:28:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8942.845698 secs 2024-07-26T10:28:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8942.845698 secs 2024-07-26T10:28:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:26 smithi115 ceph-mon[35364]: pgmap v5406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:26 smithi181 ceph-mon[35869]: pgmap v5406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:26 smithi097 ceph-mon[32231]: pgmap v5406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:28 smithi115 ceph-mon[35364]: pgmap v5407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:28 smithi181 ceph-mon[35869]: pgmap v5407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:28 smithi097 ceph-mon[32231]: pgmap v5407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:30 smithi115 ceph-mon[35364]: pgmap v5408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8947.845844 secs 2024-07-26T10:28:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:30 smithi181 ceph-mon[35869]: pgmap v5408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8947.845844 secs 2024-07-26T10:28:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:30 smithi097 ceph-mon[32231]: pgmap v5408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8947.845844 secs 2024-07-26T10:28:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:32 smithi115 ceph-mon[35364]: pgmap v5409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:32 smithi181 ceph-mon[35869]: pgmap v5409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:32 smithi097 ceph-mon[32231]: pgmap v5409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:34 smithi115 ceph-mon[35364]: pgmap v5410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:34 smithi181 ceph-mon[35869]: pgmap v5410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:34 smithi097 ceph-mon[32231]: pgmap v5410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8952.845987 secs 2024-07-26T10:28:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8952.845987 secs 2024-07-26T10:28:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8952.845987 secs 2024-07-26T10:28:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:36.666+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:28:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:36.667+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:28:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:36.667+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:28:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:36.667+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:28:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:36.668+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:36 smithi115 ceph-mon[35364]: pgmap v5411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:36 smithi181 ceph-mon[35869]: pgmap v5411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:36 smithi097 ceph-mon[32231]: pgmap v5411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:38 smithi115 ceph-mon[35364]: pgmap v5412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:38 smithi181 ceph-mon[35869]: pgmap v5412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:38 smithi097 ceph-mon[32231]: pgmap v5412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:40 smithi115 ceph-mon[35364]: pgmap v5413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8957.846122 secs 2024-07-26T10:28:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:40 smithi181 ceph-mon[35869]: pgmap v5413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8957.846122 secs 2024-07-26T10:28:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:40 smithi097 ceph-mon[32231]: pgmap v5413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8957.846122 secs 2024-07-26T10:28:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:41 smithi115 ceph-mon[35364]: pgmap v5414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:28:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:41 smithi115 ceph-mon[35364]: pgmap v5415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:41.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:28:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:41 smithi181 ceph-mon[35869]: pgmap v5414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:28:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:41 smithi181 ceph-mon[35869]: pgmap v5415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:28:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:41 smithi097 ceph-mon[32231]: pgmap v5414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:28:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:41 smithi097 ceph-mon[32231]: pgmap v5415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:28:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:43 smithi115 ceph-mon[35364]: pgmap v5416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:43 smithi181 ceph-mon[35869]: pgmap v5416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:43 smithi097 ceph-mon[32231]: pgmap v5416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8962.846281 secs 2024-07-26T10:28:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:45 smithi115 ceph-mon[35364]: pgmap v5417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8962.846281 secs 2024-07-26T10:28:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:45 smithi181 ceph-mon[35869]: pgmap v5417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8962.846281 secs 2024-07-26T10:28:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:45 smithi097 ceph-mon[32231]: pgmap v5417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:47 smithi115 ceph-mon[35364]: pgmap v5418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:47 smithi181 ceph-mon[35869]: pgmap v5418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:47 smithi097 ceph-mon[32231]: pgmap v5418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:49 smithi115 ceph-mon[35364]: pgmap v5419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:49 smithi181 ceph-mon[35869]: pgmap v5419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:49 smithi097 ceph-mon[32231]: pgmap v5419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8967.846383 secs 2024-07-26T10:28:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8967.846383 secs 2024-07-26T10:28:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8967.846383 secs 2024-07-26T10:28:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:51.670+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:28:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:51.671+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:28:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:51.671+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:28:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:51.671+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:28:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:28:51.671+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:51 smithi115 ceph-mon[35364]: pgmap v5420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:28:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:51 smithi181 ceph-mon[35869]: pgmap v5420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:28:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:51 smithi097 ceph-mon[32231]: pgmap v5420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:28:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:53 smithi115 ceph-mon[35364]: pgmap v5421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:53 smithi181 ceph-mon[35869]: pgmap v5421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:53 smithi097 ceph-mon[32231]: pgmap v5421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8972.846519 secs 2024-07-26T10:28:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:55 smithi115 ceph-mon[35364]: pgmap v5422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:28:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8972.846519 secs 2024-07-26T10:28:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:55 smithi181 ceph-mon[35869]: pgmap v5422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:28:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8972.846519 secs 2024-07-26T10:28:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:55 smithi097 ceph-mon[32231]: pgmap v5422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:28:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:57 smithi181 ceph-mon[35869]: pgmap v5423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:28:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:57 smithi097 ceph-mon[32231]: pgmap v5423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:28:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:57 smithi115 ceph-mon[35364]: pgmap v5423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:29:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:28:59 smithi097 ceph-mon[32231]: pgmap v5424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:29:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:28:59 smithi115 ceph-mon[35364]: pgmap v5424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:29:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:28:59 smithi181 ceph-mon[35869]: pgmap v5424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:29:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8977.846620 secs 2024-07-26T10:29:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8977.846620 secs 2024-07-26T10:29:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8977.846620 secs 2024-07-26T10:29:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:01 smithi097 ceph-mon[32231]: pgmap v5425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:29:02.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:01 smithi115 ceph-mon[35364]: pgmap v5425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:29:02.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:01 smithi181 ceph-mon[35869]: pgmap v5425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:29:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:04 smithi115 ceph-mon[35364]: pgmap v5426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:29:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:04 smithi181 ceph-mon[35869]: pgmap v5426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:29:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:04 smithi097 ceph-mon[32231]: pgmap v5426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:29:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8982.846788 secs 2024-07-26T10:29:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:06 smithi115 ceph-mon[35364]: pgmap v5427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:29:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8982.846788 secs 2024-07-26T10:29:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:06 smithi181 ceph-mon[35869]: pgmap v5427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:29:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8982.846788 secs 2024-07-26T10:29:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:06 smithi097 ceph-mon[32231]: pgmap v5427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:29:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:06.677+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:29:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:06.678+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:29:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:06.678+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:29:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:06.678+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:29:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:06.678+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:08 smithi115 ceph-mon[35364]: pgmap v5428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:08 smithi181 ceph-mon[35869]: pgmap v5428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:08 smithi097 ceph-mon[32231]: pgmap v5428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:10 smithi115 ceph-mon[35364]: pgmap v5429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:10 smithi181 ceph-mon[35869]: pgmap v5429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:10 smithi097 ceph-mon[32231]: pgmap v5429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8987.846950 secs 2024-07-26T10:29:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8987.846950 secs 2024-07-26T10:29:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8987.846950 secs 2024-07-26T10:29:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:12 smithi115 ceph-mon[35364]: pgmap v5430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:12 smithi181 ceph-mon[35869]: pgmap v5430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:12 smithi097 ceph-mon[32231]: pgmap v5430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:14 smithi115 ceph-mon[35364]: pgmap v5431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:14 smithi181 ceph-mon[35869]: pgmap v5431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:14 smithi097 ceph-mon[32231]: pgmap v5431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8992.847097 secs 2024-07-26T10:29:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:16 smithi115 ceph-mon[35364]: pgmap v5432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8992.847097 secs 2024-07-26T10:29:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:16 smithi181 ceph-mon[35869]: pgmap v5432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8992.847097 secs 2024-07-26T10:29:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:16 smithi097 ceph-mon[32231]: pgmap v5432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:18 smithi115 ceph-mon[35364]: pgmap v5433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:18 smithi181 ceph-mon[35869]: pgmap v5433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:18 smithi097 ceph-mon[32231]: pgmap v5433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:20 smithi115 ceph-mon[35364]: pgmap v5434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:20 smithi181 ceph-mon[35869]: pgmap v5434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:20 smithi097 ceph-mon[32231]: pgmap v5434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 8997.847180 secs 2024-07-26T10:29:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 8997.847180 secs 2024-07-26T10:29:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 8997.847180 secs 2024-07-26T10:29:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:21.680+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:29:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:21.681+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:29:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:21.681+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:29:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:21.681+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:29:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:21.681+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:22 smithi115 ceph-mon[35364]: pgmap v5435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:22 smithi181 ceph-mon[35869]: pgmap v5435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:22 smithi097 ceph-mon[32231]: pgmap v5435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:24 smithi115 ceph-mon[35364]: pgmap v5436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:24 smithi181 ceph-mon[35869]: pgmap v5436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:24 smithi097 ceph-mon[32231]: pgmap v5436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9002.847324 secs 2024-07-26T10:29:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:26 smithi115 ceph-mon[35364]: pgmap v5437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9002.847324 secs 2024-07-26T10:29:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:26 smithi181 ceph-mon[35869]: pgmap v5437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9002.847324 secs 2024-07-26T10:29:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:26 smithi097 ceph-mon[32231]: pgmap v5437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:28 smithi115 ceph-mon[35364]: pgmap v5438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:28 smithi181 ceph-mon[35869]: pgmap v5438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:28 smithi097 ceph-mon[32231]: pgmap v5438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:30 smithi115 ceph-mon[35364]: pgmap v5439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:30 smithi181 ceph-mon[35869]: pgmap v5439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:30 smithi097 ceph-mon[32231]: pgmap v5439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9007.847465 secs 2024-07-26T10:29:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9007.847465 secs 2024-07-26T10:29:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9007.847465 secs 2024-07-26T10:29:32.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:32 smithi097 ceph-mon[32231]: pgmap v5440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:32 smithi115 ceph-mon[35364]: pgmap v5440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:32 smithi181 ceph-mon[35869]: pgmap v5440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:34 smithi115 ceph-mon[35364]: pgmap v5441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:34 smithi181 ceph-mon[35869]: pgmap v5441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:34 smithi097 ceph-mon[32231]: pgmap v5441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:35 smithi115 ceph-mon[35364]: pgmap v5442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:35 smithi181 ceph-mon[35869]: pgmap v5442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:35 smithi097 ceph-mon[32231]: pgmap v5442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9012.847555 secs 2024-07-26T10:29:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9012.847555 secs 2024-07-26T10:29:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9012.847555 secs 2024-07-26T10:29:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:36.687+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:29:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:36.688+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:29:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:36.688+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:29:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:36.688+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:29:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:36.688+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:37 smithi115 ceph-mon[35364]: pgmap v5443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:37 smithi181 ceph-mon[35869]: pgmap v5443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:37 smithi097 ceph-mon[32231]: pgmap v5443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:39 smithi115 ceph-mon[35364]: pgmap v5444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:39 smithi181 ceph-mon[35869]: pgmap v5444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:39 smithi097 ceph-mon[32231]: pgmap v5444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9017.847697 secs 2024-07-26T10:29:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9017.847697 secs 2024-07-26T10:29:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9017.847697 secs 2024-07-26T10:29:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:41 smithi115 ceph-mon[35364]: pgmap v5445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:41 smithi181 ceph-mon[35869]: pgmap v5445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:41 smithi097 ceph-mon[32231]: pgmap v5445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:29:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:29:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:29:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:43 smithi181 ceph-mon[35869]: pgmap v5446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:29:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:43 smithi181 ceph-mon[35869]: pgmap v5447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:29:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:43 smithi097 ceph-mon[32231]: pgmap v5446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:29:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:43 smithi097 ceph-mon[32231]: pgmap v5447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:29:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:43 smithi115 ceph-mon[35364]: pgmap v5446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:29:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:43 smithi115 ceph-mon[35364]: pgmap v5447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:29:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:45 smithi097 ceph-mon[32231]: pgmap v5448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:29:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9022.847854 secs 2024-07-26T10:29:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:45 smithi115 ceph-mon[35364]: pgmap v5448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:29:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9022.847854 secs 2024-07-26T10:29:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:45 smithi181 ceph-mon[35869]: pgmap v5448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:29:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9022.847854 secs 2024-07-26T10:29:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:47 smithi097 ceph-mon[32231]: pgmap v5449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:47 smithi115 ceph-mon[35364]: pgmap v5449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:47 smithi181 ceph-mon[35869]: pgmap v5449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:49 smithi097 ceph-mon[32231]: pgmap v5450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-07-26T10:29:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:50 smithi115 ceph-mon[35364]: pgmap v5450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-07-26T10:29:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:50 smithi181 ceph-mon[35869]: pgmap v5450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-07-26T10:29:51.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9027.848002 secs 2024-07-26T10:29:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9027.848002 secs 2024-07-26T10:29:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9027.848002 secs 2024-07-26T10:29:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:51.690+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:29:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:51.691+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:29:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:51.691+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:29:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:51.691+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:29:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:29:51.691+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:52 smithi115 ceph-mon[35364]: pgmap v5451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:52 smithi181 ceph-mon[35869]: pgmap v5451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:52 smithi097 ceph-mon[32231]: pgmap v5451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:54 smithi115 ceph-mon[35364]: pgmap v5452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:29:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:54 smithi181 ceph-mon[35869]: pgmap v5452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:29:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:54 smithi097 ceph-mon[32231]: pgmap v5452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:29:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:56 smithi115 ceph-mon[35364]: pgmap v5453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:29:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9032.848141 secs 2024-07-26T10:29:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:56 smithi181 ceph-mon[35869]: pgmap v5453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:29:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9032.848141 secs 2024-07-26T10:29:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:56 smithi097 ceph-mon[32231]: pgmap v5453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:29:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9032.848141 secs 2024-07-26T10:29:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:29:58 smithi115 ceph-mon[35364]: pgmap v5454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:29:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:29:58 smithi181 ceph-mon[35869]: pgmap v5454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:29:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:29:58 smithi097 ceph-mon[32231]: pgmap v5454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:30:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:00 smithi115 ceph-mon[35364]: pgmap v5455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:30:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:00 smithi181 ceph-mon[35869]: pgmap v5455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:30:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:00 smithi097 ceph-mon[32231]: pgmap v5455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:30:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9037.848310 secs 2024-07-26T10:30:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9037.848310 secs 2024-07-26T10:30:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9037.848310 secs 2024-07-26T10:30:02.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:02 smithi115 ceph-mon[35364]: pgmap v5456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:30:02.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:02 smithi181 ceph-mon[35869]: pgmap v5456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:30:02.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:02 smithi097 ceph-mon[32231]: pgmap v5456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:30:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:04 smithi115 ceph-mon[35364]: pgmap v5457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:30:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:04 smithi181 ceph-mon[35869]: pgmap v5457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:30:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:04 smithi097 ceph-mon[32231]: pgmap v5457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:30:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:06 smithi115 ceph-mon[35364]: pgmap v5458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:30:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9042.848473 secs 2024-07-26T10:30:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:06 smithi181 ceph-mon[35869]: pgmap v5458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:30:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9042.848473 secs 2024-07-26T10:30:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:06 smithi097 ceph-mon[32231]: pgmap v5458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:30:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9042.848473 secs 2024-07-26T10:30:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:06.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:30:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:06.699+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:30:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:06.699+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:30:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:06.699+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:30:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:06.699+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:08 smithi115 ceph-mon[35364]: pgmap v5459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:08 smithi181 ceph-mon[35869]: pgmap v5459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:08 smithi097 ceph-mon[32231]: pgmap v5459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:10 smithi115 ceph-mon[35364]: pgmap v5460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:10 smithi181 ceph-mon[35869]: pgmap v5460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:10 smithi097 ceph-mon[32231]: pgmap v5460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9047.848575 secs 2024-07-26T10:30:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9047.848575 secs 2024-07-26T10:30:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9047.848575 secs 2024-07-26T10:30:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:12 smithi115 ceph-mon[35364]: pgmap v5461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:12 smithi181 ceph-mon[35869]: pgmap v5461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:12 smithi097 ceph-mon[32231]: pgmap v5461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:14 smithi115 ceph-mon[35364]: pgmap v5462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:14 smithi181 ceph-mon[35869]: pgmap v5462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:14 smithi097 ceph-mon[32231]: pgmap v5462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:16 smithi115 ceph-mon[35364]: pgmap v5463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9052.848709 secs 2024-07-26T10:30:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:16 smithi181 ceph-mon[35869]: pgmap v5463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9052.848709 secs 2024-07-26T10:30:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:16 smithi097 ceph-mon[32231]: pgmap v5463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9052.848709 secs 2024-07-26T10:30:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:18 smithi115 ceph-mon[35364]: pgmap v5464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:18 smithi181 ceph-mon[35869]: pgmap v5464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:18 smithi097 ceph-mon[32231]: pgmap v5464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:20 smithi115 ceph-mon[35364]: pgmap v5465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:20 smithi181 ceph-mon[35869]: pgmap v5465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:20 smithi097 ceph-mon[32231]: pgmap v5465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9057.848851 secs 2024-07-26T10:30:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9057.848851 secs 2024-07-26T10:30:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9057.848851 secs 2024-07-26T10:30:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:21.701+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:30:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:21.702+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:30:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:21.702+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:30:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:21.702+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:30:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:21.702+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:22 smithi115 ceph-mon[35364]: pgmap v5466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:22 smithi181 ceph-mon[35869]: pgmap v5466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:22 smithi097 ceph-mon[32231]: pgmap v5466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:24 smithi115 ceph-mon[35364]: pgmap v5467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:24 smithi181 ceph-mon[35869]: pgmap v5467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:24 smithi097 ceph-mon[32231]: pgmap v5467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:26 smithi115 ceph-mon[35364]: pgmap v5468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9062.848994 secs 2024-07-26T10:30:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:26 smithi181 ceph-mon[35869]: pgmap v5468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9062.848994 secs 2024-07-26T10:30:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:26 smithi097 ceph-mon[32231]: pgmap v5468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9062.848994 secs 2024-07-26T10:30:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:27 smithi115 ceph-mon[35364]: pgmap v5469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:27 smithi181 ceph-mon[35869]: pgmap v5469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:27 smithi097 ceph-mon[32231]: pgmap v5469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:29 smithi115 ceph-mon[35364]: pgmap v5470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:29 smithi181 ceph-mon[35869]: pgmap v5470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:29 smithi097 ceph-mon[32231]: pgmap v5470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9067.849149 secs 2024-07-26T10:30:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:31 smithi115 ceph-mon[35364]: pgmap v5471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9067.849149 secs 2024-07-26T10:30:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:31 smithi181 ceph-mon[35869]: pgmap v5471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9067.849149 secs 2024-07-26T10:30:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:31 smithi097 ceph-mon[32231]: pgmap v5471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:33.556 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:33 smithi097 ceph-mon[32231]: pgmap v5472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:33 smithi115 ceph-mon[35364]: pgmap v5472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:33 smithi181 ceph-mon[35869]: pgmap v5472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:35 smithi115 ceph-mon[35364]: pgmap v5473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:35 smithi181 ceph-mon[35869]: pgmap v5473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:35 smithi097 ceph-mon[32231]: pgmap v5473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:30:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:36.709+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:30:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:36.709+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:30:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:36.709+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:30:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:36.709+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:30:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:36.709+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9072.849326 secs 2024-07-26T10:30:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9072.849326 secs 2024-07-26T10:30:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9072.849326 secs 2024-07-26T10:30:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:37 smithi115 ceph-mon[35364]: pgmap v5474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:37 smithi181 ceph-mon[35869]: pgmap v5474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:37 smithi097 ceph-mon[32231]: pgmap v5474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:39 smithi115 ceph-mon[35364]: pgmap v5475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:39 smithi181 ceph-mon[35869]: pgmap v5475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:39 smithi097 ceph-mon[32231]: pgmap v5475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9077.849465 secs 2024-07-26T10:30:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9077.849465 secs 2024-07-26T10:30:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9077.849465 secs 2024-07-26T10:30:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:41 smithi115 ceph-mon[35364]: pgmap v5476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:41 smithi181 ceph-mon[35869]: pgmap v5476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:41 smithi097 ceph-mon[32231]: pgmap v5476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:43 smithi115 ceph-mon[35364]: pgmap v5477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:43 smithi181 ceph-mon[35869]: pgmap v5477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:43 smithi097 ceph-mon[32231]: pgmap v5477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:44 smithi115 ceph-mon[35364]: pgmap v5478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:30:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:44 smithi115 ceph-mon[35364]: pgmap v5479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:30:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:44 smithi181 ceph-mon[35869]: pgmap v5478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:30:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:44 smithi181 ceph-mon[35869]: pgmap v5479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:44.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:30:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:44 smithi097 ceph-mon[32231]: pgmap v5478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:30:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:44 smithi097 ceph-mon[32231]: pgmap v5479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:30:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:30:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9082.849573 secs 2024-07-26T10:30:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9082.849573 secs 2024-07-26T10:30:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9082.849573 secs 2024-07-26T10:30:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:46 smithi115 ceph-mon[35364]: pgmap v5480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:30:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:46 smithi181 ceph-mon[35869]: pgmap v5480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:30:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:46 smithi097 ceph-mon[32231]: pgmap v5480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:30:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:48 smithi115 ceph-mon[35364]: pgmap v5481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:30:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:48 smithi181 ceph-mon[35869]: pgmap v5481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:30:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:48 smithi097 ceph-mon[32231]: pgmap v5481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:30:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:50 smithi115 ceph-mon[35364]: pgmap v5482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9087.849736 secs 2024-07-26T10:30:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:50 smithi181 ceph-mon[35869]: pgmap v5482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9087.849736 secs 2024-07-26T10:30:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:50 smithi097 ceph-mon[32231]: pgmap v5482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9087.849736 secs 2024-07-26T10:30:51.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:51.710+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:30:51.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:51.712+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:30:51.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:51.712+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:30:51.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:51.712+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:30:51.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:30:51.712+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:52 smithi115 ceph-mon[35364]: pgmap v5483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:52 smithi181 ceph-mon[35869]: pgmap v5483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:52 smithi097 ceph-mon[32231]: pgmap v5483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:54 smithi115 ceph-mon[35364]: pgmap v5484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:54 smithi181 ceph-mon[35869]: pgmap v5484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:54 smithi097 ceph-mon[32231]: pgmap v5484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9092.849899 secs 2024-07-26T10:30:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9092.849899 secs 2024-07-26T10:30:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9092.849899 secs 2024-07-26T10:30:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:56 smithi181 ceph-mon[35869]: pgmap v5485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:56 smithi097 ceph-mon[32231]: pgmap v5485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:56 smithi115 ceph-mon[35364]: pgmap v5485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:30:58 smithi097 ceph-mon[32231]: pgmap v5486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:30:58 smithi115 ceph-mon[35364]: pgmap v5486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:30:58 smithi181 ceph-mon[35869]: pgmap v5486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:00 smithi097 ceph-mon[32231]: pgmap v5487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9097.850064 secs 2024-07-26T10:31:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:00 smithi115 ceph-mon[35364]: pgmap v5487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9097.850064 secs 2024-07-26T10:31:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:00 smithi181 ceph-mon[35869]: pgmap v5487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9097.850064 secs 2024-07-26T10:31:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:02 smithi097 ceph-mon[32231]: pgmap v5488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:03.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:02 smithi115 ceph-mon[35364]: pgmap v5488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:03.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:02 smithi181 ceph-mon[35869]: pgmap v5488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:04 smithi115 ceph-mon[35364]: pgmap v5489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:04 smithi181 ceph-mon[35869]: pgmap v5489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:04 smithi097 ceph-mon[32231]: pgmap v5489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9102.850226 secs 2024-07-26T10:31:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9102.850226 secs 2024-07-26T10:31:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9102.850226 secs 2024-07-26T10:31:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:06.719+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:31:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:06.720+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:31:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:06.720+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:31:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:06.720+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:31:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:06.720+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:06 smithi115 ceph-mon[35364]: pgmap v5490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:06 smithi181 ceph-mon[35869]: pgmap v5490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:06 smithi097 ceph-mon[32231]: pgmap v5490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:08.493+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:31:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:08.493+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:31:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:08.494+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:31:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:08.494+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:31:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:08 smithi115 ceph-mon[35364]: pgmap v5491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:08 smithi181 ceph-mon[35869]: pgmap v5491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:08 smithi097 ceph-mon[32231]: pgmap v5491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:10 smithi115 ceph-mon[35364]: pgmap v5492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9107.850348 secs 2024-07-26T10:31:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:10 smithi181 ceph-mon[35869]: pgmap v5492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9107.850348 secs 2024-07-26T10:31:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:10 smithi097 ceph-mon[32231]: pgmap v5492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9107.850348 secs 2024-07-26T10:31:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:12 smithi115 ceph-mon[35364]: pgmap v5493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:12 smithi181 ceph-mon[35869]: pgmap v5493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:12 smithi097 ceph-mon[32231]: pgmap v5493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:14 smithi115 ceph-mon[35364]: pgmap v5494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:14 smithi181 ceph-mon[35869]: pgmap v5494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:14 smithi097 ceph-mon[32231]: pgmap v5494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9112.850500 secs 2024-07-26T10:31:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9112.850500 secs 2024-07-26T10:31:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9112.850500 secs 2024-07-26T10:31:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:16 smithi115 ceph-mon[35364]: pgmap v5495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:16 smithi181 ceph-mon[35869]: pgmap v5495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:16 smithi097 ceph-mon[32231]: pgmap v5495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:18 smithi115 ceph-mon[35364]: pgmap v5496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:18 smithi181 ceph-mon[35869]: pgmap v5496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:18 smithi097 ceph-mon[32231]: pgmap v5496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:20 smithi115 ceph-mon[35364]: pgmap v5497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9117.850644 secs 2024-07-26T10:31:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:20 smithi181 ceph-mon[35869]: pgmap v5497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9117.850644 secs 2024-07-26T10:31:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:20 smithi097 ceph-mon[32231]: pgmap v5497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9117.850644 secs 2024-07-26T10:31:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:21.722+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:31:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:21.723+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:31:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:21.723+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:31:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:21.723+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:31:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:21.723+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:22 smithi115 ceph-mon[35364]: pgmap v5498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:22 smithi181 ceph-mon[35869]: pgmap v5498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:22 smithi097 ceph-mon[32231]: pgmap v5498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:24 smithi115 ceph-mon[35364]: pgmap v5499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:24 smithi181 ceph-mon[35869]: pgmap v5499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:24 smithi097 ceph-mon[32231]: pgmap v5499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9122.850794 secs 2024-07-26T10:31:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9122.850794 secs 2024-07-26T10:31:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9122.850794 secs 2024-07-26T10:31:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:26 smithi115 ceph-mon[35364]: pgmap v5500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:26 smithi181 ceph-mon[35869]: pgmap v5500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:26 smithi097 ceph-mon[32231]: pgmap v5500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:28 smithi115 ceph-mon[35364]: pgmap v5501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:28 smithi181 ceph-mon[35869]: pgmap v5501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:28 smithi097 ceph-mon[32231]: pgmap v5501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:30 smithi115 ceph-mon[35364]: pgmap v5502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9127.850936 secs 2024-07-26T10:31:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:30 smithi181 ceph-mon[35869]: pgmap v5502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9127.850936 secs 2024-07-26T10:31:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:30 smithi097 ceph-mon[32231]: pgmap v5502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9127.850936 secs 2024-07-26T10:31:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:32 smithi115 ceph-mon[35364]: pgmap v5503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:32 smithi181 ceph-mon[35869]: pgmap v5503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:32 smithi097 ceph-mon[32231]: pgmap v5503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:34 smithi115 ceph-mon[35364]: pgmap v5504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:34 smithi181 ceph-mon[35869]: pgmap v5504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:34 smithi097 ceph-mon[32231]: pgmap v5504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9132.851080 secs 2024-07-26T10:31:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9132.851080 secs 2024-07-26T10:31:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9132.851080 secs 2024-07-26T10:31:36.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:36.731+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:31:36.733 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:36.731+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:31:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:36.731+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:31:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:36.731+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:31:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:36.731+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:36 smithi115 ceph-mon[35364]: pgmap v5505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:36 smithi181 ceph-mon[35869]: pgmap v5505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:36 smithi097 ceph-mon[32231]: pgmap v5505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:38 smithi115 ceph-mon[35364]: pgmap v5506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:38 smithi181 ceph-mon[35869]: pgmap v5506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:38 smithi097 ceph-mon[32231]: pgmap v5506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:40 smithi115 ceph-mon[35364]: pgmap v5507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9137.851272 secs 2024-07-26T10:31:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:40 smithi181 ceph-mon[35869]: pgmap v5507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9137.851272 secs 2024-07-26T10:31:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:40 smithi097 ceph-mon[32231]: pgmap v5507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9137.851272 secs 2024-07-26T10:31:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:42 smithi115 ceph-mon[35364]: pgmap v5508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:42 smithi181 ceph-mon[35869]: pgmap v5508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:42 smithi097 ceph-mon[32231]: pgmap v5508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:44 smithi115 ceph-mon[35364]: pgmap v5509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:44 smithi181 ceph-mon[35869]: pgmap v5509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:44 smithi097 ceph-mon[32231]: pgmap v5509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9142.851377 secs 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:31:46.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:31:46.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:31:46.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:31:46.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:46.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:46.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: pgmap v5510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:31:46.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: pgmap v5511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:46.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9142.851377 secs 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:31:46.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:31:46.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:31:46.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:31:46.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:46.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:46.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: pgmap v5510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:31:46.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: pgmap v5511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:46.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9142.851377 secs 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:46.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:46.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: pgmap v5510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:31:46.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: pgmap v5511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:46.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:31:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:48 smithi115 ceph-mon[35364]: pgmap v5512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:31:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:48 smithi181 ceph-mon[35869]: pgmap v5512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:31:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:48 smithi097 ceph-mon[32231]: pgmap v5512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:31:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:50 smithi115 ceph-mon[35364]: pgmap v5513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:31:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:50 smithi181 ceph-mon[35869]: pgmap v5513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:31:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:50 smithi097 ceph-mon[32231]: pgmap v5513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:31:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9147.851525 secs 2024-07-26T10:31:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9147.851525 secs 2024-07-26T10:31:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9147.851525 secs 2024-07-26T10:31:51.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:51.732+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:31:51.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:51.733+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:31:51.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:51.733+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:31:51.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:51.733+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:31:51.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:31:51.734+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:52 smithi115 ceph-mon[35364]: pgmap v5514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:52 smithi181 ceph-mon[35869]: pgmap v5514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:52 smithi097 ceph-mon[32231]: pgmap v5514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:54 smithi115 ceph-mon[35364]: pgmap v5515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:54 smithi181 ceph-mon[35869]: pgmap v5515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:54 smithi097 ceph-mon[32231]: pgmap v5515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:55 smithi115 ceph-mon[35364]: pgmap v5516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:31:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:55 smithi181 ceph-mon[35869]: pgmap v5516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:31:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:55 smithi097 ceph-mon[32231]: pgmap v5516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:31:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9152.851622 secs 2024-07-26T10:31:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9152.851622 secs 2024-07-26T10:31:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9152.851622 secs 2024-07-26T10:31:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:31:57 smithi115 ceph-mon[35364]: pgmap v5517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:31:57 smithi181 ceph-mon[35869]: pgmap v5517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:31:57 smithi097 ceph-mon[32231]: pgmap v5517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:00 smithi115 ceph-mon[35364]: pgmap v5518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:00 smithi181 ceph-mon[35869]: pgmap v5518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:00 smithi097 ceph-mon[32231]: pgmap v5518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9157.851785 secs 2024-07-26T10:32:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9157.851785 secs 2024-07-26T10:32:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9157.851785 secs 2024-07-26T10:32:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:02 smithi115 ceph-mon[35364]: pgmap v5519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:02 smithi181 ceph-mon[35869]: pgmap v5519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:02 smithi097 ceph-mon[32231]: pgmap v5519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:04 smithi115 ceph-mon[35364]: pgmap v5520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:04 smithi181 ceph-mon[35869]: pgmap v5520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:04 smithi097 ceph-mon[32231]: pgmap v5520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:05 smithi181 ceph-mon[35869]: pgmap v5521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:05 smithi097 ceph-mon[32231]: pgmap v5521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:05 smithi115 ceph-mon[35364]: pgmap v5521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9162.851947 secs 2024-07-26T10:32:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9162.851947 secs 2024-07-26T10:32:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9162.851947 secs 2024-07-26T10:32:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:06.738+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:32:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:06.739+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:32:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:06.739+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:32:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:06.739+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:32:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:06.739+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:07 smithi097 ceph-mon[32231]: pgmap v5522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:07 smithi115 ceph-mon[35364]: pgmap v5522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:07 smithi181 ceph-mon[35869]: pgmap v5522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:10 smithi115 ceph-mon[35364]: pgmap v5523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:10 smithi181 ceph-mon[35869]: pgmap v5523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:10 smithi097 ceph-mon[32231]: pgmap v5523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9167.852090 secs 2024-07-26T10:32:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9167.852090 secs 2024-07-26T10:32:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9167.852090 secs 2024-07-26T10:32:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:12 smithi115 ceph-mon[35364]: pgmap v5524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:12 smithi181 ceph-mon[35869]: pgmap v5524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:12 smithi097 ceph-mon[32231]: pgmap v5524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:14 smithi115 ceph-mon[35364]: pgmap v5525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:32:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:14 smithi181 ceph-mon[35869]: pgmap v5525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:32:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:14 smithi097 ceph-mon[32231]: pgmap v5525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:32:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:15 smithi115 ceph-mon[35364]: pgmap v5526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:32:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:15 smithi181 ceph-mon[35869]: pgmap v5526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:32:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:15 smithi097 ceph-mon[32231]: pgmap v5526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:32:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9172.852283 secs 2024-07-26T10:32:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9172.852283 secs 2024-07-26T10:32:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9172.852283 secs 2024-07-26T10:32:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:17 smithi115 ceph-mon[35364]: pgmap v5527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:32:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:17 smithi181 ceph-mon[35869]: pgmap v5527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:32:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:17 smithi097 ceph-mon[32231]: pgmap v5527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:32:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:19 smithi115 ceph-mon[35364]: pgmap v5528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:32:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:19 smithi181 ceph-mon[35869]: pgmap v5528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:32:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:19 smithi097 ceph-mon[32231]: pgmap v5528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:32:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9177.852366 secs 2024-07-26T10:32:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9177.852366 secs 2024-07-26T10:32:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9177.852366 secs 2024-07-26T10:32:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:21 smithi115 ceph-mon[35364]: pgmap v5529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:21 smithi181 ceph-mon[35869]: pgmap v5529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:21.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:21.740+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:32:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:21.742+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:32:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:21.742+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:32:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:21.742+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:32:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:21.742+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:21 smithi097 ceph-mon[32231]: pgmap v5529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:24 smithi115 ceph-mon[35364]: pgmap v5530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:24 smithi181 ceph-mon[35869]: pgmap v5530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:24 smithi097 ceph-mon[32231]: pgmap v5530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9182.852511 secs 2024-07-26T10:32:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:26 smithi115 ceph-mon[35364]: pgmap v5531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9182.852511 secs 2024-07-26T10:32:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:26 smithi181 ceph-mon[35869]: pgmap v5531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9182.852511 secs 2024-07-26T10:32:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:26 smithi097 ceph-mon[32231]: pgmap v5531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:28 smithi115 ceph-mon[35364]: pgmap v5532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:28 smithi181 ceph-mon[35869]: pgmap v5532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:28 smithi097 ceph-mon[32231]: pgmap v5532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:29 smithi097 ceph-mon[32231]: pgmap v5533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:29 smithi115 ceph-mon[35364]: pgmap v5533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:29 smithi181 ceph-mon[35869]: pgmap v5533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9187.852649 secs 2024-07-26T10:32:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9187.852649 secs 2024-07-26T10:32:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9187.852649 secs 2024-07-26T10:32:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:31 smithi115 ceph-mon[35364]: pgmap v5534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:31 smithi181 ceph-mon[35869]: pgmap v5534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:31 smithi097 ceph-mon[32231]: pgmap v5534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:34.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:34 smithi115 ceph-mon[35364]: pgmap v5535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:34.437 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:34 smithi181 ceph-mon[35869]: pgmap v5535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:34 smithi097 ceph-mon[32231]: pgmap v5535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9192.852809 secs 2024-07-26T10:32:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:36 smithi115 ceph-mon[35364]: pgmap v5536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9192.852809 secs 2024-07-26T10:32:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:36 smithi181 ceph-mon[35869]: pgmap v5536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9192.852809 secs 2024-07-26T10:32:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:36 smithi097 ceph-mon[32231]: pgmap v5536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:36.748+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:32:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:36.749+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:32:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:36.749+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:32:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:36.749+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:32:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:36.750+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:38 smithi115 ceph-mon[35364]: pgmap v5537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:38 smithi181 ceph-mon[35869]: pgmap v5537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:38 smithi097 ceph-mon[32231]: pgmap v5537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:39 smithi115 ceph-mon[35364]: pgmap v5538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:39 smithi181 ceph-mon[35869]: pgmap v5538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:39 smithi097 ceph-mon[32231]: pgmap v5538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9197.852951 secs 2024-07-26T10:32:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9197.852951 secs 2024-07-26T10:32:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9197.852951 secs 2024-07-26T10:32:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:41 smithi115 ceph-mon[35364]: pgmap v5539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:41 smithi181 ceph-mon[35869]: pgmap v5539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:41 smithi097 ceph-mon[32231]: pgmap v5539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:44 smithi115 ceph-mon[35364]: pgmap v5540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:44 smithi181 ceph-mon[35869]: pgmap v5540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:44 smithi097 ceph-mon[32231]: pgmap v5540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9202.853109 secs 2024-07-26T10:32:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:46 smithi115 ceph-mon[35364]: pgmap v5541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9202.853109 secs 2024-07-26T10:32:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:46 smithi181 ceph-mon[35869]: pgmap v5541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:46.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9202.853109 secs 2024-07-26T10:32:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:46 smithi097 ceph-mon[32231]: pgmap v5541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:47.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:47.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:32:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:47.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:47.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:32:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:32:47.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:47.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:47.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:32:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:48 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:32:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:48 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:32:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:48 smithi115 ceph-mon[35364]: pgmap v5542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:32:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:48 smithi115 ceph-mon[35364]: pgmap v5543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:48 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:32:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:48 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:32:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:48 smithi181 ceph-mon[35869]: pgmap v5542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:32:48.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:48 smithi181 ceph-mon[35869]: pgmap v5543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:48 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:32:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:48 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:32:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:48 smithi097 ceph-mon[32231]: pgmap v5542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:32:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:48 smithi097 ceph-mon[32231]: pgmap v5543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:49 smithi115 ceph-mon[35364]: pgmap v5544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:32:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:49 smithi181 ceph-mon[35869]: pgmap v5544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:32:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:49 smithi097 ceph-mon[32231]: pgmap v5544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:32:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9207.853287 secs 2024-07-26T10:32:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9207.853287 secs 2024-07-26T10:32:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9207.853287 secs 2024-07-26T10:32:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:51 smithi115 ceph-mon[35364]: pgmap v5545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:32:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:51 smithi181 ceph-mon[35869]: pgmap v5545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:32:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:51 smithi097 ceph-mon[32231]: pgmap v5545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:32:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:51.753+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:32:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:51.754+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:32:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:51.754+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:32:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:51.754+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:32:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:32:51.754+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:53 smithi115 ceph-mon[35364]: pgmap v5546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:53 smithi181 ceph-mon[35869]: pgmap v5546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:53 smithi097 ceph-mon[32231]: pgmap v5546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:55 smithi115 ceph-mon[35364]: pgmap v5547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9212.853389 secs 2024-07-26T10:32:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:55 smithi181 ceph-mon[35869]: pgmap v5547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9212.853389 secs 2024-07-26T10:32:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:55 smithi097 ceph-mon[32231]: pgmap v5547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9212.853389 secs 2024-07-26T10:32:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:57 smithi115 ceph-mon[35364]: pgmap v5548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:57 smithi181 ceph-mon[35869]: pgmap v5548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:57 smithi097 ceph-mon[32231]: pgmap v5548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:32:59 smithi115 ceph-mon[35364]: pgmap v5549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:59.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:32:59 smithi181 ceph-mon[35869]: pgmap v5549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:32:59 smithi097 ceph-mon[32231]: pgmap v5549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9217.853489 secs 2024-07-26T10:33:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9217.853489 secs 2024-07-26T10:33:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9217.853489 secs 2024-07-26T10:33:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:01 smithi115 ceph-mon[35364]: pgmap v5550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:01 smithi181 ceph-mon[35869]: pgmap v5550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:01 smithi097 ceph-mon[32231]: pgmap v5550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:03 smithi115 ceph-mon[35364]: pgmap v5551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:03 smithi181 ceph-mon[35869]: pgmap v5551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:03 smithi097 ceph-mon[32231]: pgmap v5551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:05 smithi115 ceph-mon[35364]: pgmap v5552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9222.853635 secs 2024-07-26T10:33:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:05 smithi181 ceph-mon[35869]: pgmap v5552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9222.853635 secs 2024-07-26T10:33:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:05 smithi097 ceph-mon[32231]: pgmap v5552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9222.853635 secs 2024-07-26T10:33:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:06.760+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:33:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:06.762+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:33:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:06.762+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:33:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:06.762+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:33:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:06.762+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:07 smithi115 ceph-mon[35364]: pgmap v5553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:07 smithi181 ceph-mon[35869]: pgmap v5553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:07 smithi097 ceph-mon[32231]: pgmap v5553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:09 smithi115 ceph-mon[35364]: pgmap v5554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:09 smithi181 ceph-mon[35869]: pgmap v5554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:09 smithi097 ceph-mon[32231]: pgmap v5554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9227.853779 secs 2024-07-26T10:33:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9227.853779 secs 2024-07-26T10:33:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9227.853779 secs 2024-07-26T10:33:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:11 smithi115 ceph-mon[35364]: pgmap v5555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:11 smithi181 ceph-mon[35869]: pgmap v5555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:11 smithi097 ceph-mon[32231]: pgmap v5555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:13 smithi115 ceph-mon[35364]: pgmap v5556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:13 smithi181 ceph-mon[35869]: pgmap v5556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:13 smithi097 ceph-mon[32231]: pgmap v5556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:15 smithi115 ceph-mon[35364]: pgmap v5557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9232.853872 secs 2024-07-26T10:33:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:15 smithi181 ceph-mon[35869]: pgmap v5557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9232.853872 secs 2024-07-26T10:33:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:15 smithi097 ceph-mon[32231]: pgmap v5557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9232.853872 secs 2024-07-26T10:33:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:17 smithi115 ceph-mon[35364]: pgmap v5558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:17 smithi181 ceph-mon[35869]: pgmap v5558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:17 smithi097 ceph-mon[32231]: pgmap v5558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:19 smithi115 ceph-mon[35364]: pgmap v5559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:33:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:19 smithi181 ceph-mon[35869]: pgmap v5559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:33:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:19 smithi097 ceph-mon[32231]: pgmap v5559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:33:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9237.854028 secs 2024-07-26T10:33:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9237.854028 secs 2024-07-26T10:33:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9237.854028 secs 2024-07-26T10:33:21.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:21.764+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:33:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:21.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:33:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:21.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:33:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:21.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:33:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:21.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:21 smithi115 ceph-mon[35364]: pgmap v5560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:33:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:21 smithi181 ceph-mon[35869]: pgmap v5560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:33:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:21 smithi097 ceph-mon[32231]: pgmap v5560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:33:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:23 smithi115 ceph-mon[35364]: pgmap v5561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:33:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:23 smithi181 ceph-mon[35869]: pgmap v5561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:33:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:23 smithi097 ceph-mon[32231]: pgmap v5561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:33:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:25 smithi115 ceph-mon[35364]: pgmap v5562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:33:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9242.854170 secs 2024-07-26T10:33:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:25 smithi181 ceph-mon[35869]: pgmap v5562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:33:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9242.854170 secs 2024-07-26T10:33:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:25 smithi097 ceph-mon[32231]: pgmap v5562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:33:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9242.854170 secs 2024-07-26T10:33:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:27 smithi115 ceph-mon[35364]: pgmap v5563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:33:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:27 smithi181 ceph-mon[35869]: pgmap v5563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:33:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:27 smithi097 ceph-mon[32231]: pgmap v5563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:33:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:29 smithi115 ceph-mon[35364]: pgmap v5564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:29 smithi181 ceph-mon[35869]: pgmap v5564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:29 smithi097 ceph-mon[32231]: pgmap v5564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9247.854320 secs 2024-07-26T10:33:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9247.854320 secs 2024-07-26T10:33:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9247.854320 secs 2024-07-26T10:33:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:31 smithi097 ceph-mon[32231]: pgmap v5565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:31 smithi115 ceph-mon[35364]: pgmap v5565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:31 smithi181 ceph-mon[35869]: pgmap v5565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:33 smithi097 ceph-mon[32231]: pgmap v5566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:33 smithi115 ceph-mon[35364]: pgmap v5566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:33 smithi181 ceph-mon[35869]: pgmap v5566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:35 smithi097 ceph-mon[32231]: pgmap v5567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9252.854427 secs 2024-07-26T10:33:36.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:35 smithi115 ceph-mon[35364]: pgmap v5567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:36.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9252.854427 secs 2024-07-26T10:33:36.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:35 smithi181 ceph-mon[35869]: pgmap v5567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:36.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9252.854427 secs 2024-07-26T10:33:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:36.770+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:33:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:36.772+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:33:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:36.772+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:33:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:36.772+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:33:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:36.772+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:37 smithi115 ceph-mon[35364]: pgmap v5568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:37 smithi181 ceph-mon[35869]: pgmap v5568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:37 smithi097 ceph-mon[32231]: pgmap v5568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:39 smithi115 ceph-mon[35364]: pgmap v5569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:39 smithi181 ceph-mon[35869]: pgmap v5569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:39 smithi097 ceph-mon[32231]: pgmap v5569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9257.854533 secs 2024-07-26T10:33:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9257.854533 secs 2024-07-26T10:33:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9257.854533 secs 2024-07-26T10:33:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:41 smithi115 ceph-mon[35364]: pgmap v5570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:41 smithi181 ceph-mon[35869]: pgmap v5570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:41 smithi097 ceph-mon[32231]: pgmap v5570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:43 smithi115 ceph-mon[35364]: pgmap v5571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:43 smithi181 ceph-mon[35869]: pgmap v5571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:43 smithi097 ceph-mon[32231]: pgmap v5571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:45 smithi115 ceph-mon[35364]: pgmap v5572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9262.854631 secs 2024-07-26T10:33:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:45 smithi181 ceph-mon[35869]: pgmap v5572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9262.854631 secs 2024-07-26T10:33:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:45 smithi097 ceph-mon[32231]: pgmap v5572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9262.854631 secs 2024-07-26T10:33:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:47 smithi115 ceph-mon[35364]: pgmap v5573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:47 smithi181 ceph-mon[35869]: pgmap v5573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:47 smithi097 ceph-mon[32231]: pgmap v5573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:33:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:33:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:49 smithi115 ceph-mon[35364]: pgmap v5574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:49 smithi115 ceph-mon[35364]: pgmap v5575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:49 smithi181 ceph-mon[35869]: pgmap v5574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:50.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:49 smithi181 ceph-mon[35869]: pgmap v5575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:49 smithi097 ceph-mon[32231]: pgmap v5574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:49 smithi097 ceph-mon[32231]: pgmap v5575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9267.854793 secs 2024-07-26T10:33:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9267.854793 secs 2024-07-26T10:33:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9267.854793 secs 2024-07-26T10:33:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:51.774+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:33:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:51.775+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:33:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:51.775+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:33:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:51.775+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:33:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:33:51.775+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:51 smithi115 ceph-mon[35364]: pgmap v5576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:51 smithi181 ceph-mon[35869]: pgmap v5576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:51 smithi097 ceph-mon[32231]: pgmap v5576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:53 smithi115 ceph-mon[35364]: pgmap v5577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:53 smithi181 ceph-mon[35869]: pgmap v5577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:53 smithi097 ceph-mon[32231]: pgmap v5577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:55 smithi115 ceph-mon[35364]: pgmap v5578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9272.854936 secs 2024-07-26T10:33:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:55 smithi181 ceph-mon[35869]: pgmap v5578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9272.854936 secs 2024-07-26T10:33:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:55 smithi097 ceph-mon[32231]: pgmap v5578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9272.854936 secs 2024-07-26T10:33:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:57 smithi115 ceph-mon[35364]: pgmap v5579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:57 smithi181 ceph-mon[35869]: pgmap v5579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:57 smithi097 ceph-mon[32231]: pgmap v5579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:34:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:33:59 smithi115 ceph-mon[35364]: pgmap v5580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:33:59 smithi181 ceph-mon[35869]: pgmap v5580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:33:59 smithi097 ceph-mon[32231]: pgmap v5580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9277.855043 secs 2024-07-26T10:34:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9277.855043 secs 2024-07-26T10:34:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9277.855043 secs 2024-07-26T10:34:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:01 smithi115 ceph-mon[35364]: pgmap v5581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:01 smithi181 ceph-mon[35869]: pgmap v5581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:01 smithi097 ceph-mon[32231]: pgmap v5581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:03 smithi115 ceph-mon[35364]: pgmap v5582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:03 smithi181 ceph-mon[35869]: pgmap v5582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:03 smithi097 ceph-mon[32231]: pgmap v5582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:05 smithi115 ceph-mon[35364]: pgmap v5583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9282.855197 secs 2024-07-26T10:34:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:05 smithi181 ceph-mon[35869]: pgmap v5583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9282.855197 secs 2024-07-26T10:34:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:05 smithi097 ceph-mon[32231]: pgmap v5583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9282.855197 secs 2024-07-26T10:34:06.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:06.782+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:34:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:06.783+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:34:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:06.783+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:34:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:06.783+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:34:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:06.783+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:07 smithi115 ceph-mon[35364]: pgmap v5584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:07 smithi181 ceph-mon[35869]: pgmap v5584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:07 smithi097 ceph-mon[32231]: pgmap v5584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:08.525+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:34:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:08.525+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:34:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:08.525+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:34:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:08.525+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:34:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:09 smithi115 ceph-mon[35364]: pgmap v5585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:09 smithi181 ceph-mon[35869]: pgmap v5585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:09 smithi097 ceph-mon[32231]: pgmap v5585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9287.855344 secs 2024-07-26T10:34:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9287.855344 secs 2024-07-26T10:34:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9287.855344 secs 2024-07-26T10:34:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:11 smithi115 ceph-mon[35364]: pgmap v5586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:11 smithi181 ceph-mon[35869]: pgmap v5586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:11 smithi097 ceph-mon[32231]: pgmap v5586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:13 smithi115 ceph-mon[35364]: pgmap v5587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:13 smithi181 ceph-mon[35869]: pgmap v5587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:13 smithi097 ceph-mon[32231]: pgmap v5587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:15 smithi115 ceph-mon[35364]: pgmap v5588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9292.855487 secs 2024-07-26T10:34:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:15 smithi181 ceph-mon[35869]: pgmap v5588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9292.855487 secs 2024-07-26T10:34:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:15 smithi097 ceph-mon[32231]: pgmap v5588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9292.855487 secs 2024-07-26T10:34:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:17 smithi115 ceph-mon[35364]: pgmap v5589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:17 smithi181 ceph-mon[35869]: pgmap v5589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:17 smithi097 ceph-mon[32231]: pgmap v5589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:19 smithi115 ceph-mon[35364]: pgmap v5590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:19 smithi181 ceph-mon[35869]: pgmap v5590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:19 smithi097 ceph-mon[32231]: pgmap v5590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9297.855627 secs 2024-07-26T10:34:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9297.855627 secs 2024-07-26T10:34:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9297.855627 secs 2024-07-26T10:34:21.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:21.784+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:34:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:21.786+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:34:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:21.786+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:34:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:21.786+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:34:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:21.786+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:21 smithi115 ceph-mon[35364]: pgmap v5591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:21 smithi181 ceph-mon[35869]: pgmap v5591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:21 smithi097 ceph-mon[32231]: pgmap v5591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:23 smithi115 ceph-mon[35364]: pgmap v5592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:23 smithi181 ceph-mon[35869]: pgmap v5592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:23 smithi097 ceph-mon[32231]: pgmap v5592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:25 smithi115 ceph-mon[35364]: pgmap v5593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9302.855712 secs 2024-07-26T10:34:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:25 smithi181 ceph-mon[35869]: pgmap v5593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9302.855712 secs 2024-07-26T10:34:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:25 smithi097 ceph-mon[32231]: pgmap v5593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9302.855712 secs 2024-07-26T10:34:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:27 smithi181 ceph-mon[35869]: pgmap v5594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:27 smithi097 ceph-mon[32231]: pgmap v5594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:27 smithi115 ceph-mon[35364]: pgmap v5594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:29 smithi097 ceph-mon[32231]: pgmap v5595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:29 smithi115 ceph-mon[35364]: pgmap v5595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:29 smithi181 ceph-mon[35869]: pgmap v5595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9307.855867 secs 2024-07-26T10:34:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9307.855867 secs 2024-07-26T10:34:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9307.855867 secs 2024-07-26T10:34:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:32 smithi115 ceph-mon[35364]: pgmap v5596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:32 smithi181 ceph-mon[35869]: pgmap v5596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:32 smithi097 ceph-mon[32231]: pgmap v5596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:34 smithi115 ceph-mon[35364]: pgmap v5597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:34 smithi181 ceph-mon[35869]: pgmap v5597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:34 smithi097 ceph-mon[32231]: pgmap v5597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:36.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:36 smithi115 ceph-mon[35364]: pgmap v5598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:36.279 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9312.856031 secs 2024-07-26T10:34:36.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:36 smithi181 ceph-mon[35869]: pgmap v5598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:36.287 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9312.856031 secs 2024-07-26T10:34:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:36 smithi097 ceph-mon[32231]: pgmap v5598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9312.856031 secs 2024-07-26T10:34:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:36.788+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:34:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:36.789+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:34:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:36.789+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:34:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:36.789+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:34:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:36.789+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:38.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:38 smithi097 ceph-mon[32231]: pgmap v5599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:38 smithi115 ceph-mon[35364]: pgmap v5599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:38 smithi181 ceph-mon[35869]: pgmap v5599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:40 smithi115 ceph-mon[35364]: pgmap v5600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:40 smithi181 ceph-mon[35869]: pgmap v5600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:40 smithi097 ceph-mon[32231]: pgmap v5600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9317.856178 secs 2024-07-26T10:34:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9317.856178 secs 2024-07-26T10:34:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9317.856178 secs 2024-07-26T10:34:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:42 smithi115 ceph-mon[35364]: pgmap v5601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:42 smithi181 ceph-mon[35869]: pgmap v5601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:42 smithi097 ceph-mon[32231]: pgmap v5601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:44 smithi115 ceph-mon[35364]: pgmap v5602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:44 smithi181 ceph-mon[35869]: pgmap v5602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:44 smithi097 ceph-mon[32231]: pgmap v5602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:46 smithi115 ceph-mon[35364]: pgmap v5603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9322.856325 secs 2024-07-26T10:34:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:46 smithi181 ceph-mon[35869]: pgmap v5603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9322.856325 secs 2024-07-26T10:34:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:46 smithi097 ceph-mon[32231]: pgmap v5603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9322.856325 secs 2024-07-26T10:34:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:48 smithi115 ceph-mon[35364]: pgmap v5604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:48 smithi181 ceph-mon[35869]: pgmap v5604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:48 smithi097 ceph-mon[32231]: pgmap v5604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:50 smithi115 ceph-mon[35364]: pgmap v5605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:50 smithi181 ceph-mon[35869]: pgmap v5605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:50 smithi097 ceph-mon[32231]: pgmap v5605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:34:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:50.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9327.856428 secs 2024-07-26T10:34:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9327.856428 secs 2024-07-26T10:34:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9327.856428 secs 2024-07-26T10:34:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:34:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:51.791+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:34:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:51.792+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:34:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:51.792+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:34:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:51.792+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:34:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:51.792+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:52 smithi115 ceph-mon[35364]: pgmap v5606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:52 smithi115 ceph-mon[35364]: pgmap v5607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:52 smithi181 ceph-mon[35869]: pgmap v5606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:52 smithi181 ceph-mon[35869]: pgmap v5607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:52 smithi097 ceph-mon[32231]: pgmap v5606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:52 smithi097 ceph-mon[32231]: pgmap v5607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:53.533+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:34:53.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:53.533+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:34:53.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:53.533+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:34:53.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:34:53.533+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:34:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:54 smithi115 ceph-mon[35364]: pgmap v5608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:54 smithi181 ceph-mon[35869]: pgmap v5608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:54 smithi097 ceph-mon[32231]: pgmap v5608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:34:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:56 smithi115 ceph-mon[35364]: pgmap v5609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:34:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9332.856570 secs 2024-07-26T10:34:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:56 smithi181 ceph-mon[35869]: pgmap v5609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:34:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9332.856570 secs 2024-07-26T10:34:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:56 smithi097 ceph-mon[32231]: pgmap v5609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:34:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9332.856570 secs 2024-07-26T10:34:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:34:58 smithi115 ceph-mon[35364]: pgmap v5610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:34:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:34:58 smithi181 ceph-mon[35869]: pgmap v5610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:34:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:34:58 smithi097 ceph-mon[32231]: pgmap v5610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:35:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:00 smithi115 ceph-mon[35364]: pgmap v5611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:35:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:00 smithi181 ceph-mon[35869]: pgmap v5611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:35:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:00 smithi097 ceph-mon[32231]: pgmap v5611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:35:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9337.856715 secs 2024-07-26T10:35:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9337.856715 secs 2024-07-26T10:35:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9337.856715 secs 2024-07-26T10:35:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:02 smithi115 ceph-mon[35364]: pgmap v5612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:35:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:02 smithi181 ceph-mon[35869]: pgmap v5612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:35:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:02 smithi097 ceph-mon[32231]: pgmap v5612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:35:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:04 smithi115 ceph-mon[35364]: pgmap v5613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:04 smithi181 ceph-mon[35869]: pgmap v5613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:04 smithi097 ceph-mon[32231]: pgmap v5613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:06 smithi115 ceph-mon[35364]: pgmap v5614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:06.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9342.856806 secs 2024-07-26T10:35:06.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:06 smithi181 ceph-mon[35869]: pgmap v5614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:06.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9342.856806 secs 2024-07-26T10:35:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:06 smithi097 ceph-mon[32231]: pgmap v5614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9342.856806 secs 2024-07-26T10:35:06.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:06.798+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:35:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:06.800+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:35:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:06.800+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:35:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:06.800+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:35:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:06.800+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:08 smithi115 ceph-mon[35364]: pgmap v5615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:08 smithi181 ceph-mon[35869]: pgmap v5615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:08 smithi097 ceph-mon[32231]: pgmap v5615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:10 smithi115 ceph-mon[35364]: pgmap v5616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:10 smithi181 ceph-mon[35869]: pgmap v5616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:10 smithi097 ceph-mon[32231]: pgmap v5616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9347.856955 secs 2024-07-26T10:35:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9347.856955 secs 2024-07-26T10:35:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9347.856955 secs 2024-07-26T10:35:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:12 smithi115 ceph-mon[35364]: pgmap v5617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:12 smithi181 ceph-mon[35869]: pgmap v5617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:12 smithi097 ceph-mon[32231]: pgmap v5617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:14 smithi115 ceph-mon[35364]: pgmap v5618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:14 smithi181 ceph-mon[35869]: pgmap v5618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:14 smithi097 ceph-mon[32231]: pgmap v5618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:16 smithi115 ceph-mon[35364]: pgmap v5619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9352.857096 secs 2024-07-26T10:35:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:16 smithi181 ceph-mon[35869]: pgmap v5619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9352.857096 secs 2024-07-26T10:35:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:16 smithi097 ceph-mon[32231]: pgmap v5619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9352.857096 secs 2024-07-26T10:35:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:18 smithi115 ceph-mon[35364]: pgmap v5620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:18 smithi181 ceph-mon[35869]: pgmap v5620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:18 smithi097 ceph-mon[32231]: pgmap v5620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:20 smithi115 ceph-mon[35364]: pgmap v5621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:20 smithi181 ceph-mon[35869]: pgmap v5621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:20 smithi097 ceph-mon[32231]: pgmap v5621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9357.857276 secs 2024-07-26T10:35:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9357.857276 secs 2024-07-26T10:35:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9357.857276 secs 2024-07-26T10:35:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:21.801+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:35:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:21.802+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:35:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:21.802+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:35:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:21.802+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:35:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:21.802+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:22 smithi097 ceph-mon[32231]: pgmap v5622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:22 smithi115 ceph-mon[35364]: pgmap v5622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:22 smithi181 ceph-mon[35869]: pgmap v5622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:23 smithi097 ceph-mon[32231]: pgmap v5623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:23 smithi115 ceph-mon[35364]: pgmap v5623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:23 smithi181 ceph-mon[35869]: pgmap v5623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:25 smithi115 ceph-mon[35364]: pgmap v5624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9362.857421 secs 2024-07-26T10:35:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:25 smithi181 ceph-mon[35869]: pgmap v5624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9362.857421 secs 2024-07-26T10:35:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:25 smithi097 ceph-mon[32231]: pgmap v5624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9362.857421 secs 2024-07-26T10:35:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:27 smithi115 ceph-mon[35364]: pgmap v5625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:27 smithi181 ceph-mon[35869]: pgmap v5625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:27 smithi097 ceph-mon[32231]: pgmap v5625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:29 smithi115 ceph-mon[35364]: pgmap v5626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:29 smithi181 ceph-mon[35869]: pgmap v5626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:29 smithi097 ceph-mon[32231]: pgmap v5626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9367.857585 secs 2024-07-26T10:35:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9367.857585 secs 2024-07-26T10:35:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9367.857585 secs 2024-07-26T10:35:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:31 smithi115 ceph-mon[35364]: pgmap v5627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:31 smithi181 ceph-mon[35869]: pgmap v5627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:31 smithi097 ceph-mon[32231]: pgmap v5627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:33 smithi115 ceph-mon[35364]: pgmap v5628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:33 smithi181 ceph-mon[35869]: pgmap v5628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:33 smithi097 ceph-mon[32231]: pgmap v5628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:35 smithi115 ceph-mon[35364]: pgmap v5629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9372.857740 secs 2024-07-26T10:35:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:35 smithi181 ceph-mon[35869]: pgmap v5629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9372.857740 secs 2024-07-26T10:35:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:35 smithi097 ceph-mon[32231]: pgmap v5629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9372.857740 secs 2024-07-26T10:35:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:36.810+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:35:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:36.811+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:35:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:36.811+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:35:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:36.811+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:35:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:36.811+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:37 smithi115 ceph-mon[35364]: pgmap v5630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:37 smithi181 ceph-mon[35869]: pgmap v5630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:37 smithi097 ceph-mon[32231]: pgmap v5630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:39 smithi115 ceph-mon[35364]: pgmap v5631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:39 smithi181 ceph-mon[35869]: pgmap v5631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:39 smithi097 ceph-mon[32231]: pgmap v5631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9377.857901 secs 2024-07-26T10:35:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9377.857901 secs 2024-07-26T10:35:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9377.857901 secs 2024-07-26T10:35:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:41 smithi115 ceph-mon[35364]: pgmap v5632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:41 smithi181 ceph-mon[35869]: pgmap v5632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:41 smithi097 ceph-mon[32231]: pgmap v5632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:43 smithi115 ceph-mon[35364]: pgmap v5633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:43 smithi181 ceph-mon[35869]: pgmap v5633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:43 smithi097 ceph-mon[32231]: pgmap v5633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:45 smithi115 ceph-mon[35364]: pgmap v5634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9382.858053 secs 2024-07-26T10:35:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:45 smithi181 ceph-mon[35869]: pgmap v5634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9382.858053 secs 2024-07-26T10:35:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:45 smithi097 ceph-mon[32231]: pgmap v5634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9382.858053 secs 2024-07-26T10:35:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:47 smithi115 ceph-mon[35364]: pgmap v5635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:47 smithi181 ceph-mon[35869]: pgmap v5635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:47 smithi097 ceph-mon[32231]: pgmap v5635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:49 smithi115 ceph-mon[35364]: pgmap v5636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:49 smithi181 ceph-mon[35869]: pgmap v5636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:49 smithi097 ceph-mon[32231]: pgmap v5636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9387.858212 secs 2024-07-26T10:35:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9387.858212 secs 2024-07-26T10:35:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9387.858212 secs 2024-07-26T10:35:51.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:51.813+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:35:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:51.815+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:35:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:51.815+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:35:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:51.815+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:35:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:35:51.815+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:51.961 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:51 smithi181 ceph-mon[35869]: pgmap v5637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:51.961 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:51.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:51 smithi115 ceph-mon[35364]: pgmap v5637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:51.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:51 smithi097 ceph-mon[32231]: pgmap v5637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:53 smithi115 ceph-mon[35364]: pgmap v5638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:53 smithi181 ceph-mon[35869]: pgmap v5638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:53 smithi097 ceph-mon[32231]: pgmap v5638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:55 smithi115 ceph-mon[35364]: pgmap v5639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9392.858351 secs 2024-07-26T10:35:56.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:55 smithi115 ceph-mon[35364]: pgmap v5640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:56.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:55 smithi115 ceph-mon[35364]: pgmap v5641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T10:35:56.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:55 smithi181 ceph-mon[35869]: pgmap v5639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9392.858351 secs 2024-07-26T10:35:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:55 smithi181 ceph-mon[35869]: pgmap v5640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:55 smithi181 ceph-mon[35869]: pgmap v5641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T10:35:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:55 smithi097 ceph-mon[32231]: pgmap v5639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:35:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9392.858351 secs 2024-07-26T10:35:56.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:55 smithi097 ceph-mon[32231]: pgmap v5640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:35:56.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:55 smithi097 ceph-mon[32231]: pgmap v5641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T10:35:56.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:35:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:57 smithi115 ceph-mon[35364]: pgmap v5642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:35:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:57 smithi181 ceph-mon[35869]: pgmap v5642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:35:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:57 smithi097 ceph-mon[32231]: pgmap v5642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:35:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:35:59 smithi115 ceph-mon[35364]: pgmap v5643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:35:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:35:59 smithi181 ceph-mon[35869]: pgmap v5643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:35:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:35:59 smithi097 ceph-mon[32231]: pgmap v5643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:36:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9397.858504 secs 2024-07-26T10:36:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9397.858504 secs 2024-07-26T10:36:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9397.858504 secs 2024-07-26T10:36:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:01 smithi115 ceph-mon[35364]: pgmap v5644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T10:36:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:01 smithi181 ceph-mon[35869]: pgmap v5644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T10:36:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:01 smithi097 ceph-mon[32231]: pgmap v5644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T10:36:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:03 smithi115 ceph-mon[35364]: pgmap v5645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:36:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:03 smithi181 ceph-mon[35869]: pgmap v5645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:36:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:03 smithi097 ceph-mon[32231]: pgmap v5645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:36:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:05 smithi115 ceph-mon[35364]: pgmap v5646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:36:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:05 smithi181 ceph-mon[35869]: pgmap v5646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:36:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:05 smithi097 ceph-mon[32231]: pgmap v5646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:36:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9402.858600 secs 2024-07-26T10:36:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9402.858600 secs 2024-07-26T10:36:06.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9402.858600 secs 2024-07-26T10:36:06.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:06.821+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:36:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:06.822+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:36:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:06.822+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:36:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:06.822+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:36:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:06.822+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:07.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:07 smithi115 ceph-mon[35364]: pgmap v5647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:07.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:07 smithi181 ceph-mon[35869]: pgmap v5647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:07 smithi097 ceph-mon[32231]: pgmap v5647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:09.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:09 smithi097 ceph-mon[32231]: pgmap v5648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:09 smithi115 ceph-mon[35364]: pgmap v5648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:09 smithi181 ceph-mon[35869]: pgmap v5648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9407.858742 secs 2024-07-26T10:36:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9407.858742 secs 2024-07-26T10:36:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9407.858742 secs 2024-07-26T10:36:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:12 smithi115 ceph-mon[35364]: pgmap v5649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:12 smithi181 ceph-mon[35869]: pgmap v5649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:12 smithi097 ceph-mon[32231]: pgmap v5649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:14 smithi115 ceph-mon[35364]: pgmap v5650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:14 smithi181 ceph-mon[35869]: pgmap v5650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:14 smithi097 ceph-mon[32231]: pgmap v5650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9412.858886 secs 2024-07-26T10:36:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9412.858886 secs 2024-07-26T10:36:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9412.858886 secs 2024-07-26T10:36:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:16 smithi097 ceph-mon[32231]: pgmap v5651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:16 smithi115 ceph-mon[35364]: pgmap v5651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:16 smithi181 ceph-mon[35869]: pgmap v5651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:18 smithi097 ceph-mon[32231]: pgmap v5652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:18 smithi115 ceph-mon[35364]: pgmap v5652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:18 smithi181 ceph-mon[35869]: pgmap v5652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:20 smithi097 ceph-mon[32231]: pgmap v5653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9417.859030 secs 2024-07-26T10:36:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:20 smithi115 ceph-mon[35364]: pgmap v5653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9417.859030 secs 2024-07-26T10:36:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:20 smithi181 ceph-mon[35869]: pgmap v5653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9417.859030 secs 2024-07-26T10:36:21.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:21.824+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:36:21.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:21.825+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:36:21.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:21.825+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:36:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:21.825+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:36:21.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:21.825+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:22 smithi115 ceph-mon[35364]: pgmap v5654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:22 smithi181 ceph-mon[35869]: pgmap v5654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:22 smithi097 ceph-mon[32231]: pgmap v5654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:24 smithi115 ceph-mon[35364]: pgmap v5655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:24 smithi181 ceph-mon[35869]: pgmap v5655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:24 smithi097 ceph-mon[32231]: pgmap v5655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9422.859175 secs 2024-07-26T10:36:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9422.859175 secs 2024-07-26T10:36:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9422.859175 secs 2024-07-26T10:36:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:26 smithi115 ceph-mon[35364]: pgmap v5656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:26 smithi181 ceph-mon[35869]: pgmap v5656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:26 smithi097 ceph-mon[32231]: pgmap v5656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:28 smithi115 ceph-mon[35364]: pgmap v5657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:28 smithi181 ceph-mon[35869]: pgmap v5657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:28 smithi097 ceph-mon[32231]: pgmap v5657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:30 smithi115 ceph-mon[35364]: pgmap v5658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9427.859323 secs 2024-07-26T10:36:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:30 smithi181 ceph-mon[35869]: pgmap v5658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9427.859323 secs 2024-07-26T10:36:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:30 smithi097 ceph-mon[32231]: pgmap v5658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9427.859323 secs 2024-07-26T10:36:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:32 smithi115 ceph-mon[35364]: pgmap v5659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:32 smithi181 ceph-mon[35869]: pgmap v5659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:32 smithi097 ceph-mon[32231]: pgmap v5659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:34.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:34 smithi115 ceph-mon[35364]: pgmap v5660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:34.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:34 smithi181 ceph-mon[35869]: pgmap v5660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:34 smithi097 ceph-mon[32231]: pgmap v5660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9432.859460 secs 2024-07-26T10:36:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9432.859460 secs 2024-07-26T10:36:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9432.859460 secs 2024-07-26T10:36:36.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:36.832+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:36:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:36.833+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:36:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:36.833+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:36:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:36.833+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:36:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:36.833+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:36 smithi115 ceph-mon[35364]: pgmap v5661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:36 smithi181 ceph-mon[35869]: pgmap v5661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:36 smithi097 ceph-mon[32231]: pgmap v5661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:38 smithi115 ceph-mon[35364]: pgmap v5662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:38 smithi181 ceph-mon[35869]: pgmap v5662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:38 smithi097 ceph-mon[32231]: pgmap v5662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:40 smithi115 ceph-mon[35364]: pgmap v5663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9437.859632 secs 2024-07-26T10:36:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:40 smithi181 ceph-mon[35869]: pgmap v5663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9437.859632 secs 2024-07-26T10:36:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:40 smithi097 ceph-mon[32231]: pgmap v5663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9437.859632 secs 2024-07-26T10:36:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:42 smithi115 ceph-mon[35364]: pgmap v5664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:42 smithi181 ceph-mon[35869]: pgmap v5664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:42 smithi097 ceph-mon[32231]: pgmap v5664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:44 smithi115 ceph-mon[35364]: pgmap v5665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:44 smithi181 ceph-mon[35869]: pgmap v5665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:44 smithi097 ceph-mon[32231]: pgmap v5665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9442.859794 secs 2024-07-26T10:36:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9442.859794 secs 2024-07-26T10:36:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9442.859794 secs 2024-07-26T10:36:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:46 smithi115 ceph-mon[35364]: pgmap v5666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:46 smithi181 ceph-mon[35869]: pgmap v5666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:46 smithi097 ceph-mon[32231]: pgmap v5666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:48 smithi115 ceph-mon[35364]: pgmap v5667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:48 smithi181 ceph-mon[35869]: pgmap v5667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:48 smithi097 ceph-mon[32231]: pgmap v5667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:50 smithi115 ceph-mon[35364]: pgmap v5668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9447.859957 secs 2024-07-26T10:36:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:50 smithi181 ceph-mon[35869]: pgmap v5668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9447.859957 secs 2024-07-26T10:36:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:50 smithi097 ceph-mon[32231]: pgmap v5668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:36:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9447.859957 secs 2024-07-26T10:36:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:51.834+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:36:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:51.836+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:36:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:51.836+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:36:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:51.836+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:36:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:36:51.836+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:52 smithi115 ceph-mon[35364]: pgmap v5669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:52 smithi181 ceph-mon[35869]: pgmap v5669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:52 smithi097 ceph-mon[32231]: pgmap v5669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:36:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:54 smithi115 ceph-mon[35364]: pgmap v5670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:54 smithi181 ceph-mon[35869]: pgmap v5670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:54 smithi097 ceph-mon[32231]: pgmap v5670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:36:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9452.860115 secs 2024-07-26T10:36:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9452.860115 secs 2024-07-26T10:36:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9452.860115 secs 2024-07-26T10:36:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:56 smithi115 ceph-mon[35364]: pgmap v5671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:56 smithi181 ceph-mon[35869]: pgmap v5671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:56 smithi097 ceph-mon[32231]: pgmap v5671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:57 smithi115 ceph-mon[35364]: pgmap v5672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:36:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:36:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:57 smithi181 ceph-mon[35869]: pgmap v5672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:36:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:36:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:57 smithi097 ceph-mon[32231]: pgmap v5672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:36:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:37:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:36:59 smithi115 ceph-mon[35364]: pgmap v5673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:36:59 smithi181 ceph-mon[35869]: pgmap v5673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:36:59 smithi097 ceph-mon[32231]: pgmap v5673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9457.860309 secs 2024-07-26T10:37:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9457.860309 secs 2024-07-26T10:37:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9457.860309 secs 2024-07-26T10:37:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:01 smithi115 ceph-mon[35364]: pgmap v5674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:37:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:01 smithi181 ceph-mon[35869]: pgmap v5674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:37:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:01 smithi097 ceph-mon[32231]: pgmap v5674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:37:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:03 smithi115 ceph-mon[35364]: pgmap v5675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:37:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:03 smithi181 ceph-mon[35869]: pgmap v5675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:37:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:03 smithi097 ceph-mon[32231]: pgmap v5675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:37:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:05 smithi115 ceph-mon[35364]: pgmap v5676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9462.860414 secs 2024-07-26T10:37:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:05 smithi181 ceph-mon[35869]: pgmap v5676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9462.860414 secs 2024-07-26T10:37:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:05 smithi097 ceph-mon[32231]: pgmap v5676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9462.860414 secs 2024-07-26T10:37:06.844 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:06.842+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:37:06.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:06.844+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:37:06.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:06.844+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:37:06.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:06.844+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:37:06.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:06.844+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:07 smithi115 ceph-mon[35364]: pgmap v5677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:37:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:07 smithi181 ceph-mon[35869]: pgmap v5677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:37:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:07 smithi097 ceph-mon[32231]: pgmap v5677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:37:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:08.557+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:37:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:08.557+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:37:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:08.557+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:37:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:08.558+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:37:10.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:09 smithi097 ceph-mon[32231]: pgmap v5678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:09 smithi115 ceph-mon[35364]: pgmap v5678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:09 smithi181 ceph-mon[35869]: pgmap v5678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9467.860575 secs 2024-07-26T10:37:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9467.860575 secs 2024-07-26T10:37:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9467.860575 secs 2024-07-26T10:37:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:11 smithi115 ceph-mon[35364]: pgmap v5679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:11 smithi181 ceph-mon[35869]: pgmap v5679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:11 smithi097 ceph-mon[32231]: pgmap v5679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:13 smithi115 ceph-mon[35364]: pgmap v5680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:13 smithi181 ceph-mon[35869]: pgmap v5680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:13 smithi097 ceph-mon[32231]: pgmap v5680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:15 smithi115 ceph-mon[35364]: pgmap v5681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9472.860727 secs 2024-07-26T10:37:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:15 smithi181 ceph-mon[35869]: pgmap v5681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9472.860727 secs 2024-07-26T10:37:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:15 smithi097 ceph-mon[32231]: pgmap v5681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9472.860727 secs 2024-07-26T10:37:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:17 smithi115 ceph-mon[35364]: pgmap v5682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:17 smithi181 ceph-mon[35869]: pgmap v5682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:17 smithi097 ceph-mon[32231]: pgmap v5682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:19 smithi115 ceph-mon[35364]: pgmap v5683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:19 smithi181 ceph-mon[35869]: pgmap v5683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:19 smithi097 ceph-mon[32231]: pgmap v5683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9477.860877 secs 2024-07-26T10:37:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9477.860877 secs 2024-07-26T10:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9477.860877 secs 2024-07-26T10:37:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:21.846+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:37:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:21.847+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:37:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:21.847+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:37:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:21.847+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:37:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:21.847+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:21 smithi115 ceph-mon[35364]: pgmap v5684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:21 smithi181 ceph-mon[35869]: pgmap v5684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:21 smithi097 ceph-mon[32231]: pgmap v5684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:23 smithi115 ceph-mon[35364]: pgmap v5685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:23 smithi181 ceph-mon[35869]: pgmap v5685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:23 smithi097 ceph-mon[32231]: pgmap v5685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:25 smithi115 ceph-mon[35364]: pgmap v5686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:37:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9482.860997 secs 2024-07-26T10:37:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:25 smithi181 ceph-mon[35869]: pgmap v5686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:37:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9482.860997 secs 2024-07-26T10:37:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:25 smithi097 ceph-mon[32231]: pgmap v5686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:37:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9482.860997 secs 2024-07-26T10:37:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:27 smithi115 ceph-mon[35364]: pgmap v5687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:37:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:27 smithi181 ceph-mon[35869]: pgmap v5687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:37:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:27 smithi097 ceph-mon[32231]: pgmap v5687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:37:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:29 smithi115 ceph-mon[35364]: pgmap v5688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:37:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:29 smithi181 ceph-mon[35869]: pgmap v5688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:37:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:29 smithi097 ceph-mon[32231]: pgmap v5688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:37:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9487.861152 secs 2024-07-26T10:37:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9487.861152 secs 2024-07-26T10:37:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9487.861152 secs 2024-07-26T10:37:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:31 smithi115 ceph-mon[35364]: pgmap v5689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:31 smithi181 ceph-mon[35869]: pgmap v5689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:31 smithi097 ceph-mon[32231]: pgmap v5689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:37:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:33 smithi115 ceph-mon[35364]: pgmap v5690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:33 smithi181 ceph-mon[35869]: pgmap v5690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:33 smithi097 ceph-mon[32231]: pgmap v5690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:35 smithi115 ceph-mon[35364]: pgmap v5691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9492.861331 secs 2024-07-26T10:37:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:35 smithi181 ceph-mon[35869]: pgmap v5691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9492.861331 secs 2024-07-26T10:37:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:35 smithi097 ceph-mon[32231]: pgmap v5691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9492.861331 secs 2024-07-26T10:37:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:36.853+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:37:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:36.854+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:37:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:36.854+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:37:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:36.854+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:37:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:36.855+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:37 smithi115 ceph-mon[35364]: pgmap v5692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:37 smithi181 ceph-mon[35869]: pgmap v5692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:37 smithi097 ceph-mon[32231]: pgmap v5692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:39 smithi115 ceph-mon[35364]: pgmap v5693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:39 smithi181 ceph-mon[35869]: pgmap v5693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:39 smithi097 ceph-mon[32231]: pgmap v5693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9497.861495 secs 2024-07-26T10:37:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9497.861495 secs 2024-07-26T10:37:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9497.861495 secs 2024-07-26T10:37:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:41 smithi115 ceph-mon[35364]: pgmap v5694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:41 smithi181 ceph-mon[35869]: pgmap v5694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:41 smithi097 ceph-mon[32231]: pgmap v5694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:43 smithi115 ceph-mon[35364]: pgmap v5695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:43 smithi181 ceph-mon[35869]: pgmap v5695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:43 smithi097 ceph-mon[32231]: pgmap v5695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:45 smithi115 ceph-mon[35364]: pgmap v5696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9502.861647 secs 2024-07-26T10:37:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:45 smithi181 ceph-mon[35869]: pgmap v5696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9502.861647 secs 2024-07-26T10:37:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:45 smithi097 ceph-mon[32231]: pgmap v5696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9502.861647 secs 2024-07-26T10:37:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:47 smithi115 ceph-mon[35364]: pgmap v5697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:47 smithi181 ceph-mon[35869]: pgmap v5697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:47 smithi097 ceph-mon[32231]: pgmap v5697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:49 smithi115 ceph-mon[35364]: pgmap v5698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:49 smithi181 ceph-mon[35869]: pgmap v5698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:49 smithi097 ceph-mon[32231]: pgmap v5698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9507.861739 secs 2024-07-26T10:37:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9507.861739 secs 2024-07-26T10:37:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9507.861739 secs 2024-07-26T10:37:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:51.857+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:37:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:51.858+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:37:51.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:51.858+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:37:51.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:51.858+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:37:51.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:37:51.858+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:51 smithi097 ceph-mon[32231]: pgmap v5699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:51 smithi115 ceph-mon[35364]: pgmap v5699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:51 smithi181 ceph-mon[35869]: pgmap v5699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:53 smithi097 ceph-mon[32231]: pgmap v5700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:53 smithi115 ceph-mon[35364]: pgmap v5700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:54.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:53 smithi181 ceph-mon[35869]: pgmap v5700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:55 smithi097 ceph-mon[32231]: pgmap v5701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9512.861906 secs 2024-07-26T10:37:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:55 smithi115 ceph-mon[35364]: pgmap v5701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9512.861906 secs 2024-07-26T10:37:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:55 smithi181 ceph-mon[35869]: pgmap v5701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9512.861906 secs 2024-07-26T10:37:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:58 smithi115 ceph-mon[35364]: pgmap v5702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:58 smithi181 ceph-mon[35869]: pgmap v5702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:58 smithi097 ceph-mon[32231]: pgmap v5702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:37:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:37:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:37:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:37:59.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:59 smithi115 ceph-mon[35364]: pgmap v5703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:59.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:59 smithi115 ceph-mon[35364]: pgmap v5704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:37:59.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:37:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:37:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:37:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:37:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:37:59.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:37:59.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:59 smithi181 ceph-mon[35869]: pgmap v5703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:59.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:59 smithi181 ceph-mon[35869]: pgmap v5704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:37:59.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:37:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:38:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:38:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:38:00.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:38:00.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:38:00.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:59 smithi097 ceph-mon[32231]: pgmap v5703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:00.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:59 smithi097 ceph-mon[32231]: pgmap v5704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:00.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:37:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:38:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9517.862047 secs 2024-07-26T10:38:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9517.862047 secs 2024-07-26T10:38:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9517.862047 secs 2024-07-26T10:38:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:01 smithi115 ceph-mon[35364]: pgmap v5705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:01 smithi181 ceph-mon[35869]: pgmap v5705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:01 smithi097 ceph-mon[32231]: pgmap v5705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:03 smithi115 ceph-mon[35364]: pgmap v5706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:03 smithi181 ceph-mon[35869]: pgmap v5706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:03 smithi097 ceph-mon[32231]: pgmap v5706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:05 smithi115 ceph-mon[35364]: pgmap v5707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9522.862205 secs 2024-07-26T10:38:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:05 smithi181 ceph-mon[35869]: pgmap v5707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9522.862205 secs 2024-07-26T10:38:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:05 smithi097 ceph-mon[32231]: pgmap v5707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9522.862205 secs 2024-07-26T10:38:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:06.865+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:38:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:06.866+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:38:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:06.866+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:38:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:06.866+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:38:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:06.866+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:07 smithi115 ceph-mon[35364]: pgmap v5708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:07 smithi181 ceph-mon[35869]: pgmap v5708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:07 smithi097 ceph-mon[32231]: pgmap v5708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:09 smithi115 ceph-mon[35364]: pgmap v5709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:09 smithi181 ceph-mon[35869]: pgmap v5709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:09 smithi097 ceph-mon[32231]: pgmap v5709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9527.862361 secs 2024-07-26T10:38:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9527.862361 secs 2024-07-26T10:38:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9527.862361 secs 2024-07-26T10:38:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:11 smithi115 ceph-mon[35364]: pgmap v5710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:11 smithi181 ceph-mon[35869]: pgmap v5710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:11 smithi097 ceph-mon[32231]: pgmap v5710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:13 smithi115 ceph-mon[35364]: pgmap v5711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:13 smithi181 ceph-mon[35869]: pgmap v5711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:13 smithi097 ceph-mon[32231]: pgmap v5711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:15 smithi115 ceph-mon[35364]: pgmap v5712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9532.862517 secs 2024-07-26T10:38:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:15 smithi181 ceph-mon[35869]: pgmap v5712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9532.862517 secs 2024-07-26T10:38:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:15 smithi097 ceph-mon[32231]: pgmap v5712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9532.862517 secs 2024-07-26T10:38:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:17 smithi115 ceph-mon[35364]: pgmap v5713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:17 smithi181 ceph-mon[35869]: pgmap v5713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:17 smithi097 ceph-mon[32231]: pgmap v5713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:19 smithi115 ceph-mon[35364]: pgmap v5714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:19 smithi181 ceph-mon[35869]: pgmap v5714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:19 smithi097 ceph-mon[32231]: pgmap v5714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9537.862673 secs 2024-07-26T10:38:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9537.862673 secs 2024-07-26T10:38:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9537.862673 secs 2024-07-26T10:38:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:21.868+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:38:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:21.869+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:38:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:21.869+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:38:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:21.869+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:38:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:21.869+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:21 smithi115 ceph-mon[35364]: pgmap v5715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:21 smithi181 ceph-mon[35869]: pgmap v5715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:21 smithi097 ceph-mon[32231]: pgmap v5715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:23 smithi115 ceph-mon[35364]: pgmap v5716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:23 smithi181 ceph-mon[35869]: pgmap v5716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:23 smithi097 ceph-mon[32231]: pgmap v5716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:25 smithi181 ceph-mon[35869]: pgmap v5717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9542.862823 secs 2024-07-26T10:38:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:25 smithi097 ceph-mon[32231]: pgmap v5717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9542.862823 secs 2024-07-26T10:38:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:25 smithi115 ceph-mon[35364]: pgmap v5717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9542.862823 secs 2024-07-26T10:38:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:27 smithi097 ceph-mon[32231]: pgmap v5718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:38:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:27 smithi115 ceph-mon[35364]: pgmap v5718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:38:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:27 smithi181 ceph-mon[35869]: pgmap v5718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:38:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:29 smithi097 ceph-mon[32231]: pgmap v5719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:29 smithi115 ceph-mon[35364]: pgmap v5719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:29 smithi181 ceph-mon[35869]: pgmap v5719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9547.862980 secs 2024-07-26T10:38:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9547.862980 secs 2024-07-26T10:38:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9547.862980 secs 2024-07-26T10:38:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:31 smithi115 ceph-mon[35364]: pgmap v5720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:31 smithi181 ceph-mon[35869]: pgmap v5720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:31 smithi097 ceph-mon[32231]: pgmap v5720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:33 smithi115 ceph-mon[35364]: pgmap v5721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:38:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:33 smithi181 ceph-mon[35869]: pgmap v5721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:38:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:33 smithi097 ceph-mon[32231]: pgmap v5721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:38:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:35 smithi115 ceph-mon[35364]: pgmap v5722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9552.863158 secs 2024-07-26T10:38:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:35 smithi181 ceph-mon[35869]: pgmap v5722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9552.863158 secs 2024-07-26T10:38:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:35 smithi097 ceph-mon[32231]: pgmap v5722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:38:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9552.863158 secs 2024-07-26T10:38:36.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:36.876+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:38:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:36.877+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:38:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:36.877+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:38:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:36.877+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:38:36.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:36.877+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:37 smithi115 ceph-mon[35364]: pgmap v5723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:37 smithi181 ceph-mon[35869]: pgmap v5723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:37 smithi097 ceph-mon[32231]: pgmap v5723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:40.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:39 smithi181 ceph-mon[35869]: pgmap v5724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:40.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:39 smithi097 ceph-mon[32231]: pgmap v5724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:39 smithi115 ceph-mon[35364]: pgmap v5724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:41.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9557.863315 secs 2024-07-26T10:38:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9557.863315 secs 2024-07-26T10:38:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9557.863315 secs 2024-07-26T10:38:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:41 smithi115 ceph-mon[35364]: pgmap v5725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:41 smithi181 ceph-mon[35869]: pgmap v5725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:41 smithi097 ceph-mon[32231]: pgmap v5725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:43 smithi115 ceph-mon[35364]: pgmap v5726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:43 smithi181 ceph-mon[35869]: pgmap v5726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:43 smithi097 ceph-mon[32231]: pgmap v5726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:45 smithi115 ceph-mon[35364]: pgmap v5727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9562.863471 secs 2024-07-26T10:38:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:45 smithi181 ceph-mon[35869]: pgmap v5727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9562.863471 secs 2024-07-26T10:38:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:45 smithi097 ceph-mon[32231]: pgmap v5727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9562.863471 secs 2024-07-26T10:38:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:47 smithi115 ceph-mon[35364]: pgmap v5728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:38:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:47 smithi181 ceph-mon[35869]: pgmap v5728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:38:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:47 smithi097 ceph-mon[32231]: pgmap v5728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:38:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:49 smithi115 ceph-mon[35364]: pgmap v5729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:49 smithi181 ceph-mon[35869]: pgmap v5729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:49 smithi097 ceph-mon[32231]: pgmap v5729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9567.863629 secs 2024-07-26T10:38:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9567.863629 secs 2024-07-26T10:38:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9567.863629 secs 2024-07-26T10:38:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:51.879+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:38:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:51.880+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:38:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:51.880+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:38:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:51.880+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:38:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:38:51.880+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:51 smithi115 ceph-mon[35364]: pgmap v5730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:51 smithi181 ceph-mon[35869]: pgmap v5730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:51 smithi097 ceph-mon[32231]: pgmap v5730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:53 smithi115 ceph-mon[35364]: pgmap v5731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:53 smithi181 ceph-mon[35869]: pgmap v5731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:53 smithi097 ceph-mon[32231]: pgmap v5731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:55 smithi115 ceph-mon[35364]: pgmap v5732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9572.863793 secs 2024-07-26T10:38:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:55 smithi181 ceph-mon[35869]: pgmap v5732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9572.863793 secs 2024-07-26T10:38:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:55 smithi097 ceph-mon[32231]: pgmap v5732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9572.863793 secs 2024-07-26T10:38:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:57 smithi115 ceph-mon[35364]: pgmap v5733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:57 smithi181 ceph-mon[35869]: pgmap v5733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:57 smithi097 ceph-mon[32231]: pgmap v5733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:59 smithi115 ceph-mon[35364]: pgmap v5734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:38:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:39:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:59 smithi181 ceph-mon[35869]: pgmap v5734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:38:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:39:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:59 smithi097 ceph-mon[32231]: pgmap v5734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:38:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:39:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9577.863935 secs 2024-07-26T10:39:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9577.863935 secs 2024-07-26T10:39:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:01.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9577.863935 secs 2024-07-26T10:39:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:01.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:01.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:39:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:01 smithi115 ceph-mon[35364]: pgmap v5735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:01 smithi115 ceph-mon[35364]: pgmap v5736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:01 smithi181 ceph-mon[35869]: pgmap v5735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:01 smithi181 ceph-mon[35869]: pgmap v5736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:01 smithi097 ceph-mon[32231]: pgmap v5735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:01 smithi097 ceph-mon[32231]: pgmap v5736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:03 smithi115 ceph-mon[35364]: pgmap v5737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:03 smithi181 ceph-mon[35869]: pgmap v5737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:03 smithi097 ceph-mon[32231]: pgmap v5737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:05 smithi115 ceph-mon[35364]: pgmap v5738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:39:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9582.864085 secs 2024-07-26T10:39:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:05 smithi181 ceph-mon[35869]: pgmap v5738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:39:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9582.864085 secs 2024-07-26T10:39:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:05 smithi097 ceph-mon[32231]: pgmap v5738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:39:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9582.864085 secs 2024-07-26T10:39:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:06.884+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:39:06.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:06.885+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:39:06.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:06.885+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:39:06.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:06.885+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:39:06.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:06.886+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:07 smithi115 ceph-mon[35364]: pgmap v5739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:07 smithi181 ceph-mon[35869]: pgmap v5739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:07 smithi097 ceph-mon[32231]: pgmap v5739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:39:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:09 smithi115 ceph-mon[35364]: pgmap v5740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:09 smithi181 ceph-mon[35869]: pgmap v5740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:09 smithi097 ceph-mon[32231]: pgmap v5740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9587.864267 secs 2024-07-26T10:39:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9587.864267 secs 2024-07-26T10:39:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9587.864267 secs 2024-07-26T10:39:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:11 smithi115 ceph-mon[35364]: pgmap v5741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:11 smithi181 ceph-mon[35869]: pgmap v5741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:11 smithi097 ceph-mon[32231]: pgmap v5741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:13 smithi115 ceph-mon[35364]: pgmap v5742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:13 smithi181 ceph-mon[35869]: pgmap v5742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:13 smithi097 ceph-mon[32231]: pgmap v5742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:15 smithi115 ceph-mon[35364]: pgmap v5743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9592.864417 secs 2024-07-26T10:39:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:15 smithi181 ceph-mon[35869]: pgmap v5743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9592.864417 secs 2024-07-26T10:39:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:15 smithi097 ceph-mon[32231]: pgmap v5743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9592.864417 secs 2024-07-26T10:39:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:17 smithi115 ceph-mon[35364]: pgmap v5744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:17 smithi181 ceph-mon[35869]: pgmap v5744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:17 smithi097 ceph-mon[32231]: pgmap v5744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:19 smithi115 ceph-mon[35364]: pgmap v5745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:19 smithi181 ceph-mon[35869]: pgmap v5745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:19 smithi097 ceph-mon[32231]: pgmap v5745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9597.864524 secs 2024-07-26T10:39:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9597.864524 secs 2024-07-26T10:39:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9597.864524 secs 2024-07-26T10:39:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:21.887+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:39:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:21.889+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:39:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:21.889+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:39:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:21.889+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:39:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:21.889+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:21 smithi115 ceph-mon[35364]: pgmap v5746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:21 smithi181 ceph-mon[35869]: pgmap v5746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:21 smithi097 ceph-mon[32231]: pgmap v5746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:23 smithi115 ceph-mon[35364]: pgmap v5747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:23 smithi181 ceph-mon[35869]: pgmap v5747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:23 smithi097 ceph-mon[32231]: pgmap v5747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:25 smithi181 ceph-mon[35869]: pgmap v5748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9602.864700 secs 2024-07-26T10:39:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:25 smithi097 ceph-mon[32231]: pgmap v5748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9602.864700 secs 2024-07-26T10:39:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:25 smithi115 ceph-mon[35364]: pgmap v5748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9602.864700 secs 2024-07-26T10:39:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:27 smithi097 ceph-mon[32231]: pgmap v5749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:27 smithi115 ceph-mon[35364]: pgmap v5749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:27 smithi181 ceph-mon[35869]: pgmap v5749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:39:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:29 smithi097 ceph-mon[32231]: pgmap v5750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:29 smithi115 ceph-mon[35364]: pgmap v5750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:29 smithi181 ceph-mon[35869]: pgmap v5750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9607.864857 secs 2024-07-26T10:39:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9607.864857 secs 2024-07-26T10:39:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9607.864857 secs 2024-07-26T10:39:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:32 smithi115 ceph-mon[35364]: pgmap v5751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:32 smithi181 ceph-mon[35869]: pgmap v5751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:32 smithi097 ceph-mon[32231]: pgmap v5751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:34 smithi115 ceph-mon[35364]: pgmap v5752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:34 smithi181 ceph-mon[35869]: pgmap v5752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:34 smithi097 ceph-mon[32231]: pgmap v5752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:36 smithi115 ceph-mon[35364]: pgmap v5753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9612.865009 secs 2024-07-26T10:39:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:36 smithi181 ceph-mon[35869]: pgmap v5753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9612.865009 secs 2024-07-26T10:39:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:36 smithi097 ceph-mon[32231]: pgmap v5753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9612.865009 secs 2024-07-26T10:39:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:36.896+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:39:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:36.896+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:39:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:36.896+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:39:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:36.896+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:39:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:36.896+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:38 smithi115 ceph-mon[35364]: pgmap v5754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:38 smithi181 ceph-mon[35869]: pgmap v5754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:38 smithi097 ceph-mon[32231]: pgmap v5754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:40.288 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:40 smithi181 ceph-mon[35869]: pgmap v5755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:40.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:40 smithi097 ceph-mon[32231]: pgmap v5755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:40 smithi115 ceph-mon[35364]: pgmap v5755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9617.865179 secs 2024-07-26T10:39:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9617.865179 secs 2024-07-26T10:39:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9617.865179 secs 2024-07-26T10:39:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:42 smithi115 ceph-mon[35364]: pgmap v5756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:42 smithi181 ceph-mon[35869]: pgmap v5756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:42 smithi097 ceph-mon[32231]: pgmap v5756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:44 smithi115 ceph-mon[35364]: pgmap v5757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:44 smithi181 ceph-mon[35869]: pgmap v5757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:44 smithi097 ceph-mon[32231]: pgmap v5757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:46 smithi115 ceph-mon[35364]: pgmap v5758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9622.865367 secs 2024-07-26T10:39:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:46 smithi181 ceph-mon[35869]: pgmap v5758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9622.865367 secs 2024-07-26T10:39:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:46 smithi097 ceph-mon[32231]: pgmap v5758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9622.865367 secs 2024-07-26T10:39:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:48 smithi115 ceph-mon[35364]: pgmap v5759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:48 smithi181 ceph-mon[35869]: pgmap v5759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:48 smithi097 ceph-mon[32231]: pgmap v5759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:50 smithi115 ceph-mon[35364]: pgmap v5760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:50 smithi181 ceph-mon[35869]: pgmap v5760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:50 smithi097 ceph-mon[32231]: pgmap v5760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9627.865523 secs 2024-07-26T10:39:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9627.865523 secs 2024-07-26T10:39:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9627.865523 secs 2024-07-26T10:39:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:51.898+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:39:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:51.899+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:51.899+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:51.899+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:39:51.899+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:52 smithi115 ceph-mon[35364]: pgmap v5761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:52 smithi181 ceph-mon[35869]: pgmap v5761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:52 smithi097 ceph-mon[32231]: pgmap v5761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:54 smithi115 ceph-mon[35364]: pgmap v5762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:54 smithi181 ceph-mon[35869]: pgmap v5762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:54 smithi097 ceph-mon[32231]: pgmap v5762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:56 smithi115 ceph-mon[35364]: pgmap v5763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9632.865683 secs 2024-07-26T10:39:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:56 smithi181 ceph-mon[35869]: pgmap v5763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9632.865683 secs 2024-07-26T10:39:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:56 smithi097 ceph-mon[32231]: pgmap v5763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9632.865683 secs 2024-07-26T10:39:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:39:58 smithi115 ceph-mon[35364]: pgmap v5764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:39:58 smithi181 ceph-mon[35869]: pgmap v5764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:39:58 smithi097 ceph-mon[32231]: pgmap v5764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:00 smithi115 ceph-mon[35364]: pgmap v5765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:00 smithi181 ceph-mon[35869]: pgmap v5765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:40:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:00 smithi097 ceph-mon[32231]: pgmap v5765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:40:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9637.866710 secs 2024-07-26T10:40:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9637.866710 secs 2024-07-26T10:40:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9637.866710 secs 2024-07-26T10:40:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:02 smithi115 ceph-mon[35364]: pgmap v5766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:40:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:02 smithi181 ceph-mon[35869]: pgmap v5766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:40:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:02 smithi097 ceph-mon[32231]: pgmap v5766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:40:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:03 smithi115 ceph-mon[35364]: pgmap v5767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:40:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:03 smithi115 ceph-mon[35364]: pgmap v5768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:40:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:03 smithi181 ceph-mon[35869]: pgmap v5767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:40:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:03 smithi181 ceph-mon[35869]: pgmap v5768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:40:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:03 smithi097 ceph-mon[32231]: pgmap v5767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:40:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:03 smithi097 ceph-mon[32231]: pgmap v5768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:40:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:05 smithi115 ceph-mon[35364]: pgmap v5769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:05 smithi181 ceph-mon[35869]: pgmap v5769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:05 smithi097 ceph-mon[32231]: pgmap v5769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9642.866879 secs 2024-07-26T10:40:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9642.866879 secs 2024-07-26T10:40:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9642.866879 secs 2024-07-26T10:40:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:06.904+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:40:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:06.906+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:40:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:06.906+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:40:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:06.906+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:40:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:06.906+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:07 smithi115 ceph-mon[35364]: pgmap v5770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:07 smithi181 ceph-mon[35869]: pgmap v5770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:07 smithi097 ceph-mon[32231]: pgmap v5770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:08.588+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:40:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:08.588+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:40:08.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:08.589+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:40:08.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:08.589+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:40:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:09 smithi115 ceph-mon[35364]: pgmap v5771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:09.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:09 smithi181 ceph-mon[35869]: pgmap v5771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:09 smithi097 ceph-mon[32231]: pgmap v5771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9647.866962 secs 2024-07-26T10:40:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:11 smithi115 ceph-mon[35364]: pgmap v5772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9647.866962 secs 2024-07-26T10:40:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:11 smithi181 ceph-mon[35869]: pgmap v5772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9647.866962 secs 2024-07-26T10:40:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:11 smithi097 ceph-mon[32231]: pgmap v5772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:13 smithi115 ceph-mon[35364]: pgmap v5773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:40:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:13 smithi181 ceph-mon[35869]: pgmap v5773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:40:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:13 smithi097 ceph-mon[32231]: pgmap v5773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:40:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:15 smithi115 ceph-mon[35364]: pgmap v5774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:15 smithi181 ceph-mon[35869]: pgmap v5774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:15 smithi097 ceph-mon[32231]: pgmap v5774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9652.867112 secs 2024-07-26T10:40:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9652.867112 secs 2024-07-26T10:40:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9652.867112 secs 2024-07-26T10:40:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:17 smithi181 ceph-mon[35869]: pgmap v5775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:17 smithi097 ceph-mon[32231]: pgmap v5775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:17 smithi115 ceph-mon[35364]: pgmap v5775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:19 smithi097 ceph-mon[32231]: pgmap v5776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:19 smithi115 ceph-mon[35364]: pgmap v5776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:19 smithi181 ceph-mon[35869]: pgmap v5776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9657.867280 secs 2024-07-26T10:40:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:21 smithi097 ceph-mon[32231]: pgmap v5777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9657.867280 secs 2024-07-26T10:40:21.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:21 smithi115 ceph-mon[35364]: pgmap v5777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9657.867280 secs 2024-07-26T10:40:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:21 smithi181 ceph-mon[35869]: pgmap v5777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:21.908+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:40:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:21.909+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:40:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:21.909+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:40:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:21.909+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:40:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:21.909+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:23 smithi097 ceph-mon[32231]: pgmap v5778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:23 smithi115 ceph-mon[35364]: pgmap v5778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:23 smithi181 ceph-mon[35869]: pgmap v5778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:25 smithi115 ceph-mon[35364]: pgmap v5779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:25 smithi181 ceph-mon[35869]: pgmap v5779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:25 smithi097 ceph-mon[32231]: pgmap v5779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9662.867413 secs 2024-07-26T10:40:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9662.867413 secs 2024-07-26T10:40:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9662.867413 secs 2024-07-26T10:40:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:27 smithi115 ceph-mon[35364]: pgmap v5780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:27 smithi181 ceph-mon[35869]: pgmap v5780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:27 smithi097 ceph-mon[32231]: pgmap v5780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:29 smithi115 ceph-mon[35364]: pgmap v5781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:29 smithi181 ceph-mon[35869]: pgmap v5781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:29 smithi097 ceph-mon[32231]: pgmap v5781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9667.867574 secs 2024-07-26T10:40:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:31 smithi115 ceph-mon[35364]: pgmap v5782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9667.867574 secs 2024-07-26T10:40:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:31 smithi181 ceph-mon[35869]: pgmap v5782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9667.867574 secs 2024-07-26T10:40:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:31 smithi097 ceph-mon[32231]: pgmap v5782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:32 smithi115 ceph-mon[35364]: pgmap v5783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:32 smithi181 ceph-mon[35869]: pgmap v5783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:32 smithi097 ceph-mon[32231]: pgmap v5783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:35 smithi115 ceph-mon[35364]: pgmap v5784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:35 smithi181 ceph-mon[35869]: pgmap v5784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:35 smithi097 ceph-mon[32231]: pgmap v5784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9672.867734 secs 2024-07-26T10:40:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9672.867734 secs 2024-07-26T10:40:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9672.867734 secs 2024-07-26T10:40:36.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:36.915+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:40:36.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:36.917+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:40:36.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:36.917+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:40:36.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:36.917+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:40:36.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:36.917+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:37 smithi115 ceph-mon[35364]: pgmap v5785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:37 smithi181 ceph-mon[35869]: pgmap v5785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:37 smithi097 ceph-mon[32231]: pgmap v5785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:39 smithi115 ceph-mon[35364]: pgmap v5786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:39 smithi181 ceph-mon[35869]: pgmap v5786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:39 smithi097 ceph-mon[32231]: pgmap v5786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9677.867904 secs 2024-07-26T10:40:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:41 smithi115 ceph-mon[35364]: pgmap v5787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9677.867904 secs 2024-07-26T10:40:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:41 smithi181 ceph-mon[35869]: pgmap v5787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9677.867904 secs 2024-07-26T10:40:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:41 smithi097 ceph-mon[32231]: pgmap v5787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:43 smithi115 ceph-mon[35364]: pgmap v5788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:43 smithi181 ceph-mon[35869]: pgmap v5788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:43 smithi097 ceph-mon[32231]: pgmap v5788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:44 smithi115 ceph-mon[35364]: pgmap v5789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:44 smithi181 ceph-mon[35869]: pgmap v5789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:44 smithi097 ceph-mon[32231]: pgmap v5789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9682.868068 secs 2024-07-26T10:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9682.868068 secs 2024-07-26T10:40:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9682.868068 secs 2024-07-26T10:40:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:46 smithi115 ceph-mon[35364]: pgmap v5790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:46 smithi181 ceph-mon[35869]: pgmap v5790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:46 smithi097 ceph-mon[32231]: pgmap v5790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:49 smithi115 ceph-mon[35364]: pgmap v5791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:49 smithi181 ceph-mon[35869]: pgmap v5791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:49 smithi097 ceph-mon[32231]: pgmap v5791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9687.868277 secs 2024-07-26T10:40:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:51 smithi115 ceph-mon[35364]: pgmap v5792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9687.868277 secs 2024-07-26T10:40:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:51 smithi181 ceph-mon[35869]: pgmap v5792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9687.868277 secs 2024-07-26T10:40:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:51 smithi097 ceph-mon[32231]: pgmap v5792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:51.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:51.918+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:40:51.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:51.919+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:40:51.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:51.919+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:40:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:51.919+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:40:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:40:51.919+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:53 smithi097 ceph-mon[32231]: pgmap v5793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:53 smithi115 ceph-mon[35364]: pgmap v5793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:53 smithi181 ceph-mon[35869]: pgmap v5793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:55 smithi115 ceph-mon[35364]: pgmap v5794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:55 smithi181 ceph-mon[35869]: pgmap v5794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:55 smithi097 ceph-mon[32231]: pgmap v5794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:40:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9692.868433 secs 2024-07-26T10:40:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9692.868433 secs 2024-07-26T10:40:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9692.868433 secs 2024-07-26T10:40:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:57 smithi115 ceph-mon[35364]: pgmap v5795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:57 smithi181 ceph-mon[35869]: pgmap v5795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:57 smithi097 ceph-mon[32231]: pgmap v5795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:40:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:40:59 smithi115 ceph-mon[35364]: pgmap v5796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:40:59 smithi181 ceph-mon[35869]: pgmap v5796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:40:59 smithi097 ceph-mon[32231]: pgmap v5796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9697.868595 secs 2024-07-26T10:41:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:01 smithi181 ceph-mon[35869]: pgmap v5797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9697.868595 secs 2024-07-26T10:41:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:01 smithi097 ceph-mon[32231]: pgmap v5797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9697.868595 secs 2024-07-26T10:41:01.741 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:01 smithi115 ceph-mon[35364]: pgmap v5797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:03 smithi097 ceph-mon[32231]: pgmap v5798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:03 smithi115 ceph-mon[35364]: pgmap v5798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:03 smithi181 ceph-mon[35869]: pgmap v5798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:41:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:41:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:41:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:05 smithi115 ceph-mon[35364]: pgmap v5799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:41:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:05 smithi115 ceph-mon[35364]: pgmap v5800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:41:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:05 smithi181 ceph-mon[35869]: pgmap v5799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:41:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:05 smithi181 ceph-mon[35869]: pgmap v5800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:41:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:05 smithi097 ceph-mon[32231]: pgmap v5799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:41:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:05 smithi097 ceph-mon[32231]: pgmap v5800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:41:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9702.868757 secs 2024-07-26T10:41:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9702.868757 secs 2024-07-26T10:41:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9702.868757 secs 2024-07-26T10:41:06.928 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:06.926+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:41:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:06.927+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:41:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:06.927+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:41:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:06.927+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:41:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:06.927+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:07 smithi115 ceph-mon[35364]: pgmap v5801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:07 smithi181 ceph-mon[35869]: pgmap v5801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:07 smithi097 ceph-mon[32231]: pgmap v5801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:09 smithi115 ceph-mon[35364]: pgmap v5802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:09 smithi181 ceph-mon[35869]: pgmap v5802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:09 smithi097 ceph-mon[32231]: pgmap v5802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:11.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:11 smithi181 ceph-mon[35869]: pgmap v5803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:11.537 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9707.868904 secs 2024-07-26T10:41:11.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:11 smithi097 ceph-mon[32231]: pgmap v5803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:11.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9707.868904 secs 2024-07-26T10:41:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:11 smithi115 ceph-mon[35364]: pgmap v5803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9707.868904 secs 2024-07-26T10:41:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:13 smithi115 ceph-mon[35364]: pgmap v5804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:13 smithi181 ceph-mon[35869]: pgmap v5804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:13 smithi097 ceph-mon[32231]: pgmap v5804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:14 smithi115 ceph-mon[35364]: pgmap v5805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:41:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:14 smithi181 ceph-mon[35869]: pgmap v5805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:41:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:14 smithi097 ceph-mon[32231]: pgmap v5805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:41:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9712.869073 secs 2024-07-26T10:41:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9712.869073 secs 2024-07-26T10:41:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9712.869073 secs 2024-07-26T10:41:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:16 smithi115 ceph-mon[35364]: pgmap v5806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:16 smithi181 ceph-mon[35869]: pgmap v5806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:16 smithi097 ceph-mon[32231]: pgmap v5806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:18 smithi115 ceph-mon[35364]: pgmap v5807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:18 smithi181 ceph-mon[35869]: pgmap v5807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:18 smithi097 ceph-mon[32231]: pgmap v5807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:20 smithi115 ceph-mon[35364]: pgmap v5808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9717.869208 secs 2024-07-26T10:41:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:20 smithi181 ceph-mon[35869]: pgmap v5808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9717.869208 secs 2024-07-26T10:41:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:20 smithi097 ceph-mon[32231]: pgmap v5808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9717.869208 secs 2024-07-26T10:41:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:21.928+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:41:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:21.929+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:41:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:21.929+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:41:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:21.929+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:41:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:21.929+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:22 smithi115 ceph-mon[35364]: pgmap v5809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:22 smithi181 ceph-mon[35869]: pgmap v5809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:22 smithi097 ceph-mon[32231]: pgmap v5809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:24 smithi115 ceph-mon[35364]: pgmap v5810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:24 smithi181 ceph-mon[35869]: pgmap v5810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:24 smithi097 ceph-mon[32231]: pgmap v5810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9722.869347 secs 2024-07-26T10:41:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9722.869347 secs 2024-07-26T10:41:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9722.869347 secs 2024-07-26T10:41:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:26 smithi115 ceph-mon[35364]: pgmap v5811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:26 smithi181 ceph-mon[35869]: pgmap v5811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:26 smithi097 ceph-mon[32231]: pgmap v5811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:28.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:28 smithi115 ceph-mon[35364]: pgmap v5812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:28 smithi181 ceph-mon[35869]: pgmap v5812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:28 smithi097 ceph-mon[32231]: pgmap v5812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:30 smithi115 ceph-mon[35364]: pgmap v5813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9727.869495 secs 2024-07-26T10:41:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:30 smithi181 ceph-mon[35869]: pgmap v5813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9727.869495 secs 2024-07-26T10:41:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:30 smithi097 ceph-mon[32231]: pgmap v5813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9727.869495 secs 2024-07-26T10:41:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:32 smithi115 ceph-mon[35364]: pgmap v5814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:32 smithi181 ceph-mon[35869]: pgmap v5814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:32 smithi097 ceph-mon[32231]: pgmap v5814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:34 smithi115 ceph-mon[35364]: pgmap v5815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:34 smithi181 ceph-mon[35869]: pgmap v5815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:34 smithi097 ceph-mon[32231]: pgmap v5815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9732.869626 secs 2024-07-26T10:41:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9732.869626 secs 2024-07-26T10:41:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9732.869626 secs 2024-07-26T10:41:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:36.936+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:41:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:36.938+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:41:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:36.938+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:41:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:36.938+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:41:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:36.938+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:36 smithi115 ceph-mon[35364]: pgmap v5816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:36 smithi181 ceph-mon[35869]: pgmap v5816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:36 smithi097 ceph-mon[32231]: pgmap v5816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:38 smithi115 ceph-mon[35364]: pgmap v5817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:38 smithi181 ceph-mon[35869]: pgmap v5817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:38 smithi097 ceph-mon[32231]: pgmap v5817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:40 smithi115 ceph-mon[35364]: pgmap v5818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9737.869756 secs 2024-07-26T10:41:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:40 smithi181 ceph-mon[35869]: pgmap v5818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9737.869756 secs 2024-07-26T10:41:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:40 smithi097 ceph-mon[32231]: pgmap v5818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9737.869756 secs 2024-07-26T10:41:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:42 smithi181 ceph-mon[35869]: pgmap v5819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:42 smithi097 ceph-mon[32231]: pgmap v5819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:42 smithi115 ceph-mon[35364]: pgmap v5819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:44 smithi097 ceph-mon[32231]: pgmap v5820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:44 smithi115 ceph-mon[35364]: pgmap v5820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:44 smithi181 ceph-mon[35869]: pgmap v5820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9742.869848 secs 2024-07-26T10:41:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9742.869848 secs 2024-07-26T10:41:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9742.869848 secs 2024-07-26T10:41:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:46 smithi097 ceph-mon[32231]: pgmap v5821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:46 smithi115 ceph-mon[35364]: pgmap v5821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:46 smithi181 ceph-mon[35869]: pgmap v5821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:48 smithi115 ceph-mon[35364]: pgmap v5822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:48 smithi181 ceph-mon[35869]: pgmap v5822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:48 smithi097 ceph-mon[32231]: pgmap v5822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:50 smithi115 ceph-mon[35364]: pgmap v5823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9747.870020 secs 2024-07-26T10:41:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:50 smithi181 ceph-mon[35869]: pgmap v5823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9747.870020 secs 2024-07-26T10:41:51.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:50 smithi097 ceph-mon[32231]: pgmap v5823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:51.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9747.870020 secs 2024-07-26T10:41:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:51.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:41:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:51.941+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:41:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:51.941+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:41:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:51.941+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:41:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:41:51.941+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:52 smithi115 ceph-mon[35364]: pgmap v5824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:52 smithi181 ceph-mon[35869]: pgmap v5824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:52 smithi097 ceph-mon[32231]: pgmap v5824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:41:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:54 smithi115 ceph-mon[35364]: pgmap v5825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:54 smithi181 ceph-mon[35869]: pgmap v5825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:54 smithi097 ceph-mon[32231]: pgmap v5825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9752.870106 secs 2024-07-26T10:41:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9752.870106 secs 2024-07-26T10:41:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9752.870106 secs 2024-07-26T10:41:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:56 smithi115 ceph-mon[35364]: pgmap v5826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:56 smithi181 ceph-mon[35869]: pgmap v5826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:56 smithi097 ceph-mon[32231]: pgmap v5826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:41:58 smithi115 ceph-mon[35364]: pgmap v5827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:41:58 smithi181 ceph-mon[35869]: pgmap v5827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:41:58 smithi097 ceph-mon[32231]: pgmap v5827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:00 smithi115 ceph-mon[35364]: pgmap v5828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9757.870276 secs 2024-07-26T10:42:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:00 smithi181 ceph-mon[35869]: pgmap v5828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9757.870276 secs 2024-07-26T10:42:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:00 smithi097 ceph-mon[32231]: pgmap v5828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9757.870276 secs 2024-07-26T10:42:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:02 smithi115 ceph-mon[35364]: pgmap v5829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:02 smithi181 ceph-mon[35869]: pgmap v5829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:02 smithi097 ceph-mon[32231]: pgmap v5829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:04 smithi115 ceph-mon[35364]: pgmap v5830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:42:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:04 smithi181 ceph-mon[35869]: pgmap v5830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:42:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:04 smithi097 ceph-mon[32231]: pgmap v5830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:42:06.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9762.870382 secs 2024-07-26T10:42:06.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: pgmap v5831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: pgmap v5832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:06.228 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9762.870382 secs 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: pgmap v5831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: pgmap v5832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:42:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9762.870382 secs 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:42:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:42:06.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:42:06.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: pgmap v5831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:42:06.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: pgmap v5832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:06.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:42:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:06.944+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:42:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:06.945+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:42:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:06.945+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:42:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:06.946+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:42:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:06.946+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:08 smithi115 ceph-mon[35364]: pgmap v5833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:42:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:08 smithi181 ceph-mon[35869]: pgmap v5833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:42:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:08 smithi097 ceph-mon[32231]: pgmap v5833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:42:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:10 smithi115 ceph-mon[35364]: pgmap v5834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:10 smithi181 ceph-mon[35869]: pgmap v5834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:10 smithi097 ceph-mon[32231]: pgmap v5834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9767.870540 secs 2024-07-26T10:42:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9767.870540 secs 2024-07-26T10:42:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9767.870540 secs 2024-07-26T10:42:12.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:12 smithi097 ceph-mon[32231]: pgmap v5835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:12 smithi115 ceph-mon[35364]: pgmap v5835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:12 smithi181 ceph-mon[35869]: pgmap v5835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:14 smithi115 ceph-mon[35364]: pgmap v5836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:14 smithi181 ceph-mon[35869]: pgmap v5836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:14 smithi097 ceph-mon[32231]: pgmap v5836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9772.870680 secs 2024-07-26T10:42:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:16 smithi115 ceph-mon[35364]: pgmap v5837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:42:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9772.870680 secs 2024-07-26T10:42:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:16 smithi181 ceph-mon[35869]: pgmap v5837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:42:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9772.870680 secs 2024-07-26T10:42:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:16 smithi097 ceph-mon[32231]: pgmap v5837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:42:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:18 smithi115 ceph-mon[35364]: pgmap v5838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:18 smithi181 ceph-mon[35869]: pgmap v5838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:18 smithi097 ceph-mon[32231]: pgmap v5838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:20.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:20 smithi115 ceph-mon[35364]: pgmap v5839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:20 smithi181 ceph-mon[35869]: pgmap v5839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:20 smithi097 ceph-mon[32231]: pgmap v5839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9777.870839 secs 2024-07-26T10:42:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9777.870839 secs 2024-07-26T10:42:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9777.870839 secs 2024-07-26T10:42:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:21.948+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:42:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:21.949+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:42:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:21.949+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:42:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:21.949+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:42:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:21.949+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:22 smithi115 ceph-mon[35364]: pgmap v5840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:22 smithi181 ceph-mon[35869]: pgmap v5840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:22 smithi097 ceph-mon[32231]: pgmap v5840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:24 smithi115 ceph-mon[35364]: pgmap v5841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:24 smithi181 ceph-mon[35869]: pgmap v5841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:24 smithi097 ceph-mon[32231]: pgmap v5841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9782.870925 secs 2024-07-26T10:42:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:26 smithi115 ceph-mon[35364]: pgmap v5842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:42:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9782.870925 secs 2024-07-26T10:42:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:26 smithi181 ceph-mon[35869]: pgmap v5842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:42:26.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9782.870925 secs 2024-07-26T10:42:26.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:26 smithi097 ceph-mon[32231]: pgmap v5842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:42:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:28 smithi115 ceph-mon[35364]: pgmap v5843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:42:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:28 smithi181 ceph-mon[35869]: pgmap v5843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:42:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:28 smithi097 ceph-mon[32231]: pgmap v5843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:42:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:30 smithi115 ceph-mon[35364]: pgmap v5844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:30 smithi181 ceph-mon[35869]: pgmap v5844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:30.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:30 smithi097 ceph-mon[32231]: pgmap v5844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9787.871087 secs 2024-07-26T10:42:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9787.871087 secs 2024-07-26T10:42:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9787.871087 secs 2024-07-26T10:42:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:32 smithi097 ceph-mon[32231]: pgmap v5845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:32 smithi115 ceph-mon[35364]: pgmap v5845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:32 smithi181 ceph-mon[35869]: pgmap v5845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:34.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:34 smithi097 ceph-mon[32231]: pgmap v5846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:42:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:34 smithi115 ceph-mon[35364]: pgmap v5846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:42:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:34 smithi181 ceph-mon[35869]: pgmap v5846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:42:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9792.871300 secs 2024-07-26T10:42:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:36 smithi097 ceph-mon[32231]: pgmap v5847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:42:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9792.871300 secs 2024-07-26T10:42:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:36 smithi115 ceph-mon[35364]: pgmap v5847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:42:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9792.871300 secs 2024-07-26T10:42:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:36 smithi181 ceph-mon[35869]: pgmap v5847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:42:36.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:36.956+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:42:36.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:36.957+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:42:36.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:36.957+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:42:36.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:36.957+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:42:36.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:36.957+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:38 smithi115 ceph-mon[35364]: pgmap v5848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:42:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:38 smithi181 ceph-mon[35869]: pgmap v5848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:42:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:38 smithi097 ceph-mon[32231]: pgmap v5848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:42:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:40 smithi115 ceph-mon[35364]: pgmap v5849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:40 smithi181 ceph-mon[35869]: pgmap v5849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:40 smithi097 ceph-mon[32231]: pgmap v5849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9797.871396 secs 2024-07-26T10:42:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9797.871396 secs 2024-07-26T10:42:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9797.871396 secs 2024-07-26T10:42:42.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:42 smithi115 ceph-mon[35364]: pgmap v5850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:42.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:42 smithi181 ceph-mon[35869]: pgmap v5850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:42.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:42 smithi097 ceph-mon[32231]: pgmap v5850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:44 smithi115 ceph-mon[35364]: pgmap v5851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:44 smithi181 ceph-mon[35869]: pgmap v5851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:44 smithi097 ceph-mon[32231]: pgmap v5851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9802.871572 secs 2024-07-26T10:42:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:46 smithi115 ceph-mon[35364]: pgmap v5852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9802.871572 secs 2024-07-26T10:42:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:46 smithi181 ceph-mon[35869]: pgmap v5852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9802.871572 secs 2024-07-26T10:42:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:46 smithi097 ceph-mon[32231]: pgmap v5852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:48 smithi115 ceph-mon[35364]: pgmap v5853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:48 smithi181 ceph-mon[35869]: pgmap v5853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:48 smithi097 ceph-mon[32231]: pgmap v5853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:50 smithi115 ceph-mon[35364]: pgmap v5854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:50 smithi181 ceph-mon[35869]: pgmap v5854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:50 smithi097 ceph-mon[32231]: pgmap v5854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:42:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9807.871732 secs 2024-07-26T10:42:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9807.871732 secs 2024-07-26T10:42:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9807.871732 secs 2024-07-26T10:42:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:51.958+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:42:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:51.960+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:42:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:51.960+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:42:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:51.960+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:42:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:42:51.960+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:52 smithi115 ceph-mon[35364]: pgmap v5855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:52 smithi181 ceph-mon[35869]: pgmap v5855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:52 smithi097 ceph-mon[32231]: pgmap v5855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:42:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:53 smithi115 ceph-mon[35364]: pgmap v5856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:53 smithi181 ceph-mon[35869]: pgmap v5856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:53 smithi097 ceph-mon[32231]: pgmap v5856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9812.871888 secs 2024-07-26T10:42:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:56 smithi115 ceph-mon[35364]: pgmap v5857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9812.871888 secs 2024-07-26T10:42:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:56 smithi181 ceph-mon[35869]: pgmap v5857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9812.871888 secs 2024-07-26T10:42:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:56 smithi097 ceph-mon[32231]: pgmap v5857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:42:58 smithi115 ceph-mon[35364]: pgmap v5858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:42:58 smithi181 ceph-mon[35869]: pgmap v5858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:42:58 smithi097 ceph-mon[32231]: pgmap v5858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:00 smithi115 ceph-mon[35364]: pgmap v5859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:43:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:00 smithi181 ceph-mon[35869]: pgmap v5859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:43:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:00 smithi097 ceph-mon[32231]: pgmap v5859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:43:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9817.872046 secs 2024-07-26T10:43:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9817.872046 secs 2024-07-26T10:43:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9817.872046 secs 2024-07-26T10:43:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:02 smithi115 ceph-mon[35364]: pgmap v5860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:02 smithi181 ceph-mon[35869]: pgmap v5860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:02 smithi097 ceph-mon[32231]: pgmap v5860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:04 smithi115 ceph-mon[35364]: pgmap v5861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:04 smithi181 ceph-mon[35869]: pgmap v5861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:04 smithi097 ceph-mon[32231]: pgmap v5861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9822.872201 secs 2024-07-26T10:43:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:06 smithi115 ceph-mon[35364]: pgmap v5862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:43:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9822.872201 secs 2024-07-26T10:43:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:06 smithi181 ceph-mon[35869]: pgmap v5862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:43:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9822.872201 secs 2024-07-26T10:43:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:06 smithi097 ceph-mon[32231]: pgmap v5862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:43:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.908+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 msgr2=0x7ff3d40b5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000656) for more than 900000000 us, fault. 2024-07-26T10:43:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.908+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e0059040 comp rx=0 tx=0).stop 2024-07-26T10:43:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.908+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 msgr2=0x7ff3d40b5d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:43:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.908+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:43:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.908+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:43:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.909+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:43:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.911+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057c0 tx=0x7ff3e007a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:43:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.962+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:43:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.963+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:43:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.963+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:43:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.963+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:43:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:06.963+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:07.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:43:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:07.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:07.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:43:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:43:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:43:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:08 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:43:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:08 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:43:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:08 smithi115 ceph-mon[35364]: pgmap v5863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:43:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:08 smithi115 ceph-mon[35364]: pgmap v5864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:08 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:43:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:08 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:43:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:08 smithi181 ceph-mon[35869]: pgmap v5863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:43:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:08 smithi181 ceph-mon[35869]: pgmap v5864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:08 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:43:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:08 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:43:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:08 smithi097 ceph-mon[32231]: pgmap v5863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:43:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:08 smithi097 ceph-mon[32231]: pgmap v5864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:08.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:08.620+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:43:08.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:08.620+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:43:08.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:08.621+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T10:43:08.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:08.621+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:43:08.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:08.622+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T10:43:08.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:08.622+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:43:08.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:08.622+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:43:08.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:08.623+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00061e0 con 0x7ff3f01061a0 2024-07-26T10:43:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:10 smithi115 ceph-mon[35364]: pgmap v5865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:43:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:10 smithi181 ceph-mon[35869]: pgmap v5865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:43:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:10 smithi097 ceph-mon[32231]: pgmap v5865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:43:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9827.872363 secs 2024-07-26T10:43:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9827.872363 secs 2024-07-26T10:43:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9827.872363 secs 2024-07-26T10:43:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:12 smithi181 ceph-mon[35869]: pgmap v5866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:12 smithi097 ceph-mon[32231]: pgmap v5866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:12.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:12 smithi115 ceph-mon[35364]: pgmap v5866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:14.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:14 smithi097 ceph-mon[32231]: pgmap v5867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:14 smithi115 ceph-mon[35364]: pgmap v5867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:14 smithi181 ceph-mon[35869]: pgmap v5867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:16 smithi097 ceph-mon[32231]: pgmap v5868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9832.872525 secs 2024-07-26T10:43:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:16 smithi115 ceph-mon[35364]: pgmap v5868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9832.872525 secs 2024-07-26T10:43:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:16 smithi181 ceph-mon[35869]: pgmap v5868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9832.872525 secs 2024-07-26T10:43:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:18 smithi115 ceph-mon[35364]: pgmap v5869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:18 smithi181 ceph-mon[35869]: pgmap v5869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:18 smithi097 ceph-mon[32231]: pgmap v5869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:20 smithi115 ceph-mon[35364]: pgmap v5870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:20 smithi181 ceph-mon[35869]: pgmap v5870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:20 smithi097 ceph-mon[32231]: pgmap v5870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9837.872687 secs 2024-07-26T10:43:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9837.872687 secs 2024-07-26T10:43:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9837.872687 secs 2024-07-26T10:43:21.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:21.964+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:43:21.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:21.966+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:43:21.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:21.966+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:43:21.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:21.966+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:43:21.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:21.966+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:22 smithi115 ceph-mon[35364]: pgmap v5871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:22 smithi181 ceph-mon[35869]: pgmap v5871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:22 smithi097 ceph-mon[32231]: pgmap v5871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:24 smithi115 ceph-mon[35364]: pgmap v5872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:24 smithi181 ceph-mon[35869]: pgmap v5872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:24 smithi097 ceph-mon[32231]: pgmap v5872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:26 smithi115 ceph-mon[35364]: pgmap v5873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9842.872818 secs 2024-07-26T10:43:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:26 smithi181 ceph-mon[35869]: pgmap v5873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9842.872818 secs 2024-07-26T10:43:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:26 smithi097 ceph-mon[32231]: pgmap v5873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9842.872818 secs 2024-07-26T10:43:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:28 smithi115 ceph-mon[35364]: pgmap v5874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:28 smithi181 ceph-mon[35869]: pgmap v5874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:28 smithi097 ceph-mon[32231]: pgmap v5874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:30.382 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:30 smithi097 ceph-mon[32231]: pgmap v5875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:30 smithi115 ceph-mon[35364]: pgmap v5875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:30 smithi181 ceph-mon[35869]: pgmap v5875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9847.872955 secs 2024-07-26T10:43:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9847.872955 secs 2024-07-26T10:43:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9847.872955 secs 2024-07-26T10:43:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:32 smithi115 ceph-mon[35364]: pgmap v5876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:32 smithi181 ceph-mon[35869]: pgmap v5876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:32 smithi097 ceph-mon[32231]: pgmap v5876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:33 smithi115 ceph-mon[35364]: pgmap v5877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:33 smithi181 ceph-mon[35869]: pgmap v5877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:33 smithi097 ceph-mon[32231]: pgmap v5877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:35 smithi115 ceph-mon[35364]: pgmap v5878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9852.873119 secs 2024-07-26T10:43:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:35 smithi181 ceph-mon[35869]: pgmap v5878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9852.873119 secs 2024-07-26T10:43:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:35 smithi097 ceph-mon[32231]: pgmap v5878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9852.873119 secs 2024-07-26T10:43:36.974 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:36.972+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:43:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:36.973+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:43:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:36.973+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:43:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:36.973+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:43:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:36.973+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:37 smithi115 ceph-mon[35364]: pgmap v5879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:37.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:37 smithi181 ceph-mon[35869]: pgmap v5879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:37 smithi097 ceph-mon[32231]: pgmap v5879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:43:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:39 smithi115 ceph-mon[35364]: pgmap v5880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:39 smithi181 ceph-mon[35869]: pgmap v5880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:39 smithi097 ceph-mon[32231]: pgmap v5880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9857.873283 secs 2024-07-26T10:43:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9857.873283 secs 2024-07-26T10:43:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9857.873283 secs 2024-07-26T10:43:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:41 smithi115 ceph-mon[35364]: pgmap v5881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:41 smithi181 ceph-mon[35869]: pgmap v5881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:41 smithi097 ceph-mon[32231]: pgmap v5881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:43 smithi115 ceph-mon[35364]: pgmap v5882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:43 smithi181 ceph-mon[35869]: pgmap v5882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:43 smithi097 ceph-mon[32231]: pgmap v5882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:45 smithi115 ceph-mon[35364]: pgmap v5883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:45.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9862.873430 secs 2024-07-26T10:43:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:45 smithi181 ceph-mon[35869]: pgmap v5883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9862.873430 secs 2024-07-26T10:43:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:45 smithi097 ceph-mon[32231]: pgmap v5883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9862.873430 secs 2024-07-26T10:43:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:47 smithi115 ceph-mon[35364]: pgmap v5884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:47 smithi181 ceph-mon[35869]: pgmap v5884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:47 smithi097 ceph-mon[32231]: pgmap v5884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:49 smithi115 ceph-mon[35364]: pgmap v5885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:49 smithi181 ceph-mon[35869]: pgmap v5885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:49 smithi097 ceph-mon[32231]: pgmap v5885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9867.873571 secs 2024-07-26T10:43:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9867.873571 secs 2024-07-26T10:43:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9867.873571 secs 2024-07-26T10:43:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:51 smithi115 ceph-mon[35364]: pgmap v5886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:51.975+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:43:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:51.976+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:43:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:51.976+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:43:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:51.976+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:43:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:43:51.977+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:51 smithi181 ceph-mon[35869]: pgmap v5886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:51 smithi097 ceph-mon[32231]: pgmap v5886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:53 smithi115 ceph-mon[35364]: pgmap v5887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:53 smithi181 ceph-mon[35869]: pgmap v5887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:53 smithi097 ceph-mon[32231]: pgmap v5887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:55 smithi115 ceph-mon[35364]: pgmap v5888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9872.873728 secs 2024-07-26T10:43:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:55 smithi181 ceph-mon[35869]: pgmap v5888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9872.873728 secs 2024-07-26T10:43:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:55 smithi097 ceph-mon[32231]: pgmap v5888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9872.873728 secs 2024-07-26T10:43:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:57 smithi115 ceph-mon[35364]: pgmap v5889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:57 smithi181 ceph-mon[35869]: pgmap v5889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:57 smithi097 ceph-mon[32231]: pgmap v5889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:43:59 smithi115 ceph-mon[35364]: pgmap v5890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:43:59 smithi181 ceph-mon[35869]: pgmap v5890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:43:59 smithi097 ceph-mon[32231]: pgmap v5890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9877.873847 secs 2024-07-26T10:44:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9877.873847 secs 2024-07-26T10:44:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9877.873847 secs 2024-07-26T10:44:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:01 smithi181 ceph-mon[35869]: pgmap v5891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:01 smithi097 ceph-mon[32231]: pgmap v5891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:01 smithi115 ceph-mon[35364]: pgmap v5891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:03 smithi181 ceph-mon[35869]: pgmap v5892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:03 smithi097 ceph-mon[32231]: pgmap v5892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:03 smithi115 ceph-mon[35364]: pgmap v5892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:05 smithi097 ceph-mon[32231]: pgmap v5893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9882.874009 secs 2024-07-26T10:44:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:05 smithi115 ceph-mon[35364]: pgmap v5893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9882.874009 secs 2024-07-26T10:44:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:05 smithi181 ceph-mon[35869]: pgmap v5893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9882.874009 secs 2024-07-26T10:44:06.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:06.982+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:44:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:06.983+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:44:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:06.983+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:44:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:06.983+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:44:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:06.983+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:07 smithi097 ceph-mon[32231]: pgmap v5894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:07 smithi115 ceph-mon[35364]: pgmap v5894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:07 smithi181 ceph-mon[35869]: pgmap v5894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:44:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:09 smithi115 ceph-mon[35364]: pgmap v5895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:44:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:09 smithi115 ceph-mon[35364]: pgmap v5896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:44:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:09 smithi181 ceph-mon[35869]: pgmap v5895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:44:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:09 smithi181 ceph-mon[35869]: pgmap v5896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:44:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:09 smithi097 ceph-mon[32231]: pgmap v5895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:44:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:09 smithi097 ceph-mon[32231]: pgmap v5896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:44:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9887.874150 secs 2024-07-26T10:44:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9887.874150 secs 2024-07-26T10:44:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9887.874150 secs 2024-07-26T10:44:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:11 smithi115 ceph-mon[35364]: pgmap v5897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:11 smithi181 ceph-mon[35869]: pgmap v5897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:11 smithi097 ceph-mon[32231]: pgmap v5897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:14.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:13 smithi181 ceph-mon[35869]: pgmap v5898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:14.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:13 smithi097 ceph-mon[32231]: pgmap v5898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:13 smithi115 ceph-mon[35364]: pgmap v5898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:15 smithi115 ceph-mon[35364]: pgmap v5899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9892.874308 secs 2024-07-26T10:44:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:15 smithi181 ceph-mon[35869]: pgmap v5899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:16.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9892.874308 secs 2024-07-26T10:44:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:15 smithi097 ceph-mon[32231]: pgmap v5899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9892.874308 secs 2024-07-26T10:44:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:17 smithi115 ceph-mon[35364]: pgmap v5900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:17 smithi181 ceph-mon[35869]: pgmap v5900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:18.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:17 smithi097 ceph-mon[32231]: pgmap v5900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:44:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:19 smithi115 ceph-mon[35364]: pgmap v5901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:44:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:19 smithi181 ceph-mon[35869]: pgmap v5901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:44:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:19 smithi097 ceph-mon[32231]: pgmap v5901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:44:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9897.874455 secs 2024-07-26T10:44:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9897.874455 secs 2024-07-26T10:44:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9897.874455 secs 2024-07-26T10:44:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:21.985+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:44:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:21.986+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:44:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:21.986+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:44:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:21.986+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:44:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:21.986+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:21 smithi115 ceph-mon[35364]: pgmap v5902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:21 smithi181 ceph-mon[35869]: pgmap v5902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:21 smithi097 ceph-mon[32231]: pgmap v5902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:23 smithi115 ceph-mon[35364]: pgmap v5903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:23 smithi181 ceph-mon[35869]: pgmap v5903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:23 smithi097 ceph-mon[32231]: pgmap v5903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:25 smithi115 ceph-mon[35364]: pgmap v5904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9902.874583 secs 2024-07-26T10:44:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:25 smithi181 ceph-mon[35869]: pgmap v5904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9902.874583 secs 2024-07-26T10:44:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:25 smithi097 ceph-mon[32231]: pgmap v5904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9902.874583 secs 2024-07-26T10:44:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:27 smithi115 ceph-mon[35364]: pgmap v5905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:27 smithi181 ceph-mon[35869]: pgmap v5905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:27 smithi097 ceph-mon[32231]: pgmap v5905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:29 smithi115 ceph-mon[35364]: pgmap v5906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:29 smithi181 ceph-mon[35869]: pgmap v5906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:29 smithi097 ceph-mon[32231]: pgmap v5906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9907.874742 secs 2024-07-26T10:44:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9907.874742 secs 2024-07-26T10:44:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9907.874742 secs 2024-07-26T10:44:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:31 smithi115 ceph-mon[35364]: pgmap v5907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:31 smithi181 ceph-mon[35869]: pgmap v5907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:31 smithi097 ceph-mon[32231]: pgmap v5907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:33 smithi115 ceph-mon[35364]: pgmap v5908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:33 smithi181 ceph-mon[35869]: pgmap v5908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:33 smithi097 ceph-mon[32231]: pgmap v5908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:35 smithi115 ceph-mon[35364]: pgmap v5909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9912.874897 secs 2024-07-26T10:44:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:35 smithi181 ceph-mon[35869]: pgmap v5909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9912.874897 secs 2024-07-26T10:44:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:35 smithi097 ceph-mon[32231]: pgmap v5909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9912.874897 secs 2024-07-26T10:44:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:36.991+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:44:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:36.992+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:44:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:36.992+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:44:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:36.992+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:44:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:36.993+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:37 smithi115 ceph-mon[35364]: pgmap v5910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:37 smithi181 ceph-mon[35869]: pgmap v5910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:37 smithi097 ceph-mon[32231]: pgmap v5910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:39 smithi115 ceph-mon[35364]: pgmap v5911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:39 smithi181 ceph-mon[35869]: pgmap v5911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:39 smithi097 ceph-mon[32231]: pgmap v5911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9917.875027 secs 2024-07-26T10:44:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9917.875027 secs 2024-07-26T10:44:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9917.875027 secs 2024-07-26T10:44:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:41 smithi115 ceph-mon[35364]: pgmap v5912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:41 smithi181 ceph-mon[35869]: pgmap v5912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:41 smithi097 ceph-mon[32231]: pgmap v5912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:43 smithi115 ceph-mon[35364]: pgmap v5913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:43 smithi181 ceph-mon[35869]: pgmap v5913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:43 smithi097 ceph-mon[32231]: pgmap v5913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:45 smithi115 ceph-mon[35364]: pgmap v5914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9922.875180 secs 2024-07-26T10:44:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:45 smithi181 ceph-mon[35869]: pgmap v5914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9922.875180 secs 2024-07-26T10:44:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:45 smithi097 ceph-mon[32231]: pgmap v5914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9922.875180 secs 2024-07-26T10:44:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:47 smithi115 ceph-mon[35364]: pgmap v5915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:47 smithi181 ceph-mon[35869]: pgmap v5915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:47 smithi097 ceph-mon[32231]: pgmap v5915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:49 smithi115 ceph-mon[35364]: pgmap v5916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:49 smithi181 ceph-mon[35869]: pgmap v5916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:49 smithi097 ceph-mon[32231]: pgmap v5916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9927.875340 secs 2024-07-26T10:44:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9927.875340 secs 2024-07-26T10:44:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9927.875340 secs 2024-07-26T10:44:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:51.994+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:44:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:51.996+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:44:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:51.996+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:44:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:51.996+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:44:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:44:51.996+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:51 smithi115 ceph-mon[35364]: pgmap v5917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:51 smithi181 ceph-mon[35869]: pgmap v5917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:51 smithi097 ceph-mon[32231]: pgmap v5917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:53 smithi115 ceph-mon[35364]: pgmap v5918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:53 smithi181 ceph-mon[35869]: pgmap v5918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:53 smithi097 ceph-mon[32231]: pgmap v5918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:55 smithi115 ceph-mon[35364]: pgmap v5919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9932.875488 secs 2024-07-26T10:44:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:55 smithi181 ceph-mon[35869]: pgmap v5919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9932.875488 secs 2024-07-26T10:44:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:55 smithi097 ceph-mon[32231]: pgmap v5919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9932.875488 secs 2024-07-26T10:44:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:57 smithi115 ceph-mon[35364]: pgmap v5920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:57 smithi181 ceph-mon[35869]: pgmap v5920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:57 smithi097 ceph-mon[32231]: pgmap v5920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:44:59 smithi181 ceph-mon[35869]: pgmap v5921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:44:59 smithi097 ceph-mon[32231]: pgmap v5921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:44:59 smithi115 ceph-mon[35364]: pgmap v5921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9937.875644 secs 2024-07-26T10:45:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9937.875644 secs 2024-07-26T10:45:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9937.875644 secs 2024-07-26T10:45:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:01 smithi097 ceph-mon[32231]: pgmap v5922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:01 smithi115 ceph-mon[35364]: pgmap v5922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:01 smithi181 ceph-mon[35869]: pgmap v5922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:04 smithi115 ceph-mon[35364]: pgmap v5923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:04 smithi181 ceph-mon[35869]: pgmap v5923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:04 smithi097 ceph-mon[32231]: pgmap v5923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:06 smithi115 ceph-mon[35364]: pgmap v5924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9942.875804 secs 2024-07-26T10:45:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:06 smithi181 ceph-mon[35869]: pgmap v5924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9942.875804 secs 2024-07-26T10:45:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:06 smithi097 ceph-mon[32231]: pgmap v5924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9942.875804 secs 2024-07-26T10:45:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:06.999+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:45:07.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:07.000+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:45:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:07.000+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:45:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:07.000+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:45:07.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:07.000+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:08 smithi115 ceph-mon[35364]: pgmap v5925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:08 smithi181 ceph-mon[35869]: pgmap v5925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:08 smithi097 ceph-mon[32231]: pgmap v5925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:10 smithi115 ceph-mon[35364]: pgmap v5926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:10 smithi181 ceph-mon[35869]: pgmap v5926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:10 smithi097 ceph-mon[32231]: pgmap v5926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9947.875938 secs 2024-07-26T10:45:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9947.875938 secs 2024-07-26T10:45:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9947.875938 secs 2024-07-26T10:45:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:45:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:12 smithi115 ceph-mon[35364]: pgmap v5927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:12 smithi115 ceph-mon[35364]: pgmap v5928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:12 smithi181 ceph-mon[35869]: pgmap v5927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:12 smithi181 ceph-mon[35869]: pgmap v5928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:12 smithi097 ceph-mon[32231]: pgmap v5927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:12 smithi097 ceph-mon[32231]: pgmap v5928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:14.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:14 smithi097 ceph-mon[32231]: pgmap v5929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:14 smithi115 ceph-mon[35364]: pgmap v5929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:14 smithi181 ceph-mon[35869]: pgmap v5929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:16 smithi115 ceph-mon[35364]: pgmap v5930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9952.876092 secs 2024-07-26T10:45:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:16 smithi181 ceph-mon[35869]: pgmap v5930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9952.876092 secs 2024-07-26T10:45:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:16 smithi097 ceph-mon[32231]: pgmap v5930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9952.876092 secs 2024-07-26T10:45:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:18 smithi115 ceph-mon[35364]: pgmap v5931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:18 smithi181 ceph-mon[35869]: pgmap v5931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:18 smithi097 ceph-mon[32231]: pgmap v5931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:20 smithi115 ceph-mon[35364]: pgmap v5932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:20 smithi181 ceph-mon[35869]: pgmap v5932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:20 smithi097 ceph-mon[32231]: pgmap v5932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:45:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9957.876286 secs 2024-07-26T10:45:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9957.876286 secs 2024-07-26T10:45:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9957.876286 secs 2024-07-26T10:45:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:22.002+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:45:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:22.004+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:45:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:22.004+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:45:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:22.004+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:45:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:22.004+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:22 smithi115 ceph-mon[35364]: pgmap v5933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:45:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:22 smithi181 ceph-mon[35869]: pgmap v5933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:45:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:22 smithi097 ceph-mon[32231]: pgmap v5933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:45:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:24 smithi115 ceph-mon[35364]: pgmap v5934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:24 smithi181 ceph-mon[35869]: pgmap v5934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:24.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:24 smithi097 ceph-mon[32231]: pgmap v5934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:26 smithi115 ceph-mon[35364]: pgmap v5935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9962.876440 secs 2024-07-26T10:45:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:26 smithi181 ceph-mon[35869]: pgmap v5935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9962.876440 secs 2024-07-26T10:45:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:26 smithi097 ceph-mon[32231]: pgmap v5935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9962.876440 secs 2024-07-26T10:45:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:28 smithi115 ceph-mon[35364]: pgmap v5936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:28 smithi181 ceph-mon[35869]: pgmap v5936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:28 smithi097 ceph-mon[32231]: pgmap v5936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:30 smithi115 ceph-mon[35364]: pgmap v5937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:30 smithi181 ceph-mon[35869]: pgmap v5937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:30 smithi097 ceph-mon[32231]: pgmap v5937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9967.876602 secs 2024-07-26T10:45:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9967.876602 secs 2024-07-26T10:45:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9967.876602 secs 2024-07-26T10:45:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:32 smithi115 ceph-mon[35364]: pgmap v5938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:32 smithi181 ceph-mon[35869]: pgmap v5938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:32 smithi097 ceph-mon[32231]: pgmap v5938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:34 smithi115 ceph-mon[35364]: pgmap v5939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:34 smithi181 ceph-mon[35869]: pgmap v5939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:34 smithi097 ceph-mon[32231]: pgmap v5939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:36 smithi115 ceph-mon[35364]: pgmap v5940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9972.876761 secs 2024-07-26T10:45:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:36 smithi181 ceph-mon[35869]: pgmap v5940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9972.876761 secs 2024-07-26T10:45:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:36 smithi097 ceph-mon[32231]: pgmap v5940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9972.876761 secs 2024-07-26T10:45:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:37.009+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:45:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:37.011+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:45:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:37.011+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:45:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:37.011+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:45:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:37.011+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:38 smithi115 ceph-mon[35364]: pgmap v5941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:38 smithi181 ceph-mon[35869]: pgmap v5941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:38 smithi097 ceph-mon[32231]: pgmap v5941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:40 smithi115 ceph-mon[35364]: pgmap v5942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:40 smithi181 ceph-mon[35869]: pgmap v5942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:40 smithi097 ceph-mon[32231]: pgmap v5942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9977.876854 secs 2024-07-26T10:45:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9977.876854 secs 2024-07-26T10:45:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9977.876854 secs 2024-07-26T10:45:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:42 smithi115 ceph-mon[35364]: pgmap v5943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:42 smithi181 ceph-mon[35869]: pgmap v5943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:42 smithi097 ceph-mon[32231]: pgmap v5943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:44 smithi115 ceph-mon[35364]: pgmap v5944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:44 smithi181 ceph-mon[35869]: pgmap v5944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:44 smithi097 ceph-mon[32231]: pgmap v5944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:46 smithi115 ceph-mon[35364]: pgmap v5945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9982.877009 secs 2024-07-26T10:45:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:46 smithi181 ceph-mon[35869]: pgmap v5945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9982.877009 secs 2024-07-26T10:45:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:46 smithi097 ceph-mon[32231]: pgmap v5945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9982.877009 secs 2024-07-26T10:45:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:48 smithi115 ceph-mon[35364]: pgmap v5946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:45:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:48 smithi181 ceph-mon[35869]: pgmap v5946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:45:48.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:48 smithi097 ceph-mon[32231]: pgmap v5946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:45:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:50 smithi115 ceph-mon[35364]: pgmap v5947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:50 smithi181 ceph-mon[35869]: pgmap v5947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:50 smithi097 ceph-mon[32231]: pgmap v5947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9987.877163 secs 2024-07-26T10:45:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9987.877163 secs 2024-07-26T10:45:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9987.877163 secs 2024-07-26T10:45:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:52.013+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:45:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:52.014+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:45:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:52.014+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:45:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:52.014+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:45:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:45:52.014+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:52 smithi115 ceph-mon[35364]: pgmap v5948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:45:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:52 smithi181 ceph-mon[35869]: pgmap v5948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:45:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:52 smithi097 ceph-mon[32231]: pgmap v5948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:45:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:54 smithi181 ceph-mon[35869]: pgmap v5949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:45:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:54 smithi097 ceph-mon[32231]: pgmap v5949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:45:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:54 smithi115 ceph-mon[35364]: pgmap v5949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:45:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:56 smithi097 ceph-mon[32231]: pgmap v5950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9992.877329 secs 2024-07-26T10:45:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:56 smithi115 ceph-mon[35364]: pgmap v5950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9992.877329 secs 2024-07-26T10:45:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:56 smithi181 ceph-mon[35869]: pgmap v5950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:45:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9992.877329 secs 2024-07-26T10:45:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:45:58 smithi097 ceph-mon[32231]: pgmap v5951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:45:58 smithi115 ceph-mon[35364]: pgmap v5951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:45:58 smithi181 ceph-mon[35869]: pgmap v5951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:00 smithi097 ceph-mon[32231]: pgmap v5952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:00 smithi115 ceph-mon[35364]: pgmap v5952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:00 smithi181 ceph-mon[35869]: pgmap v5952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 9997.877429 secs 2024-07-26T10:46:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 9997.877429 secs 2024-07-26T10:46:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 9997.877429 secs 2024-07-26T10:46:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:02 smithi115 ceph-mon[35364]: pgmap v5953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:02 smithi181 ceph-mon[35869]: pgmap v5953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:02 smithi097 ceph-mon[32231]: pgmap v5953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:04 smithi115 ceph-mon[35364]: pgmap v5954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:04 smithi181 ceph-mon[35869]: pgmap v5954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:04 smithi097 ceph-mon[32231]: pgmap v5954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:06 smithi115 ceph-mon[35364]: pgmap v5955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10002.877587 secs 2024-07-26T10:46:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:06 smithi181 ceph-mon[35869]: pgmap v5955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10002.877587 secs 2024-07-26T10:46:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:06 smithi097 ceph-mon[32231]: pgmap v5955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10002.877587 secs 2024-07-26T10:46:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:07.020+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:46:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:07.022+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:46:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:07.022+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:46:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:07.022+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:46:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:07.022+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:08.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:08.651+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:46:08.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:08.651+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:46:08.654 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:08.652+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:46:08.654 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:08.652+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:46:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:08 smithi115 ceph-mon[35364]: pgmap v5956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:08 smithi181 ceph-mon[35869]: pgmap v5956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:08 smithi097 ceph-mon[32231]: pgmap v5956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:10 smithi115 ceph-mon[35364]: pgmap v5957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:10 smithi181 ceph-mon[35869]: pgmap v5957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:10 smithi097 ceph-mon[32231]: pgmap v5957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10007.877745 secs 2024-07-26T10:46:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10007.877745 secs 2024-07-26T10:46:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10007.877745 secs 2024-07-26T10:46:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:12 smithi115 ceph-mon[35364]: pgmap v5958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:12 smithi181 ceph-mon[35869]: pgmap v5958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:12 smithi097 ceph-mon[32231]: pgmap v5958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:14 smithi115 ceph-mon[35364]: pgmap v5959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:14 smithi181 ceph-mon[35869]: pgmap v5959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:14 smithi097 ceph-mon[32231]: pgmap v5959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:16 smithi097 ceph-mon[32231]: pgmap v5960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10012.877901 secs 2024-07-26T10:46:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:16 smithi115 ceph-mon[35364]: pgmap v5960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10012.877901 secs 2024-07-26T10:46:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:16 smithi181 ceph-mon[35869]: pgmap v5960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10012.877901 secs 2024-07-26T10:46:18.676 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:18 smithi097 ceph-mon[32231]: pgmap v5961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:18.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:18 smithi115 ceph-mon[35364]: pgmap v5961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:18 smithi181 ceph-mon[35869]: pgmap v5961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:20 smithi115 ceph-mon[35364]: pgmap v5962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:20.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:20 smithi181 ceph-mon[35869]: pgmap v5962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:20.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:20.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:20 smithi097 ceph-mon[32231]: pgmap v5962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:46:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10017.878056 secs 2024-07-26T10:46:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:21 smithi115 ceph-mon[35364]: pgmap v5963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:21 smithi115 ceph-mon[35364]: pgmap v5964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10017.878056 secs 2024-07-26T10:46:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:21 smithi181 ceph-mon[35869]: pgmap v5963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:21 smithi181 ceph-mon[35869]: pgmap v5964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10017.878056 secs 2024-07-26T10:46:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:21 smithi097 ceph-mon[32231]: pgmap v5963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:21 smithi097 ceph-mon[32231]: pgmap v5964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:22.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:22.022+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:46:22.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:22.024+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:46:22.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:22.024+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:46:22.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:22.024+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:46:22.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:22.024+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:23 smithi115 ceph-mon[35364]: pgmap v5965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T10:46:23.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:23 smithi181 ceph-mon[35869]: pgmap v5965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T10:46:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:23 smithi097 ceph-mon[32231]: pgmap v5965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T10:46:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:25 smithi115 ceph-mon[35364]: pgmap v5966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:46:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:25 smithi181 ceph-mon[35869]: pgmap v5966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:46:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:25 smithi097 ceph-mon[32231]: pgmap v5966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:46:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10022.878139 secs 2024-07-26T10:46:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10022.878139 secs 2024-07-26T10:46:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10022.878139 secs 2024-07-26T10:46:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:27 smithi115 ceph-mon[35364]: pgmap v5967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:27 smithi181 ceph-mon[35869]: pgmap v5967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:27 smithi097 ceph-mon[32231]: pgmap v5967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:28 smithi115 ceph-mon[35364]: pgmap v5968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:28 smithi181 ceph-mon[35869]: pgmap v5968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:28 smithi097 ceph-mon[32231]: pgmap v5968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10027.878270 secs 2024-07-26T10:46:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:31 smithi115 ceph-mon[35364]: pgmap v5969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:46:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10027.878270 secs 2024-07-26T10:46:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:31 smithi181 ceph-mon[35869]: pgmap v5969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:46:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10027.878270 secs 2024-07-26T10:46:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:31 smithi097 ceph-mon[32231]: pgmap v5969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:46:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:33 smithi115 ceph-mon[35364]: pgmap v5970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:33 smithi181 ceph-mon[35869]: pgmap v5970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:33 smithi097 ceph-mon[32231]: pgmap v5970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:35 smithi115 ceph-mon[35364]: pgmap v5971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:35.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:35 smithi181 ceph-mon[35869]: pgmap v5971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:35 smithi097 ceph-mon[32231]: pgmap v5971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10032.878402 secs 2024-07-26T10:46:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10032.878402 secs 2024-07-26T10:46:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10032.878402 secs 2024-07-26T10:46:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:37.030+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:46:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:37.031+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:46:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:37.031+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:46:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:37.031+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:46:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:37.031+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:37 smithi115 ceph-mon[35364]: pgmap v5972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:37 smithi181 ceph-mon[35869]: pgmap v5972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:37 smithi097 ceph-mon[32231]: pgmap v5972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:39 smithi115 ceph-mon[35364]: pgmap v5973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:39 smithi181 ceph-mon[35869]: pgmap v5973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:39 smithi097 ceph-mon[32231]: pgmap v5973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10037.878536 secs 2024-07-26T10:46:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:41 smithi115 ceph-mon[35364]: pgmap v5974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10037.878536 secs 2024-07-26T10:46:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:41 smithi181 ceph-mon[35869]: pgmap v5974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10037.878536 secs 2024-07-26T10:46:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:41 smithi097 ceph-mon[32231]: pgmap v5974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:43 smithi115 ceph-mon[35364]: pgmap v5975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:43 smithi181 ceph-mon[35869]: pgmap v5975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:43 smithi097 ceph-mon[32231]: pgmap v5975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:45 smithi181 ceph-mon[35869]: pgmap v5976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:45 smithi097 ceph-mon[32231]: pgmap v5976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:45 smithi115 ceph-mon[35364]: pgmap v5976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10042.878697 secs 2024-07-26T10:46:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10042.878697 secs 2024-07-26T10:46:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10042.878697 secs 2024-07-26T10:46:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:47 smithi097 ceph-mon[32231]: pgmap v5977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:47 smithi115 ceph-mon[35364]: pgmap v5977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:47 smithi181 ceph-mon[35869]: pgmap v5977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:49 smithi097 ceph-mon[32231]: pgmap v5978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:49 smithi115 ceph-mon[35364]: pgmap v5978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:49 smithi181 ceph-mon[35869]: pgmap v5978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10047.878854 secs 2024-07-26T10:46:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:51 smithi115 ceph-mon[35364]: pgmap v5979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10047.878854 secs 2024-07-26T10:46:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:51 smithi181 ceph-mon[35869]: pgmap v5979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10047.878854 secs 2024-07-26T10:46:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:51 smithi097 ceph-mon[32231]: pgmap v5979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:52.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:52.033+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:46:52.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:52.034+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:46:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:52.034+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:46:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:52.034+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:46:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:46:52.034+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:53 smithi115 ceph-mon[35364]: pgmap v5980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:53 smithi181 ceph-mon[35869]: pgmap v5980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:53 smithi097 ceph-mon[32231]: pgmap v5980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:55 smithi115 ceph-mon[35364]: pgmap v5981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:55 smithi181 ceph-mon[35869]: pgmap v5981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:55 smithi097 ceph-mon[32231]: pgmap v5981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:46:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10052.879011 secs 2024-07-26T10:46:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10052.879011 secs 2024-07-26T10:46:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10052.879011 secs 2024-07-26T10:46:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:57 smithi115 ceph-mon[35364]: pgmap v5982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:57 smithi181 ceph-mon[35869]: pgmap v5982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:57 smithi097 ceph-mon[32231]: pgmap v5982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:46:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:46:59 smithi115 ceph-mon[35364]: pgmap v5983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:46:59 smithi181 ceph-mon[35869]: pgmap v5983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:46:59 smithi097 ceph-mon[32231]: pgmap v5983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10057.879157 secs 2024-07-26T10:47:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:01 smithi115 ceph-mon[35364]: pgmap v5984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10057.879157 secs 2024-07-26T10:47:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:01 smithi181 ceph-mon[35869]: pgmap v5984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10057.879157 secs 2024-07-26T10:47:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:01 smithi097 ceph-mon[32231]: pgmap v5984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:03 smithi115 ceph-mon[35364]: pgmap v5985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:03 smithi181 ceph-mon[35869]: pgmap v5985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:03 smithi097 ceph-mon[32231]: pgmap v5985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:05 smithi115 ceph-mon[35364]: pgmap v5986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:05 smithi181 ceph-mon[35869]: pgmap v5986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:05 smithi097 ceph-mon[32231]: pgmap v5986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10062.879289 secs 2024-07-26T10:47:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10062.879289 secs 2024-07-26T10:47:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10062.879289 secs 2024-07-26T10:47:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:07.040+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:47:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:07.042+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:47:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:07.042+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:47:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:07.042+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:47:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:07.042+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:07 smithi115 ceph-mon[35364]: pgmap v5987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:07 smithi181 ceph-mon[35869]: pgmap v5987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:07 smithi097 ceph-mon[32231]: pgmap v5987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:09 smithi115 ceph-mon[35364]: pgmap v5988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:09 smithi181 ceph-mon[35869]: pgmap v5988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:09 smithi097 ceph-mon[32231]: pgmap v5988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10067.879433 secs 2024-07-26T10:47:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:11 smithi115 ceph-mon[35364]: pgmap v5989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10067.879433 secs 2024-07-26T10:47:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:11 smithi181 ceph-mon[35869]: pgmap v5989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10067.879433 secs 2024-07-26T10:47:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:11 smithi097 ceph-mon[32231]: pgmap v5989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:13 smithi115 ceph-mon[35364]: pgmap v5990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:13 smithi181 ceph-mon[35869]: pgmap v5990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:13 smithi097 ceph-mon[32231]: pgmap v5990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:15 smithi115 ceph-mon[35364]: pgmap v5991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:15 smithi181 ceph-mon[35869]: pgmap v5991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:15 smithi097 ceph-mon[32231]: pgmap v5991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10072.879600 secs 2024-07-26T10:47:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10072.879600 secs 2024-07-26T10:47:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10072.879600 secs 2024-07-26T10:47:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:17 smithi115 ceph-mon[35364]: pgmap v5992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:17 smithi181 ceph-mon[35869]: pgmap v5992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:17 smithi097 ceph-mon[32231]: pgmap v5992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:18 smithi115 ceph-mon[35364]: pgmap v5993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:18 smithi181 ceph-mon[35869]: pgmap v5993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:18 smithi097 ceph-mon[32231]: pgmap v5993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10077.879752 secs 2024-07-26T10:47:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:21 smithi115 ceph-mon[35364]: pgmap v5994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10077.879752 secs 2024-07-26T10:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:21 smithi181 ceph-mon[35869]: pgmap v5994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10077.879752 secs 2024-07-26T10:47:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:21 smithi097 ceph-mon[32231]: pgmap v5994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:22.043+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:47:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:22.044+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:47:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:22.044+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:47:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:22.044+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:47:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:22.044+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:23.298 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:23 smithi181 ceph-mon[35869]: pgmap v5995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:23 smithi115 ceph-mon[35364]: pgmap v5995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:23 smithi097 ceph-mon[32231]: pgmap v5995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:25 smithi115 ceph-mon[35364]: pgmap v5996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:25 smithi181 ceph-mon[35869]: pgmap v5996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:25 smithi097 ceph-mon[32231]: pgmap v5996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:26.343 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10082.879890 secs 2024-07-26T10:47:26.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10082.879890 secs 2024-07-26T10:47:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10082.879890 secs 2024-07-26T10:47:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:27 smithi181 ceph-mon[35869]: pgmap v5997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:27.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:27 smithi097 ceph-mon[32231]: pgmap v5997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:27.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:27.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:27 smithi115 ceph-mon[35364]: pgmap v5997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:27.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:27.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:27.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:47:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:28 smithi181 ceph-mon[35869]: pgmap v5998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:47:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:28 smithi181 ceph-mon[35869]: pgmap v5999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:47:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:28 smithi097 ceph-mon[32231]: pgmap v5998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:47:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:28 smithi097 ceph-mon[32231]: pgmap v5999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:47:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:28 smithi115 ceph-mon[35364]: pgmap v5998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:47:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:28 smithi115 ceph-mon[35364]: pgmap v5999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T10:47:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:30 smithi097 ceph-mon[32231]: pgmap v6000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:47:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:30 smithi115 ceph-mon[35364]: pgmap v6000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:47:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:30 smithi181 ceph-mon[35869]: pgmap v6000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:47:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10087.880031 secs 2024-07-26T10:47:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10087.880031 secs 2024-07-26T10:47:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10087.880031 secs 2024-07-26T10:47:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:32 smithi115 ceph-mon[35364]: pgmap v6001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:47:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:32 smithi181 ceph-mon[35869]: pgmap v6001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:47:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:32 smithi097 ceph-mon[32231]: pgmap v6001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:47:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:34 smithi115 ceph-mon[35364]: pgmap v6002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:34 smithi181 ceph-mon[35869]: pgmap v6002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:34 smithi097 ceph-mon[32231]: pgmap v6002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:36 smithi115 ceph-mon[35364]: pgmap v6003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:47:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10092.880199 secs 2024-07-26T10:47:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:36 smithi181 ceph-mon[35869]: pgmap v6003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:47:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10092.880199 secs 2024-07-26T10:47:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:36 smithi097 ceph-mon[32231]: pgmap v6003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:47:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10092.880199 secs 2024-07-26T10:47:37.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:37.051+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:47:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:37.052+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:47:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:37.052+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:47:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:37.052+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:47:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:37.052+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:38 smithi115 ceph-mon[35364]: pgmap v6004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:47:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:38 smithi181 ceph-mon[35869]: pgmap v6004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:47:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:38 smithi097 ceph-mon[32231]: pgmap v6004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:47:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:40 smithi115 ceph-mon[35364]: pgmap v6005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:40 smithi181 ceph-mon[35869]: pgmap v6005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:40 smithi097 ceph-mon[32231]: pgmap v6005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10097.880339 secs 2024-07-26T10:47:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10097.880339 secs 2024-07-26T10:47:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10097.880339 secs 2024-07-26T10:47:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:42 smithi115 ceph-mon[35364]: pgmap v6006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:42 smithi181 ceph-mon[35869]: pgmap v6006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:42 smithi097 ceph-mon[32231]: pgmap v6006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:44 smithi115 ceph-mon[35364]: pgmap v6007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:44 smithi181 ceph-mon[35869]: pgmap v6007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:44 smithi097 ceph-mon[32231]: pgmap v6007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:46.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:46 smithi181 ceph-mon[35869]: pgmap v6008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:46.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10102.880508 secs 2024-07-26T10:47:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:46 smithi115 ceph-mon[35364]: pgmap v6008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10102.880508 secs 2024-07-26T10:47:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:46 smithi097 ceph-mon[32231]: pgmap v6008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10102.880508 secs 2024-07-26T10:47:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:48 smithi115 ceph-mon[35364]: pgmap v6009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:48.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:48 smithi181 ceph-mon[35869]: pgmap v6009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:48 smithi097 ceph-mon[32231]: pgmap v6009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:50 smithi115 ceph-mon[35364]: pgmap v6010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:50 smithi181 ceph-mon[35869]: pgmap v6010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:50 smithi097 ceph-mon[32231]: pgmap v6010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10107.880671 secs 2024-07-26T10:47:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10107.880671 secs 2024-07-26T10:47:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10107.880671 secs 2024-07-26T10:47:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:52.055+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:47:52.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:52.056+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:47:52.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:52.056+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:47:52.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:52.056+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:47:52.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:47:52.056+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:52 smithi115 ceph-mon[35364]: pgmap v6011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:52 smithi181 ceph-mon[35869]: pgmap v6011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:52 smithi097 ceph-mon[32231]: pgmap v6011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:54 smithi115 ceph-mon[35364]: pgmap v6012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:54 smithi181 ceph-mon[35869]: pgmap v6012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:54 smithi097 ceph-mon[32231]: pgmap v6012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:56 smithi115 ceph-mon[35364]: pgmap v6013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10112.880830 secs 2024-07-26T10:47:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:56 smithi181 ceph-mon[35869]: pgmap v6013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10112.880830 secs 2024-07-26T10:47:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:56 smithi097 ceph-mon[32231]: pgmap v6013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10112.880830 secs 2024-07-26T10:47:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:47:58 smithi181 ceph-mon[35869]: pgmap v6014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:47:58 smithi115 ceph-mon[35364]: pgmap v6014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:47:58 smithi097 ceph-mon[32231]: pgmap v6014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:00 smithi115 ceph-mon[35364]: pgmap v6015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:00 smithi181 ceph-mon[35869]: pgmap v6015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:00 smithi097 ceph-mon[32231]: pgmap v6015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10117.880979 secs 2024-07-26T10:48:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10117.880979 secs 2024-07-26T10:48:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10117.880979 secs 2024-07-26T10:48:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:02 smithi115 ceph-mon[35364]: pgmap v6016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:02 smithi181 ceph-mon[35869]: pgmap v6016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:02 smithi097 ceph-mon[32231]: pgmap v6016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:03 smithi115 ceph-mon[35364]: pgmap v6017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:03 smithi181 ceph-mon[35869]: pgmap v6017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:03 smithi097 ceph-mon[32231]: pgmap v6017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:05 smithi115 ceph-mon[35364]: pgmap v6018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10122.881138 secs 2024-07-26T10:48:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:05 smithi181 ceph-mon[35869]: pgmap v6018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10122.881138 secs 2024-07-26T10:48:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:05 smithi097 ceph-mon[32231]: pgmap v6018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10122.881138 secs 2024-07-26T10:48:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:07.058+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:48:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:07.059+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:48:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:07.059+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:48:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:07.059+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:48:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:07.059+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:07 smithi115 ceph-mon[35364]: pgmap v6019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:07 smithi181 ceph-mon[35869]: pgmap v6019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:07 smithi097 ceph-mon[32231]: pgmap v6019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:09 smithi115 ceph-mon[35364]: pgmap v6020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:09 smithi181 ceph-mon[35869]: pgmap v6020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:09 smithi097 ceph-mon[32231]: pgmap v6020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10127.881318 secs 2024-07-26T10:48:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10127.881318 secs 2024-07-26T10:48:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10127.881318 secs 2024-07-26T10:48:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:11 smithi115 ceph-mon[35364]: pgmap v6021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:11 smithi181 ceph-mon[35869]: pgmap v6021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:11 smithi097 ceph-mon[32231]: pgmap v6021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:13 smithi115 ceph-mon[35364]: pgmap v6022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:13 smithi181 ceph-mon[35869]: pgmap v6022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:13 smithi097 ceph-mon[32231]: pgmap v6022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:15 smithi115 ceph-mon[35364]: pgmap v6023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10132.881480 secs 2024-07-26T10:48:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:15 smithi181 ceph-mon[35869]: pgmap v6023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10132.881480 secs 2024-07-26T10:48:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:15 smithi097 ceph-mon[32231]: pgmap v6023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10132.881480 secs 2024-07-26T10:48:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:17 smithi115 ceph-mon[35364]: pgmap v6024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:17 smithi181 ceph-mon[35869]: pgmap v6024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:17 smithi097 ceph-mon[32231]: pgmap v6024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:19 smithi115 ceph-mon[35364]: pgmap v6025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:19 smithi181 ceph-mon[35869]: pgmap v6025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:19 smithi097 ceph-mon[32231]: pgmap v6025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10137.881606 secs 2024-07-26T10:48:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10137.881606 secs 2024-07-26T10:48:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10137.881606 secs 2024-07-26T10:48:22.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:22.060+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:48:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:22.062+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:48:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:22.062+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:48:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:22.062+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:48:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:22.062+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:21 smithi115 ceph-mon[35364]: pgmap v6026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:21 smithi181 ceph-mon[35869]: pgmap v6026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:21 smithi097 ceph-mon[32231]: pgmap v6026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:23 smithi115 ceph-mon[35364]: pgmap v6027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:23 smithi181 ceph-mon[35869]: pgmap v6027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:23 smithi097 ceph-mon[32231]: pgmap v6027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:25 smithi115 ceph-mon[35364]: pgmap v6028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10142.881692 secs 2024-07-26T10:48:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:25 smithi181 ceph-mon[35869]: pgmap v6028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10142.881692 secs 2024-07-26T10:48:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:25 smithi097 ceph-mon[32231]: pgmap v6028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10142.881692 secs 2024-07-26T10:48:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:27 smithi115 ceph-mon[35364]: pgmap v6029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:27 smithi181 ceph-mon[35869]: pgmap v6029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:27 smithi097 ceph-mon[32231]: pgmap v6029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:28 smithi115 ceph-mon[35364]: pgmap v6030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:48:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:48:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:28 smithi181 ceph-mon[35869]: pgmap v6030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:48:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:48:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:28 smithi097 ceph-mon[32231]: pgmap v6030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:48:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:48:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10147.881824 secs 2024-07-26T10:48:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:31 smithi115 ceph-mon[35364]: pgmap v6031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:48:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10147.881824 secs 2024-07-26T10:48:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:31 smithi181 ceph-mon[35869]: pgmap v6031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:48:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10147.881824 secs 2024-07-26T10:48:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:31 smithi097 ceph-mon[32231]: pgmap v6031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:48:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:33 smithi115 ceph-mon[35364]: pgmap v6032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:48:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:33 smithi181 ceph-mon[35869]: pgmap v6032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:48:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:33 smithi097 ceph-mon[32231]: pgmap v6032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:48:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:35 smithi115 ceph-mon[35364]: pgmap v6033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:48:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:35 smithi181 ceph-mon[35869]: pgmap v6033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:48:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:35 smithi097 ceph-mon[32231]: pgmap v6033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:48:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10152.881961 secs 2024-07-26T10:48:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10152.881961 secs 2024-07-26T10:48:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10152.881961 secs 2024-07-26T10:48:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:37.069+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:48:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:37.070+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:48:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:37.070+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:48:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:37.070+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:48:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:37.070+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:37 smithi115 ceph-mon[35364]: pgmap v6034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:48:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:37 smithi181 ceph-mon[35869]: pgmap v6034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:48:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:37 smithi097 ceph-mon[32231]: pgmap v6034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:48:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:39 smithi115 ceph-mon[35364]: pgmap v6035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:48:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:39 smithi181 ceph-mon[35869]: pgmap v6035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:48:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:39 smithi097 ceph-mon[32231]: pgmap v6035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:48:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10157.882094 secs 2024-07-26T10:48:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:41 smithi115 ceph-mon[35364]: pgmap v6036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:48:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10157.882094 secs 2024-07-26T10:48:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:41 smithi181 ceph-mon[35869]: pgmap v6036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:48:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10157.882094 secs 2024-07-26T10:48:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:41 smithi097 ceph-mon[32231]: pgmap v6036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:48:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:43 smithi115 ceph-mon[35364]: pgmap v6037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:48:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:43 smithi181 ceph-mon[35869]: pgmap v6037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:48:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:43 smithi097 ceph-mon[32231]: pgmap v6037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:48:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:45 smithi115 ceph-mon[35364]: pgmap v6038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:48:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:45 smithi181 ceph-mon[35869]: pgmap v6038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:48:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:45 smithi097 ceph-mon[32231]: pgmap v6038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:48:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10162.882225 secs 2024-07-26T10:48:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10162.882225 secs 2024-07-26T10:48:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10162.882225 secs 2024-07-26T10:48:47.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:47 smithi115 ceph-mon[35364]: pgmap v6039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:47.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:47 smithi181 ceph-mon[35869]: pgmap v6039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:47.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:47 smithi097 ceph-mon[32231]: pgmap v6039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:49 smithi181 ceph-mon[35869]: pgmap v6040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:49 smithi115 ceph-mon[35364]: pgmap v6040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:49 smithi097 ceph-mon[32231]: pgmap v6040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10167.882361 secs 2024-07-26T10:48:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:51 smithi115 ceph-mon[35364]: pgmap v6041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10167.882361 secs 2024-07-26T10:48:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:51 smithi181 ceph-mon[35869]: pgmap v6041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10167.882361 secs 2024-07-26T10:48:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:51 smithi097 ceph-mon[32231]: pgmap v6041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:52.073+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:48:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:52.074+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:48:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:52.074+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:48:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:52.074+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:48:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:48:52.074+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:53 smithi115 ceph-mon[35364]: pgmap v6042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:53 smithi181 ceph-mon[35869]: pgmap v6042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:53 smithi097 ceph-mon[32231]: pgmap v6042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:55 smithi115 ceph-mon[35364]: pgmap v6043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:55 smithi181 ceph-mon[35869]: pgmap v6043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:55 smithi097 ceph-mon[32231]: pgmap v6043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10172.882491 secs 2024-07-26T10:48:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10172.882491 secs 2024-07-26T10:48:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10172.882491 secs 2024-07-26T10:48:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:57 smithi115 ceph-mon[35364]: pgmap v6044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:57 smithi181 ceph-mon[35869]: pgmap v6044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:57 smithi097 ceph-mon[32231]: pgmap v6044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:48:59 smithi115 ceph-mon[35364]: pgmap v6045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:48:59 smithi181 ceph-mon[35869]: pgmap v6045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:48:59 smithi097 ceph-mon[32231]: pgmap v6045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10177.882625 secs 2024-07-26T10:49:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:01 smithi115 ceph-mon[35364]: pgmap v6046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10177.882625 secs 2024-07-26T10:49:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:01 smithi181 ceph-mon[35869]: pgmap v6046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10177.882625 secs 2024-07-26T10:49:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:01 smithi097 ceph-mon[32231]: pgmap v6046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:03 smithi115 ceph-mon[35364]: pgmap v6047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:03 smithi181 ceph-mon[35869]: pgmap v6047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:03 smithi097 ceph-mon[32231]: pgmap v6047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:05 smithi115 ceph-mon[35364]: pgmap v6048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:05 smithi181 ceph-mon[35869]: pgmap v6048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:05 smithi097 ceph-mon[32231]: pgmap v6048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10182.882752 secs 2024-07-26T10:49:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10182.882752 secs 2024-07-26T10:49:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10182.882752 secs 2024-07-26T10:49:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:07.080+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:49:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:07.082+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:49:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:07.082+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:49:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:07.082+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:49:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:07.082+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:07 smithi115 ceph-mon[35364]: pgmap v6049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:07 smithi181 ceph-mon[35869]: pgmap v6049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:07 smithi097 ceph-mon[32231]: pgmap v6049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:08.681+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:49:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:08.681+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:49:08.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:08.682+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T10:49:08.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:08.682+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:49:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:09 smithi115 ceph-mon[35364]: pgmap v6050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:09 smithi181 ceph-mon[35869]: pgmap v6050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:09 smithi097 ceph-mon[32231]: pgmap v6050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10187.882884 secs 2024-07-26T10:49:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:11 smithi115 ceph-mon[35364]: pgmap v6051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10187.882884 secs 2024-07-26T10:49:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:11 smithi181 ceph-mon[35869]: pgmap v6051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10187.882884 secs 2024-07-26T10:49:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:11 smithi097 ceph-mon[32231]: pgmap v6051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:13 smithi115 ceph-mon[35364]: pgmap v6052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:13 smithi181 ceph-mon[35869]: pgmap v6052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:13 smithi097 ceph-mon[32231]: pgmap v6052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:15 smithi115 ceph-mon[35364]: pgmap v6053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:15 smithi181 ceph-mon[35869]: pgmap v6053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:15 smithi097 ceph-mon[32231]: pgmap v6053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10192.883018 secs 2024-07-26T10:49:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10192.883018 secs 2024-07-26T10:49:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10192.883018 secs 2024-07-26T10:49:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:17 smithi181 ceph-mon[35869]: pgmap v6054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:17 smithi097 ceph-mon[32231]: pgmap v6054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:17 smithi115 ceph-mon[35364]: pgmap v6054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:19 smithi097 ceph-mon[32231]: pgmap v6055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:19 smithi115 ceph-mon[35364]: pgmap v6055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:19 smithi181 ceph-mon[35869]: pgmap v6055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10197.883153 secs 2024-07-26T10:49:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:21 smithi097 ceph-mon[32231]: pgmap v6056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10197.883153 secs 2024-07-26T10:49:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:21 smithi115 ceph-mon[35364]: pgmap v6056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10197.883153 secs 2024-07-26T10:49:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:21 smithi181 ceph-mon[35869]: pgmap v6056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:22.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:22.083+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:49:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:22.084+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:49:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:22.084+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:49:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:22.084+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:49:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:22.084+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:23.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:23 smithi097 ceph-mon[32231]: pgmap v6057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:23 smithi115 ceph-mon[35364]: pgmap v6057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:23 smithi181 ceph-mon[35869]: pgmap v6057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:25 smithi115 ceph-mon[35364]: pgmap v6058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:25 smithi181 ceph-mon[35869]: pgmap v6058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:25 smithi097 ceph-mon[32231]: pgmap v6058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10202.883285 secs 2024-07-26T10:49:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10202.883285 secs 2024-07-26T10:49:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10202.883285 secs 2024-07-26T10:49:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:27 smithi115 ceph-mon[35364]: pgmap v6059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:27 smithi181 ceph-mon[35869]: pgmap v6059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:27 smithi097 ceph-mon[32231]: pgmap v6059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:29 smithi115 ceph-mon[35364]: pgmap v6060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:29 smithi181 ceph-mon[35869]: pgmap v6060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:29 smithi097 ceph-mon[32231]: pgmap v6060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:30 smithi115 ceph-mon[35364]: pgmap v6061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:49:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:30 smithi115 ceph-mon[35364]: pgmap v6062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:49:30.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:30.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:30.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:30 smithi181 ceph-mon[35869]: pgmap v6061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:49:30.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:30 smithi181 ceph-mon[35869]: pgmap v6062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:30.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:49:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:30 smithi097 ceph-mon[32231]: pgmap v6061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T10:49:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:30 smithi097 ceph-mon[32231]: pgmap v6062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:49:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10207.883418 secs 2024-07-26T10:49:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10207.883418 secs 2024-07-26T10:49:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10207.883418 secs 2024-07-26T10:49:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:32 smithi115 ceph-mon[35364]: pgmap v6063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:49:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:32 smithi181 ceph-mon[35869]: pgmap v6063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:49:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:32 smithi097 ceph-mon[32231]: pgmap v6063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:49:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:33 smithi115 ceph-mon[35364]: pgmap v6064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:49:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:33 smithi181 ceph-mon[35869]: pgmap v6064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:49:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:33 smithi097 ceph-mon[32231]: pgmap v6064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T10:49:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:35 smithi115 ceph-mon[35364]: pgmap v6065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:49:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:35 smithi181 ceph-mon[35869]: pgmap v6065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:49:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:35 smithi097 ceph-mon[32231]: pgmap v6065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:49:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10212.883548 secs 2024-07-26T10:49:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10212.883548 secs 2024-07-26T10:49:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10212.883548 secs 2024-07-26T10:49:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:37.090+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:49:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:37.091+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:49:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:37.091+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:49:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:37.091+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:49:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:37.091+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:37 smithi181 ceph-mon[35869]: pgmap v6066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:49:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:37 smithi097 ceph-mon[32231]: pgmap v6066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:49:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:37 smithi115 ceph-mon[35364]: pgmap v6066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:49:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:39 smithi115 ceph-mon[35364]: pgmap v6067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:49:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:39 smithi181 ceph-mon[35869]: pgmap v6067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:49:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:39 smithi097 ceph-mon[32231]: pgmap v6067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:49:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10217.883703 secs 2024-07-26T10:49:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10217.883703 secs 2024-07-26T10:49:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10217.883703 secs 2024-07-26T10:49:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:41 smithi115 ceph-mon[35364]: pgmap v6068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:49:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:41 smithi181 ceph-mon[35869]: pgmap v6068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:49:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:41 smithi097 ceph-mon[32231]: pgmap v6068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:49:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:43 smithi115 ceph-mon[35364]: pgmap v6069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:49:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:43 smithi181 ceph-mon[35869]: pgmap v6069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:49:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:43 smithi097 ceph-mon[32231]: pgmap v6069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:49:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:45 smithi115 ceph-mon[35364]: pgmap v6070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:49:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:45 smithi181 ceph-mon[35869]: pgmap v6070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:49:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:45 smithi097 ceph-mon[32231]: pgmap v6070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:49:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10222.883846 secs 2024-07-26T10:49:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10222.883846 secs 2024-07-26T10:49:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10222.883846 secs 2024-07-26T10:49:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:47 smithi181 ceph-mon[35869]: pgmap v6071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:49:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:47 smithi097 ceph-mon[32231]: pgmap v6071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:49:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:47 smithi115 ceph-mon[35364]: pgmap v6071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:49:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:49 smithi115 ceph-mon[35364]: pgmap v6072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:49:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:49 smithi181 ceph-mon[35869]: pgmap v6072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:49:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:49 smithi097 ceph-mon[32231]: pgmap v6072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:49:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10227.883990 secs 2024-07-26T10:49:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10227.883990 secs 2024-07-26T10:49:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10227.883990 secs 2024-07-26T10:49:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:51 smithi115 ceph-mon[35364]: pgmap v6073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:51 smithi181 ceph-mon[35869]: pgmap v6073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:51 smithi097 ceph-mon[32231]: pgmap v6073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:52.094 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:52.092+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:49:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:52.093+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:49:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:52.093+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:49:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:52.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:49:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:49:52.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:54 smithi115 ceph-mon[35364]: pgmap v6074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:54 smithi181 ceph-mon[35869]: pgmap v6074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:54 smithi097 ceph-mon[32231]: pgmap v6074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10232.884126 secs 2024-07-26T10:49:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:56 smithi115 ceph-mon[35364]: pgmap v6075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10232.884126 secs 2024-07-26T10:49:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:56 smithi181 ceph-mon[35869]: pgmap v6075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10232.884126 secs 2024-07-26T10:49:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:56 smithi097 ceph-mon[32231]: pgmap v6075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:49:58 smithi181 ceph-mon[35869]: pgmap v6076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:49:58 smithi097 ceph-mon[32231]: pgmap v6076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:49:58 smithi115 ceph-mon[35364]: pgmap v6076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:00 smithi181 ceph-mon[35869]: pgmap v6077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:50:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:00 smithi097 ceph-mon[32231]: pgmap v6077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:50:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:00 smithi115 ceph-mon[35364]: pgmap v6077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:50:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10237.884265 secs 2024-07-26T10:50:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10237.884265 secs 2024-07-26T10:50:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10237.884265 secs 2024-07-26T10:50:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:02 smithi097 ceph-mon[32231]: pgmap v6078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:02 smithi115 ceph-mon[35364]: pgmap v6078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:02 smithi181 ceph-mon[35869]: pgmap v6078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:04 smithi115 ceph-mon[35364]: pgmap v6079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:04 smithi181 ceph-mon[35869]: pgmap v6079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:04 smithi097 ceph-mon[32231]: pgmap v6079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10242.884398 secs 2024-07-26T10:50:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:06 smithi115 ceph-mon[35364]: pgmap v6080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:06.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10242.884398 secs 2024-07-26T10:50:06.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:06 smithi181 ceph-mon[35869]: pgmap v6080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10242.884398 secs 2024-07-26T10:50:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:06 smithi097 ceph-mon[32231]: pgmap v6080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:07.099+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:07.100+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:50:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:07.100+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:50:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:07.100+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:50:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:07.100+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:08 smithi115 ceph-mon[35364]: pgmap v6081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:08 smithi181 ceph-mon[35869]: pgmap v6081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:08 smithi097 ceph-mon[32231]: pgmap v6081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:10 smithi115 ceph-mon[35364]: pgmap v6082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:10 smithi181 ceph-mon[35869]: pgmap v6082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:10 smithi097 ceph-mon[32231]: pgmap v6082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10247.884561 secs 2024-07-26T10:50:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10247.884561 secs 2024-07-26T10:50:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10247.884561 secs 2024-07-26T10:50:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:12 smithi115 ceph-mon[35364]: pgmap v6083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:12 smithi181 ceph-mon[35869]: pgmap v6083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:12 smithi097 ceph-mon[32231]: pgmap v6083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:14 smithi115 ceph-mon[35364]: pgmap v6084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:14 smithi181 ceph-mon[35869]: pgmap v6084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:14 smithi097 ceph-mon[32231]: pgmap v6084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10252.884703 secs 2024-07-26T10:50:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:16 smithi115 ceph-mon[35364]: pgmap v6085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10252.884703 secs 2024-07-26T10:50:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:16 smithi181 ceph-mon[35869]: pgmap v6085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10252.884703 secs 2024-07-26T10:50:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:16 smithi097 ceph-mon[32231]: pgmap v6085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:18 smithi115 ceph-mon[35364]: pgmap v6086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:18 smithi181 ceph-mon[35869]: pgmap v6086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:18 smithi097 ceph-mon[32231]: pgmap v6086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:20 smithi115 ceph-mon[35364]: pgmap v6087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10257.884845 secs 2024-07-26T10:50:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:20 smithi181 ceph-mon[35869]: pgmap v6087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10257.884845 secs 2024-07-26T10:50:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:20 smithi097 ceph-mon[32231]: pgmap v6087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10257.884845 secs 2024-07-26T10:50:22.104 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:22.102+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:22.103+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:50:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:22.103+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:50:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:22.103+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:50:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:22.103+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:22 smithi115 ceph-mon[35364]: pgmap v6088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:22 smithi181 ceph-mon[35869]: pgmap v6088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:22 smithi097 ceph-mon[32231]: pgmap v6088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:24 smithi115 ceph-mon[35364]: pgmap v6089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:24 smithi181 ceph-mon[35869]: pgmap v6089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:24 smithi097 ceph-mon[32231]: pgmap v6089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10262.884986 secs 2024-07-26T10:50:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10262.884986 secs 2024-07-26T10:50:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10262.884986 secs 2024-07-26T10:50:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:26 smithi115 ceph-mon[35364]: pgmap v6090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:26 smithi181 ceph-mon[35869]: pgmap v6090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:26 smithi097 ceph-mon[32231]: pgmap v6090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:28 smithi115 ceph-mon[35364]: pgmap v6091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:28 smithi181 ceph-mon[35869]: pgmap v6091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:28 smithi097 ceph-mon[32231]: pgmap v6091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:30 smithi115 ceph-mon[35364]: pgmap v6092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10267.885149 secs 2024-07-26T10:50:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:30 smithi181 ceph-mon[35869]: pgmap v6092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10267.885149 secs 2024-07-26T10:50:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:30 smithi097 ceph-mon[32231]: pgmap v6092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:50:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10267.885149 secs 2024-07-26T10:50:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:31 smithi115 ceph-mon[35364]: pgmap v6093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:50:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:31 smithi181 ceph-mon[35869]: pgmap v6093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:50:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:31 smithi097 ceph-mon[32231]: pgmap v6093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:50:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:33 smithi115 ceph-mon[35364]: pgmap v6094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:50:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:33 smithi181 ceph-mon[35869]: pgmap v6094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:50:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:33 smithi097 ceph-mon[32231]: pgmap v6094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:50:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:35 smithi115 ceph-mon[35364]: pgmap v6095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:50:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10272.885295 secs 2024-07-26T10:50:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:35 smithi181 ceph-mon[35869]: pgmap v6095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:50:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10272.885295 secs 2024-07-26T10:50:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:35 smithi097 ceph-mon[32231]: pgmap v6095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:50:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10272.885295 secs 2024-07-26T10:50:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:37.107+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:37.108+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:50:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:37.108+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:50:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:37.108+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:50:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:37.108+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:37 smithi115 ceph-mon[35364]: pgmap v6096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:37 smithi181 ceph-mon[35869]: pgmap v6096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:37 smithi097 ceph-mon[32231]: pgmap v6096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:39 smithi115 ceph-mon[35364]: pgmap v6097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:50:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:39 smithi181 ceph-mon[35869]: pgmap v6097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:50:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:39 smithi097 ceph-mon[32231]: pgmap v6097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:50:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10277.885393 secs 2024-07-26T10:50:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10277.885393 secs 2024-07-26T10:50:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10277.885393 secs 2024-07-26T10:50:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:41 smithi115 ceph-mon[35364]: pgmap v6098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:41 smithi181 ceph-mon[35869]: pgmap v6098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:41 smithi097 ceph-mon[32231]: pgmap v6098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:50:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:43 smithi115 ceph-mon[35364]: pgmap v6099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:43 smithi181 ceph-mon[35869]: pgmap v6099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:43 smithi097 ceph-mon[32231]: pgmap v6099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:45 smithi115 ceph-mon[35364]: pgmap v6100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10282.885536 secs 2024-07-26T10:50:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:45 smithi181 ceph-mon[35869]: pgmap v6100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10282.885536 secs 2024-07-26T10:50:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:45 smithi097 ceph-mon[32231]: pgmap v6100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10282.885536 secs 2024-07-26T10:50:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:47 smithi115 ceph-mon[35364]: pgmap v6101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:47 smithi181 ceph-mon[35869]: pgmap v6101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:47 smithi097 ceph-mon[32231]: pgmap v6101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:49 smithi115 ceph-mon[35364]: pgmap v6102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:49 smithi181 ceph-mon[35869]: pgmap v6102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:49 smithi097 ceph-mon[32231]: pgmap v6102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10287.885639 secs 2024-07-26T10:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10287.885639 secs 2024-07-26T10:50:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10287.885639 secs 2024-07-26T10:50:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:52.109+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:50:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:52.110+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:50:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:52.110+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:50:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:52.110+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:50:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:50:52.111+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:51 smithi115 ceph-mon[35364]: pgmap v6103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:51 smithi181 ceph-mon[35869]: pgmap v6103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:51 smithi097 ceph-mon[32231]: pgmap v6103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:53 smithi115 ceph-mon[35364]: pgmap v6104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:53 smithi181 ceph-mon[35869]: pgmap v6104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:53 smithi097 ceph-mon[32231]: pgmap v6104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:55 smithi115 ceph-mon[35364]: pgmap v6105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:50:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10292.885780 secs 2024-07-26T10:50:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:55 smithi181 ceph-mon[35869]: pgmap v6105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:50:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10292.885780 secs 2024-07-26T10:50:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:55 smithi097 ceph-mon[32231]: pgmap v6105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:50:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10292.885780 secs 2024-07-26T10:50:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:57 smithi115 ceph-mon[35364]: pgmap v6106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:50:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:57 smithi181 ceph-mon[35869]: pgmap v6106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:50:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:57 smithi097 ceph-mon[32231]: pgmap v6106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:51:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:50:59 smithi115 ceph-mon[35364]: pgmap v6107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:50:59 smithi181 ceph-mon[35869]: pgmap v6107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:50:59 smithi097 ceph-mon[32231]: pgmap v6107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10297.885870 secs 2024-07-26T10:51:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10297.885870 secs 2024-07-26T10:51:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10297.885870 secs 2024-07-26T10:51:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:01 smithi115 ceph-mon[35364]: pgmap v6108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:01 smithi181 ceph-mon[35869]: pgmap v6108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:01 smithi097 ceph-mon[32231]: pgmap v6108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:03 smithi115 ceph-mon[35364]: pgmap v6109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:51:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:03 smithi181 ceph-mon[35869]: pgmap v6109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:51:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:03 smithi097 ceph-mon[32231]: pgmap v6109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:51:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:05 smithi115 ceph-mon[35364]: pgmap v6110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10302.886043 secs 2024-07-26T10:51:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:05 smithi181 ceph-mon[35869]: pgmap v6110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10302.886043 secs 2024-07-26T10:51:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:05 smithi097 ceph-mon[32231]: pgmap v6110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:51:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10302.886043 secs 2024-07-26T10:51:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:07.117+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:07.118+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:51:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:07.118+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:51:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:07.118+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:51:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:07.118+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:07 smithi115 ceph-mon[35364]: pgmap v6111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:07 smithi181 ceph-mon[35869]: pgmap v6111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:07 smithi097 ceph-mon[32231]: pgmap v6111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:09 smithi115 ceph-mon[35364]: pgmap v6112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:09 smithi181 ceph-mon[35869]: pgmap v6112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:09 smithi097 ceph-mon[32231]: pgmap v6112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10307.886193 secs 2024-07-26T10:51:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10307.886193 secs 2024-07-26T10:51:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10307.886193 secs 2024-07-26T10:51:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:11 smithi115 ceph-mon[35364]: pgmap v6113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:11 smithi181 ceph-mon[35869]: pgmap v6113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:11 smithi097 ceph-mon[32231]: pgmap v6113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:13 smithi115 ceph-mon[35364]: pgmap v6114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:13 smithi181 ceph-mon[35869]: pgmap v6114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:13 smithi097 ceph-mon[32231]: pgmap v6114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:15 smithi115 ceph-mon[35364]: pgmap v6115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10312.886347 secs 2024-07-26T10:51:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:15 smithi181 ceph-mon[35869]: pgmap v6115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10312.886347 secs 2024-07-26T10:51:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:15 smithi097 ceph-mon[32231]: pgmap v6115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10312.886347 secs 2024-07-26T10:51:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:17 smithi115 ceph-mon[35364]: pgmap v6116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:17 smithi181 ceph-mon[35869]: pgmap v6116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:17 smithi097 ceph-mon[32231]: pgmap v6116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:19 smithi181 ceph-mon[35869]: pgmap v6117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:19 smithi097 ceph-mon[32231]: pgmap v6117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:19 smithi115 ceph-mon[35364]: pgmap v6117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10317.886479 secs 2024-07-26T10:51:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10317.886479 secs 2024-07-26T10:51:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10317.886479 secs 2024-07-26T10:51:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:22.120+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:22.121+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:51:22.124 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:22.121+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:51:22.124 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:22.121+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:51:22.124 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:22.121+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:21 smithi097 ceph-mon[32231]: pgmap v6118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:21 smithi115 ceph-mon[35364]: pgmap v6118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:21 smithi181 ceph-mon[35869]: pgmap v6118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:23.705+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:51:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:23.705+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T10:51:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:23.706+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T10:51:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:23.706+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:51:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:23 smithi097 ceph-mon[32231]: pgmap v6119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:23 smithi115 ceph-mon[35364]: pgmap v6119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:23 smithi181 ceph-mon[35869]: pgmap v6119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:26 smithi115 ceph-mon[35364]: pgmap v6120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10322.886621 secs 2024-07-26T10:51:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:26 smithi181 ceph-mon[35869]: pgmap v6120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10322.886621 secs 2024-07-26T10:51:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:26 smithi097 ceph-mon[32231]: pgmap v6120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10322.886621 secs 2024-07-26T10:51:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:28 smithi115 ceph-mon[35364]: pgmap v6121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:28 smithi181 ceph-mon[35869]: pgmap v6121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:28 smithi097 ceph-mon[32231]: pgmap v6121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:30 smithi115 ceph-mon[35364]: pgmap v6122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:30 smithi181 ceph-mon[35869]: pgmap v6122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:30 smithi097 ceph-mon[32231]: pgmap v6122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10327.886766 secs 2024-07-26T10:51:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10327.886766 secs 2024-07-26T10:51:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10327.886766 secs 2024-07-26T10:51:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:32 smithi115 ceph-mon[35364]: pgmap v6123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:32 smithi181 ceph-mon[35869]: pgmap v6123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:32 smithi097 ceph-mon[32231]: pgmap v6123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:33.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:33 smithi115 ceph-mon[35364]: pgmap v6124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:51:33.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:33 smithi115 ceph-mon[35364]: pgmap v6125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:51:33.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:51:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:33 smithi181 ceph-mon[35869]: pgmap v6124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:51:33.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:33 smithi181 ceph-mon[35869]: pgmap v6125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:51:33.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:51:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:33 smithi097 ceph-mon[32231]: pgmap v6124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T10:51:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:33 smithi097 ceph-mon[32231]: pgmap v6125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:51:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:51:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:35 smithi115 ceph-mon[35364]: pgmap v6126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:35 smithi181 ceph-mon[35869]: pgmap v6126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:35 smithi097 ceph-mon[32231]: pgmap v6126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10332.886859 secs 2024-07-26T10:51:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10332.886859 secs 2024-07-26T10:51:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10332.886859 secs 2024-07-26T10:51:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:37.127+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:37.129+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:51:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:37.129+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:51:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:37.129+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:51:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:37.129+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:37 smithi115 ceph-mon[35364]: pgmap v6127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:37 smithi181 ceph-mon[35869]: pgmap v6127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:37 smithi097 ceph-mon[32231]: pgmap v6127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:39 smithi115 ceph-mon[35364]: pgmap v6128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:39 smithi181 ceph-mon[35869]: pgmap v6128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:39 smithi097 ceph-mon[32231]: pgmap v6128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10337.887001 secs 2024-07-26T10:51:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:41 smithi115 ceph-mon[35364]: pgmap v6129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10337.887001 secs 2024-07-26T10:51:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:41 smithi181 ceph-mon[35869]: pgmap v6129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10337.887001 secs 2024-07-26T10:51:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:41 smithi097 ceph-mon[32231]: pgmap v6129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:43 smithi115 ceph-mon[35364]: pgmap v6130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:51:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:43 smithi181 ceph-mon[35869]: pgmap v6130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:51:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:43 smithi097 ceph-mon[32231]: pgmap v6130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:51:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:45 smithi115 ceph-mon[35364]: pgmap v6131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:45 smithi181 ceph-mon[35869]: pgmap v6131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:45 smithi097 ceph-mon[32231]: pgmap v6131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:46.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10342.887120 secs 2024-07-26T10:51:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10342.887120 secs 2024-07-26T10:51:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10342.887120 secs 2024-07-26T10:51:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:47 smithi115 ceph-mon[35364]: pgmap v6132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:47 smithi181 ceph-mon[35869]: pgmap v6132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:47.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:47 smithi097 ceph-mon[32231]: pgmap v6132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:48 smithi115 ceph-mon[35364]: pgmap v6133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:48 smithi181 ceph-mon[35869]: pgmap v6133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:48 smithi097 ceph-mon[32231]: pgmap v6133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10347.887344 secs 2024-07-26T10:51:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:51 smithi115 ceph-mon[35364]: pgmap v6134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10347.887344 secs 2024-07-26T10:51:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:51 smithi181 ceph-mon[35869]: pgmap v6134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10347.887344 secs 2024-07-26T10:51:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:51 smithi097 ceph-mon[32231]: pgmap v6134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:52.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:52.131+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:51:52.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:52.133+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:51:52.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:52.133+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:51:52.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:52.133+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:51:52.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:51:52.133+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:53 smithi115 ceph-mon[35364]: pgmap v6135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:53 smithi181 ceph-mon[35869]: pgmap v6135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:53 smithi097 ceph-mon[32231]: pgmap v6135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:55 smithi115 ceph-mon[35364]: pgmap v6136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:55 smithi181 ceph-mon[35869]: pgmap v6136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:55 smithi097 ceph-mon[32231]: pgmap v6136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10352.887479 secs 2024-07-26T10:51:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10352.887479 secs 2024-07-26T10:51:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10352.887479 secs 2024-07-26T10:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:57 smithi115 ceph-mon[35364]: pgmap v6137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:57 smithi181 ceph-mon[35869]: pgmap v6137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:57 smithi097 ceph-mon[32231]: pgmap v6137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:51:59 smithi115 ceph-mon[35364]: pgmap v6138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:51:59 smithi181 ceph-mon[35869]: pgmap v6138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:51:59 smithi097 ceph-mon[32231]: pgmap v6138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10357.887617 secs 2024-07-26T10:52:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:01 smithi115 ceph-mon[35364]: pgmap v6139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10357.887617 secs 2024-07-26T10:52:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:01 smithi181 ceph-mon[35869]: pgmap v6139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10357.887617 secs 2024-07-26T10:52:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:01 smithi097 ceph-mon[32231]: pgmap v6139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:02 smithi115 ceph-mon[35364]: pgmap v6140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:02 smithi181 ceph-mon[35869]: pgmap v6140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:02 smithi097 ceph-mon[32231]: pgmap v6140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:05 smithi115 ceph-mon[35364]: pgmap v6141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:05 smithi181 ceph-mon[35869]: pgmap v6141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:05 smithi097 ceph-mon[32231]: pgmap v6141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10362.887757 secs 2024-07-26T10:52:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10362.887757 secs 2024-07-26T10:52:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10362.887757 secs 2024-07-26T10:52:07.141 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:07.139+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:07.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:07.140+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:52:07.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:07.140+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:52:07.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:07.140+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:52:07.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:07.140+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:07 smithi115 ceph-mon[35364]: pgmap v6142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:07 smithi181 ceph-mon[35869]: pgmap v6142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:07 smithi097 ceph-mon[32231]: pgmap v6142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:08.713+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006240 con 0x7ff3f01061a0 2024-07-26T10:52:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:08.713+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:52:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:08.714+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:52:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:08.714+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006240 con 0x7ff3f01061a0 2024-07-26T10:52:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:09 smithi115 ceph-mon[35364]: pgmap v6143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:09 smithi181 ceph-mon[35869]: pgmap v6143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:09 smithi097 ceph-mon[32231]: pgmap v6143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10367.887899 secs 2024-07-26T10:52:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:11 smithi115 ceph-mon[35364]: pgmap v6144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10367.887899 secs 2024-07-26T10:52:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:11 smithi181 ceph-mon[35869]: pgmap v6144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10367.887899 secs 2024-07-26T10:52:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:11 smithi097 ceph-mon[32231]: pgmap v6144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:13 smithi115 ceph-mon[35364]: pgmap v6145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:13 smithi181 ceph-mon[35869]: pgmap v6145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:13 smithi097 ceph-mon[32231]: pgmap v6145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:14 smithi097 ceph-mon[32231]: pgmap v6146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:14 smithi115 ceph-mon[35364]: pgmap v6146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:14 smithi181 ceph-mon[35869]: pgmap v6146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10372.888037 secs 2024-07-26T10:52:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10372.888037 secs 2024-07-26T10:52:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10372.888037 secs 2024-07-26T10:52:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:16 smithi115 ceph-mon[35364]: pgmap v6147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:16 smithi181 ceph-mon[35869]: pgmap v6147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:16 smithi097 ceph-mon[32231]: pgmap v6147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:19.679 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:19 smithi115 ceph-mon[35364]: pgmap v6148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:19.687 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:19 smithi181 ceph-mon[35869]: pgmap v6148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:19.706 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:19 smithi097 ceph-mon[32231]: pgmap v6148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10377.888175 secs 2024-07-26T10:52:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:21 smithi115 ceph-mon[35364]: pgmap v6149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10377.888175 secs 2024-07-26T10:52:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:21 smithi181 ceph-mon[35869]: pgmap v6149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10377.888175 secs 2024-07-26T10:52:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:21 smithi097 ceph-mon[32231]: pgmap v6149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:22.142+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:22.143+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:52:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:22.143+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:52:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:22.143+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:52:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:22.143+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:23 smithi115 ceph-mon[35364]: pgmap v6150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:23 smithi181 ceph-mon[35869]: pgmap v6150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:23 smithi097 ceph-mon[32231]: pgmap v6150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:25 smithi115 ceph-mon[35364]: pgmap v6151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:25 smithi181 ceph-mon[35869]: pgmap v6151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:25 smithi097 ceph-mon[32231]: pgmap v6151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:52:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10382.888323 secs 2024-07-26T10:52:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10382.888323 secs 2024-07-26T10:52:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10382.888323 secs 2024-07-26T10:52:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:27 smithi115 ceph-mon[35364]: pgmap v6152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:27 smithi181 ceph-mon[35869]: pgmap v6152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:27 smithi097 ceph-mon[32231]: pgmap v6152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:28 smithi115 ceph-mon[35364]: pgmap v6153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:28 smithi181 ceph-mon[35869]: pgmap v6153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:28 smithi097 ceph-mon[32231]: pgmap v6153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10387.888460 secs 2024-07-26T10:52:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:31 smithi115 ceph-mon[35364]: pgmap v6154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10387.888460 secs 2024-07-26T10:52:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:31 smithi181 ceph-mon[35869]: pgmap v6154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10387.888460 secs 2024-07-26T10:52:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:31 smithi097 ceph-mon[32231]: pgmap v6154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:33 smithi115 ceph-mon[35364]: pgmap v6155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:33 smithi181 ceph-mon[35869]: pgmap v6155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:33 smithi097 ceph-mon[32231]: pgmap v6155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:34.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:52:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:52:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:34.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:52:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:35 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:52:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:35 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:52:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:35 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:52:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:35 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:52:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:35 smithi115 ceph-mon[35364]: pgmap v6156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:52:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:35 smithi115 ceph-mon[35364]: pgmap v6157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:35 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:52:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:35 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:52:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:35 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:52:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:35 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:52:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:35 smithi181 ceph-mon[35869]: pgmap v6156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:52:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:35 smithi181 ceph-mon[35869]: pgmap v6157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:35 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T10:52:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:35 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T10:52:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:35 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T10:52:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:35 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T10:52:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:35 smithi097 ceph-mon[32231]: pgmap v6156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:52:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:35 smithi097 ceph-mon[32231]: pgmap v6157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:52:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10392.888544 secs 2024-07-26T10:52:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10392.888544 secs 2024-07-26T10:52:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10392.888544 secs 2024-07-26T10:52:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:37.150+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:37.151+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:52:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:37.151+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:52:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:37.151+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:52:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:37.151+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:37 smithi115 ceph-mon[35364]: pgmap v6158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:37 smithi181 ceph-mon[35869]: pgmap v6158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:37 smithi097 ceph-mon[32231]: pgmap v6158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:38 smithi115 ceph-mon[35364]: pgmap v6159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:38 smithi181 ceph-mon[35869]: pgmap v6159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:38 smithi097 ceph-mon[32231]: pgmap v6159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:40 smithi115 ceph-mon[35364]: pgmap v6160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10397.888687 secs 2024-07-26T10:52:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:40 smithi181 ceph-mon[35869]: pgmap v6160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10397.888687 secs 2024-07-26T10:52:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:40 smithi097 ceph-mon[32231]: pgmap v6160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10397.888687 secs 2024-07-26T10:52:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:42 smithi115 ceph-mon[35364]: pgmap v6161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:42 smithi181 ceph-mon[35869]: pgmap v6161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:42 smithi097 ceph-mon[32231]: pgmap v6161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:44 smithi115 ceph-mon[35364]: pgmap v6162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:52:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:44 smithi181 ceph-mon[35869]: pgmap v6162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:52:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:44 smithi097 ceph-mon[32231]: pgmap v6162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:52:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10402.888846 secs 2024-07-26T10:52:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10402.888846 secs 2024-07-26T10:52:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10402.888846 secs 2024-07-26T10:52:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:46 smithi115 ceph-mon[35364]: pgmap v6163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:46 smithi181 ceph-mon[35869]: pgmap v6163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:46 smithi097 ceph-mon[32231]: pgmap v6163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:48 smithi115 ceph-mon[35364]: pgmap v6164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:48 smithi181 ceph-mon[35869]: pgmap v6164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:48 smithi097 ceph-mon[32231]: pgmap v6164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:50 smithi115 ceph-mon[35364]: pgmap v6165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10407.888988 secs 2024-07-26T10:52:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:50 smithi181 ceph-mon[35869]: pgmap v6165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10407.888988 secs 2024-07-26T10:52:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:50 smithi097 ceph-mon[32231]: pgmap v6165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10407.888988 secs 2024-07-26T10:52:52.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:52.152+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:52:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:52.153+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:52:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:52.153+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:52:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:52.153+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:52:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:52:52.154+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:52 smithi115 ceph-mon[35364]: pgmap v6166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:52 smithi181 ceph-mon[35869]: pgmap v6166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:52 smithi097 ceph-mon[32231]: pgmap v6166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:54 smithi115 ceph-mon[35364]: pgmap v6167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:54 smithi181 ceph-mon[35869]: pgmap v6167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:54 smithi097 ceph-mon[32231]: pgmap v6167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10412.889123 secs 2024-07-26T10:52:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10412.889123 secs 2024-07-26T10:52:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10412.889123 secs 2024-07-26T10:52:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:56 smithi115 ceph-mon[35364]: pgmap v6168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:56 smithi181 ceph-mon[35869]: pgmap v6168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:56 smithi097 ceph-mon[32231]: pgmap v6168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:52:58 smithi115 ceph-mon[35364]: pgmap v6169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:52:58 smithi181 ceph-mon[35869]: pgmap v6169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:52:58 smithi097 ceph-mon[32231]: pgmap v6169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:00 smithi115 ceph-mon[35364]: pgmap v6170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10417.889226 secs 2024-07-26T10:53:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:00 smithi181 ceph-mon[35869]: pgmap v6170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10417.889226 secs 2024-07-26T10:53:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:00 smithi097 ceph-mon[32231]: pgmap v6170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10417.889226 secs 2024-07-26T10:53:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:02 smithi115 ceph-mon[35364]: pgmap v6171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:02 smithi181 ceph-mon[35869]: pgmap v6171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:02 smithi097 ceph-mon[32231]: pgmap v6171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:04 smithi115 ceph-mon[35364]: pgmap v6172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:04 smithi181 ceph-mon[35869]: pgmap v6172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:04 smithi097 ceph-mon[32231]: pgmap v6172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10422.889385 secs 2024-07-26T10:53:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10422.889385 secs 2024-07-26T10:53:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10422.889385 secs 2024-07-26T10:53:07.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:07.161+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:07.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:07.162+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:53:07.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:07.162+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:53:07.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:07.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:53:07.165 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:07.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:06 smithi115 ceph-mon[35364]: pgmap v6173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:06 smithi181 ceph-mon[35869]: pgmap v6173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:06 smithi097 ceph-mon[32231]: pgmap v6173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:08 smithi115 ceph-mon[35364]: pgmap v6174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:09.258 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:08 smithi097 ceph-mon[32231]: pgmap v6174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:09.259 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:08 smithi181 ceph-mon[35869]: pgmap v6174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:10 smithi115 ceph-mon[35364]: pgmap v6175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10427.889523 secs 2024-07-26T10:53:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:10 smithi181 ceph-mon[35869]: pgmap v6175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10427.889523 secs 2024-07-26T10:53:11.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:10 smithi097 ceph-mon[32231]: pgmap v6175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:11.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10427.889523 secs 2024-07-26T10:53:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:12 smithi115 ceph-mon[35364]: pgmap v6176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:12 smithi181 ceph-mon[35869]: pgmap v6176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:12 smithi097 ceph-mon[32231]: pgmap v6176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:14 smithi181 ceph-mon[35869]: pgmap v6177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:14 smithi097 ceph-mon[32231]: pgmap v6177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:14 smithi115 ceph-mon[35364]: pgmap v6177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10432.889663 secs 2024-07-26T10:53:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10432.889663 secs 2024-07-26T10:53:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10432.889663 secs 2024-07-26T10:53:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:16 smithi097 ceph-mon[32231]: pgmap v6178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:16 smithi115 ceph-mon[35364]: pgmap v6178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:16 smithi181 ceph-mon[35869]: pgmap v6178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:19.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:19 smithi115 ceph-mon[35364]: pgmap v6179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:19.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:19 smithi181 ceph-mon[35869]: pgmap v6179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:19 smithi097 ceph-mon[32231]: pgmap v6179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:21 smithi115 ceph-mon[35364]: pgmap v6180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10437.889805 secs 2024-07-26T10:53:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:21 smithi181 ceph-mon[35869]: pgmap v6180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10437.889805 secs 2024-07-26T10:53:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:21 smithi097 ceph-mon[32231]: pgmap v6180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10437.889805 secs 2024-07-26T10:53:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:22.164+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:22.165+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:22.165+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:22.165+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:22.165+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:23 smithi115 ceph-mon[35364]: pgmap v6181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:23 smithi181 ceph-mon[35869]: pgmap v6181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:23 smithi097 ceph-mon[32231]: pgmap v6181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:25 smithi115 ceph-mon[35364]: pgmap v6182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:25 smithi181 ceph-mon[35869]: pgmap v6182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:25 smithi097 ceph-mon[32231]: pgmap v6182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10442.889946 secs 2024-07-26T10:53:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10442.889946 secs 2024-07-26T10:53:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10442.889946 secs 2024-07-26T10:53:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:27 smithi115 ceph-mon[35364]: pgmap v6183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:27 smithi181 ceph-mon[35869]: pgmap v6183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:27 smithi097 ceph-mon[32231]: pgmap v6183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:29 smithi115 ceph-mon[35364]: pgmap v6184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:29 smithi181 ceph-mon[35869]: pgmap v6184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:29 smithi097 ceph-mon[32231]: pgmap v6184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:31 smithi115 ceph-mon[35364]: pgmap v6185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10447.890103 secs 2024-07-26T10:53:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:31 smithi181 ceph-mon[35869]: pgmap v6185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10447.890103 secs 2024-07-26T10:53:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:31 smithi097 ceph-mon[32231]: pgmap v6185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10447.890103 secs 2024-07-26T10:53:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:33 smithi115 ceph-mon[35364]: pgmap v6186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:33 smithi181 ceph-mon[35869]: pgmap v6186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:33 smithi097 ceph-mon[32231]: pgmap v6186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:35 smithi115 ceph-mon[35364]: pgmap v6187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:35 smithi181 ceph-mon[35869]: pgmap v6187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:35 smithi097 ceph-mon[32231]: pgmap v6187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10452.890263 secs 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: pgmap v6188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: pgmap v6189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:36.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10452.890263 secs 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: pgmap v6188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: pgmap v6189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:53:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10452.890263 secs 2024-07-26T10:53:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: pgmap v6188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: pgmap v6189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:53:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:37.172+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:37.173+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:53:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:37.173+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:53:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:37.173+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:53:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:37.173+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:38 smithi115 ceph-mon[35364]: pgmap v6190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:53:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:38 smithi181 ceph-mon[35869]: pgmap v6190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:53:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:38 smithi097 ceph-mon[32231]: pgmap v6190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:53:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:40 smithi115 ceph-mon[35364]: pgmap v6191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:53:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:40 smithi181 ceph-mon[35869]: pgmap v6191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:53:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:40 smithi097 ceph-mon[32231]: pgmap v6191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:53:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10457.890304 secs 2024-07-26T10:53:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10457.890304 secs 2024-07-26T10:53:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10457.890304 secs 2024-07-26T10:53:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:42 smithi115 ceph-mon[35364]: pgmap v6192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:53:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:42 smithi181 ceph-mon[35869]: pgmap v6192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:53:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:42 smithi097 ceph-mon[32231]: pgmap v6192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:53:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:44 smithi115 ceph-mon[35364]: pgmap v6193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:44 smithi181 ceph-mon[35869]: pgmap v6193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:44 smithi097 ceph-mon[32231]: pgmap v6193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10462.890451 secs 2024-07-26T10:53:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:46 smithi115 ceph-mon[35364]: pgmap v6194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:53:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10462.890451 secs 2024-07-26T10:53:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:46 smithi181 ceph-mon[35869]: pgmap v6194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:53:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10462.890451 secs 2024-07-26T10:53:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:46 smithi097 ceph-mon[32231]: pgmap v6194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:53:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:48 smithi115 ceph-mon[35364]: pgmap v6195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:53:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:48 smithi181 ceph-mon[35869]: pgmap v6195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:53:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:48 smithi097 ceph-mon[32231]: pgmap v6195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:53:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:50 smithi115 ceph-mon[35364]: pgmap v6196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:53:50.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:50 smithi181 ceph-mon[35869]: pgmap v6196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:53:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:50 smithi097 ceph-mon[32231]: pgmap v6196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:53:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10467.890593 secs 2024-07-26T10:53:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10467.890593 secs 2024-07-26T10:53:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10467.890593 secs 2024-07-26T10:53:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:52.174+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:53:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:52.175+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:53:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:52.175+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:53:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:52.175+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:53:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:53:52.176+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:52 smithi115 ceph-mon[35364]: pgmap v6197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:52 smithi181 ceph-mon[35869]: pgmap v6197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:52 smithi097 ceph-mon[32231]: pgmap v6197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:54 smithi115 ceph-mon[35364]: pgmap v6198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:54 smithi181 ceph-mon[35869]: pgmap v6198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:54 smithi097 ceph-mon[32231]: pgmap v6198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10472.891493 secs 2024-07-26T10:53:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:56 smithi115 ceph-mon[35364]: pgmap v6199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10472.891493 secs 2024-07-26T10:53:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:56 smithi181 ceph-mon[35869]: pgmap v6199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10472.891493 secs 2024-07-26T10:53:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:56 smithi097 ceph-mon[32231]: pgmap v6199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:53:58 smithi115 ceph-mon[35364]: pgmap v6200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:53:58 smithi181 ceph-mon[35869]: pgmap v6200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:53:58 smithi097 ceph-mon[32231]: pgmap v6200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:00 smithi115 ceph-mon[35364]: pgmap v6201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:00 smithi181 ceph-mon[35869]: pgmap v6201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:00 smithi097 ceph-mon[32231]: pgmap v6201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10477.891654 secs 2024-07-26T10:54:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10477.891654 secs 2024-07-26T10:54:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10477.891654 secs 2024-07-26T10:54:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:02 smithi115 ceph-mon[35364]: pgmap v6202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:02 smithi181 ceph-mon[35869]: pgmap v6202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:02 smithi097 ceph-mon[32231]: pgmap v6202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:04 smithi115 ceph-mon[35364]: pgmap v6203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:04 smithi181 ceph-mon[35869]: pgmap v6203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:04 smithi097 ceph-mon[32231]: pgmap v6203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10482.891815 secs 2024-07-26T10:54:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:06 smithi115 ceph-mon[35364]: pgmap v6204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10482.891815 secs 2024-07-26T10:54:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:06 smithi181 ceph-mon[35869]: pgmap v6204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10482.891815 secs 2024-07-26T10:54:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:06 smithi097 ceph-mon[32231]: pgmap v6204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:07.185 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:07.183+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:07.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:07.184+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:54:07.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:07.184+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:54:07.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:07.184+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:54:07.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:07.184+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:08 smithi115 ceph-mon[35364]: pgmap v6205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:08 smithi181 ceph-mon[35869]: pgmap v6205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:08.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:08 smithi097 ceph-mon[32231]: pgmap v6205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:10 smithi115 ceph-mon[35364]: pgmap v6206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:10 smithi181 ceph-mon[35869]: pgmap v6206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:10 smithi097 ceph-mon[32231]: pgmap v6206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10487.891971 secs 2024-07-26T10:54:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10487.891971 secs 2024-07-26T10:54:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10487.891971 secs 2024-07-26T10:54:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:12 smithi115 ceph-mon[35364]: pgmap v6207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:12 smithi181 ceph-mon[35869]: pgmap v6207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:12 smithi097 ceph-mon[32231]: pgmap v6207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:13 smithi181 ceph-mon[35869]: pgmap v6208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:13 smithi097 ceph-mon[32231]: pgmap v6208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:13 smithi115 ceph-mon[35364]: pgmap v6208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10492.892127 secs 2024-07-26T10:54:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:16 smithi115 ceph-mon[35364]: pgmap v6209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10492.892127 secs 2024-07-26T10:54:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:16 smithi181 ceph-mon[35869]: pgmap v6209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10492.892127 secs 2024-07-26T10:54:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:16 smithi097 ceph-mon[32231]: pgmap v6209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:18 smithi115 ceph-mon[35364]: pgmap v6210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:18 smithi181 ceph-mon[35869]: pgmap v6210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:18 smithi097 ceph-mon[32231]: pgmap v6210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:19 smithi115 ceph-mon[35364]: pgmap v6211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:19 smithi181 ceph-mon[35869]: pgmap v6211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:19 smithi097 ceph-mon[32231]: pgmap v6211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10497.892289 secs 2024-07-26T10:54:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10497.892289 secs 2024-07-26T10:54:21.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10497.892289 secs 2024-07-26T10:54:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:22.185+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:22.187+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:54:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:22.187+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:54:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:22.187+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:54:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:22.187+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:21 smithi115 ceph-mon[35364]: pgmap v6212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:21 smithi181 ceph-mon[35869]: pgmap v6212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:21 smithi097 ceph-mon[32231]: pgmap v6212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:24 smithi115 ceph-mon[35364]: pgmap v6213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:24.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:24 smithi181 ceph-mon[35869]: pgmap v6213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:24 smithi097 ceph-mon[32231]: pgmap v6213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10502.892452 secs 2024-07-26T10:54:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:26 smithi115 ceph-mon[35364]: pgmap v6214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10502.892452 secs 2024-07-26T10:54:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:26 smithi181 ceph-mon[35869]: pgmap v6214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10502.892452 secs 2024-07-26T10:54:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:26 smithi097 ceph-mon[32231]: pgmap v6214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:28 smithi115 ceph-mon[35364]: pgmap v6215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:28 smithi181 ceph-mon[35869]: pgmap v6215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:28 smithi097 ceph-mon[32231]: pgmap v6215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:30 smithi115 ceph-mon[35364]: pgmap v6216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:30 smithi181 ceph-mon[35869]: pgmap v6216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:30 smithi097 ceph-mon[32231]: pgmap v6216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10507.892600 secs 2024-07-26T10:54:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10507.892600 secs 2024-07-26T10:54:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10507.892600 secs 2024-07-26T10:54:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:32 smithi115 ceph-mon[35364]: pgmap v6217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:32 smithi181 ceph-mon[35869]: pgmap v6217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:32 smithi097 ceph-mon[32231]: pgmap v6217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:34 smithi115 ceph-mon[35364]: pgmap v6218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:34 smithi181 ceph-mon[35869]: pgmap v6218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:34 smithi097 ceph-mon[32231]: pgmap v6218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10512.892763 secs 2024-07-26T10:54:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:36 smithi115 ceph-mon[35364]: pgmap v6219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10512.892763 secs 2024-07-26T10:54:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:36 smithi181 ceph-mon[35869]: pgmap v6219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10512.892763 secs 2024-07-26T10:54:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:36 smithi097 ceph-mon[32231]: pgmap v6219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:37.194+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:37.195+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:54:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:37.195+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:54:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:37.195+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:54:37.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:37.195+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:54:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:38 smithi115 ceph-mon[35364]: pgmap v6220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T10:54:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:38 smithi115 ceph-mon[35364]: pgmap v6221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:38 smithi181 ceph-mon[35869]: pgmap v6220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T10:54:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:38 smithi181 ceph-mon[35869]: pgmap v6221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:38 smithi097 ceph-mon[32231]: pgmap v6220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T10:54:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:38 smithi097 ceph-mon[32231]: pgmap v6221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:39 smithi115 ceph-mon[35364]: pgmap v6222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:39 smithi181 ceph-mon[35869]: pgmap v6222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:39 smithi097 ceph-mon[32231]: pgmap v6222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10517.892932 secs 2024-07-26T10:54:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10517.892932 secs 2024-07-26T10:54:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10517.892932 secs 2024-07-26T10:54:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:41 smithi115 ceph-mon[35364]: pgmap v6223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:41 smithi181 ceph-mon[35869]: pgmap v6223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:41 smithi097 ceph-mon[32231]: pgmap v6223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:44 smithi115 ceph-mon[35364]: pgmap v6224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:44 smithi181 ceph-mon[35869]: pgmap v6224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:44 smithi097 ceph-mon[32231]: pgmap v6224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10522.893061 secs 2024-07-26T10:54:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:46 smithi115 ceph-mon[35364]: pgmap v6225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10522.893061 secs 2024-07-26T10:54:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:46 smithi181 ceph-mon[35869]: pgmap v6225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10522.893061 secs 2024-07-26T10:54:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:46 smithi097 ceph-mon[32231]: pgmap v6225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:48.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:48 smithi115 ceph-mon[35364]: pgmap v6226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:48 smithi181 ceph-mon[35869]: pgmap v6226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:48 smithi097 ceph-mon[32231]: pgmap v6226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:54:50.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:50 smithi115 ceph-mon[35364]: pgmap v6227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:50.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:50 smithi181 ceph-mon[35869]: pgmap v6227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:50.556 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:50 smithi097 ceph-mon[32231]: pgmap v6227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10527.893202 secs 2024-07-26T10:54:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10527.893202 secs 2024-07-26T10:54:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10527.893202 secs 2024-07-26T10:54:52.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:52.196+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:54:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:52.198+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:54:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:52.198+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:54:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:52.198+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:54:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:54:52.198+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:52 smithi115 ceph-mon[35364]: pgmap v6228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:52 smithi181 ceph-mon[35869]: pgmap v6228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:52 smithi097 ceph-mon[32231]: pgmap v6228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:54 smithi115 ceph-mon[35364]: pgmap v6229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:54 smithi181 ceph-mon[35869]: pgmap v6229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:54 smithi097 ceph-mon[32231]: pgmap v6229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10532.893337 secs 2024-07-26T10:54:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:56 smithi115 ceph-mon[35364]: pgmap v6230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10532.893337 secs 2024-07-26T10:54:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:56 smithi181 ceph-mon[35869]: pgmap v6230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10532.893337 secs 2024-07-26T10:54:56.782 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:56 smithi097 ceph-mon[32231]: pgmap v6230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:54:58 smithi115 ceph-mon[35364]: pgmap v6231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:54:58 smithi181 ceph-mon[35869]: pgmap v6231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:54:58 smithi097 ceph-mon[32231]: pgmap v6231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:00 smithi115 ceph-mon[35364]: pgmap v6232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:00 smithi181 ceph-mon[35869]: pgmap v6232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:00 smithi097 ceph-mon[32231]: pgmap v6232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10537.893440 secs 2024-07-26T10:55:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10537.893440 secs 2024-07-26T10:55:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10537.893440 secs 2024-07-26T10:55:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:02 smithi115 ceph-mon[35364]: pgmap v6233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:02 smithi181 ceph-mon[35869]: pgmap v6233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:02 smithi097 ceph-mon[32231]: pgmap v6233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:04 smithi115 ceph-mon[35364]: pgmap v6234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:04 smithi181 ceph-mon[35869]: pgmap v6234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:04 smithi097 ceph-mon[32231]: pgmap v6234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10542.893603 secs 2024-07-26T10:55:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:06 smithi115 ceph-mon[35364]: pgmap v6235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:55:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10542.893603 secs 2024-07-26T10:55:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:06 smithi181 ceph-mon[35869]: pgmap v6235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:55:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10542.893603 secs 2024-07-26T10:55:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:06 smithi097 ceph-mon[32231]: pgmap v6235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:55:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:07.202+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:07.203+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:55:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:07.203+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:55:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:07.203+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:55:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:07.203+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:08 smithi115 ceph-mon[35364]: pgmap v6236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:08 smithi181 ceph-mon[35869]: pgmap v6236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:08.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:08.745+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006240 con 0x7ff3f01061a0 2024-07-26T10:55:08.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:08.745+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:55:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:08.745+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:55:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:08.745+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006240 con 0x7ff3f01061a0 2024-07-26T10:55:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:08 smithi097 ceph-mon[32231]: pgmap v6236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:10 smithi115 ceph-mon[35364]: pgmap v6237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:55:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:10 smithi181 ceph-mon[35869]: pgmap v6237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:55:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:10 smithi097 ceph-mon[32231]: pgmap v6237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:55:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10547.893744 secs 2024-07-26T10:55:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10547.893744 secs 2024-07-26T10:55:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10547.893744 secs 2024-07-26T10:55:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:12 smithi115 ceph-mon[35364]: pgmap v6238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:55:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:12 smithi181 ceph-mon[35869]: pgmap v6238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:55:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:12 smithi097 ceph-mon[32231]: pgmap v6238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:55:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:14 smithi115 ceph-mon[35364]: pgmap v6239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:14 smithi181 ceph-mon[35869]: pgmap v6239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:14 smithi097 ceph-mon[32231]: pgmap v6239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10552.893881 secs 2024-07-26T10:55:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:16 smithi115 ceph-mon[35364]: pgmap v6240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10552.893881 secs 2024-07-26T10:55:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:16 smithi181 ceph-mon[35869]: pgmap v6240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10552.893881 secs 2024-07-26T10:55:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:16 smithi097 ceph-mon[32231]: pgmap v6240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:18 smithi115 ceph-mon[35364]: pgmap v6241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:18 smithi181 ceph-mon[35869]: pgmap v6241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:18 smithi097 ceph-mon[32231]: pgmap v6241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:20 smithi115 ceph-mon[35364]: pgmap v6242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:20 smithi181 ceph-mon[35869]: pgmap v6242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:20 smithi097 ceph-mon[32231]: pgmap v6242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10557.894016 secs 2024-07-26T10:55:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10557.894016 secs 2024-07-26T10:55:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10557.894016 secs 2024-07-26T10:55:22.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:22.205+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:22.206+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:55:22.208 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:22.206+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:55:22.209 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:22.206+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:55:22.209 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:22.206+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:22 smithi115 ceph-mon[35364]: pgmap v6243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:22 smithi181 ceph-mon[35869]: pgmap v6243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:22 smithi097 ceph-mon[32231]: pgmap v6243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:24 smithi115 ceph-mon[35364]: pgmap v6244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:24 smithi181 ceph-mon[35869]: pgmap v6244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:24 smithi097 ceph-mon[32231]: pgmap v6244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10562.894156 secs 2024-07-26T10:55:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:26 smithi115 ceph-mon[35364]: pgmap v6245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10562.894156 secs 2024-07-26T10:55:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:26 smithi181 ceph-mon[35869]: pgmap v6245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10562.894156 secs 2024-07-26T10:55:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:26 smithi097 ceph-mon[32231]: pgmap v6245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:28 smithi115 ceph-mon[35364]: pgmap v6246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:28 smithi181 ceph-mon[35869]: pgmap v6246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:28 smithi097 ceph-mon[32231]: pgmap v6246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:30 smithi115 ceph-mon[35364]: pgmap v6247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:30 smithi181 ceph-mon[35869]: pgmap v6247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:30 smithi097 ceph-mon[32231]: pgmap v6247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10567.894312 secs 2024-07-26T10:55:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10567.894312 secs 2024-07-26T10:55:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10567.894312 secs 2024-07-26T10:55:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:32 smithi115 ceph-mon[35364]: pgmap v6248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:32 smithi181 ceph-mon[35869]: pgmap v6248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:32 smithi097 ceph-mon[32231]: pgmap v6248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:34 smithi115 ceph-mon[35364]: pgmap v6249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:34 smithi181 ceph-mon[35869]: pgmap v6249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:34 smithi097 ceph-mon[32231]: pgmap v6249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10572.894477 secs 2024-07-26T10:55:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:36 smithi181 ceph-mon[35869]: pgmap v6250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10572.894477 secs 2024-07-26T10:55:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:36 smithi097 ceph-mon[32231]: pgmap v6250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10572.894477 secs 2024-07-26T10:55:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:36 smithi115 ceph-mon[35364]: pgmap v6250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:37.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:37.213+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:37.214+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:55:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:37.214+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:55:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:37.214+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:55:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:37.214+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:37 smithi115 ceph-mon[35364]: pgmap v6251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:37 smithi181 ceph-mon[35869]: pgmap v6251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:37 smithi097 ceph-mon[32231]: pgmap v6251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:39 smithi115 ceph-mon[35364]: pgmap v6252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:39 smithi115 ceph-mon[35364]: pgmap v6253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:39 smithi181 ceph-mon[35869]: pgmap v6252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:39 smithi181 ceph-mon[35869]: pgmap v6253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:39 smithi097 ceph-mon[32231]: pgmap v6252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:39 smithi097 ceph-mon[32231]: pgmap v6253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:55:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10577.894574 secs 2024-07-26T10:55:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10577.894574 secs 2024-07-26T10:55:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10577.894574 secs 2024-07-26T10:55:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:41 smithi115 ceph-mon[35364]: pgmap v6254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:41 smithi181 ceph-mon[35869]: pgmap v6254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:41 smithi097 ceph-mon[32231]: pgmap v6254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:44 smithi115 ceph-mon[35364]: pgmap v6255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:44 smithi181 ceph-mon[35869]: pgmap v6255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:44 smithi097 ceph-mon[32231]: pgmap v6255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10582.894736 secs 2024-07-26T10:55:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:46 smithi115 ceph-mon[35364]: pgmap v6256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10582.894736 secs 2024-07-26T10:55:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:46 smithi181 ceph-mon[35869]: pgmap v6256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10582.894736 secs 2024-07-26T10:55:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:46 smithi097 ceph-mon[32231]: pgmap v6256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:48 smithi115 ceph-mon[35364]: pgmap v6257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:48 smithi181 ceph-mon[35869]: pgmap v6257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:48 smithi097 ceph-mon[32231]: pgmap v6257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:50 smithi115 ceph-mon[35364]: pgmap v6258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:50 smithi181 ceph-mon[35869]: pgmap v6258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:50 smithi097 ceph-mon[32231]: pgmap v6258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:55:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10587.894892 secs 2024-07-26T10:55:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10587.894892 secs 2024-07-26T10:55:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10587.894892 secs 2024-07-26T10:55:52.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:52.216+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:52.217+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:55:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:52.217+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:55:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:52.217+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:55:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:55:52.217+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:52 smithi115 ceph-mon[35364]: pgmap v6259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:52 smithi181 ceph-mon[35869]: pgmap v6259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:52 smithi097 ceph-mon[32231]: pgmap v6259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:54 smithi115 ceph-mon[35364]: pgmap v6260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:54 smithi181 ceph-mon[35869]: pgmap v6260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:54 smithi097 ceph-mon[32231]: pgmap v6260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10592.895048 secs 2024-07-26T10:55:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:56 smithi115 ceph-mon[35364]: pgmap v6261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10592.895048 secs 2024-07-26T10:55:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:56 smithi181 ceph-mon[35869]: pgmap v6261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10592.895048 secs 2024-07-26T10:55:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:56 smithi097 ceph-mon[32231]: pgmap v6261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:55:58 smithi115 ceph-mon[35364]: pgmap v6262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:55:58 smithi181 ceph-mon[35869]: pgmap v6262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:55:58 smithi097 ceph-mon[32231]: pgmap v6262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:00 smithi115 ceph-mon[35364]: pgmap v6263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:00 smithi181 ceph-mon[35869]: pgmap v6263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:00 smithi097 ceph-mon[32231]: pgmap v6263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10597.895211 secs 2024-07-26T10:56:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10597.895211 secs 2024-07-26T10:56:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10597.895211 secs 2024-07-26T10:56:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:02 smithi115 ceph-mon[35364]: pgmap v6264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:02 smithi181 ceph-mon[35869]: pgmap v6264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:02 smithi097 ceph-mon[32231]: pgmap v6264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:04 smithi115 ceph-mon[35364]: pgmap v6265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:56:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:04 smithi181 ceph-mon[35869]: pgmap v6265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:56:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:04 smithi097 ceph-mon[32231]: pgmap v6265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:56:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10602.895325 secs 2024-07-26T10:56:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:06 smithi115 ceph-mon[35364]: pgmap v6266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:56:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10602.895325 secs 2024-07-26T10:56:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:06 smithi181 ceph-mon[35869]: pgmap v6266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:56:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10602.895325 secs 2024-07-26T10:56:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:06 smithi097 ceph-mon[32231]: pgmap v6266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:56:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:07.225+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:07.226+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:56:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:07.226+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:56:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:07.226+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:56:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:07.226+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:08 smithi115 ceph-mon[35364]: pgmap v6267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:56:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:08 smithi181 ceph-mon[35869]: pgmap v6267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:56:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:08 smithi097 ceph-mon[32231]: pgmap v6267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:56:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:10 smithi115 ceph-mon[35364]: pgmap v6268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:56:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:10 smithi181 ceph-mon[35869]: pgmap v6268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:56:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:10 smithi097 ceph-mon[32231]: pgmap v6268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:56:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10607.895481 secs 2024-07-26T10:56:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10607.895481 secs 2024-07-26T10:56:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10607.895481 secs 2024-07-26T10:56:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:12 smithi115 ceph-mon[35364]: pgmap v6269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:12 smithi181 ceph-mon[35869]: pgmap v6269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:12 smithi097 ceph-mon[32231]: pgmap v6269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:14 smithi115 ceph-mon[35364]: pgmap v6270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:14 smithi181 ceph-mon[35869]: pgmap v6270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:14 smithi097 ceph-mon[32231]: pgmap v6270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10612.895652 secs 2024-07-26T10:56:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:16 smithi115 ceph-mon[35364]: pgmap v6271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10612.895652 secs 2024-07-26T10:56:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:16 smithi181 ceph-mon[35869]: pgmap v6271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10612.895652 secs 2024-07-26T10:56:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:16 smithi097 ceph-mon[32231]: pgmap v6271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:18 smithi115 ceph-mon[35364]: pgmap v6272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:18 smithi181 ceph-mon[35869]: pgmap v6272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:18 smithi097 ceph-mon[32231]: pgmap v6272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:19 smithi097 ceph-mon[32231]: pgmap v6273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:19 smithi115 ceph-mon[35364]: pgmap v6273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:19 smithi181 ceph-mon[35869]: pgmap v6273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10617.895805 secs 2024-07-26T10:56:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10617.895805 secs 2024-07-26T10:56:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10617.895805 secs 2024-07-26T10:56:22.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:22.228+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:22.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:22.229+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:56:22.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:22.229+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:56:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:22.229+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:56:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:22.229+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:22 smithi115 ceph-mon[35364]: pgmap v6274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:22 smithi181 ceph-mon[35869]: pgmap v6274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:22 smithi097 ceph-mon[32231]: pgmap v6274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:24 smithi115 ceph-mon[35364]: pgmap v6275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:24 smithi181 ceph-mon[35869]: pgmap v6275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:24 smithi097 ceph-mon[32231]: pgmap v6275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10622.895963 secs 2024-07-26T10:56:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:26 smithi115 ceph-mon[35364]: pgmap v6276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10622.895963 secs 2024-07-26T10:56:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:26 smithi181 ceph-mon[35869]: pgmap v6276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10622.895963 secs 2024-07-26T10:56:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:26 smithi097 ceph-mon[32231]: pgmap v6276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:28 smithi115 ceph-mon[35364]: pgmap v6277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:28 smithi181 ceph-mon[35869]: pgmap v6277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:28 smithi097 ceph-mon[32231]: pgmap v6277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:30 smithi115 ceph-mon[35364]: pgmap v6278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:30 smithi181 ceph-mon[35869]: pgmap v6278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:30 smithi097 ceph-mon[32231]: pgmap v6278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10627.896062 secs 2024-07-26T10:56:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10627.896062 secs 2024-07-26T10:56:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10627.896062 secs 2024-07-26T10:56:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:32 smithi115 ceph-mon[35364]: pgmap v6279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:32 smithi181 ceph-mon[35869]: pgmap v6279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:32 smithi097 ceph-mon[32231]: pgmap v6279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:34 smithi115 ceph-mon[35364]: pgmap v6280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:34 smithi181 ceph-mon[35869]: pgmap v6280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:34 smithi097 ceph-mon[32231]: pgmap v6280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10632.896267 secs 2024-07-26T10:56:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:36 smithi115 ceph-mon[35364]: pgmap v6281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10632.896267 secs 2024-07-26T10:56:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:36 smithi181 ceph-mon[35869]: pgmap v6281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10632.896267 secs 2024-07-26T10:56:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:36 smithi097 ceph-mon[32231]: pgmap v6281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:37.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:37.234+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:37.238 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:37.235+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:56:37.238 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:37.235+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:56:37.238 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:37.235+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:56:37.238 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:37.236+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:38 smithi115 ceph-mon[35364]: pgmap v6282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:38 smithi181 ceph-mon[35869]: pgmap v6282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:38 smithi097 ceph-mon[32231]: pgmap v6282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:40.671 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:40 smithi115 ceph-mon[35364]: pgmap v6283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:40.671 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:40.677 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:40 smithi097 ceph-mon[32231]: pgmap v6283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:40.678 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:40.680 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:40 smithi181 ceph-mon[35869]: pgmap v6283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:40.680 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10637.896362 secs 2024-07-26T10:56:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10637.896362 secs 2024-07-26T10:56:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10637.896362 secs 2024-07-26T10:56:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:42 smithi115 ceph-mon[35364]: pgmap v6284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:42 smithi181 ceph-mon[35869]: pgmap v6284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:42 smithi097 ceph-mon[32231]: pgmap v6284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:56:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:43 smithi115 ceph-mon[35364]: pgmap v6285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:43 smithi181 ceph-mon[35869]: pgmap v6285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:43 smithi097 ceph-mon[32231]: pgmap v6285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:44 smithi115 ceph-mon[35364]: pgmap v6286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:56:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:44 smithi115 ceph-mon[35364]: pgmap v6287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:56:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:44 smithi181 ceph-mon[35869]: pgmap v6286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:56:44.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:44 smithi181 ceph-mon[35869]: pgmap v6287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:56:44.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:44 smithi097 ceph-mon[32231]: pgmap v6286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:56:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:44 smithi097 ceph-mon[32231]: pgmap v6287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:56:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:56:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10642.896524 secs 2024-07-26T10:56:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10642.896524 secs 2024-07-26T10:56:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10642.896524 secs 2024-07-26T10:56:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:46 smithi115 ceph-mon[35364]: pgmap v6288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T10:56:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:46 smithi181 ceph-mon[35869]: pgmap v6288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T10:56:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:46 smithi097 ceph-mon[32231]: pgmap v6288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T10:56:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:48 smithi115 ceph-mon[35364]: pgmap v6289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:48 smithi181 ceph-mon[35869]: pgmap v6289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:48 smithi097 ceph-mon[32231]: pgmap v6289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10647.896682 secs 2024-07-26T10:56:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:50 smithi115 ceph-mon[35364]: pgmap v6290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10647.896682 secs 2024-07-26T10:56:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:50 smithi181 ceph-mon[35869]: pgmap v6290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10647.896682 secs 2024-07-26T10:56:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:50 smithi097 ceph-mon[32231]: pgmap v6290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:56:52.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:52.237+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:52.239+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:56:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:52.239+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:56:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:52.239+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:56:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:56:52.239+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:52 smithi115 ceph-mon[35364]: pgmap v6291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T10:56:53.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:52 smithi181 ceph-mon[35869]: pgmap v6291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T10:56:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:52 smithi097 ceph-mon[32231]: pgmap v6291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T10:56:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:54 smithi115 ceph-mon[35364]: pgmap v6292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:54 smithi181 ceph-mon[35869]: pgmap v6292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:54 smithi097 ceph-mon[32231]: pgmap v6292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10652.896829 secs 2024-07-26T10:56:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10652.896829 secs 2024-07-26T10:56:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10652.896829 secs 2024-07-26T10:56:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:56 smithi097 ceph-mon[32231]: pgmap v6293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:56 smithi115 ceph-mon[35364]: pgmap v6293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:56 smithi181 ceph-mon[35869]: pgmap v6293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:56:58 smithi097 ceph-mon[32231]: pgmap v6294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:56:58 smithi115 ceph-mon[35364]: pgmap v6294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:56:58 smithi181 ceph-mon[35869]: pgmap v6294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10657.896961 secs 2024-07-26T10:57:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:01 smithi115 ceph-mon[35364]: pgmap v6295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10657.896961 secs 2024-07-26T10:57:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:01 smithi181 ceph-mon[35869]: pgmap v6295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10657.896961 secs 2024-07-26T10:57:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:01 smithi097 ceph-mon[32231]: pgmap v6295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:03 smithi115 ceph-mon[35364]: pgmap v6296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:03 smithi181 ceph-mon[35869]: pgmap v6296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:03 smithi097 ceph-mon[32231]: pgmap v6296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:05 smithi115 ceph-mon[35364]: pgmap v6297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:05 smithi181 ceph-mon[35869]: pgmap v6297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:05 smithi097 ceph-mon[32231]: pgmap v6297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10662.897098 secs 2024-07-26T10:57:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10662.897098 secs 2024-07-26T10:57:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10662.897098 secs 2024-07-26T10:57:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:07.245+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:07.247+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:57:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:07.247+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:57:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:07.247+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:57:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:07.247+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:07 smithi115 ceph-mon[35364]: pgmap v6298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:07 smithi181 ceph-mon[35869]: pgmap v6298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:07 smithi097 ceph-mon[32231]: pgmap v6298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:09 smithi115 ceph-mon[35364]: pgmap v6299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:09 smithi181 ceph-mon[35869]: pgmap v6299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:09 smithi097 ceph-mon[32231]: pgmap v6299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10667.897286 secs 2024-07-26T10:57:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:11 smithi115 ceph-mon[35364]: pgmap v6300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10667.897286 secs 2024-07-26T10:57:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:11 smithi181 ceph-mon[35869]: pgmap v6300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10667.897286 secs 2024-07-26T10:57:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:11 smithi097 ceph-mon[32231]: pgmap v6300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:13 smithi115 ceph-mon[35364]: pgmap v6301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:13 smithi181 ceph-mon[35869]: pgmap v6301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:13 smithi097 ceph-mon[32231]: pgmap v6301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:15 smithi115 ceph-mon[35364]: pgmap v6302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:15 smithi181 ceph-mon[35869]: pgmap v6302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:15 smithi097 ceph-mon[32231]: pgmap v6302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10672.897430 secs 2024-07-26T10:57:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10672.897430 secs 2024-07-26T10:57:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10672.897430 secs 2024-07-26T10:57:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:17 smithi115 ceph-mon[35364]: pgmap v6303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:17 smithi181 ceph-mon[35869]: pgmap v6303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:17 smithi097 ceph-mon[32231]: pgmap v6303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:19 smithi115 ceph-mon[35364]: pgmap v6304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:19 smithi181 ceph-mon[35869]: pgmap v6304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:19 smithi097 ceph-mon[32231]: pgmap v6304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10677.897563 secs 2024-07-26T10:57:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:21 smithi115 ceph-mon[35364]: pgmap v6305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10677.897563 secs 2024-07-26T10:57:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:21 smithi181 ceph-mon[35869]: pgmap v6305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10677.897563 secs 2024-07-26T10:57:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:21 smithi097 ceph-mon[32231]: pgmap v6305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:22.249+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:22.250+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:57:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:22.250+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:57:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:22.250+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:57:22.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:22.250+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:23 smithi115 ceph-mon[35364]: pgmap v6306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:23 smithi181 ceph-mon[35869]: pgmap v6306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:23 smithi097 ceph-mon[32231]: pgmap v6306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:25 smithi115 ceph-mon[35364]: pgmap v6307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:25.568 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:25 smithi097 ceph-mon[32231]: pgmap v6307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:25.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:25 smithi181 ceph-mon[35869]: pgmap v6307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10682.897690 secs 2024-07-26T10:57:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10682.897690 secs 2024-07-26T10:57:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10682.897690 secs 2024-07-26T10:57:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:27 smithi115 ceph-mon[35364]: pgmap v6308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:27 smithi181 ceph-mon[35869]: pgmap v6308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:27 smithi097 ceph-mon[32231]: pgmap v6308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:29 smithi115 ceph-mon[35364]: pgmap v6309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:29 smithi181 ceph-mon[35869]: pgmap v6309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:29 smithi097 ceph-mon[32231]: pgmap v6309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10687.897823 secs 2024-07-26T10:57:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:31 smithi115 ceph-mon[35364]: pgmap v6310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10687.897823 secs 2024-07-26T10:57:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:31 smithi181 ceph-mon[35869]: pgmap v6310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10687.897823 secs 2024-07-26T10:57:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:31 smithi097 ceph-mon[32231]: pgmap v6310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:33 smithi115 ceph-mon[35364]: pgmap v6311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:33 smithi181 ceph-mon[35869]: pgmap v6311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:33.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:33 smithi097 ceph-mon[32231]: pgmap v6311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:35 smithi115 ceph-mon[35364]: pgmap v6312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:35 smithi181 ceph-mon[35869]: pgmap v6312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:35 smithi097 ceph-mon[32231]: pgmap v6312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10692.897954 secs 2024-07-26T10:57:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10692.897954 secs 2024-07-26T10:57:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10692.897954 secs 2024-07-26T10:57:37.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:37.256+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:37.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:37.257+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:57:37.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:37.257+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:57:37.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:37.257+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:57:37.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:37.257+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:37 smithi115 ceph-mon[35364]: pgmap v6313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:37 smithi181 ceph-mon[35869]: pgmap v6313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:37 smithi097 ceph-mon[32231]: pgmap v6313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:39 smithi115 ceph-mon[35364]: pgmap v6314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:39 smithi181 ceph-mon[35869]: pgmap v6314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:39 smithi097 ceph-mon[32231]: pgmap v6314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10697.898090 secs 2024-07-26T10:57:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:41 smithi115 ceph-mon[35364]: pgmap v6315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10697.898090 secs 2024-07-26T10:57:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:41 smithi181 ceph-mon[35869]: pgmap v6315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10697.898090 secs 2024-07-26T10:57:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:41 smithi097 ceph-mon[32231]: pgmap v6315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:43 smithi115 ceph-mon[35364]: pgmap v6316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:43 smithi181 ceph-mon[35869]: pgmap v6316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:43 smithi097 ceph-mon[32231]: pgmap v6316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:45 smithi115 ceph-mon[35364]: pgmap v6317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:45 smithi181 ceph-mon[35869]: pgmap v6317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:45 smithi097 ceph-mon[32231]: pgmap v6317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10702.898222 secs 2024-07-26T10:57:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:46.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:46 smithi115 ceph-mon[35364]: pgmap v6318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:46.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:46 smithi115 ceph-mon[35364]: pgmap v6319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:57:46.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:57:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10702.898222 secs 2024-07-26T10:57:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:46.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:46 smithi181 ceph-mon[35869]: pgmap v6318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:46.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:46 smithi181 ceph-mon[35869]: pgmap v6319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:57:46.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:57:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10702.898222 secs 2024-07-26T10:57:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:46 smithi097 ceph-mon[32231]: pgmap v6318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:46 smithi097 ceph-mon[32231]: pgmap v6319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:57:46.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:57:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:48 smithi115 ceph-mon[35364]: pgmap v6320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:57:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:48 smithi181 ceph-mon[35869]: pgmap v6320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:57:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:48 smithi097 ceph-mon[32231]: pgmap v6320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:57:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:50 smithi115 ceph-mon[35364]: pgmap v6321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:57:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:50 smithi181 ceph-mon[35869]: pgmap v6321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:57:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:50 smithi097 ceph-mon[32231]: pgmap v6321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:57:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10707.898349 secs 2024-07-26T10:57:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10707.898349 secs 2024-07-26T10:57:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10707.898349 secs 2024-07-26T10:57:52.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:52.259+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:57:52.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:52.261+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:57:52.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:52.261+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:57:52.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:52.261+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:57:52.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:57:52.261+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:52 smithi115 ceph-mon[35364]: pgmap v6322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:57:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:52 smithi181 ceph-mon[35869]: pgmap v6322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:57:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:52 smithi097 ceph-mon[32231]: pgmap v6322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:57:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:54 smithi115 ceph-mon[35364]: pgmap v6323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:57:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:54 smithi181 ceph-mon[35869]: pgmap v6323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:57:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:54 smithi097 ceph-mon[32231]: pgmap v6323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:57:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10712.898488 secs 2024-07-26T10:57:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:56 smithi115 ceph-mon[35364]: pgmap v6324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:57:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10712.898488 secs 2024-07-26T10:57:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:56 smithi181 ceph-mon[35869]: pgmap v6324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:57:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10712.898488 secs 2024-07-26T10:57:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:56 smithi097 ceph-mon[32231]: pgmap v6324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:57:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:57:58 smithi115 ceph-mon[35364]: pgmap v6325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:57:58 smithi181 ceph-mon[35869]: pgmap v6325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:57:58 smithi097 ceph-mon[32231]: pgmap v6325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:00 smithi115 ceph-mon[35364]: pgmap v6326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:00 smithi181 ceph-mon[35869]: pgmap v6326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:00 smithi097 ceph-mon[32231]: pgmap v6326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10717.898588 secs 2024-07-26T10:58:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10717.898588 secs 2024-07-26T10:58:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10717.898588 secs 2024-07-26T10:58:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:02 smithi115 ceph-mon[35364]: pgmap v6327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:02 smithi181 ceph-mon[35869]: pgmap v6327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:02 smithi097 ceph-mon[32231]: pgmap v6327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:04 smithi181 ceph-mon[35869]: pgmap v6328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:04 smithi097 ceph-mon[32231]: pgmap v6328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:04 smithi115 ceph-mon[35364]: pgmap v6328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10722.898680 secs 2024-07-26T10:58:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:06 smithi181 ceph-mon[35869]: pgmap v6329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10722.898680 secs 2024-07-26T10:58:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:06 smithi097 ceph-mon[32231]: pgmap v6329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:06.912+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000754) for more than 900000000 us, fault. 2024-07-26T10:58:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:06.912+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057c0 tx=0x7ff3e007a000 comp rx=0 tx=0).stop 2024-07-26T10:58:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:06.912+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b53b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:58:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:06.912+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b53b0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:58:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:06.912+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:58:06.915 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:06.913+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:06.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:06.914+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e0082000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:58:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10722.898680 secs 2024-07-26T10:58:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:06 smithi115 ceph-mon[35364]: pgmap v6329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:07.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:07.262+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:07.263+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:58:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:07.263+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:58:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:07.263+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:58:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:07.263+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:08 smithi097 ceph-mon[32231]: pgmap v6330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:08.776+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:58:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:08.776+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T10:58:08.779 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:08.777+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T10:58:08.780 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:08.777+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:58:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:08 smithi115 ceph-mon[35364]: pgmap v6330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:08 smithi181 ceph-mon[35869]: pgmap v6330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:10 smithi097 ceph-mon[32231]: pgmap v6331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:10 smithi115 ceph-mon[35364]: pgmap v6331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:10 smithi181 ceph-mon[35869]: pgmap v6331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10727.898789 secs 2024-07-26T10:58:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10727.898789 secs 2024-07-26T10:58:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10727.898789 secs 2024-07-26T10:58:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:12 smithi115 ceph-mon[35364]: pgmap v6332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:12 smithi181 ceph-mon[35869]: pgmap v6332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:12 smithi097 ceph-mon[32231]: pgmap v6332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:13 smithi115 ceph-mon[35364]: pgmap v6333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:13 smithi181 ceph-mon[35869]: pgmap v6333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:13 smithi097 ceph-mon[32231]: pgmap v6333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10732.898909 secs 2024-07-26T10:58:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:16 smithi115 ceph-mon[35364]: pgmap v6334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10732.898909 secs 2024-07-26T10:58:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:16 smithi181 ceph-mon[35869]: pgmap v6334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10732.898909 secs 2024-07-26T10:58:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:16 smithi097 ceph-mon[32231]: pgmap v6334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:18 smithi115 ceph-mon[35364]: pgmap v6335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:18 smithi181 ceph-mon[35869]: pgmap v6335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:18 smithi097 ceph-mon[32231]: pgmap v6335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:20 smithi115 ceph-mon[35364]: pgmap v6336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:20 smithi181 ceph-mon[35869]: pgmap v6336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:20 smithi097 ceph-mon[32231]: pgmap v6336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10737.899035 secs 2024-07-26T10:58:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10737.899035 secs 2024-07-26T10:58:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10737.899035 secs 2024-07-26T10:58:22.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:22.265+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:22.265+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:58:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:22.265+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:58:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:22.265+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:58:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:22.265+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:22 smithi115 ceph-mon[35364]: pgmap v6337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:22 smithi181 ceph-mon[35869]: pgmap v6337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:22 smithi097 ceph-mon[32231]: pgmap v6337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:24 smithi115 ceph-mon[35364]: pgmap v6338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:24 smithi181 ceph-mon[35869]: pgmap v6338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:24 smithi097 ceph-mon[32231]: pgmap v6338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10742.899198 secs 2024-07-26T10:58:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:26 smithi115 ceph-mon[35364]: pgmap v6339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10742.899198 secs 2024-07-26T10:58:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:26 smithi181 ceph-mon[35869]: pgmap v6339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10742.899198 secs 2024-07-26T10:58:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:26 smithi097 ceph-mon[32231]: pgmap v6339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:28 smithi115 ceph-mon[35364]: pgmap v6340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:28 smithi181 ceph-mon[35869]: pgmap v6340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:28 smithi097 ceph-mon[32231]: pgmap v6340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:30 smithi115 ceph-mon[35364]: pgmap v6341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:30 smithi181 ceph-mon[35869]: pgmap v6341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:30 smithi097 ceph-mon[32231]: pgmap v6341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10747.899355 secs 2024-07-26T10:58:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10747.899355 secs 2024-07-26T10:58:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10747.899355 secs 2024-07-26T10:58:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:32 smithi097 ceph-mon[32231]: pgmap v6342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:32 smithi115 ceph-mon[35364]: pgmap v6342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:32 smithi181 ceph-mon[35869]: pgmap v6342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:34 smithi097 ceph-mon[32231]: pgmap v6343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:34 smithi115 ceph-mon[35364]: pgmap v6343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:34 smithi181 ceph-mon[35869]: pgmap v6343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10752.899494 secs 2024-07-26T10:58:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:36 smithi115 ceph-mon[35364]: pgmap v6344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10752.899494 secs 2024-07-26T10:58:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:36 smithi181 ceph-mon[35869]: pgmap v6344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10752.899494 secs 2024-07-26T10:58:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:36 smithi097 ceph-mon[32231]: pgmap v6344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:37.271+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:37.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:37.273+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:58:37.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:37.273+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:58:37.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:37.273+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:58:37.275 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:37.273+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:38 smithi115 ceph-mon[35364]: pgmap v6345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:38 smithi181 ceph-mon[35869]: pgmap v6345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:38 smithi097 ceph-mon[32231]: pgmap v6345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:39 smithi115 ceph-mon[35364]: pgmap v6346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:39 smithi181 ceph-mon[35869]: pgmap v6346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:39 smithi097 ceph-mon[32231]: pgmap v6346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10757.899597 secs 2024-07-26T10:58:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10757.899597 secs 2024-07-26T10:58:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10757.899597 secs 2024-07-26T10:58:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:41 smithi115 ceph-mon[35364]: pgmap v6347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:41 smithi181 ceph-mon[35869]: pgmap v6347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:41 smithi097 ceph-mon[32231]: pgmap v6347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:44 smithi115 ceph-mon[35364]: pgmap v6348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:44 smithi181 ceph-mon[35869]: pgmap v6348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:44 smithi097 ceph-mon[32231]: pgmap v6348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10762.899695 secs 2024-07-26T10:58:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:46 smithi115 ceph-mon[35364]: pgmap v6349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10762.899695 secs 2024-07-26T10:58:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:46 smithi181 ceph-mon[35869]: pgmap v6349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10762.899695 secs 2024-07-26T10:58:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:46 smithi097 ceph-mon[32231]: pgmap v6349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:58:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:58:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:58:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:48 smithi115 ceph-mon[35364]: pgmap v6350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:58:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:48 smithi115 ceph-mon[35364]: pgmap v6351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:58:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:48 smithi181 ceph-mon[35869]: pgmap v6350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:58:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:48 smithi181 ceph-mon[35869]: pgmap v6351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:58:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:48 smithi097 ceph-mon[32231]: pgmap v6350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:58:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:48 smithi097 ceph-mon[32231]: pgmap v6351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:58:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:50 smithi115 ceph-mon[35364]: pgmap v6352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:58:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:50 smithi181 ceph-mon[35869]: pgmap v6352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:58:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:50 smithi097 ceph-mon[32231]: pgmap v6352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T10:58:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10767.899834 secs 2024-07-26T10:58:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10767.899834 secs 2024-07-26T10:58:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10767.899834 secs 2024-07-26T10:58:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:52.275+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:58:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:52.277+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:58:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:52.277+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:58:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:52.277+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:58:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:58:52.277+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:52 smithi181 ceph-mon[35869]: pgmap v6353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:52 smithi097 ceph-mon[32231]: pgmap v6353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:52 smithi115 ceph-mon[35364]: pgmap v6353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:54 smithi097 ceph-mon[32231]: pgmap v6354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:54 smithi115 ceph-mon[35364]: pgmap v6354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:54 smithi181 ceph-mon[35869]: pgmap v6354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:56 smithi097 ceph-mon[32231]: pgmap v6355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10772.899988 secs 2024-07-26T10:58:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:56 smithi115 ceph-mon[35364]: pgmap v6355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10772.899988 secs 2024-07-26T10:58:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:56 smithi181 ceph-mon[35869]: pgmap v6355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10772.899988 secs 2024-07-26T10:58:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:58:58 smithi097 ceph-mon[32231]: pgmap v6356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:58:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:58:58 smithi115 ceph-mon[35364]: pgmap v6356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:58:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:58:58 smithi181 ceph-mon[35869]: pgmap v6356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:59:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:00 smithi115 ceph-mon[35364]: pgmap v6357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:00 smithi181 ceph-mon[35869]: pgmap v6357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:00 smithi097 ceph-mon[32231]: pgmap v6357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10777.900130 secs 2024-07-26T10:59:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10777.900130 secs 2024-07-26T10:59:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10777.900130 secs 2024-07-26T10:59:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:02 smithi115 ceph-mon[35364]: pgmap v6358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:59:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:02 smithi181 ceph-mon[35869]: pgmap v6358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:59:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:02 smithi097 ceph-mon[32231]: pgmap v6358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:59:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:03 smithi115 ceph-mon[35364]: pgmap v6359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:03 smithi181 ceph-mon[35869]: pgmap v6359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:03 smithi097 ceph-mon[32231]: pgmap v6359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:05 smithi115 ceph-mon[35364]: pgmap v6360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10782.900273 secs 2024-07-26T10:59:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:05 smithi181 ceph-mon[35869]: pgmap v6360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10782.900273 secs 2024-07-26T10:59:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:05 smithi097 ceph-mon[32231]: pgmap v6360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10782.900273 secs 2024-07-26T10:59:07.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:07.283+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:07.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:07.285+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:59:07.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:07.285+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:59:07.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:07.285+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:59:07.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:07.285+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:07 smithi115 ceph-mon[35364]: pgmap v6361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:07 smithi181 ceph-mon[35869]: pgmap v6361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:07 smithi097 ceph-mon[32231]: pgmap v6361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:09 smithi115 ceph-mon[35364]: pgmap v6362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:09 smithi181 ceph-mon[35869]: pgmap v6362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:09 smithi097 ceph-mon[32231]: pgmap v6362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10787.900439 secs 2024-07-26T10:59:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10787.900439 secs 2024-07-26T10:59:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10787.900439 secs 2024-07-26T10:59:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:11 smithi115 ceph-mon[35364]: pgmap v6363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:11 smithi181 ceph-mon[35869]: pgmap v6363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:11 smithi097 ceph-mon[32231]: pgmap v6363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:13 smithi115 ceph-mon[35364]: pgmap v6364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:13 smithi181 ceph-mon[35869]: pgmap v6364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:13 smithi097 ceph-mon[32231]: pgmap v6364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:15 smithi115 ceph-mon[35364]: pgmap v6365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10792.900582 secs 2024-07-26T10:59:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:15 smithi181 ceph-mon[35869]: pgmap v6365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10792.900582 secs 2024-07-26T10:59:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:15 smithi097 ceph-mon[32231]: pgmap v6365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10792.900582 secs 2024-07-26T10:59:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:17 smithi115 ceph-mon[35364]: pgmap v6366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:17 smithi181 ceph-mon[35869]: pgmap v6366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:17 smithi097 ceph-mon[32231]: pgmap v6366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:19 smithi115 ceph-mon[35364]: pgmap v6367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:19 smithi181 ceph-mon[35869]: pgmap v6367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:19 smithi097 ceph-mon[32231]: pgmap v6367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:21.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10797.900684 secs 2024-07-26T10:59:21.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10797.900684 secs 2024-07-26T10:59:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10797.900684 secs 2024-07-26T10:59:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:21 smithi115 ceph-mon[35364]: pgmap v6368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:21 smithi181 ceph-mon[35869]: pgmap v6368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:21 smithi097 ceph-mon[32231]: pgmap v6368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:22.286+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:22.287+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:59:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:22.287+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:59:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:22.287+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:59:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:22.288+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:23 smithi115 ceph-mon[35364]: pgmap v6369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:23 smithi181 ceph-mon[35869]: pgmap v6369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:23 smithi097 ceph-mon[32231]: pgmap v6369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:25 smithi115 ceph-mon[35364]: pgmap v6370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10802.900819 secs 2024-07-26T10:59:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:25 smithi181 ceph-mon[35869]: pgmap v6370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10802.900819 secs 2024-07-26T10:59:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:25 smithi097 ceph-mon[32231]: pgmap v6370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10802.900819 secs 2024-07-26T10:59:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:27 smithi115 ceph-mon[35364]: pgmap v6371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:27 smithi181 ceph-mon[35869]: pgmap v6371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:27 smithi097 ceph-mon[32231]: pgmap v6371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:29 smithi115 ceph-mon[35364]: pgmap v6372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:29 smithi181 ceph-mon[35869]: pgmap v6372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:29 smithi097 ceph-mon[32231]: pgmap v6372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10807.900969 secs 2024-07-26T10:59:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10807.900969 secs 2024-07-26T10:59:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10807.900969 secs 2024-07-26T10:59:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:31 smithi115 ceph-mon[35364]: pgmap v6373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:31 smithi181 ceph-mon[35869]: pgmap v6373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:31 smithi097 ceph-mon[32231]: pgmap v6373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:33 smithi115 ceph-mon[35364]: pgmap v6374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:33 smithi181 ceph-mon[35869]: pgmap v6374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:33 smithi097 ceph-mon[32231]: pgmap v6374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:35 smithi115 ceph-mon[35364]: pgmap v6375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10812.901111 secs 2024-07-26T10:59:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:35 smithi181 ceph-mon[35869]: pgmap v6375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10812.901111 secs 2024-07-26T10:59:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:35 smithi097 ceph-mon[32231]: pgmap v6375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10812.901111 secs 2024-07-26T10:59:37.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:37.295+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:37.296+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:59:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:37.296+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:59:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:37.296+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:59:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:37.296+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:37 smithi115 ceph-mon[35364]: pgmap v6376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:37 smithi181 ceph-mon[35869]: pgmap v6376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:37 smithi097 ceph-mon[32231]: pgmap v6376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:38.793+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:59:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:38.793+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T10:59:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:38.794+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T10:59:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:38.794+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T10:59:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:39 smithi115 ceph-mon[35364]: pgmap v6377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:39 smithi181 ceph-mon[35869]: pgmap v6377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:39 smithi097 ceph-mon[32231]: pgmap v6377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10817.901289 secs 2024-07-26T10:59:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10817.901289 secs 2024-07-26T10:59:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10817.901289 secs 2024-07-26T10:59:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:41 smithi181 ceph-mon[35869]: pgmap v6378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:41 smithi097 ceph-mon[32231]: pgmap v6378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:41 smithi115 ceph-mon[35364]: pgmap v6378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:43 smithi181 ceph-mon[35869]: pgmap v6379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:43 smithi097 ceph-mon[32231]: pgmap v6379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:43 smithi115 ceph-mon[35364]: pgmap v6379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:45 smithi097 ceph-mon[32231]: pgmap v6380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10822.901451 secs 2024-07-26T10:59:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:45 smithi115 ceph-mon[35364]: pgmap v6380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10822.901451 secs 2024-07-26T10:59:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:45 smithi181 ceph-mon[35869]: pgmap v6380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10822.901451 secs 2024-07-26T10:59:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:47 smithi097 ceph-mon[32231]: pgmap v6381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:47 smithi115 ceph-mon[35364]: pgmap v6381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:47 smithi181 ceph-mon[35869]: pgmap v6381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:49 smithi115 ceph-mon[35364]: pgmap v6382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:59:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:49 smithi115 ceph-mon[35364]: pgmap v6383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:59:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:49 smithi181 ceph-mon[35869]: pgmap v6382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:59:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:49 smithi181 ceph-mon[35869]: pgmap v6383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:59:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:49 smithi097 ceph-mon[32231]: pgmap v6382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T10:59:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:49 smithi097 ceph-mon[32231]: pgmap v6383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T10:59:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10827.901589 secs 2024-07-26T10:59:51.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:51 smithi115 ceph-mon[35364]: pgmap v6384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10827.901589 secs 2024-07-26T10:59:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:51 smithi181 ceph-mon[35869]: pgmap v6384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10827.901589 secs 2024-07-26T10:59:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:51 smithi097 ceph-mon[32231]: pgmap v6384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:52.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:52.297+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:59:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:52.299+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T10:59:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:52.299+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T10:59:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:52.299+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:59:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T10:59:52.299+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:53 smithi115 ceph-mon[35364]: pgmap v6385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:59:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:53 smithi181 ceph-mon[35869]: pgmap v6385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:59:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:53 smithi097 ceph-mon[32231]: pgmap v6385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:59:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:55 smithi115 ceph-mon[35364]: pgmap v6386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:55 smithi181 ceph-mon[35869]: pgmap v6386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:55 smithi097 ceph-mon[32231]: pgmap v6386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10832.901729 secs 2024-07-26T10:59:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10832.901729 secs 2024-07-26T10:59:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10832.901729 secs 2024-07-26T10:59:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:57 smithi115 ceph-mon[35364]: pgmap v6387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:57 smithi181 ceph-mon[35869]: pgmap v6387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:57 smithi097 ceph-mon[32231]: pgmap v6387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:59.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 10:59:59 smithi115 ceph-mon[35364]: pgmap v6388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:59:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 10:59:59 smithi181 ceph-mon[35869]: pgmap v6388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:59:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 10:59:59 smithi097 ceph-mon[32231]: pgmap v6388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:00:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:00:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:00:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:00:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10837.901874 secs 2024-07-26T11:00:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:01 smithi115 ceph-mon[35364]: pgmap v6389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10837.901874 secs 2024-07-26T11:00:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:01 smithi181 ceph-mon[35869]: pgmap v6389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10837.901874 secs 2024-07-26T11:00:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:01 smithi097 ceph-mon[32231]: pgmap v6389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:03 smithi115 ceph-mon[35364]: pgmap v6390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:03 smithi181 ceph-mon[35869]: pgmap v6390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:03 smithi097 ceph-mon[32231]: pgmap v6390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:05 smithi115 ceph-mon[35364]: pgmap v6391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:05 smithi181 ceph-mon[35869]: pgmap v6391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:05 smithi097 ceph-mon[32231]: pgmap v6391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10842.902027 secs 2024-07-26T11:00:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10842.902027 secs 2024-07-26T11:00:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10842.902027 secs 2024-07-26T11:00:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:07.305+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:07.308+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:00:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:07.308+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:00:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:07.308+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:00:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:07.308+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:07 smithi115 ceph-mon[35364]: pgmap v6392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:07 smithi181 ceph-mon[35869]: pgmap v6392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:07 smithi097 ceph-mon[32231]: pgmap v6392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:09 smithi115 ceph-mon[35364]: pgmap v6393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:09 smithi181 ceph-mon[35869]: pgmap v6393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:09 smithi097 ceph-mon[32231]: pgmap v6393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10847.902190 secs 2024-07-26T11:00:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:11 smithi115 ceph-mon[35364]: pgmap v6394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10847.902190 secs 2024-07-26T11:00:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:11 smithi181 ceph-mon[35869]: pgmap v6394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10847.902190 secs 2024-07-26T11:00:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:11 smithi097 ceph-mon[32231]: pgmap v6394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:13 smithi115 ceph-mon[35364]: pgmap v6395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:13 smithi181 ceph-mon[35869]: pgmap v6395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:13 smithi097 ceph-mon[32231]: pgmap v6395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:15 smithi115 ceph-mon[35364]: pgmap v6396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:15 smithi181 ceph-mon[35869]: pgmap v6396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:15 smithi097 ceph-mon[32231]: pgmap v6396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10852.902332 secs 2024-07-26T11:00:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10852.902332 secs 2024-07-26T11:00:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10852.902332 secs 2024-07-26T11:00:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:17 smithi115 ceph-mon[35364]: pgmap v6397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:17 smithi181 ceph-mon[35869]: pgmap v6397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:17 smithi097 ceph-mon[32231]: pgmap v6397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:19 smithi115 ceph-mon[35364]: pgmap v6398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:19 smithi181 ceph-mon[35869]: pgmap v6398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:19 smithi097 ceph-mon[32231]: pgmap v6398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10857.902469 secs 2024-07-26T11:00:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:21 smithi115 ceph-mon[35364]: pgmap v6399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10857.902469 secs 2024-07-26T11:00:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:21 smithi181 ceph-mon[35869]: pgmap v6399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10857.902469 secs 2024-07-26T11:00:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:21 smithi097 ceph-mon[32231]: pgmap v6399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:22.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:22.309+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:22.311+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:22.311+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:22.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:00:22.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:22.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:23 smithi097 ceph-mon[32231]: pgmap v6400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:23 smithi115 ceph-mon[35364]: pgmap v6400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:23 smithi181 ceph-mon[35869]: pgmap v6400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:25 smithi115 ceph-mon[35364]: pgmap v6401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:25 smithi181 ceph-mon[35869]: pgmap v6401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:25 smithi097 ceph-mon[32231]: pgmap v6401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:00:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10862.902605 secs 2024-07-26T11:00:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10862.902605 secs 2024-07-26T11:00:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10862.902605 secs 2024-07-26T11:00:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:27 smithi115 ceph-mon[35364]: pgmap v6402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:27 smithi181 ceph-mon[35869]: pgmap v6402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:27 smithi097 ceph-mon[32231]: pgmap v6402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:29 smithi115 ceph-mon[35364]: pgmap v6403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:29 smithi181 ceph-mon[35869]: pgmap v6403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:29 smithi097 ceph-mon[32231]: pgmap v6403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10867.902702 secs 2024-07-26T11:00:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:31 smithi115 ceph-mon[35364]: pgmap v6404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10867.902702 secs 2024-07-26T11:00:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:31 smithi181 ceph-mon[35869]: pgmap v6404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10867.902702 secs 2024-07-26T11:00:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:31 smithi097 ceph-mon[32231]: pgmap v6404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:33 smithi115 ceph-mon[35364]: pgmap v6405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:33 smithi181 ceph-mon[35869]: pgmap v6405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:33 smithi097 ceph-mon[32231]: pgmap v6405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:35 smithi115 ceph-mon[35364]: pgmap v6406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:35 smithi181 ceph-mon[35869]: pgmap v6406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:35.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:35 smithi097 ceph-mon[32231]: pgmap v6406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10872.902843 secs 2024-07-26T11:00:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10872.902843 secs 2024-07-26T11:00:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10872.902843 secs 2024-07-26T11:00:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:37.317+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:37.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:37.319+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:00:37.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:37.319+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:00:37.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:37.319+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:00:37.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:37.319+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:37 smithi181 ceph-mon[35869]: pgmap v6407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:00:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:37 smithi097 ceph-mon[32231]: pgmap v6407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:00:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:37 smithi115 ceph-mon[35364]: pgmap v6407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:00:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:39 smithi097 ceph-mon[32231]: pgmap v6408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:00:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:39 smithi115 ceph-mon[35364]: pgmap v6408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:00:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:39 smithi181 ceph-mon[35869]: pgmap v6408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:00:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10877.902943 secs 2024-07-26T11:00:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:41 smithi097 ceph-mon[32231]: pgmap v6409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:00:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10877.902943 secs 2024-07-26T11:00:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:41 smithi115 ceph-mon[35364]: pgmap v6409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:00:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10877.902943 secs 2024-07-26T11:00:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:41 smithi181 ceph-mon[35869]: pgmap v6409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:00:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:43 smithi097 ceph-mon[32231]: pgmap v6410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:00:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:43 smithi115 ceph-mon[35364]: pgmap v6410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:00:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:43 smithi181 ceph-mon[35869]: pgmap v6410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:00:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:45 smithi115 ceph-mon[35364]: pgmap v6411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:45.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:45 smithi181 ceph-mon[35869]: pgmap v6411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:45 smithi097 ceph-mon[32231]: pgmap v6411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10882.903078 secs 2024-07-26T11:00:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10882.903078 secs 2024-07-26T11:00:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10882.903078 secs 2024-07-26T11:00:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:47 smithi115 ceph-mon[35364]: pgmap v6412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:47 smithi181 ceph-mon[35869]: pgmap v6412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:47 smithi097 ceph-mon[32231]: pgmap v6412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:49 smithi115 ceph-mon[35364]: pgmap v6413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:49 smithi181 ceph-mon[35869]: pgmap v6413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:49 smithi097 ceph-mon[32231]: pgmap v6413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:00:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:00:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:00:51.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:51 smithi115 ceph-mon[35364]: pgmap v6414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:51.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:51 smithi115 ceph-mon[35364]: pgmap v6415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:51.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10887.903208 secs 2024-07-26T11:00:51.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:51 smithi181 ceph-mon[35869]: pgmap v6414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:51.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:51 smithi181 ceph-mon[35869]: pgmap v6415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:51.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10887.903208 secs 2024-07-26T11:00:51.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:51 smithi097 ceph-mon[32231]: pgmap v6414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:00:51.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:51 smithi097 ceph-mon[32231]: pgmap v6415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:51.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10887.903208 secs 2024-07-26T11:00:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:52.321+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:00:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:52.322+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:00:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:52.322+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:00:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:52.322+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:00:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:00:52.323+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:53 smithi115 ceph-mon[35364]: pgmap v6416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:53 smithi181 ceph-mon[35869]: pgmap v6416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:53 smithi097 ceph-mon[32231]: pgmap v6416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:54 smithi115 ceph-mon[35364]: pgmap v6417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:54 smithi181 ceph-mon[35869]: pgmap v6417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:54 smithi097 ceph-mon[32231]: pgmap v6417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:00:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10892.903348 secs 2024-07-26T11:00:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10892.903348 secs 2024-07-26T11:00:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10892.903348 secs 2024-07-26T11:00:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:56 smithi097 ceph-mon[32231]: pgmap v6418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:56 smithi115 ceph-mon[35364]: pgmap v6418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:56 smithi181 ceph-mon[35869]: pgmap v6418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:00:58 smithi097 ceph-mon[32231]: pgmap v6419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:00:58 smithi115 ceph-mon[35364]: pgmap v6419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:00:58 smithi181 ceph-mon[35869]: pgmap v6419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:00 smithi097 ceph-mon[32231]: pgmap v6420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:01:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10897.903494 secs 2024-07-26T11:01:01.119 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:00 smithi115 ceph-mon[35364]: pgmap v6420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:01:01.119 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10897.903494 secs 2024-07-26T11:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:00 smithi181 ceph-mon[35869]: pgmap v6420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:01:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10897.903494 secs 2024-07-26T11:01:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:02 smithi115 ceph-mon[35364]: pgmap v6421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:02 smithi181 ceph-mon[35869]: pgmap v6421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:02 smithi097 ceph-mon[32231]: pgmap v6421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:04 smithi115 ceph-mon[35364]: pgmap v6422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:04 smithi181 ceph-mon[35869]: pgmap v6422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:04 smithi097 ceph-mon[32231]: pgmap v6422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10902.903640 secs 2024-07-26T11:01:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10902.903640 secs 2024-07-26T11:01:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10902.903640 secs 2024-07-26T11:01:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:06 smithi115 ceph-mon[35364]: pgmap v6423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:06 smithi181 ceph-mon[35869]: pgmap v6423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:06 smithi097 ceph-mon[32231]: pgmap v6423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:07.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:07.328+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:01:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:07.328+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:01:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:07.328+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:01:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:07.328+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:08.811+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:01:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:08.811+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:01:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:08.812+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:01:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:08.812+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:01:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:08 smithi115 ceph-mon[35364]: pgmap v6424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:08 smithi181 ceph-mon[35869]: pgmap v6424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:08 smithi097 ceph-mon[32231]: pgmap v6424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:10 smithi115 ceph-mon[35364]: pgmap v6425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10907.903788 secs 2024-07-26T11:01:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:10 smithi181 ceph-mon[35869]: pgmap v6425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10907.903788 secs 2024-07-26T11:01:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:10 smithi097 ceph-mon[32231]: pgmap v6425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10907.903788 secs 2024-07-26T11:01:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:12 smithi115 ceph-mon[35364]: pgmap v6426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:01:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:12 smithi181 ceph-mon[35869]: pgmap v6426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:01:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:12 smithi097 ceph-mon[32231]: pgmap v6426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:01:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:14 smithi115 ceph-mon[35364]: pgmap v6427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:14 smithi181 ceph-mon[35869]: pgmap v6427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:14 smithi097 ceph-mon[32231]: pgmap v6427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10912.903948 secs 2024-07-26T11:01:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10912.903948 secs 2024-07-26T11:01:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10912.903948 secs 2024-07-26T11:01:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:16 smithi115 ceph-mon[35364]: pgmap v6428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:01:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:16 smithi181 ceph-mon[35869]: pgmap v6428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:01:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:16 smithi097 ceph-mon[32231]: pgmap v6428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:01:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:18 smithi115 ceph-mon[35364]: pgmap v6429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:01:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:18 smithi181 ceph-mon[35869]: pgmap v6429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:01:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:18 smithi097 ceph-mon[32231]: pgmap v6429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:01:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:20 smithi115 ceph-mon[35364]: pgmap v6430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10917.904113 secs 2024-07-26T11:01:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:20 smithi181 ceph-mon[35869]: pgmap v6430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10917.904113 secs 2024-07-26T11:01:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:20 smithi097 ceph-mon[32231]: pgmap v6430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:01:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10917.904113 secs 2024-07-26T11:01:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:22.329+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:22.331+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:01:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:22.331+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:01:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:22.331+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:01:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:22.331+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:22 smithi115 ceph-mon[35364]: pgmap v6431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:22 smithi181 ceph-mon[35869]: pgmap v6431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:22 smithi097 ceph-mon[32231]: pgmap v6431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:24 smithi115 ceph-mon[35364]: pgmap v6432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:24 smithi181 ceph-mon[35869]: pgmap v6432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:24 smithi097 ceph-mon[32231]: pgmap v6432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10922.904281 secs 2024-07-26T11:01:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10922.904281 secs 2024-07-26T11:01:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10922.904281 secs 2024-07-26T11:01:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:26 smithi115 ceph-mon[35364]: pgmap v6433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:26 smithi181 ceph-mon[35869]: pgmap v6433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:26 smithi097 ceph-mon[32231]: pgmap v6433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:28 smithi115 ceph-mon[35364]: pgmap v6434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:28 smithi181 ceph-mon[35869]: pgmap v6434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:28 smithi097 ceph-mon[32231]: pgmap v6434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:30 smithi115 ceph-mon[35364]: pgmap v6435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10927.904432 secs 2024-07-26T11:01:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:30 smithi181 ceph-mon[35869]: pgmap v6435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10927.904432 secs 2024-07-26T11:01:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:30 smithi097 ceph-mon[32231]: pgmap v6435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10927.904432 secs 2024-07-26T11:01:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:32 smithi115 ceph-mon[35364]: pgmap v6436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:32 smithi181 ceph-mon[35869]: pgmap v6436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:32 smithi097 ceph-mon[32231]: pgmap v6436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:34 smithi115 ceph-mon[35364]: pgmap v6437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:34 smithi181 ceph-mon[35869]: pgmap v6437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:34 smithi097 ceph-mon[32231]: pgmap v6437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10932.904529 secs 2024-07-26T11:01:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10932.904529 secs 2024-07-26T11:01:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10932.904529 secs 2024-07-26T11:01:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:36 smithi115 ceph-mon[35364]: pgmap v6438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:36 smithi181 ceph-mon[35869]: pgmap v6438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:36 smithi097 ceph-mon[32231]: pgmap v6438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:37.338+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:37.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:37.339+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:01:37.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:37.340+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:01:37.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:37.340+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:01:37.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:37.340+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:38 smithi115 ceph-mon[35364]: pgmap v6439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:38 smithi181 ceph-mon[35869]: pgmap v6439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:39.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:38 smithi097 ceph-mon[32231]: pgmap v6439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:40 smithi115 ceph-mon[35364]: pgmap v6440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10937.904671 secs 2024-07-26T11:01:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:40 smithi181 ceph-mon[35869]: pgmap v6440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10937.904671 secs 2024-07-26T11:01:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:40 smithi097 ceph-mon[32231]: pgmap v6440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10937.904671 secs 2024-07-26T11:01:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:42 smithi115 ceph-mon[35364]: pgmap v6441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:42 smithi181 ceph-mon[35869]: pgmap v6441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:42 smithi097 ceph-mon[32231]: pgmap v6441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:44 smithi115 ceph-mon[35364]: pgmap v6442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:44 smithi181 ceph-mon[35869]: pgmap v6442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:44 smithi097 ceph-mon[32231]: pgmap v6442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10942.904817 secs 2024-07-26T11:01:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10942.904817 secs 2024-07-26T11:01:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10942.904817 secs 2024-07-26T11:01:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:46 smithi115 ceph-mon[35364]: pgmap v6443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:46 smithi181 ceph-mon[35869]: pgmap v6443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:46 smithi097 ceph-mon[32231]: pgmap v6443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:48 smithi181 ceph-mon[35869]: pgmap v6444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:48 smithi097 ceph-mon[32231]: pgmap v6444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:48 smithi115 ceph-mon[35364]: pgmap v6444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:50 smithi181 ceph-mon[35869]: pgmap v6445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10947.904986 secs 2024-07-26T11:01:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:50 smithi097 ceph-mon[32231]: pgmap v6445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10947.904986 secs 2024-07-26T11:01:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:51.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:50 smithi115 ceph-mon[35364]: pgmap v6445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:51.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10947.904986 secs 2024-07-26T11:01:51.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:51 smithi097 ceph-mon[32231]: pgmap v6446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:01:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:51 smithi097 ceph-mon[32231]: pgmap v6447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:01:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:52.342+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:01:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:52.343+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:01:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:52.343+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:01:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:52.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:01:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:01:52.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:51 smithi115 ceph-mon[35364]: pgmap v6446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:01:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:51 smithi115 ceph-mon[35364]: pgmap v6447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:52.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:01:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:51 smithi181 ceph-mon[35869]: pgmap v6446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:01:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:51 smithi181 ceph-mon[35869]: pgmap v6447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:01:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:54 smithi115 ceph-mon[35364]: pgmap v6448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:01:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:54 smithi181 ceph-mon[35869]: pgmap v6448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:01:54.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:54 smithi097 ceph-mon[32231]: pgmap v6448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:01:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10952.905152 secs 2024-07-26T11:01:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:56 smithi115 ceph-mon[35364]: pgmap v6449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:01:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10952.905152 secs 2024-07-26T11:01:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:56 smithi181 ceph-mon[35869]: pgmap v6449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:01:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10952.905152 secs 2024-07-26T11:01:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:56 smithi097 ceph-mon[32231]: pgmap v6449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:01:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:01:58 smithi115 ceph-mon[35364]: pgmap v6450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:01:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:01:58 smithi181 ceph-mon[35869]: pgmap v6450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:01:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:01:58 smithi097 ceph-mon[32231]: pgmap v6450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:02:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:00 smithi115 ceph-mon[35364]: pgmap v6451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:00 smithi181 ceph-mon[35869]: pgmap v6451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:00 smithi097 ceph-mon[32231]: pgmap v6451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10957.905306 secs 2024-07-26T11:02:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10957.905306 secs 2024-07-26T11:02:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10957.905306 secs 2024-07-26T11:02:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:02 smithi115 ceph-mon[35364]: pgmap v6452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:02:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:02 smithi181 ceph-mon[35869]: pgmap v6452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:02:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:02 smithi097 ceph-mon[32231]: pgmap v6452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:02:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:04 smithi115 ceph-mon[35364]: pgmap v6453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:04 smithi181 ceph-mon[35869]: pgmap v6453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:04 smithi097 ceph-mon[32231]: pgmap v6453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10962.905437 secs 2024-07-26T11:02:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:06 smithi115 ceph-mon[35364]: pgmap v6454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10962.905437 secs 2024-07-26T11:02:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:06 smithi181 ceph-mon[35869]: pgmap v6454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10962.905437 secs 2024-07-26T11:02:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:06 smithi097 ceph-mon[32231]: pgmap v6454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:07.349+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:07.351+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:02:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:07.351+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:02:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:07.351+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:02:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:07.351+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:08 smithi115 ceph-mon[35364]: pgmap v6455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:02:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:08 smithi181 ceph-mon[35869]: pgmap v6455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:02:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:08 smithi097 ceph-mon[32231]: pgmap v6455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:02:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:10 smithi115 ceph-mon[35364]: pgmap v6456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:02:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:10 smithi181 ceph-mon[35869]: pgmap v6456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:02:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:10 smithi097 ceph-mon[32231]: pgmap v6456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:02:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10967.905607 secs 2024-07-26T11:02:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10967.905607 secs 2024-07-26T11:02:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10967.905607 secs 2024-07-26T11:02:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:12 smithi115 ceph-mon[35364]: pgmap v6457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:12 smithi181 ceph-mon[35869]: pgmap v6457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:12 smithi097 ceph-mon[32231]: pgmap v6457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:14 smithi115 ceph-mon[35364]: pgmap v6458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:02:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:14 smithi181 ceph-mon[35869]: pgmap v6458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:02:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:14 smithi097 ceph-mon[32231]: pgmap v6458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:02:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10972.905753 secs 2024-07-26T11:02:16.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:16 smithi181 ceph-mon[35869]: pgmap v6459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10972.905753 secs 2024-07-26T11:02:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:16 smithi097 ceph-mon[32231]: pgmap v6459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10972.905753 secs 2024-07-26T11:02:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:16 smithi115 ceph-mon[35364]: pgmap v6459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:18 smithi097 ceph-mon[32231]: pgmap v6460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:18 smithi115 ceph-mon[35364]: pgmap v6460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:18 smithi181 ceph-mon[35869]: pgmap v6460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:20 smithi097 ceph-mon[32231]: pgmap v6461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:20 smithi115 ceph-mon[35364]: pgmap v6461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:20 smithi181 ceph-mon[35869]: pgmap v6461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10977.905838 secs 2024-07-26T11:02:21.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10977.905838 secs 2024-07-26T11:02:21.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10977.905838 secs 2024-07-26T11:02:22.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:22.353+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:22.354+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:02:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:22.354+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:02:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:22.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:02:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:22.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:22.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:22 smithi115 ceph-mon[35364]: pgmap v6462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:22 smithi181 ceph-mon[35869]: pgmap v6462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:22 smithi097 ceph-mon[32231]: pgmap v6462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:24 smithi115 ceph-mon[35364]: pgmap v6463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:24 smithi181 ceph-mon[35869]: pgmap v6463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:24 smithi097 ceph-mon[32231]: pgmap v6463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10982.905984 secs 2024-07-26T11:02:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:26 smithi115 ceph-mon[35364]: pgmap v6464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10982.905984 secs 2024-07-26T11:02:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:26 smithi181 ceph-mon[35869]: pgmap v6464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10982.905984 secs 2024-07-26T11:02:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:26 smithi097 ceph-mon[32231]: pgmap v6464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:28 smithi115 ceph-mon[35364]: pgmap v6465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:28 smithi181 ceph-mon[35869]: pgmap v6465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:28 smithi097 ceph-mon[32231]: pgmap v6465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:30 smithi115 ceph-mon[35364]: pgmap v6466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:30 smithi181 ceph-mon[35869]: pgmap v6466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:30 smithi097 ceph-mon[32231]: pgmap v6466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10987.906147 secs 2024-07-26T11:02:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10987.906147 secs 2024-07-26T11:02:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10987.906147 secs 2024-07-26T11:02:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:32 smithi115 ceph-mon[35364]: pgmap v6467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:32 smithi181 ceph-mon[35869]: pgmap v6467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:32 smithi097 ceph-mon[32231]: pgmap v6467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:34 smithi115 ceph-mon[35364]: pgmap v6468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:34 smithi181 ceph-mon[35869]: pgmap v6468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:34 smithi097 ceph-mon[32231]: pgmap v6468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10992.906319 secs 2024-07-26T11:02:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:36 smithi115 ceph-mon[35364]: pgmap v6469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10992.906319 secs 2024-07-26T11:02:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:36 smithi181 ceph-mon[35869]: pgmap v6469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10992.906319 secs 2024-07-26T11:02:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:36 smithi097 ceph-mon[32231]: pgmap v6469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:37.361+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:37.362+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:02:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:37.362+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:02:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:37.362+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:02:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:37.362+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:38 smithi115 ceph-mon[35364]: pgmap v6470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:38 smithi181 ceph-mon[35869]: pgmap v6470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:38 smithi097 ceph-mon[32231]: pgmap v6470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:40 smithi115 ceph-mon[35364]: pgmap v6471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:40 smithi181 ceph-mon[35869]: pgmap v6471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:40 smithi097 ceph-mon[32231]: pgmap v6471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 10997.906461 secs 2024-07-26T11:02:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 10997.906461 secs 2024-07-26T11:02:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 10997.906461 secs 2024-07-26T11:02:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:42 smithi115 ceph-mon[35364]: pgmap v6472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:42 smithi181 ceph-mon[35869]: pgmap v6472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:42 smithi097 ceph-mon[32231]: pgmap v6472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:44 smithi115 ceph-mon[35364]: pgmap v6473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:44 smithi181 ceph-mon[35869]: pgmap v6473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:44 smithi097 ceph-mon[32231]: pgmap v6473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11002.906599 secs 2024-07-26T11:02:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:46 smithi115 ceph-mon[35364]: pgmap v6474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11002.906599 secs 2024-07-26T11:02:46.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:46 smithi181 ceph-mon[35869]: pgmap v6474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11002.906599 secs 2024-07-26T11:02:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:46 smithi097 ceph-mon[32231]: pgmap v6474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:48 smithi115 ceph-mon[35364]: pgmap v6475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:48 smithi181 ceph-mon[35869]: pgmap v6475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:48 smithi097 ceph-mon[32231]: pgmap v6475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:50 smithi115 ceph-mon[35364]: pgmap v6476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:50 smithi181 ceph-mon[35869]: pgmap v6476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:50 smithi097 ceph-mon[32231]: pgmap v6476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:02:51.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11007.906695 secs 2024-07-26T11:02:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11007.906695 secs 2024-07-26T11:02:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11007.906695 secs 2024-07-26T11:02:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:52.364+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:52.365+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:02:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:52.365+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:02:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:52.365+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:02:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:02:52.365+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:52 smithi115 ceph-mon[35364]: pgmap v6477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:52 smithi181 ceph-mon[35869]: pgmap v6477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:52 smithi097 ceph-mon[32231]: pgmap v6477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:02:53.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:02:53.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:02:53.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:02:53.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:53.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:53.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: pgmap v6478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:02:53.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: pgmap v6479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:53.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:02:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:02:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:02:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:02:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: pgmap v6478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:02:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: pgmap v6479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:02:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:53.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: pgmap v6478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:02:53.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: pgmap v6479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:53.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:02:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:54 smithi115 ceph-mon[35364]: pgmap v6480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:54 smithi181 ceph-mon[35869]: pgmap v6480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:54 smithi097 ceph-mon[32231]: pgmap v6480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:02:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11012.906837 secs 2024-07-26T11:02:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11012.906837 secs 2024-07-26T11:02:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11012.906837 secs 2024-07-26T11:02:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:56 smithi115 ceph-mon[35364]: pgmap v6481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:56 smithi181 ceph-mon[35869]: pgmap v6481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:56 smithi097 ceph-mon[32231]: pgmap v6481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:02:58 smithi115 ceph-mon[35364]: pgmap v6482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:02:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:02:58 smithi181 ceph-mon[35869]: pgmap v6482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:02:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:02:58 smithi097 ceph-mon[32231]: pgmap v6482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:03:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:00 smithi115 ceph-mon[35364]: pgmap v6483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:00 smithi181 ceph-mon[35869]: pgmap v6483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:00 smithi097 ceph-mon[32231]: pgmap v6483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11017.906980 secs 2024-07-26T11:03:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11017.906980 secs 2024-07-26T11:03:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11017.906980 secs 2024-07-26T11:03:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:02 smithi115 ceph-mon[35364]: pgmap v6484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:03:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:02 smithi181 ceph-mon[35869]: pgmap v6484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:03:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:02 smithi097 ceph-mon[32231]: pgmap v6484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:03:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:04 smithi115 ceph-mon[35364]: pgmap v6485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:04 smithi181 ceph-mon[35869]: pgmap v6485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:04 smithi097 ceph-mon[32231]: pgmap v6485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11022.907114 secs 2024-07-26T11:03:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11022.907114 secs 2024-07-26T11:03:06.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11022.907114 secs 2024-07-26T11:03:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:06 smithi115 ceph-mon[35364]: pgmap v6486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:06 smithi181 ceph-mon[35869]: pgmap v6486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:06 smithi097 ceph-mon[32231]: pgmap v6486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:07.373 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:07.371+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:07.372+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:03:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:07.372+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:03:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:07.372+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:03:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:07.372+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:08 smithi115 ceph-mon[35364]: pgmap v6487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:08 smithi181 ceph-mon[35869]: pgmap v6487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:08 smithi097 ceph-mon[32231]: pgmap v6487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:10 smithi115 ceph-mon[35364]: pgmap v6488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:10 smithi181 ceph-mon[35869]: pgmap v6488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:10 smithi097 ceph-mon[32231]: pgmap v6488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11027.907284 secs 2024-07-26T11:03:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11027.907284 secs 2024-07-26T11:03:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11027.907284 secs 2024-07-26T11:03:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:12 smithi115 ceph-mon[35364]: pgmap v6489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:12 smithi181 ceph-mon[35869]: pgmap v6489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:12 smithi097 ceph-mon[32231]: pgmap v6489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:14 smithi115 ceph-mon[35364]: pgmap v6490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:14 smithi181 ceph-mon[35869]: pgmap v6490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:14 smithi097 ceph-mon[32231]: pgmap v6490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11032.907445 secs 2024-07-26T11:03:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11032.907445 secs 2024-07-26T11:03:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11032.907445 secs 2024-07-26T11:03:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:16 smithi115 ceph-mon[35364]: pgmap v6491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:16 smithi181 ceph-mon[35869]: pgmap v6491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:16 smithi097 ceph-mon[32231]: pgmap v6491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:18 smithi115 ceph-mon[35364]: pgmap v6492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:18 smithi181 ceph-mon[35869]: pgmap v6492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:18 smithi097 ceph-mon[32231]: pgmap v6492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:20 smithi115 ceph-mon[35364]: pgmap v6493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:20 smithi181 ceph-mon[35869]: pgmap v6493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:20 smithi097 ceph-mon[32231]: pgmap v6493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:21.879 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11037.907585 secs 2024-07-26T11:03:21.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11037.907585 secs 2024-07-26T11:03:21.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11037.907585 secs 2024-07-26T11:03:22.376 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:22.374+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:22.375+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:03:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:22.375+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:03:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:22.375+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:03:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:22.375+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:22 smithi115 ceph-mon[35364]: pgmap v6494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:22 smithi181 ceph-mon[35869]: pgmap v6494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:22 smithi097 ceph-mon[32231]: pgmap v6494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:24 smithi115 ceph-mon[35364]: pgmap v6495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:24 smithi181 ceph-mon[35869]: pgmap v6495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:24 smithi097 ceph-mon[32231]: pgmap v6495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11042.907743 secs 2024-07-26T11:03:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11042.907743 secs 2024-07-26T11:03:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11042.907743 secs 2024-07-26T11:03:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:26 smithi115 ceph-mon[35364]: pgmap v6496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:26 smithi181 ceph-mon[35869]: pgmap v6496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:26 smithi097 ceph-mon[32231]: pgmap v6496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:28 smithi115 ceph-mon[35364]: pgmap v6497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:28 smithi181 ceph-mon[35869]: pgmap v6497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:28 smithi097 ceph-mon[32231]: pgmap v6497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:30 smithi115 ceph-mon[35364]: pgmap v6498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:30 smithi181 ceph-mon[35869]: pgmap v6498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:30 smithi097 ceph-mon[32231]: pgmap v6498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11047.907876 secs 2024-07-26T11:03:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11047.907876 secs 2024-07-26T11:03:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11047.907876 secs 2024-07-26T11:03:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:32 smithi115 ceph-mon[35364]: pgmap v6499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:32 smithi181 ceph-mon[35869]: pgmap v6499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:32 smithi097 ceph-mon[32231]: pgmap v6499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:34 smithi115 ceph-mon[35364]: pgmap v6500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:34 smithi181 ceph-mon[35869]: pgmap v6500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:34 smithi097 ceph-mon[32231]: pgmap v6500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11052.908020 secs 2024-07-26T11:03:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11052.908020 secs 2024-07-26T11:03:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11052.908020 secs 2024-07-26T11:03:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:36 smithi115 ceph-mon[35364]: pgmap v6501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:36 smithi181 ceph-mon[35869]: pgmap v6501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:36 smithi097 ceph-mon[32231]: pgmap v6501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:37.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:37.381+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:37.383+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:03:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:37.383+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:03:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:37.383+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:03:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:37.383+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:38 smithi115 ceph-mon[35364]: pgmap v6502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:38 smithi181 ceph-mon[35869]: pgmap v6502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:38 smithi097 ceph-mon[32231]: pgmap v6502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:40 smithi115 ceph-mon[35364]: pgmap v6503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:40 smithi181 ceph-mon[35869]: pgmap v6503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:40 smithi097 ceph-mon[32231]: pgmap v6503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11057.908162 secs 2024-07-26T11:03:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11057.908162 secs 2024-07-26T11:03:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11057.908162 secs 2024-07-26T11:03:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:42 smithi115 ceph-mon[35364]: pgmap v6504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:42 smithi181 ceph-mon[35869]: pgmap v6504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:42 smithi097 ceph-mon[32231]: pgmap v6504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:44 smithi115 ceph-mon[35364]: pgmap v6505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:44 smithi181 ceph-mon[35869]: pgmap v6505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:44 smithi097 ceph-mon[32231]: pgmap v6505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11062.908325 secs 2024-07-26T11:03:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11062.908325 secs 2024-07-26T11:03:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11062.908325 secs 2024-07-26T11:03:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:46 smithi115 ceph-mon[35364]: pgmap v6506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:46 smithi181 ceph-mon[35869]: pgmap v6506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:46 smithi097 ceph-mon[32231]: pgmap v6506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:03:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:48 smithi115 ceph-mon[35364]: pgmap v6507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:48 smithi181 ceph-mon[35869]: pgmap v6507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:48 smithi097 ceph-mon[32231]: pgmap v6507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:50 smithi115 ceph-mon[35364]: pgmap v6508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:50 smithi181 ceph-mon[35869]: pgmap v6508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:50 smithi097 ceph-mon[32231]: pgmap v6508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:51.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11067.908467 secs 2024-07-26T11:03:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11067.908467 secs 2024-07-26T11:03:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11067.908467 secs 2024-07-26T11:03:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:52.384+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:52.385+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:03:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:52.385+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:03:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:52.385+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:03:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:03:52.386+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:52 smithi115 ceph-mon[35364]: pgmap v6509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:52 smithi181 ceph-mon[35869]: pgmap v6509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:52 smithi097 ceph-mon[32231]: pgmap v6509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:03:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: pgmap v6510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: pgmap v6511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:54.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: pgmap v6510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: pgmap v6511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:54.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:03:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:55.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:03:55.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:03:55.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:03:55.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:55.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:55.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: pgmap v6510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:03:55.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: pgmap v6511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:55.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:03:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11072.908611 secs 2024-07-26T11:03:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11072.908611 secs 2024-07-26T11:03:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11072.908611 secs 2024-07-26T11:03:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:56 smithi115 ceph-mon[35364]: pgmap v6512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:56 smithi181 ceph-mon[35869]: pgmap v6512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:56 smithi097 ceph-mon[32231]: pgmap v6512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:03:59 smithi115 ceph-mon[35364]: pgmap v6513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:03:59 smithi181 ceph-mon[35869]: pgmap v6513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:03:59 smithi097 ceph-mon[32231]: pgmap v6513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11077.908752 secs 2024-07-26T11:04:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:01 smithi115 ceph-mon[35364]: pgmap v6514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11077.908752 secs 2024-07-26T11:04:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:01 smithi181 ceph-mon[35869]: pgmap v6514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11077.908752 secs 2024-07-26T11:04:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:01 smithi097 ceph-mon[32231]: pgmap v6514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:03 smithi115 ceph-mon[35364]: pgmap v6515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:03 smithi181 ceph-mon[35869]: pgmap v6515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:03 smithi097 ceph-mon[32231]: pgmap v6515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:05 smithi115 ceph-mon[35364]: pgmap v6516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:05 smithi181 ceph-mon[35869]: pgmap v6516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:05 smithi097 ceph-mon[32231]: pgmap v6516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11082.908913 secs 2024-07-26T11:04:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11082.908913 secs 2024-07-26T11:04:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11082.908913 secs 2024-07-26T11:04:07.394 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:07.392+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:07.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:07.393+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:04:07.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:07.393+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:04:07.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:07.393+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:04:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:07.393+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:07 smithi115 ceph-mon[35364]: pgmap v6517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:07 smithi181 ceph-mon[35869]: pgmap v6517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:07 smithi097 ceph-mon[32231]: pgmap v6517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:08.843+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:04:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:08.843+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:04:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:08.844+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:04:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:08.844+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:04:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:09 smithi115 ceph-mon[35364]: pgmap v6518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:09 smithi181 ceph-mon[35869]: pgmap v6518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:09 smithi097 ceph-mon[32231]: pgmap v6518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11087.909090 secs 2024-07-26T11:04:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:11 smithi115 ceph-mon[35364]: pgmap v6519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:04:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11087.909090 secs 2024-07-26T11:04:11.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:11 smithi181 ceph-mon[35869]: pgmap v6519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:04:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11087.909090 secs 2024-07-26T11:04:11.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:11 smithi097 ceph-mon[32231]: pgmap v6519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:04:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:13 smithi115 ceph-mon[35364]: pgmap v6520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:13 smithi181 ceph-mon[35869]: pgmap v6520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:13 smithi097 ceph-mon[32231]: pgmap v6520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:15.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:15 smithi115 ceph-mon[35364]: pgmap v6521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:04:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:15 smithi181 ceph-mon[35869]: pgmap v6521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:04:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:15 smithi097 ceph-mon[32231]: pgmap v6521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:04:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11092.909293 secs 2024-07-26T11:04:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11092.909293 secs 2024-07-26T11:04:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11092.909293 secs 2024-07-26T11:04:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:17 smithi115 ceph-mon[35364]: pgmap v6522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:04:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:17 smithi181 ceph-mon[35869]: pgmap v6522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:04:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:17 smithi097 ceph-mon[32231]: pgmap v6522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:04:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:19 smithi115 ceph-mon[35364]: pgmap v6523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:19 smithi181 ceph-mon[35869]: pgmap v6523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:19 smithi097 ceph-mon[32231]: pgmap v6523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11097.909432 secs 2024-07-26T11:04:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:21 smithi181 ceph-mon[35869]: pgmap v6524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11097.909432 secs 2024-07-26T11:04:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:21 smithi097 ceph-mon[32231]: pgmap v6524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11097.909432 secs 2024-07-26T11:04:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:21 smithi115 ceph-mon[35364]: pgmap v6524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:22.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:22.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:22.396+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:04:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:22.396+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:04:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:22.396+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:04:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:22.396+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:23 smithi181 ceph-mon[35869]: pgmap v6525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:23 smithi097 ceph-mon[32231]: pgmap v6525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:23 smithi115 ceph-mon[35364]: pgmap v6525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:25 smithi097 ceph-mon[32231]: pgmap v6526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:25 smithi115 ceph-mon[35364]: pgmap v6526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:25 smithi181 ceph-mon[35869]: pgmap v6526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11102.909528 secs 2024-07-26T11:04:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11102.909528 secs 2024-07-26T11:04:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11102.909528 secs 2024-07-26T11:04:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:27 smithi115 ceph-mon[35364]: pgmap v6527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:27 smithi181 ceph-mon[35869]: pgmap v6527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:27 smithi097 ceph-mon[32231]: pgmap v6527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:29 smithi115 ceph-mon[35364]: pgmap v6528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:29 smithi181 ceph-mon[35869]: pgmap v6528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:29 smithi097 ceph-mon[32231]: pgmap v6528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11107.909696 secs 2024-07-26T11:04:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:31 smithi115 ceph-mon[35364]: pgmap v6529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11107.909696 secs 2024-07-26T11:04:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:31 smithi181 ceph-mon[35869]: pgmap v6529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11107.909696 secs 2024-07-26T11:04:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:31 smithi097 ceph-mon[32231]: pgmap v6529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:33 smithi115 ceph-mon[35364]: pgmap v6530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:33 smithi181 ceph-mon[35869]: pgmap v6530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:33 smithi097 ceph-mon[32231]: pgmap v6530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:35 smithi115 ceph-mon[35364]: pgmap v6531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:35 smithi181 ceph-mon[35869]: pgmap v6531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:35 smithi097 ceph-mon[32231]: pgmap v6531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11112.909836 secs 2024-07-26T11:04:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11112.909836 secs 2024-07-26T11:04:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11112.909836 secs 2024-07-26T11:04:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:37.403+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:37.404+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:04:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:37.404+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:04:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:37.404+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:04:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:37.404+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:37 smithi115 ceph-mon[35364]: pgmap v6532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:37 smithi181 ceph-mon[35869]: pgmap v6532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:37 smithi097 ceph-mon[32231]: pgmap v6532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:39 smithi115 ceph-mon[35364]: pgmap v6533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:39 smithi181 ceph-mon[35869]: pgmap v6533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:39 smithi097 ceph-mon[32231]: pgmap v6533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11117.909981 secs 2024-07-26T11:04:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:41 smithi115 ceph-mon[35364]: pgmap v6534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11117.909981 secs 2024-07-26T11:04:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:41 smithi181 ceph-mon[35869]: pgmap v6534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11117.909981 secs 2024-07-26T11:04:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:41 smithi097 ceph-mon[32231]: pgmap v6534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:43 smithi115 ceph-mon[35364]: pgmap v6535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:43 smithi181 ceph-mon[35869]: pgmap v6535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:43 smithi097 ceph-mon[32231]: pgmap v6535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:45 smithi115 ceph-mon[35364]: pgmap v6536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:45 smithi181 ceph-mon[35869]: pgmap v6536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:45 smithi097 ceph-mon[32231]: pgmap v6536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11122.910140 secs 2024-07-26T11:04:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11122.910140 secs 2024-07-26T11:04:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11122.910140 secs 2024-07-26T11:04:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:47 smithi115 ceph-mon[35364]: pgmap v6537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:47 smithi181 ceph-mon[35869]: pgmap v6537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:47 smithi097 ceph-mon[32231]: pgmap v6537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:04:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:49 smithi115 ceph-mon[35364]: pgmap v6538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:49 smithi181 ceph-mon[35869]: pgmap v6538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:49 smithi097 ceph-mon[32231]: pgmap v6538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:04:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11127.910298 secs 2024-07-26T11:04:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:51 smithi115 ceph-mon[35364]: pgmap v6539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11127.910298 secs 2024-07-26T11:04:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:51 smithi181 ceph-mon[35869]: pgmap v6539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11127.910298 secs 2024-07-26T11:04:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:51 smithi097 ceph-mon[32231]: pgmap v6539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:52.405+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:52.406+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:04:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:52.406+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:04:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:52.406+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:04:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:04:52.407+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:53 smithi115 ceph-mon[35364]: pgmap v6540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:53 smithi181 ceph-mon[35869]: pgmap v6540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:53 smithi097 ceph-mon[32231]: pgmap v6540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:55 smithi115 ceph-mon[35364]: pgmap v6541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:55 smithi181 ceph-mon[35869]: pgmap v6541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:55 smithi097 ceph-mon[32231]: pgmap v6541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:04:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11132.910438 secs 2024-07-26T11:04:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11132.910438 secs 2024-07-26T11:04:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11132.910438 secs 2024-07-26T11:04:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:04:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:57 smithi115 ceph-mon[35364]: pgmap v6542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:04:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:57 smithi115 ceph-mon[35364]: pgmap v6543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:04:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:57 smithi181 ceph-mon[35869]: pgmap v6542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:04:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:57 smithi181 ceph-mon[35869]: pgmap v6543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:04:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:57 smithi097 ceph-mon[32231]: pgmap v6542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:04:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:57 smithi097 ceph-mon[32231]: pgmap v6543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:04:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:04:59 smithi115 ceph-mon[35364]: pgmap v6544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:04:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:04:59 smithi181 ceph-mon[35869]: pgmap v6544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:04:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:04:59 smithi097 ceph-mon[32231]: pgmap v6544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11137.910580 secs 2024-07-26T11:05:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:01 smithi115 ceph-mon[35364]: pgmap v6545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11137.910580 secs 2024-07-26T11:05:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:01 smithi181 ceph-mon[35869]: pgmap v6545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11137.910580 secs 2024-07-26T11:05:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:01 smithi097 ceph-mon[32231]: pgmap v6545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:03 smithi115 ceph-mon[35364]: pgmap v6546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:03 smithi181 ceph-mon[35869]: pgmap v6546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:03 smithi097 ceph-mon[32231]: pgmap v6546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:05 smithi115 ceph-mon[35364]: pgmap v6547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:05 smithi181 ceph-mon[35869]: pgmap v6547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:05 smithi097 ceph-mon[32231]: pgmap v6547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11142.910675 secs 2024-07-26T11:05:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11142.910675 secs 2024-07-26T11:05:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11142.910675 secs 2024-07-26T11:05:07.416 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:07.414+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:07.415+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:07.418 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:07.415+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:07.418 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:07.415+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:07.418 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:07.415+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:07 smithi115 ceph-mon[35364]: pgmap v6548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:05:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:07 smithi181 ceph-mon[35869]: pgmap v6548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:05:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:07 smithi097 ceph-mon[32231]: pgmap v6548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:05:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:09 smithi115 ceph-mon[35364]: pgmap v6549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:09 smithi181 ceph-mon[35869]: pgmap v6549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:09 smithi097 ceph-mon[32231]: pgmap v6549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11147.910769 secs 2024-07-26T11:05:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:11 smithi115 ceph-mon[35364]: pgmap v6550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11147.910769 secs 2024-07-26T11:05:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:11 smithi181 ceph-mon[35869]: pgmap v6550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11147.910769 secs 2024-07-26T11:05:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:11 smithi097 ceph-mon[32231]: pgmap v6550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:13 smithi115 ceph-mon[35364]: pgmap v6551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:13 smithi181 ceph-mon[35869]: pgmap v6551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:13 smithi097 ceph-mon[32231]: pgmap v6551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:15 smithi181 ceph-mon[35869]: pgmap v6552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:15 smithi097 ceph-mon[32231]: pgmap v6552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:15 smithi115 ceph-mon[35364]: pgmap v6552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11152.910911 secs 2024-07-26T11:05:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11152.910911 secs 2024-07-26T11:05:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11152.910911 secs 2024-07-26T11:05:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:17 smithi097 ceph-mon[32231]: pgmap v6553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:17 smithi115 ceph-mon[35364]: pgmap v6553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:17 smithi181 ceph-mon[35869]: pgmap v6553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:19 smithi115 ceph-mon[35364]: pgmap v6554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:19 smithi181 ceph-mon[35869]: pgmap v6554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:19 smithi097 ceph-mon[32231]: pgmap v6554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:21.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11157.911075 secs 2024-07-26T11:05:21.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:21 smithi115 ceph-mon[35364]: pgmap v6555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:21.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11157.911075 secs 2024-07-26T11:05:21.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:21 smithi181 ceph-mon[35869]: pgmap v6555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:21.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11157.911075 secs 2024-07-26T11:05:21.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:21 smithi097 ceph-mon[32231]: pgmap v6555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:22.417+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:22.418+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:22.418+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:22.418+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:22.418+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:23 smithi115 ceph-mon[35364]: pgmap v6556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:23 smithi181 ceph-mon[35869]: pgmap v6556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:23 smithi097 ceph-mon[32231]: pgmap v6556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:25 smithi115 ceph-mon[35364]: pgmap v6557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:25 smithi181 ceph-mon[35869]: pgmap v6557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:25 smithi097 ceph-mon[32231]: pgmap v6557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11162.911313 secs 2024-07-26T11:05:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11162.911313 secs 2024-07-26T11:05:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11162.911313 secs 2024-07-26T11:05:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:27 smithi115 ceph-mon[35364]: pgmap v6558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:27 smithi181 ceph-mon[35869]: pgmap v6558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:27 smithi097 ceph-mon[32231]: pgmap v6558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:29 smithi115 ceph-mon[35364]: pgmap v6559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:29 smithi181 ceph-mon[35869]: pgmap v6559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:29 smithi097 ceph-mon[32231]: pgmap v6559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11167.911467 secs 2024-07-26T11:05:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:31 smithi115 ceph-mon[35364]: pgmap v6560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11167.911467 secs 2024-07-26T11:05:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:31 smithi181 ceph-mon[35869]: pgmap v6560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11167.911467 secs 2024-07-26T11:05:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:31 smithi097 ceph-mon[32231]: pgmap v6560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:33 smithi115 ceph-mon[35364]: pgmap v6561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:33 smithi181 ceph-mon[35869]: pgmap v6561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:33 smithi097 ceph-mon[32231]: pgmap v6561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:35 smithi115 ceph-mon[35364]: pgmap v6562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:35 smithi181 ceph-mon[35869]: pgmap v6562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:35 smithi097 ceph-mon[32231]: pgmap v6562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11172.911566 secs 2024-07-26T11:05:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11172.911566 secs 2024-07-26T11:05:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11172.911566 secs 2024-07-26T11:05:37.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:37.425+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:37.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:37.426+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:37.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:37.426+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:37.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:37.426+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:37.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:37.426+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:37 smithi115 ceph-mon[35364]: pgmap v6563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:37 smithi181 ceph-mon[35869]: pgmap v6563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:37 smithi097 ceph-mon[32231]: pgmap v6563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:39 smithi115 ceph-mon[35364]: pgmap v6564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:39 smithi181 ceph-mon[35869]: pgmap v6564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:39 smithi097 ceph-mon[32231]: pgmap v6564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11177.911698 secs 2024-07-26T11:05:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:41 smithi115 ceph-mon[35364]: pgmap v6565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11177.911698 secs 2024-07-26T11:05:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:41 smithi181 ceph-mon[35869]: pgmap v6565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11177.911698 secs 2024-07-26T11:05:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:41 smithi097 ceph-mon[32231]: pgmap v6565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:43 smithi115 ceph-mon[35364]: pgmap v6566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:43 smithi181 ceph-mon[35869]: pgmap v6566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:43 smithi097 ceph-mon[32231]: pgmap v6566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:44 smithi115 ceph-mon[35364]: pgmap v6567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:44 smithi181 ceph-mon[35869]: pgmap v6567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:44 smithi097 ceph-mon[32231]: pgmap v6567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11182.911840 secs 2024-07-26T11:05:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11182.911840 secs 2024-07-26T11:05:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11182.911840 secs 2024-07-26T11:05:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:46 smithi115 ceph-mon[35364]: pgmap v6568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:46 smithi181 ceph-mon[35869]: pgmap v6568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:47.318 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:46 smithi097 ceph-mon[32231]: pgmap v6568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:49 smithi115 ceph-mon[35364]: pgmap v6569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:49 smithi181 ceph-mon[35869]: pgmap v6569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:49 smithi097 ceph-mon[32231]: pgmap v6569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11187.912000 secs 2024-07-26T11:05:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:51 smithi115 ceph-mon[35364]: pgmap v6570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11187.912000 secs 2024-07-26T11:05:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:51 smithi181 ceph-mon[35869]: pgmap v6570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11187.912000 secs 2024-07-26T11:05:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:51 smithi097 ceph-mon[32231]: pgmap v6570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:52.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:52.429+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:05:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:52.429+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:05:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:52.429+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:05:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:05:52.429+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:53 smithi115 ceph-mon[35364]: pgmap v6571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:53 smithi181 ceph-mon[35869]: pgmap v6571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:53 smithi097 ceph-mon[32231]: pgmap v6571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:55 smithi115 ceph-mon[35364]: pgmap v6572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:55.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:55 smithi181 ceph-mon[35869]: pgmap v6572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:55 smithi097 ceph-mon[32231]: pgmap v6572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:05:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11192.912139 secs 2024-07-26T11:05:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11192.912139 secs 2024-07-26T11:05:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11192.912139 secs 2024-07-26T11:05:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:57 smithi115 ceph-mon[35364]: pgmap v6573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:57 smithi181 ceph-mon[35869]: pgmap v6573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:57 smithi097 ceph-mon[32231]: pgmap v6573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:05:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:05:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:59 smithi115 ceph-mon[35364]: pgmap v6574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:05:59 smithi115 ceph-mon[35364]: pgmap v6575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:59 smithi181 ceph-mon[35869]: pgmap v6574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:05:59 smithi181 ceph-mon[35869]: pgmap v6575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:05:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:59 smithi097 ceph-mon[32231]: pgmap v6574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:05:59 smithi097 ceph-mon[32231]: pgmap v6575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11197.912282 secs 2024-07-26T11:06:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:01 smithi115 ceph-mon[35364]: pgmap v6576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11197.912282 secs 2024-07-26T11:06:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:01 smithi181 ceph-mon[35869]: pgmap v6576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11197.912282 secs 2024-07-26T11:06:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:01 smithi097 ceph-mon[32231]: pgmap v6576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:03 smithi115 ceph-mon[35364]: pgmap v6577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:03 smithi181 ceph-mon[35869]: pgmap v6577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:03 smithi097 ceph-mon[32231]: pgmap v6577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:05 smithi115 ceph-mon[35364]: pgmap v6578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:05 smithi181 ceph-mon[35869]: pgmap v6578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:05 smithi097 ceph-mon[32231]: pgmap v6578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11202.912445 secs 2024-07-26T11:06:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11202.912445 secs 2024-07-26T11:06:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11202.912445 secs 2024-07-26T11:06:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:07.435+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:07.437+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:06:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:07.437+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:06:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:07.437+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:06:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:07.437+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:07 smithi115 ceph-mon[35364]: pgmap v6579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:07 smithi181 ceph-mon[35869]: pgmap v6579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:07 smithi097 ceph-mon[32231]: pgmap v6579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:06:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:09 smithi115 ceph-mon[35364]: pgmap v6580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:06:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:09 smithi181 ceph-mon[35869]: pgmap v6580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:06:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:09 smithi097 ceph-mon[32231]: pgmap v6580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:06:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11207.912585 secs 2024-07-26T11:06:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:11 smithi115 ceph-mon[35364]: pgmap v6581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11207.912585 secs 2024-07-26T11:06:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:11 smithi181 ceph-mon[35869]: pgmap v6581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11207.912585 secs 2024-07-26T11:06:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:11 smithi097 ceph-mon[32231]: pgmap v6581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:13 smithi115 ceph-mon[35364]: pgmap v6582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:13 smithi181 ceph-mon[35869]: pgmap v6582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:13 smithi097 ceph-mon[32231]: pgmap v6582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:15 smithi115 ceph-mon[35364]: pgmap v6583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:15 smithi181 ceph-mon[35869]: pgmap v6583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:15 smithi097 ceph-mon[32231]: pgmap v6583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11212.912733 secs 2024-07-26T11:06:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11212.912733 secs 2024-07-26T11:06:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11212.912733 secs 2024-07-26T11:06:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:17 smithi115 ceph-mon[35364]: pgmap v6584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:17 smithi181 ceph-mon[35869]: pgmap v6584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:17 smithi097 ceph-mon[32231]: pgmap v6584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:19 smithi115 ceph-mon[35364]: pgmap v6585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:06:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:19 smithi181 ceph-mon[35869]: pgmap v6585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:06:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:19 smithi097 ceph-mon[32231]: pgmap v6585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:06:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11217.912893 secs 2024-07-26T11:06:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:21 smithi115 ceph-mon[35364]: pgmap v6586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:06:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11217.912893 secs 2024-07-26T11:06:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:21 smithi181 ceph-mon[35869]: pgmap v6586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:06:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11217.912893 secs 2024-07-26T11:06:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:21 smithi097 ceph-mon[32231]: pgmap v6586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:06:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:22.438+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:22.440+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:06:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:22.440+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:06:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:22.440+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:06:22.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:22.440+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:23 smithi181 ceph-mon[35869]: pgmap v6587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:06:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:23 smithi097 ceph-mon[32231]: pgmap v6587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:06:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:23 smithi115 ceph-mon[35364]: pgmap v6587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:06:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:25 smithi181 ceph-mon[35869]: pgmap v6588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:06:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:25 smithi097 ceph-mon[32231]: pgmap v6588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:06:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:25 smithi115 ceph-mon[35364]: pgmap v6588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:06:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11222.913056 secs 2024-07-26T11:06:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11222.913056 secs 2024-07-26T11:06:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11222.913056 secs 2024-07-26T11:06:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:27 smithi097 ceph-mon[32231]: pgmap v6589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:27 smithi115 ceph-mon[35364]: pgmap v6589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:27 smithi181 ceph-mon[35869]: pgmap v6589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:29 smithi115 ceph-mon[35364]: pgmap v6590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:29 smithi181 ceph-mon[35869]: pgmap v6590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:29 smithi097 ceph-mon[32231]: pgmap v6590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11227.913198 secs 2024-07-26T11:06:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:31 smithi115 ceph-mon[35364]: pgmap v6591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11227.913198 secs 2024-07-26T11:06:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:31 smithi181 ceph-mon[35869]: pgmap v6591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11227.913198 secs 2024-07-26T11:06:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:31 smithi097 ceph-mon[32231]: pgmap v6591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:33 smithi115 ceph-mon[35364]: pgmap v6592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:33 smithi181 ceph-mon[35869]: pgmap v6592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:33.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:33 smithi097 ceph-mon[32231]: pgmap v6592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:35 smithi115 ceph-mon[35364]: pgmap v6593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:35 smithi181 ceph-mon[35869]: pgmap v6593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:35 smithi097 ceph-mon[32231]: pgmap v6593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11232.913351 secs 2024-07-26T11:06:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11232.913351 secs 2024-07-26T11:06:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11232.913351 secs 2024-07-26T11:06:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:37.447+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:37.448+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:06:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:37.448+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:06:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:37.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:06:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:37.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:37 smithi115 ceph-mon[35364]: pgmap v6594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:37 smithi181 ceph-mon[35869]: pgmap v6594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:37 smithi097 ceph-mon[32231]: pgmap v6594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:39 smithi115 ceph-mon[35364]: pgmap v6595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:39 smithi181 ceph-mon[35869]: pgmap v6595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:39 smithi097 ceph-mon[32231]: pgmap v6595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11237.913510 secs 2024-07-26T11:06:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:41 smithi115 ceph-mon[35364]: pgmap v6596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11237.913510 secs 2024-07-26T11:06:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:41 smithi181 ceph-mon[35869]: pgmap v6596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11237.913510 secs 2024-07-26T11:06:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:41 smithi097 ceph-mon[32231]: pgmap v6596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:43 smithi115 ceph-mon[35364]: pgmap v6597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:43 smithi181 ceph-mon[35869]: pgmap v6597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:43 smithi097 ceph-mon[32231]: pgmap v6597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:45 smithi115 ceph-mon[35364]: pgmap v6598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:45 smithi181 ceph-mon[35869]: pgmap v6598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:45 smithi097 ceph-mon[32231]: pgmap v6598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11242.913668 secs 2024-07-26T11:06:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11242.913668 secs 2024-07-26T11:06:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11242.913668 secs 2024-07-26T11:06:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:47 smithi115 ceph-mon[35364]: pgmap v6599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:47 smithi181 ceph-mon[35869]: pgmap v6599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:47 smithi097 ceph-mon[32231]: pgmap v6599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:49 smithi115 ceph-mon[35364]: pgmap v6600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:49 smithi181 ceph-mon[35869]: pgmap v6600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:49 smithi097 ceph-mon[32231]: pgmap v6600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11247.913806 secs 2024-07-26T11:06:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:51 smithi115 ceph-mon[35364]: pgmap v6601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11247.913806 secs 2024-07-26T11:06:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:51 smithi181 ceph-mon[35869]: pgmap v6601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11247.913806 secs 2024-07-26T11:06:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:51 smithi097 ceph-mon[32231]: pgmap v6601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:52.450+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:52.451+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:06:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:52.451+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:06:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:52.451+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:06:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:06:52.451+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:53 smithi115 ceph-mon[35364]: pgmap v6602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:53 smithi181 ceph-mon[35869]: pgmap v6602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:53 smithi097 ceph-mon[32231]: pgmap v6602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:55 smithi115 ceph-mon[35364]: pgmap v6603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:55 smithi181 ceph-mon[35869]: pgmap v6603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:55 smithi097 ceph-mon[32231]: pgmap v6603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:06:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11252.913968 secs 2024-07-26T11:06:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11252.913968 secs 2024-07-26T11:06:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11252.913968 secs 2024-07-26T11:06:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:57 smithi115 ceph-mon[35364]: pgmap v6604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:57 smithi181 ceph-mon[35869]: pgmap v6604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:57 smithi097 ceph-mon[32231]: pgmap v6604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:06:59 smithi115 ceph-mon[35364]: pgmap v6605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:06:59 smithi181 ceph-mon[35869]: pgmap v6605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:06:59 smithi097 ceph-mon[32231]: pgmap v6605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11257.914059 secs 2024-07-26T11:07:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:01 smithi115 ceph-mon[35364]: pgmap v6606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11257.914059 secs 2024-07-26T11:07:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:01 smithi181 ceph-mon[35869]: pgmap v6606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11257.914059 secs 2024-07-26T11:07:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:01 smithi097 ceph-mon[32231]: pgmap v6606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:02.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:03 smithi115 ceph-mon[35364]: pgmap v6607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:03 smithi181 ceph-mon[35869]: pgmap v6607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:03.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:03.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:03.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:03 smithi097 ceph-mon[32231]: pgmap v6607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:07:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:04 smithi115 ceph-mon[35364]: pgmap v6608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:07:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:04 smithi115 ceph-mon[35364]: pgmap v6609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:04 smithi181 ceph-mon[35869]: pgmap v6608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:07:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:04 smithi181 ceph-mon[35869]: pgmap v6609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:04 smithi097 ceph-mon[32231]: pgmap v6608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:07:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:04 smithi097 ceph-mon[32231]: pgmap v6609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:06 smithi115 ceph-mon[35364]: pgmap v6610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:07:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11262.914197 secs 2024-07-26T11:07:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:06 smithi181 ceph-mon[35869]: pgmap v6610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:07:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11262.914197 secs 2024-07-26T11:07:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:06 smithi097 ceph-mon[32231]: pgmap v6610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:07:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11262.914197 secs 2024-07-26T11:07:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:07.458+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:07.459+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:07:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:07.459+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:07:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:07.459+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:07:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:07.459+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:08 smithi115 ceph-mon[35364]: pgmap v6611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:07:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:08 smithi181 ceph-mon[35869]: pgmap v6611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:07:08.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:08 smithi097 ceph-mon[32231]: pgmap v6611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:07:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:08.877+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:07:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:08.877+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:07:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:08.877+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:07:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:08.878+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:07:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:10 smithi115 ceph-mon[35364]: pgmap v6612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:07:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:10 smithi181 ceph-mon[35869]: pgmap v6612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:07:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:10 smithi097 ceph-mon[32231]: pgmap v6612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:07:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11267.914334 secs 2024-07-26T11:07:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11267.914334 secs 2024-07-26T11:07:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11267.914334 secs 2024-07-26T11:07:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:12 smithi181 ceph-mon[35869]: pgmap v6613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:07:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:12 smithi097 ceph-mon[32231]: pgmap v6613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:07:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:12 smithi115 ceph-mon[35364]: pgmap v6613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:07:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:14 smithi097 ceph-mon[32231]: pgmap v6614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:14 smithi115 ceph-mon[35364]: pgmap v6614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:14 smithi181 ceph-mon[35869]: pgmap v6614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:16 smithi097 ceph-mon[32231]: pgmap v6615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11272.914478 secs 2024-07-26T11:07:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:16 smithi115 ceph-mon[35364]: pgmap v6615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11272.914478 secs 2024-07-26T11:07:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:16 smithi181 ceph-mon[35869]: pgmap v6615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11272.914478 secs 2024-07-26T11:07:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:18 smithi097 ceph-mon[32231]: pgmap v6616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:07:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:18 smithi115 ceph-mon[35364]: pgmap v6616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:07:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:18 smithi181 ceph-mon[35869]: pgmap v6616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:07:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:20 smithi115 ceph-mon[35364]: pgmap v6617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:20 smithi181 ceph-mon[35869]: pgmap v6617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:20 smithi097 ceph-mon[32231]: pgmap v6617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:21.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11277.914620 secs 2024-07-26T11:07:21.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11277.914620 secs 2024-07-26T11:07:21.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11277.914620 secs 2024-07-26T11:07:22.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:22.462+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:22.462+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:07:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:22.462+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:07:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:22.462+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:07:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:22.462+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:22.915 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:22 smithi181 ceph-mon[35869]: pgmap v6618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:07:22.945 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:22 smithi097 ceph-mon[32231]: pgmap v6618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:07:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:22 smithi115 ceph-mon[35364]: pgmap v6618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:07:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:24 smithi115 ceph-mon[35364]: pgmap v6619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:07:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:24 smithi181 ceph-mon[35869]: pgmap v6619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:07:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:24 smithi097 ceph-mon[32231]: pgmap v6619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:07:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:26 smithi115 ceph-mon[35364]: pgmap v6620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11282.914716 secs 2024-07-26T11:07:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:26 smithi181 ceph-mon[35869]: pgmap v6620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11282.914716 secs 2024-07-26T11:07:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:26 smithi097 ceph-mon[32231]: pgmap v6620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:07:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11282.914716 secs 2024-07-26T11:07:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:28 smithi115 ceph-mon[35364]: pgmap v6621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:28 smithi181 ceph-mon[35869]: pgmap v6621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:28 smithi097 ceph-mon[32231]: pgmap v6621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:30 smithi115 ceph-mon[35364]: pgmap v6622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:30 smithi181 ceph-mon[35869]: pgmap v6622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:30 smithi097 ceph-mon[32231]: pgmap v6622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11287.914863 secs 2024-07-26T11:07:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11287.914863 secs 2024-07-26T11:07:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11287.914863 secs 2024-07-26T11:07:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:32 smithi115 ceph-mon[35364]: pgmap v6623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:32 smithi181 ceph-mon[35869]: pgmap v6623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:32 smithi097 ceph-mon[32231]: pgmap v6623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:34 smithi115 ceph-mon[35364]: pgmap v6624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:34 smithi181 ceph-mon[35869]: pgmap v6624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:34 smithi097 ceph-mon[32231]: pgmap v6624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:36 smithi115 ceph-mon[35364]: pgmap v6625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11292.915007 secs 2024-07-26T11:07:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:36 smithi181 ceph-mon[35869]: pgmap v6625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11292.915007 secs 2024-07-26T11:07:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:36 smithi097 ceph-mon[32231]: pgmap v6625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11292.915007 secs 2024-07-26T11:07:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:37.469+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:37.470+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:07:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:37.470+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:07:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:37.470+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:07:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:37.470+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:38 smithi115 ceph-mon[35364]: pgmap v6626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:38 smithi181 ceph-mon[35869]: pgmap v6626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:38 smithi097 ceph-mon[32231]: pgmap v6626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:40 smithi115 ceph-mon[35364]: pgmap v6627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:40 smithi181 ceph-mon[35869]: pgmap v6627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:40 smithi097 ceph-mon[32231]: pgmap v6627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11297.915146 secs 2024-07-26T11:07:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11297.915146 secs 2024-07-26T11:07:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11297.915146 secs 2024-07-26T11:07:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:42 smithi115 ceph-mon[35364]: pgmap v6628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:42 smithi181 ceph-mon[35869]: pgmap v6628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:42 smithi097 ceph-mon[32231]: pgmap v6628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:44 smithi115 ceph-mon[35364]: pgmap v6629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:44 smithi181 ceph-mon[35869]: pgmap v6629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:44 smithi097 ceph-mon[32231]: pgmap v6629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:46 smithi115 ceph-mon[35364]: pgmap v6630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11302.915274 secs 2024-07-26T11:07:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:46 smithi181 ceph-mon[35869]: pgmap v6630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11302.915274 secs 2024-07-26T11:07:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:46 smithi097 ceph-mon[32231]: pgmap v6630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11302.915274 secs 2024-07-26T11:07:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:47 smithi115 ceph-mon[35364]: pgmap v6631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:47 smithi181 ceph-mon[35869]: pgmap v6631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:47 smithi097 ceph-mon[32231]: pgmap v6631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:49 smithi115 ceph-mon[35364]: pgmap v6632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:49 smithi181 ceph-mon[35869]: pgmap v6632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:49 smithi097 ceph-mon[32231]: pgmap v6632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:51.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11307.915379 secs 2024-07-26T11:07:51.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11307.915379 secs 2024-07-26T11:07:51.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11307.915379 secs 2024-07-26T11:07:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:51 smithi115 ceph-mon[35364]: pgmap v6633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:51 smithi181 ceph-mon[35869]: pgmap v6633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:51 smithi097 ceph-mon[32231]: pgmap v6633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:52.472+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:52.473+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:07:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:52.473+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:07:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:52.473+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:07:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:52.473+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:53.885+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:07:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:53.885+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:07:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:53.886+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:07:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:07:53.886+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:07:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:53 smithi115 ceph-mon[35364]: pgmap v6634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:53 smithi181 ceph-mon[35869]: pgmap v6634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:53 smithi097 ceph-mon[32231]: pgmap v6634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:55 smithi115 ceph-mon[35364]: pgmap v6635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11312.915517 secs 2024-07-26T11:07:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:55 smithi181 ceph-mon[35869]: pgmap v6635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11312.915517 secs 2024-07-26T11:07:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:55 smithi097 ceph-mon[32231]: pgmap v6635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11312.915517 secs 2024-07-26T11:07:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:57 smithi115 ceph-mon[35364]: pgmap v6636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:57 smithi181 ceph-mon[35869]: pgmap v6636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:57 smithi097 ceph-mon[32231]: pgmap v6636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:07:59 smithi115 ceph-mon[35364]: pgmap v6637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:07:59 smithi181 ceph-mon[35869]: pgmap v6637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:07:59 smithi097 ceph-mon[32231]: pgmap v6637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11317.915662 secs 2024-07-26T11:08:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11317.915662 secs 2024-07-26T11:08:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11317.915662 secs 2024-07-26T11:08:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:01 smithi115 ceph-mon[35364]: pgmap v6638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:01 smithi181 ceph-mon[35869]: pgmap v6638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:01 smithi097 ceph-mon[32231]: pgmap v6638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:03 smithi115 ceph-mon[35364]: pgmap v6639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:03 smithi181 ceph-mon[35869]: pgmap v6639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:03 smithi097 ceph-mon[32231]: pgmap v6639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:05.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:05.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:05 smithi115 ceph-mon[35364]: pgmap v6640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:05.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:05 smithi115 ceph-mon[35364]: pgmap v6641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:05.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:05.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:05.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:05.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:05.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:05.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:05 smithi181 ceph-mon[35869]: pgmap v6640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:05.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:05 smithi181 ceph-mon[35869]: pgmap v6641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:05.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:05 smithi097 ceph-mon[32231]: pgmap v6640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:05 smithi097 ceph-mon[32231]: pgmap v6641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:08:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11322.915756 secs 2024-07-26T11:08:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11322.915756 secs 2024-07-26T11:08:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11322.915756 secs 2024-07-26T11:08:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:07.480+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:07.481+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:08:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:07.481+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:08:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:07.481+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:08:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:07.481+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:07 smithi115 ceph-mon[35364]: pgmap v6642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:07 smithi181 ceph-mon[35869]: pgmap v6642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:07 smithi097 ceph-mon[32231]: pgmap v6642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:09 smithi181 ceph-mon[35869]: pgmap v6643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:09 smithi097 ceph-mon[32231]: pgmap v6643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:09 smithi115 ceph-mon[35364]: pgmap v6643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11327.915907 secs 2024-07-26T11:08:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:11 smithi097 ceph-mon[32231]: pgmap v6644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11327.915907 secs 2024-07-26T11:08:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:11 smithi115 ceph-mon[35364]: pgmap v6644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11327.915907 secs 2024-07-26T11:08:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:11 smithi181 ceph-mon[35869]: pgmap v6644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:13 smithi097 ceph-mon[32231]: pgmap v6645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:13 smithi115 ceph-mon[35364]: pgmap v6645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:13 smithi181 ceph-mon[35869]: pgmap v6645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:08:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:15 smithi115 ceph-mon[35364]: pgmap v6646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:08:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:15 smithi181 ceph-mon[35869]: pgmap v6646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:08:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:15 smithi097 ceph-mon[32231]: pgmap v6646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:08:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11332.916059 secs 2024-07-26T11:08:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11332.916059 secs 2024-07-26T11:08:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11332.916059 secs 2024-07-26T11:08:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:17 smithi115 ceph-mon[35364]: pgmap v6647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:17 smithi181 ceph-mon[35869]: pgmap v6647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:17 smithi097 ceph-mon[32231]: pgmap v6647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:20 smithi115 ceph-mon[35364]: pgmap v6648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:20 smithi181 ceph-mon[35869]: pgmap v6648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:20 smithi097 ceph-mon[32231]: pgmap v6648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:21.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11337.916148 secs 2024-07-26T11:08:21.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11337.916148 secs 2024-07-26T11:08:21.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11337.916148 secs 2024-07-26T11:08:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:22.482+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:22.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:08:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:22.484+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:08:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:22.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:08:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:22.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:22 smithi115 ceph-mon[35364]: pgmap v6649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:22 smithi181 ceph-mon[35869]: pgmap v6649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:22 smithi097 ceph-mon[32231]: pgmap v6649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:24 smithi115 ceph-mon[35364]: pgmap v6650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:24 smithi181 ceph-mon[35869]: pgmap v6650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:24 smithi097 ceph-mon[32231]: pgmap v6650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:26 smithi115 ceph-mon[35364]: pgmap v6651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11342.916293 secs 2024-07-26T11:08:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:26 smithi181 ceph-mon[35869]: pgmap v6651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11342.916293 secs 2024-07-26T11:08:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:26 smithi097 ceph-mon[32231]: pgmap v6651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11342.916293 secs 2024-07-26T11:08:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:28 smithi115 ceph-mon[35364]: pgmap v6652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:28 smithi181 ceph-mon[35869]: pgmap v6652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:28 smithi097 ceph-mon[32231]: pgmap v6652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:30 smithi115 ceph-mon[35364]: pgmap v6653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:30 smithi181 ceph-mon[35869]: pgmap v6653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:30 smithi097 ceph-mon[32231]: pgmap v6653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11347.916434 secs 2024-07-26T11:08:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11347.916434 secs 2024-07-26T11:08:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11347.916434 secs 2024-07-26T11:08:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:32 smithi115 ceph-mon[35364]: pgmap v6654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:32 smithi181 ceph-mon[35869]: pgmap v6654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:32 smithi097 ceph-mon[32231]: pgmap v6654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:34 smithi115 ceph-mon[35364]: pgmap v6655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:34 smithi181 ceph-mon[35869]: pgmap v6655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:34 smithi097 ceph-mon[32231]: pgmap v6655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:36 smithi115 ceph-mon[35364]: pgmap v6656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11352.916570 secs 2024-07-26T11:08:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:36 smithi181 ceph-mon[35869]: pgmap v6656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11352.916570 secs 2024-07-26T11:08:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:36 smithi097 ceph-mon[32231]: pgmap v6656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11352.916570 secs 2024-07-26T11:08:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:37.490+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:37.493+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:08:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:37.493+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:08:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:37.493+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:08:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:37.493+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:38 smithi115 ceph-mon[35364]: pgmap v6657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:38 smithi181 ceph-mon[35869]: pgmap v6657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:38 smithi097 ceph-mon[32231]: pgmap v6657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:40 smithi115 ceph-mon[35364]: pgmap v6658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:40 smithi181 ceph-mon[35869]: pgmap v6658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:40 smithi097 ceph-mon[32231]: pgmap v6658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11357.916717 secs 2024-07-26T11:08:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11357.916717 secs 2024-07-26T11:08:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11357.916717 secs 2024-07-26T11:08:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:42 smithi115 ceph-mon[35364]: pgmap v6659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:42 smithi181 ceph-mon[35869]: pgmap v6659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:42 smithi097 ceph-mon[32231]: pgmap v6659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:44 smithi115 ceph-mon[35364]: pgmap v6660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:44 smithi181 ceph-mon[35869]: pgmap v6660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:44 smithi097 ceph-mon[32231]: pgmap v6660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:46 smithi115 ceph-mon[35364]: pgmap v6661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11362.916890 secs 2024-07-26T11:08:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:46 smithi181 ceph-mon[35869]: pgmap v6661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11362.916890 secs 2024-07-26T11:08:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:46 smithi097 ceph-mon[32231]: pgmap v6661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11362.916890 secs 2024-07-26T11:08:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:48 smithi115 ceph-mon[35364]: pgmap v6662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:48 smithi181 ceph-mon[35869]: pgmap v6662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:48 smithi097 ceph-mon[32231]: pgmap v6662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:50 smithi115 ceph-mon[35364]: pgmap v6663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:50 smithi181 ceph-mon[35869]: pgmap v6663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:50 smithi097 ceph-mon[32231]: pgmap v6663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11367.917031 secs 2024-07-26T11:08:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11367.917031 secs 2024-07-26T11:08:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11367.917031 secs 2024-07-26T11:08:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:52.495+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:52.497+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:08:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:52.497+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:08:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:52.497+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:08:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:08:52.497+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:52 smithi115 ceph-mon[35364]: pgmap v6664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:52 smithi181 ceph-mon[35869]: pgmap v6664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:52 smithi097 ceph-mon[32231]: pgmap v6664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:54 smithi115 ceph-mon[35364]: pgmap v6665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:54 smithi181 ceph-mon[35869]: pgmap v6665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:54 smithi097 ceph-mon[32231]: pgmap v6665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:08:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:56 smithi181 ceph-mon[35869]: pgmap v6666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11372.917168 secs 2024-07-26T11:08:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:56 smithi097 ceph-mon[32231]: pgmap v6666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11372.917168 secs 2024-07-26T11:08:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:56 smithi115 ceph-mon[35364]: pgmap v6666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11372.917168 secs 2024-07-26T11:08:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:08:58 smithi097 ceph-mon[32231]: pgmap v6667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:08:58 smithi115 ceph-mon[35364]: pgmap v6667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:08:58 smithi181 ceph-mon[35869]: pgmap v6667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:00 smithi097 ceph-mon[32231]: pgmap v6668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:00 smithi115 ceph-mon[35364]: pgmap v6668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:00 smithi181 ceph-mon[35869]: pgmap v6668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11377.917307 secs 2024-07-26T11:09:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11377.917307 secs 2024-07-26T11:09:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11377.917307 secs 2024-07-26T11:09:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:02 smithi115 ceph-mon[35364]: pgmap v6669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:02 smithi181 ceph-mon[35869]: pgmap v6669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:02 smithi097 ceph-mon[32231]: pgmap v6669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:04 smithi115 ceph-mon[35364]: pgmap v6670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:04 smithi181 ceph-mon[35869]: pgmap v6670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:04 smithi097 ceph-mon[32231]: pgmap v6670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:06 smithi115 ceph-mon[35364]: pgmap v6671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11382.917449 secs 2024-07-26T11:09:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:06 smithi181 ceph-mon[35869]: pgmap v6671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11382.917449 secs 2024-07-26T11:09:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:06 smithi097 ceph-mon[32231]: pgmap v6671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11382.917449 secs 2024-07-26T11:09:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:07.504+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:07.505+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:09:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:07.505+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:09:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:07.505+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:09:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:07.505+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:07.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:07.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:07.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:07.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:07.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:08.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:08.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:08.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:09:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:08 smithi115 ceph-mon[35364]: pgmap v6672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:08 smithi115 ceph-mon[35364]: pgmap v6673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:08 smithi115 ceph-mon[35364]: pgmap v6674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:09:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:08 smithi181 ceph-mon[35869]: pgmap v6672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:08 smithi181 ceph-mon[35869]: pgmap v6673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:08 smithi181 ceph-mon[35869]: pgmap v6674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:09:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:08 smithi097 ceph-mon[32231]: pgmap v6672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:08 smithi097 ceph-mon[32231]: pgmap v6673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:08 smithi097 ceph-mon[32231]: pgmap v6674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:09:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:09 smithi115 ceph-mon[35364]: pgmap v6675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:09:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:09 smithi181 ceph-mon[35869]: pgmap v6675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:09:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:09 smithi097 ceph-mon[32231]: pgmap v6675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:09:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11387.917547 secs 2024-07-26T11:09:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:11 smithi115 ceph-mon[35364]: pgmap v6676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:09:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11387.917547 secs 2024-07-26T11:09:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:11 smithi181 ceph-mon[35869]: pgmap v6676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:09:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11387.917547 secs 2024-07-26T11:09:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:11 smithi097 ceph-mon[32231]: pgmap v6676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:09:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:13 smithi115 ceph-mon[35364]: pgmap v6677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:09:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:13 smithi181 ceph-mon[35869]: pgmap v6677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:09:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:13 smithi097 ceph-mon[32231]: pgmap v6677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:09:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:15 smithi115 ceph-mon[35364]: pgmap v6678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:09:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:15 smithi181 ceph-mon[35869]: pgmap v6678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:09:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:15 smithi097 ceph-mon[32231]: pgmap v6678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:09:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11392.917650 secs 2024-07-26T11:09:17.266 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11392.917650 secs 2024-07-26T11:09:17.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11392.917650 secs 2024-07-26T11:09:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:17 smithi115 ceph-mon[35364]: pgmap v6679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:17 smithi181 ceph-mon[35869]: pgmap v6679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:17 smithi097 ceph-mon[32231]: pgmap v6679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:19 smithi115 ceph-mon[35364]: pgmap v6680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:19 smithi181 ceph-mon[35869]: pgmap v6680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:19 smithi097 ceph-mon[32231]: pgmap v6680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:21.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11397.917801 secs 2024-07-26T11:09:21.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11397.917801 secs 2024-07-26T11:09:21.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11397.917801 secs 2024-07-26T11:09:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:21 smithi115 ceph-mon[35364]: pgmap v6681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:21 smithi181 ceph-mon[35869]: pgmap v6681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:21 smithi097 ceph-mon[32231]: pgmap v6681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:22.506+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:22.508+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:09:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:22.508+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:09:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:22.508+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:09:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:22.508+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:23 smithi115 ceph-mon[35364]: pgmap v6682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:23 smithi181 ceph-mon[35869]: pgmap v6682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:23 smithi097 ceph-mon[32231]: pgmap v6682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:25 smithi115 ceph-mon[35364]: pgmap v6683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11402.917962 secs 2024-07-26T11:09:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:25 smithi181 ceph-mon[35869]: pgmap v6683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11402.917962 secs 2024-07-26T11:09:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:25 smithi097 ceph-mon[32231]: pgmap v6683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11402.917962 secs 2024-07-26T11:09:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:27 smithi115 ceph-mon[35364]: pgmap v6684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:27 smithi181 ceph-mon[35869]: pgmap v6684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:27 smithi097 ceph-mon[32231]: pgmap v6684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:29 smithi115 ceph-mon[35364]: pgmap v6685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:29 smithi181 ceph-mon[35869]: pgmap v6685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:29 smithi097 ceph-mon[32231]: pgmap v6685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11407.918108 secs 2024-07-26T11:09:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11407.918108 secs 2024-07-26T11:09:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11407.918108 secs 2024-07-26T11:09:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:31 smithi115 ceph-mon[35364]: pgmap v6686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:31 smithi181 ceph-mon[35869]: pgmap v6686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:31 smithi097 ceph-mon[32231]: pgmap v6686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:33 smithi115 ceph-mon[35364]: pgmap v6687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:33 smithi181 ceph-mon[35869]: pgmap v6687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:33 smithi097 ceph-mon[32231]: pgmap v6687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:35 smithi115 ceph-mon[35364]: pgmap v6688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11412.918296 secs 2024-07-26T11:09:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:35 smithi181 ceph-mon[35869]: pgmap v6688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11412.918296 secs 2024-07-26T11:09:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:35 smithi097 ceph-mon[32231]: pgmap v6688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11412.918296 secs 2024-07-26T11:09:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:37.514+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:37.515+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:09:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:37.515+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:09:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:37.515+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:09:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:37.515+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:37 smithi115 ceph-mon[35364]: pgmap v6689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:09:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:37 smithi181 ceph-mon[35869]: pgmap v6689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:09:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:37 smithi097 ceph-mon[32231]: pgmap v6689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:09:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:39 smithi115 ceph-mon[35364]: pgmap v6690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:39 smithi181 ceph-mon[35869]: pgmap v6690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:39 smithi097 ceph-mon[32231]: pgmap v6690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11417.918403 secs 2024-07-26T11:09:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11417.918403 secs 2024-07-26T11:09:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11417.918403 secs 2024-07-26T11:09:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:41 smithi115 ceph-mon[35364]: pgmap v6691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:41 smithi181 ceph-mon[35869]: pgmap v6691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:41 smithi097 ceph-mon[32231]: pgmap v6691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:43 smithi115 ceph-mon[35364]: pgmap v6692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:43 smithi181 ceph-mon[35869]: pgmap v6692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:43 smithi097 ceph-mon[32231]: pgmap v6692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:45 smithi115 ceph-mon[35364]: pgmap v6693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11422.918502 secs 2024-07-26T11:09:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:45 smithi181 ceph-mon[35869]: pgmap v6693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11422.918502 secs 2024-07-26T11:09:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:45 smithi097 ceph-mon[32231]: pgmap v6693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11422.918502 secs 2024-07-26T11:09:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:47 smithi115 ceph-mon[35364]: pgmap v6694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:47 smithi181 ceph-mon[35869]: pgmap v6694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:47 smithi097 ceph-mon[32231]: pgmap v6694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:49 smithi115 ceph-mon[35364]: pgmap v6695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:49 smithi181 ceph-mon[35869]: pgmap v6695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:49 smithi097 ceph-mon[32231]: pgmap v6695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11427.918606 secs 2024-07-26T11:09:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11427.918606 secs 2024-07-26T11:09:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11427.918606 secs 2024-07-26T11:09:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:51 smithi115 ceph-mon[35364]: pgmap v6696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:51 smithi181 ceph-mon[35869]: pgmap v6696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:51 smithi097 ceph-mon[32231]: pgmap v6696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:52.517+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:52.519+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:09:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:52.519+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:09:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:52.519+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:09:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:09:52.519+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:53 smithi115 ceph-mon[35364]: pgmap v6697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:53 smithi181 ceph-mon[35869]: pgmap v6697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:53 smithi097 ceph-mon[32231]: pgmap v6697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:55 smithi115 ceph-mon[35364]: pgmap v6698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11432.918767 secs 2024-07-26T11:09:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:55 smithi181 ceph-mon[35869]: pgmap v6698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11432.918767 secs 2024-07-26T11:09:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:55 smithi097 ceph-mon[32231]: pgmap v6698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11432.918767 secs 2024-07-26T11:09:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:57 smithi115 ceph-mon[35364]: pgmap v6699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:57 smithi181 ceph-mon[35869]: pgmap v6699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:57 smithi097 ceph-mon[32231]: pgmap v6699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:09:59 smithi181 ceph-mon[35869]: pgmap v6700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:09:59 smithi097 ceph-mon[32231]: pgmap v6700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:09:59 smithi115 ceph-mon[35364]: pgmap v6700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11437.918909 secs 2024-07-26T11:10:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11437.918909 secs 2024-07-26T11:10:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11437.918909 secs 2024-07-26T11:10:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:01 smithi097 ceph-mon[32231]: pgmap v6701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:01 smithi115 ceph-mon[35364]: pgmap v6701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:01 smithi181 ceph-mon[35869]: pgmap v6701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:04 smithi115 ceph-mon[35364]: pgmap v6702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:04 smithi181 ceph-mon[35869]: pgmap v6702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:04 smithi097 ceph-mon[32231]: pgmap v6702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:06 smithi115 ceph-mon[35364]: pgmap v6703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11442.919056 secs 2024-07-26T11:10:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:06 smithi181 ceph-mon[35869]: pgmap v6703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11442.919056 secs 2024-07-26T11:10:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:06 smithi097 ceph-mon[32231]: pgmap v6703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11442.919056 secs 2024-07-26T11:10:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:07.521+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:07.523+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:10:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:07.523+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:10:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:07.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:07.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:08 smithi115 ceph-mon[35364]: pgmap v6704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:08 smithi181 ceph-mon[35869]: pgmap v6704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:08 smithi097 ceph-mon[32231]: pgmap v6704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:08.911+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:10:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:08.911+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:10:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:08.911+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:10:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:08.911+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:10:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:09 smithi115 ceph-mon[35364]: pgmap v6705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:10:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:09 smithi115 ceph-mon[35364]: pgmap v6706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:10:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:09 smithi181 ceph-mon[35869]: pgmap v6705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:10:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:09 smithi181 ceph-mon[35869]: pgmap v6706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:09.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:10:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:09 smithi097 ceph-mon[32231]: pgmap v6705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:10:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:09 smithi097 ceph-mon[32231]: pgmap v6706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:10:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11447.919272 secs 2024-07-26T11:10:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:11 smithi115 ceph-mon[35364]: pgmap v6707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11447.919272 secs 2024-07-26T11:10:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:11 smithi181 ceph-mon[35869]: pgmap v6707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11447.919272 secs 2024-07-26T11:10:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:11 smithi097 ceph-mon[32231]: pgmap v6707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:13 smithi115 ceph-mon[35364]: pgmap v6708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:10:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:13 smithi181 ceph-mon[35869]: pgmap v6708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:10:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:13 smithi097 ceph-mon[32231]: pgmap v6708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:10:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:15 smithi115 ceph-mon[35364]: pgmap v6709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:15 smithi181 ceph-mon[35869]: pgmap v6709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:15 smithi097 ceph-mon[32231]: pgmap v6709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11452.919374 secs 2024-07-26T11:10:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11452.919374 secs 2024-07-26T11:10:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11452.919374 secs 2024-07-26T11:10:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:17 smithi097 ceph-mon[32231]: pgmap v6710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:17 smithi115 ceph-mon[35364]: pgmap v6710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:17 smithi181 ceph-mon[35869]: pgmap v6710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:19 smithi097 ceph-mon[32231]: pgmap v6711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:10:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:19 smithi115 ceph-mon[35364]: pgmap v6711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:10:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:19 smithi181 ceph-mon[35869]: pgmap v6711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:10:21.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11457.919475 secs 2024-07-26T11:10:21.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:21 smithi115 ceph-mon[35364]: pgmap v6712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:21.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11457.919475 secs 2024-07-26T11:10:21.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:21 smithi181 ceph-mon[35869]: pgmap v6712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11457.919475 secs 2024-07-26T11:10:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:21 smithi097 ceph-mon[32231]: pgmap v6712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:22.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:22.525+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:10:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:22.525+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:10:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:22.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:22.525+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:23 smithi115 ceph-mon[35364]: pgmap v6713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:23 smithi181 ceph-mon[35869]: pgmap v6713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:23 smithi097 ceph-mon[32231]: pgmap v6713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:25 smithi115 ceph-mon[35364]: pgmap v6714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:25 smithi181 ceph-mon[35869]: pgmap v6714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:25 smithi097 ceph-mon[32231]: pgmap v6714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11462.919614 secs 2024-07-26T11:10:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11462.919614 secs 2024-07-26T11:10:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11462.919614 secs 2024-07-26T11:10:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:27 smithi115 ceph-mon[35364]: pgmap v6715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:27 smithi181 ceph-mon[35869]: pgmap v6715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:27 smithi097 ceph-mon[32231]: pgmap v6715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:29 smithi115 ceph-mon[35364]: pgmap v6716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:29 smithi181 ceph-mon[35869]: pgmap v6716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:29 smithi097 ceph-mon[32231]: pgmap v6716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11467.919773 secs 2024-07-26T11:10:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:31 smithi115 ceph-mon[35364]: pgmap v6717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11467.919773 secs 2024-07-26T11:10:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:31 smithi181 ceph-mon[35869]: pgmap v6717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11467.919773 secs 2024-07-26T11:10:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:31 smithi097 ceph-mon[32231]: pgmap v6717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:33 smithi115 ceph-mon[35364]: pgmap v6718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:33 smithi181 ceph-mon[35869]: pgmap v6718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:33 smithi097 ceph-mon[32231]: pgmap v6718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:35 smithi115 ceph-mon[35364]: pgmap v6719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:35 smithi181 ceph-mon[35869]: pgmap v6719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:35 smithi097 ceph-mon[32231]: pgmap v6719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11472.919930 secs 2024-07-26T11:10:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11472.919930 secs 2024-07-26T11:10:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11472.919930 secs 2024-07-26T11:10:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:37.532+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:37.534+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:10:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:37.534+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:10:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:37.534+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:37.534+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:37 smithi115 ceph-mon[35364]: pgmap v6720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:37 smithi181 ceph-mon[35869]: pgmap v6720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:37 smithi097 ceph-mon[32231]: pgmap v6720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:39 smithi115 ceph-mon[35364]: pgmap v6721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:39 smithi181 ceph-mon[35869]: pgmap v6721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:39 smithi097 ceph-mon[32231]: pgmap v6721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11477.920088 secs 2024-07-26T11:10:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:41 smithi115 ceph-mon[35364]: pgmap v6722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11477.920088 secs 2024-07-26T11:10:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:41 smithi181 ceph-mon[35869]: pgmap v6722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11477.920088 secs 2024-07-26T11:10:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:41 smithi097 ceph-mon[32231]: pgmap v6722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:43 smithi115 ceph-mon[35364]: pgmap v6723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:43 smithi181 ceph-mon[35869]: pgmap v6723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:43 smithi097 ceph-mon[32231]: pgmap v6723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:45 smithi115 ceph-mon[35364]: pgmap v6724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:45 smithi181 ceph-mon[35869]: pgmap v6724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:45 smithi097 ceph-mon[32231]: pgmap v6724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11482.920285 secs 2024-07-26T11:10:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11482.920285 secs 2024-07-26T11:10:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11482.920285 secs 2024-07-26T11:10:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:47 smithi115 ceph-mon[35364]: pgmap v6725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:47 smithi181 ceph-mon[35869]: pgmap v6725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:47 smithi097 ceph-mon[32231]: pgmap v6725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:49 smithi115 ceph-mon[35364]: pgmap v6726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:49 smithi181 ceph-mon[35869]: pgmap v6726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:49 smithi097 ceph-mon[32231]: pgmap v6726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11487.920427 secs 2024-07-26T11:10:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:51 smithi115 ceph-mon[35364]: pgmap v6727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11487.920427 secs 2024-07-26T11:10:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:51 smithi181 ceph-mon[35869]: pgmap v6727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11487.920427 secs 2024-07-26T11:10:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:51 smithi097 ceph-mon[32231]: pgmap v6727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:52.536+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:52.537+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:10:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:52.537+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:10:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:52.537+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:10:52.537+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:53 smithi115 ceph-mon[35364]: pgmap v6728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:53 smithi181 ceph-mon[35869]: pgmap v6728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:53 smithi097 ceph-mon[32231]: pgmap v6728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:55 smithi115 ceph-mon[35364]: pgmap v6729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:55 smithi181 ceph-mon[35869]: pgmap v6729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:55 smithi097 ceph-mon[32231]: pgmap v6729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11492.920565 secs 2024-07-26T11:10:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11492.920565 secs 2024-07-26T11:10:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11492.920565 secs 2024-07-26T11:10:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:57 smithi115 ceph-mon[35364]: pgmap v6730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:57 smithi181 ceph-mon[35869]: pgmap v6730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:57 smithi097 ceph-mon[32231]: pgmap v6730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:10:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:10:59 smithi115 ceph-mon[35364]: pgmap v6731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:10:59 smithi181 ceph-mon[35869]: pgmap v6731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:10:59 smithi097 ceph-mon[32231]: pgmap v6731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11497.920710 secs 2024-07-26T11:11:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:01 smithi115 ceph-mon[35364]: pgmap v6732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11497.920710 secs 2024-07-26T11:11:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:01 smithi181 ceph-mon[35869]: pgmap v6732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11497.920710 secs 2024-07-26T11:11:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:01 smithi097 ceph-mon[32231]: pgmap v6732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:03 smithi115 ceph-mon[35364]: pgmap v6733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:03 smithi181 ceph-mon[35869]: pgmap v6733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:03 smithi097 ceph-mon[32231]: pgmap v6733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:05 smithi115 ceph-mon[35364]: pgmap v6734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:11:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:05 smithi181 ceph-mon[35869]: pgmap v6734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:11:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:05 smithi097 ceph-mon[32231]: pgmap v6734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:11:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11502.920851 secs 2024-07-26T11:11:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11502.920851 secs 2024-07-26T11:11:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11502.920851 secs 2024-07-26T11:11:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:07.544+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:07.545+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:11:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:07.545+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:11:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:07.545+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:07.545+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:07 smithi115 ceph-mon[35364]: pgmap v6735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:07 smithi181 ceph-mon[35869]: pgmap v6735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:07 smithi097 ceph-mon[32231]: pgmap v6735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:09 smithi181 ceph-mon[35869]: pgmap v6736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:09.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:09 smithi097 ceph-mon[32231]: pgmap v6736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:09 smithi115 ceph-mon[35364]: pgmap v6736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:10 smithi097 ceph-mon[32231]: pgmap v6737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:11:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:10 smithi097 ceph-mon[32231]: pgmap v6738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:11:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:11:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:10 smithi115 ceph-mon[35364]: pgmap v6737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:11:10.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:10 smithi115 ceph-mon[35364]: pgmap v6738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:11:10.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:11:10.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:10.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:10.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:10 smithi181 ceph-mon[35869]: pgmap v6737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:11:10.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:10 smithi181 ceph-mon[35869]: pgmap v6738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:11:10.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:11:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11507.921017 secs 2024-07-26T11:11:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11507.921017 secs 2024-07-26T11:11:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11507.921017 secs 2024-07-26T11:11:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:12 smithi115 ceph-mon[35364]: pgmap v6739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:12 smithi181 ceph-mon[35869]: pgmap v6739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:12 smithi097 ceph-mon[32231]: pgmap v6739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:14 smithi115 ceph-mon[35364]: pgmap v6740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:11:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:14 smithi181 ceph-mon[35869]: pgmap v6740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:11:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:14 smithi097 ceph-mon[32231]: pgmap v6740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:11:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11512.921152 secs 2024-07-26T11:11:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:16 smithi115 ceph-mon[35364]: pgmap v6741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11512.921152 secs 2024-07-26T11:11:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:16 smithi181 ceph-mon[35869]: pgmap v6741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11512.921152 secs 2024-07-26T11:11:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:16 smithi097 ceph-mon[32231]: pgmap v6741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:18 smithi115 ceph-mon[35364]: pgmap v6742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:18 smithi181 ceph-mon[35869]: pgmap v6742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:18 smithi097 ceph-mon[32231]: pgmap v6742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:20 smithi115 ceph-mon[35364]: pgmap v6743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:11:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:20 smithi181 ceph-mon[35869]: pgmap v6743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:11:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:20 smithi097 ceph-mon[32231]: pgmap v6743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:11:21.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11517.921295 secs 2024-07-26T11:11:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11517.921295 secs 2024-07-26T11:11:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11517.921295 secs 2024-07-26T11:11:22.549 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:22.546+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:22.548+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:11:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:22.548+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:11:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:22.548+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:22.548+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:22 smithi115 ceph-mon[35364]: pgmap v6744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:22 smithi181 ceph-mon[35869]: pgmap v6744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:23.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:22 smithi097 ceph-mon[32231]: pgmap v6744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:24 smithi115 ceph-mon[35364]: pgmap v6745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:11:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:24 smithi181 ceph-mon[35869]: pgmap v6745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:11:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:24 smithi097 ceph-mon[32231]: pgmap v6745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:11:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:26 smithi115 ceph-mon[35364]: 1 slow requests, 1 included below; oldest blocked for > 11522.921413 secs 2024-07-26T11:11:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:26 smithi115 ceph-mon[35364]: slow request 11522.921413 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:11:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:26 smithi115 ceph-mon[35364]: pgmap v6746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:11:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:26 smithi181 ceph-mon[35869]: 1 slow requests, 1 included below; oldest blocked for > 11522.921413 secs 2024-07-26T11:11:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:26 smithi181 ceph-mon[35869]: slow request 11522.921413 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:11:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:26 smithi181 ceph-mon[35869]: pgmap v6746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:11:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:26 smithi097 ceph-mon[32231]: 1 slow requests, 1 included below; oldest blocked for > 11522.921413 secs 2024-07-26T11:11:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:26 smithi097 ceph-mon[32231]: slow request 11522.921413 seconds old, received at 2024-07-26T07:59:22.006071+0000: client_request(client.15360:123670 rmdir #0x10000000399/blog-13 2024-07-26T07:59:22.005533+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:11:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:26 smithi097 ceph-mon[32231]: pgmap v6746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:11:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:28 smithi115 ceph-mon[35364]: pgmap v6747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:11:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:28 smithi181 ceph-mon[35869]: pgmap v6747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:11:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:28 smithi097 ceph-mon[32231]: pgmap v6747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:11:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:30 smithi115 ceph-mon[35364]: pgmap v6748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:30 smithi181 ceph-mon[35869]: pgmap v6748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:30 smithi097 ceph-mon[32231]: pgmap v6748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11527.921514 secs 2024-07-26T11:11:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11527.921514 secs 2024-07-26T11:11:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11527.921514 secs 2024-07-26T11:11:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:32 smithi115 ceph-mon[35364]: pgmap v6749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:32 smithi181 ceph-mon[35869]: pgmap v6749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:32 smithi097 ceph-mon[32231]: pgmap v6749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:33 smithi115 ceph-mon[35364]: pgmap v6750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:33 smithi181 ceph-mon[35869]: pgmap v6750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:33 smithi097 ceph-mon[32231]: pgmap v6750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11532.921677 secs 2024-07-26T11:11:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:36 smithi115 ceph-mon[35364]: pgmap v6751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11532.921677 secs 2024-07-26T11:11:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:36 smithi181 ceph-mon[35869]: pgmap v6751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11532.921677 secs 2024-07-26T11:11:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:36 smithi097 ceph-mon[32231]: pgmap v6751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:37.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:37.555+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:37.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:37.557+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:11:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:37.557+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:11:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:37.557+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:37.557+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:38 smithi115 ceph-mon[35364]: pgmap v6752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:38 smithi181 ceph-mon[35869]: pgmap v6752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:38 smithi097 ceph-mon[32231]: pgmap v6752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:40 smithi115 ceph-mon[35364]: pgmap v6753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:40 smithi181 ceph-mon[35869]: pgmap v6753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:40 smithi097 ceph-mon[32231]: pgmap v6753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11537.921810 secs 2024-07-26T11:11:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11537.921810 secs 2024-07-26T11:11:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11537.921810 secs 2024-07-26T11:11:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:42 smithi115 ceph-mon[35364]: pgmap v6754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:42 smithi181 ceph-mon[35869]: pgmap v6754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:42 smithi097 ceph-mon[32231]: pgmap v6754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:44 smithi115 ceph-mon[35364]: pgmap v6755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:44 smithi181 ceph-mon[35869]: pgmap v6755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:44 smithi097 ceph-mon[32231]: pgmap v6755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11542.921950 secs 2024-07-26T11:11:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:46 smithi115 ceph-mon[35364]: pgmap v6756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11542.921950 secs 2024-07-26T11:11:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:46 smithi181 ceph-mon[35869]: pgmap v6756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11542.921950 secs 2024-07-26T11:11:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:46 smithi097 ceph-mon[32231]: pgmap v6756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:47 smithi097 ceph-mon[32231]: pgmap v6757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:47 smithi115 ceph-mon[35364]: pgmap v6757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:47 smithi181 ceph-mon[35869]: pgmap v6757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:50 smithi115 ceph-mon[35364]: pgmap v6758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:50 smithi181 ceph-mon[35869]: pgmap v6758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:50 smithi097 ceph-mon[32231]: pgmap v6758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:51.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11547.922085 secs 2024-07-26T11:11:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11547.922085 secs 2024-07-26T11:11:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11547.922085 secs 2024-07-26T11:11:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:52.559+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:52.560+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:11:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:52.560+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:11:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:52.560+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:11:52.560+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:52 smithi115 ceph-mon[35364]: pgmap v6759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:52 smithi181 ceph-mon[35869]: pgmap v6759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:52 smithi097 ceph-mon[32231]: pgmap v6759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:11:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:54 smithi115 ceph-mon[35364]: pgmap v6760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:54 smithi181 ceph-mon[35869]: pgmap v6760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:54 smithi097 ceph-mon[32231]: pgmap v6760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11552.922217 secs 2024-07-26T11:11:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:56 smithi115 ceph-mon[35364]: pgmap v6761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11552.922217 secs 2024-07-26T11:11:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:56 smithi181 ceph-mon[35869]: pgmap v6761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11552.922217 secs 2024-07-26T11:11:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:56 smithi097 ceph-mon[32231]: pgmap v6761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:11:58 smithi115 ceph-mon[35364]: pgmap v6762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:11:58 smithi181 ceph-mon[35869]: pgmap v6762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:11:58 smithi097 ceph-mon[32231]: pgmap v6762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:00 smithi115 ceph-mon[35364]: pgmap v6763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:00 smithi181 ceph-mon[35869]: pgmap v6763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:00 smithi097 ceph-mon[32231]: pgmap v6763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11557.922365 secs 2024-07-26T11:12:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11557.922365 secs 2024-07-26T11:12:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11557.922365 secs 2024-07-26T11:12:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:02 smithi115 ceph-mon[35364]: pgmap v6764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:02 smithi181 ceph-mon[35869]: pgmap v6764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:02 smithi097 ceph-mon[32231]: pgmap v6764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:03 smithi115 ceph-mon[35364]: pgmap v6765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:03 smithi181 ceph-mon[35869]: pgmap v6765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:03 smithi097 ceph-mon[32231]: pgmap v6765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11562.922504 secs 2024-07-26T11:12:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:06 smithi115 ceph-mon[35364]: pgmap v6766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11562.922504 secs 2024-07-26T11:12:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:06 smithi181 ceph-mon[35869]: pgmap v6766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11562.922504 secs 2024-07-26T11:12:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:06 smithi097 ceph-mon[32231]: pgmap v6766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:07.567+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:07.568+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:12:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:07.568+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:12:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:07.568+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:12:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:07.568+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:08 smithi115 ceph-mon[35364]: pgmap v6767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:08 smithi181 ceph-mon[35869]: pgmap v6767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:08 smithi097 ceph-mon[32231]: pgmap v6767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:10 smithi115 ceph-mon[35364]: pgmap v6768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:10 smithi181 ceph-mon[35869]: pgmap v6768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:10 smithi097 ceph-mon[32231]: pgmap v6768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11567.922599 secs 2024-07-26T11:12:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:12:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11567.922599 secs 2024-07-26T11:12:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:12:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11567.922599 secs 2024-07-26T11:12:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:12:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:12 smithi115 ceph-mon[35364]: pgmap v6769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:12:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:12 smithi115 ceph-mon[35364]: pgmap v6770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:12 smithi181 ceph-mon[35869]: pgmap v6769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:12:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:12 smithi181 ceph-mon[35869]: pgmap v6770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:12 smithi097 ceph-mon[32231]: pgmap v6769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:12:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:12 smithi097 ceph-mon[32231]: pgmap v6770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:14 smithi115 ceph-mon[35364]: pgmap v6771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:14 smithi181 ceph-mon[35869]: pgmap v6771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:14 smithi097 ceph-mon[32231]: pgmap v6771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11572.922693 secs 2024-07-26T11:12:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:16 smithi115 ceph-mon[35364]: pgmap v6772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11572.922693 secs 2024-07-26T11:12:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:16 smithi181 ceph-mon[35869]: pgmap v6772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11572.922693 secs 2024-07-26T11:12:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:16 smithi097 ceph-mon[32231]: pgmap v6772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:18 smithi115 ceph-mon[35364]: pgmap v6773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:12:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:18 smithi181 ceph-mon[35869]: pgmap v6773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:12:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:18 smithi097 ceph-mon[32231]: pgmap v6773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:12:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:20 smithi115 ceph-mon[35364]: pgmap v6774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:12:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:20 smithi181 ceph-mon[35869]: pgmap v6774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:12:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:20 smithi097 ceph-mon[32231]: pgmap v6774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:12:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11577.922843 secs 2024-07-26T11:12:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11577.922843 secs 2024-07-26T11:12:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11577.922843 secs 2024-07-26T11:12:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:22.569+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:22.571+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:12:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:22.571+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:12:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:22.571+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:12:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:22.571+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:22 smithi115 ceph-mon[35364]: pgmap v6775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:12:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:22 smithi181 ceph-mon[35869]: pgmap v6775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:12:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:22 smithi097 ceph-mon[32231]: pgmap v6775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:12:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:24 smithi115 ceph-mon[35364]: pgmap v6776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:12:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:24 smithi181 ceph-mon[35869]: pgmap v6776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:12:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:24 smithi097 ceph-mon[32231]: pgmap v6776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:12:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11582.922986 secs 2024-07-26T11:12:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:26 smithi115 ceph-mon[35364]: pgmap v6777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:12:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11582.922986 secs 2024-07-26T11:12:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:26 smithi181 ceph-mon[35869]: pgmap v6777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:12:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11582.922986 secs 2024-07-26T11:12:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:26 smithi097 ceph-mon[32231]: pgmap v6777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:12:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:28 smithi115 ceph-mon[35364]: pgmap v6778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:12:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:28 smithi181 ceph-mon[35869]: pgmap v6778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:12:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:28 smithi097 ceph-mon[32231]: pgmap v6778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:12:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:30 smithi181 ceph-mon[35869]: pgmap v6779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:12:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:30 smithi097 ceph-mon[32231]: pgmap v6779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:12:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:30 smithi115 ceph-mon[35364]: pgmap v6779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:12:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11587.923069 secs 2024-07-26T11:12:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11587.923069 secs 2024-07-26T11:12:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11587.923069 secs 2024-07-26T11:12:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:32 smithi097 ceph-mon[32231]: pgmap v6780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:32 smithi115 ceph-mon[35364]: pgmap v6780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:32 smithi181 ceph-mon[35869]: pgmap v6780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:34 smithi097 ceph-mon[32231]: pgmap v6781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:34 smithi115 ceph-mon[35364]: pgmap v6781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:34 smithi181 ceph-mon[35869]: pgmap v6781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11592.923212 secs 2024-07-26T11:12:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:36 smithi115 ceph-mon[35364]: pgmap v6782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11592.923212 secs 2024-07-26T11:12:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:36 smithi181 ceph-mon[35869]: pgmap v6782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11592.923212 secs 2024-07-26T11:12:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:36 smithi097 ceph-mon[32231]: pgmap v6782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:37.578+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:37.579+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:12:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:37.579+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:12:37.582 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:37.579+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:12:37.582 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:37.580+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:38 smithi115 ceph-mon[35364]: pgmap v6783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:38 smithi181 ceph-mon[35869]: pgmap v6783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:38 smithi097 ceph-mon[32231]: pgmap v6783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:40 smithi115 ceph-mon[35364]: pgmap v6784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:12:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:40 smithi181 ceph-mon[35869]: pgmap v6784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:12:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:40 smithi097 ceph-mon[32231]: pgmap v6784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:12:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11597.923346 secs 2024-07-26T11:12:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11597.923346 secs 2024-07-26T11:12:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11597.923346 secs 2024-07-26T11:12:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:42 smithi115 ceph-mon[35364]: pgmap v6785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:42 smithi181 ceph-mon[35869]: pgmap v6785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:42 smithi097 ceph-mon[32231]: pgmap v6785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:44 smithi115 ceph-mon[35364]: pgmap v6786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:44 smithi181 ceph-mon[35869]: pgmap v6786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:44 smithi097 ceph-mon[32231]: pgmap v6786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11602.923480 secs 2024-07-26T11:12:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:46 smithi115 ceph-mon[35364]: pgmap v6787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11602.923480 secs 2024-07-26T11:12:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:46 smithi181 ceph-mon[35869]: pgmap v6787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11602.923480 secs 2024-07-26T11:12:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:46 smithi097 ceph-mon[32231]: pgmap v6787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:48 smithi115 ceph-mon[35364]: pgmap v6788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:48 smithi181 ceph-mon[35869]: pgmap v6788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:48 smithi097 ceph-mon[32231]: pgmap v6788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:50 smithi115 ceph-mon[35364]: pgmap v6789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:12:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:50 smithi181 ceph-mon[35869]: pgmap v6789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:12:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:50 smithi097 ceph-mon[32231]: pgmap v6789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:12:51.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11607.923644 secs 2024-07-26T11:12:51.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11607.923644 secs 2024-07-26T11:12:51.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11607.923644 secs 2024-07-26T11:12:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:52.582+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:52.583+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:12:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:52.583+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:12:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:52.583+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:12:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:12:52.584+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:52 smithi115 ceph-mon[35364]: pgmap v6790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:52 smithi181 ceph-mon[35869]: pgmap v6790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:52 smithi097 ceph-mon[32231]: pgmap v6790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:12:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:54 smithi115 ceph-mon[35364]: pgmap v6791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:54 smithi181 ceph-mon[35869]: pgmap v6791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:54 smithi097 ceph-mon[32231]: pgmap v6791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11612.923799 secs 2024-07-26T11:12:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:56 smithi115 ceph-mon[35364]: pgmap v6792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11612.923799 secs 2024-07-26T11:12:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:56 smithi181 ceph-mon[35869]: pgmap v6792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11612.923799 secs 2024-07-26T11:12:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:56 smithi097 ceph-mon[32231]: pgmap v6792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:12:58 smithi115 ceph-mon[35364]: pgmap v6793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:12:58 smithi181 ceph-mon[35869]: pgmap v6793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:12:58 smithi097 ceph-mon[32231]: pgmap v6793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:00 smithi115 ceph-mon[35364]: pgmap v6794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:13:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:00 smithi181 ceph-mon[35869]: pgmap v6794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:13:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:00 smithi097 ceph-mon[32231]: pgmap v6794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:13:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11617.923941 secs 2024-07-26T11:13:01.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11617.923941 secs 2024-07-26T11:13:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11617.923941 secs 2024-07-26T11:13:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:02 smithi115 ceph-mon[35364]: pgmap v6795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:02 smithi181 ceph-mon[35869]: pgmap v6795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:02 smithi097 ceph-mon[32231]: pgmap v6795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:04 smithi115 ceph-mon[35364]: pgmap v6796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:04 smithi181 ceph-mon[35869]: pgmap v6796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:04 smithi097 ceph-mon[32231]: pgmap v6796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:06.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.915+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 msgr2=0x7ff3d40b5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T11:13:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.915+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 msgr2=0x7ff3d40b5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:13:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.915+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e0082000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.915+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e0082000 comp rx=0 tx=0).stop 2024-07-26T11:13:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.915+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 msgr2=0x7ff3d40b5d00 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T11:13:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.915+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 msgr2=0x7ff3d40b5d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:13:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.915+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b58f0 0x7ff3d40b5d00 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.915+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.916+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:06.917+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005d50 tx=0x7ff3e0059000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11622.924090 secs 2024-07-26T11:13:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:06 smithi115 ceph-mon[35364]: pgmap v6797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11622.924090 secs 2024-07-26T11:13:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:06 smithi181 ceph-mon[35869]: pgmap v6797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11622.924090 secs 2024-07-26T11:13:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:06 smithi097 ceph-mon[32231]: pgmap v6797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:07.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:07.585+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:07.586+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:13:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:07.586+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:13:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:07.586+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:07.586+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:08.943+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:13:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:08.943+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:13:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:08.944+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:13:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:08.944+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:13:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:08 smithi115 ceph-mon[35364]: pgmap v6798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:08 smithi181 ceph-mon[35869]: pgmap v6798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:08 smithi097 ceph-mon[32231]: pgmap v6798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:10 smithi115 ceph-mon[35364]: pgmap v6799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:13:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:10 smithi181 ceph-mon[35869]: pgmap v6799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:13:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:10 smithi097 ceph-mon[32231]: pgmap v6799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:13:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11627.924279 secs 2024-07-26T11:13:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11627.924279 secs 2024-07-26T11:13:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11627.924279 secs 2024-07-26T11:13:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: pgmap v6800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:12.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: pgmap v6800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:12.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:12.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:12.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:13:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: pgmap v6800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:13.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:13.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:13.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:13.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:13.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:13:13.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:13.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:13.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:13:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:13 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:13:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:13 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:13:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:13 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:13:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:13 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:13:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:13 smithi115 ceph-mon[35364]: pgmap v6801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:13:13.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:13 smithi115 ceph-mon[35364]: pgmap v6802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:13 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:13:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:13 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:13:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:13 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:13:13.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:13 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:13:13.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:13 smithi181 ceph-mon[35869]: pgmap v6801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:13:13.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:13 smithi181 ceph-mon[35869]: pgmap v6802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:13 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:13:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:13 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:13:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:13 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:13:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:13 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:13:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:13 smithi097 ceph-mon[32231]: pgmap v6801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:13:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:13 smithi097 ceph-mon[32231]: pgmap v6802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:15 smithi115 ceph-mon[35364]: pgmap v6803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:15 smithi181 ceph-mon[35869]: pgmap v6803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:15 smithi097 ceph-mon[32231]: pgmap v6803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11632.924422 secs 2024-07-26T11:13:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11632.924422 secs 2024-07-26T11:13:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11632.924422 secs 2024-07-26T11:13:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:17 smithi115 ceph-mon[35364]: pgmap v6804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:17 smithi181 ceph-mon[35869]: pgmap v6804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:17 smithi097 ceph-mon[32231]: pgmap v6804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:18 smithi181 ceph-mon[35869]: pgmap v6805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:13:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:18 smithi097 ceph-mon[32231]: pgmap v6805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:13:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:18 smithi115 ceph-mon[35364]: pgmap v6805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:13:21.928 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11637.924566 secs 2024-07-26T11:13:21.929 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:21 smithi115 ceph-mon[35364]: pgmap v6806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11637.924566 secs 2024-07-26T11:13:21.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:21 smithi181 ceph-mon[35869]: pgmap v6806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11637.924566 secs 2024-07-26T11:13:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:21 smithi097 ceph-mon[32231]: pgmap v6806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:22.587+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:22.589+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:13:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:22.589+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:13:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:22.589+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:22.589+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:23 smithi115 ceph-mon[35364]: pgmap v6807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:13:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:23 smithi181 ceph-mon[35869]: pgmap v6807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:13:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:23 smithi097 ceph-mon[32231]: pgmap v6807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:13:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:25 smithi115 ceph-mon[35364]: pgmap v6808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:25 smithi181 ceph-mon[35869]: pgmap v6808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:25 smithi097 ceph-mon[32231]: pgmap v6808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11642.924666 secs 2024-07-26T11:13:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11642.924666 secs 2024-07-26T11:13:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11642.924666 secs 2024-07-26T11:13:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:27 smithi115 ceph-mon[35364]: pgmap v6809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:27 smithi181 ceph-mon[35869]: pgmap v6809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:27 smithi097 ceph-mon[32231]: pgmap v6809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:29 smithi115 ceph-mon[35364]: pgmap v6810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:29 smithi181 ceph-mon[35869]: pgmap v6810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:29 smithi097 ceph-mon[32231]: pgmap v6810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11647.924805 secs 2024-07-26T11:13:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:31 smithi115 ceph-mon[35364]: pgmap v6811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11647.924805 secs 2024-07-26T11:13:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:31 smithi181 ceph-mon[35869]: pgmap v6811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11647.924805 secs 2024-07-26T11:13:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:31 smithi097 ceph-mon[32231]: pgmap v6811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:33 smithi115 ceph-mon[35364]: pgmap v6812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:33 smithi181 ceph-mon[35869]: pgmap v6812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:33 smithi097 ceph-mon[32231]: pgmap v6812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:35 smithi115 ceph-mon[35364]: pgmap v6813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:35 smithi181 ceph-mon[35869]: pgmap v6813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:35 smithi097 ceph-mon[32231]: pgmap v6813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11652.924949 secs 2024-07-26T11:13:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11652.924949 secs 2024-07-26T11:13:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11652.924949 secs 2024-07-26T11:13:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:37.595+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:37.597+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:13:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:37.597+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:13:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:37.597+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:37.597+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:37 smithi115 ceph-mon[35364]: pgmap v6814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:37 smithi181 ceph-mon[35869]: pgmap v6814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:37 smithi097 ceph-mon[32231]: pgmap v6814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:13:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:39 smithi115 ceph-mon[35364]: pgmap v6815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:39 smithi181 ceph-mon[35869]: pgmap v6815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:39 smithi097 ceph-mon[32231]: pgmap v6815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11657.925124 secs 2024-07-26T11:13:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:41 smithi115 ceph-mon[35364]: pgmap v6816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11657.925124 secs 2024-07-26T11:13:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:41 smithi181 ceph-mon[35869]: pgmap v6816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11657.925124 secs 2024-07-26T11:13:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:41 smithi097 ceph-mon[32231]: pgmap v6816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:43 smithi115 ceph-mon[35364]: pgmap v6817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:43 smithi181 ceph-mon[35869]: pgmap v6817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:43 smithi097 ceph-mon[32231]: pgmap v6817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:45 smithi115 ceph-mon[35364]: pgmap v6818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:45 smithi181 ceph-mon[35869]: pgmap v6818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:45 smithi097 ceph-mon[32231]: pgmap v6818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11662.925268 secs 2024-07-26T11:13:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11662.925268 secs 2024-07-26T11:13:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11662.925268 secs 2024-07-26T11:13:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:47 smithi115 ceph-mon[35364]: pgmap v6819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:47 smithi181 ceph-mon[35869]: pgmap v6819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:47 smithi097 ceph-mon[32231]: pgmap v6819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:49 smithi115 ceph-mon[35364]: pgmap v6820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:49 smithi181 ceph-mon[35869]: pgmap v6820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:49 smithi097 ceph-mon[32231]: pgmap v6820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11667.925410 secs 2024-07-26T11:13:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:51 smithi115 ceph-mon[35364]: pgmap v6821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11667.925410 secs 2024-07-26T11:13:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:51 smithi181 ceph-mon[35869]: pgmap v6821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11667.925410 secs 2024-07-26T11:13:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:51 smithi097 ceph-mon[32231]: pgmap v6821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:52.598+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:52.599+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:13:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:52.599+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:13:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:52.599+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:13:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:13:52.600+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:53 smithi115 ceph-mon[35364]: pgmap v6822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:53 smithi181 ceph-mon[35869]: pgmap v6822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:53 smithi097 ceph-mon[32231]: pgmap v6822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:55 smithi115 ceph-mon[35364]: pgmap v6823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:55 smithi181 ceph-mon[35869]: pgmap v6823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:55 smithi097 ceph-mon[32231]: pgmap v6823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:56.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11672.925589 secs 2024-07-26T11:13:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11672.925589 secs 2024-07-26T11:13:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11672.925589 secs 2024-07-26T11:13:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:57 smithi115 ceph-mon[35364]: pgmap v6824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:57 smithi181 ceph-mon[35869]: pgmap v6824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:57 smithi097 ceph-mon[32231]: pgmap v6824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:13:58 smithi181 ceph-mon[35869]: pgmap v6825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:13:58 smithi097 ceph-mon[32231]: pgmap v6825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:13:58 smithi115 ceph-mon[35364]: pgmap v6825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11677.925731 secs 2024-07-26T11:14:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:01 smithi115 ceph-mon[35364]: pgmap v6826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11677.925731 secs 2024-07-26T11:14:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:01 smithi181 ceph-mon[35869]: pgmap v6826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11677.925731 secs 2024-07-26T11:14:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:01 smithi097 ceph-mon[32231]: pgmap v6826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:03 smithi115 ceph-mon[35364]: pgmap v6827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:03 smithi181 ceph-mon[35869]: pgmap v6827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:03 smithi097 ceph-mon[32231]: pgmap v6827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:05 smithi115 ceph-mon[35364]: pgmap v6828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:05 smithi181 ceph-mon[35869]: pgmap v6828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:05 smithi097 ceph-mon[32231]: pgmap v6828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11682.925876 secs 2024-07-26T11:14:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11682.925876 secs 2024-07-26T11:14:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11682.925876 secs 2024-07-26T11:14:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:07.606+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:07.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:07.608+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:14:07.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:07.608+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:14:07.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:07.608+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:14:07.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:07.608+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:07 smithi115 ceph-mon[35364]: pgmap v6829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:07 smithi181 ceph-mon[35869]: pgmap v6829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:07 smithi097 ceph-mon[32231]: pgmap v6829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:09 smithi115 ceph-mon[35364]: pgmap v6830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:09 smithi181 ceph-mon[35869]: pgmap v6830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:09 smithi097 ceph-mon[32231]: pgmap v6830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11687.925963 secs 2024-07-26T11:14:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:11 smithi115 ceph-mon[35364]: pgmap v6831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11687.925963 secs 2024-07-26T11:14:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:11 smithi181 ceph-mon[35869]: pgmap v6831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11687.925963 secs 2024-07-26T11:14:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:11 smithi097 ceph-mon[32231]: pgmap v6831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:13 smithi115 ceph-mon[35364]: pgmap v6832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:13 smithi181 ceph-mon[35869]: pgmap v6832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:13 smithi097 ceph-mon[32231]: pgmap v6832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:14.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:14.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:14.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:14:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:14.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:14.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:14.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:14.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:14.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:14:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:14:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:15.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:15 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:15 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:14:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:15 smithi115 ceph-mon[35364]: pgmap v6833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:14:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:15 smithi115 ceph-mon[35364]: pgmap v6834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:15 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:14:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:15 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:14:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:15 smithi181 ceph-mon[35869]: pgmap v6833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:14:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:15 smithi181 ceph-mon[35869]: pgmap v6834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:15 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:14:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:15 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:14:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:15 smithi097 ceph-mon[32231]: pgmap v6833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:14:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:15 smithi097 ceph-mon[32231]: pgmap v6834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11692.926106 secs 2024-07-26T11:14:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11692.926106 secs 2024-07-26T11:14:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11692.926106 secs 2024-07-26T11:14:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:17 smithi115 ceph-mon[35364]: pgmap v6835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:17 smithi181 ceph-mon[35869]: pgmap v6835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:17 smithi097 ceph-mon[32231]: pgmap v6835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:19 smithi115 ceph-mon[35364]: pgmap v6836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:19 smithi181 ceph-mon[35869]: pgmap v6836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:19 smithi097 ceph-mon[32231]: pgmap v6836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11697.926293 secs 2024-07-26T11:14:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:21 smithi115 ceph-mon[35364]: pgmap v6837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11697.926293 secs 2024-07-26T11:14:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:21 smithi181 ceph-mon[35869]: pgmap v6837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11697.926293 secs 2024-07-26T11:14:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:21 smithi097 ceph-mon[32231]: pgmap v6837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:22.610+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:22.612+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:14:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:22.612+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:14:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:22.612+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:14:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:22.612+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:23 smithi115 ceph-mon[35364]: pgmap v6838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:23.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:23 smithi181 ceph-mon[35869]: pgmap v6838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:23 smithi097 ceph-mon[32231]: pgmap v6838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:25 smithi115 ceph-mon[35364]: pgmap v6839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:25 smithi181 ceph-mon[35869]: pgmap v6839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:25 smithi097 ceph-mon[32231]: pgmap v6839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11702.926381 secs 2024-07-26T11:14:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11702.926381 secs 2024-07-26T11:14:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11702.926381 secs 2024-07-26T11:14:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:27 smithi115 ceph-mon[35364]: pgmap v6840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:27 smithi181 ceph-mon[35869]: pgmap v6840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:27 smithi097 ceph-mon[32231]: pgmap v6840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:29 smithi115 ceph-mon[35364]: pgmap v6841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:29 smithi181 ceph-mon[35869]: pgmap v6841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:29 smithi097 ceph-mon[32231]: pgmap v6841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11707.926524 secs 2024-07-26T11:14:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:31 smithi115 ceph-mon[35364]: pgmap v6842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11707.926524 secs 2024-07-26T11:14:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:31 smithi181 ceph-mon[35869]: pgmap v6842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11707.926524 secs 2024-07-26T11:14:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:31 smithi097 ceph-mon[32231]: pgmap v6842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:33 smithi115 ceph-mon[35364]: pgmap v6843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:33 smithi181 ceph-mon[35869]: pgmap v6843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:33 smithi097 ceph-mon[32231]: pgmap v6843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:34 smithi181 ceph-mon[35869]: pgmap v6844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:14:35.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:34 smithi097 ceph-mon[32231]: pgmap v6844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:14:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:34 smithi115 ceph-mon[35364]: pgmap v6844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:14:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11712.926666 secs 2024-07-26T11:14:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11712.926666 secs 2024-07-26T11:14:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11712.926666 secs 2024-07-26T11:14:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:36 smithi115 ceph-mon[35364]: pgmap v6845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:36 smithi181 ceph-mon[35869]: pgmap v6845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:36 smithi097 ceph-mon[32231]: pgmap v6845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:37.618+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:37.619+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:14:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:37.619+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:14:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:37.619+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:14:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:37.620+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:39 smithi115 ceph-mon[35364]: pgmap v6846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:39 smithi181 ceph-mon[35869]: pgmap v6846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:39 smithi097 ceph-mon[32231]: pgmap v6846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11717.926763 secs 2024-07-26T11:14:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:41 smithi115 ceph-mon[35364]: pgmap v6847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11717.926763 secs 2024-07-26T11:14:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:41 smithi181 ceph-mon[35869]: pgmap v6847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11717.926763 secs 2024-07-26T11:14:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:41 smithi097 ceph-mon[32231]: pgmap v6847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:43 smithi115 ceph-mon[35364]: pgmap v6848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:43 smithi181 ceph-mon[35869]: pgmap v6848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:43 smithi097 ceph-mon[32231]: pgmap v6848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:45 smithi115 ceph-mon[35364]: pgmap v6849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:14:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:45 smithi181 ceph-mon[35869]: pgmap v6849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:14:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:45 smithi097 ceph-mon[32231]: pgmap v6849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:14:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11722.926896 secs 2024-07-26T11:14:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11722.926896 secs 2024-07-26T11:14:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11722.926896 secs 2024-07-26T11:14:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:47 smithi115 ceph-mon[35364]: pgmap v6850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:47 smithi181 ceph-mon[35869]: pgmap v6850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:47 smithi097 ceph-mon[32231]: pgmap v6850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:49 smithi115 ceph-mon[35364]: pgmap v6851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:49 smithi181 ceph-mon[35869]: pgmap v6851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:49 smithi097 ceph-mon[32231]: pgmap v6851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11727.927033 secs 2024-07-26T11:14:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:51 smithi115 ceph-mon[35364]: pgmap v6852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11727.927033 secs 2024-07-26T11:14:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:51 smithi181 ceph-mon[35869]: pgmap v6852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11727.927033 secs 2024-07-26T11:14:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:51 smithi097 ceph-mon[32231]: pgmap v6852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:52.623+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:52.623+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:14:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:52.623+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:14:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:52.623+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:14:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:14:52.623+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:53 smithi115 ceph-mon[35364]: pgmap v6853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:53 smithi181 ceph-mon[35869]: pgmap v6853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:53 smithi097 ceph-mon[32231]: pgmap v6853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:55 smithi115 ceph-mon[35364]: pgmap v6854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:14:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:55 smithi181 ceph-mon[35869]: pgmap v6854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:14:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:55 smithi097 ceph-mon[32231]: pgmap v6854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:14:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11732.927169 secs 2024-07-26T11:14:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11732.927169 secs 2024-07-26T11:14:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11732.927169 secs 2024-07-26T11:14:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:57 smithi115 ceph-mon[35364]: pgmap v6855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:57 smithi181 ceph-mon[35869]: pgmap v6855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:57 smithi097 ceph-mon[32231]: pgmap v6855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:14:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:14:59 smithi115 ceph-mon[35364]: pgmap v6856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:14:59 smithi181 ceph-mon[35869]: pgmap v6856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:14:59 smithi097 ceph-mon[32231]: pgmap v6856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11737.927305 secs 2024-07-26T11:15:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:01 smithi115 ceph-mon[35364]: pgmap v6857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11737.927305 secs 2024-07-26T11:15:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:01 smithi181 ceph-mon[35869]: pgmap v6857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11737.927305 secs 2024-07-26T11:15:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:01 smithi097 ceph-mon[32231]: pgmap v6857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:03 smithi115 ceph-mon[35364]: pgmap v6858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:03 smithi181 ceph-mon[35869]: pgmap v6858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:03 smithi097 ceph-mon[32231]: pgmap v6858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:05 smithi115 ceph-mon[35364]: pgmap v6859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:15:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:05 smithi181 ceph-mon[35869]: pgmap v6859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:15:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:05 smithi097 ceph-mon[32231]: pgmap v6859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:15:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11742.927437 secs 2024-07-26T11:15:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11742.927437 secs 2024-07-26T11:15:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11742.927437 secs 2024-07-26T11:15:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:07 smithi115 ceph-mon[35364]: pgmap v6860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:15:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:07 smithi181 ceph-mon[35869]: pgmap v6860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:15:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:07 smithi097 ceph-mon[32231]: pgmap v6860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:15:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:07.629+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:07.630+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:15:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:07.630+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:15:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:07.630+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:15:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:07.630+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:09 smithi115 ceph-mon[35364]: pgmap v6861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:09 smithi181 ceph-mon[35869]: pgmap v6861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:09 smithi097 ceph-mon[32231]: pgmap v6861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11747.927534 secs 2024-07-26T11:15:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:11 smithi115 ceph-mon[35364]: pgmap v6862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11747.927534 secs 2024-07-26T11:15:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:11 smithi181 ceph-mon[35869]: pgmap v6862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11747.927534 secs 2024-07-26T11:15:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:11 smithi097 ceph-mon[32231]: pgmap v6862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:13 smithi115 ceph-mon[35364]: pgmap v6863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:13 smithi181 ceph-mon[35869]: pgmap v6863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:13 smithi097 ceph-mon[32231]: pgmap v6863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:15 smithi115 ceph-mon[35364]: pgmap v6864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:15:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:15 smithi181 ceph-mon[35869]: pgmap v6864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:15:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:15 smithi097 ceph-mon[32231]: pgmap v6864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:15:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11752.927669 secs 2024-07-26T11:15:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11752.927669 secs 2024-07-26T11:15:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11752.927669 secs 2024-07-26T11:15:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:17 smithi115 ceph-mon[35364]: pgmap v6865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:17 smithi115 ceph-mon[35364]: pgmap v6866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:17 smithi181 ceph-mon[35869]: pgmap v6865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:15:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:17 smithi181 ceph-mon[35869]: pgmap v6866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:17 smithi097 ceph-mon[32231]: pgmap v6865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:15:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:17 smithi097 ceph-mon[32231]: pgmap v6866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:19 smithi115 ceph-mon[35364]: pgmap v6867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:19 smithi181 ceph-mon[35869]: pgmap v6867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:19 smithi097 ceph-mon[32231]: pgmap v6867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:21 smithi115 ceph-mon[35364]: pgmap v6868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11757.927800 secs 2024-07-26T11:15:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:21 smithi181 ceph-mon[35869]: pgmap v6868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11757.927800 secs 2024-07-26T11:15:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:21 smithi097 ceph-mon[32231]: pgmap v6868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11757.927800 secs 2024-07-26T11:15:22.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:22.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:22.634+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:15:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:22.634+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:15:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:22.634+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:15:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:22.634+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:23 smithi115 ceph-mon[35364]: pgmap v6869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:23 smithi181 ceph-mon[35869]: pgmap v6869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:23 smithi097 ceph-mon[32231]: pgmap v6869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:25 smithi181 ceph-mon[35869]: pgmap v6870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:25 smithi097 ceph-mon[32231]: pgmap v6870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:25 smithi115 ceph-mon[35364]: pgmap v6870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:15:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11762.927932 secs 2024-07-26T11:15:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11762.927932 secs 2024-07-26T11:15:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11762.927932 secs 2024-07-26T11:15:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:27 smithi097 ceph-mon[32231]: pgmap v6871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:27 smithi115 ceph-mon[35364]: pgmap v6871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:27 smithi181 ceph-mon[35869]: pgmap v6871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:29 smithi115 ceph-mon[35364]: pgmap v6872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:29 smithi181 ceph-mon[35869]: pgmap v6872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:29 smithi097 ceph-mon[32231]: pgmap v6872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:31 smithi115 ceph-mon[35364]: pgmap v6873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11767.928082 secs 2024-07-26T11:15:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:31 smithi181 ceph-mon[35869]: pgmap v6873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11767.928082 secs 2024-07-26T11:15:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:31 smithi097 ceph-mon[32231]: pgmap v6873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11767.928082 secs 2024-07-26T11:15:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:33 smithi115 ceph-mon[35364]: pgmap v6874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:33 smithi181 ceph-mon[35869]: pgmap v6874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:33 smithi097 ceph-mon[32231]: pgmap v6874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:35 smithi115 ceph-mon[35364]: pgmap v6875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:15:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:35 smithi181 ceph-mon[35869]: pgmap v6875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:15:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:35 smithi097 ceph-mon[32231]: pgmap v6875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:15:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11772.928287 secs 2024-07-26T11:15:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11772.928287 secs 2024-07-26T11:15:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11772.928287 secs 2024-07-26T11:15:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:37.641+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:37.642+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:15:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:37.642+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:15:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:37.642+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:15:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:37.642+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:37 smithi115 ceph-mon[35364]: pgmap v6876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:37 smithi181 ceph-mon[35869]: pgmap v6876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:37 smithi097 ceph-mon[32231]: pgmap v6876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:39 smithi115 ceph-mon[35364]: pgmap v6877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:39 smithi181 ceph-mon[35869]: pgmap v6877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:39 smithi097 ceph-mon[32231]: pgmap v6877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11777.928452 secs 2024-07-26T11:15:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:41 smithi115 ceph-mon[35364]: pgmap v6878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11777.928452 secs 2024-07-26T11:15:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:41 smithi181 ceph-mon[35869]: pgmap v6878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11777.928452 secs 2024-07-26T11:15:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:41 smithi097 ceph-mon[32231]: pgmap v6878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:43 smithi115 ceph-mon[35364]: pgmap v6879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:43 smithi181 ceph-mon[35869]: pgmap v6879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:43 smithi097 ceph-mon[32231]: pgmap v6879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:45 smithi115 ceph-mon[35364]: pgmap v6880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:15:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:45 smithi181 ceph-mon[35869]: pgmap v6880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:15:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:45 smithi097 ceph-mon[32231]: pgmap v6880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:15:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11782.928587 secs 2024-07-26T11:15:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11782.928587 secs 2024-07-26T11:15:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11782.928587 secs 2024-07-26T11:15:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:47 smithi115 ceph-mon[35364]: pgmap v6881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:47 smithi181 ceph-mon[35869]: pgmap v6881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:47 smithi097 ceph-mon[32231]: pgmap v6881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:49 smithi115 ceph-mon[35364]: pgmap v6882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:49 smithi181 ceph-mon[35869]: pgmap v6882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:49 smithi097 ceph-mon[32231]: pgmap v6882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11787.928746 secs 2024-07-26T11:15:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:51 smithi115 ceph-mon[35364]: pgmap v6883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:15:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11787.928746 secs 2024-07-26T11:15:51.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:51 smithi181 ceph-mon[35869]: pgmap v6883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:15:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11787.928746 secs 2024-07-26T11:15:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:51 smithi097 ceph-mon[32231]: pgmap v6883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:15:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:52.643+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:52.644+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:15:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:52.644+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:15:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:52.644+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:15:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:15:52.644+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:53.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:53 smithi115 ceph-mon[35364]: pgmap v6884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:53 smithi181 ceph-mon[35869]: pgmap v6884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:53.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:53 smithi097 ceph-mon[32231]: pgmap v6884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:55 smithi115 ceph-mon[35364]: pgmap v6885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:15:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:55 smithi181 ceph-mon[35869]: pgmap v6885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:15:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:55 smithi097 ceph-mon[32231]: pgmap v6885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:15:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11792.928885 secs 2024-07-26T11:15:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11792.928885 secs 2024-07-26T11:15:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11792.928885 secs 2024-07-26T11:15:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:57 smithi115 ceph-mon[35364]: pgmap v6886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:57 smithi181 ceph-mon[35869]: pgmap v6886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:57 smithi097 ceph-mon[32231]: pgmap v6886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:15:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:15:59 smithi115 ceph-mon[35364]: pgmap v6887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:15:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:15:59 smithi181 ceph-mon[35869]: pgmap v6887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:15:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:15:59 smithi097 ceph-mon[32231]: pgmap v6887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:16:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11797.929027 secs 2024-07-26T11:16:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:01 smithi115 ceph-mon[35364]: pgmap v6888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11797.929027 secs 2024-07-26T11:16:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:01 smithi181 ceph-mon[35869]: pgmap v6888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11797.929027 secs 2024-07-26T11:16:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:01 smithi097 ceph-mon[32231]: pgmap v6888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:03 smithi115 ceph-mon[35364]: pgmap v6889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:03 smithi181 ceph-mon[35869]: pgmap v6889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:03 smithi097 ceph-mon[32231]: pgmap v6889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:05 smithi115 ceph-mon[35364]: pgmap v6890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:05 smithi181 ceph-mon[35869]: pgmap v6890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:05 smithi097 ceph-mon[32231]: pgmap v6890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11802.929196 secs 2024-07-26T11:16:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11802.929196 secs 2024-07-26T11:16:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11802.929196 secs 2024-07-26T11:16:07.653 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:07.651+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.654 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:07.652+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:16:07.654 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:07.652+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:16:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:07.652+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:16:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:07.652+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:07 smithi115 ceph-mon[35364]: pgmap v6891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:07 smithi181 ceph-mon[35869]: pgmap v6891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:07 smithi097 ceph-mon[32231]: pgmap v6891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:08.976+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:16:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:08.976+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:16:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:08.977+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:16:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:08.977+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:16:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:08.977+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:16:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:08.977+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:16:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:08.978+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:16:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:08.978+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:16:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:09 smithi115 ceph-mon[35364]: pgmap v6892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:09 smithi181 ceph-mon[35869]: pgmap v6892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:09.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:09 smithi097 ceph-mon[32231]: pgmap v6892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11807.929357 secs 2024-07-26T11:16:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:11 smithi115 ceph-mon[35364]: pgmap v6893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11807.929357 secs 2024-07-26T11:16:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:11 smithi181 ceph-mon[35869]: pgmap v6893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11807.929357 secs 2024-07-26T11:16:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:11 smithi097 ceph-mon[32231]: pgmap v6893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:13 smithi115 ceph-mon[35364]: pgmap v6894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:13 smithi181 ceph-mon[35869]: pgmap v6894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:13 smithi097 ceph-mon[32231]: pgmap v6894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:15 smithi115 ceph-mon[35364]: pgmap v6895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:15 smithi181 ceph-mon[35869]: pgmap v6895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:15 smithi097 ceph-mon[32231]: pgmap v6895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11812.929499 secs 2024-07-26T11:16:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11812.929499 secs 2024-07-26T11:16:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11812.929499 secs 2024-07-26T11:16:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:17 smithi115 ceph-mon[35364]: pgmap v6896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:17 smithi181 ceph-mon[35869]: pgmap v6896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:17 smithi097 ceph-mon[32231]: pgmap v6896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:18.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:16:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:19 smithi181 ceph-mon[35869]: pgmap v6897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:19 smithi181 ceph-mon[35869]: pgmap v6898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:19 smithi097 ceph-mon[32231]: pgmap v6897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:19 smithi097 ceph-mon[32231]: pgmap v6898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:19 smithi115 ceph-mon[35364]: pgmap v6897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:19 smithi115 ceph-mon[35364]: pgmap v6898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:21 smithi115 ceph-mon[35364]: pgmap v6899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11817.929641 secs 2024-07-26T11:16:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:21 smithi181 ceph-mon[35869]: pgmap v6899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11817.929641 secs 2024-07-26T11:16:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:21 smithi097 ceph-mon[32231]: pgmap v6899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11817.929641 secs 2024-07-26T11:16:22.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:22.654+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:22.655+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:16:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:22.655+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:16:22.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:22.655+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:16:22.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:22.656+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:23 smithi115 ceph-mon[35364]: pgmap v6900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:23 smithi181 ceph-mon[35869]: pgmap v6900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:23 smithi097 ceph-mon[32231]: pgmap v6900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:25 smithi115 ceph-mon[35364]: pgmap v6901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:25 smithi181 ceph-mon[35869]: pgmap v6901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:25 smithi097 ceph-mon[32231]: pgmap v6901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11822.929814 secs 2024-07-26T11:16:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11822.929814 secs 2024-07-26T11:16:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11822.929814 secs 2024-07-26T11:16:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:27 smithi115 ceph-mon[35364]: pgmap v6902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:27 smithi181 ceph-mon[35869]: pgmap v6902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:27 smithi097 ceph-mon[32231]: pgmap v6902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:16:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:29 smithi115 ceph-mon[35364]: pgmap v6903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:16:29.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:29 smithi181 ceph-mon[35869]: pgmap v6903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:16:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:29 smithi097 ceph-mon[32231]: pgmap v6903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:16:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:31 smithi115 ceph-mon[35364]: pgmap v6904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11827.929960 secs 2024-07-26T11:16:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:31 smithi181 ceph-mon[35869]: pgmap v6904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11827.929960 secs 2024-07-26T11:16:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:31 smithi097 ceph-mon[32231]: pgmap v6904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11827.929960 secs 2024-07-26T11:16:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:33 smithi115 ceph-mon[35364]: pgmap v6905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:33 smithi181 ceph-mon[35869]: pgmap v6905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:33 smithi097 ceph-mon[32231]: pgmap v6905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:35 smithi115 ceph-mon[35364]: pgmap v6906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:35 smithi181 ceph-mon[35869]: pgmap v6906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:35 smithi097 ceph-mon[32231]: pgmap v6906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11832.930100 secs 2024-07-26T11:16:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11832.930100 secs 2024-07-26T11:16:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11832.930100 secs 2024-07-26T11:16:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:37.663+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:37.664+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:16:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:37.664+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:16:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:37.664+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:16:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:37.664+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:37 smithi115 ceph-mon[35364]: pgmap v6907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:37 smithi181 ceph-mon[35869]: pgmap v6907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:37 smithi097 ceph-mon[32231]: pgmap v6907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:39 smithi115 ceph-mon[35364]: pgmap v6908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:39 smithi181 ceph-mon[35869]: pgmap v6908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:39 smithi097 ceph-mon[32231]: pgmap v6908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:41 smithi115 ceph-mon[35364]: pgmap v6909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11837.930285 secs 2024-07-26T11:16:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:41 smithi181 ceph-mon[35869]: pgmap v6909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11837.930285 secs 2024-07-26T11:16:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:41 smithi097 ceph-mon[32231]: pgmap v6909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11837.930285 secs 2024-07-26T11:16:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:43 smithi115 ceph-mon[35364]: pgmap v6910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:43 smithi181 ceph-mon[35869]: pgmap v6910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:43 smithi097 ceph-mon[32231]: pgmap v6910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:45 smithi115 ceph-mon[35364]: pgmap v6911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:45 smithi181 ceph-mon[35869]: pgmap v6911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:45 smithi097 ceph-mon[32231]: pgmap v6911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11842.930418 secs 2024-07-26T11:16:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11842.930418 secs 2024-07-26T11:16:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11842.930418 secs 2024-07-26T11:16:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:47 smithi115 ceph-mon[35364]: pgmap v6912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:47 smithi181 ceph-mon[35869]: pgmap v6912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:47 smithi097 ceph-mon[32231]: pgmap v6912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:49 smithi115 ceph-mon[35364]: pgmap v6913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:49 smithi181 ceph-mon[35869]: pgmap v6913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:50.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:49 smithi097 ceph-mon[32231]: pgmap v6913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11847.930556 secs 2024-07-26T11:16:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:51 smithi115 ceph-mon[35364]: pgmap v6914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11847.930556 secs 2024-07-26T11:16:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:51 smithi181 ceph-mon[35869]: pgmap v6914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11847.930556 secs 2024-07-26T11:16:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:51 smithi097 ceph-mon[32231]: pgmap v6914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:52.664+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:52.666+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:16:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:52.666+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:16:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:52.666+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:16:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:16:52.666+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:53 smithi115 ceph-mon[35364]: pgmap v6915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:53 smithi181 ceph-mon[35869]: pgmap v6915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:53 smithi097 ceph-mon[32231]: pgmap v6915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:55 smithi115 ceph-mon[35364]: pgmap v6916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:55 smithi181 ceph-mon[35869]: pgmap v6916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:55 smithi097 ceph-mon[32231]: pgmap v6916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11852.930653 secs 2024-07-26T11:16:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11852.930653 secs 2024-07-26T11:16:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11852.930653 secs 2024-07-26T11:16:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:57 smithi115 ceph-mon[35364]: pgmap v6917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:57 smithi181 ceph-mon[35869]: pgmap v6917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:57 smithi097 ceph-mon[32231]: pgmap v6917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:16:59 smithi115 ceph-mon[35364]: pgmap v6918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:16:59 smithi181 ceph-mon[35869]: pgmap v6918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:16:59 smithi097 ceph-mon[32231]: pgmap v6918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11857.930807 secs 2024-07-26T11:17:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:01 smithi115 ceph-mon[35364]: pgmap v6919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11857.930807 secs 2024-07-26T11:17:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:01 smithi181 ceph-mon[35869]: pgmap v6919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11857.930807 secs 2024-07-26T11:17:02.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:01 smithi097 ceph-mon[32231]: pgmap v6919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:03 smithi115 ceph-mon[35364]: pgmap v6920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:03 smithi181 ceph-mon[35869]: pgmap v6920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:03 smithi097 ceph-mon[32231]: pgmap v6920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:05 smithi181 ceph-mon[35869]: pgmap v6921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:05 smithi097 ceph-mon[32231]: pgmap v6921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:05 smithi115 ceph-mon[35364]: pgmap v6921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11862.930947 secs 2024-07-26T11:17:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11862.930947 secs 2024-07-26T11:17:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11862.930947 secs 2024-07-26T11:17:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:07.673+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:07.674+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:17:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:07.674+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:17:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:07.674+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:17:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:07.674+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:07 smithi097 ceph-mon[32231]: pgmap v6922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:07 smithi115 ceph-mon[35364]: pgmap v6922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:07 smithi181 ceph-mon[35869]: pgmap v6922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:09 smithi097 ceph-mon[32231]: pgmap v6923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:09 smithi115 ceph-mon[35364]: pgmap v6923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:09 smithi181 ceph-mon[35869]: pgmap v6923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:10 smithi115 ceph-mon[35364]: pgmap v6924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:10 smithi181 ceph-mon[35869]: pgmap v6924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:10 smithi097 ceph-mon[32231]: pgmap v6924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11867.931091 secs 2024-07-26T11:17:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11867.931091 secs 2024-07-26T11:17:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11867.931091 secs 2024-07-26T11:17:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:12 smithi115 ceph-mon[35364]: pgmap v6925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:12 smithi181 ceph-mon[35869]: pgmap v6925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:12 smithi097 ceph-mon[32231]: pgmap v6925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:15 smithi115 ceph-mon[35364]: pgmap v6926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:15 smithi181 ceph-mon[35869]: pgmap v6926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:15 smithi097 ceph-mon[32231]: pgmap v6926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11872.931270 secs 2024-07-26T11:17:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11872.931270 secs 2024-07-26T11:17:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11872.931270 secs 2024-07-26T11:17:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:17 smithi115 ceph-mon[35364]: pgmap v6927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:17 smithi181 ceph-mon[35869]: pgmap v6927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:17.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:17 smithi097 ceph-mon[32231]: pgmap v6927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:19.337 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:19 smithi115 ceph-mon[35364]: pgmap v6928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:19.337 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:19.378 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:19 smithi181 ceph-mon[35869]: pgmap v6928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:19.378 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:19.444 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:19 smithi097 ceph-mon[32231]: pgmap v6928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:19.444 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11877.931406 secs 2024-07-26T11:17:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:21 smithi115 ceph-mon[35364]: pgmap v6929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11877.931406 secs 2024-07-26T11:17:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:21 smithi181 ceph-mon[35869]: pgmap v6929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11877.931406 secs 2024-07-26T11:17:21.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:21 smithi097 ceph-mon[32231]: pgmap v6929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:22.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:22.675+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:22.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:22.676+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:17:22.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:22.676+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:17:22.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:22.676+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:17:22.679 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:22.677+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:23.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:23 smithi115 ceph-mon[35364]: pgmap v6930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:23 smithi181 ceph-mon[35869]: pgmap v6930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:23 smithi097 ceph-mon[32231]: pgmap v6930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:25.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:25 smithi097 ceph-mon[32231]: pgmap v6931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:25 smithi115 ceph-mon[35364]: pgmap v6931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:25 smithi181 ceph-mon[35869]: pgmap v6931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11882.931502 secs 2024-07-26T11:17:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11882.931502 secs 2024-07-26T11:17:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11882.931502 secs 2024-07-26T11:17:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:27 smithi115 ceph-mon[35364]: pgmap v6932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:27 smithi181 ceph-mon[35869]: pgmap v6932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:27 smithi097 ceph-mon[32231]: pgmap v6932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:28 smithi115 ceph-mon[35364]: pgmap v6933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:28 smithi115 ceph-mon[35364]: pgmap v6934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:28 smithi181 ceph-mon[35869]: pgmap v6933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:28 smithi181 ceph-mon[35869]: pgmap v6934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:29.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:29.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:28 smithi097 ceph-mon[32231]: pgmap v6933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:29.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:28 smithi097 ceph-mon[32231]: pgmap v6934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:29.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:30 smithi181 ceph-mon[35869]: pgmap v6935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11887.931639 secs 2024-07-26T11:17:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:30 smithi097 ceph-mon[32231]: pgmap v6935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11887.931639 secs 2024-07-26T11:17:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:30 smithi115 ceph-mon[35364]: pgmap v6935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11887.931639 secs 2024-07-26T11:17:33.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:32 smithi181 ceph-mon[35869]: pgmap v6936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:32 smithi097 ceph-mon[32231]: pgmap v6936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:32 smithi115 ceph-mon[35364]: pgmap v6936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:34 smithi097 ceph-mon[32231]: pgmap v6937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:34 smithi115 ceph-mon[35364]: pgmap v6937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:34 smithi181 ceph-mon[35869]: pgmap v6937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:36.344 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11892.931779 secs 2024-07-26T11:17:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11892.931779 secs 2024-07-26T11:17:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11892.931779 secs 2024-07-26T11:17:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:37 smithi115 ceph-mon[35364]: pgmap v6938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:37 smithi181 ceph-mon[35869]: pgmap v6938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:37 smithi097 ceph-mon[32231]: pgmap v6938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:37.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:37.684+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:37.687+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:17:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:37.687+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:17:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:37.687+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:17:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:37.687+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:39 smithi115 ceph-mon[35364]: pgmap v6939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:17:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:39 smithi181 ceph-mon[35869]: pgmap v6939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:17:39.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:39 smithi097 ceph-mon[32231]: pgmap v6939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:17:41.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:41 smithi115 ceph-mon[35364]: pgmap v6940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:41.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11897.931946 secs 2024-07-26T11:17:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:41 smithi181 ceph-mon[35869]: pgmap v6940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11897.931946 secs 2024-07-26T11:17:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:41 smithi097 ceph-mon[32231]: pgmap v6940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11897.931946 secs 2024-07-26T11:17:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:43 smithi115 ceph-mon[35364]: pgmap v6941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:43 smithi181 ceph-mon[35869]: pgmap v6941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:43 smithi097 ceph-mon[32231]: pgmap v6941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:45 smithi115 ceph-mon[35364]: pgmap v6942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:45 smithi181 ceph-mon[35869]: pgmap v6942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:45 smithi097 ceph-mon[32231]: pgmap v6942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11902.932096 secs 2024-07-26T11:17:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11902.932096 secs 2024-07-26T11:17:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11902.932096 secs 2024-07-26T11:17:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:47 smithi115 ceph-mon[35364]: pgmap v6943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:47 smithi181 ceph-mon[35869]: pgmap v6943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:47 smithi097 ceph-mon[32231]: pgmap v6943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:49 smithi115 ceph-mon[35364]: pgmap v6944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:49 smithi181 ceph-mon[35869]: pgmap v6944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:49 smithi097 ceph-mon[32231]: pgmap v6944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:51 smithi115 ceph-mon[35364]: pgmap v6945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11907.932274 secs 2024-07-26T11:17:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:51 smithi181 ceph-mon[35869]: pgmap v6945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11907.932274 secs 2024-07-26T11:17:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:51 smithi097 ceph-mon[32231]: pgmap v6945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11907.932274 secs 2024-07-26T11:17:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:52.687+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:52.688+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:17:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:52.688+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:17:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:52.688+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:17:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:17:52.689+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:53 smithi115 ceph-mon[35364]: pgmap v6946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:53 smithi181 ceph-mon[35869]: pgmap v6946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:53 smithi097 ceph-mon[32231]: pgmap v6946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:55 smithi115 ceph-mon[35364]: pgmap v6947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:55 smithi181 ceph-mon[35869]: pgmap v6947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:55 smithi097 ceph-mon[32231]: pgmap v6947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11912.932408 secs 2024-07-26T11:17:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11912.932408 secs 2024-07-26T11:17:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11912.932408 secs 2024-07-26T11:17:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:57 smithi115 ceph-mon[35364]: pgmap v6948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:57 smithi181 ceph-mon[35869]: pgmap v6948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:57 smithi097 ceph-mon[32231]: pgmap v6948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:17:59 smithi115 ceph-mon[35364]: pgmap v6949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:17:59 smithi181 ceph-mon[35869]: pgmap v6949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:17:59 smithi097 ceph-mon[32231]: pgmap v6949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:01 smithi115 ceph-mon[35364]: pgmap v6950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11917.932550 secs 2024-07-26T11:18:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:01 smithi181 ceph-mon[35869]: pgmap v6950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11917.932550 secs 2024-07-26T11:18:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:01 smithi097 ceph-mon[32231]: pgmap v6950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11917.932550 secs 2024-07-26T11:18:03.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:03 smithi097 ceph-mon[32231]: pgmap v6951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:03 smithi115 ceph-mon[35364]: pgmap v6951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:03 smithi181 ceph-mon[35869]: pgmap v6951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:05 smithi115 ceph-mon[35364]: pgmap v6952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:05 smithi181 ceph-mon[35869]: pgmap v6952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:05 smithi097 ceph-mon[32231]: pgmap v6952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11922.932684 secs 2024-07-26T11:18:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11922.932684 secs 2024-07-26T11:18:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11922.932684 secs 2024-07-26T11:18:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:07.696+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:07.697+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:18:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:07.697+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:18:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:07.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:18:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:07.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:07 smithi115 ceph-mon[35364]: pgmap v6953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:07 smithi181 ceph-mon[35869]: pgmap v6953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:07 smithi097 ceph-mon[32231]: pgmap v6953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:09 smithi115 ceph-mon[35364]: pgmap v6954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:09 smithi181 ceph-mon[35869]: pgmap v6954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:09 smithi097 ceph-mon[32231]: pgmap v6954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:11 smithi115 ceph-mon[35364]: pgmap v6955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11927.932822 secs 2024-07-26T11:18:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:11 smithi181 ceph-mon[35869]: pgmap v6955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11927.932822 secs 2024-07-26T11:18:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:11 smithi097 ceph-mon[32231]: pgmap v6955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11927.932822 secs 2024-07-26T11:18:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:13 smithi115 ceph-mon[35364]: pgmap v6956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:13 smithi181 ceph-mon[35869]: pgmap v6956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:13 smithi097 ceph-mon[32231]: pgmap v6956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:15 smithi115 ceph-mon[35364]: pgmap v6957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:15 smithi181 ceph-mon[35869]: pgmap v6957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:15 smithi097 ceph-mon[32231]: pgmap v6957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11932.932959 secs 2024-07-26T11:18:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11932.932959 secs 2024-07-26T11:18:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11932.932959 secs 2024-07-26T11:18:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:17 smithi115 ceph-mon[35364]: pgmap v6958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:17 smithi181 ceph-mon[35869]: pgmap v6958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:17 smithi097 ceph-mon[32231]: pgmap v6958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:19 smithi115 ceph-mon[35364]: pgmap v6959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:19 smithi181 ceph-mon[35869]: pgmap v6959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:19 smithi097 ceph-mon[32231]: pgmap v6959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11937.933094 secs 2024-07-26T11:18:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:21 smithi115 ceph-mon[35364]: pgmap v6960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11937.933094 secs 2024-07-26T11:18:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:21 smithi181 ceph-mon[35869]: pgmap v6960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11937.933094 secs 2024-07-26T11:18:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:21 smithi097 ceph-mon[32231]: pgmap v6960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:22.698+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:22.700+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:18:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:22.700+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:18:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:22.700+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:18:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:22.700+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:23 smithi115 ceph-mon[35364]: pgmap v6961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:23 smithi181 ceph-mon[35869]: pgmap v6961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:23 smithi097 ceph-mon[32231]: pgmap v6961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:25 smithi115 ceph-mon[35364]: pgmap v6962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:25 smithi181 ceph-mon[35869]: pgmap v6962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:25 smithi097 ceph-mon[32231]: pgmap v6962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11942.933293 secs 2024-07-26T11:18:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11942.933293 secs 2024-07-26T11:18:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11942.933293 secs 2024-07-26T11:18:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:27 smithi115 ceph-mon[35364]: pgmap v6963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:27 smithi181 ceph-mon[35869]: pgmap v6963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:27 smithi097 ceph-mon[32231]: pgmap v6963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:29 smithi181 ceph-mon[35869]: pgmap v6964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:29 smithi097 ceph-mon[32231]: pgmap v6964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:30.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:29 smithi115 ceph-mon[35364]: pgmap v6964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:31.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11947.933374 secs 2024-07-26T11:18:31.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:31 smithi115 ceph-mon[35364]: pgmap v6965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11947.933374 secs 2024-07-26T11:18:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:31 smithi097 ceph-mon[32231]: pgmap v6965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:32.023 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11947.933374 secs 2024-07-26T11:18:32.023 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:31 smithi181 ceph-mon[35869]: pgmap v6965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:33.862 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:33 smithi115 ceph-mon[35364]: pgmap v6966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:33.941 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:33 smithi181 ceph-mon[35869]: pgmap v6966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:33 smithi097 ceph-mon[32231]: pgmap v6966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: pgmap v6967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: pgmap v6968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: pgmap v6969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: pgmap v6967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: pgmap v6968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: pgmap v6969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: pgmap v6967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: pgmap v6968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:18:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: pgmap v6969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:18:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11952.933516 secs 2024-07-26T11:18:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11952.933516 secs 2024-07-26T11:18:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11952.933516 secs 2024-07-26T11:18:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:37.707+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:37.708+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:18:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:37.708+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:18:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:37.708+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:18:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:37.708+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:37 smithi115 ceph-mon[35364]: pgmap v6970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:18:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:37 smithi181 ceph-mon[35869]: pgmap v6970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:18:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:37 smithi097 ceph-mon[32231]: pgmap v6970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:18:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:39 smithi115 ceph-mon[35364]: pgmap v6971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:18:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:39 smithi181 ceph-mon[35869]: pgmap v6971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:18:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:39 smithi097 ceph-mon[32231]: pgmap v6971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:18:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11957.933668 secs 2024-07-26T11:18:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:41 smithi115 ceph-mon[35364]: pgmap v6972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:18:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11957.933668 secs 2024-07-26T11:18:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:41 smithi181 ceph-mon[35869]: pgmap v6972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:18:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11957.933668 secs 2024-07-26T11:18:41.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:41 smithi097 ceph-mon[32231]: pgmap v6972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:18:43.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:43 smithi115 ceph-mon[35364]: pgmap v6973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:18:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:43 smithi181 ceph-mon[35869]: pgmap v6973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:18:43.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:43 smithi097 ceph-mon[32231]: pgmap v6973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:18:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:45 smithi115 ceph-mon[35364]: pgmap v6974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:18:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:45 smithi181 ceph-mon[35869]: pgmap v6974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:18:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:45 smithi097 ceph-mon[32231]: pgmap v6974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:18:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11962.933807 secs 2024-07-26T11:18:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11962.933807 secs 2024-07-26T11:18:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11962.933807 secs 2024-07-26T11:18:47.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:47 smithi115 ceph-mon[35364]: pgmap v6975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:47 smithi181 ceph-mon[35869]: pgmap v6975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:47.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:47 smithi097 ceph-mon[32231]: pgmap v6975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:49 smithi097 ceph-mon[32231]: pgmap v6976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:49 smithi115 ceph-mon[35364]: pgmap v6976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:49 smithi181 ceph-mon[35869]: pgmap v6976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11967.933956 secs 2024-07-26T11:18:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:51 smithi097 ceph-mon[32231]: pgmap v6977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:51.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11967.933956 secs 2024-07-26T11:18:51.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:51 smithi115 ceph-mon[35364]: pgmap v6977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:51.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11967.933956 secs 2024-07-26T11:18:51.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:51 smithi181 ceph-mon[35869]: pgmap v6977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:52.710+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:52.711+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:18:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:52.711+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:18:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:52.711+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:18:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:18:52.711+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:53 smithi181 ceph-mon[35869]: pgmap v6978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:18:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:53 smithi115 ceph-mon[35364]: pgmap v6978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:18:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:53 smithi097 ceph-mon[32231]: pgmap v6978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:18:55.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:55 smithi115 ceph-mon[35364]: pgmap v6979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:55 smithi181 ceph-mon[35869]: pgmap v6979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:55 smithi097 ceph-mon[32231]: pgmap v6979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11972.934048 secs 2024-07-26T11:18:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11972.934048 secs 2024-07-26T11:18:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11972.934048 secs 2024-07-26T11:18:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:57 smithi115 ceph-mon[35364]: pgmap v6980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:57 smithi181 ceph-mon[35869]: pgmap v6980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:57 smithi097 ceph-mon[32231]: pgmap v6980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:18:59 smithi115 ceph-mon[35364]: pgmap v6981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:18:59 smithi181 ceph-mon[35869]: pgmap v6981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:18:59 smithi097 ceph-mon[32231]: pgmap v6981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11977.934208 secs 2024-07-26T11:19:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:01 smithi115 ceph-mon[35364]: pgmap v6982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11977.934208 secs 2024-07-26T11:19:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:01 smithi181 ceph-mon[35869]: pgmap v6982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11977.934208 secs 2024-07-26T11:19:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:01 smithi097 ceph-mon[32231]: pgmap v6982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:03.815 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:03 smithi181 ceph-mon[35869]: pgmap v6983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:03 smithi115 ceph-mon[35364]: pgmap v6983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:03 smithi097 ceph-mon[32231]: pgmap v6983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:05 smithi115 ceph-mon[35364]: pgmap v6984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:05 smithi181 ceph-mon[35869]: pgmap v6984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:05 smithi097 ceph-mon[32231]: pgmap v6984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11982.934322 secs 2024-07-26T11:19:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11982.934322 secs 2024-07-26T11:19:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11982.934322 secs 2024-07-26T11:19:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:07.718+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:07.719+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:19:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:07.719+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:19:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:07.719+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:19:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:07.719+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:07 smithi115 ceph-mon[35364]: pgmap v6985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:07 smithi181 ceph-mon[35869]: pgmap v6985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:08.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:07 smithi097 ceph-mon[32231]: pgmap v6985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:09.008+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:19:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:09.008+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:19:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:09.009+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:19:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:09.009+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:19:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:09 smithi115 ceph-mon[35364]: pgmap v6986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:09 smithi181 ceph-mon[35869]: pgmap v6986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:09 smithi097 ceph-mon[32231]: pgmap v6986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11987.934462 secs 2024-07-26T11:19:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:11 smithi115 ceph-mon[35364]: pgmap v6987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11987.934462 secs 2024-07-26T11:19:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:11 smithi181 ceph-mon[35869]: pgmap v6987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11987.934462 secs 2024-07-26T11:19:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:11 smithi097 ceph-mon[32231]: pgmap v6987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:13 smithi115 ceph-mon[35364]: pgmap v6988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:13 smithi181 ceph-mon[35869]: pgmap v6988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:13 smithi097 ceph-mon[32231]: pgmap v6988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:15 smithi115 ceph-mon[35364]: pgmap v6989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:15 smithi181 ceph-mon[35869]: pgmap v6989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:15 smithi097 ceph-mon[32231]: pgmap v6989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11992.934634 secs 2024-07-26T11:19:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11992.934634 secs 2024-07-26T11:19:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11992.934634 secs 2024-07-26T11:19:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:17 smithi115 ceph-mon[35364]: pgmap v6990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:17 smithi181 ceph-mon[35869]: pgmap v6990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:17 smithi097 ceph-mon[32231]: pgmap v6990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:19 smithi115 ceph-mon[35364]: pgmap v6991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:19 smithi181 ceph-mon[35869]: pgmap v6991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:19 smithi097 ceph-mon[32231]: pgmap v6991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 11997.934723 secs 2024-07-26T11:19:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:21 smithi115 ceph-mon[35364]: pgmap v6992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 11997.934723 secs 2024-07-26T11:19:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:21 smithi181 ceph-mon[35869]: pgmap v6992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 11997.934723 secs 2024-07-26T11:19:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:21 smithi097 ceph-mon[32231]: pgmap v6992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:22.723 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:22.721+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:22.724 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:22.722+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:19:22.724 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:22.722+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:19:22.724 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:22.722+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:19:22.724 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:22.722+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:23 smithi115 ceph-mon[35364]: pgmap v6993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:23 smithi181 ceph-mon[35869]: pgmap v6993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:23 smithi097 ceph-mon[32231]: pgmap v6993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:25 smithi115 ceph-mon[35364]: pgmap v6994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:25 smithi181 ceph-mon[35869]: pgmap v6994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:25 smithi097 ceph-mon[32231]: pgmap v6994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:19:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12002.934879 secs 2024-07-26T11:19:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12002.934879 secs 2024-07-26T11:19:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12002.934879 secs 2024-07-26T11:19:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:27 smithi115 ceph-mon[35364]: pgmap v6995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:27 smithi181 ceph-mon[35869]: pgmap v6995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:27 smithi097 ceph-mon[32231]: pgmap v6995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:29 smithi115 ceph-mon[35364]: pgmap v6996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:29 smithi181 ceph-mon[35869]: pgmap v6996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:29 smithi097 ceph-mon[32231]: pgmap v6996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12007.935024 secs 2024-07-26T11:19:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:31 smithi115 ceph-mon[35364]: pgmap v6997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12007.935024 secs 2024-07-26T11:19:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:31 smithi181 ceph-mon[35869]: pgmap v6997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12007.935024 secs 2024-07-26T11:19:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:31 smithi097 ceph-mon[32231]: pgmap v6997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:33 smithi115 ceph-mon[35364]: pgmap v6998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:33 smithi181 ceph-mon[35869]: pgmap v6998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:33 smithi097 ceph-mon[32231]: pgmap v6998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:35 smithi115 ceph-mon[35364]: pgmap v6999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:19:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:35 smithi181 ceph-mon[35869]: pgmap v6999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:19:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:35 smithi097 ceph-mon[32231]: pgmap v6999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:19:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12012.935126 secs 2024-07-26T11:19:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:19:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12012.935126 secs 2024-07-26T11:19:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:19:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12012.935126 secs 2024-07-26T11:19:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:37.728+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:37.730+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:37.730+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:37.730+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:37.730+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:37 smithi115 ceph-mon[35364]: pgmap v7000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:19:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:37 smithi115 ceph-mon[35364]: pgmap v7001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:37 smithi181 ceph-mon[35869]: pgmap v7000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:19:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:37 smithi181 ceph-mon[35869]: pgmap v7001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:37 smithi097 ceph-mon[32231]: pgmap v7000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:19:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:37 smithi097 ceph-mon[32231]: pgmap v7001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:39 smithi115 ceph-mon[35364]: pgmap v7002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:39 smithi181 ceph-mon[35869]: pgmap v7002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:39 smithi097 ceph-mon[32231]: pgmap v7002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:41 smithi115 ceph-mon[35364]: pgmap v7003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:19:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12017.935269 secs 2024-07-26T11:19:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:41 smithi181 ceph-mon[35869]: pgmap v7003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:19:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12017.935269 secs 2024-07-26T11:19:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:41 smithi097 ceph-mon[32231]: pgmap v7003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:19:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12017.935269 secs 2024-07-26T11:19:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:43 smithi115 ceph-mon[35364]: pgmap v7004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:43 smithi181 ceph-mon[35869]: pgmap v7004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:43 smithi097 ceph-mon[32231]: pgmap v7004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:19:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:45 smithi181 ceph-mon[35869]: pgmap v7005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:45 smithi097 ceph-mon[32231]: pgmap v7005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:45 smithi115 ceph-mon[35364]: pgmap v7005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12022.935412 secs 2024-07-26T11:19:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12022.935412 secs 2024-07-26T11:19:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12022.935412 secs 2024-07-26T11:19:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:47 smithi097 ceph-mon[32231]: pgmap v7006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:19:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:47 smithi115 ceph-mon[35364]: pgmap v7006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:19:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:47 smithi181 ceph-mon[35869]: pgmap v7006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:19:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:49 smithi115 ceph-mon[35364]: pgmap v7007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:19:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:49 smithi181 ceph-mon[35869]: pgmap v7007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:19:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:49 smithi097 ceph-mon[32231]: pgmap v7007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:19:52.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:51 smithi115 ceph-mon[35364]: pgmap v7008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:19:52.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12027.935558 secs 2024-07-26T11:19:52.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:51 smithi181 ceph-mon[35869]: pgmap v7008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:19:52.037 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12027.935558 secs 2024-07-26T11:19:52.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:51 smithi097 ceph-mon[32231]: pgmap v7008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:19:52.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12027.935558 secs 2024-07-26T11:19:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:52.732+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:52.733+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:19:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:52.733+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:19:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:52.733+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:19:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:19:52.733+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:53 smithi181 ceph-mon[35869]: pgmap v7009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:19:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:53 smithi115 ceph-mon[35364]: pgmap v7009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:19:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:53 smithi097 ceph-mon[32231]: pgmap v7009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:19:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:55 smithi115 ceph-mon[35364]: pgmap v7010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:19:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:55 smithi181 ceph-mon[35869]: pgmap v7010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:19:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:55 smithi097 ceph-mon[32231]: pgmap v7010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:19:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12032.935688 secs 2024-07-26T11:19:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12032.935688 secs 2024-07-26T11:19:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12032.935688 secs 2024-07-26T11:19:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:57 smithi115 ceph-mon[35364]: pgmap v7011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:57 smithi181 ceph-mon[35869]: pgmap v7011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:57 smithi097 ceph-mon[32231]: pgmap v7011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:19:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:19:58 smithi115 ceph-mon[35364]: pgmap v7012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:19:58 smithi181 ceph-mon[35869]: pgmap v7012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:19:58 smithi097 ceph-mon[32231]: pgmap v7012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:00 smithi115 ceph-mon[35364]: pgmap v7013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:00 smithi181 ceph-mon[35869]: pgmap v7013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:00 smithi097 ceph-mon[32231]: pgmap v7013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12037.935839 secs 2024-07-26T11:20:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12037.935839 secs 2024-07-26T11:20:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12037.935839 secs 2024-07-26T11:20:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:02 smithi115 ceph-mon[35364]: pgmap v7014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:02 smithi181 ceph-mon[35869]: pgmap v7014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:02 smithi097 ceph-mon[32231]: pgmap v7014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:04 smithi115 ceph-mon[35364]: pgmap v7015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:04 smithi181 ceph-mon[35869]: pgmap v7015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:04 smithi097 ceph-mon[32231]: pgmap v7015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12042.935987 secs 2024-07-26T11:20:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:06 smithi115 ceph-mon[35364]: pgmap v7016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12042.935987 secs 2024-07-26T11:20:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:06 smithi181 ceph-mon[35869]: pgmap v7016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12042.935987 secs 2024-07-26T11:20:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:06 smithi097 ceph-mon[32231]: pgmap v7016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:07.740+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:07.741+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:20:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:07.741+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:20:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:07.741+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:07.741+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:08 smithi115 ceph-mon[35364]: pgmap v7017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:08 smithi181 ceph-mon[35869]: pgmap v7017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:08 smithi097 ceph-mon[32231]: pgmap v7017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:10 smithi115 ceph-mon[35364]: pgmap v7018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:10 smithi181 ceph-mon[35869]: pgmap v7018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:10 smithi097 ceph-mon[32231]: pgmap v7018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12047.936136 secs 2024-07-26T11:20:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12047.936136 secs 2024-07-26T11:20:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12047.936136 secs 2024-07-26T11:20:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:12 smithi115 ceph-mon[35364]: pgmap v7019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:12 smithi181 ceph-mon[35869]: pgmap v7019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:12 smithi097 ceph-mon[32231]: pgmap v7019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:14 smithi115 ceph-mon[35364]: pgmap v7020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:14 smithi181 ceph-mon[35869]: pgmap v7020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:14 smithi097 ceph-mon[32231]: pgmap v7020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12052.936318 secs 2024-07-26T11:20:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12052.936318 secs 2024-07-26T11:20:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12052.936318 secs 2024-07-26T11:20:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:16 smithi115 ceph-mon[35364]: pgmap v7021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:16 smithi181 ceph-mon[35869]: pgmap v7021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:16 smithi097 ceph-mon[32231]: pgmap v7021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:18 smithi115 ceph-mon[35364]: pgmap v7022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:20:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:18 smithi181 ceph-mon[35869]: pgmap v7022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:20:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:18 smithi097 ceph-mon[32231]: pgmap v7022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:20:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:21 smithi115 ceph-mon[35364]: pgmap v7023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:20:21.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12057.936456 secs 2024-07-26T11:20:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:21 smithi181 ceph-mon[35869]: pgmap v7023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:20:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12057.936456 secs 2024-07-26T11:20:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:21 smithi097 ceph-mon[32231]: pgmap v7023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:20:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12057.936456 secs 2024-07-26T11:20:22.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:22.742+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:22.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:22.744+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:20:22.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:22.744+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:20:22.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:22.744+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:22.746 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:22.744+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:23.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:23 smithi115 ceph-mon[35364]: pgmap v7024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:20:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:23 smithi181 ceph-mon[35869]: pgmap v7024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:20:24.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:23 smithi097 ceph-mon[32231]: pgmap v7024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:20:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:25 smithi115 ceph-mon[35364]: pgmap v7025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:25 smithi181 ceph-mon[35869]: pgmap v7025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:25 smithi097 ceph-mon[32231]: pgmap v7025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12062.936602 secs 2024-07-26T11:20:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12062.936602 secs 2024-07-26T11:20:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12062.936602 secs 2024-07-26T11:20:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:27 smithi115 ceph-mon[35364]: pgmap v7026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:20:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:27 smithi181 ceph-mon[35869]: pgmap v7026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:20:28.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:27 smithi097 ceph-mon[32231]: pgmap v7026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:20:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:29 smithi115 ceph-mon[35364]: pgmap v7027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:20:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:29 smithi181 ceph-mon[35869]: pgmap v7027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:20:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:29 smithi097 ceph-mon[32231]: pgmap v7027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:20:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:31 smithi115 ceph-mon[35364]: pgmap v7028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12067.936744 secs 2024-07-26T11:20:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:31 smithi181 ceph-mon[35869]: pgmap v7028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12067.936744 secs 2024-07-26T11:20:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:31 smithi097 ceph-mon[32231]: pgmap v7028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12067.936744 secs 2024-07-26T11:20:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:33 smithi115 ceph-mon[35364]: pgmap v7029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:33 smithi181 ceph-mon[35869]: pgmap v7029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:34.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:33 smithi097 ceph-mon[32231]: pgmap v7029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:35 smithi115 ceph-mon[35364]: pgmap v7030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:35 smithi181 ceph-mon[35869]: pgmap v7030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:35 smithi097 ceph-mon[32231]: pgmap v7030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12072.936888 secs 2024-07-26T11:20:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12072.936888 secs 2024-07-26T11:20:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12072.936888 secs 2024-07-26T11:20:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:37.749+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:37.753 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:37.751+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:20:37.753 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:37.751+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:20:37.753 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:37.751+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:37.753 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:37.751+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:37 smithi097 ceph-mon[32231]: pgmap v7031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:38.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:20:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:37 smithi115 ceph-mon[35364]: pgmap v7031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:20:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:37 smithi181 ceph-mon[35869]: pgmap v7031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:38.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:20:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:38 smithi097 ceph-mon[32231]: pgmap v7032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:20:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:38 smithi097 ceph-mon[32231]: pgmap v7033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:20:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:38 smithi115 ceph-mon[35364]: pgmap v7032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:20:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:38 smithi115 ceph-mon[35364]: pgmap v7033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:20:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:38 smithi181 ceph-mon[35869]: pgmap v7032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:20:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:38 smithi181 ceph-mon[35869]: pgmap v7033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:20:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:40 smithi115 ceph-mon[35364]: pgmap v7034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:40 smithi181 ceph-mon[35869]: pgmap v7034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:40 smithi097 ceph-mon[32231]: pgmap v7034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12077.937026 secs 2024-07-26T11:20:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12077.937026 secs 2024-07-26T11:20:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12077.937026 secs 2024-07-26T11:20:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:42 smithi115 ceph-mon[35364]: pgmap v7035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:42 smithi181 ceph-mon[35869]: pgmap v7035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:42 smithi097 ceph-mon[32231]: pgmap v7035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:44 smithi115 ceph-mon[35364]: pgmap v7036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:20:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:44 smithi181 ceph-mon[35869]: pgmap v7036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:20:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:44 smithi097 ceph-mon[32231]: pgmap v7036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:20:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12082.937168 secs 2024-07-26T11:20:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:46 smithi115 ceph-mon[35364]: pgmap v7037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:20:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12082.937168 secs 2024-07-26T11:20:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:46 smithi181 ceph-mon[35869]: pgmap v7037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:20:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12082.937168 secs 2024-07-26T11:20:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:46 smithi097 ceph-mon[32231]: pgmap v7037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:20:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:48 smithi115 ceph-mon[35364]: pgmap v7038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:20:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:48 smithi181 ceph-mon[35869]: pgmap v7038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:20:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:48 smithi097 ceph-mon[32231]: pgmap v7038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:20:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:50 smithi115 ceph-mon[35364]: pgmap v7039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:20:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:50 smithi181 ceph-mon[35869]: pgmap v7039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:20:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:50 smithi097 ceph-mon[32231]: pgmap v7039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:20:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12087.937347 secs 2024-07-26T11:20:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12087.937347 secs 2024-07-26T11:20:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12087.937347 secs 2024-07-26T11:20:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:52.752+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:52.755+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:20:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:52.755+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:20:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:52.755+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:20:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:20:52.755+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:53.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:52 smithi115 ceph-mon[35364]: pgmap v7040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:53.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:52 smithi181 ceph-mon[35869]: pgmap v7040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:53.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:52 smithi097 ceph-mon[32231]: pgmap v7040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:54.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:53 smithi115 ceph-mon[35364]: pgmap v7041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:20:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:53 smithi181 ceph-mon[35869]: pgmap v7041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:20:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:53 smithi097 ceph-mon[32231]: pgmap v7041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:20:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12092.937484 secs 2024-07-26T11:20:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:56 smithi115 ceph-mon[35364]: pgmap v7042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:20:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12092.937484 secs 2024-07-26T11:20:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:56 smithi181 ceph-mon[35869]: pgmap v7042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:20:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12092.937484 secs 2024-07-26T11:20:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:56 smithi097 ceph-mon[32231]: pgmap v7042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:20:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:20:58 smithi115 ceph-mon[35364]: pgmap v7043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:20:58 smithi181 ceph-mon[35869]: pgmap v7043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:20:58 smithi097 ceph-mon[32231]: pgmap v7043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:00 smithi115 ceph-mon[35364]: pgmap v7044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:00 smithi181 ceph-mon[35869]: pgmap v7044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:00 smithi097 ceph-mon[32231]: pgmap v7044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12097.937620 secs 2024-07-26T11:21:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12097.937620 secs 2024-07-26T11:21:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12097.937620 secs 2024-07-26T11:21:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:02 smithi115 ceph-mon[35364]: pgmap v7045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:02 smithi181 ceph-mon[35869]: pgmap v7045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:02 smithi097 ceph-mon[32231]: pgmap v7045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:04 smithi115 ceph-mon[35364]: pgmap v7046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:04 smithi181 ceph-mon[35869]: pgmap v7046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:04 smithi097 ceph-mon[32231]: pgmap v7046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12102.937754 secs 2024-07-26T11:21:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:06 smithi115 ceph-mon[35364]: pgmap v7047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12102.937754 secs 2024-07-26T11:21:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:06 smithi181 ceph-mon[35869]: pgmap v7047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12102.937754 secs 2024-07-26T11:21:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:06 smithi097 ceph-mon[32231]: pgmap v7047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:07.762+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:07.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:07.763+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:21:07.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:07.763+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:21:07.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:07.763+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:21:07.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:07.763+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:08 smithi115 ceph-mon[35364]: pgmap v7048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:08 smithi181 ceph-mon[35869]: pgmap v7048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:08 smithi097 ceph-mon[32231]: pgmap v7048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:10 smithi115 ceph-mon[35364]: pgmap v7049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:10 smithi181 ceph-mon[35869]: pgmap v7049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:10 smithi097 ceph-mon[32231]: pgmap v7049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12107.937895 secs 2024-07-26T11:21:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12107.937895 secs 2024-07-26T11:21:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12107.937895 secs 2024-07-26T11:21:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:12 smithi115 ceph-mon[35364]: pgmap v7050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:12 smithi181 ceph-mon[35869]: pgmap v7050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:12 smithi097 ceph-mon[32231]: pgmap v7050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:14 smithi115 ceph-mon[35364]: pgmap v7051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:14 smithi181 ceph-mon[35869]: pgmap v7051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:14 smithi097 ceph-mon[32231]: pgmap v7051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12112.938055 secs 2024-07-26T11:21:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:16 smithi115 ceph-mon[35364]: pgmap v7052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12112.938055 secs 2024-07-26T11:21:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:16 smithi181 ceph-mon[35869]: pgmap v7052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12112.938055 secs 2024-07-26T11:21:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:16 smithi097 ceph-mon[32231]: pgmap v7052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:18 smithi115 ceph-mon[35364]: pgmap v7053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:18 smithi181 ceph-mon[35869]: pgmap v7053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:18 smithi097 ceph-mon[32231]: pgmap v7053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:20 smithi115 ceph-mon[35364]: pgmap v7054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:20 smithi181 ceph-mon[35869]: pgmap v7054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:20 smithi097 ceph-mon[32231]: pgmap v7054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12117.938218 secs 2024-07-26T11:21:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12117.938218 secs 2024-07-26T11:21:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12117.938218 secs 2024-07-26T11:21:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:22 smithi115 ceph-mon[35364]: pgmap v7055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:22 smithi181 ceph-mon[35869]: pgmap v7055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:22 smithi097 ceph-mon[32231]: pgmap v7055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:22.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:22.766+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:21:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:22.766+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:21:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:22.766+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:21:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:22.766+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:24 smithi115 ceph-mon[35364]: pgmap v7056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:24 smithi181 ceph-mon[35869]: pgmap v7056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:24 smithi097 ceph-mon[32231]: pgmap v7056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12122.938363 secs 2024-07-26T11:21:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:26 smithi115 ceph-mon[35364]: pgmap v7057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12122.938363 secs 2024-07-26T11:21:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:26 smithi181 ceph-mon[35869]: pgmap v7057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12122.938363 secs 2024-07-26T11:21:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:26 smithi097 ceph-mon[32231]: pgmap v7057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:28 smithi115 ceph-mon[35364]: pgmap v7058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:28 smithi181 ceph-mon[35869]: pgmap v7058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:28 smithi097 ceph-mon[32231]: pgmap v7058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:30 smithi115 ceph-mon[35364]: pgmap v7059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:30 smithi181 ceph-mon[35869]: pgmap v7059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:30 smithi097 ceph-mon[32231]: pgmap v7059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12127.938457 secs 2024-07-26T11:21:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12127.938457 secs 2024-07-26T11:21:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12127.938457 secs 2024-07-26T11:21:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:32 smithi115 ceph-mon[35364]: pgmap v7060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:32 smithi181 ceph-mon[35869]: pgmap v7060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:32 smithi097 ceph-mon[32231]: pgmap v7060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:34 smithi181 ceph-mon[35869]: pgmap v7061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:34 smithi097 ceph-mon[32231]: pgmap v7061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:34 smithi115 ceph-mon[35364]: pgmap v7061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12132.938539 secs 2024-07-26T11:21:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:36 smithi097 ceph-mon[32231]: pgmap v7062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12132.938539 secs 2024-07-26T11:21:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:36 smithi115 ceph-mon[35364]: pgmap v7062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12132.938539 secs 2024-07-26T11:21:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:36 smithi181 ceph-mon[35869]: pgmap v7062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:37.772+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:37.773+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:21:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:37.773+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:21:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:37.773+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:21:37.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:37.773+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:38 smithi097 ceph-mon[32231]: pgmap v7063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:38 smithi115 ceph-mon[35364]: pgmap v7063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:38 smithi181 ceph-mon[35869]: pgmap v7063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:39 smithi115 ceph-mon[35364]: pgmap v7064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:21:39.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:39 smithi115 ceph-mon[35364]: pgmap v7065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:39.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:21:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:39 smithi181 ceph-mon[35869]: pgmap v7064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:21:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:39 smithi181 ceph-mon[35869]: pgmap v7065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:39.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:21:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:39 smithi097 ceph-mon[32231]: pgmap v7064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:21:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:39 smithi097 ceph-mon[32231]: pgmap v7065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:40.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:21:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12137.938637 secs 2024-07-26T11:21:41.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:41 smithi181 ceph-mon[35869]: pgmap v7066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:21:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12137.938637 secs 2024-07-26T11:21:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:41 smithi097 ceph-mon[32231]: pgmap v7066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:21:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12137.938637 secs 2024-07-26T11:21:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:41 smithi115 ceph-mon[35364]: pgmap v7066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:21:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:43 smithi097 ceph-mon[32231]: pgmap v7067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:43 smithi115 ceph-mon[35364]: pgmap v7067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:43 smithi181 ceph-mon[35869]: pgmap v7067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:45 smithi097 ceph-mon[32231]: pgmap v7068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:21:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:45 smithi115 ceph-mon[35364]: pgmap v7068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:21:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:45 smithi181 ceph-mon[35869]: pgmap v7068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:21:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12142.938725 secs 2024-07-26T11:21:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12142.938725 secs 2024-07-26T11:21:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12142.938725 secs 2024-07-26T11:21:48.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:47 smithi115 ceph-mon[35364]: pgmap v7069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:47 smithi181 ceph-mon[35869]: pgmap v7069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:47 smithi097 ceph-mon[32231]: pgmap v7069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:49 smithi115 ceph-mon[35364]: pgmap v7070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:21:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:49 smithi181 ceph-mon[35869]: pgmap v7070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:21:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:49 smithi097 ceph-mon[32231]: pgmap v7070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:21:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12147.938884 secs 2024-07-26T11:21:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:51 smithi115 ceph-mon[35364]: pgmap v7071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12147.938884 secs 2024-07-26T11:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:51 smithi181 ceph-mon[35869]: pgmap v7071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12147.938884 secs 2024-07-26T11:21:52.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:51 smithi097 ceph-mon[32231]: pgmap v7071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:52.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:52.775+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:52.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:52.776+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:21:52.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:52.776+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:21:52.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:52.776+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:21:52.778 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:21:52.776+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:54.028 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:53 smithi115 ceph-mon[35364]: pgmap v7072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:54.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:53 smithi181 ceph-mon[35869]: pgmap v7072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:54.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:53 smithi097 ceph-mon[32231]: pgmap v7072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:55 smithi115 ceph-mon[35364]: pgmap v7073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:55 smithi181 ceph-mon[35869]: pgmap v7073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:56.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:55 smithi097 ceph-mon[32231]: pgmap v7073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12152.939028 secs 2024-07-26T11:21:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12152.939028 secs 2024-07-26T11:21:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12152.939028 secs 2024-07-26T11:21:58.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:57 smithi115 ceph-mon[35364]: pgmap v7074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:57 smithi181 ceph-mon[35869]: pgmap v7074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:58.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:57 smithi097 ceph-mon[32231]: pgmap v7074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:21:59 smithi115 ceph-mon[35364]: pgmap v7075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:21:59 smithi181 ceph-mon[35869]: pgmap v7075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:21:59 smithi097 ceph-mon[32231]: pgmap v7075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12157.939170 secs 2024-07-26T11:22:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:01 smithi115 ceph-mon[35364]: pgmap v7076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12157.939170 secs 2024-07-26T11:22:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:01 smithi181 ceph-mon[35869]: pgmap v7076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12157.939170 secs 2024-07-26T11:22:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:01 smithi097 ceph-mon[32231]: pgmap v7076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:02 smithi115 ceph-mon[35364]: pgmap v7077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:02 smithi181 ceph-mon[35869]: pgmap v7077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:02 smithi097 ceph-mon[32231]: pgmap v7077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:05 smithi097 ceph-mon[32231]: pgmap v7078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:05 smithi115 ceph-mon[35364]: pgmap v7078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:05 smithi181 ceph-mon[35869]: pgmap v7078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12162.939311 secs 2024-07-26T11:22:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12162.939311 secs 2024-07-26T11:22:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12162.939311 secs 2024-07-26T11:22:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:07.783+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:07.784+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:22:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:07.784+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:22:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:07.784+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:22:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:07.784+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:07 smithi115 ceph-mon[35364]: pgmap v7079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:07 smithi181 ceph-mon[35869]: pgmap v7079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:07 smithi097 ceph-mon[32231]: pgmap v7079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:09.041+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T11:22:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:09.041+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:22:09.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:09.042+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T11:22:09.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:09.042+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:22:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:09 smithi115 ceph-mon[35364]: pgmap v7080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:09 smithi181 ceph-mon[35869]: pgmap v7080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:09 smithi097 ceph-mon[32231]: pgmap v7080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12167.939417 secs 2024-07-26T11:22:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:11 smithi115 ceph-mon[35364]: pgmap v7081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12167.939417 secs 2024-07-26T11:22:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:11 smithi181 ceph-mon[35869]: pgmap v7081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12167.939417 secs 2024-07-26T11:22:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:11 smithi097 ceph-mon[32231]: pgmap v7081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:13 smithi115 ceph-mon[35364]: pgmap v7082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:13 smithi181 ceph-mon[35869]: pgmap v7082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:13 smithi097 ceph-mon[32231]: pgmap v7082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:15 smithi115 ceph-mon[35364]: pgmap v7083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:15 smithi181 ceph-mon[35869]: pgmap v7083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:15 smithi097 ceph-mon[32231]: pgmap v7083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12172.939559 secs 2024-07-26T11:22:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12172.939559 secs 2024-07-26T11:22:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12172.939559 secs 2024-07-26T11:22:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:17 smithi115 ceph-mon[35364]: pgmap v7084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:17 smithi181 ceph-mon[35869]: pgmap v7084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:17 smithi097 ceph-mon[32231]: pgmap v7084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:19 smithi115 ceph-mon[35364]: pgmap v7085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:19 smithi181 ceph-mon[35869]: pgmap v7085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:19 smithi097 ceph-mon[32231]: pgmap v7085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12177.939710 secs 2024-07-26T11:22:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:21 smithi115 ceph-mon[35364]: pgmap v7086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12177.939710 secs 2024-07-26T11:22:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:21 smithi181 ceph-mon[35869]: pgmap v7086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12177.939710 secs 2024-07-26T11:22:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:21 smithi097 ceph-mon[32231]: pgmap v7086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:22.785+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:22.786+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:22:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:22.786+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:22:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:22.786+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:22:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:22.786+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:23 smithi115 ceph-mon[35364]: pgmap v7087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:23 smithi181 ceph-mon[35869]: pgmap v7087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:23 smithi097 ceph-mon[32231]: pgmap v7087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:24 smithi115 ceph-mon[35364]: pgmap v7088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:24 smithi181 ceph-mon[35869]: pgmap v7088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:24 smithi097 ceph-mon[32231]: pgmap v7088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12182.939852 secs 2024-07-26T11:22:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12182.939852 secs 2024-07-26T11:22:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12182.939852 secs 2024-07-26T11:22:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:26 smithi097 ceph-mon[32231]: pgmap v7089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:26 smithi115 ceph-mon[35364]: pgmap v7089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:26 smithi181 ceph-mon[35869]: pgmap v7089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:29 smithi097 ceph-mon[32231]: pgmap v7090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:29 smithi115 ceph-mon[35364]: pgmap v7090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:29 smithi181 ceph-mon[35869]: pgmap v7090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12187.939997 secs 2024-07-26T11:22:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:31 smithi115 ceph-mon[35364]: pgmap v7091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12187.939997 secs 2024-07-26T11:22:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:31 smithi181 ceph-mon[35869]: pgmap v7091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12187.939997 secs 2024-07-26T11:22:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:31 smithi097 ceph-mon[32231]: pgmap v7091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:33 smithi115 ceph-mon[35364]: pgmap v7092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:33 smithi181 ceph-mon[35869]: pgmap v7092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:33 smithi097 ceph-mon[32231]: pgmap v7092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:35 smithi115 ceph-mon[35364]: pgmap v7093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:35 smithi181 ceph-mon[35869]: pgmap v7093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:35 smithi097 ceph-mon[32231]: pgmap v7093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12192.940142 secs 2024-07-26T11:22:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12192.940142 secs 2024-07-26T11:22:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12192.940142 secs 2024-07-26T11:22:37.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:37.793+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:37.795+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:22:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:37.795+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:22:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:37.795+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:22:37.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:37.795+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:37 smithi115 ceph-mon[35364]: pgmap v7094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:37 smithi181 ceph-mon[35869]: pgmap v7094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:38.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:37 smithi097 ceph-mon[32231]: pgmap v7094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:39 smithi115 ceph-mon[35364]: pgmap v7095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:39 smithi181 ceph-mon[35869]: pgmap v7095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:39 smithi097 ceph-mon[32231]: pgmap v7095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:22:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:22:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:22:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12197.940296 secs 2024-07-26T11:22:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:41 smithi115 ceph-mon[35364]: pgmap v7096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:22:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:41 smithi115 ceph-mon[35364]: pgmap v7097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:22:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12197.940296 secs 2024-07-26T11:22:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:41 smithi181 ceph-mon[35869]: pgmap v7096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:22:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:41 smithi181 ceph-mon[35869]: pgmap v7097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:22:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12197.940296 secs 2024-07-26T11:22:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:41 smithi097 ceph-mon[32231]: pgmap v7096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:22:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:41 smithi097 ceph-mon[32231]: pgmap v7097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:22:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:43 smithi115 ceph-mon[35364]: pgmap v7098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:22:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:43 smithi181 ceph-mon[35869]: pgmap v7098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:22:44.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:43 smithi097 ceph-mon[32231]: pgmap v7098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:22:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:45 smithi115 ceph-mon[35364]: pgmap v7099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:22:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:45 smithi181 ceph-mon[35869]: pgmap v7099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:22:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:45 smithi097 ceph-mon[32231]: pgmap v7099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:22:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12202.940442 secs 2024-07-26T11:22:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:46 smithi115 ceph-mon[35364]: pgmap v7100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:22:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12202.940442 secs 2024-07-26T11:22:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:46 smithi181 ceph-mon[35869]: pgmap v7100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:22:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12202.940442 secs 2024-07-26T11:22:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:46 smithi097 ceph-mon[32231]: pgmap v7100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:22:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:49 smithi115 ceph-mon[35364]: pgmap v7101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:22:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:49 smithi181 ceph-mon[35869]: pgmap v7101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:22:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:49 smithi097 ceph-mon[32231]: pgmap v7101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:22:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12207.940593 secs 2024-07-26T11:22:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:51 smithi115 ceph-mon[35364]: pgmap v7102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:22:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12207.940593 secs 2024-07-26T11:22:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:51 smithi181 ceph-mon[35869]: pgmap v7102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:22:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12207.940593 secs 2024-07-26T11:22:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:51 smithi097 ceph-mon[32231]: pgmap v7102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:22:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:52.797+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:52.798+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:22:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:52.798+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:22:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:52.798+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:22:52.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:22:52.798+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:53 smithi115 ceph-mon[35364]: pgmap v7103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:53 smithi181 ceph-mon[35869]: pgmap v7103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:53 smithi097 ceph-mon[32231]: pgmap v7103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:55 smithi115 ceph-mon[35364]: pgmap v7104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:22:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:55 smithi181 ceph-mon[35869]: pgmap v7104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:22:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:55 smithi097 ceph-mon[32231]: pgmap v7104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:22:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12212.940733 secs 2024-07-26T11:22:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12212.940733 secs 2024-07-26T11:22:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12212.940733 secs 2024-07-26T11:22:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:57 smithi115 ceph-mon[35364]: pgmap v7105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:22:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:57 smithi181 ceph-mon[35869]: pgmap v7105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:22:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:57 smithi097 ceph-mon[32231]: pgmap v7105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:22:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:22:59 smithi115 ceph-mon[35364]: pgmap v7106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:22:59 smithi181 ceph-mon[35869]: pgmap v7106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:59.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:22:59 smithi097 ceph-mon[32231]: pgmap v7106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12217.940883 secs 2024-07-26T11:23:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:01 smithi115 ceph-mon[35364]: pgmap v7107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12217.940883 secs 2024-07-26T11:23:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:01 smithi181 ceph-mon[35869]: pgmap v7107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12217.940883 secs 2024-07-26T11:23:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:01 smithi097 ceph-mon[32231]: pgmap v7107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:03 smithi115 ceph-mon[35364]: pgmap v7108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:03 smithi181 ceph-mon[35869]: pgmap v7108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:03 smithi097 ceph-mon[32231]: pgmap v7108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:05 smithi115 ceph-mon[35364]: pgmap v7109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:05 smithi181 ceph-mon[35869]: pgmap v7109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:05 smithi097 ceph-mon[32231]: pgmap v7109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12222.941023 secs 2024-07-26T11:23:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12222.941023 secs 2024-07-26T11:23:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12222.941023 secs 2024-07-26T11:23:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:07 smithi115 ceph-mon[35364]: pgmap v7110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:07 smithi181 ceph-mon[35869]: pgmap v7110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:07 smithi097 ceph-mon[32231]: pgmap v7110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:07.806+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:07.807+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:23:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:07.807+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:23:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:07.807+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:23:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:07.807+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:09 smithi181 ceph-mon[35869]: pgmap v7111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:09 smithi097 ceph-mon[32231]: pgmap v7111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:09 smithi115 ceph-mon[35364]: pgmap v7111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12227.941108 secs 2024-07-26T11:23:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:11 smithi097 ceph-mon[32231]: pgmap v7112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12227.941108 secs 2024-07-26T11:23:11.740 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:11 smithi115 ceph-mon[35364]: pgmap v7112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:11.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12227.941108 secs 2024-07-26T11:23:11.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:11 smithi181 ceph-mon[35869]: pgmap v7112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:13.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:13 smithi097 ceph-mon[32231]: pgmap v7113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:13 smithi115 ceph-mon[35364]: pgmap v7113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:13 smithi181 ceph-mon[35869]: pgmap v7113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:15 smithi097 ceph-mon[32231]: pgmap v7114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:15 smithi115 ceph-mon[35364]: pgmap v7114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:15 smithi181 ceph-mon[35869]: pgmap v7114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12232.941213 secs 2024-07-26T11:23:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12232.941213 secs 2024-07-26T11:23:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12232.941213 secs 2024-07-26T11:23:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:17 smithi115 ceph-mon[35364]: pgmap v7115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:17 smithi181 ceph-mon[35869]: pgmap v7115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:17 smithi097 ceph-mon[32231]: pgmap v7115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:19 smithi115 ceph-mon[35364]: pgmap v7116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:19 smithi181 ceph-mon[35869]: pgmap v7116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:19 smithi097 ceph-mon[32231]: pgmap v7116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12237.941346 secs 2024-07-26T11:23:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:21 smithi115 ceph-mon[35364]: pgmap v7117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12237.941346 secs 2024-07-26T11:23:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:21 smithi181 ceph-mon[35869]: pgmap v7117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12237.941346 secs 2024-07-26T11:23:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:21 smithi097 ceph-mon[32231]: pgmap v7117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:22.808+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:22.809+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:23:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:22.809+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:23:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:22.809+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:23:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:22.809+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:23 smithi115 ceph-mon[35364]: pgmap v7118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:23 smithi181 ceph-mon[35869]: pgmap v7118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:23 smithi097 ceph-mon[32231]: pgmap v7118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:25.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:25 smithi097 ceph-mon[32231]: pgmap v7119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:25 smithi115 ceph-mon[35364]: pgmap v7119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:25 smithi181 ceph-mon[35869]: pgmap v7119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12242.941445 secs 2024-07-26T11:23:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12242.941445 secs 2024-07-26T11:23:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12242.941445 secs 2024-07-26T11:23:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:27 smithi115 ceph-mon[35364]: pgmap v7120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:27 smithi181 ceph-mon[35869]: pgmap v7120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:27 smithi097 ceph-mon[32231]: pgmap v7120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:29.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:29 smithi115 ceph-mon[35364]: pgmap v7121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:29 smithi181 ceph-mon[35869]: pgmap v7121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:29 smithi097 ceph-mon[32231]: pgmap v7121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12247.941586 secs 2024-07-26T11:23:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:31 smithi115 ceph-mon[35364]: pgmap v7122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12247.941586 secs 2024-07-26T11:23:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:31 smithi181 ceph-mon[35869]: pgmap v7122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12247.941586 secs 2024-07-26T11:23:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:31 smithi097 ceph-mon[32231]: pgmap v7122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:33.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:33 smithi115 ceph-mon[35364]: pgmap v7123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:33 smithi181 ceph-mon[35869]: pgmap v7123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:33.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:33 smithi097 ceph-mon[32231]: pgmap v7123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:35 smithi115 ceph-mon[35364]: pgmap v7124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:35 smithi181 ceph-mon[35869]: pgmap v7124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:35 smithi097 ceph-mon[32231]: pgmap v7124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12252.941686 secs 2024-07-26T11:23:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12252.941686 secs 2024-07-26T11:23:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12252.941686 secs 2024-07-26T11:23:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:37 smithi115 ceph-mon[35364]: pgmap v7125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:37 smithi181 ceph-mon[35869]: pgmap v7125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:37.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:37 smithi097 ceph-mon[32231]: pgmap v7125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:37.815+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:37.817+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:23:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:37.817+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:23:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:37.817+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:23:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:37.817+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:39.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:39 smithi115 ceph-mon[35364]: pgmap v7126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:39 smithi181 ceph-mon[35869]: pgmap v7126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:39.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:39 smithi097 ceph-mon[32231]: pgmap v7126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12257.941832 secs 2024-07-26T11:23:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:41 smithi115 ceph-mon[35364]: pgmap v7127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12257.941832 secs 2024-07-26T11:23:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:41 smithi181 ceph-mon[35869]: pgmap v7127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:41.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12257.941832 secs 2024-07-26T11:23:41.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:41 smithi097 ceph-mon[32231]: pgmap v7127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:41.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:23:42.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:23:42.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:23:42.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:23:42.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:42.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:42.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: pgmap v7128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:23:42.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: pgmap v7129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:42.727 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:23:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: pgmap v7128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: pgmap v7129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:23:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:42.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: pgmap v7128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:23:42.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: pgmap v7129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:42.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:23:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:44 smithi115 ceph-mon[35364]: pgmap v7130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:44 smithi181 ceph-mon[35869]: pgmap v7130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:44 smithi097 ceph-mon[32231]: pgmap v7130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12262.941975 secs 2024-07-26T11:23:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:46 smithi115 ceph-mon[35364]: pgmap v7131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12262.941975 secs 2024-07-26T11:23:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:46 smithi181 ceph-mon[35869]: pgmap v7131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12262.941975 secs 2024-07-26T11:23:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:46 smithi097 ceph-mon[32231]: pgmap v7131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:48 smithi115 ceph-mon[35364]: pgmap v7132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:23:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:48 smithi181 ceph-mon[35869]: pgmap v7132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:23:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:48 smithi097 ceph-mon[32231]: pgmap v7132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:23:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:50 smithi115 ceph-mon[35364]: pgmap v7133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:50 smithi181 ceph-mon[35869]: pgmap v7133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:50 smithi097 ceph-mon[32231]: pgmap v7133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12267.942076 secs 2024-07-26T11:23:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12267.942076 secs 2024-07-26T11:23:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12267.942076 secs 2024-07-26T11:23:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:52.818+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:52.820+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:23:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:52.820+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:23:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:52.820+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:23:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:23:52.820+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:52 smithi115 ceph-mon[35364]: pgmap v7134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:23:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:52 smithi181 ceph-mon[35869]: pgmap v7134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:23:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:52 smithi097 ceph-mon[32231]: pgmap v7134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:23:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:54 smithi115 ceph-mon[35364]: pgmap v7135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:54 smithi181 ceph-mon[35869]: pgmap v7135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:54 smithi097 ceph-mon[32231]: pgmap v7135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12272.942219 secs 2024-07-26T11:23:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:56 smithi115 ceph-mon[35364]: pgmap v7136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12272.942219 secs 2024-07-26T11:23:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:56 smithi181 ceph-mon[35869]: pgmap v7136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12272.942219 secs 2024-07-26T11:23:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:56 smithi097 ceph-mon[32231]: pgmap v7136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:23:58 smithi115 ceph-mon[35364]: pgmap v7137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:23:58 smithi181 ceph-mon[35869]: pgmap v7137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:23:58 smithi097 ceph-mon[32231]: pgmap v7137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:00 smithi115 ceph-mon[35364]: pgmap v7138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:24:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:00 smithi181 ceph-mon[35869]: pgmap v7138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:24:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:00 smithi097 ceph-mon[32231]: pgmap v7138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:24:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12277.942358 secs 2024-07-26T11:24:01.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12277.942358 secs 2024-07-26T11:24:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12277.942358 secs 2024-07-26T11:24:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:02 smithi115 ceph-mon[35364]: pgmap v7139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:02 smithi181 ceph-mon[35869]: pgmap v7139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:02 smithi097 ceph-mon[32231]: pgmap v7139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:04 smithi115 ceph-mon[35364]: pgmap v7140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:04 smithi181 ceph-mon[35869]: pgmap v7140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:04 smithi097 ceph-mon[32231]: pgmap v7140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12282.942505 secs 2024-07-26T11:24:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:06 smithi115 ceph-mon[35364]: pgmap v7141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12282.942505 secs 2024-07-26T11:24:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:06 smithi181 ceph-mon[35869]: pgmap v7141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12282.942505 secs 2024-07-26T11:24:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:06 smithi097 ceph-mon[32231]: pgmap v7141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:07.826+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:07.827+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:24:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:07.827+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:24:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:07.827+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:24:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:07.828+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:08 smithi115 ceph-mon[35364]: pgmap v7142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:08 smithi181 ceph-mon[35869]: pgmap v7142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:08 smithi097 ceph-mon[32231]: pgmap v7142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:10 smithi181 ceph-mon[35869]: pgmap v7143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:24:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:10 smithi097 ceph-mon[32231]: pgmap v7143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:24:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:10 smithi115 ceph-mon[35364]: pgmap v7143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:24:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12287.942654 secs 2024-07-26T11:24:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12287.942654 secs 2024-07-26T11:24:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12287.942654 secs 2024-07-26T11:24:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:12 smithi097 ceph-mon[32231]: pgmap v7144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:12 smithi115 ceph-mon[35364]: pgmap v7144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:12 smithi181 ceph-mon[35869]: pgmap v7144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:14 smithi097 ceph-mon[32231]: pgmap v7145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:14 smithi115 ceph-mon[35364]: pgmap v7145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:14 smithi181 ceph-mon[35869]: pgmap v7145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12292.942740 secs 2024-07-26T11:24:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:16 smithi115 ceph-mon[35364]: pgmap v7146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12292.942740 secs 2024-07-26T11:24:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:16 smithi181 ceph-mon[35869]: pgmap v7146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12292.942740 secs 2024-07-26T11:24:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:16 smithi097 ceph-mon[32231]: pgmap v7146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:18 smithi115 ceph-mon[35364]: pgmap v7147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:18 smithi181 ceph-mon[35869]: pgmap v7147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:19.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:18 smithi097 ceph-mon[32231]: pgmap v7147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:20 smithi115 ceph-mon[35364]: pgmap v7148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:20 smithi181 ceph-mon[35869]: pgmap v7148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:20 smithi097 ceph-mon[32231]: pgmap v7148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12297.942836 secs 2024-07-26T11:24:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12297.942836 secs 2024-07-26T11:24:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12297.942836 secs 2024-07-26T11:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:22.830+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:22.831+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:24:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:22.831+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:22.831+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:22.832+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:22 smithi115 ceph-mon[35364]: pgmap v7149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:22 smithi181 ceph-mon[35869]: pgmap v7149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:22 smithi097 ceph-mon[32231]: pgmap v7149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:24.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:24.066+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T11:24:24.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:24.066+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:24:24.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:24.067+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:24:24.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:24.067+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:24:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:24 smithi115 ceph-mon[35364]: pgmap v7150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:24 smithi181 ceph-mon[35869]: pgmap v7150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:24 smithi097 ceph-mon[32231]: pgmap v7150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:26.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12302.942978 secs 2024-07-26T11:24:26.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12302.942978 secs 2024-07-26T11:24:26.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12302.942978 secs 2024-07-26T11:24:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:26 smithi115 ceph-mon[35364]: pgmap v7151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:26 smithi181 ceph-mon[35869]: pgmap v7151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:26 smithi097 ceph-mon[32231]: pgmap v7151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:28 smithi115 ceph-mon[35364]: pgmap v7152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:28 smithi181 ceph-mon[35869]: pgmap v7152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:28 smithi097 ceph-mon[32231]: pgmap v7152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:30 smithi115 ceph-mon[35364]: pgmap v7153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12307.943117 secs 2024-07-26T11:24:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:30 smithi181 ceph-mon[35869]: pgmap v7153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12307.943117 secs 2024-07-26T11:24:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:30 smithi097 ceph-mon[32231]: pgmap v7153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12307.943117 secs 2024-07-26T11:24:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:32 smithi115 ceph-mon[35364]: pgmap v7154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:32 smithi181 ceph-mon[35869]: pgmap v7154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:32 smithi097 ceph-mon[32231]: pgmap v7154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:33 smithi115 ceph-mon[35364]: pgmap v7155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:33 smithi181 ceph-mon[35869]: pgmap v7155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:33 smithi097 ceph-mon[32231]: pgmap v7155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12312.943291 secs 2024-07-26T11:24:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12312.943291 secs 2024-07-26T11:24:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12312.943291 secs 2024-07-26T11:24:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:36 smithi115 ceph-mon[35364]: pgmap v7156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:36 smithi181 ceph-mon[35869]: pgmap v7156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:36 smithi097 ceph-mon[32231]: pgmap v7156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:37.839+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:37.840+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:24:37.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:37.840+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:24:37.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:37.840+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:24:37.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:37.841+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:38 smithi115 ceph-mon[35364]: pgmap v7157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:38 smithi181 ceph-mon[35869]: pgmap v7157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:39.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:38 smithi097 ceph-mon[32231]: pgmap v7157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:40 smithi115 ceph-mon[35364]: pgmap v7158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12317.943412 secs 2024-07-26T11:24:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:40 smithi181 ceph-mon[35869]: pgmap v7158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12317.943412 secs 2024-07-26T11:24:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:40 smithi097 ceph-mon[32231]: pgmap v7158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12317.943412 secs 2024-07-26T11:24:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:42 smithi115 ceph-mon[35364]: pgmap v7159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:42 smithi181 ceph-mon[35869]: pgmap v7159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:42 smithi097 ceph-mon[32231]: pgmap v7159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:43.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:24:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:43.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:24:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:24:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:24:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:44 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:24:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:44 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:24:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:44 smithi115 ceph-mon[35364]: pgmap v7160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:24:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:44 smithi115 ceph-mon[35364]: pgmap v7161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:44 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:24:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:44 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:24:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:44 smithi181 ceph-mon[35869]: pgmap v7160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:24:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:44 smithi181 ceph-mon[35869]: pgmap v7161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:44 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:24:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:44 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:24:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:44 smithi097 ceph-mon[32231]: pgmap v7160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:24:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:44 smithi097 ceph-mon[32231]: pgmap v7161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12322.943559 secs 2024-07-26T11:24:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12322.943559 secs 2024-07-26T11:24:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12322.943559 secs 2024-07-26T11:24:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:46 smithi115 ceph-mon[35364]: pgmap v7162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:46 smithi181 ceph-mon[35869]: pgmap v7162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:46 smithi097 ceph-mon[32231]: pgmap v7162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:48 smithi115 ceph-mon[35364]: pgmap v7163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:24:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:48 smithi181 ceph-mon[35869]: pgmap v7163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:24:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:48 smithi097 ceph-mon[32231]: pgmap v7163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:24:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:50 smithi115 ceph-mon[35364]: pgmap v7164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12327.943708 secs 2024-07-26T11:24:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:50 smithi181 ceph-mon[35869]: pgmap v7164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12327.943708 secs 2024-07-26T11:24:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:50 smithi097 ceph-mon[32231]: pgmap v7164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12327.943708 secs 2024-07-26T11:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:52.843+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:52.844+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:52.844+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:52.844+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:24:52.844+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:52 smithi115 ceph-mon[35364]: pgmap v7165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:52 smithi181 ceph-mon[35869]: pgmap v7165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:52 smithi097 ceph-mon[32231]: pgmap v7165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:54 smithi115 ceph-mon[35364]: pgmap v7166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:24:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:54 smithi181 ceph-mon[35869]: pgmap v7166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:24:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:54 smithi097 ceph-mon[32231]: pgmap v7166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:24:56.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12332.943850 secs 2024-07-26T11:24:56.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12332.943850 secs 2024-07-26T11:24:56.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12332.943850 secs 2024-07-26T11:24:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:56 smithi115 ceph-mon[35364]: pgmap v7167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:56 smithi181 ceph-mon[35869]: pgmap v7167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:56 smithi097 ceph-mon[32231]: pgmap v7167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:24:58 smithi115 ceph-mon[35364]: pgmap v7168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:24:58 smithi181 ceph-mon[35869]: pgmap v7168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:24:58 smithi097 ceph-mon[32231]: pgmap v7168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:00 smithi115 ceph-mon[35364]: pgmap v7169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12337.943994 secs 2024-07-26T11:25:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:00 smithi181 ceph-mon[35869]: pgmap v7169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12337.943994 secs 2024-07-26T11:25:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:00 smithi097 ceph-mon[32231]: pgmap v7169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12337.943994 secs 2024-07-26T11:25:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:02 smithi115 ceph-mon[35364]: pgmap v7170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:02 smithi181 ceph-mon[35869]: pgmap v7170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:02 smithi097 ceph-mon[32231]: pgmap v7170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:04 smithi115 ceph-mon[35364]: pgmap v7171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:25:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:04 smithi181 ceph-mon[35869]: pgmap v7171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:25:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:04 smithi097 ceph-mon[32231]: pgmap v7171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:25:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12342.944155 secs 2024-07-26T11:25:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12342.944155 secs 2024-07-26T11:25:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12342.944155 secs 2024-07-26T11:25:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:06 smithi115 ceph-mon[35364]: pgmap v7172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:25:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:06 smithi181 ceph-mon[35869]: pgmap v7172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:25:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:06 smithi097 ceph-mon[32231]: pgmap v7172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:25:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:07.849+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:07.851+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:25:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:07.851+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:25:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:07.851+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:07.851+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:07 smithi115 ceph-mon[35364]: pgmap v7173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:25:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:07 smithi181 ceph-mon[35869]: pgmap v7173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:25:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:07 smithi097 ceph-mon[32231]: pgmap v7173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:25:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:09.075+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:25:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:09.075+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:25:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:09.076+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:25:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:09.076+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:25:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:10 smithi115 ceph-mon[35364]: pgmap v7174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:25:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12347.944324 secs 2024-07-26T11:25:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:10 smithi181 ceph-mon[35869]: pgmap v7174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:25:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12347.944324 secs 2024-07-26T11:25:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:10 smithi097 ceph-mon[32231]: pgmap v7174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:25:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12347.944324 secs 2024-07-26T11:25:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:12 smithi115 ceph-mon[35364]: pgmap v7175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:12 smithi181 ceph-mon[35869]: pgmap v7175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:12 smithi097 ceph-mon[32231]: pgmap v7175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:14 smithi115 ceph-mon[35364]: pgmap v7176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:14 smithi181 ceph-mon[35869]: pgmap v7176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:14 smithi097 ceph-mon[32231]: pgmap v7176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12352.944468 secs 2024-07-26T11:25:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12352.944468 secs 2024-07-26T11:25:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12352.944468 secs 2024-07-26T11:25:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:16 smithi115 ceph-mon[35364]: pgmap v7177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:16 smithi181 ceph-mon[35869]: pgmap v7177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:16 smithi097 ceph-mon[32231]: pgmap v7177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:18 smithi115 ceph-mon[35364]: pgmap v7178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:18 smithi181 ceph-mon[35869]: pgmap v7178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:18.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:18 smithi097 ceph-mon[32231]: pgmap v7178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:20 smithi115 ceph-mon[35364]: pgmap v7179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12357.944610 secs 2024-07-26T11:25:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:20 smithi181 ceph-mon[35869]: pgmap v7179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12357.944610 secs 2024-07-26T11:25:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:20 smithi097 ceph-mon[32231]: pgmap v7179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:20.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12357.944610 secs 2024-07-26T11:25:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:22 smithi115 ceph-mon[35364]: pgmap v7180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:22 smithi181 ceph-mon[35869]: pgmap v7180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:22 smithi097 ceph-mon[32231]: pgmap v7180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:22.853+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:22.854+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:25:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:22.854+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:25:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:22.854+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:22.854+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:24 smithi115 ceph-mon[35364]: pgmap v7181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:24 smithi181 ceph-mon[35869]: pgmap v7181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:24 smithi097 ceph-mon[32231]: pgmap v7181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12362.944708 secs 2024-07-26T11:25:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12362.944708 secs 2024-07-26T11:25:25.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12362.944708 secs 2024-07-26T11:25:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:26 smithi115 ceph-mon[35364]: pgmap v7182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:26 smithi181 ceph-mon[35869]: pgmap v7182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:26 smithi097 ceph-mon[32231]: pgmap v7182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:28 smithi115 ceph-mon[35364]: pgmap v7183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:28 smithi181 ceph-mon[35869]: pgmap v7183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:28.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:28 smithi097 ceph-mon[32231]: pgmap v7183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:30 smithi115 ceph-mon[35364]: pgmap v7184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12367.944852 secs 2024-07-26T11:25:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:30 smithi181 ceph-mon[35869]: pgmap v7184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12367.944852 secs 2024-07-26T11:25:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:30 smithi097 ceph-mon[32231]: pgmap v7184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12367.944852 secs 2024-07-26T11:25:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:32 smithi115 ceph-mon[35364]: pgmap v7185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:32 smithi181 ceph-mon[35869]: pgmap v7185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:32.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:32 smithi097 ceph-mon[32231]: pgmap v7185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:34 smithi115 ceph-mon[35364]: pgmap v7186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:34 smithi181 ceph-mon[35869]: pgmap v7186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:34 smithi097 ceph-mon[32231]: pgmap v7186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12372.944998 secs 2024-07-26T11:25:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12372.944998 secs 2024-07-26T11:25:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12372.944998 secs 2024-07-26T11:25:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:36 smithi097 ceph-mon[32231]: pgmap v7187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:36 smithi115 ceph-mon[35364]: pgmap v7187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:36 smithi181 ceph-mon[35869]: pgmap v7187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:37.861+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:37.862+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:25:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:37.862+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:25:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:37.862+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:37.863+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:38 smithi097 ceph-mon[32231]: pgmap v7188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:38 smithi115 ceph-mon[35364]: pgmap v7188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:38 smithi181 ceph-mon[35869]: pgmap v7188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:40 smithi097 ceph-mon[32231]: pgmap v7189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12377.945154 secs 2024-07-26T11:25:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:40 smithi115 ceph-mon[35364]: pgmap v7189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12377.945154 secs 2024-07-26T11:25:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:40 smithi181 ceph-mon[35869]: pgmap v7189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12377.945154 secs 2024-07-26T11:25:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:42 smithi115 ceph-mon[35364]: pgmap v7190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:42 smithi181 ceph-mon[35869]: pgmap v7190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:42 smithi097 ceph-mon[32231]: pgmap v7190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:44 smithi115 ceph-mon[35364]: pgmap v7191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:44 smithi181 ceph-mon[35869]: pgmap v7191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:44 smithi097 ceph-mon[32231]: pgmap v7191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12382.945297 secs 2024-07-26T11:25:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12382.945297 secs 2024-07-26T11:25:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12382.945297 secs 2024-07-26T11:25:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:45.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:25:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:46 smithi115 ceph-mon[35364]: pgmap v7192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:25:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:46 smithi115 ceph-mon[35364]: pgmap v7193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:46 smithi181 ceph-mon[35869]: pgmap v7192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:25:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:46 smithi181 ceph-mon[35869]: pgmap v7193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:46 smithi097 ceph-mon[32231]: pgmap v7192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:25:46.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:46 smithi097 ceph-mon[32231]: pgmap v7193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:48 smithi115 ceph-mon[35364]: pgmap v7194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:48 smithi181 ceph-mon[35869]: pgmap v7194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:48.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:48 smithi097 ceph-mon[32231]: pgmap v7194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:50 smithi115 ceph-mon[35364]: pgmap v7195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12387.945439 secs 2024-07-26T11:25:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:50 smithi181 ceph-mon[35869]: pgmap v7195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12387.945439 secs 2024-07-26T11:25:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:50 smithi097 ceph-mon[32231]: pgmap v7195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12387.945439 secs 2024-07-26T11:25:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:52 smithi115 ceph-mon[35364]: pgmap v7196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:52 smithi181 ceph-mon[35869]: pgmap v7196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:52.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:52 smithi097 ceph-mon[32231]: pgmap v7196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:52.865+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:52.866+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:25:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:52.866+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:25:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:52.866+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:25:52.866+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:54 smithi115 ceph-mon[35364]: pgmap v7197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:25:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:54 smithi181 ceph-mon[35869]: pgmap v7197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:25:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:54 smithi097 ceph-mon[32231]: pgmap v7197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:25:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12392.945587 secs 2024-07-26T11:25:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12392.945587 secs 2024-07-26T11:25:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12392.945587 secs 2024-07-26T11:25:56.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:56 smithi115 ceph-mon[35364]: pgmap v7198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:56.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:56 smithi181 ceph-mon[35869]: pgmap v7198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:56.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:56 smithi097 ceph-mon[32231]: pgmap v7198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:25:58 smithi115 ceph-mon[35364]: pgmap v7199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:25:58 smithi181 ceph-mon[35869]: pgmap v7199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:25:58 smithi097 ceph-mon[32231]: pgmap v7199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:00 smithi115 ceph-mon[35364]: pgmap v7200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:26:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12397.945728 secs 2024-07-26T11:26:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:00 smithi181 ceph-mon[35869]: pgmap v7200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:26:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12397.945728 secs 2024-07-26T11:26:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:00 smithi097 ceph-mon[32231]: pgmap v7200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:26:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12397.945728 secs 2024-07-26T11:26:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:02 smithi115 ceph-mon[35364]: pgmap v7201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:02 smithi181 ceph-mon[35869]: pgmap v7201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:02.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:02 smithi097 ceph-mon[32231]: pgmap v7201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:04 smithi115 ceph-mon[35364]: pgmap v7202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:26:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:04 smithi181 ceph-mon[35869]: pgmap v7202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:26:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:04 smithi097 ceph-mon[32231]: pgmap v7202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:26:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12402.945813 secs 2024-07-26T11:26:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12402.945813 secs 2024-07-26T11:26:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12402.945813 secs 2024-07-26T11:26:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:06 smithi115 ceph-mon[35364]: pgmap v7203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:06 smithi181 ceph-mon[35869]: pgmap v7203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:06 smithi097 ceph-mon[32231]: pgmap v7203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:07.874 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:07.872+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:07.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:07.873+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:26:07.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:07.873+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:26:07.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:07.873+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:07.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:07.874+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:08 smithi115 ceph-mon[35364]: pgmap v7204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:08 smithi181 ceph-mon[35869]: pgmap v7204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:08 smithi097 ceph-mon[32231]: pgmap v7204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:10 smithi115 ceph-mon[35364]: pgmap v7205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:26:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12407.945957 secs 2024-07-26T11:26:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:10 smithi181 ceph-mon[35869]: pgmap v7205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:26:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12407.945957 secs 2024-07-26T11:26:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:10 smithi097 ceph-mon[32231]: pgmap v7205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:26:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12407.945957 secs 2024-07-26T11:26:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:12 smithi115 ceph-mon[35364]: pgmap v7206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:12 smithi181 ceph-mon[35869]: pgmap v7206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:12 smithi097 ceph-mon[32231]: pgmap v7206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:14 smithi115 ceph-mon[35364]: pgmap v7207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:14 smithi181 ceph-mon[35869]: pgmap v7207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:14 smithi097 ceph-mon[32231]: pgmap v7207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12412.946103 secs 2024-07-26T11:26:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12412.946103 secs 2024-07-26T11:26:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12412.946103 secs 2024-07-26T11:26:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:16 smithi115 ceph-mon[35364]: pgmap v7208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:16 smithi181 ceph-mon[35869]: pgmap v7208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:16 smithi097 ceph-mon[32231]: pgmap v7208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:18 smithi115 ceph-mon[35364]: pgmap v7209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:18 smithi181 ceph-mon[35869]: pgmap v7209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:18 smithi097 ceph-mon[32231]: pgmap v7209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:20 smithi115 ceph-mon[35364]: pgmap v7210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12417.946285 secs 2024-07-26T11:26:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:20 smithi181 ceph-mon[35869]: pgmap v7210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12417.946285 secs 2024-07-26T11:26:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:20 smithi097 ceph-mon[32231]: pgmap v7210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12417.946285 secs 2024-07-26T11:26:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:22 smithi115 ceph-mon[35364]: pgmap v7211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:22 smithi181 ceph-mon[35869]: pgmap v7211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:22 smithi097 ceph-mon[32231]: pgmap v7211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:22.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:22.876+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:22.877+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:26:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:22.877+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:26:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:22.877+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:22.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:22.877+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:24 smithi115 ceph-mon[35364]: pgmap v7212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:24 smithi181 ceph-mon[35869]: pgmap v7212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:24 smithi097 ceph-mon[32231]: pgmap v7212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12422.946435 secs 2024-07-26T11:26:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12422.946435 secs 2024-07-26T11:26:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12422.946435 secs 2024-07-26T11:26:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:26 smithi115 ceph-mon[35364]: pgmap v7213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:26 smithi181 ceph-mon[35869]: pgmap v7213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:26 smithi097 ceph-mon[32231]: pgmap v7213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:28 smithi115 ceph-mon[35364]: pgmap v7214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:28 smithi181 ceph-mon[35869]: pgmap v7214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:28 smithi097 ceph-mon[32231]: pgmap v7214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:30 smithi181 ceph-mon[35869]: pgmap v7215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:30.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12427.946576 secs 2024-07-26T11:26:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:30 smithi097 ceph-mon[32231]: pgmap v7215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12427.946576 secs 2024-07-26T11:26:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:30 smithi115 ceph-mon[35364]: pgmap v7215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12427.946576 secs 2024-07-26T11:26:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:32 smithi097 ceph-mon[32231]: pgmap v7216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:32 smithi115 ceph-mon[35364]: pgmap v7216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:32 smithi181 ceph-mon[35869]: pgmap v7216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:34 smithi097 ceph-mon[32231]: pgmap v7217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:34 smithi115 ceph-mon[35364]: pgmap v7217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:34 smithi181 ceph-mon[35869]: pgmap v7217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12432.946721 secs 2024-07-26T11:26:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12432.946721 secs 2024-07-26T11:26:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12432.946721 secs 2024-07-26T11:26:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:36 smithi115 ceph-mon[35364]: pgmap v7218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:36 smithi181 ceph-mon[35869]: pgmap v7218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:36 smithi097 ceph-mon[32231]: pgmap v7218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:37.884+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:37.885+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:26:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:37.885+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:26:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:37.885+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:37.885+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:38 smithi115 ceph-mon[35364]: pgmap v7219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:38 smithi181 ceph-mon[35869]: pgmap v7219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:38 smithi097 ceph-mon[32231]: pgmap v7219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:40 smithi115 ceph-mon[35364]: pgmap v7220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12437.946873 secs 2024-07-26T11:26:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:40 smithi181 ceph-mon[35869]: pgmap v7220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12437.946873 secs 2024-07-26T11:26:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:40 smithi097 ceph-mon[32231]: pgmap v7220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:26:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12437.946873 secs 2024-07-26T11:26:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:42 smithi115 ceph-mon[35364]: pgmap v7221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:42 smithi181 ceph-mon[35869]: pgmap v7221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:42 smithi097 ceph-mon[32231]: pgmap v7221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:26:44.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:44 smithi115 ceph-mon[35364]: pgmap v7222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:44 smithi181 ceph-mon[35869]: pgmap v7222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:44 smithi097 ceph-mon[32231]: pgmap v7222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12442.946973 secs 2024-07-26T11:26:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12442.946973 secs 2024-07-26T11:26:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12442.946973 secs 2024-07-26T11:26:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:46 smithi115 ceph-mon[35364]: pgmap v7223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:46 smithi181 ceph-mon[35869]: pgmap v7223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:46.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:46 smithi097 ceph-mon[32231]: pgmap v7223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:26:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:48 smithi115 ceph-mon[35364]: pgmap v7224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:48 smithi115 ceph-mon[35364]: pgmap v7225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:48 smithi181 ceph-mon[35869]: pgmap v7224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:48 smithi181 ceph-mon[35869]: pgmap v7225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:48 smithi097 ceph-mon[32231]: pgmap v7224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:49.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:48 smithi097 ceph-mon[32231]: pgmap v7225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:50 smithi115 ceph-mon[35364]: pgmap v7226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12447.947115 secs 2024-07-26T11:26:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:50 smithi181 ceph-mon[35869]: pgmap v7226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12447.947115 secs 2024-07-26T11:26:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:50 smithi097 ceph-mon[32231]: pgmap v7226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12447.947115 secs 2024-07-26T11:26:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:52.886+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:52.888+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:26:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:52.888+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:26:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:52.888+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:26:52.888+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:52 smithi115 ceph-mon[35364]: pgmap v7227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:52 smithi181 ceph-mon[35869]: pgmap v7227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:52 smithi097 ceph-mon[32231]: pgmap v7227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:54 smithi115 ceph-mon[35364]: pgmap v7228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:54 smithi181 ceph-mon[35869]: pgmap v7228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:54 smithi097 ceph-mon[32231]: pgmap v7228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12452.947297 secs 2024-07-26T11:26:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12452.947297 secs 2024-07-26T11:26:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12452.947297 secs 2024-07-26T11:26:56.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:56 smithi115 ceph-mon[35364]: pgmap v7229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:56.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:56 smithi181 ceph-mon[35869]: pgmap v7229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:56.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:56 smithi097 ceph-mon[32231]: pgmap v7229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:26:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:26:58 smithi115 ceph-mon[35364]: pgmap v7230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:26:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:26:58 smithi181 ceph-mon[35869]: pgmap v7230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:26:59.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:26:58 smithi097 ceph-mon[32231]: pgmap v7230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:27:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:00 smithi115 ceph-mon[35364]: pgmap v7231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12457.947385 secs 2024-07-26T11:27:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:00 smithi181 ceph-mon[35869]: pgmap v7231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12457.947385 secs 2024-07-26T11:27:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:00 smithi097 ceph-mon[32231]: pgmap v7231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12457.947385 secs 2024-07-26T11:27:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:02 smithi115 ceph-mon[35364]: pgmap v7232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:02 smithi181 ceph-mon[35869]: pgmap v7232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:02 smithi097 ceph-mon[32231]: pgmap v7232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:04 smithi115 ceph-mon[35364]: pgmap v7233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:04 smithi181 ceph-mon[35869]: pgmap v7233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:04 smithi097 ceph-mon[32231]: pgmap v7233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12462.947553 secs 2024-07-26T11:27:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12462.947553 secs 2024-07-26T11:27:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12462.947553 secs 2024-07-26T11:27:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:06 smithi115 ceph-mon[35364]: pgmap v7234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:06 smithi181 ceph-mon[35869]: pgmap v7234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:07.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:06 smithi097 ceph-mon[32231]: pgmap v7234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:07.896 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:07.894+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:07.895+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:27:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:07.895+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:07.895+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:07.896+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:08 smithi115 ceph-mon[35364]: pgmap v7235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:08 smithi181 ceph-mon[35869]: pgmap v7235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:09.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:08 smithi097 ceph-mon[32231]: pgmap v7235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:10 smithi115 ceph-mon[35364]: pgmap v7236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12467.947703 secs 2024-07-26T11:27:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:10 smithi181 ceph-mon[35869]: pgmap v7236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12467.947703 secs 2024-07-26T11:27:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:10 smithi097 ceph-mon[32231]: pgmap v7236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12467.947703 secs 2024-07-26T11:27:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:12 smithi115 ceph-mon[35364]: pgmap v7237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:12 smithi181 ceph-mon[35869]: pgmap v7237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:12 smithi097 ceph-mon[32231]: pgmap v7237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:14 smithi115 ceph-mon[35364]: pgmap v7238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:14 smithi181 ceph-mon[35869]: pgmap v7238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:14 smithi097 ceph-mon[32231]: pgmap v7238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12472.947844 secs 2024-07-26T11:27:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12472.947844 secs 2024-07-26T11:27:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12472.947844 secs 2024-07-26T11:27:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:16 smithi115 ceph-mon[35364]: pgmap v7239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:16 smithi181 ceph-mon[35869]: pgmap v7239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:16 smithi097 ceph-mon[32231]: pgmap v7239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:18 smithi115 ceph-mon[35364]: pgmap v7240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:18 smithi181 ceph-mon[35869]: pgmap v7240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:18 smithi097 ceph-mon[32231]: pgmap v7240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:20 smithi115 ceph-mon[35364]: pgmap v7241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12477.947991 secs 2024-07-26T11:27:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:20 smithi181 ceph-mon[35869]: pgmap v7241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12477.947991 secs 2024-07-26T11:27:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:20 smithi097 ceph-mon[32231]: pgmap v7241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12477.947991 secs 2024-07-26T11:27:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:22.898+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:22.899+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:27:22.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:22.899+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:22.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:22.899+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:22.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:22.899+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:22 smithi115 ceph-mon[35364]: pgmap v7242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:22 smithi181 ceph-mon[35869]: pgmap v7242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:22 smithi097 ceph-mon[32231]: pgmap v7242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:24 smithi115 ceph-mon[35364]: pgmap v7243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:24 smithi181 ceph-mon[35869]: pgmap v7243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:24 smithi097 ceph-mon[32231]: pgmap v7243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:26.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12482.948164 secs 2024-07-26T11:27:26.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12482.948164 secs 2024-07-26T11:27:26.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12482.948164 secs 2024-07-26T11:27:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:26 smithi115 ceph-mon[35364]: pgmap v7244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:26 smithi181 ceph-mon[35869]: pgmap v7244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:27.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:26 smithi097 ceph-mon[32231]: pgmap v7244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:28 smithi115 ceph-mon[35364]: pgmap v7245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:28 smithi181 ceph-mon[35869]: pgmap v7245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:28 smithi097 ceph-mon[32231]: pgmap v7245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:29 smithi115 ceph-mon[35364]: pgmap v7246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:29 smithi181 ceph-mon[35869]: pgmap v7246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:29 smithi097 ceph-mon[32231]: pgmap v7246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12487.948338 secs 2024-07-26T11:27:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12487.948338 secs 2024-07-26T11:27:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12487.948338 secs 2024-07-26T11:27:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:31 smithi115 ceph-mon[35364]: pgmap v7247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:31 smithi181 ceph-mon[35869]: pgmap v7247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:32.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:31 smithi097 ceph-mon[32231]: pgmap v7247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:34 smithi115 ceph-mon[35364]: pgmap v7248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:34 smithi181 ceph-mon[35869]: pgmap v7248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:34 smithi097 ceph-mon[32231]: pgmap v7248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12492.948482 secs 2024-07-26T11:27:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12492.948482 secs 2024-07-26T11:27:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12492.948482 secs 2024-07-26T11:27:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:36 smithi115 ceph-mon[35364]: pgmap v7249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:36 smithi181 ceph-mon[35869]: pgmap v7249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:36 smithi097 ceph-mon[32231]: pgmap v7249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:37.906+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:37.907+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:27:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:37.907+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:37.907+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:37.907+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:38 smithi115 ceph-mon[35364]: pgmap v7250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:38 smithi181 ceph-mon[35869]: pgmap v7250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:38.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:38 smithi097 ceph-mon[32231]: pgmap v7250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:40 smithi115 ceph-mon[35364]: pgmap v7251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12497.948581 secs 2024-07-26T11:27:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:40 smithi181 ceph-mon[35869]: pgmap v7251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12497.948581 secs 2024-07-26T11:27:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:40 smithi097 ceph-mon[32231]: pgmap v7251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12497.948581 secs 2024-07-26T11:27:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:42 smithi115 ceph-mon[35364]: pgmap v7252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:42 smithi181 ceph-mon[35869]: pgmap v7252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:42 smithi097 ceph-mon[32231]: pgmap v7252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:44 smithi115 ceph-mon[35364]: pgmap v7253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:44 smithi181 ceph-mon[35869]: pgmap v7253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:44 smithi097 ceph-mon[32231]: pgmap v7253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12502.948666 secs 2024-07-26T11:27:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12502.948666 secs 2024-07-26T11:27:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12502.948666 secs 2024-07-26T11:27:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:46 smithi115 ceph-mon[35364]: pgmap v7254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:46 smithi181 ceph-mon[35869]: pgmap v7254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:46 smithi097 ceph-mon[32231]: pgmap v7254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:48 smithi097 ceph-mon[32231]: pgmap v7255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:48 smithi115 ceph-mon[35364]: pgmap v7255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:48 smithi181 ceph-mon[35869]: pgmap v7255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:50 smithi115 ceph-mon[35364]: pgmap v7256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12507.948770 secs 2024-07-26T11:27:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:50 smithi181 ceph-mon[35869]: pgmap v7256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12507.948770 secs 2024-07-26T11:27:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:50 smithi097 ceph-mon[32231]: pgmap v7256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12507.948770 secs 2024-07-26T11:27:51.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.980 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:51.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:52.908+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:52.909+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:27:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:52.909+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:52.909+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:27:52.909+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:52 smithi115 ceph-mon[35364]: pgmap v7257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:52 smithi115 ceph-mon[35364]: pgmap v7258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:52 smithi115 ceph-mon[35364]: pgmap v7259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:27:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:52 smithi181 ceph-mon[35869]: pgmap v7257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:52.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:52.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:52 smithi181 ceph-mon[35869]: pgmap v7258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:52.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:52 smithi181 ceph-mon[35869]: pgmap v7259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:27:52.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:52 smithi097 ceph-mon[32231]: pgmap v7257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:52 smithi097 ceph-mon[32231]: pgmap v7258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:52 smithi097 ceph-mon[32231]: pgmap v7259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:27:53.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:27:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:54 smithi115 ceph-mon[35364]: pgmap v7260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:27:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:54 smithi181 ceph-mon[35869]: pgmap v7260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:27:55.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:54 smithi097 ceph-mon[32231]: pgmap v7260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-26T11:27:56.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12512.948934 secs 2024-07-26T11:27:56.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12512.948934 secs 2024-07-26T11:27:56.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12512.948934 secs 2024-07-26T11:27:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:56 smithi115 ceph-mon[35364]: pgmap v7261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:56 smithi181 ceph-mon[35869]: pgmap v7261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:57.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:56 smithi097 ceph-mon[32231]: pgmap v7261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:27:58 smithi115 ceph-mon[35364]: pgmap v7262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:27:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:27:58 smithi181 ceph-mon[35869]: pgmap v7262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:27:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:27:58 smithi097 ceph-mon[32231]: pgmap v7262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:28:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:00 smithi115 ceph-mon[35364]: pgmap v7263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12517.949073 secs 2024-07-26T11:28:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:00 smithi181 ceph-mon[35869]: pgmap v7263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12517.949073 secs 2024-07-26T11:28:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:00 smithi097 ceph-mon[32231]: pgmap v7263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12517.949073 secs 2024-07-26T11:28:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:02 smithi115 ceph-mon[35364]: pgmap v7264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:28:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:02 smithi181 ceph-mon[35869]: pgmap v7264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:28:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:02 smithi097 ceph-mon[32231]: pgmap v7264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:28:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:04 smithi115 ceph-mon[35364]: pgmap v7265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:04 smithi181 ceph-mon[35869]: pgmap v7265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:04 smithi097 ceph-mon[32231]: pgmap v7265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12522.949292 secs 2024-07-26T11:28:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12522.949292 secs 2024-07-26T11:28:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12522.949292 secs 2024-07-26T11:28:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.918+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T11:28:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.918+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:28:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.918+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005d50 tx=0x7ff3e0059000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.918+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005d50 tx=0x7ff3e0059000 comp rx=0 tx=0).stop 2024-07-26T11:28:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.918+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T11:28:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.918+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:28:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.918+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:28:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.918+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 0x7ff3d40b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:28:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.919+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 0x7ff3d40b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:06.921+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 0x7ff3d40b59d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40b6840 tx=0x7ff3e0071000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:28:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:06 smithi115 ceph-mon[35364]: pgmap v7266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:06 smithi181 ceph-mon[35869]: pgmap v7266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:06 smithi097 ceph-mon[32231]: pgmap v7266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:07.911+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:07.912+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:28:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:07.912+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:07.912+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:07.912+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:09.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:09.107+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:28:09.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:09.107+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:28:09.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:09.107+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:28:09.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:09.107+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:28:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:08 smithi115 ceph-mon[35364]: pgmap v7267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:08 smithi181 ceph-mon[35869]: pgmap v7267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:08 smithi097 ceph-mon[32231]: pgmap v7267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:10 smithi115 ceph-mon[35364]: pgmap v7268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12527.949457 secs 2024-07-26T11:28:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:10 smithi181 ceph-mon[35869]: pgmap v7268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12527.949457 secs 2024-07-26T11:28:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:10 smithi097 ceph-mon[32231]: pgmap v7268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12527.949457 secs 2024-07-26T11:28:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:12 smithi115 ceph-mon[35364]: pgmap v7269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:12 smithi181 ceph-mon[35869]: pgmap v7269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:12 smithi097 ceph-mon[32231]: pgmap v7269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:14 smithi115 ceph-mon[35364]: pgmap v7270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:14 smithi181 ceph-mon[35869]: pgmap v7270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:14 smithi097 ceph-mon[32231]: pgmap v7270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12532.949576 secs 2024-07-26T11:28:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12532.949576 secs 2024-07-26T11:28:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12532.949576 secs 2024-07-26T11:28:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:16 smithi115 ceph-mon[35364]: pgmap v7271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:16 smithi181 ceph-mon[35869]: pgmap v7271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:16 smithi097 ceph-mon[32231]: pgmap v7271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:17 smithi115 ceph-mon[35364]: pgmap v7272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:17 smithi181 ceph-mon[35869]: pgmap v7272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:17 smithi097 ceph-mon[32231]: pgmap v7272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:20 smithi115 ceph-mon[35364]: pgmap v7273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12537.949705 secs 2024-07-26T11:28:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:20 smithi181 ceph-mon[35869]: pgmap v7273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12537.949705 secs 2024-07-26T11:28:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:20 smithi097 ceph-mon[32231]: pgmap v7273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12537.949705 secs 2024-07-26T11:28:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:22.914+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:22.916+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:28:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:22.916+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:22.916+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:22.916+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:22 smithi115 ceph-mon[35364]: pgmap v7274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:22 smithi181 ceph-mon[35869]: pgmap v7274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:22 smithi097 ceph-mon[32231]: pgmap v7274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:24 smithi115 ceph-mon[35364]: pgmap v7275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:24 smithi181 ceph-mon[35869]: pgmap v7275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:24 smithi097 ceph-mon[32231]: pgmap v7275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12542.949847 secs 2024-07-26T11:28:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:25 smithi115 ceph-mon[35364]: pgmap v7276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12542.949847 secs 2024-07-26T11:28:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:25 smithi181 ceph-mon[35869]: pgmap v7276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12542.949847 secs 2024-07-26T11:28:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:25 smithi097 ceph-mon[32231]: pgmap v7276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:28 smithi115 ceph-mon[35364]: pgmap v7277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:28 smithi181 ceph-mon[35869]: pgmap v7277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:28 smithi097 ceph-mon[32231]: pgmap v7277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:30 smithi115 ceph-mon[35364]: pgmap v7278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12547.950023 secs 2024-07-26T11:28:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:30 smithi181 ceph-mon[35869]: pgmap v7278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12547.950023 secs 2024-07-26T11:28:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:30 smithi097 ceph-mon[32231]: pgmap v7278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12547.950023 secs 2024-07-26T11:28:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:32 smithi115 ceph-mon[35364]: pgmap v7279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:32 smithi181 ceph-mon[35869]: pgmap v7279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:32 smithi097 ceph-mon[32231]: pgmap v7279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:34 smithi115 ceph-mon[35364]: pgmap v7280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:34 smithi181 ceph-mon[35869]: pgmap v7280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:34 smithi097 ceph-mon[32231]: pgmap v7280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12552.950169 secs 2024-07-26T11:28:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12552.950169 secs 2024-07-26T11:28:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12552.950169 secs 2024-07-26T11:28:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:36 smithi115 ceph-mon[35364]: pgmap v7281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:36 smithi181 ceph-mon[35869]: pgmap v7281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:36 smithi097 ceph-mon[32231]: pgmap v7281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:37.923+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:37.925+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:28:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:37.925+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:37.925+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:37.925+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:38 smithi115 ceph-mon[35364]: pgmap v7282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:38 smithi181 ceph-mon[35869]: pgmap v7282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:38 smithi097 ceph-mon[32231]: pgmap v7282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:40 smithi115 ceph-mon[35364]: pgmap v7283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12557.950330 secs 2024-07-26T11:28:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:40 smithi181 ceph-mon[35869]: pgmap v7283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12557.950330 secs 2024-07-26T11:28:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:40 smithi097 ceph-mon[32231]: pgmap v7283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12557.950330 secs 2024-07-26T11:28:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:42 smithi115 ceph-mon[35364]: pgmap v7284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:42 smithi181 ceph-mon[35869]: pgmap v7284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:42 smithi097 ceph-mon[32231]: pgmap v7284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:44 smithi115 ceph-mon[35364]: pgmap v7285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:45.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:44 smithi181 ceph-mon[35869]: pgmap v7285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:44 smithi097 ceph-mon[32231]: pgmap v7285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12562.950495 secs 2024-07-26T11:28:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12562.950495 secs 2024-07-26T11:28:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12562.950495 secs 2024-07-26T11:28:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:46 smithi115 ceph-mon[35364]: pgmap v7286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:46 smithi181 ceph-mon[35869]: pgmap v7286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:46 smithi097 ceph-mon[32231]: pgmap v7286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:48 smithi115 ceph-mon[35364]: pgmap v7287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:48 smithi181 ceph-mon[35869]: pgmap v7287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:48 smithi097 ceph-mon[32231]: pgmap v7287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:49 smithi115 ceph-mon[35364]: pgmap v7288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:49 smithi181 ceph-mon[35869]: pgmap v7288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:49 smithi097 ceph-mon[32231]: pgmap v7288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12567.950591 secs 2024-07-26T11:28:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12567.950591 secs 2024-07-26T11:28:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12567.950591 secs 2024-07-26T11:28:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:51 smithi115 ceph-mon[35364]: pgmap v7289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:51 smithi181 ceph-mon[35869]: pgmap v7289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:51 smithi097 ceph-mon[32231]: pgmap v7289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:52.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:52.925+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:52.927+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:28:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:52.927+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:52.927+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:28:52.927+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:53 smithi097 ceph-mon[32231]: pgmap v7290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:28:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:53 smithi097 ceph-mon[32231]: pgmap v7291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:28:54.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:28:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:53 smithi115 ceph-mon[35364]: pgmap v7290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:28:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:53 smithi115 ceph-mon[35364]: pgmap v7291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:28:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:28:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:53 smithi181 ceph-mon[35869]: pgmap v7290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:28:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:53 smithi181 ceph-mon[35869]: pgmap v7291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:28:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:28:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12572.950749 secs 2024-07-26T11:28:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12572.950749 secs 2024-07-26T11:28:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12572.950749 secs 2024-07-26T11:28:56.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:56 smithi115 ceph-mon[35364]: pgmap v7292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:28:56.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:56 smithi181 ceph-mon[35869]: pgmap v7292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:28:56.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:56 smithi097 ceph-mon[32231]: pgmap v7292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:28:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:28:58 smithi115 ceph-mon[35364]: pgmap v7293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:28:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:28:58 smithi181 ceph-mon[35869]: pgmap v7293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:28:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:28:58 smithi097 ceph-mon[32231]: pgmap v7293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:29:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:00 smithi115 ceph-mon[35364]: pgmap v7294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12577.950886 secs 2024-07-26T11:29:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:00 smithi181 ceph-mon[35869]: pgmap v7294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12577.950886 secs 2024-07-26T11:29:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:00 smithi097 ceph-mon[32231]: pgmap v7294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12577.950886 secs 2024-07-26T11:29:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:02 smithi181 ceph-mon[35869]: pgmap v7295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:02 smithi097 ceph-mon[32231]: pgmap v7295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:02.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:02 smithi115 ceph-mon[35364]: pgmap v7295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:04.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:04 smithi097 ceph-mon[32231]: pgmap v7296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:29:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:04 smithi115 ceph-mon[35364]: pgmap v7296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:29:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:04 smithi181 ceph-mon[35869]: pgmap v7296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:29:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12582.951020 secs 2024-07-26T11:29:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12582.951020 secs 2024-07-26T11:29:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12582.951020 secs 2024-07-26T11:29:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:06 smithi115 ceph-mon[35364]: pgmap v7297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:06 smithi181 ceph-mon[35869]: pgmap v7297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:06 smithi097 ceph-mon[32231]: pgmap v7297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:07.934+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:07.935+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:29:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:07.935+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:07.935+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:07.935+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:08 smithi115 ceph-mon[35364]: pgmap v7298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:08 smithi181 ceph-mon[35869]: pgmap v7298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:08 smithi097 ceph-mon[32231]: pgmap v7298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:10 smithi115 ceph-mon[35364]: pgmap v7299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12587.951160 secs 2024-07-26T11:29:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:10 smithi181 ceph-mon[35869]: pgmap v7299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12587.951160 secs 2024-07-26T11:29:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:10 smithi097 ceph-mon[32231]: pgmap v7299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12587.951160 secs 2024-07-26T11:29:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:12 smithi115 ceph-mon[35364]: pgmap v7300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:12 smithi181 ceph-mon[35869]: pgmap v7300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:12.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:12 smithi097 ceph-mon[32231]: pgmap v7300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:14 smithi115 ceph-mon[35364]: pgmap v7301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:14 smithi181 ceph-mon[35869]: pgmap v7301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:14 smithi097 ceph-mon[32231]: pgmap v7301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12592.951304 secs 2024-07-26T11:29:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12592.951304 secs 2024-07-26T11:29:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12592.951304 secs 2024-07-26T11:29:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:16 smithi115 ceph-mon[35364]: pgmap v7302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:16 smithi181 ceph-mon[35869]: pgmap v7302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:16 smithi097 ceph-mon[32231]: pgmap v7302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:18 smithi115 ceph-mon[35364]: pgmap v7303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:18 smithi181 ceph-mon[35869]: pgmap v7303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:18 smithi097 ceph-mon[32231]: pgmap v7303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:20 smithi115 ceph-mon[35364]: pgmap v7304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12597.951443 secs 2024-07-26T11:29:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:20 smithi181 ceph-mon[35869]: pgmap v7304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12597.951443 secs 2024-07-26T11:29:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:20 smithi097 ceph-mon[32231]: pgmap v7304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12597.951443 secs 2024-07-26T11:29:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:22 smithi115 ceph-mon[35364]: pgmap v7305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:22 smithi181 ceph-mon[35869]: pgmap v7305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:22 smithi097 ceph-mon[32231]: pgmap v7305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:22.936+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:22.938+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:29:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:22.938+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:22.938+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:22.938+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:24.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:24 smithi115 ceph-mon[35364]: pgmap v7306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:24 smithi181 ceph-mon[35869]: pgmap v7306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:24 smithi097 ceph-mon[32231]: pgmap v7306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12602.951587 secs 2024-07-26T11:29:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12602.951587 secs 2024-07-26T11:29:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12602.951587 secs 2024-07-26T11:29:26.627 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:26 smithi115 ceph-mon[35364]: pgmap v7307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:26.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:26 smithi181 ceph-mon[35869]: pgmap v7307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:26.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:26 smithi097 ceph-mon[32231]: pgmap v7307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:28 smithi115 ceph-mon[35364]: pgmap v7308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:28 smithi181 ceph-mon[35869]: pgmap v7308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:28.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:28 smithi097 ceph-mon[32231]: pgmap v7308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:30 smithi115 ceph-mon[35364]: pgmap v7309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12607.951722 secs 2024-07-26T11:29:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:30 smithi181 ceph-mon[35869]: pgmap v7309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12607.951722 secs 2024-07-26T11:29:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:30 smithi097 ceph-mon[32231]: pgmap v7309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12607.951722 secs 2024-07-26T11:29:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:32 smithi115 ceph-mon[35364]: pgmap v7310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:32 smithi181 ceph-mon[35869]: pgmap v7310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:32 smithi097 ceph-mon[32231]: pgmap v7310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:34 smithi115 ceph-mon[35364]: pgmap v7311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:34 smithi181 ceph-mon[35869]: pgmap v7311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:34.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:34 smithi097 ceph-mon[32231]: pgmap v7311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12612.951809 secs 2024-07-26T11:29:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12612.951809 secs 2024-07-26T11:29:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12612.951809 secs 2024-07-26T11:29:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:36 smithi115 ceph-mon[35364]: pgmap v7312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:36 smithi181 ceph-mon[35869]: pgmap v7312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:36 smithi097 ceph-mon[32231]: pgmap v7312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:37.943+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:37.945+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:29:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:37.945+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:37.945+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:37.945+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:38 smithi115 ceph-mon[35364]: pgmap v7313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:38 smithi181 ceph-mon[35869]: pgmap v7313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:38 smithi097 ceph-mon[32231]: pgmap v7313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:40 smithi115 ceph-mon[35364]: pgmap v7314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12617.951948 secs 2024-07-26T11:29:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:40 smithi181 ceph-mon[35869]: pgmap v7314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12617.951948 secs 2024-07-26T11:29:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:40 smithi097 ceph-mon[32231]: pgmap v7314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12617.951948 secs 2024-07-26T11:29:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:42 smithi115 ceph-mon[35364]: pgmap v7315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:42 smithi181 ceph-mon[35869]: pgmap v7315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:42.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:42 smithi097 ceph-mon[32231]: pgmap v7315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:44 smithi115 ceph-mon[35364]: pgmap v7316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:44 smithi181 ceph-mon[35869]: pgmap v7316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:44 smithi097 ceph-mon[32231]: pgmap v7316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:45.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12622.952093 secs 2024-07-26T11:29:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12622.952093 secs 2024-07-26T11:29:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12622.952093 secs 2024-07-26T11:29:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:46 smithi115 ceph-mon[35364]: pgmap v7317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:46 smithi181 ceph-mon[35869]: pgmap v7317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:46 smithi097 ceph-mon[32231]: pgmap v7317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:48.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:48 smithi115 ceph-mon[35364]: pgmap v7318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:48 smithi181 ceph-mon[35869]: pgmap v7318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:48 smithi097 ceph-mon[32231]: pgmap v7318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:50 smithi115 ceph-mon[35364]: pgmap v7319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12627.952289 secs 2024-07-26T11:29:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:50 smithi181 ceph-mon[35869]: pgmap v7319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12627.952289 secs 2024-07-26T11:29:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:50 smithi097 ceph-mon[32231]: pgmap v7319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:29:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12627.952289 secs 2024-07-26T11:29:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:52 smithi115 ceph-mon[35364]: pgmap v7320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:52 smithi181 ceph-mon[35869]: pgmap v7320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:52 smithi097 ceph-mon[32231]: pgmap v7320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:52.947+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:52.948+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:29:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:52.948+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:52.948+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:29:52.948+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:54 smithi115 ceph-mon[35364]: pgmap v7321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:54.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:54 smithi181 ceph-mon[35869]: pgmap v7321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:54 smithi097 ceph-mon[32231]: pgmap v7321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:54.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:55 smithi181 ceph-mon[35869]: pgmap v7322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:29:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:55 smithi181 ceph-mon[35869]: pgmap v7323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:29:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12632.952390 secs 2024-07-26T11:29:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:55 smithi097 ceph-mon[32231]: pgmap v7322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:29:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:55 smithi097 ceph-mon[32231]: pgmap v7323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:29:55.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12632.952390 secs 2024-07-26T11:29:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:55 smithi115 ceph-mon[35364]: pgmap v7322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:29:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:55 smithi115 ceph-mon[35364]: pgmap v7323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:29:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12632.952390 secs 2024-07-26T11:29:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:57 smithi115 ceph-mon[35364]: pgmap v7324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:57 smithi181 ceph-mon[35869]: pgmap v7324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:57 smithi097 ceph-mon[32231]: pgmap v7324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:29:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:29:59 smithi115 ceph-mon[35364]: pgmap v7325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:29:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:29:59 smithi181 ceph-mon[35869]: pgmap v7325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:30:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:29:59 smithi097 ceph-mon[32231]: pgmap v7325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:30:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12637.952550 secs 2024-07-26T11:30:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12637.952550 secs 2024-07-26T11:30:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12637.952550 secs 2024-07-26T11:30:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:01 smithi115 ceph-mon[35364]: pgmap v7326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:30:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:01 smithi181 ceph-mon[35869]: pgmap v7326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:30:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:01 smithi097 ceph-mon[32231]: pgmap v7326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:30:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:03 smithi115 ceph-mon[35364]: pgmap v7327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:30:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:03 smithi181 ceph-mon[35869]: pgmap v7327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:30:04.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:03 smithi097 ceph-mon[32231]: pgmap v7327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:30:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:05 smithi115 ceph-mon[35364]: pgmap v7328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12642.952694 secs 2024-07-26T11:30:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:05 smithi181 ceph-mon[35869]: pgmap v7328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12642.952694 secs 2024-07-26T11:30:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:05 smithi097 ceph-mon[32231]: pgmap v7328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:06.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12642.952694 secs 2024-07-26T11:30:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:07.953+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:30:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:07.954+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:30:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:07.954+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:30:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:07.954+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:30:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:07.955+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:07 smithi115 ceph-mon[35364]: pgmap v7329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:07 smithi181 ceph-mon[35869]: pgmap v7329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:07 smithi097 ceph-mon[32231]: pgmap v7329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:09 smithi115 ceph-mon[35364]: pgmap v7330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:09 smithi181 ceph-mon[35869]: pgmap v7330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:09 smithi097 ceph-mon[32231]: pgmap v7330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12647.952837 secs 2024-07-26T11:30:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12647.952837 secs 2024-07-26T11:30:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12647.952837 secs 2024-07-26T11:30:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:11 smithi115 ceph-mon[35364]: pgmap v7331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:11 smithi181 ceph-mon[35869]: pgmap v7331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:11 smithi097 ceph-mon[32231]: pgmap v7331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:13 smithi115 ceph-mon[35364]: pgmap v7332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:13 smithi181 ceph-mon[35869]: pgmap v7332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:14.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:13 smithi097 ceph-mon[32231]: pgmap v7332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:15 smithi115 ceph-mon[35364]: pgmap v7333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:30:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12652.952983 secs 2024-07-26T11:30:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:15 smithi181 ceph-mon[35869]: pgmap v7333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:30:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12652.952983 secs 2024-07-26T11:30:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:15 smithi097 ceph-mon[32231]: pgmap v7333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:30:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12652.952983 secs 2024-07-26T11:30:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:17 smithi181 ceph-mon[35869]: pgmap v7334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:30:18.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:17 smithi097 ceph-mon[32231]: pgmap v7334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:30:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:17 smithi115 ceph-mon[35364]: pgmap v7334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:30:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:19 smithi097 ceph-mon[32231]: pgmap v7335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:19 smithi115 ceph-mon[35364]: pgmap v7335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:19 smithi181 ceph-mon[35869]: pgmap v7335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12657.953125 secs 2024-07-26T11:30:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12657.953125 secs 2024-07-26T11:30:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12657.953125 secs 2024-07-26T11:30:22.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:21 smithi097 ceph-mon[32231]: pgmap v7336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:21 smithi115 ceph-mon[35364]: pgmap v7336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:21 smithi181 ceph-mon[35869]: pgmap v7336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:22.957+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:30:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:22.959+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:30:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:22.959+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:30:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:22.959+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:30:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:22.959+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:23 smithi115 ceph-mon[35364]: pgmap v7337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:23 smithi181 ceph-mon[35869]: pgmap v7337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:23 smithi097 ceph-mon[32231]: pgmap v7337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:25 smithi115 ceph-mon[35364]: pgmap v7338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12662.953272 secs 2024-07-26T11:30:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:25 smithi181 ceph-mon[35869]: pgmap v7338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12662.953272 secs 2024-07-26T11:30:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:25 smithi097 ceph-mon[32231]: pgmap v7338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12662.953272 secs 2024-07-26T11:30:28.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:27 smithi115 ceph-mon[35364]: pgmap v7339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:28.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:27 smithi181 ceph-mon[35869]: pgmap v7339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:28.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:27 smithi097 ceph-mon[32231]: pgmap v7339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:29 smithi181 ceph-mon[35869]: pgmap v7340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:29 smithi115 ceph-mon[35364]: pgmap v7340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:29 smithi097 ceph-mon[32231]: pgmap v7340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12667.953378 secs 2024-07-26T11:30:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12667.953378 secs 2024-07-26T11:30:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12667.953378 secs 2024-07-26T11:30:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:31 smithi115 ceph-mon[35364]: pgmap v7341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:31 smithi181 ceph-mon[35869]: pgmap v7341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:31 smithi097 ceph-mon[32231]: pgmap v7341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:33 smithi115 ceph-mon[35364]: pgmap v7342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:30:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:33 smithi181 ceph-mon[35869]: pgmap v7342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:30:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:33 smithi097 ceph-mon[32231]: pgmap v7342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:30:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:35 smithi115 ceph-mon[35364]: pgmap v7343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12672.953527 secs 2024-07-26T11:30:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:35 smithi181 ceph-mon[35869]: pgmap v7343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12672.953527 secs 2024-07-26T11:30:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:35 smithi097 ceph-mon[32231]: pgmap v7343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12672.953527 secs 2024-07-26T11:30:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:37.966+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:30:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:37.967+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:30:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:37.967+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:30:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:37.967+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:30:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:37.967+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:37 smithi115 ceph-mon[35364]: pgmap v7344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:37 smithi181 ceph-mon[35869]: pgmap v7344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:37 smithi097 ceph-mon[32231]: pgmap v7344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:39 smithi115 ceph-mon[35364]: pgmap v7345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:39 smithi181 ceph-mon[35869]: pgmap v7345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:39 smithi097 ceph-mon[32231]: pgmap v7345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12677.953633 secs 2024-07-26T11:30:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:40 smithi115 ceph-mon[35364]: pgmap v7346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12677.953633 secs 2024-07-26T11:30:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:40 smithi181 ceph-mon[35869]: pgmap v7346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12677.953633 secs 2024-07-26T11:30:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:40 smithi097 ceph-mon[32231]: pgmap v7346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:43 smithi115 ceph-mon[35364]: pgmap v7347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:43 smithi181 ceph-mon[35869]: pgmap v7347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:43 smithi097 ceph-mon[32231]: pgmap v7347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:45 smithi115 ceph-mon[35364]: pgmap v7348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12682.953773 secs 2024-07-26T11:30:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:45 smithi181 ceph-mon[35869]: pgmap v7348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12682.953773 secs 2024-07-26T11:30:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:45 smithi097 ceph-mon[32231]: pgmap v7348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12682.953773 secs 2024-07-26T11:30:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:47 smithi115 ceph-mon[35364]: pgmap v7349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:47 smithi181 ceph-mon[35869]: pgmap v7349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:48.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:47 smithi097 ceph-mon[32231]: pgmap v7349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:49 smithi115 ceph-mon[35364]: pgmap v7350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:49 smithi181 ceph-mon[35869]: pgmap v7350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:49 smithi097 ceph-mon[32231]: pgmap v7350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12687.953933 secs 2024-07-26T11:30:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12687.953933 secs 2024-07-26T11:30:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12687.953933 secs 2024-07-26T11:30:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:51 smithi115 ceph-mon[35364]: pgmap v7351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:51 smithi181 ceph-mon[35869]: pgmap v7351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:51 smithi097 ceph-mon[32231]: pgmap v7351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:52.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:52.969+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:30:52.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:52.970+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:30:52.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:52.970+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:30:52.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:52.970+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:30:52.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:30:52.970+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:53 smithi115 ceph-mon[35364]: pgmap v7352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:53 smithi181 ceph-mon[35869]: pgmap v7352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:53 smithi097 ceph-mon[32231]: pgmap v7352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:55 smithi115 ceph-mon[35364]: pgmap v7353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:30:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12692.954068 secs 2024-07-26T11:30:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:55 smithi181 ceph-mon[35869]: pgmap v7353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:30:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12692.954068 secs 2024-07-26T11:30:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:55 smithi097 ceph-mon[32231]: pgmap v7353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:30:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12692.954068 secs 2024-07-26T11:30:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:57.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:30:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:30:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:30:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:57 smithi097 ceph-mon[32231]: pgmap v7354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:57 smithi097 ceph-mon[32231]: pgmap v7355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:30:58.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:57 smithi115 ceph-mon[35364]: pgmap v7354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:58.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:57 smithi115 ceph-mon[35364]: pgmap v7355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:30:58.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:57 smithi181 ceph-mon[35869]: pgmap v7354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:58.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:57 smithi181 ceph-mon[35869]: pgmap v7355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:31:00.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:30:59 smithi097 ceph-mon[32231]: pgmap v7356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:30:59 smithi115 ceph-mon[35364]: pgmap v7356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:30:59 smithi181 ceph-mon[35869]: pgmap v7356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12697.954223 secs 2024-07-26T11:31:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12697.954223 secs 2024-07-26T11:31:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12697.954223 secs 2024-07-26T11:31:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:01 smithi115 ceph-mon[35364]: pgmap v7357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:01 smithi181 ceph-mon[35869]: pgmap v7357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:01 smithi097 ceph-mon[32231]: pgmap v7357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:03 smithi115 ceph-mon[35364]: pgmap v7358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:03 smithi181 ceph-mon[35869]: pgmap v7358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:03 smithi097 ceph-mon[32231]: pgmap v7358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:05 smithi115 ceph-mon[35364]: pgmap v7359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12702.954352 secs 2024-07-26T11:31:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:05 smithi181 ceph-mon[35869]: pgmap v7359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12702.954352 secs 2024-07-26T11:31:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:05 smithi097 ceph-mon[32231]: pgmap v7359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12702.954352 secs 2024-07-26T11:31:07.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:07.977+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:31:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:07.979+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:31:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:07.979+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:31:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:07.979+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:31:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:07.979+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:07 smithi115 ceph-mon[35364]: pgmap v7360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:07 smithi181 ceph-mon[35869]: pgmap v7360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:08.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:07 smithi097 ceph-mon[32231]: pgmap v7360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:09.140+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:31:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:09.140+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:31:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:09.140+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:31:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:09.141+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:31:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:09 smithi115 ceph-mon[35364]: pgmap v7361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:09 smithi181 ceph-mon[35869]: pgmap v7361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:10.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:09 smithi097 ceph-mon[32231]: pgmap v7361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12707.954447 secs 2024-07-26T11:31:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12707.954447 secs 2024-07-26T11:31:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12707.954447 secs 2024-07-26T11:31:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:11 smithi115 ceph-mon[35364]: pgmap v7362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:11 smithi181 ceph-mon[35869]: pgmap v7362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:12.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:11 smithi097 ceph-mon[32231]: pgmap v7362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:13 smithi115 ceph-mon[35364]: pgmap v7363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:13 smithi181 ceph-mon[35869]: pgmap v7363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:13 smithi097 ceph-mon[32231]: pgmap v7363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:15 smithi115 ceph-mon[35364]: pgmap v7364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:31:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12712.954590 secs 2024-07-26T11:31:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:15 smithi181 ceph-mon[35869]: pgmap v7364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:31:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12712.954590 secs 2024-07-26T11:31:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:15 smithi097 ceph-mon[32231]: pgmap v7364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:31:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12712.954590 secs 2024-07-26T11:31:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:17 smithi115 ceph-mon[35364]: pgmap v7365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:17 smithi181 ceph-mon[35869]: pgmap v7365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:17 smithi097 ceph-mon[32231]: pgmap v7365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:19 smithi115 ceph-mon[35364]: pgmap v7366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:31:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:19 smithi181 ceph-mon[35869]: pgmap v7366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:31:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:19 smithi097 ceph-mon[32231]: pgmap v7366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:31:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12717.954742 secs 2024-07-26T11:31:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12717.954742 secs 2024-07-26T11:31:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12717.954742 secs 2024-07-26T11:31:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:21 smithi115 ceph-mon[35364]: pgmap v7367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:31:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:21 smithi181 ceph-mon[35869]: pgmap v7367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:31:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:21 smithi097 ceph-mon[32231]: pgmap v7367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:31:22.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:22.981+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:31:22.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:22.983+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:31:22.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:22.983+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:31:22.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:22.983+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:31:22.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:22.983+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:23 smithi115 ceph-mon[35364]: pgmap v7368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:23 smithi181 ceph-mon[35869]: pgmap v7368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:23 smithi097 ceph-mon[32231]: pgmap v7368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:25 smithi115 ceph-mon[35364]: pgmap v7369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:31:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12722.954892 secs 2024-07-26T11:31:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:25 smithi181 ceph-mon[35869]: pgmap v7369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:31:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12722.954892 secs 2024-07-26T11:31:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:25 smithi097 ceph-mon[32231]: pgmap v7369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:31:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12722.954892 secs 2024-07-26T11:31:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:27 smithi115 ceph-mon[35364]: pgmap v7370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:27 smithi181 ceph-mon[35869]: pgmap v7370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:27 smithi097 ceph-mon[32231]: pgmap v7370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:29 smithi115 ceph-mon[35364]: pgmap v7371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:29 smithi181 ceph-mon[35869]: pgmap v7371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:29 smithi097 ceph-mon[32231]: pgmap v7371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12727.955027 secs 2024-07-26T11:31:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12727.955027 secs 2024-07-26T11:31:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12727.955027 secs 2024-07-26T11:31:32.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:31 smithi115 ceph-mon[35364]: pgmap v7372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:31 smithi181 ceph-mon[35869]: pgmap v7372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:31 smithi097 ceph-mon[32231]: pgmap v7372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:33 smithi115 ceph-mon[35364]: pgmap v7373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:33 smithi181 ceph-mon[35869]: pgmap v7373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:34.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:33 smithi097 ceph-mon[32231]: pgmap v7373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:35 smithi115 ceph-mon[35364]: pgmap v7374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12732.955163 secs 2024-07-26T11:31:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:35 smithi181 ceph-mon[35869]: pgmap v7374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12732.955163 secs 2024-07-26T11:31:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:35 smithi097 ceph-mon[32231]: pgmap v7374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12732.955163 secs 2024-07-26T11:31:37.991 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:37.989+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:31:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:37.991+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:31:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:37.991+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:31:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:37.991+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:31:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:37.991+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:37 smithi115 ceph-mon[35364]: pgmap v7375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:37 smithi181 ceph-mon[35869]: pgmap v7375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:37 smithi097 ceph-mon[32231]: pgmap v7375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:39 smithi115 ceph-mon[35364]: pgmap v7376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:39 smithi181 ceph-mon[35869]: pgmap v7376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:39 smithi097 ceph-mon[32231]: pgmap v7376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12737.955311 secs 2024-07-26T11:31:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:40 smithi097 ceph-mon[32231]: pgmap v7377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12737.955311 secs 2024-07-26T11:31:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:40 smithi115 ceph-mon[35364]: pgmap v7377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12737.955311 secs 2024-07-26T11:31:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:40 smithi181 ceph-mon[35869]: pgmap v7377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:43 smithi115 ceph-mon[35364]: pgmap v7378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:43 smithi181 ceph-mon[35869]: pgmap v7378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:43 smithi097 ceph-mon[32231]: pgmap v7378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:45 smithi115 ceph-mon[35364]: pgmap v7379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12742.955453 secs 2024-07-26T11:31:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:45 smithi181 ceph-mon[35869]: pgmap v7379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12742.955453 secs 2024-07-26T11:31:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:45 smithi097 ceph-mon[32231]: pgmap v7379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12742.955453 secs 2024-07-26T11:31:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:47 smithi115 ceph-mon[35364]: pgmap v7380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:47 smithi181 ceph-mon[35869]: pgmap v7380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:47.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:47 smithi097 ceph-mon[32231]: pgmap v7380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:49 smithi115 ceph-mon[35364]: pgmap v7381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:49 smithi181 ceph-mon[35869]: pgmap v7381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:49 smithi097 ceph-mon[32231]: pgmap v7381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12747.955597 secs 2024-07-26T11:31:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12747.955597 secs 2024-07-26T11:31:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12747.955597 secs 2024-07-26T11:31:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:51 smithi115 ceph-mon[35364]: pgmap v7382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:51 smithi181 ceph-mon[35869]: pgmap v7382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:51 smithi097 ceph-mon[32231]: pgmap v7382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:52.993+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:31:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:52.994+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:31:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:52.994+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:31:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:52.994+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:31:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:31:52.994+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:53 smithi115 ceph-mon[35364]: pgmap v7383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:53 smithi181 ceph-mon[35869]: pgmap v7383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:53 smithi097 ceph-mon[32231]: pgmap v7383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:55 smithi115 ceph-mon[35364]: pgmap v7384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12752.955749 secs 2024-07-26T11:31:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:55 smithi181 ceph-mon[35869]: pgmap v7384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12752.955749 secs 2024-07-26T11:31:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:55 smithi097 ceph-mon[32231]: pgmap v7384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:31:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12752.955749 secs 2024-07-26T11:31:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:57 smithi115 ceph-mon[35364]: pgmap v7385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:57 smithi181 ceph-mon[35869]: pgmap v7385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:57 smithi097 ceph-mon[32231]: pgmap v7385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:31:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:58 smithi115 ceph-mon[35364]: pgmap v7386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:31:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:58 smithi115 ceph-mon[35364]: pgmap v7387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:31:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:31:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:58 smithi181 ceph-mon[35869]: pgmap v7386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:31:58.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:58 smithi181 ceph-mon[35869]: pgmap v7387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:58.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:31:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:31:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:58 smithi097 ceph-mon[32231]: pgmap v7386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:31:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:58 smithi097 ceph-mon[32231]: pgmap v7387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:31:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:32:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:00 smithi115 ceph-mon[35364]: pgmap v7388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12757.955896 secs 2024-07-26T11:32:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:00 smithi181 ceph-mon[35869]: pgmap v7388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12757.955896 secs 2024-07-26T11:32:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:00 smithi097 ceph-mon[32231]: pgmap v7388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12757.955896 secs 2024-07-26T11:32:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:02 smithi115 ceph-mon[35364]: pgmap v7389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:32:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:02 smithi181 ceph-mon[35869]: pgmap v7389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:32:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:02 smithi097 ceph-mon[32231]: pgmap v7389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:32:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:04 smithi115 ceph-mon[35364]: pgmap v7390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:32:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:04 smithi181 ceph-mon[35869]: pgmap v7390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:32:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:04 smithi097 ceph-mon[32231]: pgmap v7390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:32:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12762.956039 secs 2024-07-26T11:32:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12762.956039 secs 2024-07-26T11:32:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12762.956039 secs 2024-07-26T11:32:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:06 smithi115 ceph-mon[35364]: pgmap v7391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:32:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:06 smithi181 ceph-mon[35869]: pgmap v7391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:32:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:06 smithi097 ceph-mon[32231]: pgmap v7391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:32:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:08.001+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:32:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:08.002+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:32:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:08.002+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:32:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:08.002+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:32:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:08.002+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:08.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:08 smithi115 ceph-mon[35364]: pgmap v7392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:32:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:08 smithi181 ceph-mon[35869]: pgmap v7392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:32:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:08 smithi097 ceph-mon[32231]: pgmap v7392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:32:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:10 smithi115 ceph-mon[35364]: pgmap v7393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12767.956182 secs 2024-07-26T11:32:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:10 smithi181 ceph-mon[35869]: pgmap v7393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12767.956182 secs 2024-07-26T11:32:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:10 smithi097 ceph-mon[32231]: pgmap v7393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12767.956182 secs 2024-07-26T11:32:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:12 smithi115 ceph-mon[35364]: pgmap v7394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:12 smithi181 ceph-mon[35869]: pgmap v7394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:12 smithi097 ceph-mon[32231]: pgmap v7394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:14 smithi115 ceph-mon[35364]: pgmap v7395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:14 smithi181 ceph-mon[35869]: pgmap v7395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:14 smithi097 ceph-mon[32231]: pgmap v7395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12772.956325 secs 2024-07-26T11:32:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12772.956325 secs 2024-07-26T11:32:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12772.956325 secs 2024-07-26T11:32:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:16 smithi115 ceph-mon[35364]: pgmap v7396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:16 smithi181 ceph-mon[35869]: pgmap v7396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:16 smithi097 ceph-mon[32231]: pgmap v7396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:18 smithi115 ceph-mon[35364]: pgmap v7397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:18 smithi181 ceph-mon[35869]: pgmap v7397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:18 smithi097 ceph-mon[32231]: pgmap v7397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:20 smithi115 ceph-mon[35364]: pgmap v7398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12777.956475 secs 2024-07-26T11:32:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:20 smithi181 ceph-mon[35869]: pgmap v7398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12777.956475 secs 2024-07-26T11:32:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:20 smithi097 ceph-mon[32231]: pgmap v7398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12777.956475 secs 2024-07-26T11:32:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:22 smithi115 ceph-mon[35364]: pgmap v7399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:22 smithi181 ceph-mon[35869]: pgmap v7399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:22 smithi097 ceph-mon[32231]: pgmap v7399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:23.004+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:32:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:23.005+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:32:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:23.005+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:32:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:23.005+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:32:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:23.005+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:24 smithi115 ceph-mon[35364]: pgmap v7400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:24 smithi181 ceph-mon[35869]: pgmap v7400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:24 smithi097 ceph-mon[32231]: pgmap v7400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12782.956616 secs 2024-07-26T11:32:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12782.956616 secs 2024-07-26T11:32:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12782.956616 secs 2024-07-26T11:32:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:26 smithi115 ceph-mon[35364]: pgmap v7401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:26 smithi181 ceph-mon[35869]: pgmap v7401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:26 smithi097 ceph-mon[32231]: pgmap v7401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:28 smithi115 ceph-mon[35364]: pgmap v7402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:28 smithi181 ceph-mon[35869]: pgmap v7402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:28 smithi097 ceph-mon[32231]: pgmap v7402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:30 smithi115 ceph-mon[35364]: pgmap v7403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12787.956755 secs 2024-07-26T11:32:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:30 smithi181 ceph-mon[35869]: pgmap v7403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12787.956755 secs 2024-07-26T11:32:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:30 smithi097 ceph-mon[32231]: pgmap v7403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12787.956755 secs 2024-07-26T11:32:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:32 smithi115 ceph-mon[35364]: pgmap v7404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:32 smithi181 ceph-mon[35869]: pgmap v7404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:32 smithi097 ceph-mon[32231]: pgmap v7404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:34 smithi115 ceph-mon[35364]: pgmap v7405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:34 smithi181 ceph-mon[35869]: pgmap v7405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:34 smithi097 ceph-mon[32231]: pgmap v7405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12792.956860 secs 2024-07-26T11:32:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12792.956860 secs 2024-07-26T11:32:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12792.956860 secs 2024-07-26T11:32:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:36 smithi115 ceph-mon[35364]: pgmap v7406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:36 smithi181 ceph-mon[35869]: pgmap v7406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:36 smithi097 ceph-mon[32231]: pgmap v7406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:38.011+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:32:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:38.013+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:32:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:38.013+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:32:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:38.013+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:32:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:38.013+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:38.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:38 smithi115 ceph-mon[35364]: pgmap v7407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:38 smithi181 ceph-mon[35869]: pgmap v7407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:38 smithi097 ceph-mon[32231]: pgmap v7407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:39.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:39.156+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:32:39.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:39.156+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:32:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:39.157+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:32:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:39.157+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:32:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:40 smithi181 ceph-mon[35869]: pgmap v7408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12797.957007 secs 2024-07-26T11:32:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:40 smithi097 ceph-mon[32231]: pgmap v7408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12797.957007 secs 2024-07-26T11:32:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:40 smithi115 ceph-mon[35364]: pgmap v7408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12797.957007 secs 2024-07-26T11:32:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:42 smithi097 ceph-mon[32231]: pgmap v7409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:42 smithi115 ceph-mon[35364]: pgmap v7409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:42 smithi181 ceph-mon[35869]: pgmap v7409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:44 smithi097 ceph-mon[32231]: pgmap v7410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:44 smithi115 ceph-mon[35364]: pgmap v7410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:44 smithi181 ceph-mon[35869]: pgmap v7410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12802.957156 secs 2024-07-26T11:32:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12802.957156 secs 2024-07-26T11:32:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12802.957156 secs 2024-07-26T11:32:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:46 smithi115 ceph-mon[35364]: pgmap v7411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:46 smithi181 ceph-mon[35869]: pgmap v7411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:46 smithi097 ceph-mon[32231]: pgmap v7411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:48 smithi115 ceph-mon[35364]: pgmap v7412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:48 smithi181 ceph-mon[35869]: pgmap v7412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:48 smithi097 ceph-mon[32231]: pgmap v7412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:50 smithi115 ceph-mon[35364]: pgmap v7413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12807.957347 secs 2024-07-26T11:32:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:50 smithi181 ceph-mon[35869]: pgmap v7413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12807.957347 secs 2024-07-26T11:32:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:50 smithi097 ceph-mon[32231]: pgmap v7413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12807.957347 secs 2024-07-26T11:32:53.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:53.015+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:32:53.018 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:53.017+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:32:53.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:53.017+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:32:53.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:53.017+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:32:53.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:32:53.017+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:52 smithi115 ceph-mon[35364]: pgmap v7414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:52 smithi181 ceph-mon[35869]: pgmap v7414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:52 smithi097 ceph-mon[32231]: pgmap v7414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:32:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:54 smithi115 ceph-mon[35364]: pgmap v7415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:54 smithi181 ceph-mon[35869]: pgmap v7415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:54 smithi097 ceph-mon[32231]: pgmap v7415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12812.957489 secs 2024-07-26T11:32:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12812.957489 secs 2024-07-26T11:32:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12812.957489 secs 2024-07-26T11:32:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:56 smithi115 ceph-mon[35364]: pgmap v7416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:56 smithi181 ceph-mon[35869]: pgmap v7416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:56 smithi097 ceph-mon[32231]: pgmap v7416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:58 smithi115 ceph-mon[35364]: pgmap v7417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:58 smithi181 ceph-mon[35869]: pgmap v7417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:58 smithi097 ceph-mon[32231]: pgmap v7417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:59 smithi115 ceph-mon[35364]: pgmap v7418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:33:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:59 smithi115 ceph-mon[35364]: pgmap v7419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:33:00.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:32:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:33:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:59 smithi181 ceph-mon[35869]: pgmap v7418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:33:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:59 smithi181 ceph-mon[35869]: pgmap v7419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:33:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:32:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:33:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:59 smithi097 ceph-mon[32231]: pgmap v7418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:33:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:59 smithi097 ceph-mon[32231]: pgmap v7419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:33:00.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:32:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:33:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12817.957638 secs 2024-07-26T11:33:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12817.957638 secs 2024-07-26T11:33:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12817.957638 secs 2024-07-26T11:33:02.154 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:01 smithi097 ceph-mon[32231]: pgmap v7420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:33:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:01 smithi115 ceph-mon[35364]: pgmap v7420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:33:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:01 smithi181 ceph-mon[35869]: pgmap v7420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:33:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:03 smithi115 ceph-mon[35364]: pgmap v7421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:03 smithi181 ceph-mon[35869]: pgmap v7421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:03 smithi097 ceph-mon[32231]: pgmap v7421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:05 smithi115 ceph-mon[35364]: pgmap v7422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:33:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:05 smithi181 ceph-mon[35869]: pgmap v7422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:33:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:05 smithi097 ceph-mon[32231]: pgmap v7422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:33:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12822.957770 secs 2024-07-26T11:33:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12822.957770 secs 2024-07-26T11:33:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12822.957770 secs 2024-07-26T11:33:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:07 smithi115 ceph-mon[35364]: pgmap v7423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:33:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:07 smithi181 ceph-mon[35869]: pgmap v7423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:33:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:07 smithi097 ceph-mon[32231]: pgmap v7423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:33:08.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:08.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:33:08.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:08.020+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:33:08.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:08.020+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:33:08.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:08.020+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:33:08.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:08.020+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:09 smithi115 ceph-mon[35364]: pgmap v7424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:33:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:09 smithi181 ceph-mon[35869]: pgmap v7424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:33:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:09 smithi097 ceph-mon[32231]: pgmap v7424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:33:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12827.957912 secs 2024-07-26T11:33:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12827.957912 secs 2024-07-26T11:33:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12827.957912 secs 2024-07-26T11:33:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:11 smithi115 ceph-mon[35364]: pgmap v7425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:11 smithi181 ceph-mon[35869]: pgmap v7425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:11 smithi097 ceph-mon[32231]: pgmap v7425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:13 smithi115 ceph-mon[35364]: pgmap v7426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:13 smithi181 ceph-mon[35869]: pgmap v7426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:13 smithi097 ceph-mon[32231]: pgmap v7426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:15 smithi115 ceph-mon[35364]: pgmap v7427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:15 smithi181 ceph-mon[35869]: pgmap v7427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:15 smithi097 ceph-mon[32231]: pgmap v7427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12832.958055 secs 2024-07-26T11:33:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12832.958055 secs 2024-07-26T11:33:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12832.958055 secs 2024-07-26T11:33:17.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:17 smithi115 ceph-mon[35364]: pgmap v7428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:17 smithi181 ceph-mon[35869]: pgmap v7428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:17 smithi097 ceph-mon[32231]: pgmap v7428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:19 smithi115 ceph-mon[35364]: pgmap v7429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:19 smithi181 ceph-mon[35869]: pgmap v7429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:19.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:19 smithi097 ceph-mon[32231]: pgmap v7429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12837.958153 secs 2024-07-26T11:33:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12837.958153 secs 2024-07-26T11:33:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12837.958153 secs 2024-07-26T11:33:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:21 smithi115 ceph-mon[35364]: pgmap v7430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:21 smithi181 ceph-mon[35869]: pgmap v7430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:21 smithi097 ceph-mon[32231]: pgmap v7430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:23.022+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:33:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:23.023+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:33:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:23.023+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:33:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:23.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:33:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:23.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:23 smithi115 ceph-mon[35364]: pgmap v7431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:23 smithi181 ceph-mon[35869]: pgmap v7431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:23 smithi097 ceph-mon[32231]: pgmap v7431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:25 smithi115 ceph-mon[35364]: pgmap v7432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:25 smithi181 ceph-mon[35869]: pgmap v7432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:25 smithi097 ceph-mon[32231]: pgmap v7432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12842.958315 secs 2024-07-26T11:33:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12842.958315 secs 2024-07-26T11:33:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12842.958315 secs 2024-07-26T11:33:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:27 smithi115 ceph-mon[35364]: pgmap v7433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:27 smithi181 ceph-mon[35869]: pgmap v7433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:27 smithi097 ceph-mon[32231]: pgmap v7433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:29.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:29 smithi115 ceph-mon[35364]: pgmap v7434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:29.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:29 smithi181 ceph-mon[35869]: pgmap v7434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:29.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:29 smithi097 ceph-mon[32231]: pgmap v7434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12847.958458 secs 2024-07-26T11:33:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12847.958458 secs 2024-07-26T11:33:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12847.958458 secs 2024-07-26T11:33:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:31 smithi115 ceph-mon[35364]: pgmap v7435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:31 smithi181 ceph-mon[35869]: pgmap v7435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:31 smithi097 ceph-mon[32231]: pgmap v7435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:33.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:33 smithi097 ceph-mon[32231]: pgmap v7436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:33.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:33 smithi115 ceph-mon[35364]: pgmap v7436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:33 smithi181 ceph-mon[35869]: pgmap v7436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:35 smithi115 ceph-mon[35364]: pgmap v7437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:35 smithi181 ceph-mon[35869]: pgmap v7437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:35.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:35 smithi097 ceph-mon[32231]: pgmap v7437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12852.958569 secs 2024-07-26T11:33:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12852.958569 secs 2024-07-26T11:33:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12852.958569 secs 2024-07-26T11:33:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:37 smithi115 ceph-mon[35364]: pgmap v7438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:37 smithi181 ceph-mon[35869]: pgmap v7438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:37 smithi097 ceph-mon[32231]: pgmap v7438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:38.030+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:33:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:38.032+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:33:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:38.032+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:33:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:38.032+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:33:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:38.032+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:39.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:39 smithi115 ceph-mon[35364]: pgmap v7439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:39 smithi181 ceph-mon[35869]: pgmap v7439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:39 smithi097 ceph-mon[32231]: pgmap v7439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12857.958710 secs 2024-07-26T11:33:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12857.958710 secs 2024-07-26T11:33:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12857.958710 secs 2024-07-26T11:33:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:41 smithi115 ceph-mon[35364]: pgmap v7440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:41 smithi181 ceph-mon[35869]: pgmap v7440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:41 smithi097 ceph-mon[32231]: pgmap v7440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:43 smithi115 ceph-mon[35364]: pgmap v7441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:43 smithi181 ceph-mon[35869]: pgmap v7441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:43 smithi097 ceph-mon[32231]: pgmap v7441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:45 smithi115 ceph-mon[35364]: pgmap v7442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:45 smithi181 ceph-mon[35869]: pgmap v7442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:45 smithi097 ceph-mon[32231]: pgmap v7442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12862.958845 secs 2024-07-26T11:33:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12862.958845 secs 2024-07-26T11:33:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12862.958845 secs 2024-07-26T11:33:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:47 smithi115 ceph-mon[35364]: pgmap v7443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:47 smithi181 ceph-mon[35869]: pgmap v7443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:47 smithi097 ceph-mon[32231]: pgmap v7443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:49 smithi115 ceph-mon[35364]: pgmap v7444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:49 smithi181 ceph-mon[35869]: pgmap v7444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:49 smithi097 ceph-mon[32231]: pgmap v7444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12867.959004 secs 2024-07-26T11:33:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12867.959004 secs 2024-07-26T11:33:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12867.959004 secs 2024-07-26T11:33:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:51 smithi115 ceph-mon[35364]: pgmap v7445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:51 smithi181 ceph-mon[35869]: pgmap v7445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:51 smithi097 ceph-mon[32231]: pgmap v7445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:53.034+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:33:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:53.035+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:33:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:53.035+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:33:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:53.035+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:33:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:33:53.035+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:53 smithi115 ceph-mon[35364]: pgmap v7446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:53 smithi181 ceph-mon[35869]: pgmap v7446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:53 smithi097 ceph-mon[32231]: pgmap v7446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:33:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:55 smithi115 ceph-mon[35364]: pgmap v7447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:55 smithi181 ceph-mon[35869]: pgmap v7447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:55 smithi097 ceph-mon[32231]: pgmap v7447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12872.959142 secs 2024-07-26T11:33:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12872.959142 secs 2024-07-26T11:33:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12872.959142 secs 2024-07-26T11:33:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:57 smithi115 ceph-mon[35364]: pgmap v7448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:57 smithi181 ceph-mon[35869]: pgmap v7448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:57.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:57 smithi097 ceph-mon[32231]: pgmap v7448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:59.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:33:59 smithi115 ceph-mon[35364]: pgmap v7449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:59.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:33:59 smithi181 ceph-mon[35869]: pgmap v7449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:33:59 smithi097 ceph-mon[32231]: pgmap v7449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12877.959283 secs 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:34:01.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:34:01.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:01.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:01.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: pgmap v7450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:01.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: pgmap v7451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:01.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12877.959283 secs 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:34:01.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:34:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:34:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:34:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: pgmap v7450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: pgmap v7451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:01.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:34:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12877.959283 secs 2024-07-26T11:34:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: pgmap v7450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:01.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: pgmap v7451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:01.504 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:34:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:03 smithi115 ceph-mon[35364]: pgmap v7452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:03 smithi181 ceph-mon[35869]: pgmap v7452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:03 smithi097 ceph-mon[32231]: pgmap v7452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:05 smithi115 ceph-mon[35364]: pgmap v7453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12882.959416 secs 2024-07-26T11:34:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:05 smithi181 ceph-mon[35869]: pgmap v7453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12882.959416 secs 2024-07-26T11:34:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:05 smithi097 ceph-mon[32231]: pgmap v7453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12882.959416 secs 2024-07-26T11:34:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:07 smithi115 ceph-mon[35364]: pgmap v7454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:07 smithi181 ceph-mon[35869]: pgmap v7454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:07 smithi097 ceph-mon[32231]: pgmap v7454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:08.040+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:34:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:08.042+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:34:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:08.042+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:34:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:08.042+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:34:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:08.042+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:09.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:09.173+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:34:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:09.173+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:34:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:09.174+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:34:09.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:09.174+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:34:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:09 smithi115 ceph-mon[35364]: pgmap v7455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:09 smithi181 ceph-mon[35869]: pgmap v7455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:09 smithi097 ceph-mon[32231]: pgmap v7455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12887.959562 secs 2024-07-26T11:34:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12887.959562 secs 2024-07-26T11:34:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12887.959562 secs 2024-07-26T11:34:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:11 smithi115 ceph-mon[35364]: pgmap v7456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:11 smithi181 ceph-mon[35869]: pgmap v7456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:11 smithi097 ceph-mon[32231]: pgmap v7456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:13 smithi115 ceph-mon[35364]: pgmap v7457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:13 smithi181 ceph-mon[35869]: pgmap v7457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:13.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:13 smithi097 ceph-mon[32231]: pgmap v7457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:15 smithi115 ceph-mon[35364]: pgmap v7458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12892.959712 secs 2024-07-26T11:34:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:15 smithi181 ceph-mon[35869]: pgmap v7458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12892.959712 secs 2024-07-26T11:34:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:15 smithi097 ceph-mon[32231]: pgmap v7458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12892.959712 secs 2024-07-26T11:34:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:17 smithi115 ceph-mon[35364]: pgmap v7459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:17 smithi181 ceph-mon[35869]: pgmap v7459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:17 smithi097 ceph-mon[32231]: pgmap v7459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:19 smithi181 ceph-mon[35869]: pgmap v7460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:19 smithi097 ceph-mon[32231]: pgmap v7460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:19 smithi115 ceph-mon[35364]: pgmap v7460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12897.959855 secs 2024-07-26T11:34:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12897.959855 secs 2024-07-26T11:34:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12897.959855 secs 2024-07-26T11:34:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:21 smithi097 ceph-mon[32231]: pgmap v7461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:21 smithi115 ceph-mon[35364]: pgmap v7461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:21.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:21 smithi181 ceph-mon[35869]: pgmap v7461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:23.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:23.044+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:34:23.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:23.045+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:34:23.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:23.045+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:34:23.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:23.045+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:34:23.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:23.045+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:23.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:23 smithi097 ceph-mon[32231]: pgmap v7462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:23 smithi115 ceph-mon[35364]: pgmap v7462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:23 smithi181 ceph-mon[35869]: pgmap v7462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:25 smithi115 ceph-mon[35364]: pgmap v7463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12902.959999 secs 2024-07-26T11:34:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:25 smithi181 ceph-mon[35869]: pgmap v7463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12902.959999 secs 2024-07-26T11:34:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:25 smithi097 ceph-mon[32231]: pgmap v7463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12902.959999 secs 2024-07-26T11:34:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:27 smithi115 ceph-mon[35364]: pgmap v7464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:27 smithi181 ceph-mon[35869]: pgmap v7464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:27 smithi097 ceph-mon[32231]: pgmap v7464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:29.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:29 smithi115 ceph-mon[35364]: pgmap v7465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:29.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:29 smithi181 ceph-mon[35869]: pgmap v7465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:30.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:29 smithi097 ceph-mon[32231]: pgmap v7465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12907.960137 secs 2024-07-26T11:34:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12907.960137 secs 2024-07-26T11:34:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12907.960137 secs 2024-07-26T11:34:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:31 smithi115 ceph-mon[35364]: pgmap v7466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:31 smithi181 ceph-mon[35869]: pgmap v7466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:31 smithi097 ceph-mon[32231]: pgmap v7466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:33.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:33 smithi097 ceph-mon[32231]: pgmap v7467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:33.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:33 smithi115 ceph-mon[35364]: pgmap v7467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:33 smithi181 ceph-mon[35869]: pgmap v7467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:35 smithi115 ceph-mon[35364]: pgmap v7468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:34:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12912.960277 secs 2024-07-26T11:34:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:35 smithi181 ceph-mon[35869]: pgmap v7468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:34:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12912.960277 secs 2024-07-26T11:34:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:35 smithi097 ceph-mon[32231]: pgmap v7468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:34:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12912.960277 secs 2024-07-26T11:34:37.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:37 smithi115 ceph-mon[35364]: pgmap v7469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:34:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:37 smithi181 ceph-mon[35869]: pgmap v7469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:34:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:37 smithi097 ceph-mon[32231]: pgmap v7469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:34:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:38.052+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:34:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:38.053+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:34:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:38.053+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:34:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:38.054+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:34:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:38.054+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:39 smithi115 ceph-mon[35364]: pgmap v7470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:39 smithi181 ceph-mon[35869]: pgmap v7470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:39 smithi097 ceph-mon[32231]: pgmap v7470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12917.960417 secs 2024-07-26T11:34:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12917.960417 secs 2024-07-26T11:34:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12917.960417 secs 2024-07-26T11:34:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:41 smithi115 ceph-mon[35364]: pgmap v7471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:34:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:41 smithi181 ceph-mon[35869]: pgmap v7471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:34:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:41 smithi097 ceph-mon[32231]: pgmap v7471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:34:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:43 smithi115 ceph-mon[35364]: pgmap v7472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:43 smithi181 ceph-mon[35869]: pgmap v7472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:43 smithi097 ceph-mon[32231]: pgmap v7472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:45 smithi115 ceph-mon[35364]: pgmap v7473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12922.960568 secs 2024-07-26T11:34:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:45 smithi181 ceph-mon[35869]: pgmap v7473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12922.960568 secs 2024-07-26T11:34:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:45 smithi097 ceph-mon[32231]: pgmap v7473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12922.960568 secs 2024-07-26T11:34:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:47 smithi115 ceph-mon[35364]: pgmap v7474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:47 smithi181 ceph-mon[35869]: pgmap v7474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:47 smithi097 ceph-mon[32231]: pgmap v7474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:49 smithi115 ceph-mon[35364]: pgmap v7475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:49 smithi181 ceph-mon[35869]: pgmap v7475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:49 smithi097 ceph-mon[32231]: pgmap v7475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12927.960712 secs 2024-07-26T11:34:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12927.960712 secs 2024-07-26T11:34:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12927.960712 secs 2024-07-26T11:34:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:51 smithi115 ceph-mon[35364]: pgmap v7476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:51 smithi181 ceph-mon[35869]: pgmap v7476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:51 smithi097 ceph-mon[32231]: pgmap v7476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:53.056+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:34:53.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:53.057+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:34:53.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:53.057+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:34:53.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:53.057+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:34:53.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:34:53.057+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:53 smithi115 ceph-mon[35364]: pgmap v7477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:53 smithi181 ceph-mon[35869]: pgmap v7477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:53 smithi097 ceph-mon[32231]: pgmap v7477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:55 smithi115 ceph-mon[35364]: pgmap v7478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12932.960854 secs 2024-07-26T11:34:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:55 smithi181 ceph-mon[35869]: pgmap v7478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12932.960854 secs 2024-07-26T11:34:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:55 smithi097 ceph-mon[32231]: pgmap v7478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12932.960854 secs 2024-07-26T11:34:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:57 smithi115 ceph-mon[35364]: pgmap v7479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:57 smithi181 ceph-mon[35869]: pgmap v7479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:57 smithi097 ceph-mon[32231]: pgmap v7479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:34:59 smithi115 ceph-mon[35364]: pgmap v7480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:34:59 smithi181 ceph-mon[35869]: pgmap v7480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:34:59 smithi097 ceph-mon[32231]: pgmap v7480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12937.960998 secs 2024-07-26T11:35:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12937.960998 secs 2024-07-26T11:35:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12937.960998 secs 2024-07-26T11:35:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:01 smithi115 ceph-mon[35364]: pgmap v7481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:01 smithi181 ceph-mon[35869]: pgmap v7481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:01 smithi097 ceph-mon[32231]: pgmap v7481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:02.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:02.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:35:02.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:35:02.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:02.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:02.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: pgmap v7482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:02.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: pgmap v7483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:02.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:35:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: pgmap v7482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: pgmap v7483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:03.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: pgmap v7482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:03.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: pgmap v7483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:03.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:35:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:04 smithi115 ceph-mon[35364]: pgmap v7484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:04 smithi181 ceph-mon[35869]: pgmap v7484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:04 smithi097 ceph-mon[32231]: pgmap v7484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12942.961140 secs 2024-07-26T11:35:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12942.961140 secs 2024-07-26T11:35:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12942.961140 secs 2024-07-26T11:35:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:06 smithi115 ceph-mon[35364]: pgmap v7485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:35:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:06 smithi181 ceph-mon[35869]: pgmap v7485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:35:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:06 smithi097 ceph-mon[32231]: pgmap v7485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:35:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:08.064+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:08.065+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:35:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:08.065+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:35:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:08.065+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:08.065+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:08 smithi115 ceph-mon[35364]: pgmap v7486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:35:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:08 smithi181 ceph-mon[35869]: pgmap v7486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:35:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:08 smithi097 ceph-mon[32231]: pgmap v7486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:35:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:10 smithi115 ceph-mon[35364]: pgmap v7487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:35:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12947.961288 secs 2024-07-26T11:35:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:10 smithi181 ceph-mon[35869]: pgmap v7487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:35:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12947.961288 secs 2024-07-26T11:35:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:10 smithi097 ceph-mon[32231]: pgmap v7487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:35:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12947.961288 secs 2024-07-26T11:35:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:12 smithi115 ceph-mon[35364]: pgmap v7488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:35:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:12 smithi181 ceph-mon[35869]: pgmap v7488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:35:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:12 smithi097 ceph-mon[32231]: pgmap v7488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:35:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:14 smithi115 ceph-mon[35364]: pgmap v7489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:14 smithi181 ceph-mon[35869]: pgmap v7489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:14 smithi097 ceph-mon[32231]: pgmap v7489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12952.961434 secs 2024-07-26T11:35:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12952.961434 secs 2024-07-26T11:35:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12952.961434 secs 2024-07-26T11:35:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:16 smithi115 ceph-mon[35364]: pgmap v7490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:16 smithi181 ceph-mon[35869]: pgmap v7490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:16 smithi097 ceph-mon[32231]: pgmap v7490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:18 smithi115 ceph-mon[35364]: pgmap v7491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:18 smithi181 ceph-mon[35869]: pgmap v7491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:18 smithi097 ceph-mon[32231]: pgmap v7491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:20 smithi115 ceph-mon[35364]: pgmap v7492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12957.961582 secs 2024-07-26T11:35:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:20 smithi181 ceph-mon[35869]: pgmap v7492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12957.961582 secs 2024-07-26T11:35:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:20 smithi097 ceph-mon[32231]: pgmap v7492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12957.961582 secs 2024-07-26T11:35:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:23.066+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:23.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:23.067+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:35:23.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:23.067+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:35:23.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:23.067+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:23.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:23.067+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:22 smithi115 ceph-mon[35364]: pgmap v7493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:22 smithi181 ceph-mon[35869]: pgmap v7493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:22 smithi097 ceph-mon[32231]: pgmap v7493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:24 smithi115 ceph-mon[35364]: pgmap v7494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:24 smithi181 ceph-mon[35869]: pgmap v7494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:24 smithi097 ceph-mon[32231]: pgmap v7494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12962.961728 secs 2024-07-26T11:35:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12962.961728 secs 2024-07-26T11:35:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12962.961728 secs 2024-07-26T11:35:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:26 smithi115 ceph-mon[35364]: pgmap v7495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:26 smithi181 ceph-mon[35869]: pgmap v7495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:26 smithi097 ceph-mon[32231]: pgmap v7495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:28 smithi115 ceph-mon[35364]: pgmap v7496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:28 smithi181 ceph-mon[35869]: pgmap v7496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:28 smithi097 ceph-mon[32231]: pgmap v7496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:30 smithi115 ceph-mon[35364]: pgmap v7497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:35:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12967.961879 secs 2024-07-26T11:35:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:30 smithi181 ceph-mon[35869]: pgmap v7497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:35:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12967.961879 secs 2024-07-26T11:35:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:30 smithi097 ceph-mon[32231]: pgmap v7497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:35:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12967.961879 secs 2024-07-26T11:35:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:32 smithi115 ceph-mon[35364]: pgmap v7498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:32 smithi181 ceph-mon[35869]: pgmap v7498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:32 smithi097 ceph-mon[32231]: pgmap v7498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:34 smithi115 ceph-mon[35364]: pgmap v7499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:35:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:34 smithi181 ceph-mon[35869]: pgmap v7499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:35:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:34 smithi097 ceph-mon[32231]: pgmap v7499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:35:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12972.962000 secs 2024-07-26T11:35:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12972.962000 secs 2024-07-26T11:35:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12972.962000 secs 2024-07-26T11:35:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:36 smithi097 ceph-mon[32231]: pgmap v7500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:36 smithi115 ceph-mon[35364]: pgmap v7500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:36 smithi181 ceph-mon[35869]: pgmap v7500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:35:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:38.074+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:38.075+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:38.075+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:38.075+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:38.075+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:38 smithi115 ceph-mon[35364]: pgmap v7501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:38 smithi181 ceph-mon[35869]: pgmap v7501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:38 smithi097 ceph-mon[32231]: pgmap v7501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:40 smithi115 ceph-mon[35364]: pgmap v7502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12977.962164 secs 2024-07-26T11:35:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:40 smithi181 ceph-mon[35869]: pgmap v7502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12977.962164 secs 2024-07-26T11:35:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:40 smithi097 ceph-mon[32231]: pgmap v7502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12977.962164 secs 2024-07-26T11:35:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:42 smithi115 ceph-mon[35364]: pgmap v7503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:42 smithi181 ceph-mon[35869]: pgmap v7503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:42 smithi097 ceph-mon[32231]: pgmap v7503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:44 smithi115 ceph-mon[35364]: pgmap v7504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:44 smithi181 ceph-mon[35869]: pgmap v7504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:44 smithi097 ceph-mon[32231]: pgmap v7504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12982.962325 secs 2024-07-26T11:35:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12982.962325 secs 2024-07-26T11:35:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12982.962325 secs 2024-07-26T11:35:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:46 smithi115 ceph-mon[35364]: pgmap v7505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:46 smithi181 ceph-mon[35869]: pgmap v7505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:46 smithi097 ceph-mon[32231]: pgmap v7505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:48 smithi115 ceph-mon[35364]: pgmap v7506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:48 smithi181 ceph-mon[35869]: pgmap v7506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:48 smithi097 ceph-mon[32231]: pgmap v7506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:50 smithi115 ceph-mon[35364]: pgmap v7507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12987.962424 secs 2024-07-26T11:35:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:50 smithi181 ceph-mon[35869]: pgmap v7507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12987.962424 secs 2024-07-26T11:35:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:50 smithi097 ceph-mon[32231]: pgmap v7507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12987.962424 secs 2024-07-26T11:35:53.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:53.077+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:35:53.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:53.078+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:35:53.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:53.078+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:35:53.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:53.078+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:35:53.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:35:53.078+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:52 smithi115 ceph-mon[35364]: pgmap v7508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:52 smithi181 ceph-mon[35869]: pgmap v7508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:53.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:52 smithi097 ceph-mon[32231]: pgmap v7508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:54 smithi115 ceph-mon[35364]: pgmap v7509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:55.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:54 smithi181 ceph-mon[35869]: pgmap v7509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:54 smithi097 ceph-mon[32231]: pgmap v7509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12992.962572 secs 2024-07-26T11:35:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12992.962572 secs 2024-07-26T11:35:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12992.962572 secs 2024-07-26T11:35:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:56 smithi115 ceph-mon[35364]: pgmap v7510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:56 smithi181 ceph-mon[35869]: pgmap v7510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:56 smithi097 ceph-mon[32231]: pgmap v7510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:35:58 smithi115 ceph-mon[35364]: pgmap v7511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:35:58 smithi181 ceph-mon[35869]: pgmap v7511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:35:58 smithi097 ceph-mon[32231]: pgmap v7511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:00 smithi115 ceph-mon[35364]: pgmap v7512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 12997.962662 secs 2024-07-26T11:36:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:00 smithi181 ceph-mon[35869]: pgmap v7512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 12997.962662 secs 2024-07-26T11:36:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:00 smithi097 ceph-mon[32231]: pgmap v7512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 12997.962662 secs 2024-07-26T11:36:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:02 smithi115 ceph-mon[35364]: pgmap v7513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:02 smithi181 ceph-mon[35869]: pgmap v7513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:02 smithi097 ceph-mon[32231]: pgmap v7513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:04.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:36:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:04 smithi115 ceph-mon[35364]: pgmap v7514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:04 smithi115 ceph-mon[35364]: pgmap v7515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:04 smithi181 ceph-mon[35869]: pgmap v7514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:04 smithi181 ceph-mon[35869]: pgmap v7515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:04 smithi097 ceph-mon[32231]: pgmap v7514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:04 smithi097 ceph-mon[32231]: pgmap v7515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13002.962761 secs 2024-07-26T11:36:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13002.962761 secs 2024-07-26T11:36:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13002.962761 secs 2024-07-26T11:36:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:06 smithi097 ceph-mon[32231]: pgmap v7516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:06 smithi115 ceph-mon[35364]: pgmap v7516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:06 smithi181 ceph-mon[35869]: pgmap v7516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:08.085+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:36:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:08.086+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:36:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:08.086+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:36:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:08.086+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:36:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:08.086+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:09.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:08 smithi097 ceph-mon[32231]: pgmap v7517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:08 smithi115 ceph-mon[35364]: pgmap v7517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:08 smithi181 ceph-mon[35869]: pgmap v7517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:10 smithi097 ceph-mon[32231]: pgmap v7518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13007.962930 secs 2024-07-26T11:36:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:10 smithi115 ceph-mon[35364]: pgmap v7518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13007.962930 secs 2024-07-26T11:36:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:10 smithi181 ceph-mon[35869]: pgmap v7518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13007.962930 secs 2024-07-26T11:36:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:12 smithi115 ceph-mon[35364]: pgmap v7519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:12 smithi181 ceph-mon[35869]: pgmap v7519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:12 smithi097 ceph-mon[32231]: pgmap v7519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:36:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:14 smithi115 ceph-mon[35364]: pgmap v7520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:36:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:14 smithi181 ceph-mon[35869]: pgmap v7520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:36:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:14 smithi097 ceph-mon[32231]: pgmap v7520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:36:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13012.963069 secs 2024-07-26T11:36:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13012.963069 secs 2024-07-26T11:36:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13012.963069 secs 2024-07-26T11:36:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:16 smithi115 ceph-mon[35364]: pgmap v7521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:16 smithi181 ceph-mon[35869]: pgmap v7521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:16 smithi097 ceph-mon[32231]: pgmap v7521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:18 smithi115 ceph-mon[35364]: pgmap v7522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:18 smithi181 ceph-mon[35869]: pgmap v7522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:18 smithi097 ceph-mon[32231]: pgmap v7522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:20 smithi115 ceph-mon[35364]: pgmap v7523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13017.963212 secs 2024-07-26T11:36:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:20 smithi181 ceph-mon[35869]: pgmap v7523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13017.963212 secs 2024-07-26T11:36:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:20 smithi097 ceph-mon[32231]: pgmap v7523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13017.963212 secs 2024-07-26T11:36:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:23.088+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:23.090+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:23.090+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:23.090+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:23.090+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:22 smithi115 ceph-mon[35364]: pgmap v7524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:22 smithi181 ceph-mon[35869]: pgmap v7524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:22 smithi097 ceph-mon[32231]: pgmap v7524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:24 smithi115 ceph-mon[35364]: pgmap v7525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:24 smithi181 ceph-mon[35869]: pgmap v7525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:24 smithi097 ceph-mon[32231]: pgmap v7525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13022.963358 secs 2024-07-26T11:36:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13022.963358 secs 2024-07-26T11:36:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13022.963358 secs 2024-07-26T11:36:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:26 smithi115 ceph-mon[35364]: pgmap v7526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:26 smithi181 ceph-mon[35869]: pgmap v7526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:26 smithi097 ceph-mon[32231]: pgmap v7526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:28 smithi115 ceph-mon[35364]: pgmap v7527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:28 smithi181 ceph-mon[35869]: pgmap v7527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:28 smithi097 ceph-mon[32231]: pgmap v7527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:30 smithi115 ceph-mon[35364]: pgmap v7528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13027.963493 secs 2024-07-26T11:36:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:30 smithi181 ceph-mon[35869]: pgmap v7528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13027.963493 secs 2024-07-26T11:36:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:30 smithi097 ceph-mon[32231]: pgmap v7528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13027.963493 secs 2024-07-26T11:36:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:32 smithi115 ceph-mon[35364]: pgmap v7529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:36:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:32 smithi181 ceph-mon[35869]: pgmap v7529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:36:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:32 smithi097 ceph-mon[32231]: pgmap v7529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:36:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:34 smithi115 ceph-mon[35364]: pgmap v7530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:34 smithi181 ceph-mon[35869]: pgmap v7530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:34 smithi097 ceph-mon[32231]: pgmap v7530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13032.963598 secs 2024-07-26T11:36:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13032.963598 secs 2024-07-26T11:36:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13032.963598 secs 2024-07-26T11:36:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:36 smithi115 ceph-mon[35364]: pgmap v7531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:36:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:36 smithi181 ceph-mon[35869]: pgmap v7531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:36:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:36 smithi097 ceph-mon[32231]: pgmap v7531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:36:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:38.092+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:36:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:38.093+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:36:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:38.093+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:36:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:38.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:36:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:38.093+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:38 smithi115 ceph-mon[35364]: pgmap v7532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:36:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:38 smithi181 ceph-mon[35869]: pgmap v7532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:36:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:38 smithi097 ceph-mon[32231]: pgmap v7532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:36:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:40 smithi115 ceph-mon[35364]: pgmap v7533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13037.963780 secs 2024-07-26T11:36:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:40 smithi181 ceph-mon[35869]: pgmap v7533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13037.963780 secs 2024-07-26T11:36:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:40 smithi097 ceph-mon[32231]: pgmap v7533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:36:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13037.963780 secs 2024-07-26T11:36:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:42 smithi115 ceph-mon[35364]: pgmap v7534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:42 smithi181 ceph-mon[35869]: pgmap v7534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:42 smithi097 ceph-mon[32231]: pgmap v7534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:44 smithi115 ceph-mon[35364]: pgmap v7535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:44 smithi181 ceph-mon[35869]: pgmap v7535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:44 smithi097 ceph-mon[32231]: pgmap v7535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13042.963911 secs 2024-07-26T11:36:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13042.963911 secs 2024-07-26T11:36:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13042.963911 secs 2024-07-26T11:36:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:46 smithi097 ceph-mon[32231]: pgmap v7536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:46 smithi115 ceph-mon[35364]: pgmap v7536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:46 smithi181 ceph-mon[35869]: pgmap v7536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:49.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:48 smithi097 ceph-mon[32231]: pgmap v7537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:48 smithi115 ceph-mon[35364]: pgmap v7537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:48 smithi181 ceph-mon[35869]: pgmap v7537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:50 smithi097 ceph-mon[32231]: pgmap v7538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13047.964088 secs 2024-07-26T11:36:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:50 smithi115 ceph-mon[35364]: pgmap v7538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13047.964088 secs 2024-07-26T11:36:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:50 smithi181 ceph-mon[35869]: pgmap v7538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13047.964088 secs 2024-07-26T11:36:53.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:53.096+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:36:53.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:53.096+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:36:53.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:53.096+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:36:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:53.096+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:36:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:36:53.097+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:53 smithi115 ceph-mon[35364]: pgmap v7539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:53 smithi181 ceph-mon[35869]: pgmap v7539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:53 smithi097 ceph-mon[32231]: pgmap v7539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:54 smithi115 ceph-mon[35364]: pgmap v7540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:54 smithi181 ceph-mon[35869]: pgmap v7540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:54 smithi097 ceph-mon[32231]: pgmap v7540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13052.964283 secs 2024-07-26T11:36:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13052.964283 secs 2024-07-26T11:36:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13052.964283 secs 2024-07-26T11:36:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:56 smithi115 ceph-mon[35364]: pgmap v7541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:56 smithi181 ceph-mon[35869]: pgmap v7541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:56 smithi097 ceph-mon[32231]: pgmap v7541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:59.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:36:58 smithi115 ceph-mon[35364]: pgmap v7542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:59.187 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:36:58 smithi181 ceph-mon[35869]: pgmap v7542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:59.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:36:58 smithi097 ceph-mon[32231]: pgmap v7542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:00 smithi115 ceph-mon[35364]: pgmap v7543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13057.964294 secs 2024-07-26T11:37:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:00 smithi181 ceph-mon[35869]: pgmap v7543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13057.964294 secs 2024-07-26T11:37:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:00 smithi097 ceph-mon[32231]: pgmap v7543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13057.964294 secs 2024-07-26T11:37:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:02 smithi115 ceph-mon[35364]: pgmap v7544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:02 smithi181 ceph-mon[35869]: pgmap v7544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:03.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:02 smithi097 ceph-mon[32231]: pgmap v7544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:04 smithi115 ceph-mon[35364]: pgmap v7545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:04 smithi181 ceph-mon[35869]: pgmap v7545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:04 smithi097 ceph-mon[32231]: pgmap v7545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:05.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13062.964380 secs 2024-07-26T11:37:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13062.964380 secs 2024-07-26T11:37:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:06.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13062.964380 secs 2024-07-26T11:37:06.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:06.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:06.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:37:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:06 smithi115 ceph-mon[35364]: pgmap v7546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:07.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:06 smithi115 ceph-mon[35364]: pgmap v7547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:06 smithi181 ceph-mon[35869]: pgmap v7546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:06 smithi181 ceph-mon[35869]: pgmap v7547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:06 smithi097 ceph-mon[32231]: pgmap v7546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:06 smithi097 ceph-mon[32231]: pgmap v7547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:08.103+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:37:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:08.104+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:37:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:08.104+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:37:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:08.104+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:37:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:08.104+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:09.205+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:37:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:09.205+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:37:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:09.207+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:37:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:09.207+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:37:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:08 smithi115 ceph-mon[35364]: pgmap v7548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:08 smithi181 ceph-mon[35869]: pgmap v7548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:08 smithi097 ceph-mon[32231]: pgmap v7548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:10 smithi115 ceph-mon[35364]: pgmap v7549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13067.964529 secs 2024-07-26T11:37:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:10 smithi181 ceph-mon[35869]: pgmap v7549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13067.964529 secs 2024-07-26T11:37:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:10 smithi097 ceph-mon[32231]: pgmap v7549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13067.964529 secs 2024-07-26T11:37:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:12 smithi115 ceph-mon[35364]: pgmap v7550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:12 smithi181 ceph-mon[35869]: pgmap v7550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:12 smithi097 ceph-mon[32231]: pgmap v7550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:14 smithi115 ceph-mon[35364]: pgmap v7551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:14 smithi181 ceph-mon[35869]: pgmap v7551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:14 smithi097 ceph-mon[32231]: pgmap v7551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:37:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13072.964644 secs 2024-07-26T11:37:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13072.964644 secs 2024-07-26T11:37:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13072.964644 secs 2024-07-26T11:37:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:16 smithi115 ceph-mon[35364]: pgmap v7552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:16 smithi181 ceph-mon[35869]: pgmap v7552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:16 smithi097 ceph-mon[32231]: pgmap v7552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:18 smithi115 ceph-mon[35364]: pgmap v7553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:18 smithi181 ceph-mon[35869]: pgmap v7553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:18 smithi097 ceph-mon[32231]: pgmap v7553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:20 smithi115 ceph-mon[35364]: pgmap v7554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13077.964806 secs 2024-07-26T11:37:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:20 smithi181 ceph-mon[35869]: pgmap v7554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13077.964806 secs 2024-07-26T11:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:20 smithi097 ceph-mon[32231]: pgmap v7554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13077.964806 secs 2024-07-26T11:37:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:23.106+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:37:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:23.107+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:37:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:23.107+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:37:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:23.107+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:37:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:23.107+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:22 smithi115 ceph-mon[35364]: pgmap v7555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:22 smithi181 ceph-mon[35869]: pgmap v7555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:22 smithi097 ceph-mon[32231]: pgmap v7555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:24 smithi115 ceph-mon[35364]: pgmap v7556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:24 smithi181 ceph-mon[35869]: pgmap v7556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:24 smithi097 ceph-mon[32231]: pgmap v7556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13082.964940 secs 2024-07-26T11:37:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13082.964940 secs 2024-07-26T11:37:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13082.964940 secs 2024-07-26T11:37:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:26 smithi181 ceph-mon[35869]: pgmap v7557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:26 smithi097 ceph-mon[32231]: pgmap v7557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:26 smithi115 ceph-mon[35364]: pgmap v7557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:28 smithi097 ceph-mon[32231]: pgmap v7558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:29.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:28 smithi115 ceph-mon[35364]: pgmap v7558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:28 smithi181 ceph-mon[35869]: pgmap v7558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:30 smithi097 ceph-mon[32231]: pgmap v7559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13087.965082 secs 2024-07-26T11:37:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:30 smithi115 ceph-mon[35364]: pgmap v7559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13087.965082 secs 2024-07-26T11:37:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:30 smithi181 ceph-mon[35869]: pgmap v7559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13087.965082 secs 2024-07-26T11:37:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:33 smithi115 ceph-mon[35364]: pgmap v7560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:33 smithi181 ceph-mon[35869]: pgmap v7560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:33 smithi097 ceph-mon[32231]: pgmap v7560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:35.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:35 smithi097 ceph-mon[32231]: pgmap v7561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:35 smithi115 ceph-mon[35364]: pgmap v7561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:35 smithi181 ceph-mon[35869]: pgmap v7561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13092.965226 secs 2024-07-26T11:37:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13092.965226 secs 2024-07-26T11:37:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13092.965226 secs 2024-07-26T11:37:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:37 smithi115 ceph-mon[35364]: pgmap v7562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:37 smithi181 ceph-mon[35869]: pgmap v7562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:37 smithi097 ceph-mon[32231]: pgmap v7562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:38.114+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:37:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:38.115+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:37:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:38.115+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:37:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:38.115+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:37:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:38.115+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:38 smithi115 ceph-mon[35364]: pgmap v7563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:38 smithi181 ceph-mon[35869]: pgmap v7563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:38 smithi097 ceph-mon[32231]: pgmap v7563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:40 smithi115 ceph-mon[35364]: pgmap v7564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13097.965369 secs 2024-07-26T11:37:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:40 smithi181 ceph-mon[35869]: pgmap v7564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13097.965369 secs 2024-07-26T11:37:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:40 smithi097 ceph-mon[32231]: pgmap v7564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13097.965369 secs 2024-07-26T11:37:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:42 smithi115 ceph-mon[35364]: pgmap v7565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:42 smithi181 ceph-mon[35869]: pgmap v7565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:42 smithi097 ceph-mon[32231]: pgmap v7565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:44 smithi115 ceph-mon[35364]: pgmap v7566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:44 smithi181 ceph-mon[35869]: pgmap v7566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:44 smithi097 ceph-mon[32231]: pgmap v7566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13102.965511 secs 2024-07-26T11:37:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13102.965511 secs 2024-07-26T11:37:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13102.965511 secs 2024-07-26T11:37:47.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:46 smithi115 ceph-mon[35364]: pgmap v7567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:46 smithi181 ceph-mon[35869]: pgmap v7567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:46 smithi097 ceph-mon[32231]: pgmap v7567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:48 smithi115 ceph-mon[35364]: pgmap v7568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:48 smithi181 ceph-mon[35869]: pgmap v7568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:48 smithi097 ceph-mon[32231]: pgmap v7568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:50 smithi115 ceph-mon[35364]: pgmap v7569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13107.965654 secs 2024-07-26T11:37:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:50 smithi181 ceph-mon[35869]: pgmap v7569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13107.965654 secs 2024-07-26T11:37:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:50 smithi097 ceph-mon[32231]: pgmap v7569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13107.965654 secs 2024-07-26T11:37:53.118 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:53.117+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:37:53.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:53.118+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:37:53.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:53.118+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:37:53.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:53.118+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:37:53.120 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:37:53.118+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:52 smithi115 ceph-mon[35364]: pgmap v7570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:52 smithi181 ceph-mon[35869]: pgmap v7570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:52 smithi097 ceph-mon[32231]: pgmap v7570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:55.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:54 smithi115 ceph-mon[35364]: pgmap v7571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:54 smithi181 ceph-mon[35869]: pgmap v7571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:54 smithi097 ceph-mon[32231]: pgmap v7571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13112.965820 secs 2024-07-26T11:37:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13112.965820 secs 2024-07-26T11:37:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13112.965820 secs 2024-07-26T11:37:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:56 smithi115 ceph-mon[35364]: pgmap v7572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:56 smithi181 ceph-mon[35869]: pgmap v7572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:56 smithi097 ceph-mon[32231]: pgmap v7572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:37:58 smithi115 ceph-mon[35364]: pgmap v7573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:37:58 smithi181 ceph-mon[35869]: pgmap v7573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:37:58 smithi097 ceph-mon[32231]: pgmap v7573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:00 smithi115 ceph-mon[35364]: pgmap v7574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13117.965979 secs 2024-07-26T11:38:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:00 smithi181 ceph-mon[35869]: pgmap v7574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13117.965979 secs 2024-07-26T11:38:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:00 smithi097 ceph-mon[32231]: pgmap v7574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13117.965979 secs 2024-07-26T11:38:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:02 smithi115 ceph-mon[35364]: pgmap v7575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:02 smithi181 ceph-mon[35869]: pgmap v7575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:02 smithi097 ceph-mon[32231]: pgmap v7575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:04 smithi115 ceph-mon[35364]: pgmap v7576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:04 smithi181 ceph-mon[35869]: pgmap v7576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:04 smithi097 ceph-mon[32231]: pgmap v7576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13122.966127 secs 2024-07-26T11:38:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13122.966127 secs 2024-07-26T11:38:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13122.966127 secs 2024-07-26T11:38:07.179 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:06 smithi181 ceph-mon[35869]: pgmap v7577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:07.180 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:07.201 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:06 smithi115 ceph-mon[35364]: pgmap v7577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:07.201 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:06 smithi097 ceph-mon[32231]: pgmap v7577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:08.123+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:38:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:08.124+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:38:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:08.124+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:38:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:08.124+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:38:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:08.124+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:08 smithi115 ceph-mon[35364]: pgmap v7578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:08 smithi181 ceph-mon[35869]: pgmap v7578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:08 smithi097 ceph-mon[32231]: pgmap v7578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:10.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:10 smithi115 ceph-mon[35364]: pgmap v7579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:10 smithi181 ceph-mon[35869]: pgmap v7579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:10 smithi097 ceph-mon[32231]: pgmap v7579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13127.966309 secs 2024-07-26T11:38:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:11 smithi115 ceph-mon[35364]: pgmap v7580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:38:11.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:11 smithi115 ceph-mon[35364]: pgmap v7581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:38:11.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13127.966309 secs 2024-07-26T11:38:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:11.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:11.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:11.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:11 smithi181 ceph-mon[35869]: pgmap v7580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:38:11.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:11 smithi181 ceph-mon[35869]: pgmap v7581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:38:11.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13127.966309 secs 2024-07-26T11:38:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:11 smithi097 ceph-mon[32231]: pgmap v7580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:38:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:11 smithi097 ceph-mon[32231]: pgmap v7581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:38:11.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:38:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:13 smithi097 ceph-mon[32231]: pgmap v7582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:38:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:13 smithi115 ceph-mon[35364]: pgmap v7582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:38:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:13 smithi181 ceph-mon[35869]: pgmap v7582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:38:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:15 smithi097 ceph-mon[32231]: pgmap v7583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:38:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13132.966420 secs 2024-07-26T11:38:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:15 smithi115 ceph-mon[35364]: pgmap v7583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:38:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13132.966420 secs 2024-07-26T11:38:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:15 smithi181 ceph-mon[35869]: pgmap v7583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:38:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13132.966420 secs 2024-07-26T11:38:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:18 smithi115 ceph-mon[35364]: pgmap v7584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:18 smithi181 ceph-mon[35869]: pgmap v7584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:18 smithi097 ceph-mon[32231]: pgmap v7584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:20 smithi115 ceph-mon[35364]: pgmap v7585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:20 smithi181 ceph-mon[35869]: pgmap v7585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:20 smithi097 ceph-mon[32231]: pgmap v7585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13137.966580 secs 2024-07-26T11:38:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13137.966580 secs 2024-07-26T11:38:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13137.966580 secs 2024-07-26T11:38:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:22 smithi115 ceph-mon[35364]: pgmap v7586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:38:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:22 smithi181 ceph-mon[35869]: pgmap v7586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:38:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:22 smithi097 ceph-mon[32231]: pgmap v7586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:38:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:23.126+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:38:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:23.127+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:38:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:23.127+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:38:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:23.127+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:38:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:23.127+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:24 smithi115 ceph-mon[35364]: pgmap v7587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:24.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:24 smithi181 ceph-mon[35869]: pgmap v7587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:24 smithi097 ceph-mon[32231]: pgmap v7587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:25 smithi115 ceph-mon[35364]: pgmap v7588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:25 smithi181 ceph-mon[35869]: pgmap v7588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:25 smithi097 ceph-mon[32231]: pgmap v7588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13142.966754 secs 2024-07-26T11:38:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13142.966754 secs 2024-07-26T11:38:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13142.966754 secs 2024-07-26T11:38:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:27 smithi115 ceph-mon[35364]: pgmap v7589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:27 smithi181 ceph-mon[35869]: pgmap v7589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:27 smithi097 ceph-mon[32231]: pgmap v7589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:29 smithi097 ceph-mon[32231]: pgmap v7590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:30 smithi115 ceph-mon[35364]: pgmap v7590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:29 smithi181 ceph-mon[35869]: pgmap v7590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13147.966936 secs 2024-07-26T11:38:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13147.966936 secs 2024-07-26T11:38:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13147.966936 secs 2024-07-26T11:38:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:32 smithi115 ceph-mon[35364]: pgmap v7591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:32 smithi181 ceph-mon[35869]: pgmap v7591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:32 smithi097 ceph-mon[32231]: pgmap v7591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:34 smithi115 ceph-mon[35364]: pgmap v7592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:34 smithi181 ceph-mon[35869]: pgmap v7592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:34 smithi097 ceph-mon[32231]: pgmap v7592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:35.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:35 smithi097 ceph-mon[32231]: pgmap v7593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:35 smithi115 ceph-mon[35364]: pgmap v7593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:35 smithi181 ceph-mon[35869]: pgmap v7593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13152.967094 secs 2024-07-26T11:38:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13152.967094 secs 2024-07-26T11:38:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13152.967094 secs 2024-07-26T11:38:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:37 smithi115 ceph-mon[35364]: pgmap v7594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:37 smithi181 ceph-mon[35869]: pgmap v7594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:37 smithi097 ceph-mon[32231]: pgmap v7594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:38.130+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:38:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:38.131+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:38:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:38.131+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:38:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:38.131+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:38:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:38.131+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:40 smithi115 ceph-mon[35364]: pgmap v7595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:40 smithi181 ceph-mon[35869]: pgmap v7595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:40 smithi097 ceph-mon[32231]: pgmap v7595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13157.967295 secs 2024-07-26T11:38:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13157.967295 secs 2024-07-26T11:38:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13157.967295 secs 2024-07-26T11:38:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:42 smithi115 ceph-mon[35364]: pgmap v7596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:42 smithi181 ceph-mon[35869]: pgmap v7596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:42 smithi097 ceph-mon[32231]: pgmap v7596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:44 smithi115 ceph-mon[35364]: pgmap v7597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:44 smithi181 ceph-mon[35869]: pgmap v7597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:44 smithi097 ceph-mon[32231]: pgmap v7597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:45 smithi115 ceph-mon[35364]: pgmap v7598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:45 smithi181 ceph-mon[35869]: pgmap v7598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:45 smithi097 ceph-mon[32231]: pgmap v7598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13162.967453 secs 2024-07-26T11:38:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13162.967453 secs 2024-07-26T11:38:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13162.967453 secs 2024-07-26T11:38:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:47 smithi115 ceph-mon[35364]: pgmap v7599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:47 smithi181 ceph-mon[35869]: pgmap v7599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:47 smithi097 ceph-mon[32231]: pgmap v7599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:50 smithi115 ceph-mon[35364]: pgmap v7600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:50 smithi181 ceph-mon[35869]: pgmap v7600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:50 smithi097 ceph-mon[32231]: pgmap v7600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13167.967617 secs 2024-07-26T11:38:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13167.967617 secs 2024-07-26T11:38:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13167.967617 secs 2024-07-26T11:38:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:52 smithi115 ceph-mon[35364]: pgmap v7601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:52 smithi181 ceph-mon[35869]: pgmap v7601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:52 smithi097 ceph-mon[32231]: pgmap v7601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:53.133+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:38:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:53.134+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:38:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:53.134+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:38:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:53.134+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:38:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:38:53.134+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:53 smithi115 ceph-mon[35364]: pgmap v7602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:53 smithi181 ceph-mon[35869]: pgmap v7602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:53 smithi097 ceph-mon[32231]: pgmap v7602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:56 smithi115 ceph-mon[35364]: pgmap v7603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13172.967770 secs 2024-07-26T11:38:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:56 smithi181 ceph-mon[35869]: pgmap v7603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13172.967770 secs 2024-07-26T11:38:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:56 smithi097 ceph-mon[32231]: pgmap v7603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13172.967770 secs 2024-07-26T11:38:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:38:58 smithi115 ceph-mon[35364]: pgmap v7604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:38:58 smithi181 ceph-mon[35869]: pgmap v7604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:38:58 smithi097 ceph-mon[32231]: pgmap v7604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:00.279 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:00 smithi115 ceph-mon[35364]: pgmap v7605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:00.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:00 smithi181 ceph-mon[35869]: pgmap v7605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:00 smithi097 ceph-mon[32231]: pgmap v7605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13177.967917 secs 2024-07-26T11:39:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13177.967917 secs 2024-07-26T11:39:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13177.967917 secs 2024-07-26T11:39:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:02 smithi115 ceph-mon[35364]: pgmap v7606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:02 smithi181 ceph-mon[35869]: pgmap v7606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:02 smithi097 ceph-mon[32231]: pgmap v7606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:03 smithi115 ceph-mon[35364]: pgmap v7607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:03.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:03 smithi181 ceph-mon[35869]: pgmap v7607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:03.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:03 smithi097 ceph-mon[32231]: pgmap v7607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:06 smithi115 ceph-mon[35364]: pgmap v7608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13182.968052 secs 2024-07-26T11:39:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:06 smithi181 ceph-mon[35869]: pgmap v7608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13182.968052 secs 2024-07-26T11:39:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:06 smithi097 ceph-mon[32231]: pgmap v7608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13182.968052 secs 2024-07-26T11:39:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:08.141+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:39:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:08.143+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:39:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:08.143+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:39:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:08.143+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:39:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:08.143+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:08 smithi115 ceph-mon[35364]: pgmap v7609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:08 smithi181 ceph-mon[35869]: pgmap v7609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:08 smithi097 ceph-mon[32231]: pgmap v7609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:09 smithi115 ceph-mon[35364]: pgmap v7610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:09 smithi181 ceph-mon[35869]: pgmap v7610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:09 smithi097 ceph-mon[32231]: pgmap v7610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13187.968197 secs 2024-07-26T11:39:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13187.968197 secs 2024-07-26T11:39:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13187.968197 secs 2024-07-26T11:39:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:11 smithi115 ceph-mon[35364]: pgmap v7611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:11 smithi181 ceph-mon[35869]: pgmap v7611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:11 smithi097 ceph-mon[32231]: pgmap v7611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:14.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: pgmap v7612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:14.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: pgmap v7613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:14.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: pgmap v7612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: pgmap v7613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:14.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: pgmap v7612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: pgmap v7613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:14.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:39:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:15 smithi097 ceph-mon[32231]: pgmap v7614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13192.968349 secs 2024-07-26T11:39:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:15 smithi115 ceph-mon[35364]: pgmap v7614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13192.968349 secs 2024-07-26T11:39:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:15 smithi181 ceph-mon[35869]: pgmap v7614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13192.968349 secs 2024-07-26T11:39:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:17 smithi097 ceph-mon[32231]: pgmap v7615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:17 smithi115 ceph-mon[35364]: pgmap v7615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:17 smithi181 ceph-mon[35869]: pgmap v7615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:20 smithi115 ceph-mon[35364]: pgmap v7616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:20 smithi181 ceph-mon[35869]: pgmap v7616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:20 smithi097 ceph-mon[32231]: pgmap v7616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13197.968503 secs 2024-07-26T11:39:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13197.968503 secs 2024-07-26T11:39:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13197.968503 secs 2024-07-26T11:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:22 smithi115 ceph-mon[35364]: pgmap v7617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:22 smithi181 ceph-mon[35869]: pgmap v7617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:22 smithi097 ceph-mon[32231]: pgmap v7617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:23.145+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:39:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:23.146+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:39:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:23.146+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:39:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:23.146+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:39:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:23.147+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:24 smithi115 ceph-mon[35364]: pgmap v7618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:24 smithi181 ceph-mon[35869]: pgmap v7618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:24 smithi097 ceph-mon[32231]: pgmap v7618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:26 smithi115 ceph-mon[35364]: pgmap v7619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13202.968641 secs 2024-07-26T11:39:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:26 smithi181 ceph-mon[35869]: pgmap v7619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13202.968641 secs 2024-07-26T11:39:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:26 smithi097 ceph-mon[32231]: pgmap v7619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13202.968641 secs 2024-07-26T11:39:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:27 smithi115 ceph-mon[35364]: pgmap v7620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:27 smithi181 ceph-mon[35869]: pgmap v7620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:27 smithi097 ceph-mon[32231]: pgmap v7620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:29 smithi097 ceph-mon[32231]: pgmap v7621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:29 smithi115 ceph-mon[35364]: pgmap v7621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:29 smithi181 ceph-mon[35869]: pgmap v7621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13207.968777 secs 2024-07-26T11:39:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13207.968777 secs 2024-07-26T11:39:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13207.968777 secs 2024-07-26T11:39:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:32 smithi115 ceph-mon[35364]: pgmap v7622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:32 smithi181 ceph-mon[35869]: pgmap v7622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:32 smithi097 ceph-mon[32231]: pgmap v7622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:34 smithi115 ceph-mon[35364]: pgmap v7623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:34 smithi181 ceph-mon[35869]: pgmap v7623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:34 smithi097 ceph-mon[32231]: pgmap v7623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:36.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:36 smithi097 ceph-mon[32231]: pgmap v7624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:36.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13212.968874 secs 2024-07-26T11:39:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:36 smithi115 ceph-mon[35364]: pgmap v7624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13212.968874 secs 2024-07-26T11:39:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:36 smithi181 ceph-mon[35869]: pgmap v7624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13212.968874 secs 2024-07-26T11:39:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:38.154+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:39:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:38.155+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:39:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:38.155+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:39:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:38.155+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:39:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:38.156+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:38 smithi115 ceph-mon[35364]: pgmap v7625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:38 smithi181 ceph-mon[35869]: pgmap v7625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:38 smithi097 ceph-mon[32231]: pgmap v7625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:40 smithi115 ceph-mon[35364]: pgmap v7626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:40 smithi181 ceph-mon[35869]: pgmap v7626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:40 smithi097 ceph-mon[32231]: pgmap v7626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13217.969011 secs 2024-07-26T11:39:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:41 smithi115 ceph-mon[35364]: pgmap v7627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13217.969011 secs 2024-07-26T11:39:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:41 smithi181 ceph-mon[35869]: pgmap v7627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13217.969011 secs 2024-07-26T11:39:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:41 smithi097 ceph-mon[32231]: pgmap v7627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:44 smithi115 ceph-mon[35364]: pgmap v7628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:44 smithi181 ceph-mon[35869]: pgmap v7628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:44 smithi097 ceph-mon[32231]: pgmap v7628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:46 smithi115 ceph-mon[35364]: pgmap v7629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13222.969168 secs 2024-07-26T11:39:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:46 smithi181 ceph-mon[35869]: pgmap v7629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13222.969168 secs 2024-07-26T11:39:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:46 smithi097 ceph-mon[32231]: pgmap v7629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13222.969168 secs 2024-07-26T11:39:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:48 smithi115 ceph-mon[35364]: pgmap v7630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:48 smithi181 ceph-mon[35869]: pgmap v7630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:48 smithi097 ceph-mon[32231]: pgmap v7630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:50 smithi115 ceph-mon[35364]: pgmap v7631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:50 smithi181 ceph-mon[35869]: pgmap v7631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:50 smithi097 ceph-mon[32231]: pgmap v7631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13227.969308 secs 2024-07-26T11:39:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13227.969308 secs 2024-07-26T11:39:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13227.969308 secs 2024-07-26T11:39:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:52 smithi115 ceph-mon[35364]: pgmap v7632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:52 smithi181 ceph-mon[35869]: pgmap v7632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:52 smithi097 ceph-mon[32231]: pgmap v7632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:39:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:53.157+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:39:53.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:53.159+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:39:53.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:53.159+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:39:53.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:53.159+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:39:53.161 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:39:53.159+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:53 smithi115 ceph-mon[35364]: pgmap v7633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:53 smithi181 ceph-mon[35869]: pgmap v7633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:53 smithi097 ceph-mon[32231]: pgmap v7633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:56 smithi115 ceph-mon[35364]: pgmap v7634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13232.969408 secs 2024-07-26T11:39:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:56 smithi181 ceph-mon[35869]: pgmap v7634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13232.969408 secs 2024-07-26T11:39:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:56 smithi097 ceph-mon[32231]: pgmap v7634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:39:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13232.969408 secs 2024-07-26T11:39:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:39:58 smithi115 ceph-mon[35364]: pgmap v7635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:39:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:39:58 smithi181 ceph-mon[35869]: pgmap v7635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:39:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:39:58 smithi097 ceph-mon[32231]: pgmap v7635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:00.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:00 smithi115 ceph-mon[35364]: pgmap v7636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:00.278 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:00.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:00 smithi181 ceph-mon[35869]: pgmap v7636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:00.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:00 smithi097 ceph-mon[32231]: pgmap v7636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:00.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13237.969551 secs 2024-07-26T11:40:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13237.969551 secs 2024-07-26T11:40:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13237.969551 secs 2024-07-26T11:40:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:02 smithi115 ceph-mon[35364]: pgmap v7637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:02 smithi181 ceph-mon[35869]: pgmap v7637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:02 smithi097 ceph-mon[32231]: pgmap v7637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:03 smithi115 ceph-mon[35364]: pgmap v7638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:03 smithi181 ceph-mon[35869]: pgmap v7638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:03 smithi097 ceph-mon[32231]: pgmap v7638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:06 smithi115 ceph-mon[35364]: pgmap v7639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13242.969648 secs 2024-07-26T11:40:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:06 smithi181 ceph-mon[35869]: pgmap v7639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13242.969648 secs 2024-07-26T11:40:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:06 smithi097 ceph-mon[32231]: pgmap v7639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13242.969648 secs 2024-07-26T11:40:08.168 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:08.166+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:08.167+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:40:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:08.167+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:40:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:08.167+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:08.167+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:08 smithi115 ceph-mon[35364]: pgmap v7640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:08 smithi181 ceph-mon[35869]: pgmap v7640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:08 smithi097 ceph-mon[32231]: pgmap v7640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:09.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:09.241+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:40:09.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:09.241+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:40:09.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:09.242+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:40:09.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:09.242+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:40:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:10 smithi115 ceph-mon[35364]: pgmap v7641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:10 smithi181 ceph-mon[35869]: pgmap v7641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:10 smithi097 ceph-mon[32231]: pgmap v7641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13247.969776 secs 2024-07-26T11:40:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13247.969776 secs 2024-07-26T11:40:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13247.969776 secs 2024-07-26T11:40:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:12 smithi115 ceph-mon[35364]: pgmap v7642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:12 smithi181 ceph-mon[35869]: pgmap v7642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:12 smithi097 ceph-mon[32231]: pgmap v7642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:13 smithi115 ceph-mon[35364]: pgmap v7643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:13 smithi181 ceph-mon[35869]: pgmap v7643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:13 smithi097 ceph-mon[32231]: pgmap v7643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:15 smithi115 ceph-mon[35364]: pgmap v7644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:40:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:15 smithi115 ceph-mon[35364]: pgmap v7645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:40:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:15 smithi181 ceph-mon[35869]: pgmap v7644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:40:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:15 smithi181 ceph-mon[35869]: pgmap v7645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:40:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:15 smithi097 ceph-mon[32231]: pgmap v7644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:40:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:15 smithi097 ceph-mon[32231]: pgmap v7645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:15.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:40:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13252.969926 secs 2024-07-26T11:40:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13252.969926 secs 2024-07-26T11:40:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13252.969926 secs 2024-07-26T11:40:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:17 smithi115 ceph-mon[35364]: pgmap v7646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:40:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:17 smithi181 ceph-mon[35869]: pgmap v7646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:40:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:17 smithi097 ceph-mon[32231]: pgmap v7646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:40:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:19 smithi115 ceph-mon[35364]: pgmap v7647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:40:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:19 smithi181 ceph-mon[35869]: pgmap v7647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:40:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:19 smithi097 ceph-mon[32231]: pgmap v7647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:40:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13257.970059 secs 2024-07-26T11:40:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13257.970059 secs 2024-07-26T11:40:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13257.970059 secs 2024-07-26T11:40:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:21 smithi115 ceph-mon[35364]: pgmap v7648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:40:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:21 smithi181 ceph-mon[35869]: pgmap v7648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:40:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:21 smithi097 ceph-mon[32231]: pgmap v7648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:40:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:23.168+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:23.170+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:40:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:23.170+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:40:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:23.170+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:23.170+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:23 smithi115 ceph-mon[35364]: pgmap v7649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:23 smithi181 ceph-mon[35869]: pgmap v7649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:23 smithi097 ceph-mon[32231]: pgmap v7649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:25 smithi115 ceph-mon[35364]: pgmap v7650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13262.970206 secs 2024-07-26T11:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:25 smithi181 ceph-mon[35869]: pgmap v7650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13262.970206 secs 2024-07-26T11:40:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:25 smithi097 ceph-mon[32231]: pgmap v7650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13262.970206 secs 2024-07-26T11:40:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:27 smithi115 ceph-mon[35364]: pgmap v7651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:27 smithi181 ceph-mon[35869]: pgmap v7651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:27 smithi097 ceph-mon[32231]: pgmap v7651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:29 smithi115 ceph-mon[35364]: pgmap v7652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:29 smithi181 ceph-mon[35869]: pgmap v7652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:29 smithi097 ceph-mon[32231]: pgmap v7652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13267.970352 secs 2024-07-26T11:40:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13267.970352 secs 2024-07-26T11:40:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13267.970352 secs 2024-07-26T11:40:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:31 smithi115 ceph-mon[35364]: pgmap v7653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:31 smithi181 ceph-mon[35869]: pgmap v7653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:31 smithi097 ceph-mon[32231]: pgmap v7653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:33 smithi181 ceph-mon[35869]: pgmap v7654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:33 smithi115 ceph-mon[35364]: pgmap v7654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:33 smithi097 ceph-mon[32231]: pgmap v7654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:35 smithi115 ceph-mon[35364]: pgmap v7655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13272.970503 secs 2024-07-26T11:40:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:35 smithi181 ceph-mon[35869]: pgmap v7655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13272.970503 secs 2024-07-26T11:40:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:35 smithi097 ceph-mon[32231]: pgmap v7655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13272.970503 secs 2024-07-26T11:40:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:37 smithi181 ceph-mon[35869]: pgmap v7656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:37 smithi097 ceph-mon[32231]: pgmap v7656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:37 smithi115 ceph-mon[35364]: pgmap v7656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:38.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:38.175+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:38.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:38.176+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:40:38.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:38.176+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:40:38.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:38.176+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:38.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:38.176+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:39 smithi097 ceph-mon[32231]: pgmap v7657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:39 smithi115 ceph-mon[35364]: pgmap v7657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:39 smithi181 ceph-mon[35869]: pgmap v7657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13277.970641 secs 2024-07-26T11:40:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13277.970641 secs 2024-07-26T11:40:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13277.970641 secs 2024-07-26T11:40:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:41 smithi115 ceph-mon[35364]: pgmap v7658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:40:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:41 smithi181 ceph-mon[35869]: pgmap v7658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:40:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:41 smithi097 ceph-mon[32231]: pgmap v7658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:40:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:43 smithi181 ceph-mon[35869]: pgmap v7659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:43 smithi115 ceph-mon[35364]: pgmap v7659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:43 smithi097 ceph-mon[32231]: pgmap v7659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:45 smithi115 ceph-mon[35364]: pgmap v7660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:40:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13282.970736 secs 2024-07-26T11:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:45 smithi181 ceph-mon[35869]: pgmap v7660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:40:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13282.970736 secs 2024-07-26T11:40:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:45 smithi097 ceph-mon[32231]: pgmap v7660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:40:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13282.970736 secs 2024-07-26T11:40:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:47 smithi115 ceph-mon[35364]: pgmap v7661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:47 smithi181 ceph-mon[35869]: pgmap v7661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:47 smithi097 ceph-mon[32231]: pgmap v7661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:49.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:49 smithi115 ceph-mon[35364]: pgmap v7662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:40:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:49 smithi181 ceph-mon[35869]: pgmap v7662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:40:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:49 smithi097 ceph-mon[32231]: pgmap v7662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:40:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13287.970819 secs 2024-07-26T11:40:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13287.970819 secs 2024-07-26T11:40:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13287.970819 secs 2024-07-26T11:40:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:51 smithi115 ceph-mon[35364]: pgmap v7663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:40:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:51 smithi181 ceph-mon[35869]: pgmap v7663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:40:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:51 smithi097 ceph-mon[32231]: pgmap v7663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:40:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:53.177+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:53.178+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:40:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:53.178+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:40:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:53.178+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:53.178+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:53 smithi115 ceph-mon[35364]: pgmap v7664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:53 smithi181 ceph-mon[35869]: pgmap v7664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:53 smithi097 ceph-mon[32231]: pgmap v7664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:54.250+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:40:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:54.250+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:40:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:54.251+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:40:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:40:54.251+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:40:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:55 smithi115 ceph-mon[35364]: pgmap v7665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:55.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13292.970953 secs 2024-07-26T11:40:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:55 smithi181 ceph-mon[35869]: pgmap v7665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13292.970953 secs 2024-07-26T11:40:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:55 smithi097 ceph-mon[32231]: pgmap v7665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13292.970953 secs 2024-07-26T11:40:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:57 smithi115 ceph-mon[35364]: pgmap v7666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:57 smithi181 ceph-mon[35869]: pgmap v7666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:57 smithi097 ceph-mon[32231]: pgmap v7666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:40:59 smithi115 ceph-mon[35364]: pgmap v7667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:40:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:40:59 smithi181 ceph-mon[35869]: pgmap v7667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:40:59 smithi097 ceph-mon[32231]: pgmap v7667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:00.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13297.971112 secs 2024-07-26T11:41:00.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13297.971112 secs 2024-07-26T11:41:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13297.971112 secs 2024-07-26T11:41:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:01 smithi115 ceph-mon[35364]: pgmap v7668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:01 smithi181 ceph-mon[35869]: pgmap v7668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:01 smithi097 ceph-mon[32231]: pgmap v7668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:03.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:03 smithi115 ceph-mon[35364]: pgmap v7669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:03 smithi181 ceph-mon[35869]: pgmap v7669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:03 smithi097 ceph-mon[32231]: pgmap v7669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:05 smithi115 ceph-mon[35364]: pgmap v7670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13302.971288 secs 2024-07-26T11:41:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:05 smithi181 ceph-mon[35869]: pgmap v7670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13302.971288 secs 2024-07-26T11:41:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:05 smithi097 ceph-mon[32231]: pgmap v7670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13302.971288 secs 2024-07-26T11:41:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:07 smithi115 ceph-mon[35364]: pgmap v7671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:07 smithi181 ceph-mon[35869]: pgmap v7671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:07 smithi097 ceph-mon[32231]: pgmap v7671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:08.179+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:08.181+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:41:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:08.181+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:41:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:08.181+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:08.181+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:09.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:09 smithi115 ceph-mon[35364]: pgmap v7672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:09 smithi181 ceph-mon[35869]: pgmap v7672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:09 smithi097 ceph-mon[32231]: pgmap v7672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13307.971430 secs 2024-07-26T11:41:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13307.971430 secs 2024-07-26T11:41:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13307.971430 secs 2024-07-26T11:41:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:11 smithi115 ceph-mon[35364]: pgmap v7673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:11 smithi181 ceph-mon[35869]: pgmap v7673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:11 smithi097 ceph-mon[32231]: pgmap v7673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:13 smithi115 ceph-mon[35364]: pgmap v7674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:13 smithi181 ceph-mon[35869]: pgmap v7674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:13 smithi097 ceph-mon[32231]: pgmap v7674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:15 smithi115 ceph-mon[35364]: pgmap v7675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13312.971517 secs 2024-07-26T11:41:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:15 smithi181 ceph-mon[35869]: pgmap v7675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13312.971517 secs 2024-07-26T11:41:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:15 smithi097 ceph-mon[32231]: pgmap v7675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13312.971517 secs 2024-07-26T11:41:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:16 smithi115 ceph-mon[35364]: pgmap v7676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:41:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:16 smithi115 ceph-mon[35364]: pgmap v7677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:41:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:16 smithi181 ceph-mon[35869]: pgmap v7676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:41:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:16 smithi181 ceph-mon[35869]: pgmap v7677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:41:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:16 smithi097 ceph-mon[32231]: pgmap v7676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:41:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:16 smithi097 ceph-mon[32231]: pgmap v7677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:41:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:18 smithi115 ceph-mon[35364]: pgmap v7678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:18 smithi181 ceph-mon[35869]: pgmap v7678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:18 smithi097 ceph-mon[32231]: pgmap v7678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:20 smithi115 ceph-mon[35364]: pgmap v7679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:41:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13317.971682 secs 2024-07-26T11:41:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:20 smithi181 ceph-mon[35869]: pgmap v7679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:41:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13317.971682 secs 2024-07-26T11:41:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:20 smithi097 ceph-mon[32231]: pgmap v7679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:41:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13317.971682 secs 2024-07-26T11:41:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:23.184+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:23.185+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:41:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:23.185+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:41:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:23.185+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:23.185+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:22 smithi115 ceph-mon[35364]: pgmap v7680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:22 smithi181 ceph-mon[35869]: pgmap v7680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:22 smithi097 ceph-mon[32231]: pgmap v7680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:41:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:24 smithi115 ceph-mon[35364]: pgmap v7681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:24 smithi181 ceph-mon[35869]: pgmap v7681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:24 smithi097 ceph-mon[32231]: pgmap v7681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13322.971843 secs 2024-07-26T11:41:26.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13322.971843 secs 2024-07-26T11:41:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13322.971843 secs 2024-07-26T11:41:27.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:26 smithi115 ceph-mon[35364]: pgmap v7682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:26 smithi181 ceph-mon[35869]: pgmap v7682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:26 smithi097 ceph-mon[32231]: pgmap v7682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:28 smithi115 ceph-mon[35364]: pgmap v7683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:28 smithi181 ceph-mon[35869]: pgmap v7683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:28 smithi097 ceph-mon[32231]: pgmap v7683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:30 smithi115 ceph-mon[35364]: pgmap v7684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13327.971991 secs 2024-07-26T11:41:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:30 smithi181 ceph-mon[35869]: pgmap v7684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13327.971991 secs 2024-07-26T11:41:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:30 smithi097 ceph-mon[32231]: pgmap v7684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13327.971991 secs 2024-07-26T11:41:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:32 smithi115 ceph-mon[35364]: pgmap v7685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:32 smithi181 ceph-mon[35869]: pgmap v7685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:32 smithi097 ceph-mon[32231]: pgmap v7685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:35.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:34 smithi115 ceph-mon[35364]: pgmap v7686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:34 smithi181 ceph-mon[35869]: pgmap v7686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:34 smithi097 ceph-mon[32231]: pgmap v7686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13332.972132 secs 2024-07-26T11:41:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13332.972132 secs 2024-07-26T11:41:36.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13332.972132 secs 2024-07-26T11:41:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:36 smithi115 ceph-mon[35364]: pgmap v7687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:36 smithi181 ceph-mon[35869]: pgmap v7687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:36 smithi097 ceph-mon[32231]: pgmap v7687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:38.191+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:38.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:38.192+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:41:38.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:38.192+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:41:38.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:38.192+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:38.195 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:38.193+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:38 smithi115 ceph-mon[35364]: pgmap v7688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:38 smithi181 ceph-mon[35869]: pgmap v7688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:38 smithi097 ceph-mon[32231]: pgmap v7688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:40 smithi115 ceph-mon[35364]: pgmap v7689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13337.972284 secs 2024-07-26T11:41:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:40 smithi181 ceph-mon[35869]: pgmap v7689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13337.972284 secs 2024-07-26T11:41:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:40 smithi097 ceph-mon[32231]: pgmap v7689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:41.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13337.972284 secs 2024-07-26T11:41:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:42 smithi115 ceph-mon[35364]: pgmap v7690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:42 smithi181 ceph-mon[35869]: pgmap v7690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:42 smithi097 ceph-mon[32231]: pgmap v7690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:44 smithi115 ceph-mon[35364]: pgmap v7691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:44 smithi181 ceph-mon[35869]: pgmap v7691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:44 smithi097 ceph-mon[32231]: pgmap v7691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13342.972429 secs 2024-07-26T11:41:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13342.972429 secs 2024-07-26T11:41:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13342.972429 secs 2024-07-26T11:41:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:46 smithi115 ceph-mon[35364]: pgmap v7692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:46 smithi181 ceph-mon[35869]: pgmap v7692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:46 smithi097 ceph-mon[32231]: pgmap v7692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:48 smithi115 ceph-mon[35364]: pgmap v7693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:41:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:48 smithi181 ceph-mon[35869]: pgmap v7693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:41:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:48 smithi097 ceph-mon[32231]: pgmap v7693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:41:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:50 smithi181 ceph-mon[35869]: pgmap v7694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13347.972536 secs 2024-07-26T11:41:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:50 smithi097 ceph-mon[32231]: pgmap v7694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13347.972536 secs 2024-07-26T11:41:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:50 smithi115 ceph-mon[35364]: pgmap v7694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13347.972536 secs 2024-07-26T11:41:53.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:53.195+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:53.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:53.197+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:41:53.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:53.197+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:41:53.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:53.197+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:53.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:41:53.197+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:52 smithi181 ceph-mon[35869]: pgmap v7695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:41:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:52 smithi097 ceph-mon[32231]: pgmap v7695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:41:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:52 smithi115 ceph-mon[35364]: pgmap v7695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:41:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:54 smithi097 ceph-mon[32231]: pgmap v7696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:54 smithi115 ceph-mon[35364]: pgmap v7696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:54 smithi181 ceph-mon[35869]: pgmap v7696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13352.972673 secs 2024-07-26T11:41:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13352.972673 secs 2024-07-26T11:41:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13352.972673 secs 2024-07-26T11:41:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:57 smithi115 ceph-mon[35364]: pgmap v7697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:57 smithi181 ceph-mon[35869]: pgmap v7697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:57 smithi097 ceph-mon[32231]: pgmap v7697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:41:59 smithi115 ceph-mon[35364]: pgmap v7698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:41:59 smithi181 ceph-mon[35869]: pgmap v7698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:41:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:41:59 smithi097 ceph-mon[32231]: pgmap v7698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:01.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:01 smithi115 ceph-mon[35364]: pgmap v7699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:01.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13357.972829 secs 2024-07-26T11:42:01.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:01 smithi181 ceph-mon[35869]: pgmap v7699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:01.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13357.972829 secs 2024-07-26T11:42:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:01 smithi097 ceph-mon[32231]: pgmap v7699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:01.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13357.972829 secs 2024-07-26T11:42:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:03 smithi115 ceph-mon[35364]: pgmap v7700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:03 smithi181 ceph-mon[35869]: pgmap v7700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:03 smithi097 ceph-mon[32231]: pgmap v7700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:05 smithi115 ceph-mon[35364]: pgmap v7701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:05 smithi181 ceph-mon[35869]: pgmap v7701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:05 smithi097 ceph-mon[32231]: pgmap v7701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13362.972971 secs 2024-07-26T11:42:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13362.972971 secs 2024-07-26T11:42:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13362.972971 secs 2024-07-26T11:42:07.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:07 smithi097 ceph-mon[32231]: pgmap v7702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:07 smithi115 ceph-mon[35364]: pgmap v7702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:07 smithi181 ceph-mon[35869]: pgmap v7702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:08.204+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:08.205+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:42:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:08.205+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:42:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:08.205+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:08.205+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:09 smithi115 ceph-mon[35364]: pgmap v7703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:09 smithi181 ceph-mon[35869]: pgmap v7703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:09 smithi097 ceph-mon[32231]: pgmap v7703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:11 smithi115 ceph-mon[35364]: pgmap v7704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13367.973108 secs 2024-07-26T11:42:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:11 smithi181 ceph-mon[35869]: pgmap v7704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13367.973108 secs 2024-07-26T11:42:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:11 smithi097 ceph-mon[32231]: pgmap v7704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13367.973108 secs 2024-07-26T11:42:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:13 smithi115 ceph-mon[35364]: pgmap v7705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:13 smithi181 ceph-mon[35869]: pgmap v7705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:13 smithi097 ceph-mon[32231]: pgmap v7705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:14 smithi115 ceph-mon[35364]: pgmap v7706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:14 smithi181 ceph-mon[35869]: pgmap v7706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:14 smithi097 ceph-mon[32231]: pgmap v7706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13372.973294 secs 2024-07-26T11:42:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13372.973294 secs 2024-07-26T11:42:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13372.973294 secs 2024-07-26T11:42:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:16 smithi181 ceph-mon[35869]: pgmap v7707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:16 smithi097 ceph-mon[32231]: pgmap v7707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:16 smithi115 ceph-mon[35364]: pgmap v7707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:42:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:42:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:17.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:42:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:18 smithi115 ceph-mon[35364]: pgmap v7708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:42:18.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:18 smithi115 ceph-mon[35364]: pgmap v7709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:42:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:18 smithi181 ceph-mon[35869]: pgmap v7708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:42:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:18 smithi181 ceph-mon[35869]: pgmap v7709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:42:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:18 smithi097 ceph-mon[32231]: pgmap v7708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T11:42:18.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:18 smithi097 ceph-mon[32231]: pgmap v7709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:42:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:20 smithi115 ceph-mon[35364]: pgmap v7710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13377.973428 secs 2024-07-26T11:42:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:20 smithi181 ceph-mon[35869]: pgmap v7710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13377.973428 secs 2024-07-26T11:42:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:20 smithi097 ceph-mon[32231]: pgmap v7710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13377.973428 secs 2024-07-26T11:42:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:22 smithi115 ceph-mon[35364]: pgmap v7711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:22 smithi181 ceph-mon[35869]: pgmap v7711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:22 smithi097 ceph-mon[32231]: pgmap v7711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:23.209 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:23.206+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:23.208+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:42:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:23.208+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:42:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:23.208+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:23.208+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:24 smithi115 ceph-mon[35364]: pgmap v7712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:24 smithi181 ceph-mon[35869]: pgmap v7712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:24 smithi097 ceph-mon[32231]: pgmap v7712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13382.973569 secs 2024-07-26T11:42:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13382.973569 secs 2024-07-26T11:42:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13382.973569 secs 2024-07-26T11:42:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:26 smithi115 ceph-mon[35364]: pgmap v7713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:26 smithi181 ceph-mon[35869]: pgmap v7713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:26 smithi097 ceph-mon[32231]: pgmap v7713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:28.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:28 smithi115 ceph-mon[35364]: pgmap v7714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:28 smithi181 ceph-mon[35869]: pgmap v7714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:28 smithi097 ceph-mon[32231]: pgmap v7714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:30 smithi115 ceph-mon[35364]: pgmap v7715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:30.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13387.973699 secs 2024-07-26T11:42:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:30 smithi181 ceph-mon[35869]: pgmap v7715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13387.973699 secs 2024-07-26T11:42:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:30 smithi097 ceph-mon[32231]: pgmap v7715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13387.973699 secs 2024-07-26T11:42:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:32 smithi115 ceph-mon[35364]: pgmap v7716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:32 smithi181 ceph-mon[35869]: pgmap v7716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:32 smithi097 ceph-mon[32231]: pgmap v7716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:34.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:34 smithi115 ceph-mon[35364]: pgmap v7717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:34.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:34 smithi181 ceph-mon[35869]: pgmap v7717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:34 smithi097 ceph-mon[32231]: pgmap v7717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13392.973839 secs 2024-07-26T11:42:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13392.973839 secs 2024-07-26T11:42:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13392.973839 secs 2024-07-26T11:42:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:36 smithi115 ceph-mon[35364]: pgmap v7718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:36 smithi181 ceph-mon[35869]: pgmap v7718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:36 smithi097 ceph-mon[32231]: pgmap v7718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:38.212+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:38.213+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:42:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:38.213+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:42:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:38.213+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:38.213+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:38 smithi115 ceph-mon[35364]: pgmap v7719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:38 smithi181 ceph-mon[35869]: pgmap v7719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:38 smithi097 ceph-mon[32231]: pgmap v7719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:40 smithi115 ceph-mon[35364]: pgmap v7720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13397.973993 secs 2024-07-26T11:42:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:40 smithi181 ceph-mon[35869]: pgmap v7720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13397.973993 secs 2024-07-26T11:42:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:40 smithi097 ceph-mon[32231]: pgmap v7720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:40.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13397.973993 secs 2024-07-26T11:42:42.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:42 smithi115 ceph-mon[35364]: pgmap v7721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:42 smithi181 ceph-mon[35869]: pgmap v7721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:42 smithi097 ceph-mon[32231]: pgmap v7721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:44 smithi115 ceph-mon[35364]: pgmap v7722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:44 smithi181 ceph-mon[35869]: pgmap v7722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:44 smithi097 ceph-mon[32231]: pgmap v7722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13402.974154 secs 2024-07-26T11:42:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13402.974154 secs 2024-07-26T11:42:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13402.974154 secs 2024-07-26T11:42:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:46 smithi115 ceph-mon[35364]: pgmap v7723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:46 smithi181 ceph-mon[35869]: pgmap v7723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:46 smithi097 ceph-mon[32231]: pgmap v7723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:48 smithi115 ceph-mon[35364]: pgmap v7724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:48 smithi181 ceph-mon[35869]: pgmap v7724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:48 smithi097 ceph-mon[32231]: pgmap v7724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:50 smithi115 ceph-mon[35364]: pgmap v7725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13407.974338 secs 2024-07-26T11:42:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:50 smithi181 ceph-mon[35869]: pgmap v7725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13407.974338 secs 2024-07-26T11:42:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:50 smithi097 ceph-mon[32231]: pgmap v7725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13407.974338 secs 2024-07-26T11:42:52.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:52 smithi115 ceph-mon[35364]: pgmap v7726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:52 smithi181 ceph-mon[35869]: pgmap v7726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:52 smithi097 ceph-mon[32231]: pgmap v7726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:53.215+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:53.216+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:42:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:53.216+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:42:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:53.216+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:42:53.216+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:54 smithi115 ceph-mon[35364]: pgmap v7727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:54 smithi181 ceph-mon[35869]: pgmap v7727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:54 smithi097 ceph-mon[32231]: pgmap v7727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13412.974474 secs 2024-07-26T11:42:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13412.974474 secs 2024-07-26T11:42:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13412.974474 secs 2024-07-26T11:42:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:56 smithi115 ceph-mon[35364]: pgmap v7728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:56 smithi181 ceph-mon[35869]: pgmap v7728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:56 smithi097 ceph-mon[32231]: pgmap v7728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:42:58 smithi181 ceph-mon[35869]: pgmap v7729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:42:58 smithi097 ceph-mon[32231]: pgmap v7729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:42:58 smithi115 ceph-mon[35364]: pgmap v7729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:00 smithi097 ceph-mon[32231]: pgmap v7730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13417.974609 secs 2024-07-26T11:43:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:00 smithi115 ceph-mon[35364]: pgmap v7730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13417.974609 secs 2024-07-26T11:43:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:00 smithi181 ceph-mon[35869]: pgmap v7730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13417.974609 secs 2024-07-26T11:43:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:02 smithi097 ceph-mon[32231]: pgmap v7731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:02.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:02 smithi115 ceph-mon[35364]: pgmap v7731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:02.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:02 smithi181 ceph-mon[35869]: pgmap v7731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:04.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:04 smithi097 ceph-mon[32231]: pgmap v7732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:04 smithi115 ceph-mon[35364]: pgmap v7732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:04 smithi181 ceph-mon[35869]: pgmap v7732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13422.974748 secs 2024-07-26T11:43:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13422.974748 secs 2024-07-26T11:43:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13422.974748 secs 2024-07-26T11:43:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.922+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 msgr2=0x7ff3d40b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T11:43:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.922+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 msgr2=0x7ff3d40b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:43:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.922+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 0x7ff3d40b59d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40b6840 tx=0x7ff3e0071000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.922+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 0x7ff3d40b59d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40b6840 tx=0x7ff3e0071000 comp rx=0 tx=0).stop 2024-07-26T11:43:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.922+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 msgr2=0x7ff3d40b59d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T11:43:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.922+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 msgr2=0x7ff3d40b59d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:43:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.922+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b55c0 0x7ff3d40b59d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:43:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.922+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 0x7ff3d40b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:43:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.923+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 0x7ff3d40b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:06.924+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 0x7ff3d40b5f10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057c0 tx=0x7ff3e00a4000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:43:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:06 smithi115 ceph-mon[35364]: pgmap v7733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:06 smithi181 ceph-mon[35869]: pgmap v7733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:06 smithi097 ceph-mon[32231]: pgmap v7733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:08.218+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:08.219+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:43:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:08.219+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:43:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:08.219+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:08.219+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:08.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:08 smithi097 ceph-mon[32231]: pgmap v7734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:08 smithi115 ceph-mon[35364]: pgmap v7734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:08 smithi181 ceph-mon[35869]: pgmap v7734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:09.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:09.274+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:43:09.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:09.274+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:43:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:09.275+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:43:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:09.276+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:43:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:10 smithi115 ceph-mon[35364]: pgmap v7735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13427.974889 secs 2024-07-26T11:43:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:10 smithi181 ceph-mon[35869]: pgmap v7735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13427.974889 secs 2024-07-26T11:43:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:10 smithi097 ceph-mon[32231]: pgmap v7735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13427.974889 secs 2024-07-26T11:43:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:12 smithi115 ceph-mon[35364]: pgmap v7736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:12 smithi181 ceph-mon[35869]: pgmap v7736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:12 smithi097 ceph-mon[32231]: pgmap v7736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:14 smithi115 ceph-mon[35364]: pgmap v7737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:14 smithi181 ceph-mon[35869]: pgmap v7737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:14 smithi097 ceph-mon[32231]: pgmap v7737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13432.975025 secs 2024-07-26T11:43:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13432.975025 secs 2024-07-26T11:43:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13432.975025 secs 2024-07-26T11:43:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:16 smithi115 ceph-mon[35364]: pgmap v7738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:16 smithi181 ceph-mon[35869]: pgmap v7738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:16 smithi097 ceph-mon[32231]: pgmap v7738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:18 smithi115 ceph-mon[35364]: pgmap v7739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:18 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:18 smithi181 ceph-mon[35869]: pgmap v7739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:18 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:18 smithi097 ceph-mon[32231]: pgmap v7739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:19.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:18 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:19 smithi115 ceph-mon[35364]: pgmap v7740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:19.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:43:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:19 smithi181 ceph-mon[35869]: pgmap v7740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:43:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:19 smithi097 ceph-mon[32231]: pgmap v7740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:20.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:43:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13437.975190 secs 2024-07-26T11:43:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13437.975190 secs 2024-07-26T11:43:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13437.975190 secs 2024-07-26T11:43:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:21 smithi115 ceph-mon[35364]: pgmap v7741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:43:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:21 smithi181 ceph-mon[35869]: pgmap v7741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:43:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:21 smithi097 ceph-mon[32231]: pgmap v7741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:43:23.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:23.221+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:23.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:23.222+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:43:23.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:23.222+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:43:23.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:23.222+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:23.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:23.223+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:23 smithi097 ceph-mon[32231]: pgmap v7742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:23 smithi115 ceph-mon[35364]: pgmap v7742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:23 smithi181 ceph-mon[35869]: pgmap v7742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:25 smithi097 ceph-mon[32231]: pgmap v7743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:43:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13442.975334 secs 2024-07-26T11:43:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:25 smithi115 ceph-mon[35364]: pgmap v7743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:43:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13442.975334 secs 2024-07-26T11:43:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:25 smithi181 ceph-mon[35869]: pgmap v7743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:43:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13442.975334 secs 2024-07-26T11:43:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:27 smithi097 ceph-mon[32231]: pgmap v7744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:43:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:27 smithi115 ceph-mon[35364]: pgmap v7744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:43:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:27 smithi181 ceph-mon[35869]: pgmap v7744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:43:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:29 smithi115 ceph-mon[35364]: pgmap v7745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:29 smithi181 ceph-mon[35869]: pgmap v7745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:29 smithi097 ceph-mon[32231]: pgmap v7745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:43:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13447.975429 secs 2024-07-26T11:43:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13447.975429 secs 2024-07-26T11:43:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13447.975429 secs 2024-07-26T11:43:32.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:31 smithi115 ceph-mon[35364]: pgmap v7746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:32.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:31 smithi181 ceph-mon[35869]: pgmap v7746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:31 smithi097 ceph-mon[32231]: pgmap v7746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:33 smithi115 ceph-mon[35364]: pgmap v7747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:33 smithi181 ceph-mon[35869]: pgmap v7747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:33 smithi097 ceph-mon[32231]: pgmap v7747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:35 smithi115 ceph-mon[35364]: pgmap v7748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13452.975527 secs 2024-07-26T11:43:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:35 smithi181 ceph-mon[35869]: pgmap v7748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13452.975527 secs 2024-07-26T11:43:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:35 smithi097 ceph-mon[32231]: pgmap v7748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13452.975527 secs 2024-07-26T11:43:38.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:37 smithi097 ceph-mon[32231]: pgmap v7749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:37 smithi115 ceph-mon[35364]: pgmap v7749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:38.226+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:38.227+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:38.227+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:38.227+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:38.227+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:37 smithi181 ceph-mon[35869]: pgmap v7749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:39 smithi115 ceph-mon[35364]: pgmap v7750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:39 smithi181 ceph-mon[35869]: pgmap v7750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:39 smithi097 ceph-mon[32231]: pgmap v7750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13457.975626 secs 2024-07-26T11:43:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13457.975626 secs 2024-07-26T11:43:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13457.975626 secs 2024-07-26T11:43:42.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:41 smithi115 ceph-mon[35364]: pgmap v7751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:41 smithi181 ceph-mon[35869]: pgmap v7751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:41 smithi097 ceph-mon[32231]: pgmap v7751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:43 smithi115 ceph-mon[35364]: pgmap v7752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:43 smithi181 ceph-mon[35869]: pgmap v7752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:43 smithi097 ceph-mon[32231]: pgmap v7752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:45 smithi115 ceph-mon[35364]: pgmap v7753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13462.975771 secs 2024-07-26T11:43:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:45 smithi181 ceph-mon[35869]: pgmap v7753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13462.975771 secs 2024-07-26T11:43:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:45 smithi097 ceph-mon[32231]: pgmap v7753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13462.975771 secs 2024-07-26T11:43:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:47 smithi115 ceph-mon[35364]: pgmap v7754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:47 smithi181 ceph-mon[35869]: pgmap v7754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:47 smithi097 ceph-mon[32231]: pgmap v7754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:49 smithi115 ceph-mon[35364]: pgmap v7755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:49 smithi181 ceph-mon[35869]: pgmap v7755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:49 smithi097 ceph-mon[32231]: pgmap v7755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13467.975916 secs 2024-07-26T11:43:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13467.975916 secs 2024-07-26T11:43:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13467.975916 secs 2024-07-26T11:43:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:51 smithi115 ceph-mon[35364]: pgmap v7756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:51 smithi181 ceph-mon[35869]: pgmap v7756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:51 smithi097 ceph-mon[32231]: pgmap v7756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:53.229+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:43:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:53.230+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:43:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:53.230+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:43:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:53.230+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:43:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:43:53.230+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:53 smithi115 ceph-mon[35364]: pgmap v7757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:53 smithi181 ceph-mon[35869]: pgmap v7757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:53 smithi097 ceph-mon[32231]: pgmap v7757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:55 smithi115 ceph-mon[35364]: pgmap v7758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13472.976065 secs 2024-07-26T11:43:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:55 smithi181 ceph-mon[35869]: pgmap v7758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13472.976065 secs 2024-07-26T11:43:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:55 smithi097 ceph-mon[32231]: pgmap v7758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13472.976065 secs 2024-07-26T11:43:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:57 smithi115 ceph-mon[35364]: pgmap v7759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:57 smithi181 ceph-mon[35869]: pgmap v7759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:57 smithi097 ceph-mon[32231]: pgmap v7759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:43:59 smithi115 ceph-mon[35364]: pgmap v7760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:43:59 smithi181 ceph-mon[35869]: pgmap v7760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:43:59 smithi097 ceph-mon[32231]: pgmap v7760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13477.976160 secs 2024-07-26T11:44:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13477.976160 secs 2024-07-26T11:44:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13477.976160 secs 2024-07-26T11:44:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:01 smithi115 ceph-mon[35364]: pgmap v7761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:01 smithi181 ceph-mon[35869]: pgmap v7761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:01 smithi097 ceph-mon[32231]: pgmap v7761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:04.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:03 smithi115 ceph-mon[35364]: pgmap v7762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:03 smithi181 ceph-mon[35869]: pgmap v7762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:03 smithi097 ceph-mon[32231]: pgmap v7762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:05 smithi115 ceph-mon[35364]: pgmap v7763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13482.976306 secs 2024-07-26T11:44:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:05 smithi181 ceph-mon[35869]: pgmap v7763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13482.976306 secs 2024-07-26T11:44:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:05 smithi097 ceph-mon[32231]: pgmap v7763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13482.976306 secs 2024-07-26T11:44:08.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:07 smithi115 ceph-mon[35364]: pgmap v7764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:07 smithi181 ceph-mon[35869]: pgmap v7764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:44:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:08.237+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:44:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:08.238+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:44:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:08.238+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:44:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:08.238+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:44:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:08.238+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:07 smithi097 ceph-mon[32231]: pgmap v7764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:44:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:09 smithi115 ceph-mon[35364]: pgmap v7765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:09 smithi181 ceph-mon[35869]: pgmap v7765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:09 smithi097 ceph-mon[32231]: pgmap v7765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13487.976474 secs 2024-07-26T11:44:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13487.976474 secs 2024-07-26T11:44:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13487.976474 secs 2024-07-26T11:44:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:11 smithi115 ceph-mon[35364]: pgmap v7766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:11 smithi181 ceph-mon[35869]: pgmap v7766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:11 smithi097 ceph-mon[32231]: pgmap v7766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:13 smithi115 ceph-mon[35364]: pgmap v7767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:13 smithi181 ceph-mon[35869]: pgmap v7767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:13 smithi097 ceph-mon[32231]: pgmap v7767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:15 smithi115 ceph-mon[35364]: pgmap v7768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13492.976618 secs 2024-07-26T11:44:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:15 smithi181 ceph-mon[35869]: pgmap v7768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13492.976618 secs 2024-07-26T11:44:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:15 smithi097 ceph-mon[32231]: pgmap v7768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13492.976618 secs 2024-07-26T11:44:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:17 smithi181 ceph-mon[35869]: pgmap v7769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:17 smithi097 ceph-mon[32231]: pgmap v7769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:17 smithi115 ceph-mon[35364]: pgmap v7769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:19 smithi097 ceph-mon[32231]: pgmap v7770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:19 smithi115 ceph-mon[35364]: pgmap v7770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:20.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:19 smithi181 ceph-mon[35869]: pgmap v7770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:20.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13497.976752 secs 2024-07-26T11:44:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:21.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:44:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13497.976752 secs 2024-07-26T11:44:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:21.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:44:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13497.976752 secs 2024-07-26T11:44:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:44:21.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:21.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:21.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:44:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:22 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:44:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:22 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:44:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:22 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:44:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:22 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:44:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:22 smithi115 ceph-mon[35364]: pgmap v7771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:22 smithi115 ceph-mon[35364]: pgmap v7772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:22 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:44:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:22 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:44:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:22 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:44:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:22 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:44:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:22 smithi181 ceph-mon[35869]: pgmap v7771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:22 smithi181 ceph-mon[35869]: pgmap v7772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:22 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:44:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:22 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:44:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:22 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:44:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:22 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:44:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:22 smithi097 ceph-mon[32231]: pgmap v7771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:22 smithi097 ceph-mon[32231]: pgmap v7772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:23.240+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:44:23.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:23.241+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:44:23.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:23.241+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:44:23.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:23.241+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:44:23.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:23.241+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:24 smithi115 ceph-mon[35364]: pgmap v7773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:44:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:24 smithi181 ceph-mon[35869]: pgmap v7773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:44:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:24 smithi097 ceph-mon[32231]: pgmap v7773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:44:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:26 smithi115 ceph-mon[35364]: pgmap v7774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13502.976896 secs 2024-07-26T11:44:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:26 smithi181 ceph-mon[35869]: pgmap v7774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13502.976896 secs 2024-07-26T11:44:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:26 smithi097 ceph-mon[32231]: pgmap v7774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:26.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13502.976896 secs 2024-07-26T11:44:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:28 smithi115 ceph-mon[35364]: pgmap v7775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:28 smithi181 ceph-mon[35869]: pgmap v7775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:28 smithi097 ceph-mon[32231]: pgmap v7775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:30 smithi115 ceph-mon[35364]: pgmap v7776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:30 smithi181 ceph-mon[35869]: pgmap v7776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:30 smithi097 ceph-mon[32231]: pgmap v7776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13507.977039 secs 2024-07-26T11:44:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13507.977039 secs 2024-07-26T11:44:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13507.977039 secs 2024-07-26T11:44:32.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:32 smithi115 ceph-mon[35364]: pgmap v7777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:32.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:32 smithi181 ceph-mon[35869]: pgmap v7777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:32 smithi097 ceph-mon[32231]: pgmap v7777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:34 smithi115 ceph-mon[35364]: pgmap v7778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:34 smithi181 ceph-mon[35869]: pgmap v7778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:34.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:34 smithi097 ceph-mon[32231]: pgmap v7778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:36 smithi115 ceph-mon[35364]: pgmap v7779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13512.977183 secs 2024-07-26T11:44:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:36 smithi181 ceph-mon[35869]: pgmap v7779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13512.977183 secs 2024-07-26T11:44:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:36 smithi097 ceph-mon[32231]: pgmap v7779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13512.977183 secs 2024-07-26T11:44:38.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:38.248+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:44:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:38.249+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:44:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:38.249+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:44:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:38.249+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:44:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:38.249+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:38.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:38 smithi115 ceph-mon[35364]: pgmap v7780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:38 smithi181 ceph-mon[35869]: pgmap v7780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:38 smithi097 ceph-mon[32231]: pgmap v7780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:40 smithi115 ceph-mon[35364]: pgmap v7781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:40 smithi181 ceph-mon[35869]: pgmap v7781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:40 smithi097 ceph-mon[32231]: pgmap v7781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13517.977329 secs 2024-07-26T11:44:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13517.977329 secs 2024-07-26T11:44:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13517.977329 secs 2024-07-26T11:44:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:42 smithi115 ceph-mon[35364]: pgmap v7782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:42 smithi181 ceph-mon[35869]: pgmap v7782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:42 smithi097 ceph-mon[32231]: pgmap v7782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:44 smithi115 ceph-mon[35364]: pgmap v7783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:44 smithi181 ceph-mon[35869]: pgmap v7783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:44 smithi097 ceph-mon[32231]: pgmap v7783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:46 smithi115 ceph-mon[35364]: pgmap v7784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13522.977477 secs 2024-07-26T11:44:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:46 smithi181 ceph-mon[35869]: pgmap v7784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13522.977477 secs 2024-07-26T11:44:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:46 smithi097 ceph-mon[32231]: pgmap v7784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13522.977477 secs 2024-07-26T11:44:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:47 smithi115 ceph-mon[35364]: pgmap v7785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:47 smithi181 ceph-mon[35869]: pgmap v7785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:47 smithi097 ceph-mon[32231]: pgmap v7785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:49 smithi115 ceph-mon[35364]: pgmap v7786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:49 smithi181 ceph-mon[35869]: pgmap v7786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:49 smithi097 ceph-mon[32231]: pgmap v7786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13527.977624 secs 2024-07-26T11:44:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13527.977624 secs 2024-07-26T11:44:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13527.977624 secs 2024-07-26T11:44:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:51 smithi181 ceph-mon[35869]: pgmap v7787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:51 smithi097 ceph-mon[32231]: pgmap v7787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:51 smithi115 ceph-mon[35364]: pgmap v7787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:53.251+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:44:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:53.253+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:44:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:53.253+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:44:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:53.253+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:44:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:44:53.253+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:53 smithi097 ceph-mon[32231]: pgmap v7788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:53 smithi115 ceph-mon[35364]: pgmap v7788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:53 smithi181 ceph-mon[35869]: pgmap v7788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:55 smithi097 ceph-mon[32231]: pgmap v7789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:55.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13532.977778 secs 2024-07-26T11:44:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:55 smithi115 ceph-mon[35364]: pgmap v7789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:55.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13532.977778 secs 2024-07-26T11:44:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:55 smithi181 ceph-mon[35869]: pgmap v7789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13532.977778 secs 2024-07-26T11:44:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:57 smithi115 ceph-mon[35364]: pgmap v7790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:57 smithi181 ceph-mon[35869]: pgmap v7790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:57 smithi097 ceph-mon[32231]: pgmap v7790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:44:59 smithi115 ceph-mon[35364]: pgmap v7791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:44:59 smithi181 ceph-mon[35869]: pgmap v7791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:44:59 smithi097 ceph-mon[32231]: pgmap v7791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13537.977912 secs 2024-07-26T11:45:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13537.977912 secs 2024-07-26T11:45:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13537.977912 secs 2024-07-26T11:45:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:01 smithi115 ceph-mon[35364]: pgmap v7792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:01 smithi181 ceph-mon[35869]: pgmap v7792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:01 smithi097 ceph-mon[32231]: pgmap v7792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:03.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:03 smithi181 ceph-mon[35869]: pgmap v7793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:03 smithi115 ceph-mon[35364]: pgmap v7793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:03 smithi097 ceph-mon[32231]: pgmap v7793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:05 smithi115 ceph-mon[35364]: pgmap v7794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13542.978055 secs 2024-07-26T11:45:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:05 smithi181 ceph-mon[35869]: pgmap v7794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13542.978055 secs 2024-07-26T11:45:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:05 smithi097 ceph-mon[32231]: pgmap v7794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13542.978055 secs 2024-07-26T11:45:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:07 smithi115 ceph-mon[35364]: pgmap v7795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:07 smithi181 ceph-mon[35869]: pgmap v7795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:07 smithi097 ceph-mon[32231]: pgmap v7795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:08.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:08.260+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:45:08.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:08.262+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:45:08.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:08.262+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:45:08.263 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:08.262+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:45:08.264 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:08.262+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:09.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:09 smithi097 ceph-mon[32231]: pgmap v7796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:09 smithi115 ceph-mon[35364]: pgmap v7796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:09 smithi181 ceph-mon[35869]: pgmap v7796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13547.978207 secs 2024-07-26T11:45:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13547.978207 secs 2024-07-26T11:45:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13547.978207 secs 2024-07-26T11:45:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:11 smithi115 ceph-mon[35364]: pgmap v7797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:11 smithi181 ceph-mon[35869]: pgmap v7797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:11 smithi097 ceph-mon[32231]: pgmap v7797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:13 smithi115 ceph-mon[35364]: pgmap v7798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:13 smithi181 ceph-mon[35869]: pgmap v7798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:13 smithi097 ceph-mon[32231]: pgmap v7798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:15 smithi115 ceph-mon[35364]: pgmap v7799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13552.978350 secs 2024-07-26T11:45:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:15 smithi181 ceph-mon[35869]: pgmap v7799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13552.978350 secs 2024-07-26T11:45:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:15 smithi097 ceph-mon[32231]: pgmap v7799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:15.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13552.978350 secs 2024-07-26T11:45:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:17 smithi115 ceph-mon[35364]: pgmap v7800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:17 smithi181 ceph-mon[35869]: pgmap v7800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:17 smithi097 ceph-mon[32231]: pgmap v7800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:19 smithi115 ceph-mon[35364]: pgmap v7801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:19 smithi181 ceph-mon[35869]: pgmap v7801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:19 smithi097 ceph-mon[32231]: pgmap v7801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13557.978482 secs 2024-07-26T11:45:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13557.978482 secs 2024-07-26T11:45:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13557.978482 secs 2024-07-26T11:45:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:21 smithi115 ceph-mon[35364]: pgmap v7802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:45:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:21.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:21 smithi181 ceph-mon[35869]: pgmap v7802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:45:21.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:21 smithi097 ceph-mon[32231]: pgmap v7802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:45:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:45:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: pgmap v7803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: pgmap v7804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:45:22.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:45:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:45:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:45:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:22.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: pgmap v7803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:22.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: pgmap v7804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:45:22.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:45:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: pgmap v7803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: pgmap v7804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:45:22.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:45:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:23.263+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:45:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:23.264+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:45:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:23.264+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:45:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:23.264+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:45:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:23.264+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:24 smithi115 ceph-mon[35364]: pgmap v7805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:45:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:24 smithi181 ceph-mon[35869]: pgmap v7805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:45:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:24 smithi097 ceph-mon[32231]: pgmap v7805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:45:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13562.978580 secs 2024-07-26T11:45:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13562.978580 secs 2024-07-26T11:45:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13562.978580 secs 2024-07-26T11:45:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:26 smithi115 ceph-mon[35364]: pgmap v7806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:45:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:26 smithi181 ceph-mon[35869]: pgmap v7806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:45:27.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:26 smithi097 ceph-mon[32231]: pgmap v7806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:45:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:28 smithi115 ceph-mon[35364]: pgmap v7807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:28 smithi181 ceph-mon[35869]: pgmap v7807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:28 smithi097 ceph-mon[32231]: pgmap v7807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:30 smithi115 ceph-mon[35364]: pgmap v7808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13567.978719 secs 2024-07-26T11:45:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:30 smithi181 ceph-mon[35869]: pgmap v7808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13567.978719 secs 2024-07-26T11:45:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:30 smithi097 ceph-mon[32231]: pgmap v7808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13567.978719 secs 2024-07-26T11:45:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:32 smithi115 ceph-mon[35364]: pgmap v7809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:32 smithi181 ceph-mon[35869]: pgmap v7809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:32 smithi097 ceph-mon[32231]: pgmap v7809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:34 smithi115 ceph-mon[35364]: pgmap v7810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:34 smithi181 ceph-mon[35869]: pgmap v7810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:34 smithi097 ceph-mon[32231]: pgmap v7810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13572.978816 secs 2024-07-26T11:45:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13572.978816 secs 2024-07-26T11:45:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13572.978816 secs 2024-07-26T11:45:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:36 smithi181 ceph-mon[35869]: pgmap v7811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:36 smithi097 ceph-mon[32231]: pgmap v7811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:36 smithi115 ceph-mon[35364]: pgmap v7811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:38.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:38.271+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:45:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:38.272+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:45:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:38.272+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:45:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:38.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:45:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:38.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:38 smithi181 ceph-mon[35869]: pgmap v7812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:38 smithi097 ceph-mon[32231]: pgmap v7812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:38 smithi115 ceph-mon[35364]: pgmap v7812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:40 smithi097 ceph-mon[32231]: pgmap v7813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13577.978903 secs 2024-07-26T11:45:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:40 smithi115 ceph-mon[35364]: pgmap v7813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13577.978903 secs 2024-07-26T11:45:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:40 smithi181 ceph-mon[35869]: pgmap v7813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13577.978903 secs 2024-07-26T11:45:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:42 smithi097 ceph-mon[32231]: pgmap v7814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:42 smithi115 ceph-mon[35364]: pgmap v7814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:42 smithi181 ceph-mon[35869]: pgmap v7814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:45.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:44 smithi097 ceph-mon[32231]: pgmap v7815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:44 smithi115 ceph-mon[35364]: pgmap v7815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:44 smithi181 ceph-mon[35869]: pgmap v7815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13582.979042 secs 2024-07-26T11:45:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13582.979042 secs 2024-07-26T11:45:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13582.979042 secs 2024-07-26T11:45:47.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:46 smithi115 ceph-mon[35364]: pgmap v7816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:46 smithi181 ceph-mon[35869]: pgmap v7816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:46 smithi097 ceph-mon[32231]: pgmap v7816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:48 smithi115 ceph-mon[35364]: pgmap v7817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:48 smithi181 ceph-mon[35869]: pgmap v7817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:48 smithi097 ceph-mon[32231]: pgmap v7817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:50 smithi115 ceph-mon[35364]: pgmap v7818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13587.979173 secs 2024-07-26T11:45:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:50 smithi181 ceph-mon[35869]: pgmap v7818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13587.979173 secs 2024-07-26T11:45:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:50 smithi097 ceph-mon[32231]: pgmap v7818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13587.979173 secs 2024-07-26T11:45:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:52 smithi115 ceph-mon[35364]: pgmap v7819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:52 smithi181 ceph-mon[35869]: pgmap v7819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:52 smithi097 ceph-mon[32231]: pgmap v7819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:53.274+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:45:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:53.275+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:45:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:53.275+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:45:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:53.275+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:45:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:45:53.275+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:55.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:54 smithi115 ceph-mon[35364]: pgmap v7820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:54 smithi181 ceph-mon[35869]: pgmap v7820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:54 smithi097 ceph-mon[32231]: pgmap v7820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:56.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13592.979308 secs 2024-07-26T11:45:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13592.979308 secs 2024-07-26T11:45:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13592.979308 secs 2024-07-26T11:45:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:56 smithi115 ceph-mon[35364]: pgmap v7821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:56 smithi181 ceph-mon[35869]: pgmap v7821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:56 smithi097 ceph-mon[32231]: pgmap v7821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:45:58 smithi115 ceph-mon[35364]: pgmap v7822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:45:58 smithi181 ceph-mon[35869]: pgmap v7822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:45:58 smithi097 ceph-mon[32231]: pgmap v7822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:00 smithi115 ceph-mon[35364]: pgmap v7823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13597.979447 secs 2024-07-26T11:46:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:00 smithi181 ceph-mon[35869]: pgmap v7823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13597.979447 secs 2024-07-26T11:46:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:00 smithi097 ceph-mon[32231]: pgmap v7823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13597.979447 secs 2024-07-26T11:46:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:02 smithi115 ceph-mon[35364]: pgmap v7824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:02 smithi181 ceph-mon[35869]: pgmap v7824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:02 smithi097 ceph-mon[32231]: pgmap v7824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:04 smithi115 ceph-mon[35364]: pgmap v7825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:04 smithi181 ceph-mon[35869]: pgmap v7825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:04 smithi097 ceph-mon[32231]: pgmap v7825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13602.979591 secs 2024-07-26T11:46:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13602.979591 secs 2024-07-26T11:46:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13602.979591 secs 2024-07-26T11:46:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:06 smithi115 ceph-mon[35364]: pgmap v7826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:06 smithi181 ceph-mon[35869]: pgmap v7826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:06 smithi097 ceph-mon[32231]: pgmap v7826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:08.282+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:46:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:08.283+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:46:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:08.283+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:46:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:08.283+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:46:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:08.283+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:08 smithi115 ceph-mon[35364]: pgmap v7827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:08 smithi181 ceph-mon[35869]: pgmap v7827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:08 smithi097 ceph-mon[32231]: pgmap v7827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:09.308+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc0036c0 con 0x55d25070b2f0 2024-07-26T11:46:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:09.308+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:46:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:09.308+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc0036c0 con 0x55d25070b2f0 2024-07-26T11:46:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:09.309+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:46:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:10 smithi115 ceph-mon[35364]: pgmap v7828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13607.979732 secs 2024-07-26T11:46:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:10 smithi181 ceph-mon[35869]: pgmap v7828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13607.979732 secs 2024-07-26T11:46:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:10 smithi097 ceph-mon[32231]: pgmap v7828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13607.979732 secs 2024-07-26T11:46:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:12 smithi115 ceph-mon[35364]: pgmap v7829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:12 smithi181 ceph-mon[35869]: pgmap v7829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:12 smithi097 ceph-mon[32231]: pgmap v7829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:15.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:14 smithi115 ceph-mon[35364]: pgmap v7830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:14 smithi181 ceph-mon[35869]: pgmap v7830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:14 smithi097 ceph-mon[32231]: pgmap v7830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13612.979876 secs 2024-07-26T11:46:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13612.979876 secs 2024-07-26T11:46:16.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13612.979876 secs 2024-07-26T11:46:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:16 smithi115 ceph-mon[35364]: pgmap v7831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:16 smithi181 ceph-mon[35869]: pgmap v7831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:16 smithi097 ceph-mon[32231]: pgmap v7831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:19.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:18 smithi115 ceph-mon[35364]: pgmap v7832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:18 smithi181 ceph-mon[35869]: pgmap v7832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:18 smithi097 ceph-mon[32231]: pgmap v7832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:20 smithi115 ceph-mon[35364]: pgmap v7833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13617.979977 secs 2024-07-26T11:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:20 smithi181 ceph-mon[35869]: pgmap v7833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13617.979977 secs 2024-07-26T11:46:21.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:20 smithi097 ceph-mon[32231]: pgmap v7833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:21.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13617.979977 secs 2024-07-26T11:46:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:22 smithi115 ceph-mon[35364]: pgmap v7834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:23.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:22 smithi181 ceph-mon[35869]: pgmap v7834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:22 smithi097 ceph-mon[32231]: pgmap v7834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:23.283+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:46:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:23.285+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:46:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:23.285+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:46:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:23.285+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:46:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:23.285+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:24.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:24.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:24.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:24.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:46:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:24 smithi115 ceph-mon[35364]: pgmap v7835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:24 smithi115 ceph-mon[35364]: pgmap v7836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:24 smithi181 ceph-mon[35869]: pgmap v7835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:24 smithi181 ceph-mon[35869]: pgmap v7836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:24 smithi097 ceph-mon[32231]: pgmap v7835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:24 smithi097 ceph-mon[32231]: pgmap v7836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13622.980123 secs 2024-07-26T11:46:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13622.980123 secs 2024-07-26T11:46:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13622.980123 secs 2024-07-26T11:46:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:26 smithi115 ceph-mon[35364]: pgmap v7837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:26 smithi181 ceph-mon[35869]: pgmap v7837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:26 smithi097 ceph-mon[32231]: pgmap v7837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:28 smithi115 ceph-mon[35364]: pgmap v7838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:28 smithi181 ceph-mon[35869]: pgmap v7838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:28 smithi097 ceph-mon[32231]: pgmap v7838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:30 smithi115 ceph-mon[35364]: pgmap v7839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13627.980285 secs 2024-07-26T11:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:30 smithi181 ceph-mon[35869]: pgmap v7839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13627.980285 secs 2024-07-26T11:46:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:30 smithi097 ceph-mon[32231]: pgmap v7839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13627.980285 secs 2024-07-26T11:46:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:32 smithi115 ceph-mon[35364]: pgmap v7840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:32 smithi181 ceph-mon[35869]: pgmap v7840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:32 smithi097 ceph-mon[32231]: pgmap v7840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:46:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:34 smithi181 ceph-mon[35869]: pgmap v7841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:46:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:34 smithi097 ceph-mon[32231]: pgmap v7841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:46:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:34 smithi115 ceph-mon[35364]: pgmap v7841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:46:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13632.980395 secs 2024-07-26T11:46:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13632.980395 secs 2024-07-26T11:46:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13632.980395 secs 2024-07-26T11:46:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:36 smithi097 ceph-mon[32231]: pgmap v7842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:36 smithi115 ceph-mon[35364]: pgmap v7842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:36 smithi181 ceph-mon[35869]: pgmap v7842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:38.294 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:38.292+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:46:38.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:38.293+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:46:38.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:38.293+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:46:38.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:38.293+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:46:38.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:38.293+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:38 smithi097 ceph-mon[32231]: pgmap v7843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:38 smithi115 ceph-mon[35364]: pgmap v7843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:38 smithi181 ceph-mon[35869]: pgmap v7843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:41 smithi115 ceph-mon[35364]: pgmap v7844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13637.980526 secs 2024-07-26T11:46:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:41 smithi181 ceph-mon[35869]: pgmap v7844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13637.980526 secs 2024-07-26T11:46:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:41 smithi097 ceph-mon[32231]: pgmap v7844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13637.980526 secs 2024-07-26T11:46:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:43 smithi115 ceph-mon[35364]: pgmap v7845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:43 smithi181 ceph-mon[35869]: pgmap v7845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:43 smithi097 ceph-mon[32231]: pgmap v7845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:45 smithi115 ceph-mon[35364]: pgmap v7846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:45 smithi181 ceph-mon[35869]: pgmap v7846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:45 smithi097 ceph-mon[32231]: pgmap v7846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13642.980669 secs 2024-07-26T11:46:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13642.980669 secs 2024-07-26T11:46:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13642.980669 secs 2024-07-26T11:46:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:47 smithi115 ceph-mon[35364]: pgmap v7847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:47 smithi181 ceph-mon[35869]: pgmap v7847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:47 smithi097 ceph-mon[32231]: pgmap v7847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:49 smithi115 ceph-mon[35364]: pgmap v7848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:49 smithi181 ceph-mon[35869]: pgmap v7848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:49 smithi097 ceph-mon[32231]: pgmap v7848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:51 smithi115 ceph-mon[35364]: pgmap v7849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13647.980841 secs 2024-07-26T11:46:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:51 smithi181 ceph-mon[35869]: pgmap v7849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13647.980841 secs 2024-07-26T11:46:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:51 smithi097 ceph-mon[32231]: pgmap v7849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13647.980841 secs 2024-07-26T11:46:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:53.297+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:46:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:53.298+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:46:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:53.298+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:46:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:53.298+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:46:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:46:53.298+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:53 smithi115 ceph-mon[35364]: pgmap v7850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:53 smithi181 ceph-mon[35869]: pgmap v7850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:53 smithi097 ceph-mon[32231]: pgmap v7850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:55 smithi115 ceph-mon[35364]: pgmap v7851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:55 smithi181 ceph-mon[35869]: pgmap v7851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:55 smithi097 ceph-mon[32231]: pgmap v7851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13652.980984 secs 2024-07-26T11:46:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13652.980984 secs 2024-07-26T11:46:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13652.980984 secs 2024-07-26T11:46:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:57 smithi115 ceph-mon[35364]: pgmap v7852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:57 smithi181 ceph-mon[35869]: pgmap v7852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:57 smithi097 ceph-mon[32231]: pgmap v7852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:59.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:46:59 smithi115 ceph-mon[35364]: pgmap v7853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:46:59 smithi181 ceph-mon[35869]: pgmap v7853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:46:59 smithi097 ceph-mon[32231]: pgmap v7853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:01 smithi115 ceph-mon[35364]: pgmap v7854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13657.981122 secs 2024-07-26T11:47:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:01 smithi181 ceph-mon[35869]: pgmap v7854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13657.981122 secs 2024-07-26T11:47:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:01 smithi097 ceph-mon[32231]: pgmap v7854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13657.981122 secs 2024-07-26T11:47:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:03 smithi115 ceph-mon[35364]: pgmap v7855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:03 smithi181 ceph-mon[35869]: pgmap v7855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:03 smithi097 ceph-mon[32231]: pgmap v7855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:05 smithi115 ceph-mon[35364]: pgmap v7856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:05 smithi181 ceph-mon[35869]: pgmap v7856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:05 smithi097 ceph-mon[32231]: pgmap v7856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13662.981301 secs 2024-07-26T11:47:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13662.981301 secs 2024-07-26T11:47:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13662.981301 secs 2024-07-26T11:47:07.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:07 smithi115 ceph-mon[35364]: pgmap v7857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:07 smithi181 ceph-mon[35869]: pgmap v7857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:07.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:07 smithi097 ceph-mon[32231]: pgmap v7857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:08.300+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:47:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:08.300+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:47:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:08.300+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:47:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:08.300+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:47:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:08.300+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:09 smithi115 ceph-mon[35364]: pgmap v7858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:09 smithi181 ceph-mon[35869]: pgmap v7858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:09 smithi097 ceph-mon[32231]: pgmap v7858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:11 smithi115 ceph-mon[35364]: pgmap v7859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13667.981409 secs 2024-07-26T11:47:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:11 smithi181 ceph-mon[35869]: pgmap v7859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13667.981409 secs 2024-07-26T11:47:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:11 smithi097 ceph-mon[32231]: pgmap v7859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13667.981409 secs 2024-07-26T11:47:12.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:12 smithi115 ceph-mon[35364]: pgmap v7860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:12 smithi181 ceph-mon[35869]: pgmap v7860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:12 smithi097 ceph-mon[32231]: pgmap v7860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:14 smithi115 ceph-mon[35364]: pgmap v7861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:14 smithi181 ceph-mon[35869]: pgmap v7861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:14 smithi097 ceph-mon[32231]: pgmap v7861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13672.981546 secs 2024-07-26T11:47:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13672.981546 secs 2024-07-26T11:47:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13672.981546 secs 2024-07-26T11:47:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:16 smithi115 ceph-mon[35364]: pgmap v7862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:16 smithi181 ceph-mon[35869]: pgmap v7862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:16 smithi097 ceph-mon[32231]: pgmap v7862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:18.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:18 smithi115 ceph-mon[35364]: pgmap v7863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:18 smithi181 ceph-mon[35869]: pgmap v7863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:18 smithi097 ceph-mon[32231]: pgmap v7863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:20 smithi115 ceph-mon[35364]: pgmap v7864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13677.981684 secs 2024-07-26T11:47:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:20 smithi181 ceph-mon[35869]: pgmap v7864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13677.981684 secs 2024-07-26T11:47:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:20 smithi097 ceph-mon[32231]: pgmap v7864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13677.981684 secs 2024-07-26T11:47:22.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:22 smithi115 ceph-mon[35364]: pgmap v7865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:22 smithi181 ceph-mon[35869]: pgmap v7865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:22 smithi097 ceph-mon[32231]: pgmap v7865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:23.304 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:23.302+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:47:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:23.303+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:47:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:23.303+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:47:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:23.303+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:47:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:23.303+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:24 smithi115 ceph-mon[35364]: pgmap v7866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:24 smithi181 ceph-mon[35869]: pgmap v7866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:24 smithi097 ceph-mon[32231]: pgmap v7866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:25.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13682.981818 secs 2024-07-26T11:47:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13682.981818 secs 2024-07-26T11:47:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13682.981818 secs 2024-07-26T11:47:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:26.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:47:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:26 smithi115 ceph-mon[35364]: pgmap v7867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:26 smithi115 ceph-mon[35364]: pgmap v7868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:26 smithi181 ceph-mon[35869]: pgmap v7867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:26 smithi181 ceph-mon[35869]: pgmap v7868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:26 smithi097 ceph-mon[32231]: pgmap v7867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:26 smithi097 ceph-mon[32231]: pgmap v7868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:28 smithi115 ceph-mon[35364]: pgmap v7869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:28 smithi181 ceph-mon[35869]: pgmap v7869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:28 smithi097 ceph-mon[32231]: pgmap v7869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:30 smithi115 ceph-mon[35364]: pgmap v7870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13687.981900 secs 2024-07-26T11:47:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:30 smithi181 ceph-mon[35869]: pgmap v7870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13687.981900 secs 2024-07-26T11:47:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:30 smithi097 ceph-mon[32231]: pgmap v7870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13687.981900 secs 2024-07-26T11:47:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:32 smithi181 ceph-mon[35869]: pgmap v7871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:32 smithi097 ceph-mon[32231]: pgmap v7871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:32 smithi115 ceph-mon[35364]: pgmap v7871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:34 smithi097 ceph-mon[32231]: pgmap v7872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:35.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:34 smithi115 ceph-mon[35364]: pgmap v7872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:34 smithi181 ceph-mon[35869]: pgmap v7872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:47:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13692.982040 secs 2024-07-26T11:47:36.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13692.982040 secs 2024-07-26T11:47:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13692.982040 secs 2024-07-26T11:47:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:36 smithi097 ceph-mon[32231]: pgmap v7873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:36 smithi115 ceph-mon[35364]: pgmap v7873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:36 smithi181 ceph-mon[35869]: pgmap v7873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:38.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:38.310+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:47:38.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:38.312+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:47:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:38.312+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:47:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:38.312+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:47:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:38.312+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:38 smithi115 ceph-mon[35364]: pgmap v7874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:38 smithi181 ceph-mon[35869]: pgmap v7874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:38 smithi097 ceph-mon[32231]: pgmap v7874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:41.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:40 smithi097 ceph-mon[32231]: pgmap v7875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:41.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13697.982213 secs 2024-07-26T11:47:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:40 smithi115 ceph-mon[35364]: pgmap v7875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13697.982213 secs 2024-07-26T11:47:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:40 smithi181 ceph-mon[35869]: pgmap v7875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13697.982213 secs 2024-07-26T11:47:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:42 smithi115 ceph-mon[35364]: pgmap v7876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:42 smithi181 ceph-mon[35869]: pgmap v7876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:42 smithi097 ceph-mon[32231]: pgmap v7876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:44 smithi115 ceph-mon[35364]: pgmap v7877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:44 smithi181 ceph-mon[35869]: pgmap v7877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:44 smithi097 ceph-mon[32231]: pgmap v7877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13702.982296 secs 2024-07-26T11:47:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13702.982296 secs 2024-07-26T11:47:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13702.982296 secs 2024-07-26T11:47:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:46 smithi115 ceph-mon[35364]: pgmap v7878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:46 smithi181 ceph-mon[35869]: pgmap v7878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:46 smithi097 ceph-mon[32231]: pgmap v7878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:49.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:48 smithi115 ceph-mon[35364]: pgmap v7879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:48 smithi181 ceph-mon[35869]: pgmap v7879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:48 smithi097 ceph-mon[32231]: pgmap v7879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:50 smithi115 ceph-mon[35364]: pgmap v7880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13707.982457 secs 2024-07-26T11:47:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:50 smithi181 ceph-mon[35869]: pgmap v7880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13707.982457 secs 2024-07-26T11:47:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:50 smithi097 ceph-mon[32231]: pgmap v7880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13707.982457 secs 2024-07-26T11:47:53.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:52 smithi115 ceph-mon[35364]: pgmap v7881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:52 smithi181 ceph-mon[35869]: pgmap v7881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:52 smithi097 ceph-mon[32231]: pgmap v7881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:53.317 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:53.315+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:47:53.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:53.316+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:47:53.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:53.316+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:47:53.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:53.316+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:47:53.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:47:53.316+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:54 smithi115 ceph-mon[35364]: pgmap v7882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:54 smithi181 ceph-mon[35869]: pgmap v7882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:54 smithi097 ceph-mon[32231]: pgmap v7882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13712.982609 secs 2024-07-26T11:47:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13712.982609 secs 2024-07-26T11:47:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13712.982609 secs 2024-07-26T11:47:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:56 smithi115 ceph-mon[35364]: pgmap v7883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:56 smithi181 ceph-mon[35869]: pgmap v7883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:56 smithi097 ceph-mon[32231]: pgmap v7883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:59.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:47:58 smithi115 ceph-mon[35364]: pgmap v7884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:47:58 smithi181 ceph-mon[35869]: pgmap v7884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:47:58 smithi097 ceph-mon[32231]: pgmap v7884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:00 smithi115 ceph-mon[35364]: pgmap v7885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13717.982749 secs 2024-07-26T11:48:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:00 smithi181 ceph-mon[35869]: pgmap v7885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13717.982749 secs 2024-07-26T11:48:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:00 smithi097 ceph-mon[32231]: pgmap v7885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13717.982749 secs 2024-07-26T11:48:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:02 smithi115 ceph-mon[35364]: pgmap v7886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:02 smithi181 ceph-mon[35869]: pgmap v7886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:02 smithi097 ceph-mon[32231]: pgmap v7886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:04 smithi115 ceph-mon[35364]: pgmap v7887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:04 smithi181 ceph-mon[35869]: pgmap v7887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:04 smithi097 ceph-mon[32231]: pgmap v7887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:06.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13722.982893 secs 2024-07-26T11:48:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13722.982893 secs 2024-07-26T11:48:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13722.982893 secs 2024-07-26T11:48:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:06 smithi115 ceph-mon[35364]: pgmap v7888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:06 smithi181 ceph-mon[35869]: pgmap v7888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:07.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:06 smithi097 ceph-mon[32231]: pgmap v7888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:08.321+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:48:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:08.322+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:48:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:08.322+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:48:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:08.322+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:48:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:08.323+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:09.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:08 smithi115 ceph-mon[35364]: pgmap v7889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:08 smithi181 ceph-mon[35869]: pgmap v7889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:08 smithi097 ceph-mon[32231]: pgmap v7889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:10 smithi115 ceph-mon[35364]: pgmap v7890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13727.983047 secs 2024-07-26T11:48:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:10 smithi181 ceph-mon[35869]: pgmap v7890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13727.983047 secs 2024-07-26T11:48:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:10 smithi097 ceph-mon[32231]: pgmap v7890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13727.983047 secs 2024-07-26T11:48:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:12 smithi115 ceph-mon[35364]: pgmap v7891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:12 smithi181 ceph-mon[35869]: pgmap v7891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:12 smithi097 ceph-mon[32231]: pgmap v7891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:14 smithi115 ceph-mon[35364]: pgmap v7892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:14 smithi181 ceph-mon[35869]: pgmap v7892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:14 smithi097 ceph-mon[32231]: pgmap v7892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13732.983197 secs 2024-07-26T11:48:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13732.983197 secs 2024-07-26T11:48:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13732.983197 secs 2024-07-26T11:48:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:16 smithi115 ceph-mon[35364]: pgmap v7893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:16 smithi181 ceph-mon[35869]: pgmap v7893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:16 smithi097 ceph-mon[32231]: pgmap v7893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:18 smithi115 ceph-mon[35364]: pgmap v7894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:18 smithi181 ceph-mon[35869]: pgmap v7894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:18 smithi097 ceph-mon[32231]: pgmap v7894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:20 smithi115 ceph-mon[35364]: pgmap v7895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13737.983336 secs 2024-07-26T11:48:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:20 smithi181 ceph-mon[35869]: pgmap v7895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13737.983336 secs 2024-07-26T11:48:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:20 smithi097 ceph-mon[32231]: pgmap v7895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13737.983336 secs 2024-07-26T11:48:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:22 smithi115 ceph-mon[35364]: pgmap v7896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:22 smithi181 ceph-mon[35869]: pgmap v7896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:22 smithi097 ceph-mon[32231]: pgmap v7896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:23.325+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:48:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:23.326+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:48:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:23.326+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:48:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:23.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:48:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:23.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:24 smithi115 ceph-mon[35364]: pgmap v7897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:24 smithi181 ceph-mon[35869]: pgmap v7897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:24 smithi097 ceph-mon[32231]: pgmap v7897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13742.983486 secs 2024-07-26T11:48:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13742.983486 secs 2024-07-26T11:48:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13742.983486 secs 2024-07-26T11:48:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:26 smithi181 ceph-mon[35869]: pgmap v7898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:26 smithi097 ceph-mon[32231]: pgmap v7898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:26 smithi115 ceph-mon[35364]: pgmap v7898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:28 smithi097 ceph-mon[32231]: pgmap v7899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:29.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:28 smithi115 ceph-mon[35364]: pgmap v7899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:29.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:28 smithi181 ceph-mon[35869]: pgmap v7899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:30.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:30 smithi097 ceph-mon[32231]: pgmap v7900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13747.983575 secs 2024-07-26T11:48:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:30 smithi115 ceph-mon[35364]: pgmap v7900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13747.983575 secs 2024-07-26T11:48:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:30 smithi181 ceph-mon[35869]: pgmap v7900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13747.983575 secs 2024-07-26T11:48:33.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:32 smithi097 ceph-mon[32231]: pgmap v7901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:32 smithi115 ceph-mon[35364]: pgmap v7901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:32 smithi181 ceph-mon[35869]: pgmap v7901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:35.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:35 smithi097 ceph-mon[32231]: pgmap v7902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:35.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:35 smithi115 ceph-mon[35364]: pgmap v7902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:35 smithi181 ceph-mon[35869]: pgmap v7902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13752.983723 secs 2024-07-26T11:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:36 smithi115 ceph-mon[35364]: pgmap v7903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:36 smithi115 ceph-mon[35364]: pgmap v7904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:36.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13752.983723 secs 2024-07-26T11:48:36.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:36.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:36.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:36.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:36.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:36 smithi181 ceph-mon[35869]: pgmap v7903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:48:36.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:36 smithi181 ceph-mon[35869]: pgmap v7904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:36.386 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13752.983723 secs 2024-07-26T11:48:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:36 smithi097 ceph-mon[32231]: pgmap v7903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:48:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:36 smithi097 ceph-mon[32231]: pgmap v7904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:36.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:48:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:38.334+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:38.335+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:38.335+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:38.335+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:38.335+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:38 smithi115 ceph-mon[35364]: pgmap v7905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:38 smithi181 ceph-mon[35869]: pgmap v7905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:38 smithi097 ceph-mon[32231]: pgmap v7905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:40 smithi115 ceph-mon[35364]: pgmap v7906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:40 smithi181 ceph-mon[35869]: pgmap v7906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:40 smithi097 ceph-mon[32231]: pgmap v7906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:41.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13757.983861 secs 2024-07-26T11:48:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13757.983861 secs 2024-07-26T11:48:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13757.983861 secs 2024-07-26T11:48:42.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:42 smithi097 ceph-mon[32231]: pgmap v7907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:42 smithi115 ceph-mon[35364]: pgmap v7907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:42 smithi181 ceph-mon[35869]: pgmap v7907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:44 smithi115 ceph-mon[35364]: pgmap v7908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:44 smithi181 ceph-mon[35869]: pgmap v7908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:44 smithi097 ceph-mon[32231]: pgmap v7908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:46.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13762.984005 secs 2024-07-26T11:48:46.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:46 smithi115 ceph-mon[35364]: pgmap v7909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13762.984005 secs 2024-07-26T11:48:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:46 smithi181 ceph-mon[35869]: pgmap v7909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13762.984005 secs 2024-07-26T11:48:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:46 smithi097 ceph-mon[32231]: pgmap v7909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:48 smithi115 ceph-mon[35364]: pgmap v7910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:48 smithi181 ceph-mon[35869]: pgmap v7910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:48 smithi097 ceph-mon[32231]: pgmap v7910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:50 smithi115 ceph-mon[35364]: pgmap v7911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:50 smithi181 ceph-mon[35869]: pgmap v7911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:50 smithi097 ceph-mon[32231]: pgmap v7911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13767.984167 secs 2024-07-26T11:48:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13767.984167 secs 2024-07-26T11:48:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13767.984167 secs 2024-07-26T11:48:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:52 smithi115 ceph-mon[35364]: pgmap v7912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:52.479 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:52 smithi097 ceph-mon[32231]: pgmap v7912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:52 smithi181 ceph-mon[35869]: pgmap v7912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:53.337+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:48:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:53.338+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:48:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:53.338+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:48:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:53.338+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:48:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:48:53.339+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:53 smithi115 ceph-mon[35364]: pgmap v7913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:53 smithi181 ceph-mon[35869]: pgmap v7913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:53 smithi097 ceph-mon[32231]: pgmap v7913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13772.984331 secs 2024-07-26T11:48:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:56 smithi115 ceph-mon[35364]: pgmap v7914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13772.984331 secs 2024-07-26T11:48:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:56 smithi181 ceph-mon[35869]: pgmap v7914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13772.984331 secs 2024-07-26T11:48:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:56 smithi097 ceph-mon[32231]: pgmap v7914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:48:58 smithi115 ceph-mon[35364]: pgmap v7915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:48:58 smithi181 ceph-mon[35869]: pgmap v7915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:48:58 smithi097 ceph-mon[32231]: pgmap v7915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:00 smithi115 ceph-mon[35364]: pgmap v7916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:00 smithi181 ceph-mon[35869]: pgmap v7916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:00 smithi097 ceph-mon[32231]: pgmap v7916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13777.984460 secs 2024-07-26T11:49:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13777.984460 secs 2024-07-26T11:49:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13777.984460 secs 2024-07-26T11:49:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:02 smithi115 ceph-mon[35364]: pgmap v7917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:02.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:02 smithi181 ceph-mon[35869]: pgmap v7917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:02 smithi097 ceph-mon[32231]: pgmap v7917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:04 smithi115 ceph-mon[35364]: pgmap v7918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:04 smithi181 ceph-mon[35869]: pgmap v7918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:04 smithi097 ceph-mon[32231]: pgmap v7918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13782.984604 secs 2024-07-26T11:49:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:06 smithi115 ceph-mon[35364]: pgmap v7919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13782.984604 secs 2024-07-26T11:49:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:06 smithi181 ceph-mon[35869]: pgmap v7919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13782.984604 secs 2024-07-26T11:49:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:06 smithi097 ceph-mon[32231]: pgmap v7919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:07 smithi115 ceph-mon[35364]: pgmap v7920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:07 smithi181 ceph-mon[35869]: pgmap v7920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:07 smithi097 ceph-mon[32231]: pgmap v7920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:08.345+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:49:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:08.346+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:49:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:08.346+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:49:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:08.346+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:49:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:08.347+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:09.340+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:49:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:09.340+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:49:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:09.341+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T11:49:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:09.341+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:49:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:09.341+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T11:49:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:09.341+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:49:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:09.342+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0006220 con 0x7ff3f01061a0 2024-07-26T11:49:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:09.342+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:49:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:10 smithi115 ceph-mon[35364]: pgmap v7921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:10 smithi181 ceph-mon[35869]: pgmap v7921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:10 smithi097 ceph-mon[32231]: pgmap v7921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:11.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13787.984746 secs 2024-07-26T11:49:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13787.984746 secs 2024-07-26T11:49:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13787.984746 secs 2024-07-26T11:49:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:12 smithi115 ceph-mon[35364]: pgmap v7922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:12 smithi181 ceph-mon[35869]: pgmap v7922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:12.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:12 smithi097 ceph-mon[32231]: pgmap v7922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:14 smithi115 ceph-mon[35364]: pgmap v7923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:14 smithi181 ceph-mon[35869]: pgmap v7923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:14 smithi097 ceph-mon[32231]: pgmap v7923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13792.984901 secs 2024-07-26T11:49:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:16 smithi115 ceph-mon[35364]: pgmap v7924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:49:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13792.984901 secs 2024-07-26T11:49:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:16 smithi181 ceph-mon[35869]: pgmap v7924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:49:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13792.984901 secs 2024-07-26T11:49:16.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:16 smithi097 ceph-mon[32231]: pgmap v7924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:49:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:18 smithi115 ceph-mon[35364]: pgmap v7925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:18 smithi181 ceph-mon[35869]: pgmap v7925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:18 smithi097 ceph-mon[32231]: pgmap v7925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:20 smithi115 ceph-mon[35364]: pgmap v7926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:49:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:20 smithi181 ceph-mon[35869]: pgmap v7926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:49:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:20 smithi097 ceph-mon[32231]: pgmap v7926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:49:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13797.985040 secs 2024-07-26T11:49:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:21 smithi115 ceph-mon[35364]: pgmap v7927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13797.985040 secs 2024-07-26T11:49:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:21 smithi181 ceph-mon[35869]: pgmap v7927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13797.985040 secs 2024-07-26T11:49:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:21 smithi097 ceph-mon[32231]: pgmap v7927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:23.351 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:23.349+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:49:23.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:23.350+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:49:23.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:23.351+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:49:23.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:23.351+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:49:23.353 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:23.351+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:24 smithi115 ceph-mon[35364]: pgmap v7928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:49:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:24 smithi181 ceph-mon[35869]: pgmap v7928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:49:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:24 smithi097 ceph-mon[32231]: pgmap v7928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:49:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13802.985184 secs 2024-07-26T11:49:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:26 smithi115 ceph-mon[35364]: pgmap v7929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:49:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13802.985184 secs 2024-07-26T11:49:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:26 smithi181 ceph-mon[35869]: pgmap v7929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:49:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13802.985184 secs 2024-07-26T11:49:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:26 smithi097 ceph-mon[32231]: pgmap v7929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:49:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:28 smithi115 ceph-mon[35364]: pgmap v7930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:28 smithi181 ceph-mon[35869]: pgmap v7930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:28 smithi097 ceph-mon[32231]: pgmap v7930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:30 smithi115 ceph-mon[35364]: pgmap v7931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:30 smithi181 ceph-mon[35869]: pgmap v7931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:30 smithi097 ceph-mon[32231]: pgmap v7931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13807.985323 secs 2024-07-26T11:49:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13807.985323 secs 2024-07-26T11:49:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13807.985323 secs 2024-07-26T11:49:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:32 smithi115 ceph-mon[35364]: pgmap v7932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:32 smithi181 ceph-mon[35869]: pgmap v7932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:32 smithi097 ceph-mon[32231]: pgmap v7932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:33 smithi115 ceph-mon[35364]: pgmap v7933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:33 smithi181 ceph-mon[35869]: pgmap v7933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:33 smithi097 ceph-mon[32231]: pgmap v7933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13812.985463 secs 2024-07-26T11:49:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:36 smithi115 ceph-mon[35364]: pgmap v7934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13812.985463 secs 2024-07-26T11:49:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:36 smithi181 ceph-mon[35869]: pgmap v7934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13812.985463 secs 2024-07-26T11:49:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:36 smithi097 ceph-mon[32231]: pgmap v7934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:38.274 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:38 smithi181 ceph-mon[35869]: pgmap v7935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:38.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:38.355+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:49:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:38.356+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:49:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:38.356+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:49:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:38.356+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:49:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:38.356+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:38 smithi115 ceph-mon[35364]: pgmap v7935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:38 smithi097 ceph-mon[32231]: pgmap v7935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:40.361 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:40 smithi181 ceph-mon[35869]: pgmap v7936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:40.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:40 smithi115 ceph-mon[35364]: pgmap v7936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:40.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:40 smithi097 ceph-mon[32231]: pgmap v7936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13817.985571 secs 2024-07-26T11:49:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13817.985571 secs 2024-07-26T11:49:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13817.985571 secs 2024-07-26T11:49:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:42 smithi115 ceph-mon[35364]: pgmap v7937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:42 smithi181 ceph-mon[35869]: pgmap v7937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:42 smithi097 ceph-mon[32231]: pgmap v7937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:42.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:49:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:43 smithi115 ceph-mon[35364]: pgmap v7938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:49:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:43 smithi115 ceph-mon[35364]: pgmap v7939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:49:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:43 smithi181 ceph-mon[35869]: pgmap v7938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:49:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:43 smithi181 ceph-mon[35869]: pgmap v7939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:49:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:43 smithi097 ceph-mon[32231]: pgmap v7938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:49:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:43 smithi097 ceph-mon[32231]: pgmap v7939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:49:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:44 smithi115 ceph-mon[35364]: pgmap v7940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:44 smithi181 ceph-mon[35869]: pgmap v7940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:44 smithi097 ceph-mon[32231]: pgmap v7940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13822.985716 secs 2024-07-26T11:49:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13822.985716 secs 2024-07-26T11:49:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13822.985716 secs 2024-07-26T11:49:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:46 smithi115 ceph-mon[35364]: pgmap v7941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:49:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:46 smithi181 ceph-mon[35869]: pgmap v7941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:49:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:46 smithi097 ceph-mon[32231]: pgmap v7941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:49:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:48 smithi115 ceph-mon[35364]: pgmap v7942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:49:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:48 smithi181 ceph-mon[35869]: pgmap v7942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:49:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:48 smithi097 ceph-mon[32231]: pgmap v7942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T11:49:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:50 smithi115 ceph-mon[35364]: pgmap v7943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:49:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13827.985855 secs 2024-07-26T11:49:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:50 smithi181 ceph-mon[35869]: pgmap v7943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:49:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13827.985855 secs 2024-07-26T11:49:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:50 smithi097 ceph-mon[32231]: pgmap v7943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:49:50.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13827.985855 secs 2024-07-26T11:49:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:52 smithi115 ceph-mon[35364]: pgmap v7944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:52 smithi181 ceph-mon[35869]: pgmap v7944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:52 smithi097 ceph-mon[32231]: pgmap v7944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:53.358+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:49:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:53.359+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:49:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:53.359+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:49:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:53.359+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:49:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:49:53.359+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:54 smithi115 ceph-mon[35364]: pgmap v7945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:54 smithi181 ceph-mon[35869]: pgmap v7945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:54 smithi097 ceph-mon[32231]: pgmap v7945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:49:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13832.986003 secs 2024-07-26T11:49:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13832.986003 secs 2024-07-26T11:49:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13832.986003 secs 2024-07-26T11:49:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:56 smithi115 ceph-mon[35364]: pgmap v7946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:56 smithi181 ceph-mon[35869]: pgmap v7946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:56 smithi097 ceph-mon[32231]: pgmap v7946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:49:58 smithi115 ceph-mon[35364]: pgmap v7947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:49:58 smithi181 ceph-mon[35869]: pgmap v7947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:49:58 smithi097 ceph-mon[32231]: pgmap v7947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:00 smithi115 ceph-mon[35364]: pgmap v7948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13837.986159 secs 2024-07-26T11:50:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:00 smithi181 ceph-mon[35869]: pgmap v7948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13837.986159 secs 2024-07-26T11:50:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:00 smithi097 ceph-mon[32231]: pgmap v7948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13837.986159 secs 2024-07-26T11:50:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:02 smithi115 ceph-mon[35364]: pgmap v7949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:02 smithi181 ceph-mon[35869]: pgmap v7949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:02 smithi097 ceph-mon[32231]: pgmap v7949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:04 smithi115 ceph-mon[35364]: pgmap v7950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:04 smithi181 ceph-mon[35869]: pgmap v7950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:04 smithi097 ceph-mon[32231]: pgmap v7950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13842.986323 secs 2024-07-26T11:50:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13842.986323 secs 2024-07-26T11:50:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13842.986323 secs 2024-07-26T11:50:06.681 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:06 smithi181 ceph-mon[35869]: pgmap v7951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:06 smithi115 ceph-mon[35364]: pgmap v7951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:06 smithi097 ceph-mon[32231]: pgmap v7951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:08.364+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:08.366+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:50:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:08.366+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:50:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:08.366+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:08.366+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:08 smithi181 ceph-mon[35869]: pgmap v7952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:08 smithi115 ceph-mon[35364]: pgmap v7952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:08 smithi097 ceph-mon[32231]: pgmap v7952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:10 smithi115 ceph-mon[35364]: pgmap v7953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13847.986437 secs 2024-07-26T11:50:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:10 smithi181 ceph-mon[35869]: pgmap v7953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13847.986437 secs 2024-07-26T11:50:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:10 smithi097 ceph-mon[32231]: pgmap v7953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13847.986437 secs 2024-07-26T11:50:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:12 smithi115 ceph-mon[35364]: pgmap v7954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:12 smithi181 ceph-mon[35869]: pgmap v7954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:12 smithi097 ceph-mon[32231]: pgmap v7954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:14 smithi115 ceph-mon[35364]: pgmap v7955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:14 smithi181 ceph-mon[35869]: pgmap v7955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:14 smithi097 ceph-mon[32231]: pgmap v7955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13852.986542 secs 2024-07-26T11:50:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13852.986542 secs 2024-07-26T11:50:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13852.986542 secs 2024-07-26T11:50:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:16 smithi181 ceph-mon[35869]: pgmap v7956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:16 smithi097 ceph-mon[32231]: pgmap v7956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:16 smithi115 ceph-mon[35364]: pgmap v7956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:18 smithi181 ceph-mon[35869]: pgmap v7957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:18 smithi097 ceph-mon[32231]: pgmap v7957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:18 smithi115 ceph-mon[35364]: pgmap v7957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:20 smithi097 ceph-mon[32231]: pgmap v7958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:50:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13857.986685 secs 2024-07-26T11:50:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:20 smithi115 ceph-mon[35364]: pgmap v7958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:50:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13857.986685 secs 2024-07-26T11:50:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:20 smithi181 ceph-mon[35869]: pgmap v7958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:50:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13857.986685 secs 2024-07-26T11:50:22.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:22 smithi097 ceph-mon[32231]: pgmap v7959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:22 smithi115 ceph-mon[35364]: pgmap v7959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:22 smithi181 ceph-mon[35869]: pgmap v7959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:23.368+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:23.369+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:50:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:23.369+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:50:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:23.369+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:23.369+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:24 smithi115 ceph-mon[35364]: pgmap v7960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:50:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:24 smithi181 ceph-mon[35869]: pgmap v7960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:50:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:24 smithi097 ceph-mon[32231]: pgmap v7960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:50:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13862.986779 secs 2024-07-26T11:50:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13862.986779 secs 2024-07-26T11:50:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13862.986779 secs 2024-07-26T11:50:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:26 smithi115 ceph-mon[35364]: pgmap v7961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:26 smithi181 ceph-mon[35869]: pgmap v7961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:26 smithi097 ceph-mon[32231]: pgmap v7961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:28 smithi115 ceph-mon[35364]: pgmap v7962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:50:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:28 smithi181 ceph-mon[35869]: pgmap v7962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:50:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:28 smithi097 ceph-mon[32231]: pgmap v7962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:50:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:30 smithi115 ceph-mon[35364]: pgmap v7963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:50:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13867.986917 secs 2024-07-26T11:50:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:30 smithi181 ceph-mon[35869]: pgmap v7963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:50:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13867.986917 secs 2024-07-26T11:50:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:30 smithi097 ceph-mon[32231]: pgmap v7963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:50:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13867.986917 secs 2024-07-26T11:50:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:32 smithi115 ceph-mon[35364]: pgmap v7964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:32 smithi181 ceph-mon[35869]: pgmap v7964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:32 smithi097 ceph-mon[32231]: pgmap v7964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:34.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:34 smithi115 ceph-mon[35364]: pgmap v7965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:34 smithi181 ceph-mon[35869]: pgmap v7965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:34 smithi097 ceph-mon[32231]: pgmap v7965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13872.987061 secs 2024-07-26T11:50:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13872.987061 secs 2024-07-26T11:50:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13872.987061 secs 2024-07-26T11:50:36.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:36 smithi115 ceph-mon[35364]: pgmap v7966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:36.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:36 smithi181 ceph-mon[35869]: pgmap v7966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:36 smithi097 ceph-mon[32231]: pgmap v7966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:38.376+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:38.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:50:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:38.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:50:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:38.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:38.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:38 smithi115 ceph-mon[35364]: pgmap v7967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:38 smithi181 ceph-mon[35869]: pgmap v7967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:38 smithi097 ceph-mon[32231]: pgmap v7967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:40 smithi115 ceph-mon[35364]: pgmap v7968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:50:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13877.987187 secs 2024-07-26T11:50:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:40 smithi181 ceph-mon[35869]: pgmap v7968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:50:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13877.987187 secs 2024-07-26T11:50:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:40 smithi097 ceph-mon[32231]: pgmap v7968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:50:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13877.987187 secs 2024-07-26T11:50:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:42 smithi115 ceph-mon[35364]: pgmap v7969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:50:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:42 smithi181 ceph-mon[35869]: pgmap v7969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:50:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:42 smithi097 ceph-mon[32231]: pgmap v7969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:50:43.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:43 smithi115 ceph-mon[35364]: pgmap v7970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:43 smithi115 ceph-mon[35364]: pgmap v7971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:50:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:43 smithi181 ceph-mon[35869]: pgmap v7970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:43 smithi181 ceph-mon[35869]: pgmap v7971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:50:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:43 smithi097 ceph-mon[32231]: pgmap v7970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:43 smithi097 ceph-mon[32231]: pgmap v7971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:50:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:45 smithi115 ceph-mon[35364]: pgmap v7972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13882.987349 secs 2024-07-26T11:50:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:45 smithi181 ceph-mon[35869]: pgmap v7972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13882.987349 secs 2024-07-26T11:50:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:45 smithi097 ceph-mon[32231]: pgmap v7972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13882.987349 secs 2024-07-26T11:50:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:47 smithi115 ceph-mon[35364]: pgmap v7973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:47 smithi181 ceph-mon[35869]: pgmap v7973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:47 smithi097 ceph-mon[32231]: pgmap v7973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:50.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:49 smithi115 ceph-mon[35364]: pgmap v7974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:49 smithi181 ceph-mon[35869]: pgmap v7974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:49 smithi097 ceph-mon[32231]: pgmap v7974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13887.987490 secs 2024-07-26T11:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13887.987490 secs 2024-07-26T11:50:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13887.987490 secs 2024-07-26T11:50:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:51 smithi115 ceph-mon[35364]: pgmap v7975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:51 smithi181 ceph-mon[35869]: pgmap v7975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:51 smithi097 ceph-mon[32231]: pgmap v7975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:53.379+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:50:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:53.380+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:50:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:53.380+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:50:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:53.380+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:50:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:50:53.380+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:53 smithi115 ceph-mon[35364]: pgmap v7976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:53 smithi181 ceph-mon[35869]: pgmap v7976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:53 smithi097 ceph-mon[32231]: pgmap v7976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:55 smithi115 ceph-mon[35364]: pgmap v7977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13892.987635 secs 2024-07-26T11:50:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:55 smithi181 ceph-mon[35869]: pgmap v7977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13892.987635 secs 2024-07-26T11:50:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:55 smithi097 ceph-mon[32231]: pgmap v7977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13892.987635 secs 2024-07-26T11:50:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:57 smithi115 ceph-mon[35364]: pgmap v7978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:57 smithi181 ceph-mon[35869]: pgmap v7978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:57 smithi097 ceph-mon[32231]: pgmap v7978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:50:59 smithi115 ceph-mon[35364]: pgmap v7979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:50:59 smithi181 ceph-mon[35869]: pgmap v7979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:50:59 smithi097 ceph-mon[32231]: pgmap v7979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13897.987734 secs 2024-07-26T11:51:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13897.987734 secs 2024-07-26T11:51:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13897.987734 secs 2024-07-26T11:51:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:01 smithi115 ceph-mon[35364]: pgmap v7980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:01 smithi181 ceph-mon[35869]: pgmap v7980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:01 smithi097 ceph-mon[32231]: pgmap v7980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:03 smithi115 ceph-mon[35364]: pgmap v7981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:03 smithi181 ceph-mon[35869]: pgmap v7981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:03 smithi097 ceph-mon[32231]: pgmap v7981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:05 smithi115 ceph-mon[35364]: pgmap v7982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13902.987877 secs 2024-07-26T11:51:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:05 smithi181 ceph-mon[35869]: pgmap v7982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13902.987877 secs 2024-07-26T11:51:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:05 smithi097 ceph-mon[32231]: pgmap v7982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13902.987877 secs 2024-07-26T11:51:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:07 smithi115 ceph-mon[35364]: pgmap v7983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:07 smithi181 ceph-mon[35869]: pgmap v7983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:07 smithi097 ceph-mon[32231]: pgmap v7983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:08.386+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:51:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:08.387+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:51:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:08.387+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:51:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:08.387+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:51:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:08.387+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:09 smithi115 ceph-mon[35364]: pgmap v7984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:09 smithi181 ceph-mon[35869]: pgmap v7984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:09 smithi097 ceph-mon[32231]: pgmap v7984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13907.988018 secs 2024-07-26T11:51:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13907.988018 secs 2024-07-26T11:51:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13907.988018 secs 2024-07-26T11:51:12.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:11 smithi115 ceph-mon[35364]: pgmap v7985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:11 smithi181 ceph-mon[35869]: pgmap v7985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:11 smithi097 ceph-mon[32231]: pgmap v7985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:13 smithi115 ceph-mon[35364]: pgmap v7986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:13 smithi181 ceph-mon[35869]: pgmap v7986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:13 smithi097 ceph-mon[32231]: pgmap v7986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:15 smithi115 ceph-mon[35364]: pgmap v7987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13912.988114 secs 2024-07-26T11:51:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:15 smithi181 ceph-mon[35869]: pgmap v7987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13912.988114 secs 2024-07-26T11:51:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:15 smithi097 ceph-mon[32231]: pgmap v7987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13912.988114 secs 2024-07-26T11:51:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:17 smithi115 ceph-mon[35364]: pgmap v7988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:17 smithi181 ceph-mon[35869]: pgmap v7988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:17 smithi097 ceph-mon[32231]: pgmap v7988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:19 smithi115 ceph-mon[35364]: pgmap v7989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:19 smithi181 ceph-mon[35869]: pgmap v7989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:19 smithi097 ceph-mon[32231]: pgmap v7989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13917.988301 secs 2024-07-26T11:51:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13917.988301 secs 2024-07-26T11:51:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13917.988301 secs 2024-07-26T11:51:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:21 smithi115 ceph-mon[35364]: pgmap v7990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:21 smithi181 ceph-mon[35869]: pgmap v7990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:21 smithi097 ceph-mon[32231]: pgmap v7990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:23.389+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:51:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:23.390+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:51:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:23.390+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:51:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:23.390+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:51:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:23.391+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:23 smithi181 ceph-mon[35869]: pgmap v7991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:23 smithi097 ceph-mon[32231]: pgmap v7991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:23 smithi115 ceph-mon[35364]: pgmap v7991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:25 smithi097 ceph-mon[32231]: pgmap v7992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:26.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13922.988458 secs 2024-07-26T11:51:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:25 smithi115 ceph-mon[35364]: pgmap v7992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13922.988458 secs 2024-07-26T11:51:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:25 smithi181 ceph-mon[35869]: pgmap v7992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13922.988458 secs 2024-07-26T11:51:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:27 smithi097 ceph-mon[32231]: pgmap v7993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:27 smithi115 ceph-mon[35364]: pgmap v7993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:27 smithi181 ceph-mon[35869]: pgmap v7993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:30.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:29 smithi097 ceph-mon[32231]: pgmap v7994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:29 smithi115 ceph-mon[35364]: pgmap v7994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:29 smithi181 ceph-mon[35869]: pgmap v7994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13927.988591 secs 2024-07-26T11:51:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13927.988591 secs 2024-07-26T11:51:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13927.988591 secs 2024-07-26T11:51:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:32 smithi115 ceph-mon[35364]: pgmap v7995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:32 smithi181 ceph-mon[35869]: pgmap v7995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:32 smithi097 ceph-mon[32231]: pgmap v7995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:34 smithi115 ceph-mon[35364]: pgmap v7996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:34 smithi181 ceph-mon[35869]: pgmap v7996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:34 smithi097 ceph-mon[32231]: pgmap v7996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:36.279 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:36 smithi115 ceph-mon[35364]: pgmap v7997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:36.279 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13932.988739 secs 2024-07-26T11:51:36.285 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:36 smithi181 ceph-mon[35869]: pgmap v7997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:36.285 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13932.988739 secs 2024-07-26T11:51:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:36 smithi097 ceph-mon[32231]: pgmap v7997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:36.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13932.988739 secs 2024-07-26T11:51:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:38.398+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:51:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:38.400+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:51:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:38.400+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:51:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:38.400+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:51:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:38.400+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:38 smithi115 ceph-mon[35364]: pgmap v7998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:38 smithi181 ceph-mon[35869]: pgmap v7998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:38 smithi097 ceph-mon[32231]: pgmap v7998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:40 smithi115 ceph-mon[35364]: pgmap v7999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:40 smithi181 ceph-mon[35869]: pgmap v7999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:40 smithi097 ceph-mon[32231]: pgmap v7999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13937.988880 secs 2024-07-26T11:51:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13937.988880 secs 2024-07-26T11:51:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13937.988880 secs 2024-07-26T11:51:42.404 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:42 smithi097 ceph-mon[32231]: pgmap v8000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:42 smithi115 ceph-mon[35364]: pgmap v8000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:42 smithi181 ceph-mon[35869]: pgmap v8000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:44 smithi115 ceph-mon[35364]: pgmap v8001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:44.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:44 smithi181 ceph-mon[35869]: pgmap v8001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:44 smithi097 ceph-mon[32231]: pgmap v8001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:51:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:51:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:45.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:51:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:46 smithi115 ceph-mon[35364]: pgmap v8002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:51:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:46 smithi115 ceph-mon[35364]: pgmap v8003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13942.989039 secs 2024-07-26T11:51:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:46 smithi181 ceph-mon[35869]: pgmap v8002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:51:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:46 smithi181 ceph-mon[35869]: pgmap v8003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13942.989039 secs 2024-07-26T11:51:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:46 smithi097 ceph-mon[32231]: pgmap v8002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:51:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:46 smithi097 ceph-mon[32231]: pgmap v8003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13942.989039 secs 2024-07-26T11:51:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:48 smithi115 ceph-mon[35364]: pgmap v8004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:48 smithi181 ceph-mon[35869]: pgmap v8004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:48 smithi097 ceph-mon[32231]: pgmap v8004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:50 smithi115 ceph-mon[35364]: pgmap v8005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:50 smithi181 ceph-mon[35869]: pgmap v8005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:50 smithi097 ceph-mon[32231]: pgmap v8005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:51:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13947.989182 secs 2024-07-26T11:51:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13947.989182 secs 2024-07-26T11:51:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13947.989182 secs 2024-07-26T11:51:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:52 smithi115 ceph-mon[35364]: pgmap v8006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:52 smithi181 ceph-mon[35869]: pgmap v8006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:52 smithi097 ceph-mon[32231]: pgmap v8006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:53.401+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:51:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:53.402+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:51:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:53.402+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:51:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:53.402+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:51:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:51:53.402+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:54 smithi115 ceph-mon[35364]: pgmap v8007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:51:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:54 smithi181 ceph-mon[35869]: pgmap v8007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:51:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:54 smithi097 ceph-mon[32231]: pgmap v8007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:51:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:56 smithi115 ceph-mon[35364]: pgmap v8008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13952.989316 secs 2024-07-26T11:51:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:56 smithi181 ceph-mon[35869]: pgmap v8008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13952.989316 secs 2024-07-26T11:51:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:56 smithi097 ceph-mon[32231]: pgmap v8008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13952.989316 secs 2024-07-26T11:51:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:51:58 smithi115 ceph-mon[35364]: pgmap v8009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:51:58 smithi181 ceph-mon[35869]: pgmap v8009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:51:58 smithi097 ceph-mon[32231]: pgmap v8009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:00 smithi115 ceph-mon[35364]: pgmap v8010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:00 smithi181 ceph-mon[35869]: pgmap v8010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:00 smithi097 ceph-mon[32231]: pgmap v8010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13957.989456 secs 2024-07-26T11:52:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13957.989456 secs 2024-07-26T11:52:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13957.989456 secs 2024-07-26T11:52:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:02 smithi115 ceph-mon[35364]: pgmap v8011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:02 smithi181 ceph-mon[35869]: pgmap v8011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:02 smithi097 ceph-mon[32231]: pgmap v8011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:04 smithi115 ceph-mon[35364]: pgmap v8012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:04 smithi181 ceph-mon[35869]: pgmap v8012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:04 smithi097 ceph-mon[32231]: pgmap v8012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:06 smithi115 ceph-mon[35364]: pgmap v8013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13962.989555 secs 2024-07-26T11:52:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:06 smithi181 ceph-mon[35869]: pgmap v8013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13962.989555 secs 2024-07-26T11:52:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:06 smithi097 ceph-mon[32231]: pgmap v8013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13962.989555 secs 2024-07-26T11:52:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:08.409+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:52:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:08.410+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:52:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:08.410+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:52:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:08.410+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:52:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:08.410+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:08 smithi115 ceph-mon[35364]: pgmap v8014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:08 smithi181 ceph-mon[35869]: pgmap v8014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:08 smithi097 ceph-mon[32231]: pgmap v8014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:09.374+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:52:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:09.374+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:52:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:09.375+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:52:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:09.375+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:52:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:10 smithi115 ceph-mon[35364]: pgmap v8015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:10 smithi181 ceph-mon[35869]: pgmap v8015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:10 smithi097 ceph-mon[32231]: pgmap v8015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13967.989696 secs 2024-07-26T11:52:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13967.989696 secs 2024-07-26T11:52:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13967.989696 secs 2024-07-26T11:52:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:12 smithi115 ceph-mon[35364]: pgmap v8016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:12 smithi181 ceph-mon[35869]: pgmap v8016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:12 smithi097 ceph-mon[32231]: pgmap v8016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:13 smithi115 ceph-mon[35364]: pgmap v8017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:13 smithi181 ceph-mon[35869]: pgmap v8017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:13 smithi097 ceph-mon[32231]: pgmap v8017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:15 smithi115 ceph-mon[35364]: pgmap v8018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:15 smithi181 ceph-mon[35869]: pgmap v8018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:15 smithi097 ceph-mon[32231]: pgmap v8018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13972.989825 secs 2024-07-26T11:52:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13972.989825 secs 2024-07-26T11:52:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13972.989825 secs 2024-07-26T11:52:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:17 smithi115 ceph-mon[35364]: pgmap v8019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:17 smithi181 ceph-mon[35869]: pgmap v8019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:17 smithi097 ceph-mon[32231]: pgmap v8019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:19 smithi115 ceph-mon[35364]: pgmap v8020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:19 smithi181 ceph-mon[35869]: pgmap v8020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:19 smithi097 ceph-mon[32231]: pgmap v8020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13977.989964 secs 2024-07-26T11:52:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13977.989964 secs 2024-07-26T11:52:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13977.989964 secs 2024-07-26T11:52:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:21 smithi115 ceph-mon[35364]: pgmap v8021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:21 smithi181 ceph-mon[35869]: pgmap v8021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:21 smithi097 ceph-mon[32231]: pgmap v8021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:23.413+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:52:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:23.414+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:52:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:23.414+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:52:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:23.414+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:52:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:23.414+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:23 smithi115 ceph-mon[35364]: pgmap v8022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:23 smithi181 ceph-mon[35869]: pgmap v8022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:23 smithi097 ceph-mon[32231]: pgmap v8022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:25 smithi115 ceph-mon[35364]: pgmap v8023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13982.990121 secs 2024-07-26T11:52:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:25 smithi181 ceph-mon[35869]: pgmap v8023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13982.990121 secs 2024-07-26T11:52:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:25 smithi097 ceph-mon[32231]: pgmap v8023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13982.990121 secs 2024-07-26T11:52:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:27 smithi115 ceph-mon[35364]: pgmap v8024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:27 smithi181 ceph-mon[35869]: pgmap v8024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:27 smithi097 ceph-mon[32231]: pgmap v8024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:29 smithi115 ceph-mon[35364]: pgmap v8025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:29 smithi181 ceph-mon[35869]: pgmap v8025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:29 smithi097 ceph-mon[32231]: pgmap v8025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13987.990286 secs 2024-07-26T11:52:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13987.990286 secs 2024-07-26T11:52:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13987.990286 secs 2024-07-26T11:52:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:31 smithi115 ceph-mon[35364]: pgmap v8026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:31 smithi181 ceph-mon[35869]: pgmap v8026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:31 smithi097 ceph-mon[32231]: pgmap v8026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:33 smithi115 ceph-mon[35364]: pgmap v8027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:33 smithi181 ceph-mon[35869]: pgmap v8027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:33 smithi097 ceph-mon[32231]: pgmap v8027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:35 smithi115 ceph-mon[35364]: pgmap v8028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13992.990438 secs 2024-07-26T11:52:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:35 smithi181 ceph-mon[35869]: pgmap v8028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13992.990438 secs 2024-07-26T11:52:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:35 smithi097 ceph-mon[32231]: pgmap v8028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13992.990438 secs 2024-07-26T11:52:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:37 smithi115 ceph-mon[35364]: pgmap v8029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:37 smithi181 ceph-mon[35869]: pgmap v8029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:37 smithi097 ceph-mon[32231]: pgmap v8029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:38.421+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:52:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:38.422+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:52:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:38.422+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:52:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:38.422+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:52:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:38.422+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:39 smithi115 ceph-mon[35364]: pgmap v8030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:39 smithi181 ceph-mon[35869]: pgmap v8030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:39 smithi097 ceph-mon[32231]: pgmap v8030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 13997.990579 secs 2024-07-26T11:52:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 13997.990579 secs 2024-07-26T11:52:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 13997.990579 secs 2024-07-26T11:52:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:41 smithi115 ceph-mon[35364]: pgmap v8031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:41 smithi181 ceph-mon[35869]: pgmap v8031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:41 smithi097 ceph-mon[32231]: pgmap v8031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:43.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:43 smithi115 ceph-mon[35364]: pgmap v8032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:43 smithi181 ceph-mon[35869]: pgmap v8032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:44.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:43 smithi097 ceph-mon[32231]: pgmap v8032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:45 smithi115 ceph-mon[35364]: pgmap v8033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14002.990670 secs 2024-07-26T11:52:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:45 smithi181 ceph-mon[35869]: pgmap v8033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14002.990670 secs 2024-07-26T11:52:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:45 smithi097 ceph-mon[32231]: pgmap v8033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:46.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14002.990670 secs 2024-07-26T11:52:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:46 smithi115 ceph-mon[35364]: pgmap v8034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:52:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:46 smithi115 ceph-mon[35364]: pgmap v8035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:52:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:46 smithi181 ceph-mon[35869]: pgmap v8034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:52:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:46 smithi181 ceph-mon[35869]: pgmap v8035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:52:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:46 smithi097 ceph-mon[32231]: pgmap v8034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:52:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:46 smithi097 ceph-mon[32231]: pgmap v8035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:52:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:48 smithi181 ceph-mon[35869]: pgmap v8036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:48 smithi097 ceph-mon[32231]: pgmap v8036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:48 smithi115 ceph-mon[35364]: pgmap v8036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:50 smithi097 ceph-mon[32231]: pgmap v8037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14007.990824 secs 2024-07-26T11:52:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:50 smithi115 ceph-mon[35364]: pgmap v8037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14007.990824 secs 2024-07-26T11:52:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:50 smithi181 ceph-mon[35869]: pgmap v8037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14007.990824 secs 2024-07-26T11:52:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:52 smithi097 ceph-mon[32231]: pgmap v8038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:53.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:52 smithi115 ceph-mon[35364]: pgmap v8038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:52 smithi181 ceph-mon[35869]: pgmap v8038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:53.423+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:52:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:53.424+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:52:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:53.424+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:52:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:53.424+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:52:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:52:53.424+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:54 smithi097 ceph-mon[32231]: pgmap v8039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:54 smithi115 ceph-mon[35364]: pgmap v8039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:54 smithi181 ceph-mon[35869]: pgmap v8039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14012.990961 secs 2024-07-26T11:52:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14012.990961 secs 2024-07-26T11:52:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14012.990961 secs 2024-07-26T11:52:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:56 smithi115 ceph-mon[35364]: pgmap v8040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:56 smithi181 ceph-mon[35869]: pgmap v8040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:56 smithi097 ceph-mon[32231]: pgmap v8040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:52:58 smithi115 ceph-mon[35364]: pgmap v8041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:52:58 smithi181 ceph-mon[35869]: pgmap v8041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:52:58 smithi097 ceph-mon[32231]: pgmap v8041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:00 smithi115 ceph-mon[35364]: pgmap v8042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14017.991106 secs 2024-07-26T11:53:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:00 smithi181 ceph-mon[35869]: pgmap v8042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14017.991106 secs 2024-07-26T11:53:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:00 smithi097 ceph-mon[32231]: pgmap v8042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:01.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14017.991106 secs 2024-07-26T11:53:03.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:02 smithi115 ceph-mon[35364]: pgmap v8043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:02 smithi181 ceph-mon[35869]: pgmap v8043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:02 smithi097 ceph-mon[32231]: pgmap v8043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:04 smithi115 ceph-mon[35364]: pgmap v8044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:04 smithi181 ceph-mon[35869]: pgmap v8044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:04 smithi097 ceph-mon[32231]: pgmap v8044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14022.991284 secs 2024-07-26T11:53:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14022.991284 secs 2024-07-26T11:53:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14022.991284 secs 2024-07-26T11:53:07.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:06 smithi115 ceph-mon[35364]: pgmap v8045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:07.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:06 smithi181 ceph-mon[35869]: pgmap v8045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:06 smithi097 ceph-mon[32231]: pgmap v8045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:08.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:53:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:08.429+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:53:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:08.429+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:53:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:08.429+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:53:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:08.429+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:08 smithi115 ceph-mon[35364]: pgmap v8046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:08 smithi181 ceph-mon[35869]: pgmap v8046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:08 smithi097 ceph-mon[32231]: pgmap v8046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:10 smithi115 ceph-mon[35364]: pgmap v8047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:11.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14027.991396 secs 2024-07-26T11:53:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:10 smithi181 ceph-mon[35869]: pgmap v8047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14027.991396 secs 2024-07-26T11:53:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:10 smithi097 ceph-mon[32231]: pgmap v8047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:11.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14027.991396 secs 2024-07-26T11:53:13.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:12 smithi115 ceph-mon[35364]: pgmap v8048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:12 smithi181 ceph-mon[35869]: pgmap v8048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:12 smithi097 ceph-mon[32231]: pgmap v8048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:14 smithi115 ceph-mon[35364]: pgmap v8049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:14 smithi181 ceph-mon[35869]: pgmap v8049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:15.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:14 smithi097 ceph-mon[32231]: pgmap v8049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14032.991509 secs 2024-07-26T11:53:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14032.991509 secs 2024-07-26T11:53:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14032.991509 secs 2024-07-26T11:53:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:16 smithi115 ceph-mon[35364]: pgmap v8050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:16 smithi181 ceph-mon[35869]: pgmap v8050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:16 smithi097 ceph-mon[32231]: pgmap v8050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:18 smithi115 ceph-mon[35364]: pgmap v8051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:18 smithi181 ceph-mon[35869]: pgmap v8051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:18 smithi097 ceph-mon[32231]: pgmap v8051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:20 smithi115 ceph-mon[35364]: pgmap v8052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:21.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14037.991610 secs 2024-07-26T11:53:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:20 smithi181 ceph-mon[35869]: pgmap v8052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14037.991610 secs 2024-07-26T11:53:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:20 smithi097 ceph-mon[32231]: pgmap v8052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14037.991610 secs 2024-07-26T11:53:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:22 smithi115 ceph-mon[35364]: pgmap v8053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:22 smithi181 ceph-mon[35869]: pgmap v8053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:23.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:22 smithi097 ceph-mon[32231]: pgmap v8053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:23.431+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:53:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:23.433+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:53:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:23.433+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:53:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:23.433+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:53:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:23.433+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:24 smithi115 ceph-mon[35364]: pgmap v8054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:24 smithi181 ceph-mon[35869]: pgmap v8054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:24 smithi097 ceph-mon[32231]: pgmap v8054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14042.991772 secs 2024-07-26T11:53:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14042.991772 secs 2024-07-26T11:53:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14042.991772 secs 2024-07-26T11:53:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:26 smithi115 ceph-mon[35364]: pgmap v8055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:26 smithi181 ceph-mon[35869]: pgmap v8055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:26 smithi097 ceph-mon[32231]: pgmap v8055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:29.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:28 smithi115 ceph-mon[35364]: pgmap v8056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:28 smithi181 ceph-mon[35869]: pgmap v8056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:28 smithi097 ceph-mon[32231]: pgmap v8056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:30 smithi115 ceph-mon[35364]: pgmap v8057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14047.991933 secs 2024-07-26T11:53:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:30 smithi181 ceph-mon[35869]: pgmap v8057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14047.991933 secs 2024-07-26T11:53:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:30 smithi097 ceph-mon[32231]: pgmap v8057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14047.991933 secs 2024-07-26T11:53:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:32 smithi115 ceph-mon[35364]: pgmap v8058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:32 smithi181 ceph-mon[35869]: pgmap v8058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:32 smithi097 ceph-mon[32231]: pgmap v8058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:34 smithi115 ceph-mon[35364]: pgmap v8059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:34 smithi181 ceph-mon[35869]: pgmap v8059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:34 smithi097 ceph-mon[32231]: pgmap v8059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14052.992094 secs 2024-07-26T11:53:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14052.992094 secs 2024-07-26T11:53:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14052.992094 secs 2024-07-26T11:53:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:36 smithi115 ceph-mon[35364]: pgmap v8060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:36 smithi181 ceph-mon[35869]: pgmap v8060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:36 smithi097 ceph-mon[32231]: pgmap v8060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:38.440+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:53:38.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:38.441+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:53:38.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:38.441+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:53:38.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:38.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:53:38.444 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:38.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:39.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:38 smithi115 ceph-mon[35364]: pgmap v8061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:38 smithi181 ceph-mon[35869]: pgmap v8061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:38 smithi097 ceph-mon[32231]: pgmap v8061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:40 smithi115 ceph-mon[35364]: pgmap v8062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14057.992291 secs 2024-07-26T11:53:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:40 smithi181 ceph-mon[35869]: pgmap v8062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14057.992291 secs 2024-07-26T11:53:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:40 smithi097 ceph-mon[32231]: pgmap v8062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14057.992291 secs 2024-07-26T11:53:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:42 smithi115 ceph-mon[35364]: pgmap v8063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:42 smithi181 ceph-mon[35869]: pgmap v8063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:42 smithi097 ceph-mon[32231]: pgmap v8063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:45.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:44 smithi115 ceph-mon[35364]: pgmap v8064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:44 smithi181 ceph-mon[35869]: pgmap v8064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:45.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:44 smithi097 ceph-mon[32231]: pgmap v8064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14062.992397 secs 2024-07-26T11:53:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14062.992397 secs 2024-07-26T11:53:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14062.992397 secs 2024-07-26T11:53:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:46 smithi181 ceph-mon[35869]: pgmap v8065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:46 smithi097 ceph-mon[32231]: pgmap v8065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:46 smithi115 ceph-mon[35364]: pgmap v8065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:47.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:47 smithi097 ceph-mon[32231]: pgmap v8066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:53:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:47 smithi097 ceph-mon[32231]: pgmap v8067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:48.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:53:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:47 smithi115 ceph-mon[35364]: pgmap v8066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:53:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:47 smithi115 ceph-mon[35364]: pgmap v8067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:53:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:47 smithi181 ceph-mon[35869]: pgmap v8066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:53:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:47 smithi181 ceph-mon[35869]: pgmap v8067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:53:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:50 smithi115 ceph-mon[35364]: pgmap v8068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:53:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:50 smithi181 ceph-mon[35869]: pgmap v8068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:53:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:50 smithi097 ceph-mon[32231]: pgmap v8068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:53:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14067.992571 secs 2024-07-26T11:53:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14067.992571 secs 2024-07-26T11:53:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14067.992571 secs 2024-07-26T11:53:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:52 smithi115 ceph-mon[35364]: pgmap v8069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:52 smithi181 ceph-mon[35869]: pgmap v8069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:52 smithi097 ceph-mon[32231]: pgmap v8069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:53.444+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:53:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:53.445+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:53:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:53.445+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:53:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:53.445+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:53:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:53:53.445+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:54 smithi115 ceph-mon[35364]: pgmap v8070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:53:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:54 smithi181 ceph-mon[35869]: pgmap v8070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:53:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:54 smithi097 ceph-mon[32231]: pgmap v8070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:53:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14072.992729 secs 2024-07-26T11:53:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:56 smithi115 ceph-mon[35364]: pgmap v8071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14072.992729 secs 2024-07-26T11:53:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:56 smithi181 ceph-mon[35869]: pgmap v8071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14072.992729 secs 2024-07-26T11:53:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:56 smithi097 ceph-mon[32231]: pgmap v8071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:53:58 smithi115 ceph-mon[35364]: pgmap v8072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:53:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:53:58 smithi181 ceph-mon[35869]: pgmap v8072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:53:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:53:58 smithi097 ceph-mon[32231]: pgmap v8072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:54:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:00 smithi115 ceph-mon[35364]: pgmap v8073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:00 smithi181 ceph-mon[35869]: pgmap v8073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:00 smithi097 ceph-mon[32231]: pgmap v8073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14077.992878 secs 2024-07-26T11:54:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14077.992878 secs 2024-07-26T11:54:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14077.992878 secs 2024-07-26T11:54:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:02 smithi115 ceph-mon[35364]: pgmap v8074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:02 smithi181 ceph-mon[35869]: pgmap v8074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:02 smithi097 ceph-mon[32231]: pgmap v8074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:04 smithi115 ceph-mon[35364]: pgmap v8075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:04 smithi181 ceph-mon[35869]: pgmap v8075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:04 smithi097 ceph-mon[32231]: pgmap v8075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14082.993027 secs 2024-07-26T11:54:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:06 smithi115 ceph-mon[35364]: pgmap v8076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14082.993027 secs 2024-07-26T11:54:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:06 smithi181 ceph-mon[35869]: pgmap v8076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14082.993027 secs 2024-07-26T11:54:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:06 smithi097 ceph-mon[32231]: pgmap v8076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:08.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:08.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:54:08.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:08.449+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:54:08.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:08.449+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:54:08.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:08.449+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:54:08.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:08.449+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:08 smithi115 ceph-mon[35364]: pgmap v8077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:08 smithi181 ceph-mon[35869]: pgmap v8077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:08 smithi097 ceph-mon[32231]: pgmap v8077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:10.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:10 smithi115 ceph-mon[35364]: pgmap v8078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:10 smithi181 ceph-mon[35869]: pgmap v8078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:10 smithi097 ceph-mon[32231]: pgmap v8078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14087.993173 secs 2024-07-26T11:54:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14087.993173 secs 2024-07-26T11:54:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14087.993173 secs 2024-07-26T11:54:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:12 smithi115 ceph-mon[35364]: pgmap v8079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:12 smithi181 ceph-mon[35869]: pgmap v8079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:12 smithi097 ceph-mon[32231]: pgmap v8079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:14 smithi181 ceph-mon[35869]: pgmap v8080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:14 smithi097 ceph-mon[32231]: pgmap v8080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:14 smithi115 ceph-mon[35364]: pgmap v8080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:15 smithi097 ceph-mon[32231]: pgmap v8081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:15 smithi115 ceph-mon[35364]: pgmap v8081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:15 smithi181 ceph-mon[35869]: pgmap v8081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14092.993326 secs 2024-07-26T11:54:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14092.993326 secs 2024-07-26T11:54:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14092.993326 secs 2024-07-26T11:54:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:17 smithi115 ceph-mon[35364]: pgmap v8082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:17 smithi181 ceph-mon[35869]: pgmap v8082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:17 smithi097 ceph-mon[32231]: pgmap v8082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:20 smithi115 ceph-mon[35364]: pgmap v8083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:20 smithi181 ceph-mon[35869]: pgmap v8083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:20 smithi097 ceph-mon[32231]: pgmap v8083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14097.993477 secs 2024-07-26T11:54:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14097.993477 secs 2024-07-26T11:54:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14097.993477 secs 2024-07-26T11:54:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:22 smithi115 ceph-mon[35364]: pgmap v8084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:22 smithi181 ceph-mon[35869]: pgmap v8084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:22 smithi097 ceph-mon[32231]: pgmap v8084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:23.451+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:54:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:23.452+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:54:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:23.452+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:54:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:23.452+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:54:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:23.453+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:24 smithi115 ceph-mon[35364]: pgmap v8085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:24 smithi181 ceph-mon[35869]: pgmap v8085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:24 smithi097 ceph-mon[32231]: pgmap v8085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14102.993638 secs 2024-07-26T11:54:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:26 smithi115 ceph-mon[35364]: pgmap v8086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14102.993638 secs 2024-07-26T11:54:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:26 smithi181 ceph-mon[35869]: pgmap v8086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14102.993638 secs 2024-07-26T11:54:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:26 smithi097 ceph-mon[32231]: pgmap v8086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:28 smithi115 ceph-mon[35364]: pgmap v8087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:28 smithi181 ceph-mon[35869]: pgmap v8087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:28 smithi097 ceph-mon[32231]: pgmap v8087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:30 smithi115 ceph-mon[35364]: pgmap v8088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:30 smithi181 ceph-mon[35869]: pgmap v8088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:30 smithi097 ceph-mon[32231]: pgmap v8088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14107.993788 secs 2024-07-26T11:54:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14107.993788 secs 2024-07-26T11:54:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14107.993788 secs 2024-07-26T11:54:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:32 smithi115 ceph-mon[35364]: pgmap v8089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:32 smithi181 ceph-mon[35869]: pgmap v8089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:32 smithi097 ceph-mon[32231]: pgmap v8089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:34 smithi115 ceph-mon[35364]: pgmap v8090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:54:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:34 smithi181 ceph-mon[35869]: pgmap v8090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:54:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:34 smithi097 ceph-mon[32231]: pgmap v8090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:54:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14112.993968 secs 2024-07-26T11:54:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:36 smithi115 ceph-mon[35364]: pgmap v8091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:54:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14112.993968 secs 2024-07-26T11:54:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:36 smithi181 ceph-mon[35869]: pgmap v8091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:54:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14112.993968 secs 2024-07-26T11:54:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:36 smithi097 ceph-mon[32231]: pgmap v8091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:54:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:38.459+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:54:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:38.460+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:54:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:38.461+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:54:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:38.461+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:54:38.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:38.461+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:38 smithi181 ceph-mon[35869]: pgmap v8092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:38 smithi097 ceph-mon[32231]: pgmap v8092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:38 smithi115 ceph-mon[35364]: pgmap v8092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:40 smithi097 ceph-mon[32231]: pgmap v8093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:54:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:40 smithi115 ceph-mon[35364]: pgmap v8093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:54:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:40 smithi181 ceph-mon[35869]: pgmap v8093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:54:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14117.994119 secs 2024-07-26T11:54:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:41 smithi097 ceph-mon[32231]: pgmap v8094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14117.994119 secs 2024-07-26T11:54:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:41 smithi115 ceph-mon[35364]: pgmap v8094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14117.994119 secs 2024-07-26T11:54:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:41 smithi181 ceph-mon[35869]: pgmap v8094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:44 smithi115 ceph-mon[35364]: pgmap v8095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:44 smithi181 ceph-mon[35869]: pgmap v8095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:44 smithi097 ceph-mon[32231]: pgmap v8095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14122.994283 secs 2024-07-26T11:54:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:46 smithi115 ceph-mon[35364]: pgmap v8096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14122.994283 secs 2024-07-26T11:54:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:46 smithi181 ceph-mon[35869]: pgmap v8096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14122.994283 secs 2024-07-26T11:54:46.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:46 smithi097 ceph-mon[32231]: pgmap v8096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:48 smithi115 ceph-mon[35364]: pgmap v8097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:48.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:48 smithi181 ceph-mon[35869]: pgmap v8097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:48 smithi097 ceph-mon[32231]: pgmap v8097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:49.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:54:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:49.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:49.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:54:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:50 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:54:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:50 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:54:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:50 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:54:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:50 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:54:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:50 smithi115 ceph-mon[35364]: pgmap v8098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:50.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:50 smithi115 ceph-mon[35364]: pgmap v8099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:50 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:54:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:50 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:54:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:50 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:54:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:50 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:54:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:50 smithi181 ceph-mon[35869]: pgmap v8098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:50.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:50 smithi181 ceph-mon[35869]: pgmap v8099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:50 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T11:54:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:50 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T11:54:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:50 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T11:54:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:50 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T11:54:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:50 smithi097 ceph-mon[32231]: pgmap v8098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:50.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:50 smithi097 ceph-mon[32231]: pgmap v8099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14127.994436 secs 2024-07-26T11:54:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14127.994436 secs 2024-07-26T11:54:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14127.994436 secs 2024-07-26T11:54:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:52 smithi115 ceph-mon[35364]: pgmap v8100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:52 smithi181 ceph-mon[35869]: pgmap v8100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:52 smithi097 ceph-mon[32231]: pgmap v8100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:53.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:53.463+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:54:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:53.464+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:54:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:53.464+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:54:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:53.464+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:54:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:54:53.464+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:54 smithi115 ceph-mon[35364]: pgmap v8101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:54 smithi181 ceph-mon[35869]: pgmap v8101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:54 smithi097 ceph-mon[32231]: pgmap v8101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:56 smithi115 ceph-mon[35364]: pgmap v8102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14132.994593 secs 2024-07-26T11:54:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:56 smithi181 ceph-mon[35869]: pgmap v8102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14132.994593 secs 2024-07-26T11:54:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:56 smithi097 ceph-mon[32231]: pgmap v8102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:56.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14132.994593 secs 2024-07-26T11:54:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:54:58 smithi115 ceph-mon[35364]: pgmap v8103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:54:58 smithi181 ceph-mon[35869]: pgmap v8103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:54:58 smithi097 ceph-mon[32231]: pgmap v8103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:00 smithi115 ceph-mon[35364]: pgmap v8104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:00 smithi181 ceph-mon[35869]: pgmap v8104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:00 smithi097 ceph-mon[32231]: pgmap v8104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14137.994750 secs 2024-07-26T11:55:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14137.994750 secs 2024-07-26T11:55:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14137.994750 secs 2024-07-26T11:55:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:02 smithi097 ceph-mon[32231]: pgmap v8105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:02 smithi115 ceph-mon[35364]: pgmap v8105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:02 smithi181 ceph-mon[35869]: pgmap v8105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:03 smithi115 ceph-mon[35364]: pgmap v8106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:03 smithi181 ceph-mon[35869]: pgmap v8106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:03 smithi097 ceph-mon[32231]: pgmap v8106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:05 smithi115 ceph-mon[35364]: pgmap v8107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14142.994902 secs 2024-07-26T11:55:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:05 smithi181 ceph-mon[35869]: pgmap v8107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14142.994902 secs 2024-07-26T11:55:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:05 smithi097 ceph-mon[32231]: pgmap v8107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14142.994902 secs 2024-07-26T11:55:07.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:07 smithi115 ceph-mon[35364]: pgmap v8108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:07 smithi181 ceph-mon[35869]: pgmap v8108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:07 smithi097 ceph-mon[32231]: pgmap v8108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:08.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:08.472+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:08.473+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:55:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:08.473+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:55:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:08.473+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:08.473+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:09.407+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:55:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:09.407+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:55:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:09.407+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:55:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:09.408+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:55:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:09 smithi115 ceph-mon[35364]: pgmap v8109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:09 smithi181 ceph-mon[35869]: pgmap v8109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:09 smithi097 ceph-mon[32231]: pgmap v8109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14147.995006 secs 2024-07-26T11:55:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14147.995006 secs 2024-07-26T11:55:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14147.995006 secs 2024-07-26T11:55:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:11 smithi115 ceph-mon[35364]: pgmap v8110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:11 smithi181 ceph-mon[35869]: pgmap v8110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:11 smithi097 ceph-mon[32231]: pgmap v8110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:13 smithi115 ceph-mon[35364]: pgmap v8111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:13 smithi181 ceph-mon[35869]: pgmap v8111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:13 smithi097 ceph-mon[32231]: pgmap v8111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:15 smithi115 ceph-mon[35364]: pgmap v8112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14152.995167 secs 2024-07-26T11:55:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:15 smithi181 ceph-mon[35869]: pgmap v8112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14152.995167 secs 2024-07-26T11:55:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:15 smithi097 ceph-mon[32231]: pgmap v8112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14152.995167 secs 2024-07-26T11:55:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:17 smithi115 ceph-mon[35364]: pgmap v8113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:17 smithi181 ceph-mon[35869]: pgmap v8113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:17 smithi097 ceph-mon[32231]: pgmap v8113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:19 smithi115 ceph-mon[35364]: pgmap v8114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:19 smithi181 ceph-mon[35869]: pgmap v8114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:19 smithi097 ceph-mon[32231]: pgmap v8114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14157.995320 secs 2024-07-26T11:55:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14157.995320 secs 2024-07-26T11:55:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14157.995320 secs 2024-07-26T11:55:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:21 smithi115 ceph-mon[35364]: pgmap v8115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:21 smithi181 ceph-mon[35869]: pgmap v8115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:21 smithi097 ceph-mon[32231]: pgmap v8115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:23.475+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:23.477+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:55:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:23.477+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:55:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:23.477+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:23.477+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:23 smithi115 ceph-mon[35364]: pgmap v8116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:23 smithi181 ceph-mon[35869]: pgmap v8116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:23 smithi097 ceph-mon[32231]: pgmap v8116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:25 smithi115 ceph-mon[35364]: pgmap v8117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:25.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14162.995475 secs 2024-07-26T11:55:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:25 smithi181 ceph-mon[35869]: pgmap v8117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14162.995475 secs 2024-07-26T11:55:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:25 smithi097 ceph-mon[32231]: pgmap v8117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14162.995475 secs 2024-07-26T11:55:27.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:27 smithi115 ceph-mon[35364]: pgmap v8118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:27 smithi181 ceph-mon[35869]: pgmap v8118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:27 smithi097 ceph-mon[32231]: pgmap v8118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:29 smithi115 ceph-mon[35364]: pgmap v8119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:29 smithi181 ceph-mon[35869]: pgmap v8119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:29 smithi097 ceph-mon[32231]: pgmap v8119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14167.995627 secs 2024-07-26T11:55:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14167.995627 secs 2024-07-26T11:55:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14167.995627 secs 2024-07-26T11:55:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:31 smithi115 ceph-mon[35364]: pgmap v8120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:31 smithi181 ceph-mon[35869]: pgmap v8120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:31 smithi097 ceph-mon[32231]: pgmap v8120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:33 smithi181 ceph-mon[35869]: pgmap v8121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:33 smithi097 ceph-mon[32231]: pgmap v8121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:33 smithi115 ceph-mon[35364]: pgmap v8121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:35 smithi097 ceph-mon[32231]: pgmap v8122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:55:36.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14172.995779 secs 2024-07-26T11:55:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:35 smithi115 ceph-mon[35364]: pgmap v8122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:55:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14172.995779 secs 2024-07-26T11:55:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:35 smithi181 ceph-mon[35869]: pgmap v8122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:55:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14172.995779 secs 2024-07-26T11:55:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:37 smithi097 ceph-mon[32231]: pgmap v8123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:37 smithi115 ceph-mon[35364]: pgmap v8123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:37 smithi181 ceph-mon[35869]: pgmap v8123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:38.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:38.483+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:38.485+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:55:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:38.485+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:55:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:38.485+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:38.485+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:39 smithi097 ceph-mon[32231]: pgmap v8124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:55:40.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:39 smithi115 ceph-mon[35364]: pgmap v8124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:55:40.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:39 smithi181 ceph-mon[35869]: pgmap v8124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:55:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14177.995935 secs 2024-07-26T11:55:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14177.995935 secs 2024-07-26T11:55:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14177.995935 secs 2024-07-26T11:55:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:41 smithi115 ceph-mon[35364]: pgmap v8125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:55:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:41 smithi181 ceph-mon[35869]: pgmap v8125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:55:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:41 smithi097 ceph-mon[32231]: pgmap v8125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:55:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:43 smithi115 ceph-mon[35364]: pgmap v8126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:43 smithi181 ceph-mon[35869]: pgmap v8126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:43 smithi097 ceph-mon[32231]: pgmap v8126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:46.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:45 smithi097 ceph-mon[32231]: pgmap v8127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:46.055 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14182.996077 secs 2024-07-26T11:55:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:45 smithi115 ceph-mon[35364]: pgmap v8127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14182.996077 secs 2024-07-26T11:55:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:45 smithi181 ceph-mon[35869]: pgmap v8127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14182.996077 secs 2024-07-26T11:55:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:47 smithi115 ceph-mon[35364]: pgmap v8128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:47 smithi181 ceph-mon[35869]: pgmap v8128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:47 smithi097 ceph-mon[32231]: pgmap v8128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:49 smithi115 ceph-mon[35364]: pgmap v8129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:49 smithi181 ceph-mon[35869]: pgmap v8129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:49 smithi097 ceph-mon[32231]: pgmap v8129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:50.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14187.996284 secs 2024-07-26T11:55:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:55:51.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:55:51.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:51.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:51.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: pgmap v8130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:55:51.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: pgmap v8131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:51.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:55:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14187.996284 secs 2024-07-26T11:55:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: pgmap v8130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: pgmap v8131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:55:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14187.996284 secs 2024-07-26T11:55:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:55:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T11:55:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T11:55:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:51.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:51.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: pgmap v8130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:55:51.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: pgmap v8131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:51.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:55:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:53 smithi115 ceph-mon[35364]: pgmap v8132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:53 smithi181 ceph-mon[35869]: pgmap v8132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:53.488+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:53.489+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:55:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:53.489+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:55:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:53.489+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:55:53.489+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:53 smithi097 ceph-mon[32231]: pgmap v8132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:55 smithi115 ceph-mon[35364]: pgmap v8133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:55 smithi181 ceph-mon[35869]: pgmap v8133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:55 smithi097 ceph-mon[32231]: pgmap v8133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14192.996434 secs 2024-07-26T11:55:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14192.996434 secs 2024-07-26T11:55:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14192.996434 secs 2024-07-26T11:55:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:57 smithi115 ceph-mon[35364]: pgmap v8134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:57 smithi181 ceph-mon[35869]: pgmap v8134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:57 smithi097 ceph-mon[32231]: pgmap v8134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:55:59 smithi115 ceph-mon[35364]: pgmap v8135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:55:59 smithi181 ceph-mon[35869]: pgmap v8135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:55:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:55:59 smithi097 ceph-mon[32231]: pgmap v8135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14197.996591 secs 2024-07-26T11:56:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:01 smithi115 ceph-mon[35364]: pgmap v8136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14197.996591 secs 2024-07-26T11:56:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:01 smithi181 ceph-mon[35869]: pgmap v8136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14197.996591 secs 2024-07-26T11:56:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:01 smithi097 ceph-mon[32231]: pgmap v8136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:03 smithi115 ceph-mon[35364]: pgmap v8137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:03 smithi181 ceph-mon[35869]: pgmap v8137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:03 smithi097 ceph-mon[32231]: pgmap v8137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:05.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:05 smithi115 ceph-mon[35364]: pgmap v8138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:05 smithi181 ceph-mon[35869]: pgmap v8138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:05 smithi097 ceph-mon[32231]: pgmap v8138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14202.996748 secs 2024-07-26T11:56:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14202.996748 secs 2024-07-26T11:56:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14202.996748 secs 2024-07-26T11:56:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:07 smithi115 ceph-mon[35364]: pgmap v8139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:56:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:07 smithi181 ceph-mon[35869]: pgmap v8139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:56:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:07 smithi097 ceph-mon[32231]: pgmap v8139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:56:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:08.491+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:08.492+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:56:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:08.492+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:56:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:08.492+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:08.493+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:09 smithi115 ceph-mon[35364]: pgmap v8140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:09 smithi181 ceph-mon[35869]: pgmap v8140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:09 smithi097 ceph-mon[32231]: pgmap v8140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14207.996853 secs 2024-07-26T11:56:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:11 smithi115 ceph-mon[35364]: pgmap v8141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14207.996853 secs 2024-07-26T11:56:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:11 smithi181 ceph-mon[35869]: pgmap v8141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14207.996853 secs 2024-07-26T11:56:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:11 smithi097 ceph-mon[32231]: pgmap v8141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:13 smithi115 ceph-mon[35364]: pgmap v8142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:13 smithi181 ceph-mon[35869]: pgmap v8142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:13 smithi097 ceph-mon[32231]: pgmap v8142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:15.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:15 smithi115 ceph-mon[35364]: pgmap v8143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:15 smithi181 ceph-mon[35869]: pgmap v8143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:15 smithi097 ceph-mon[32231]: pgmap v8143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14212.997005 secs 2024-07-26T11:56:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14212.997005 secs 2024-07-26T11:56:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14212.997005 secs 2024-07-26T11:56:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:17 smithi115 ceph-mon[35364]: pgmap v8144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:17 smithi181 ceph-mon[35869]: pgmap v8144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:17 smithi097 ceph-mon[32231]: pgmap v8144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:19 smithi115 ceph-mon[35364]: pgmap v8145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:19 smithi181 ceph-mon[35869]: pgmap v8145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:19 smithi097 ceph-mon[32231]: pgmap v8145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14217.997172 secs 2024-07-26T11:56:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:21 smithi115 ceph-mon[35364]: pgmap v8146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14217.997172 secs 2024-07-26T11:56:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:21 smithi181 ceph-mon[35869]: pgmap v8146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14217.997172 secs 2024-07-26T11:56:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:21 smithi097 ceph-mon[32231]: pgmap v8146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:23 smithi115 ceph-mon[35364]: pgmap v8147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:23 smithi181 ceph-mon[35869]: pgmap v8147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:23.494+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:23.496+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:56:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:23.496+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:56:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:23.496+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:23.496+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:23 smithi097 ceph-mon[32231]: pgmap v8147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:25 smithi115 ceph-mon[35364]: pgmap v8148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:25 smithi181 ceph-mon[35869]: pgmap v8148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:25 smithi097 ceph-mon[32231]: pgmap v8148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14222.997331 secs 2024-07-26T11:56:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14222.997331 secs 2024-07-26T11:56:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14222.997331 secs 2024-07-26T11:56:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:27 smithi097 ceph-mon[32231]: pgmap v8149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:27 smithi115 ceph-mon[35364]: pgmap v8149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:27 smithi181 ceph-mon[35869]: pgmap v8149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:29 smithi097 ceph-mon[32231]: pgmap v8150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:29 smithi115 ceph-mon[35364]: pgmap v8150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:29 smithi181 ceph-mon[35869]: pgmap v8150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14227.997431 secs 2024-07-26T11:56:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:31 smithi097 ceph-mon[32231]: pgmap v8151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14227.997431 secs 2024-07-26T11:56:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:31 smithi115 ceph-mon[35364]: pgmap v8151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14227.997431 secs 2024-07-26T11:56:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:31 smithi181 ceph-mon[35869]: pgmap v8151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:33 smithi115 ceph-mon[35364]: pgmap v8152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:33 smithi181 ceph-mon[35869]: pgmap v8152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:33 smithi097 ceph-mon[32231]: pgmap v8152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:34 smithi115 ceph-mon[35364]: pgmap v8153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:34 smithi181 ceph-mon[35869]: pgmap v8153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:34 smithi097 ceph-mon[32231]: pgmap v8153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14232.997533 secs 2024-07-26T11:56:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14232.997533 secs 2024-07-26T11:56:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14232.997533 secs 2024-07-26T11:56:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:36 smithi115 ceph-mon[35364]: pgmap v8154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:36 smithi181 ceph-mon[35869]: pgmap v8154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:36 smithi097 ceph-mon[32231]: pgmap v8154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:38.505 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:38.503+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:38.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:38.505+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:56:38.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:38.505+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:56:38.506 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:38.505+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:38.507 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:38.505+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:39 smithi115 ceph-mon[35364]: pgmap v8155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:39 smithi181 ceph-mon[35869]: pgmap v8155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:39 smithi097 ceph-mon[32231]: pgmap v8155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14237.997684 secs 2024-07-26T11:56:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:41 smithi115 ceph-mon[35364]: pgmap v8156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14237.997684 secs 2024-07-26T11:56:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:41 smithi181 ceph-mon[35869]: pgmap v8156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14237.997684 secs 2024-07-26T11:56:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:41 smithi097 ceph-mon[32231]: pgmap v8156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:43 smithi115 ceph-mon[35364]: pgmap v8157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:43 smithi181 ceph-mon[35869]: pgmap v8157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:43 smithi097 ceph-mon[32231]: pgmap v8157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:45 smithi115 ceph-mon[35364]: pgmap v8158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:45 smithi181 ceph-mon[35869]: pgmap v8158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:45 smithi097 ceph-mon[32231]: pgmap v8158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:46.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14242.997840 secs 2024-07-26T11:56:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14242.997840 secs 2024-07-26T11:56:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14242.997840 secs 2024-07-26T11:56:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:47 smithi115 ceph-mon[35364]: pgmap v8159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:47 smithi181 ceph-mon[35869]: pgmap v8159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:47 smithi097 ceph-mon[32231]: pgmap v8159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:49 smithi115 ceph-mon[35364]: pgmap v8160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:49 smithi181 ceph-mon[35869]: pgmap v8160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:49 smithi097 ceph-mon[32231]: pgmap v8160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14247.997993 secs 2024-07-26T11:56:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:51 smithi115 ceph-mon[35364]: pgmap v8161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14247.997993 secs 2024-07-26T11:56:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:51 smithi181 ceph-mon[35869]: pgmap v8161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14247.997993 secs 2024-07-26T11:56:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:51 smithi097 ceph-mon[32231]: pgmap v8161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:52.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:52.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:56:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:53 smithi115 ceph-mon[35364]: pgmap v8162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:56:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:53 smithi115 ceph-mon[35364]: pgmap v8163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:53 smithi181 ceph-mon[35869]: pgmap v8162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:56:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:53 smithi181 ceph-mon[35869]: pgmap v8163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:53 smithi097 ceph-mon[32231]: pgmap v8162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:56:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:53 smithi097 ceph-mon[32231]: pgmap v8163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:53.508+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:53.509+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:56:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:53.509+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:56:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:53.509+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:56:53.509+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:55 smithi115 ceph-mon[35364]: pgmap v8164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:55 smithi181 ceph-mon[35869]: pgmap v8164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:55 smithi097 ceph-mon[32231]: pgmap v8164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14252.998146 secs 2024-07-26T11:56:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14252.998146 secs 2024-07-26T11:56:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14252.998146 secs 2024-07-26T11:56:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:57 smithi115 ceph-mon[35364]: pgmap v8165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:57 smithi181 ceph-mon[35869]: pgmap v8165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:57 smithi097 ceph-mon[32231]: pgmap v8165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:56:59 smithi115 ceph-mon[35364]: pgmap v8166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:56:59 smithi181 ceph-mon[35869]: pgmap v8166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:56:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:56:59 smithi097 ceph-mon[32231]: pgmap v8166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:01.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:01 smithi115 ceph-mon[35364]: pgmap v8167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:01.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14257.998325 secs 2024-07-26T11:57:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:01 smithi181 ceph-mon[35869]: pgmap v8167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14257.998325 secs 2024-07-26T11:57:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:01 smithi097 ceph-mon[32231]: pgmap v8167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14257.998325 secs 2024-07-26T11:57:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:03 smithi115 ceph-mon[35364]: pgmap v8168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:03 smithi181 ceph-mon[35869]: pgmap v8168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:03 smithi097 ceph-mon[32231]: pgmap v8168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:05 smithi181 ceph-mon[35869]: pgmap v8169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:05 smithi097 ceph-mon[32231]: pgmap v8169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:05 smithi115 ceph-mon[35364]: pgmap v8169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14262.998436 secs 2024-07-26T11:57:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14262.998436 secs 2024-07-26T11:57:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14262.998436 secs 2024-07-26T11:57:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:07 smithi097 ceph-mon[32231]: pgmap v8170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:07 smithi115 ceph-mon[35364]: pgmap v8170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:07 smithi181 ceph-mon[35869]: pgmap v8170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:08.516+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:08.517+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:57:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:08.517+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:57:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:08.517+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:08.517+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:09 smithi097 ceph-mon[32231]: pgmap v8171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:09.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:09 smithi115 ceph-mon[35364]: pgmap v8171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:09 smithi181 ceph-mon[35869]: pgmap v8171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:11.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:11 smithi115 ceph-mon[35364]: pgmap v8172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:11.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14267.998524 secs 2024-07-26T11:57:11.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:11 smithi181 ceph-mon[35869]: pgmap v8172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:11.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14267.998524 secs 2024-07-26T11:57:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:11 smithi097 ceph-mon[32231]: pgmap v8172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14267.998524 secs 2024-07-26T11:57:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:13 smithi115 ceph-mon[35364]: pgmap v8173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:13 smithi181 ceph-mon[35869]: pgmap v8173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:13 smithi097 ceph-mon[32231]: pgmap v8173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:15 smithi115 ceph-mon[35364]: pgmap v8174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:15 smithi181 ceph-mon[35869]: pgmap v8174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:15 smithi097 ceph-mon[32231]: pgmap v8174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:16.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14272.998617 secs 2024-07-26T11:57:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14272.998617 secs 2024-07-26T11:57:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14272.998617 secs 2024-07-26T11:57:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:17 smithi115 ceph-mon[35364]: pgmap v8175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:17 smithi181 ceph-mon[35869]: pgmap v8175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:17 smithi097 ceph-mon[32231]: pgmap v8175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:18 smithi115 ceph-mon[35364]: pgmap v8176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:18.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:18 smithi181 ceph-mon[35869]: pgmap v8176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:18 smithi097 ceph-mon[32231]: pgmap v8176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:21 smithi115 ceph-mon[35364]: pgmap v8177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14277.998762 secs 2024-07-26T11:57:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:21 smithi181 ceph-mon[35869]: pgmap v8177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14277.998762 secs 2024-07-26T11:57:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:21 smithi097 ceph-mon[32231]: pgmap v8177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14277.998762 secs 2024-07-26T11:57:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:23 smithi115 ceph-mon[35364]: pgmap v8178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:23 smithi181 ceph-mon[35869]: pgmap v8178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:23 smithi097 ceph-mon[32231]: pgmap v8178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:23.518+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:23.519+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:57:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:23.519+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:57:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:23.520+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:23.520+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:24.431+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:57:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:24.431+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:57:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:24.432+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:57:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:24.432+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T11:57:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:25 smithi115 ceph-mon[35364]: pgmap v8179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:25 smithi181 ceph-mon[35869]: pgmap v8179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:25 smithi097 ceph-mon[32231]: pgmap v8179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14282.998857 secs 2024-07-26T11:57:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14282.998857 secs 2024-07-26T11:57:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14282.998857 secs 2024-07-26T11:57:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:27 smithi115 ceph-mon[35364]: pgmap v8180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:27 smithi181 ceph-mon[35869]: pgmap v8180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:27.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:27 smithi097 ceph-mon[32231]: pgmap v8180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:29 smithi115 ceph-mon[35364]: pgmap v8181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:29 smithi181 ceph-mon[35869]: pgmap v8181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:29 smithi097 ceph-mon[32231]: pgmap v8181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:31 smithi115 ceph-mon[35364]: pgmap v8182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14287.998998 secs 2024-07-26T11:57:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:31 smithi181 ceph-mon[35869]: pgmap v8182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14287.998998 secs 2024-07-26T11:57:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:31 smithi097 ceph-mon[32231]: pgmap v8182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14287.998998 secs 2024-07-26T11:57:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:33 smithi115 ceph-mon[35364]: pgmap v8183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:33 smithi181 ceph-mon[35869]: pgmap v8183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:33 smithi097 ceph-mon[32231]: pgmap v8183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:35.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:35 smithi115 ceph-mon[35364]: pgmap v8184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:35 smithi181 ceph-mon[35869]: pgmap v8184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:35 smithi097 ceph-mon[32231]: pgmap v8184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14292.999158 secs 2024-07-26T11:57:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14292.999158 secs 2024-07-26T11:57:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14292.999158 secs 2024-07-26T11:57:37.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:37 smithi115 ceph-mon[35364]: pgmap v8185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:37 smithi181 ceph-mon[35869]: pgmap v8185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:37.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:37 smithi097 ceph-mon[32231]: pgmap v8185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:38.524 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:38.522+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:38.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:38.523+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:57:38.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:38.523+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:57:38.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:38.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:38.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:38.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:39 smithi115 ceph-mon[35364]: pgmap v8186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:39 smithi181 ceph-mon[35869]: pgmap v8186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:39 smithi097 ceph-mon[32231]: pgmap v8186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:41 smithi115 ceph-mon[35364]: pgmap v8187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14297.999307 secs 2024-07-26T11:57:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:41 smithi181 ceph-mon[35869]: pgmap v8187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14297.999307 secs 2024-07-26T11:57:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:41 smithi097 ceph-mon[32231]: pgmap v8187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14297.999307 secs 2024-07-26T11:57:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:43 smithi115 ceph-mon[35364]: pgmap v8188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:43 smithi181 ceph-mon[35869]: pgmap v8188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:43 smithi097 ceph-mon[32231]: pgmap v8188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:45 smithi115 ceph-mon[35364]: pgmap v8189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:45 smithi181 ceph-mon[35869]: pgmap v8189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:45 smithi097 ceph-mon[32231]: pgmap v8189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14302.999458 secs 2024-07-26T11:57:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14302.999458 secs 2024-07-26T11:57:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14302.999458 secs 2024-07-26T11:57:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:47 smithi115 ceph-mon[35364]: pgmap v8190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:47 smithi181 ceph-mon[35869]: pgmap v8190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:47 smithi097 ceph-mon[32231]: pgmap v8190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:49 smithi115 ceph-mon[35364]: pgmap v8191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:49 smithi181 ceph-mon[35869]: pgmap v8191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:49 smithi097 ceph-mon[32231]: pgmap v8191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:51 smithi115 ceph-mon[35364]: pgmap v8192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14307.999618 secs 2024-07-26T11:57:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:51 smithi181 ceph-mon[35869]: pgmap v8192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14307.999618 secs 2024-07-26T11:57:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:51 smithi097 ceph-mon[32231]: pgmap v8192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14307.999618 secs 2024-07-26T11:57:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:53 smithi115 ceph-mon[35364]: pgmap v8193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:53 smithi181 ceph-mon[35869]: pgmap v8193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:53 smithi097 ceph-mon[32231]: pgmap v8193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:53.524+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:53.526+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:57:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:53.526+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:57:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:53.526+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:57:53.526+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:57:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:55 smithi115 ceph-mon[35364]: pgmap v8194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:55.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:55 smithi115 ceph-mon[35364]: pgmap v8195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:55 smithi181 ceph-mon[35869]: pgmap v8194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:55 smithi181 ceph-mon[35869]: pgmap v8195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:55 smithi097 ceph-mon[32231]: pgmap v8194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:55 smithi097 ceph-mon[32231]: pgmap v8195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14312.999712 secs 2024-07-26T11:57:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14312.999712 secs 2024-07-26T11:57:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14312.999712 secs 2024-07-26T11:57:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:57 smithi115 ceph-mon[35364]: pgmap v8196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:57 smithi181 ceph-mon[35869]: pgmap v8196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:57 smithi097 ceph-mon[32231]: pgmap v8196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:57:59 smithi115 ceph-mon[35364]: pgmap v8197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:57:59 smithi181 ceph-mon[35869]: pgmap v8197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:57:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:57:59 smithi097 ceph-mon[32231]: pgmap v8197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:01 smithi181 ceph-mon[35869]: pgmap v8198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14317.999859 secs 2024-07-26T11:58:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:01 smithi097 ceph-mon[32231]: pgmap v8198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14317.999859 secs 2024-07-26T11:58:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:01 smithi115 ceph-mon[35364]: pgmap v8198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14317.999859 secs 2024-07-26T11:58:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:03 smithi097 ceph-mon[32231]: pgmap v8199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:03 smithi115 ceph-mon[35364]: pgmap v8199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:03 smithi181 ceph-mon[35869]: pgmap v8199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:05.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:05 smithi097 ceph-mon[32231]: pgmap v8200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:05 smithi115 ceph-mon[35364]: pgmap v8200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:05 smithi181 ceph-mon[35869]: pgmap v8200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14323.000011 secs 2024-07-26T11:58:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14323.000011 secs 2024-07-26T11:58:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14323.000011 secs 2024-07-26T11:58:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 msgr2=0x7ff3d40b5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T11:58:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 msgr2=0x7ff3d40b5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:58:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 0x7ff3d40b5f10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057c0 tx=0x7ff3e00a4000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 0x7ff3d40b5f10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057c0 tx=0x7ff3e00a4000 comp rx=0 tx=0).stop 2024-07-26T11:58:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 msgr2=0x7ff3d40b5f10 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T11:58:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 msgr2=0x7ff3d40b5f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:58:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4650 0x7ff3d40b5f10 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:58:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40c2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:58:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.925+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40c2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:06.927+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40c2420 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006300 tx=0x7ff3e007b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:58:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:07 smithi115 ceph-mon[35364]: pgmap v8201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:07 smithi181 ceph-mon[35869]: pgmap v8201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:07 smithi097 ceph-mon[32231]: pgmap v8201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:08.530 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:08.529+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:08.529+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:58:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:08.529+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:58:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:08.529+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:08.529+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:09.440+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:58:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:09.440+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T11:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:09.440+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T11:58:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:09.441+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T11:58:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:09 smithi115 ceph-mon[35364]: pgmap v8202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:09 smithi181 ceph-mon[35869]: pgmap v8202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:09 smithi097 ceph-mon[32231]: pgmap v8202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:10 smithi115 ceph-mon[35364]: pgmap v8203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:10 smithi181 ceph-mon[35869]: pgmap v8203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:10 smithi097 ceph-mon[32231]: pgmap v8203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:11.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14328.000166 secs 2024-07-26T11:58:11.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14328.000166 secs 2024-07-26T11:58:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14328.000166 secs 2024-07-26T11:58:12.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:12 smithi115 ceph-mon[35364]: pgmap v8204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:12 smithi181 ceph-mon[35869]: pgmap v8204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:12 smithi097 ceph-mon[32231]: pgmap v8204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:15 smithi115 ceph-mon[35364]: pgmap v8205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:15 smithi181 ceph-mon[35869]: pgmap v8205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:15 smithi097 ceph-mon[32231]: pgmap v8205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:16.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14333.000318 secs 2024-07-26T11:58:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14333.000318 secs 2024-07-26T11:58:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14333.000318 secs 2024-07-26T11:58:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:17 smithi115 ceph-mon[35364]: pgmap v8206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:17 smithi181 ceph-mon[35869]: pgmap v8206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:17 smithi097 ceph-mon[32231]: pgmap v8206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:19 smithi115 ceph-mon[35364]: pgmap v8207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:19 smithi181 ceph-mon[35869]: pgmap v8207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:19 smithi097 ceph-mon[32231]: pgmap v8207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:21 smithi115 ceph-mon[35364]: pgmap v8208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14338.000473 secs 2024-07-26T11:58:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:21 smithi181 ceph-mon[35869]: pgmap v8208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14338.000473 secs 2024-07-26T11:58:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:21 smithi097 ceph-mon[32231]: pgmap v8208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14338.000473 secs 2024-07-26T11:58:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:23 smithi115 ceph-mon[35364]: pgmap v8209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:23 smithi181 ceph-mon[35869]: pgmap v8209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:23 smithi097 ceph-mon[32231]: pgmap v8209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:23.531+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:23.532+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:58:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:23.532+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:58:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:23.532+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:23.532+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:25 smithi115 ceph-mon[35364]: pgmap v8210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:25 smithi181 ceph-mon[35869]: pgmap v8210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:25 smithi097 ceph-mon[32231]: pgmap v8210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14343.000638 secs 2024-07-26T11:58:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14343.000638 secs 2024-07-26T11:58:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14343.000638 secs 2024-07-26T11:58:27.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:27 smithi115 ceph-mon[35364]: pgmap v8211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:27 smithi181 ceph-mon[35869]: pgmap v8211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:27 smithi097 ceph-mon[32231]: pgmap v8211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:29.478 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:29 smithi115 ceph-mon[35364]: pgmap v8212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:29 smithi181 ceph-mon[35869]: pgmap v8212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:29 smithi097 ceph-mon[32231]: pgmap v8212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:31 smithi115 ceph-mon[35364]: pgmap v8213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14348.000815 secs 2024-07-26T11:58:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:31 smithi181 ceph-mon[35869]: pgmap v8213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14348.000815 secs 2024-07-26T11:58:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:31 smithi097 ceph-mon[32231]: pgmap v8213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14348.000815 secs 2024-07-26T11:58:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:33 smithi115 ceph-mon[35364]: pgmap v8214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:33 smithi181 ceph-mon[35869]: pgmap v8214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:33 smithi097 ceph-mon[32231]: pgmap v8214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:35 smithi115 ceph-mon[35364]: pgmap v8215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:35 smithi181 ceph-mon[35869]: pgmap v8215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:35 smithi097 ceph-mon[32231]: pgmap v8215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14353.000971 secs 2024-07-26T11:58:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14353.000971 secs 2024-07-26T11:58:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14353.000971 secs 2024-07-26T11:58:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:37 smithi115 ceph-mon[35364]: pgmap v8216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:37 smithi181 ceph-mon[35869]: pgmap v8216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:37 smithi097 ceph-mon[32231]: pgmap v8216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:38.539+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:38.542 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:38.541+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:58:38.542 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:38.541+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:58:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:38.541+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:38.541+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:39 smithi115 ceph-mon[35364]: pgmap v8217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:39 smithi181 ceph-mon[35869]: pgmap v8217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:39 smithi097 ceph-mon[32231]: pgmap v8217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14358.001124 secs 2024-07-26T11:58:41.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:41 smithi115 ceph-mon[35364]: pgmap v8218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14358.001124 secs 2024-07-26T11:58:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:41 smithi181 ceph-mon[35869]: pgmap v8218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14358.001124 secs 2024-07-26T11:58:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:41 smithi097 ceph-mon[32231]: pgmap v8218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:43 smithi115 ceph-mon[35364]: pgmap v8219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:43 smithi181 ceph-mon[35869]: pgmap v8219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:43 smithi097 ceph-mon[32231]: pgmap v8219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:45.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:45 smithi115 ceph-mon[35364]: pgmap v8220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:45 smithi181 ceph-mon[35869]: pgmap v8220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:45 smithi097 ceph-mon[32231]: pgmap v8220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14363.001311 secs 2024-07-26T11:58:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14363.001311 secs 2024-07-26T11:58:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14363.001311 secs 2024-07-26T11:58:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:47 smithi115 ceph-mon[35364]: pgmap v8221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:47 smithi181 ceph-mon[35869]: pgmap v8221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:47 smithi097 ceph-mon[32231]: pgmap v8221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:49.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:49 smithi115 ceph-mon[35364]: pgmap v8222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:49 smithi181 ceph-mon[35869]: pgmap v8222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:49 smithi097 ceph-mon[32231]: pgmap v8222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14368.001453 secs 2024-07-26T11:58:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:51 smithi115 ceph-mon[35364]: pgmap v8223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14368.001453 secs 2024-07-26T11:58:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:51 smithi181 ceph-mon[35869]: pgmap v8223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14368.001453 secs 2024-07-26T11:58:51.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:51 smithi097 ceph-mon[32231]: pgmap v8223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:53.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:53 smithi115 ceph-mon[35364]: pgmap v8224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:53 smithi181 ceph-mon[35869]: pgmap v8224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:53 smithi097 ceph-mon[32231]: pgmap v8224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:53.543+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:58:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:53.544+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:58:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:53.545+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:58:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:53.545+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:58:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:58:53.545+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:55 smithi181 ceph-mon[35869]: pgmap v8225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:55 smithi097 ceph-mon[32231]: pgmap v8225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:55.640 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:55 smithi115 ceph-mon[35364]: pgmap v8225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:55.640 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14373.001530 secs 2024-07-26T11:58:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14373.001530 secs 2024-07-26T11:58:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14373.001530 secs 2024-07-26T11:58:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:57 smithi097 ceph-mon[32231]: pgmap v8226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:57 smithi115 ceph-mon[35364]: pgmap v8226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:57 smithi181 ceph-mon[35869]: pgmap v8226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:58 smithi115 ceph-mon[35364]: pgmap v8227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:58 smithi181 ceph-mon[35869]: pgmap v8227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:58 smithi097 ceph-mon[32231]: pgmap v8227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:58.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:59 smithi115 ceph-mon[35364]: pgmap v8228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:59 smithi115 ceph-mon[35364]: pgmap v8229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:58:59.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:58:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:59 smithi181 ceph-mon[35869]: pgmap v8228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:59 smithi181 ceph-mon[35869]: pgmap v8229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:58:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:58:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:58:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:59 smithi097 ceph-mon[32231]: pgmap v8228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:58:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:59 smithi097 ceph-mon[32231]: pgmap v8229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:58:59.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:58:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T11:59:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14378.001682 secs 2024-07-26T11:59:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14378.001682 secs 2024-07-26T11:59:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14378.001682 secs 2024-07-26T11:59:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:01 smithi115 ceph-mon[35364]: pgmap v8230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:59:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:01 smithi181 ceph-mon[35869]: pgmap v8230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:59:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:01 smithi097 ceph-mon[32231]: pgmap v8230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:59:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:03 smithi115 ceph-mon[35364]: pgmap v8231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:59:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:03 smithi181 ceph-mon[35869]: pgmap v8231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:59:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:03 smithi097 ceph-mon[32231]: pgmap v8231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:59:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:05 smithi115 ceph-mon[35364]: pgmap v8232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:59:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:05 smithi181 ceph-mon[35869]: pgmap v8232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:59:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:05 smithi097 ceph-mon[32231]: pgmap v8232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:59:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14383.001834 secs 2024-07-26T11:59:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14383.001834 secs 2024-07-26T11:59:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14383.001834 secs 2024-07-26T11:59:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:07 smithi115 ceph-mon[35364]: pgmap v8233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:59:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:07 smithi181 ceph-mon[35869]: pgmap v8233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:59:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:07 smithi097 ceph-mon[32231]: pgmap v8233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:59:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:08.552+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:59:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:08.553+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:59:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:08.553+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:59:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:08.553+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:59:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:08.553+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:09 smithi115 ceph-mon[35364]: pgmap v8234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:59:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:09 smithi181 ceph-mon[35869]: pgmap v8234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:59:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:09 smithi097 ceph-mon[32231]: pgmap v8234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:59:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14388.001988 secs 2024-07-26T11:59:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14388.001988 secs 2024-07-26T11:59:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14388.001988 secs 2024-07-26T11:59:11.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:11 smithi115 ceph-mon[35364]: pgmap v8235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:11 smithi181 ceph-mon[35869]: pgmap v8235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:11 smithi097 ceph-mon[32231]: pgmap v8235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:13 smithi181 ceph-mon[35869]: pgmap v8236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:13 smithi115 ceph-mon[35364]: pgmap v8236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:13 smithi097 ceph-mon[32231]: pgmap v8236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:15 smithi115 ceph-mon[35364]: pgmap v8237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:15 smithi181 ceph-mon[35869]: pgmap v8237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:15 smithi097 ceph-mon[32231]: pgmap v8237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:16.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14393.002164 secs 2024-07-26T11:59:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14393.002164 secs 2024-07-26T11:59:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14393.002164 secs 2024-07-26T11:59:17.687 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:17 smithi097 ceph-mon[32231]: pgmap v8238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:17 smithi115 ceph-mon[35364]: pgmap v8238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:17 smithi181 ceph-mon[35869]: pgmap v8238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:19 smithi115 ceph-mon[35364]: pgmap v8239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:19 smithi181 ceph-mon[35869]: pgmap v8239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:19 smithi097 ceph-mon[32231]: pgmap v8239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14398.002320 secs 2024-07-26T11:59:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14398.002320 secs 2024-07-26T11:59:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14398.002320 secs 2024-07-26T11:59:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:21 smithi115 ceph-mon[35364]: pgmap v8240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:21 smithi181 ceph-mon[35869]: pgmap v8240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:21 smithi097 ceph-mon[32231]: pgmap v8240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:23.556+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:59:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:23.557+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:59:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:23.557+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:59:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:23.557+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:59:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:23.557+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:23 smithi115 ceph-mon[35364]: pgmap v8241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:23 smithi181 ceph-mon[35869]: pgmap v8241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:23 smithi097 ceph-mon[32231]: pgmap v8241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:25 smithi115 ceph-mon[35364]: pgmap v8242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:25 smithi181 ceph-mon[35869]: pgmap v8242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:25 smithi097 ceph-mon[32231]: pgmap v8242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14403.002476 secs 2024-07-26T11:59:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14403.002476 secs 2024-07-26T11:59:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14403.002476 secs 2024-07-26T11:59:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:27 smithi115 ceph-mon[35364]: pgmap v8243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:27 smithi181 ceph-mon[35869]: pgmap v8243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:27 smithi097 ceph-mon[32231]: pgmap v8243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:29 smithi115 ceph-mon[35364]: pgmap v8244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:29 smithi181 ceph-mon[35869]: pgmap v8244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:29 smithi097 ceph-mon[32231]: pgmap v8244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14408.002632 secs 2024-07-26T11:59:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14408.002632 secs 2024-07-26T11:59:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14408.002632 secs 2024-07-26T11:59:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:31 smithi115 ceph-mon[35364]: pgmap v8245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:31 smithi181 ceph-mon[35869]: pgmap v8245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:31 smithi097 ceph-mon[32231]: pgmap v8245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:34 smithi115 ceph-mon[35364]: pgmap v8246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:34 smithi181 ceph-mon[35869]: pgmap v8246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:34 smithi097 ceph-mon[32231]: pgmap v8246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14413.002737 secs 2024-07-26T11:59:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:36 smithi115 ceph-mon[35364]: pgmap v8247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14413.002737 secs 2024-07-26T11:59:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:36 smithi181 ceph-mon[35869]: pgmap v8247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14413.002737 secs 2024-07-26T11:59:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:36 smithi097 ceph-mon[32231]: pgmap v8247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:38.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:38.562+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:59:38.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:38.563+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:59:38.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:38.563+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:59:38.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:38.563+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:59:38.565 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:38.563+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:38 smithi115 ceph-mon[35364]: pgmap v8248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:38 smithi181 ceph-mon[35869]: pgmap v8248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:38 smithi097 ceph-mon[32231]: pgmap v8248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:40 smithi115 ceph-mon[35364]: pgmap v8249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14418.002893 secs 2024-07-26T11:59:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:40 smithi181 ceph-mon[35869]: pgmap v8249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14418.002893 secs 2024-07-26T11:59:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:40 smithi097 ceph-mon[32231]: pgmap v8249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14418.002893 secs 2024-07-26T11:59:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:42 smithi115 ceph-mon[35364]: pgmap v8250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:42 smithi181 ceph-mon[35869]: pgmap v8250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:42 smithi097 ceph-mon[32231]: pgmap v8250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:44 smithi115 ceph-mon[35364]: pgmap v8251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:59:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:44 smithi181 ceph-mon[35869]: pgmap v8251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:59:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:44 smithi097 ceph-mon[32231]: pgmap v8251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:59:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14423.003042 secs 2024-07-26T11:59:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14423.003042 secs 2024-07-26T11:59:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14423.003042 secs 2024-07-26T11:59:46.654 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:46 smithi097 ceph-mon[32231]: pgmap v8252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:59:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:46 smithi115 ceph-mon[35364]: pgmap v8252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:59:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:46 smithi181 ceph-mon[35869]: pgmap v8252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:59:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:48 smithi115 ceph-mon[35364]: pgmap v8253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:59:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:48 smithi181 ceph-mon[35869]: pgmap v8253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:59:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:48 smithi097 ceph-mon[32231]: pgmap v8253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:59:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:50 smithi115 ceph-mon[35364]: pgmap v8254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:59:50.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14428.003209 secs 2024-07-26T11:59:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:50 smithi181 ceph-mon[35869]: pgmap v8254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:59:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14428.003209 secs 2024-07-26T11:59:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:50 smithi097 ceph-mon[32231]: pgmap v8254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:59:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14428.003209 secs 2024-07-26T11:59:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:52 smithi115 ceph-mon[35364]: pgmap v8255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:52 smithi181 ceph-mon[35869]: pgmap v8255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:52 smithi097 ceph-mon[32231]: pgmap v8255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:53.565+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:59:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:53.566+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T11:59:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:53.566+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:59:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:53.566+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:59:53.568 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T11:59:53.566+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:54 smithi115 ceph-mon[35364]: pgmap v8256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:54 smithi181 ceph-mon[35869]: pgmap v8256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:54 smithi097 ceph-mon[32231]: pgmap v8256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14433.003369 secs 2024-07-26T11:59:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14433.003369 secs 2024-07-26T11:59:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14433.003369 secs 2024-07-26T11:59:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:56 smithi115 ceph-mon[35364]: pgmap v8257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:56 smithi181 ceph-mon[35869]: pgmap v8257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:56 smithi097 ceph-mon[32231]: pgmap v8257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 11:59:58 smithi115 ceph-mon[35364]: pgmap v8258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 11:59:58 smithi181 ceph-mon[35869]: pgmap v8258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 11:59:58 smithi097 ceph-mon[32231]: pgmap v8258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:00 smithi115 ceph-mon[35364]: pgmap v8259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:00.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14438.003475 secs 2024-07-26T12:00:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:00 smithi181 ceph-mon[35869]: pgmap v8259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14438.003475 secs 2024-07-26T12:00:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:00 smithi097 ceph-mon[32231]: pgmap v8259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:00.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14438.003475 secs 2024-07-26T12:00:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:01 smithi181 ceph-mon[35869]: pgmap v8260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:01 smithi181 ceph-mon[35869]: pgmap v8261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:01.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:00:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:01 smithi097 ceph-mon[32231]: pgmap v8260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:01 smithi097 ceph-mon[32231]: pgmap v8261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:01.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:00:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:01 smithi115 ceph-mon[35364]: pgmap v8260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:01 smithi115 ceph-mon[35364]: pgmap v8261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:00:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:03 smithi115 ceph-mon[35364]: pgmap v8262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:03 smithi181 ceph-mon[35869]: pgmap v8262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:03 smithi097 ceph-mon[32231]: pgmap v8262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:05 smithi115 ceph-mon[35364]: pgmap v8263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14443.003578 secs 2024-07-26T12:00:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:05 smithi181 ceph-mon[35869]: pgmap v8263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14443.003578 secs 2024-07-26T12:00:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:05 smithi097 ceph-mon[32231]: pgmap v8263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14443.003578 secs 2024-07-26T12:00:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:07 smithi115 ceph-mon[35364]: pgmap v8264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:07 smithi181 ceph-mon[35869]: pgmap v8264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:07 smithi097 ceph-mon[32231]: pgmap v8264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:08.573+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:00:08.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:08.574+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:00:08.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:08.574+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:00:08.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:08.574+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:00:08.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:08.574+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:09 smithi115 ceph-mon[35364]: pgmap v8265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:09 smithi181 ceph-mon[35869]: pgmap v8265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:09 smithi097 ceph-mon[32231]: pgmap v8265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14448.003750 secs 2024-07-26T12:00:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14448.003750 secs 2024-07-26T12:00:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14448.003750 secs 2024-07-26T12:00:12.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:11 smithi115 ceph-mon[35364]: pgmap v8266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:12.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:11 smithi181 ceph-mon[35869]: pgmap v8266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:11 smithi097 ceph-mon[32231]: pgmap v8266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:13.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:13 smithi115 ceph-mon[35364]: pgmap v8267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:13 smithi181 ceph-mon[35869]: pgmap v8267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:13 smithi097 ceph-mon[32231]: pgmap v8267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:15 smithi115 ceph-mon[35364]: pgmap v8268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14453.003905 secs 2024-07-26T12:00:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:15 smithi181 ceph-mon[35869]: pgmap v8268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14453.003905 secs 2024-07-26T12:00:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:15 smithi097 ceph-mon[32231]: pgmap v8268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14453.003905 secs 2024-07-26T12:00:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:17 smithi115 ceph-mon[35364]: pgmap v8269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:17 smithi181 ceph-mon[35869]: pgmap v8269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:18.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:17 smithi097 ceph-mon[32231]: pgmap v8269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:19 smithi115 ceph-mon[35364]: pgmap v8270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:19 smithi181 ceph-mon[35869]: pgmap v8270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:19 smithi097 ceph-mon[32231]: pgmap v8270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14458.004070 secs 2024-07-26T12:00:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14458.004070 secs 2024-07-26T12:00:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14458.004070 secs 2024-07-26T12:00:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:21 smithi115 ceph-mon[35364]: pgmap v8271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:21 smithi181 ceph-mon[35869]: pgmap v8271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:21 smithi097 ceph-mon[32231]: pgmap v8271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:23.577+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:00:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:23.578+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:00:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:23.578+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:00:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:23.578+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:00:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:23.578+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:23 smithi115 ceph-mon[35364]: pgmap v8272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:23 smithi181 ceph-mon[35869]: pgmap v8272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:23 smithi097 ceph-mon[32231]: pgmap v8272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:25 smithi115 ceph-mon[35364]: pgmap v8273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14463.004223 secs 2024-07-26T12:00:26.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:25 smithi181 ceph-mon[35869]: pgmap v8273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:26.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14463.004223 secs 2024-07-26T12:00:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:25 smithi097 ceph-mon[32231]: pgmap v8273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14463.004223 secs 2024-07-26T12:00:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:27 smithi115 ceph-mon[35364]: pgmap v8274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:27 smithi181 ceph-mon[35869]: pgmap v8274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:27 smithi097 ceph-mon[32231]: pgmap v8274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:29 smithi115 ceph-mon[35364]: pgmap v8275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:29 smithi181 ceph-mon[35869]: pgmap v8275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:29 smithi097 ceph-mon[32231]: pgmap v8275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14468.004348 secs 2024-07-26T12:00:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14468.004348 secs 2024-07-26T12:00:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14468.004348 secs 2024-07-26T12:00:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:31 smithi115 ceph-mon[35364]: pgmap v8276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:31 smithi181 ceph-mon[35869]: pgmap v8276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:31 smithi097 ceph-mon[32231]: pgmap v8276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:33 smithi115 ceph-mon[35364]: pgmap v8277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:33 smithi181 ceph-mon[35869]: pgmap v8277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:33 smithi097 ceph-mon[32231]: pgmap v8277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:35 smithi115 ceph-mon[35364]: pgmap v8278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14473.004484 secs 2024-07-26T12:00:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:35 smithi181 ceph-mon[35869]: pgmap v8278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14473.004484 secs 2024-07-26T12:00:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:35 smithi097 ceph-mon[32231]: pgmap v8278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:36.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14473.004484 secs 2024-07-26T12:00:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:37 smithi115 ceph-mon[35364]: pgmap v8279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:37 smithi181 ceph-mon[35869]: pgmap v8279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:37 smithi097 ceph-mon[32231]: pgmap v8279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:38.586+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:00:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:38.587+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:00:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:38.587+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:00:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:38.587+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:00:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:38.587+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:39 smithi115 ceph-mon[35364]: pgmap v8280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:39 smithi181 ceph-mon[35869]: pgmap v8280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:39 smithi097 ceph-mon[32231]: pgmap v8280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14478.004582 secs 2024-07-26T12:00:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14478.004582 secs 2024-07-26T12:00:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14478.004582 secs 2024-07-26T12:00:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:41 smithi115 ceph-mon[35364]: pgmap v8281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:41 smithi181 ceph-mon[35869]: pgmap v8281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:41 smithi097 ceph-mon[32231]: pgmap v8281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:43 smithi115 ceph-mon[35364]: pgmap v8282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:43 smithi181 ceph-mon[35869]: pgmap v8282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:43 smithi097 ceph-mon[32231]: pgmap v8282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:45 smithi115 ceph-mon[35364]: pgmap v8283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:00:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14483.004726 secs 2024-07-26T12:00:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:45 smithi181 ceph-mon[35869]: pgmap v8283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:00:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14483.004726 secs 2024-07-26T12:00:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:45 smithi097 ceph-mon[32231]: pgmap v8283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:00:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14483.004726 secs 2024-07-26T12:00:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:47 smithi181 ceph-mon[35869]: pgmap v8284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:00:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:47 smithi097 ceph-mon[32231]: pgmap v8284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:00:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:47 smithi115 ceph-mon[35364]: pgmap v8284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:00:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:49 smithi097 ceph-mon[32231]: pgmap v8285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:00:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:49 smithi115 ceph-mon[35364]: pgmap v8285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:00:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:49 smithi181 ceph-mon[35869]: pgmap v8285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:00:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14488.004865 secs 2024-07-26T12:00:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14488.004865 secs 2024-07-26T12:00:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14488.004865 secs 2024-07-26T12:00:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:51 smithi097 ceph-mon[32231]: pgmap v8286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:00:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:52 smithi115 ceph-mon[35364]: pgmap v8286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:00:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:52 smithi181 ceph-mon[35869]: pgmap v8286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:00:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:53.589+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:00:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:53.591+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:00:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:53.591+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:00:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:53.591+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:00:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:00:53.591+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:54 smithi115 ceph-mon[35364]: pgmap v8287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:54 smithi181 ceph-mon[35869]: pgmap v8287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:54 smithi097 ceph-mon[32231]: pgmap v8287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:56 smithi115 ceph-mon[35364]: pgmap v8288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14493.004951 secs 2024-07-26T12:00:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:56 smithi181 ceph-mon[35869]: pgmap v8288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14493.004951 secs 2024-07-26T12:00:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:56 smithi097 ceph-mon[32231]: pgmap v8288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:00:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14493.004951 secs 2024-07-26T12:00:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:00:58 smithi115 ceph-mon[35364]: pgmap v8289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:00:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:00:58 smithi181 ceph-mon[35869]: pgmap v8289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:00:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:00:58 smithi097 ceph-mon[32231]: pgmap v8289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:00 smithi115 ceph-mon[35364]: pgmap v8290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:01:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:00 smithi181 ceph-mon[35869]: pgmap v8290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:01:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:00 smithi097 ceph-mon[32231]: pgmap v8290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:01:01.385 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14498.005092 secs 2024-07-26T12:01:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14498.005092 secs 2024-07-26T12:01:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14498.005092 secs 2024-07-26T12:01:02.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:02 smithi115 ceph-mon[35364]: pgmap v8291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:01:02.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:02 smithi181 ceph-mon[35869]: pgmap v8291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:01:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:02 smithi097 ceph-mon[32231]: pgmap v8291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:01:02.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:03 smithi115 ceph-mon[35364]: pgmap v8292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:03 smithi115 ceph-mon[35364]: pgmap v8293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:01:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:03 smithi181 ceph-mon[35869]: pgmap v8292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:03 smithi181 ceph-mon[35869]: pgmap v8293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:01:03.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:03.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:03.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:03 smithi097 ceph-mon[32231]: pgmap v8292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:03.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:03 smithi097 ceph-mon[32231]: pgmap v8293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:03.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:01:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:05 smithi115 ceph-mon[35364]: pgmap v8294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:05.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:05 smithi181 ceph-mon[35869]: pgmap v8294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:05 smithi097 ceph-mon[32231]: pgmap v8294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14503.005261 secs 2024-07-26T12:01:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14503.005261 secs 2024-07-26T12:01:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14503.005261 secs 2024-07-26T12:01:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:07 smithi115 ceph-mon[35364]: pgmap v8295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:01:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:07 smithi181 ceph-mon[35869]: pgmap v8295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:01:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:07 smithi097 ceph-mon[32231]: pgmap v8295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:01:08.599 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:08.598+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:01:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:08.599+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:01:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:08.599+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:01:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:08.599+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:01:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:08.599+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:09 smithi115 ceph-mon[35364]: pgmap v8296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:01:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:09.473+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:01:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:09.473+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:01:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:09.474+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:01:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:09.474+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:01:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:09 smithi181 ceph-mon[35869]: pgmap v8296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:01:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:09 smithi097 ceph-mon[32231]: pgmap v8296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:01:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14508.005364 secs 2024-07-26T12:01:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:11 smithi115 ceph-mon[35364]: pgmap v8297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:01:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14508.005364 secs 2024-07-26T12:01:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:11 smithi181 ceph-mon[35869]: pgmap v8297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:01:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14508.005364 secs 2024-07-26T12:01:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:11 smithi097 ceph-mon[32231]: pgmap v8297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:01:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:13 smithi181 ceph-mon[35869]: pgmap v8298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:13 smithi097 ceph-mon[32231]: pgmap v8298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:13.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:13 smithi115 ceph-mon[35364]: pgmap v8298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:15 smithi097 ceph-mon[32231]: pgmap v8299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:15 smithi115 ceph-mon[35364]: pgmap v8299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:15 smithi181 ceph-mon[35869]: pgmap v8299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14513.005497 secs 2024-07-26T12:01:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14513.005497 secs 2024-07-26T12:01:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14513.005497 secs 2024-07-26T12:01:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:17 smithi115 ceph-mon[35364]: pgmap v8300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:17 smithi181 ceph-mon[35869]: pgmap v8300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:17 smithi097 ceph-mon[32231]: pgmap v8300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:19 smithi115 ceph-mon[35364]: pgmap v8301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:19 smithi181 ceph-mon[35869]: pgmap v8301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:19 smithi097 ceph-mon[32231]: pgmap v8301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14518.005630 secs 2024-07-26T12:01:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:21 smithi115 ceph-mon[35364]: pgmap v8302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14518.005630 secs 2024-07-26T12:01:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:21 smithi181 ceph-mon[35869]: pgmap v8302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14518.005630 secs 2024-07-26T12:01:21.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:21 smithi097 ceph-mon[32231]: pgmap v8302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:23.601+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:01:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:23.602+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:01:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:23.602+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:01:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:23.602+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:01:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:23.602+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:23 smithi115 ceph-mon[35364]: pgmap v8303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:23 smithi181 ceph-mon[35869]: pgmap v8303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:23 smithi097 ceph-mon[32231]: pgmap v8303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:25 smithi115 ceph-mon[35364]: pgmap v8304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:25 smithi181 ceph-mon[35869]: pgmap v8304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:25 smithi097 ceph-mon[32231]: pgmap v8304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14523.005785 secs 2024-07-26T12:01:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14523.005785 secs 2024-07-26T12:01:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14523.005785 secs 2024-07-26T12:01:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:27 smithi115 ceph-mon[35364]: pgmap v8305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:27 smithi181 ceph-mon[35869]: pgmap v8305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:27 smithi097 ceph-mon[32231]: pgmap v8305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:29 smithi115 ceph-mon[35364]: pgmap v8306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:29 smithi181 ceph-mon[35869]: pgmap v8306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:29 smithi097 ceph-mon[32231]: pgmap v8306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14528.005925 secs 2024-07-26T12:01:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:31 smithi115 ceph-mon[35364]: pgmap v8307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14528.005925 secs 2024-07-26T12:01:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:31 smithi181 ceph-mon[35869]: pgmap v8307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14528.005925 secs 2024-07-26T12:01:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:31 smithi097 ceph-mon[32231]: pgmap v8307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:33 smithi115 ceph-mon[35364]: pgmap v8308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:33 smithi181 ceph-mon[35869]: pgmap v8308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:33 smithi097 ceph-mon[32231]: pgmap v8308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:35 smithi115 ceph-mon[35364]: pgmap v8309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:35 smithi181 ceph-mon[35869]: pgmap v8309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:35 smithi097 ceph-mon[32231]: pgmap v8309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14533.006081 secs 2024-07-26T12:01:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:36 smithi115 ceph-mon[35364]: pgmap v8310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14533.006081 secs 2024-07-26T12:01:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:36 smithi181 ceph-mon[35869]: pgmap v8310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14533.006081 secs 2024-07-26T12:01:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:36 smithi097 ceph-mon[32231]: pgmap v8310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:38.609+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:01:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:38.610+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:01:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:38.610+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:01:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:38.610+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:01:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:38.610+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:39 smithi115 ceph-mon[35364]: pgmap v8311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:39 smithi181 ceph-mon[35869]: pgmap v8311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:39 smithi097 ceph-mon[32231]: pgmap v8311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14538.006218 secs 2024-07-26T12:01:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:41 smithi181 ceph-mon[35869]: pgmap v8312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14538.006218 secs 2024-07-26T12:01:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:41 smithi097 ceph-mon[32231]: pgmap v8312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14538.006218 secs 2024-07-26T12:01:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:41 smithi115 ceph-mon[35364]: pgmap v8312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:43 smithi097 ceph-mon[32231]: pgmap v8313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:43.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:43 smithi115 ceph-mon[35364]: pgmap v8313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:43 smithi181 ceph-mon[35869]: pgmap v8313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:45 smithi097 ceph-mon[32231]: pgmap v8314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:45 smithi115 ceph-mon[35364]: pgmap v8314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:45 smithi181 ceph-mon[35869]: pgmap v8314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14543.006355 secs 2024-07-26T12:01:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14543.006355 secs 2024-07-26T12:01:46.780 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14543.006355 secs 2024-07-26T12:01:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:47 smithi115 ceph-mon[35364]: pgmap v8315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:47 smithi181 ceph-mon[35869]: pgmap v8315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:47 smithi097 ceph-mon[32231]: pgmap v8315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:49 smithi115 ceph-mon[35364]: pgmap v8316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:49 smithi181 ceph-mon[35869]: pgmap v8316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:49 smithi097 ceph-mon[32231]: pgmap v8316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14548.006452 secs 2024-07-26T12:01:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:51 smithi115 ceph-mon[35364]: pgmap v8317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14548.006452 secs 2024-07-26T12:01:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:51 smithi181 ceph-mon[35869]: pgmap v8317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14548.006452 secs 2024-07-26T12:01:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:51 smithi097 ceph-mon[32231]: pgmap v8317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:53.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:53.612+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:01:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:53.613+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:01:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:53.613+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:01:53.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:53.613+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:01:53.615 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:01:53.613+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:53 smithi115 ceph-mon[35364]: pgmap v8318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:53 smithi181 ceph-mon[35869]: pgmap v8318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:53 smithi097 ceph-mon[32231]: pgmap v8318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:55 smithi115 ceph-mon[35364]: pgmap v8319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:55 smithi181 ceph-mon[35869]: pgmap v8319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:55 smithi097 ceph-mon[32231]: pgmap v8319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14553.006595 secs 2024-07-26T12:01:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14553.006595 secs 2024-07-26T12:01:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14553.006595 secs 2024-07-26T12:01:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:57 smithi115 ceph-mon[35364]: pgmap v8320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:57 smithi181 ceph-mon[35869]: pgmap v8320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:57 smithi097 ceph-mon[32231]: pgmap v8320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:01:59 smithi115 ceph-mon[35364]: pgmap v8321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:01:59 smithi181 ceph-mon[35869]: pgmap v8321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:01:59 smithi097 ceph-mon[32231]: pgmap v8321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14558.006734 secs 2024-07-26T12:02:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:01 smithi115 ceph-mon[35364]: pgmap v8322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14558.006734 secs 2024-07-26T12:02:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:01 smithi181 ceph-mon[35869]: pgmap v8322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14558.006734 secs 2024-07-26T12:02:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:01 smithi097 ceph-mon[32231]: pgmap v8322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:03 smithi115 ceph-mon[35364]: pgmap v8323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:03 smithi181 ceph-mon[35869]: pgmap v8323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:03 smithi097 ceph-mon[32231]: pgmap v8323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:04.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:02:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:02:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:02:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:05 smithi115 ceph-mon[35364]: pgmap v8324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:02:05.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:05 smithi115 ceph-mon[35364]: pgmap v8325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:02:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:05 smithi181 ceph-mon[35869]: pgmap v8324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:02:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:05 smithi181 ceph-mon[35869]: pgmap v8325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:02:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:05 smithi097 ceph-mon[32231]: pgmap v8324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:02:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:05 smithi097 ceph-mon[32231]: pgmap v8325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:02:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14563.006864 secs 2024-07-26T12:02:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:06 smithi115 ceph-mon[35364]: pgmap v8326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:02:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14563.006864 secs 2024-07-26T12:02:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:06 smithi181 ceph-mon[35869]: pgmap v8326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:02:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14563.006864 secs 2024-07-26T12:02:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:06 smithi097 ceph-mon[32231]: pgmap v8326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:02:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:08.619+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:02:08.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:08.621+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:02:08.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:08.621+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:02:08.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:08.621+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:02:08.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:08.621+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:08 smithi115 ceph-mon[35364]: pgmap v8327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:08 smithi181 ceph-mon[35869]: pgmap v8327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:08 smithi097 ceph-mon[32231]: pgmap v8327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:10 smithi115 ceph-mon[35364]: pgmap v8328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14568.006999 secs 2024-07-26T12:02:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:10 smithi181 ceph-mon[35869]: pgmap v8328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14568.006999 secs 2024-07-26T12:02:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:10 smithi097 ceph-mon[32231]: pgmap v8328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14568.006999 secs 2024-07-26T12:02:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:12 smithi115 ceph-mon[35364]: pgmap v8329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:12 smithi181 ceph-mon[35869]: pgmap v8329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:12 smithi097 ceph-mon[32231]: pgmap v8329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:14 smithi115 ceph-mon[35364]: pgmap v8330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:14 smithi181 ceph-mon[35869]: pgmap v8330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:14 smithi097 ceph-mon[32231]: pgmap v8330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14573.007143 secs 2024-07-26T12:02:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14573.007143 secs 2024-07-26T12:02:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14573.007143 secs 2024-07-26T12:02:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:16 smithi115 ceph-mon[35364]: pgmap v8331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:16 smithi181 ceph-mon[35869]: pgmap v8331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:16 smithi097 ceph-mon[32231]: pgmap v8331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:18 smithi115 ceph-mon[35364]: pgmap v8332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:18 smithi181 ceph-mon[35869]: pgmap v8332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:18 smithi097 ceph-mon[32231]: pgmap v8332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:20 smithi115 ceph-mon[35364]: pgmap v8333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14578.007308 secs 2024-07-26T12:02:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:20 smithi181 ceph-mon[35869]: pgmap v8333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14578.007308 secs 2024-07-26T12:02:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:20 smithi097 ceph-mon[32231]: pgmap v8333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14578.007308 secs 2024-07-26T12:02:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:22 smithi115 ceph-mon[35364]: pgmap v8334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:22 smithi181 ceph-mon[35869]: pgmap v8334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:22 smithi097 ceph-mon[32231]: pgmap v8334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:23.623+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:02:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:23.624+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:02:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:23.624+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:02:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:23.624+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:02:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:23.624+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:24.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:24 smithi115 ceph-mon[35364]: pgmap v8335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:24 smithi181 ceph-mon[35869]: pgmap v8335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:24 smithi097 ceph-mon[32231]: pgmap v8335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14583.007456 secs 2024-07-26T12:02:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14583.007456 secs 2024-07-26T12:02:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14583.007456 secs 2024-07-26T12:02:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:26 smithi115 ceph-mon[35364]: pgmap v8336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:26 smithi181 ceph-mon[35869]: pgmap v8336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:26 smithi097 ceph-mon[32231]: pgmap v8336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:28.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:28 smithi115 ceph-mon[35364]: pgmap v8337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:28 smithi181 ceph-mon[35869]: pgmap v8337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:28 smithi097 ceph-mon[32231]: pgmap v8337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:30 smithi181 ceph-mon[35869]: pgmap v8338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14588.007580 secs 2024-07-26T12:02:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:30 smithi097 ceph-mon[32231]: pgmap v8338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14588.007580 secs 2024-07-26T12:02:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:30 smithi115 ceph-mon[35364]: pgmap v8338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14588.007580 secs 2024-07-26T12:02:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:32 smithi097 ceph-mon[32231]: pgmap v8339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:32 smithi115 ceph-mon[35364]: pgmap v8339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:32 smithi181 ceph-mon[35869]: pgmap v8339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:34 smithi097 ceph-mon[32231]: pgmap v8340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:34 smithi115 ceph-mon[35364]: pgmap v8340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:34 smithi181 ceph-mon[35869]: pgmap v8340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14593.007725 secs 2024-07-26T12:02:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14593.007725 secs 2024-07-26T12:02:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14593.007725 secs 2024-07-26T12:02:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:36 smithi115 ceph-mon[35364]: pgmap v8341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:36 smithi181 ceph-mon[35869]: pgmap v8341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:36 smithi097 ceph-mon[32231]: pgmap v8341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:38.631+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:02:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:38.632+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:02:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:38.632+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:02:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:38.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:02:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:38.632+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:38 smithi115 ceph-mon[35364]: pgmap v8342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:38 smithi181 ceph-mon[35869]: pgmap v8342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:38 smithi097 ceph-mon[32231]: pgmap v8342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:40 smithi115 ceph-mon[35364]: pgmap v8343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14598.007870 secs 2024-07-26T12:02:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:40 smithi181 ceph-mon[35869]: pgmap v8343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14598.007870 secs 2024-07-26T12:02:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:40 smithi097 ceph-mon[32231]: pgmap v8343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14598.007870 secs 2024-07-26T12:02:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:42 smithi115 ceph-mon[35364]: pgmap v8344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:42 smithi181 ceph-mon[35869]: pgmap v8344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:42 smithi097 ceph-mon[32231]: pgmap v8344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:45.037 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:44 smithi181 ceph-mon[35869]: pgmap v8345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:44 smithi115 ceph-mon[35364]: pgmap v8345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:44 smithi097 ceph-mon[32231]: pgmap v8345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:46.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14603.008010 secs 2024-07-26T12:02:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14603.008010 secs 2024-07-26T12:02:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14603.008010 secs 2024-07-26T12:02:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:46 smithi115 ceph-mon[35364]: pgmap v8346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:46 smithi181 ceph-mon[35869]: pgmap v8346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:46 smithi097 ceph-mon[32231]: pgmap v8346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:48 smithi115 ceph-mon[35364]: pgmap v8347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:48 smithi181 ceph-mon[35869]: pgmap v8347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:48 smithi097 ceph-mon[32231]: pgmap v8347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:50 smithi115 ceph-mon[35364]: pgmap v8348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14608.008169 secs 2024-07-26T12:02:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:50 smithi181 ceph-mon[35869]: pgmap v8348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14608.008169 secs 2024-07-26T12:02:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:50 smithi097 ceph-mon[32231]: pgmap v8348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14608.008169 secs 2024-07-26T12:02:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:52 smithi115 ceph-mon[35364]: pgmap v8349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:52 smithi181 ceph-mon[35869]: pgmap v8349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:52 smithi097 ceph-mon[32231]: pgmap v8349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:53.634+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:02:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:53.635+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:02:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:53.635+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:02:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:53.635+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:02:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:02:53.635+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:54 smithi115 ceph-mon[35364]: pgmap v8350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:54 smithi181 ceph-mon[35869]: pgmap v8350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:54 smithi097 ceph-mon[32231]: pgmap v8350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14613.008337 secs 2024-07-26T12:02:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14613.008337 secs 2024-07-26T12:02:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14613.008337 secs 2024-07-26T12:02:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:56 smithi115 ceph-mon[35364]: pgmap v8351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:56 smithi181 ceph-mon[35869]: pgmap v8351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:56 smithi097 ceph-mon[32231]: pgmap v8351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:02:58 smithi115 ceph-mon[35364]: pgmap v8352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:02:58 smithi181 ceph-mon[35869]: pgmap v8352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:02:58 smithi097 ceph-mon[32231]: pgmap v8352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:00 smithi115 ceph-mon[35364]: pgmap v8353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14618.008479 secs 2024-07-26T12:03:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:00 smithi181 ceph-mon[35869]: pgmap v8353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14618.008479 secs 2024-07-26T12:03:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:00 smithi097 ceph-mon[32231]: pgmap v8353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14618.008479 secs 2024-07-26T12:03:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:02 smithi115 ceph-mon[35364]: pgmap v8354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:02 smithi181 ceph-mon[35869]: pgmap v8354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:02 smithi097 ceph-mon[32231]: pgmap v8354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:04 smithi115 ceph-mon[35364]: pgmap v8355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:05.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:04 smithi181 ceph-mon[35869]: pgmap v8355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:04 smithi097 ceph-mon[32231]: pgmap v8355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:05.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:05 smithi115 ceph-mon[35364]: pgmap v8356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:05 smithi115 ceph-mon[35364]: pgmap v8357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:03:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14623.008578 secs 2024-07-26T12:03:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:05 smithi181 ceph-mon[35869]: pgmap v8356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:03:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:05 smithi181 ceph-mon[35869]: pgmap v8357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:03:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:03:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14623.008578 secs 2024-07-26T12:03:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:05 smithi097 ceph-mon[32231]: pgmap v8356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:03:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:05 smithi097 ceph-mon[32231]: pgmap v8357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:03:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:03:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14623.008578 secs 2024-07-26T12:03:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:08 smithi115 ceph-mon[35364]: pgmap v8358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:03:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:08 smithi181 ceph-mon[35869]: pgmap v8358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:03:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:08 smithi097 ceph-mon[32231]: pgmap v8358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:03:08.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:08.642+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:03:08.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:08.644+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:03:08.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:08.644+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:03:08.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:08.644+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:03:08.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:08.644+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:10 smithi115 ceph-mon[35364]: pgmap v8359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:10 smithi181 ceph-mon[35869]: pgmap v8359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:10 smithi097 ceph-mon[32231]: pgmap v8359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14628.008673 secs 2024-07-26T12:03:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14628.008673 secs 2024-07-26T12:03:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14628.008673 secs 2024-07-26T12:03:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:12 smithi115 ceph-mon[35364]: pgmap v8360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:03:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:12 smithi181 ceph-mon[35869]: pgmap v8360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:03:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:12 smithi097 ceph-mon[32231]: pgmap v8360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:03:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:14 smithi115 ceph-mon[35364]: pgmap v8361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:03:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:14 smithi181 ceph-mon[35869]: pgmap v8361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:03:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:14 smithi097 ceph-mon[32231]: pgmap v8361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:03:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:16 smithi115 ceph-mon[35364]: pgmap v8362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:03:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14633.008821 secs 2024-07-26T12:03:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:16 smithi181 ceph-mon[35869]: pgmap v8362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:03:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14633.008821 secs 2024-07-26T12:03:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:16 smithi097 ceph-mon[32231]: pgmap v8362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:03:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14633.008821 secs 2024-07-26T12:03:18.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:18 smithi115 ceph-mon[35364]: pgmap v8363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:18 smithi181 ceph-mon[35869]: pgmap v8363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:18 smithi097 ceph-mon[32231]: pgmap v8363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:20 smithi115 ceph-mon[35364]: pgmap v8364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:20 smithi181 ceph-mon[35869]: pgmap v8364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:20 smithi097 ceph-mon[32231]: pgmap v8364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14638.008971 secs 2024-07-26T12:03:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14638.008971 secs 2024-07-26T12:03:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14638.008971 secs 2024-07-26T12:03:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:22 smithi115 ceph-mon[35364]: pgmap v8365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:22 smithi181 ceph-mon[35869]: pgmap v8365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:22 smithi097 ceph-mon[32231]: pgmap v8365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:23.648 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:23.647+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:03:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:23.648+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:03:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:23.648+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:03:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:23.648+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:03:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:23.648+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:24 smithi115 ceph-mon[35364]: pgmap v8366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:24 smithi181 ceph-mon[35869]: pgmap v8366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:24 smithi097 ceph-mon[32231]: pgmap v8366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:26 smithi115 ceph-mon[35364]: pgmap v8367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14643.009112 secs 2024-07-26T12:03:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:26 smithi181 ceph-mon[35869]: pgmap v8367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14643.009112 secs 2024-07-26T12:03:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:26 smithi097 ceph-mon[32231]: pgmap v8367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14643.009112 secs 2024-07-26T12:03:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:28 smithi115 ceph-mon[35364]: pgmap v8368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:28 smithi181 ceph-mon[35869]: pgmap v8368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:28 smithi097 ceph-mon[32231]: pgmap v8368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:30 smithi115 ceph-mon[35364]: pgmap v8369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:30 smithi181 ceph-mon[35869]: pgmap v8369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:30 smithi097 ceph-mon[32231]: pgmap v8369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14648.009288 secs 2024-07-26T12:03:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14648.009288 secs 2024-07-26T12:03:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14648.009288 secs 2024-07-26T12:03:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:32 smithi115 ceph-mon[35364]: pgmap v8370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:32 smithi181 ceph-mon[35869]: pgmap v8370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:32 smithi097 ceph-mon[32231]: pgmap v8370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:34.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:34 smithi115 ceph-mon[35364]: pgmap v8371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:34 smithi181 ceph-mon[35869]: pgmap v8371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:34 smithi097 ceph-mon[32231]: pgmap v8371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14653.009384 secs 2024-07-26T12:03:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:36 smithi115 ceph-mon[35364]: pgmap v8372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14653.009384 secs 2024-07-26T12:03:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:36 smithi181 ceph-mon[35869]: pgmap v8372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14653.009384 secs 2024-07-26T12:03:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:36 smithi097 ceph-mon[32231]: pgmap v8372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:38 smithi115 ceph-mon[35364]: pgmap v8373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:38 smithi181 ceph-mon[35869]: pgmap v8373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:38 smithi097 ceph-mon[32231]: pgmap v8373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:38.656+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:03:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:38.657+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:03:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:38.657+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:03:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:38.657+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:03:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:38.658+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:40 smithi115 ceph-mon[35364]: pgmap v8374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:40 smithi181 ceph-mon[35869]: pgmap v8374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:40 smithi097 ceph-mon[32231]: pgmap v8374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14658.009524 secs 2024-07-26T12:03:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14658.009524 secs 2024-07-26T12:03:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14658.009524 secs 2024-07-26T12:03:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:42 smithi115 ceph-mon[35364]: pgmap v8375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:42 smithi181 ceph-mon[35869]: pgmap v8375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:42 smithi097 ceph-mon[32231]: pgmap v8375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:44 smithi181 ceph-mon[35869]: pgmap v8376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:44 smithi097 ceph-mon[32231]: pgmap v8376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:44 smithi115 ceph-mon[35364]: pgmap v8376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14663.009681 secs 2024-07-26T12:03:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:46 smithi097 ceph-mon[32231]: pgmap v8377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:46.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14663.009681 secs 2024-07-26T12:03:46.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:46 smithi115 ceph-mon[35364]: pgmap v8377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14663.009681 secs 2024-07-26T12:03:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:46 smithi181 ceph-mon[35869]: pgmap v8377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:48 smithi097 ceph-mon[32231]: pgmap v8378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:48 smithi115 ceph-mon[35364]: pgmap v8378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:48 smithi181 ceph-mon[35869]: pgmap v8378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:50 smithi097 ceph-mon[32231]: pgmap v8379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:50 smithi115 ceph-mon[35364]: pgmap v8379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:50 smithi181 ceph-mon[35869]: pgmap v8379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14668.009770 secs 2024-07-26T12:03:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14668.009770 secs 2024-07-26T12:03:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14668.009770 secs 2024-07-26T12:03:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:52 smithi115 ceph-mon[35364]: pgmap v8380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:52 smithi181 ceph-mon[35869]: pgmap v8380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:52 smithi097 ceph-mon[32231]: pgmap v8380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:53.659+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:53.660+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:53.661+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:53.661+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:03:53.661+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:54 smithi115 ceph-mon[35364]: pgmap v8381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:54 smithi181 ceph-mon[35869]: pgmap v8381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:54 smithi097 ceph-mon[32231]: pgmap v8381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14673.009907 secs 2024-07-26T12:03:56.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:56 smithi115 ceph-mon[35364]: pgmap v8382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14673.009907 secs 2024-07-26T12:03:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:56 smithi181 ceph-mon[35869]: pgmap v8382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14673.009907 secs 2024-07-26T12:03:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:56 smithi097 ceph-mon[32231]: pgmap v8382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:03:58 smithi115 ceph-mon[35364]: pgmap v8383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:03:58 smithi181 ceph-mon[35869]: pgmap v8383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:03:58 smithi097 ceph-mon[32231]: pgmap v8383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:00 smithi115 ceph-mon[35364]: pgmap v8384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:00 smithi181 ceph-mon[35869]: pgmap v8384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:00 smithi097 ceph-mon[32231]: pgmap v8384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14678.010049 secs 2024-07-26T12:04:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14678.010049 secs 2024-07-26T12:04:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14678.010049 secs 2024-07-26T12:04:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:02 smithi115 ceph-mon[35364]: pgmap v8385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:02 smithi181 ceph-mon[35869]: pgmap v8385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:02 smithi097 ceph-mon[32231]: pgmap v8385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:04 smithi115 ceph-mon[35364]: pgmap v8386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:04 smithi181 ceph-mon[35869]: pgmap v8386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:04 smithi097 ceph-mon[32231]: pgmap v8386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14683.010187 secs 2024-07-26T12:04:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:06 smithi115 ceph-mon[35364]: pgmap v8387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14683.010187 secs 2024-07-26T12:04:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:06 smithi181 ceph-mon[35869]: pgmap v8387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14683.010187 secs 2024-07-26T12:04:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:06 smithi097 ceph-mon[32231]: pgmap v8387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:07.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:04:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:04:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:07.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:04:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:08.663+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:04:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:08.664+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:04:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:08.664+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:04:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:08.664+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:04:08.667 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:08.664+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:08 smithi115 ceph-mon[35364]: pgmap v8388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:04:08.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:08 smithi115 ceph-mon[35364]: pgmap v8389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:08 smithi181 ceph-mon[35869]: pgmap v8388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:04:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:08 smithi181 ceph-mon[35869]: pgmap v8389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:08 smithi097 ceph-mon[32231]: pgmap v8388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:04:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:08 smithi097 ceph-mon[32231]: pgmap v8389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:09.506+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:04:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:09.506+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:04:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:09.507+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:04:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:09.507+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:04:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:10 smithi115 ceph-mon[35364]: pgmap v8390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:04:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:10 smithi181 ceph-mon[35869]: pgmap v8390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:04:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:10 smithi097 ceph-mon[32231]: pgmap v8390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:04:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14688.010351 secs 2024-07-26T12:04:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14688.010351 secs 2024-07-26T12:04:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14688.010351 secs 2024-07-26T12:04:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:12 smithi115 ceph-mon[35364]: pgmap v8391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:12 smithi181 ceph-mon[35869]: pgmap v8391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:12 smithi097 ceph-mon[32231]: pgmap v8391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:13 smithi115 ceph-mon[35364]: pgmap v8392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:13 smithi181 ceph-mon[35869]: pgmap v8392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:13 smithi097 ceph-mon[32231]: pgmap v8392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:15 smithi115 ceph-mon[35364]: pgmap v8393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14693.010487 secs 2024-07-26T12:04:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:15 smithi181 ceph-mon[35869]: pgmap v8393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14693.010487 secs 2024-07-26T12:04:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:15 smithi097 ceph-mon[32231]: pgmap v8393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14693.010487 secs 2024-07-26T12:04:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:17 smithi115 ceph-mon[35364]: pgmap v8394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:17 smithi181 ceph-mon[35869]: pgmap v8394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:17 smithi097 ceph-mon[32231]: pgmap v8394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:19 smithi115 ceph-mon[35364]: pgmap v8395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:19 smithi181 ceph-mon[35869]: pgmap v8395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:19 smithi097 ceph-mon[32231]: pgmap v8395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14698.010649 secs 2024-07-26T12:04:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14698.010649 secs 2024-07-26T12:04:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14698.010649 secs 2024-07-26T12:04:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:21 smithi115 ceph-mon[35364]: pgmap v8396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:21 smithi181 ceph-mon[35869]: pgmap v8396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:22.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:21 smithi097 ceph-mon[32231]: pgmap v8396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:23.666+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:04:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:23.667+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:04:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:23.667+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:04:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:23.667+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:04:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:23.667+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:23 smithi115 ceph-mon[35364]: pgmap v8397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:23 smithi181 ceph-mon[35869]: pgmap v8397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:23 smithi097 ceph-mon[32231]: pgmap v8397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:25 smithi115 ceph-mon[35364]: pgmap v8398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14703.010785 secs 2024-07-26T12:04:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:25 smithi181 ceph-mon[35869]: pgmap v8398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14703.010785 secs 2024-07-26T12:04:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:25 smithi097 ceph-mon[32231]: pgmap v8398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14703.010785 secs 2024-07-26T12:04:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:27 smithi115 ceph-mon[35364]: pgmap v8399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:27 smithi181 ceph-mon[35869]: pgmap v8399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:27 smithi097 ceph-mon[32231]: pgmap v8399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:29 smithi115 ceph-mon[35364]: pgmap v8400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:29 smithi181 ceph-mon[35869]: pgmap v8400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:29 smithi097 ceph-mon[32231]: pgmap v8400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14708.010923 secs 2024-07-26T12:04:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14708.010923 secs 2024-07-26T12:04:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14708.010923 secs 2024-07-26T12:04:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:31 smithi115 ceph-mon[35364]: pgmap v8401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:31 smithi181 ceph-mon[35869]: pgmap v8401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:31 smithi097 ceph-mon[32231]: pgmap v8401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:34.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:33 smithi115 ceph-mon[35364]: pgmap v8402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:33 smithi181 ceph-mon[35869]: pgmap v8402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:33 smithi097 ceph-mon[32231]: pgmap v8402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:35 smithi115 ceph-mon[35364]: pgmap v8403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:36.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14713.011070 secs 2024-07-26T12:04:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:35 smithi181 ceph-mon[35869]: pgmap v8403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14713.011070 secs 2024-07-26T12:04:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:35 smithi097 ceph-mon[32231]: pgmap v8403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14713.011070 secs 2024-07-26T12:04:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:37 smithi115 ceph-mon[35364]: pgmap v8404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:37 smithi181 ceph-mon[35869]: pgmap v8404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:37 smithi097 ceph-mon[32231]: pgmap v8404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:38.669 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:38.668+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:04:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:38.669+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:04:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:38.669+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:04:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:38.669+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:04:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:38.669+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:39 smithi115 ceph-mon[35364]: pgmap v8405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:39 smithi181 ceph-mon[35869]: pgmap v8405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:39 smithi097 ceph-mon[32231]: pgmap v8405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14718.011257 secs 2024-07-26T12:04:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14718.011257 secs 2024-07-26T12:04:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14718.011257 secs 2024-07-26T12:04:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:41 smithi115 ceph-mon[35364]: pgmap v8406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:41 smithi181 ceph-mon[35869]: pgmap v8406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:41 smithi097 ceph-mon[32231]: pgmap v8406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:43 smithi115 ceph-mon[35364]: pgmap v8407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:43 smithi181 ceph-mon[35869]: pgmap v8407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:43 smithi097 ceph-mon[32231]: pgmap v8407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:45 smithi181 ceph-mon[35869]: pgmap v8408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14723.011371 secs 2024-07-26T12:04:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:45 smithi097 ceph-mon[32231]: pgmap v8408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14723.011371 secs 2024-07-26T12:04:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:45 smithi115 ceph-mon[35364]: pgmap v8408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:46.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14723.011371 secs 2024-07-26T12:04:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:47 smithi097 ceph-mon[32231]: pgmap v8409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:48.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:47 smithi115 ceph-mon[35364]: pgmap v8409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:47 smithi181 ceph-mon[35869]: pgmap v8409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:49 smithi097 ceph-mon[32231]: pgmap v8410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:49 smithi115 ceph-mon[35364]: pgmap v8410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:49 smithi181 ceph-mon[35869]: pgmap v8410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14728.011505 secs 2024-07-26T12:04:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14728.011505 secs 2024-07-26T12:04:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14728.011505 secs 2024-07-26T12:04:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:52 smithi115 ceph-mon[35364]: pgmap v8411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:52 smithi181 ceph-mon[35869]: pgmap v8411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:52 smithi097 ceph-mon[32231]: pgmap v8411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:53.671+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:53.673+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:53.673+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:53.673+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:04:53.673+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:54.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:54 smithi115 ceph-mon[35364]: pgmap v8412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:54 smithi181 ceph-mon[35869]: pgmap v8412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:54 smithi097 ceph-mon[32231]: pgmap v8412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:56 smithi115 ceph-mon[35364]: pgmap v8413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:04:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14733.011640 secs 2024-07-26T12:04:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:56 smithi181 ceph-mon[35869]: pgmap v8413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:04:56.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14733.011640 secs 2024-07-26T12:04:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:56 smithi097 ceph-mon[32231]: pgmap v8413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:04:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14733.011640 secs 2024-07-26T12:04:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:04:58 smithi115 ceph-mon[35364]: pgmap v8414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:04:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:04:58 smithi181 ceph-mon[35869]: pgmap v8414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:04:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:04:58 smithi097 ceph-mon[32231]: pgmap v8414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:05:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:00 smithi115 ceph-mon[35364]: pgmap v8415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:05:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:00 smithi181 ceph-mon[35869]: pgmap v8415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:05:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:00 smithi097 ceph-mon[32231]: pgmap v8415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:05:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14738.011788 secs 2024-07-26T12:05:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14738.011788 secs 2024-07-26T12:05:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14738.011788 secs 2024-07-26T12:05:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:02 smithi115 ceph-mon[35364]: pgmap v8416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:05:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:02 smithi181 ceph-mon[35869]: pgmap v8416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:05:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:02 smithi097 ceph-mon[32231]: pgmap v8416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:05:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:04 smithi115 ceph-mon[35364]: pgmap v8417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:05:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:04 smithi181 ceph-mon[35869]: pgmap v8417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:05:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:04 smithi097 ceph-mon[32231]: pgmap v8417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:05:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:06 smithi115 ceph-mon[35364]: pgmap v8418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14743.011936 secs 2024-07-26T12:05:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:06 smithi181 ceph-mon[35869]: pgmap v8418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14743.011936 secs 2024-07-26T12:05:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:06 smithi097 ceph-mon[32231]: pgmap v8418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14743.011936 secs 2024-07-26T12:05:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: pgmap v8419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:08.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:05:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: pgmap v8419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:08.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: pgmap v8419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:08.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:08.503 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:05:08.681 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:08.680+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:08.681+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:05:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:08.681+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:05:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:08.681+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:08.681+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:09 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:05:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:09 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:05:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:09 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:05:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:09 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:05:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:09 smithi115 ceph-mon[35364]: pgmap v8420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:05:09.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:09 smithi115 ceph-mon[35364]: pgmap v8421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:09 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:05:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:09 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:05:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:09 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:05:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:09 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:05:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:09 smithi181 ceph-mon[35869]: pgmap v8420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:05:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:09 smithi181 ceph-mon[35869]: pgmap v8421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:09 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:05:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:09 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:05:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:09 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:05:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:09 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:05:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:09 smithi097 ceph-mon[32231]: pgmap v8420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:05:09.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:09 smithi097 ceph-mon[32231]: pgmap v8421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:11 smithi115 ceph-mon[35364]: pgmap v8422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14748.012102 secs 2024-07-26T12:05:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:11 smithi181 ceph-mon[35869]: pgmap v8422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14748.012102 secs 2024-07-26T12:05:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:11 smithi097 ceph-mon[32231]: pgmap v8422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14748.012102 secs 2024-07-26T12:05:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:13 smithi115 ceph-mon[35364]: pgmap v8423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:05:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:13 smithi181 ceph-mon[35869]: pgmap v8423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:05:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:13 smithi097 ceph-mon[32231]: pgmap v8423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:05:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:15 smithi115 ceph-mon[35364]: pgmap v8424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:15 smithi181 ceph-mon[35869]: pgmap v8424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:15 smithi097 ceph-mon[32231]: pgmap v8424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14753.012283 secs 2024-07-26T12:05:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14753.012283 secs 2024-07-26T12:05:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14753.012283 secs 2024-07-26T12:05:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:17 smithi115 ceph-mon[35364]: pgmap v8425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:17 smithi181 ceph-mon[35869]: pgmap v8425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:17 smithi097 ceph-mon[32231]: pgmap v8425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:19 smithi115 ceph-mon[35364]: pgmap v8426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:19 smithi181 ceph-mon[35869]: pgmap v8426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:19 smithi097 ceph-mon[32231]: pgmap v8426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:21 smithi115 ceph-mon[35364]: pgmap v8427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14758.012420 secs 2024-07-26T12:05:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:21 smithi181 ceph-mon[35869]: pgmap v8427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14758.012420 secs 2024-07-26T12:05:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:21 smithi097 ceph-mon[32231]: pgmap v8427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14758.012420 secs 2024-07-26T12:05:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:23 smithi115 ceph-mon[35364]: pgmap v8428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:23 smithi181 ceph-mon[35869]: pgmap v8428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:23 smithi097 ceph-mon[32231]: pgmap v8428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:23.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:23.683+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:23.685+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:05:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:23.685+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:05:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:23.685+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:23.685+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:25 smithi115 ceph-mon[35364]: pgmap v8429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:25 smithi181 ceph-mon[35869]: pgmap v8429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:25 smithi097 ceph-mon[32231]: pgmap v8429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14763.012560 secs 2024-07-26T12:05:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14763.012560 secs 2024-07-26T12:05:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14763.012560 secs 2024-07-26T12:05:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:27 smithi115 ceph-mon[35364]: pgmap v8430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:27 smithi181 ceph-mon[35869]: pgmap v8430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:27 smithi097 ceph-mon[32231]: pgmap v8430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:29 smithi115 ceph-mon[35364]: pgmap v8431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:29.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:29 smithi181 ceph-mon[35869]: pgmap v8431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:29 smithi097 ceph-mon[32231]: pgmap v8431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:31 smithi115 ceph-mon[35364]: pgmap v8432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14768.012713 secs 2024-07-26T12:05:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:31 smithi181 ceph-mon[35869]: pgmap v8432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14768.012713 secs 2024-07-26T12:05:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:31 smithi097 ceph-mon[32231]: pgmap v8432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14768.012713 secs 2024-07-26T12:05:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:33 smithi115 ceph-mon[35364]: pgmap v8433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:33.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:33 smithi181 ceph-mon[35869]: pgmap v8433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:33 smithi097 ceph-mon[32231]: pgmap v8433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:35 smithi115 ceph-mon[35364]: pgmap v8434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:35 smithi181 ceph-mon[35869]: pgmap v8434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:35 smithi097 ceph-mon[32231]: pgmap v8434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14773.012856 secs 2024-07-26T12:05:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14773.012856 secs 2024-07-26T12:05:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14773.012856 secs 2024-07-26T12:05:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:37 smithi097 ceph-mon[32231]: pgmap v8435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:37 smithi115 ceph-mon[35364]: pgmap v8435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:37 smithi181 ceph-mon[35869]: pgmap v8435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:38.692+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:38.694+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:05:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:38.694+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:05:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:38.694+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:38.694+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:39 smithi097 ceph-mon[32231]: pgmap v8436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:39.523 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:39.522+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:05:39.523 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:39.522+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:05:39.524 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:39.523+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:05:39.524 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:39.523+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:05:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:39 smithi115 ceph-mon[35364]: pgmap v8436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:39 smithi181 ceph-mon[35869]: pgmap v8436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:41 smithi097 ceph-mon[32231]: pgmap v8437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14778.013001 secs 2024-07-26T12:05:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:41 smithi115 ceph-mon[35364]: pgmap v8437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14778.013001 secs 2024-07-26T12:05:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:41 smithi181 ceph-mon[35869]: pgmap v8437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14778.013001 secs 2024-07-26T12:05:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:43 smithi115 ceph-mon[35364]: pgmap v8438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:43 smithi181 ceph-mon[35869]: pgmap v8438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:43 smithi097 ceph-mon[32231]: pgmap v8438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:45 smithi115 ceph-mon[35364]: pgmap v8439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:45 smithi181 ceph-mon[35869]: pgmap v8439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:45 smithi097 ceph-mon[32231]: pgmap v8439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14783.013145 secs 2024-07-26T12:05:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14783.013145 secs 2024-07-26T12:05:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14783.013145 secs 2024-07-26T12:05:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:47 smithi115 ceph-mon[35364]: pgmap v8440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:47 smithi181 ceph-mon[35869]: pgmap v8440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:47 smithi097 ceph-mon[32231]: pgmap v8440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:49.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:49 smithi181 ceph-mon[35869]: pgmap v8441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:49.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:49 smithi097 ceph-mon[32231]: pgmap v8441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:49 smithi115 ceph-mon[35364]: pgmap v8441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:51 smithi115 ceph-mon[35364]: pgmap v8442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14788.013285 secs 2024-07-26T12:05:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:51 smithi181 ceph-mon[35869]: pgmap v8442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14788.013285 secs 2024-07-26T12:05:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:51 smithi097 ceph-mon[32231]: pgmap v8442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14788.013285 secs 2024-07-26T12:05:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:53.696+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:05:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:53.697+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:05:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:53.697+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:05:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:53.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:05:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:05:53.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:53 smithi115 ceph-mon[35364]: pgmap v8443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:53 smithi181 ceph-mon[35869]: pgmap v8443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:53 smithi097 ceph-mon[32231]: pgmap v8443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:55 smithi115 ceph-mon[35364]: pgmap v8444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:55 smithi181 ceph-mon[35869]: pgmap v8444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:55 smithi097 ceph-mon[32231]: pgmap v8444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14793.013430 secs 2024-07-26T12:05:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14793.013430 secs 2024-07-26T12:05:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14793.013430 secs 2024-07-26T12:05:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:57 smithi115 ceph-mon[35364]: pgmap v8445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:57 smithi181 ceph-mon[35869]: pgmap v8445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:57 smithi097 ceph-mon[32231]: pgmap v8445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:05:59 smithi115 ceph-mon[35364]: pgmap v8446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:59.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:05:59 smithi181 ceph-mon[35869]: pgmap v8446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:05:59 smithi097 ceph-mon[32231]: pgmap v8446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:01 smithi115 ceph-mon[35364]: pgmap v8447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:06:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14798.013525 secs 2024-07-26T12:06:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:01 smithi181 ceph-mon[35869]: pgmap v8447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:06:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14798.013525 secs 2024-07-26T12:06:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:01 smithi097 ceph-mon[32231]: pgmap v8447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:06:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14798.013525 secs 2024-07-26T12:06:03.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:03 smithi115 ceph-mon[35364]: pgmap v8448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:06:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:03 smithi181 ceph-mon[35869]: pgmap v8448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:06:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:03 smithi097 ceph-mon[32231]: pgmap v8448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:06:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:05 smithi115 ceph-mon[35364]: pgmap v8449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:06:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:05 smithi181 ceph-mon[35869]: pgmap v8449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:06:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:05 smithi097 ceph-mon[32231]: pgmap v8449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:06:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14803.013667 secs 2024-07-26T12:06:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14803.013667 secs 2024-07-26T12:06:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14803.013667 secs 2024-07-26T12:06:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:07 smithi115 ceph-mon[35364]: pgmap v8450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:07 smithi181 ceph-mon[35869]: pgmap v8450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:07 smithi097 ceph-mon[32231]: pgmap v8450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:08.704+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:06:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:08.705+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:06:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:08.705+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:06:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:08.705+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:06:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:08.705+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:09 smithi115 ceph-mon[35364]: pgmap v8451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:06:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:09 smithi181 ceph-mon[35869]: pgmap v8451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:06:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:09 smithi097 ceph-mon[32231]: pgmap v8451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:06:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:10.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:06:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:10.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:06:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:10.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:06:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:11 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:06:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:11 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:06:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:11 smithi115 ceph-mon[35364]: pgmap v8452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:06:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:11 smithi115 ceph-mon[35364]: pgmap v8453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14808.013767 secs 2024-07-26T12:06:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:11 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:06:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:11 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:06:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:11 smithi181 ceph-mon[35869]: pgmap v8452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:06:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:11 smithi181 ceph-mon[35869]: pgmap v8453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14808.013767 secs 2024-07-26T12:06:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:11 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:06:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:11 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:06:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:11 smithi097 ceph-mon[32231]: pgmap v8452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:06:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:11 smithi097 ceph-mon[32231]: pgmap v8453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14808.013767 secs 2024-07-26T12:06:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:12 smithi115 ceph-mon[35364]: pgmap v8454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:12 smithi181 ceph-mon[35869]: pgmap v8454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:12 smithi097 ceph-mon[32231]: pgmap v8454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:14 smithi115 ceph-mon[35364]: pgmap v8455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:06:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:14 smithi181 ceph-mon[35869]: pgmap v8455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:06:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:14 smithi097 ceph-mon[32231]: pgmap v8455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:06:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14813.013913 secs 2024-07-26T12:06:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14813.013913 secs 2024-07-26T12:06:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14813.013913 secs 2024-07-26T12:06:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:16 smithi181 ceph-mon[35869]: pgmap v8456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:16 smithi097 ceph-mon[32231]: pgmap v8456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:16 smithi115 ceph-mon[35364]: pgmap v8456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:18 smithi097 ceph-mon[32231]: pgmap v8457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:18 smithi115 ceph-mon[35364]: pgmap v8457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:18 smithi181 ceph-mon[35869]: pgmap v8457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:20 smithi097 ceph-mon[32231]: pgmap v8458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:20.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14818.014058 secs 2024-07-26T12:06:20.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:20 smithi115 ceph-mon[35364]: pgmap v8458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:20.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14818.014058 secs 2024-07-26T12:06:20.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:20 smithi181 ceph-mon[35869]: pgmap v8458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:20.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14818.014058 secs 2024-07-26T12:06:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:22 smithi097 ceph-mon[32231]: pgmap v8459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:22 smithi115 ceph-mon[35364]: pgmap v8459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:22 smithi181 ceph-mon[35869]: pgmap v8459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:23.708+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:06:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:23.709+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:06:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:23.709+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:06:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:23.709+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:06:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:23.709+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:24 smithi115 ceph-mon[35364]: pgmap v8460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:24 smithi181 ceph-mon[35869]: pgmap v8460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:24 smithi097 ceph-mon[32231]: pgmap v8460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14823.014217 secs 2024-07-26T12:06:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14823.014217 secs 2024-07-26T12:06:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14823.014217 secs 2024-07-26T12:06:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:26 smithi115 ceph-mon[35364]: pgmap v8461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:26 smithi181 ceph-mon[35869]: pgmap v8461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:26 smithi097 ceph-mon[32231]: pgmap v8461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:28 smithi115 ceph-mon[35364]: pgmap v8462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:28 smithi181 ceph-mon[35869]: pgmap v8462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:28 smithi097 ceph-mon[32231]: pgmap v8462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:30 smithi115 ceph-mon[35364]: pgmap v8463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14828.014320 secs 2024-07-26T12:06:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:30 smithi181 ceph-mon[35869]: pgmap v8463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14828.014320 secs 2024-07-26T12:06:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:30 smithi097 ceph-mon[32231]: pgmap v8463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14828.014320 secs 2024-07-26T12:06:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:32 smithi115 ceph-mon[35364]: pgmap v8464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:32 smithi181 ceph-mon[35869]: pgmap v8464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:32 smithi097 ceph-mon[32231]: pgmap v8464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:34 smithi115 ceph-mon[35364]: pgmap v8465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:34 smithi181 ceph-mon[35869]: pgmap v8465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:34 smithi097 ceph-mon[32231]: pgmap v8465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14833.014459 secs 2024-07-26T12:06:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14833.014459 secs 2024-07-26T12:06:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14833.014459 secs 2024-07-26T12:06:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:36 smithi115 ceph-mon[35364]: pgmap v8466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:36 smithi181 ceph-mon[35869]: pgmap v8466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:36 smithi097 ceph-mon[32231]: pgmap v8466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:38.715+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:06:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:38.716+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:06:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:38.717+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:06:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:38.717+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:06:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:38.717+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:38 smithi115 ceph-mon[35364]: pgmap v8467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:38 smithi181 ceph-mon[35869]: pgmap v8467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:38 smithi097 ceph-mon[32231]: pgmap v8467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:40 smithi115 ceph-mon[35364]: pgmap v8468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14838.014555 secs 2024-07-26T12:06:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:40 smithi181 ceph-mon[35869]: pgmap v8468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14838.014555 secs 2024-07-26T12:06:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:40 smithi097 ceph-mon[32231]: pgmap v8468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14838.014555 secs 2024-07-26T12:06:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:42 smithi115 ceph-mon[35364]: pgmap v8469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:42 smithi181 ceph-mon[35869]: pgmap v8469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:42 smithi097 ceph-mon[32231]: pgmap v8469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:44 smithi115 ceph-mon[35364]: pgmap v8470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:44 smithi181 ceph-mon[35869]: pgmap v8470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:44 smithi097 ceph-mon[32231]: pgmap v8470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14843.014726 secs 2024-07-26T12:06:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14843.014726 secs 2024-07-26T12:06:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14843.014726 secs 2024-07-26T12:06:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:46 smithi115 ceph-mon[35364]: pgmap v8471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:46 smithi181 ceph-mon[35869]: pgmap v8471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:46 smithi097 ceph-mon[32231]: pgmap v8471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:48.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:48 smithi115 ceph-mon[35364]: pgmap v8472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:48 smithi181 ceph-mon[35869]: pgmap v8472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:48 smithi097 ceph-mon[32231]: pgmap v8472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:50 smithi115 ceph-mon[35364]: pgmap v8473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14848.014869 secs 2024-07-26T12:06:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:50 smithi181 ceph-mon[35869]: pgmap v8473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14848.014869 secs 2024-07-26T12:06:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:50 smithi097 ceph-mon[32231]: pgmap v8473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14848.014869 secs 2024-07-26T12:06:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:52 smithi115 ceph-mon[35364]: pgmap v8474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:52 smithi181 ceph-mon[35869]: pgmap v8474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:52 smithi097 ceph-mon[32231]: pgmap v8474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:06:53.721 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:53.719+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:06:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:53.720+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:06:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:53.720+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:06:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:53.720+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:06:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:06:53.721+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:54 smithi115 ceph-mon[35364]: pgmap v8475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:54 smithi181 ceph-mon[35869]: pgmap v8475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:54 smithi097 ceph-mon[32231]: pgmap v8475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14853.015006 secs 2024-07-26T12:06:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14853.015006 secs 2024-07-26T12:06:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14853.015006 secs 2024-07-26T12:06:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:56 smithi115 ceph-mon[35364]: pgmap v8476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:56 smithi181 ceph-mon[35869]: pgmap v8476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:56 smithi097 ceph-mon[32231]: pgmap v8476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:06:58 smithi115 ceph-mon[35364]: pgmap v8477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:06:58 smithi181 ceph-mon[35869]: pgmap v8477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:06:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:06:58 smithi097 ceph-mon[32231]: pgmap v8477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:00 smithi115 ceph-mon[35364]: pgmap v8478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14858.015145 secs 2024-07-26T12:07:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:00 smithi181 ceph-mon[35869]: pgmap v8478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14858.015145 secs 2024-07-26T12:07:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:00 smithi097 ceph-mon[32231]: pgmap v8478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14858.015145 secs 2024-07-26T12:07:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:02 smithi115 ceph-mon[35364]: pgmap v8479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:02 smithi181 ceph-mon[35869]: pgmap v8479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:02 smithi097 ceph-mon[32231]: pgmap v8479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:04 smithi115 ceph-mon[35364]: pgmap v8480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:04 smithi181 ceph-mon[35869]: pgmap v8480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:04 smithi097 ceph-mon[32231]: pgmap v8480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14863.015314 secs 2024-07-26T12:07:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14863.015314 secs 2024-07-26T12:07:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14863.015314 secs 2024-07-26T12:07:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:06 smithi115 ceph-mon[35364]: pgmap v8481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:06 smithi181 ceph-mon[35869]: pgmap v8481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:06 smithi097 ceph-mon[32231]: pgmap v8481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:08.730 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:08.728+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:07:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:08.730+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:07:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:08.730+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:07:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:08.730+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:07:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:08.730+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:08 smithi115 ceph-mon[35364]: pgmap v8482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:08 smithi181 ceph-mon[35869]: pgmap v8482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:08 smithi097 ceph-mon[32231]: pgmap v8482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:09.539+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:07:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:09.539+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:07:09.541 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:09.539+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:07:09.541 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:09.540+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:07:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:10 smithi115 ceph-mon[35364]: pgmap v8483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14868.015395 secs 2024-07-26T12:07:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:10 smithi181 ceph-mon[35869]: pgmap v8483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14868.015395 secs 2024-07-26T12:07:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:10 smithi097 ceph-mon[32231]: pgmap v8483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14868.015395 secs 2024-07-26T12:07:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:11.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:11 smithi115 ceph-mon[35364]: pgmap v8484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:07:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:11 smithi115 ceph-mon[35364]: pgmap v8485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:11.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:11 smithi181 ceph-mon[35869]: pgmap v8484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:07:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:11 smithi181 ceph-mon[35869]: pgmap v8485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:11 smithi097 ceph-mon[32231]: pgmap v8484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:07:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:11 smithi097 ceph-mon[32231]: pgmap v8485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:12.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:07:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:13 smithi115 ceph-mon[35364]: pgmap v8486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:13 smithi181 ceph-mon[35869]: pgmap v8486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:13 smithi097 ceph-mon[32231]: pgmap v8486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:15 smithi115 ceph-mon[35364]: pgmap v8487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:15 smithi181 ceph-mon[35869]: pgmap v8487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:15 smithi097 ceph-mon[32231]: pgmap v8487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:07:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14873.015478 secs 2024-07-26T12:07:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14873.015478 secs 2024-07-26T12:07:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14873.015478 secs 2024-07-26T12:07:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:17 smithi115 ceph-mon[35364]: pgmap v8488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:17 smithi181 ceph-mon[35869]: pgmap v8488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:17 smithi097 ceph-mon[32231]: pgmap v8488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:19 smithi115 ceph-mon[35364]: pgmap v8489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:19 smithi181 ceph-mon[35869]: pgmap v8489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:19 smithi097 ceph-mon[32231]: pgmap v8489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14878.015617 secs 2024-07-26T12:07:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14878.015617 secs 2024-07-26T12:07:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14878.015617 secs 2024-07-26T12:07:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:21 smithi115 ceph-mon[35364]: pgmap v8490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:21 smithi181 ceph-mon[35869]: pgmap v8490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:21 smithi097 ceph-mon[32231]: pgmap v8490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:07:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:23 smithi115 ceph-mon[35364]: pgmap v8491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:23 smithi181 ceph-mon[35869]: pgmap v8491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:23.732+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:07:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:23.734+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:07:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:23.734+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:07:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:23.734+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:07:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:23.734+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:23 smithi097 ceph-mon[32231]: pgmap v8491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:25 smithi115 ceph-mon[35364]: pgmap v8492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:25 smithi181 ceph-mon[35869]: pgmap v8492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:25 smithi097 ceph-mon[32231]: pgmap v8492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14883.015762 secs 2024-07-26T12:07:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14883.015762 secs 2024-07-26T12:07:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14883.015762 secs 2024-07-26T12:07:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:27 smithi181 ceph-mon[35869]: pgmap v8493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:27 smithi097 ceph-mon[32231]: pgmap v8493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:27 smithi115 ceph-mon[35364]: pgmap v8493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:29 smithi115 ceph-mon[35364]: pgmap v8494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:29 smithi181 ceph-mon[35869]: pgmap v8494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:29 smithi097 ceph-mon[32231]: pgmap v8494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14888.015904 secs 2024-07-26T12:07:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14888.015904 secs 2024-07-26T12:07:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14888.015904 secs 2024-07-26T12:07:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:31 smithi181 ceph-mon[35869]: pgmap v8495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:31 smithi097 ceph-mon[32231]: pgmap v8495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:31 smithi115 ceph-mon[35364]: pgmap v8495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:33 smithi115 ceph-mon[35364]: pgmap v8496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:33 smithi181 ceph-mon[35869]: pgmap v8496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:33 smithi097 ceph-mon[32231]: pgmap v8496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:35 smithi115 ceph-mon[35364]: pgmap v8497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:35 smithi181 ceph-mon[35869]: pgmap v8497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:35 smithi097 ceph-mon[32231]: pgmap v8497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14893.016061 secs 2024-07-26T12:07:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14893.016061 secs 2024-07-26T12:07:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14893.016061 secs 2024-07-26T12:07:38.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:38 smithi115 ceph-mon[35364]: pgmap v8498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:38 smithi181 ceph-mon[35869]: pgmap v8498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:38.741+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:07:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:38.744+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:07:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:38.744+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:07:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:38.744+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:07:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:38.744+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:38 smithi097 ceph-mon[32231]: pgmap v8498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:40 smithi115 ceph-mon[35364]: pgmap v8499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:40.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14898.016215 secs 2024-07-26T12:07:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:40 smithi181 ceph-mon[35869]: pgmap v8499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14898.016215 secs 2024-07-26T12:07:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:40 smithi097 ceph-mon[32231]: pgmap v8499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14898.016215 secs 2024-07-26T12:07:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:42 smithi181 ceph-mon[35869]: pgmap v8500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:42 smithi097 ceph-mon[32231]: pgmap v8500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:42 smithi115 ceph-mon[35364]: pgmap v8500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:44 smithi097 ceph-mon[32231]: pgmap v8501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:44 smithi115 ceph-mon[35364]: pgmap v8501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:44 smithi181 ceph-mon[35869]: pgmap v8501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14903.016357 secs 2024-07-26T12:07:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14903.016357 secs 2024-07-26T12:07:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14903.016357 secs 2024-07-26T12:07:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:46 smithi097 ceph-mon[32231]: pgmap v8502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:46 smithi115 ceph-mon[35364]: pgmap v8502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:46 smithi181 ceph-mon[35869]: pgmap v8502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:48 smithi115 ceph-mon[35364]: pgmap v8503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:48 smithi181 ceph-mon[35869]: pgmap v8503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:48 smithi097 ceph-mon[32231]: pgmap v8503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:50.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:50 smithi115 ceph-mon[35364]: pgmap v8504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:50.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14908.016442 secs 2024-07-26T12:07:50.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:50 smithi181 ceph-mon[35869]: pgmap v8504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:50.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14908.016442 secs 2024-07-26T12:07:50.804 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:50 smithi097 ceph-mon[32231]: pgmap v8504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:50.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14908.016442 secs 2024-07-26T12:07:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:52 smithi115 ceph-mon[35364]: pgmap v8505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:52 smithi181 ceph-mon[35869]: pgmap v8505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:52 smithi097 ceph-mon[32231]: pgmap v8505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:07:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:53.747+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:07:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:53.748+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:07:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:53.748+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:07:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:53.748+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:07:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:07:53.748+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:54 smithi115 ceph-mon[35364]: pgmap v8506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:54 smithi181 ceph-mon[35869]: pgmap v8506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:54 smithi097 ceph-mon[32231]: pgmap v8506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14913.016585 secs 2024-07-26T12:07:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14913.016585 secs 2024-07-26T12:07:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14913.016585 secs 2024-07-26T12:07:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:56 smithi115 ceph-mon[35364]: pgmap v8507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:56 smithi181 ceph-mon[35869]: pgmap v8507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:56 smithi097 ceph-mon[32231]: pgmap v8507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:07:58 smithi115 ceph-mon[35364]: pgmap v8508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:07:58 smithi181 ceph-mon[35869]: pgmap v8508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:07:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:07:58 smithi097 ceph-mon[32231]: pgmap v8508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:00 smithi115 ceph-mon[35364]: pgmap v8509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14918.016713 secs 2024-07-26T12:08:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:00 smithi181 ceph-mon[35869]: pgmap v8509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14918.016713 secs 2024-07-26T12:08:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:00 smithi097 ceph-mon[32231]: pgmap v8509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14918.016713 secs 2024-07-26T12:08:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:02 smithi115 ceph-mon[35364]: pgmap v8510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:02 smithi181 ceph-mon[35869]: pgmap v8510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:02 smithi097 ceph-mon[32231]: pgmap v8510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:04 smithi115 ceph-mon[35364]: pgmap v8511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:04 smithi181 ceph-mon[35869]: pgmap v8511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:04 smithi097 ceph-mon[32231]: pgmap v8511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14923.016888 secs 2024-07-26T12:08:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14923.016888 secs 2024-07-26T12:08:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14923.016888 secs 2024-07-26T12:08:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:06 smithi115 ceph-mon[35364]: pgmap v8512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:06 smithi181 ceph-mon[35869]: pgmap v8512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:06 smithi097 ceph-mon[32231]: pgmap v8512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:08.755+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:08:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:08.757+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:08:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:08.757+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:08:08.759 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:08.757+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:08:08.759 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:08.757+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:08 smithi115 ceph-mon[35364]: pgmap v8513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:08 smithi181 ceph-mon[35869]: pgmap v8513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:08 smithi097 ceph-mon[32231]: pgmap v8513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:10 smithi115 ceph-mon[35364]: pgmap v8514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14928.017029 secs 2024-07-26T12:08:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:10 smithi181 ceph-mon[35869]: pgmap v8514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14928.017029 secs 2024-07-26T12:08:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:10 smithi097 ceph-mon[32231]: pgmap v8514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14928.017029 secs 2024-07-26T12:08:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:12 smithi115 ceph-mon[35364]: pgmap v8515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:12 smithi181 ceph-mon[35869]: pgmap v8515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:12 smithi097 ceph-mon[32231]: pgmap v8515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:13.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:14 smithi115 ceph-mon[35364]: pgmap v8516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:14 smithi115 ceph-mon[35364]: pgmap v8517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:14.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:14 smithi181 ceph-mon[35869]: pgmap v8516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:14 smithi181 ceph-mon[35869]: pgmap v8517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:14.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:14 smithi097 ceph-mon[32231]: pgmap v8516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:14 smithi097 ceph-mon[32231]: pgmap v8517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:14.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:08:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:15 smithi115 ceph-mon[35364]: pgmap v8518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:15 smithi181 ceph-mon[35869]: pgmap v8518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:15 smithi097 ceph-mon[32231]: pgmap v8518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14933.017168 secs 2024-07-26T12:08:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14933.017168 secs 2024-07-26T12:08:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14933.017168 secs 2024-07-26T12:08:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:17 smithi097 ceph-mon[32231]: pgmap v8519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:17 smithi115 ceph-mon[35364]: pgmap v8519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:17 smithi181 ceph-mon[35869]: pgmap v8519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:19 smithi115 ceph-mon[35364]: pgmap v8520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:19 smithi181 ceph-mon[35869]: pgmap v8520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:19 smithi097 ceph-mon[32231]: pgmap v8520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14938.017316 secs 2024-07-26T12:08:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14938.017316 secs 2024-07-26T12:08:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14938.017316 secs 2024-07-26T12:08:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:22 smithi115 ceph-mon[35364]: pgmap v8521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:22 smithi181 ceph-mon[35869]: pgmap v8521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:22 smithi097 ceph-mon[32231]: pgmap v8521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:08:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:23 smithi115 ceph-mon[35364]: pgmap v8522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:23 smithi181 ceph-mon[35869]: pgmap v8522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:23 smithi097 ceph-mon[32231]: pgmap v8522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:23.759+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:08:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:23.760+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:08:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:23.760+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:08:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:23.760+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:08:23.762 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:23.760+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:25 smithi115 ceph-mon[35364]: pgmap v8523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:25 smithi181 ceph-mon[35869]: pgmap v8523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:25 smithi097 ceph-mon[32231]: pgmap v8523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14943.017482 secs 2024-07-26T12:08:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14943.017482 secs 2024-07-26T12:08:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14943.017482 secs 2024-07-26T12:08:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:27 smithi181 ceph-mon[35869]: pgmap v8524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:27 smithi097 ceph-mon[32231]: pgmap v8524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:27 smithi115 ceph-mon[35364]: pgmap v8524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:30 smithi115 ceph-mon[35364]: pgmap v8525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:30 smithi181 ceph-mon[35869]: pgmap v8525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:30.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:30 smithi097 ceph-mon[32231]: pgmap v8525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14948.017657 secs 2024-07-26T12:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14948.017657 secs 2024-07-26T12:08:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14948.017657 secs 2024-07-26T12:08:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:32 smithi097 ceph-mon[32231]: pgmap v8526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:32 smithi115 ceph-mon[35364]: pgmap v8526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:32 smithi181 ceph-mon[35869]: pgmap v8526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:34 smithi097 ceph-mon[32231]: pgmap v8527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:34 smithi115 ceph-mon[35364]: pgmap v8527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:34 smithi181 ceph-mon[35869]: pgmap v8527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14953.017799 secs 2024-07-26T12:08:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14953.017799 secs 2024-07-26T12:08:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14953.017799 secs 2024-07-26T12:08:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:36 smithi115 ceph-mon[35364]: pgmap v8528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:36 smithi181 ceph-mon[35869]: pgmap v8528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:36 smithi097 ceph-mon[32231]: pgmap v8528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:38.768+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:08:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:38.769+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:08:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:38.769+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:08:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:38.769+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:08:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:38.770+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:38 smithi115 ceph-mon[35364]: pgmap v8529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:38 smithi181 ceph-mon[35869]: pgmap v8529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:38 smithi097 ceph-mon[32231]: pgmap v8529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:40 smithi115 ceph-mon[35364]: pgmap v8530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14958.019185 secs 2024-07-26T12:08:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:40 smithi181 ceph-mon[35869]: pgmap v8530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14958.019185 secs 2024-07-26T12:08:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:40 smithi097 ceph-mon[32231]: pgmap v8530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14958.019185 secs 2024-07-26T12:08:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:42 smithi115 ceph-mon[35364]: pgmap v8531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:42 smithi181 ceph-mon[35869]: pgmap v8531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:42 smithi097 ceph-mon[32231]: pgmap v8531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:44 smithi115 ceph-mon[35364]: pgmap v8532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:44 smithi181 ceph-mon[35869]: pgmap v8532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:44 smithi097 ceph-mon[32231]: pgmap v8532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14963.019332 secs 2024-07-26T12:08:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14963.019332 secs 2024-07-26T12:08:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14963.019332 secs 2024-07-26T12:08:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:46 smithi115 ceph-mon[35364]: pgmap v8533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:46 smithi181 ceph-mon[35869]: pgmap v8533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:46 smithi097 ceph-mon[32231]: pgmap v8533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:48 smithi115 ceph-mon[35364]: pgmap v8534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:48 smithi181 ceph-mon[35869]: pgmap v8534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:48 smithi097 ceph-mon[32231]: pgmap v8534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:50.805 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:50 smithi181 ceph-mon[35869]: pgmap v8535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:50.805 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14968.019473 secs 2024-07-26T12:08:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:50 smithi115 ceph-mon[35364]: pgmap v8535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14968.019473 secs 2024-07-26T12:08:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:50 smithi097 ceph-mon[32231]: pgmap v8535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14968.019473 secs 2024-07-26T12:08:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:52 smithi115 ceph-mon[35364]: pgmap v8536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:52 smithi181 ceph-mon[35869]: pgmap v8536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:52 smithi097 ceph-mon[32231]: pgmap v8536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:08:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:53.772+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:08:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:53.773+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:08:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:53.773+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:08:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:53.773+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:08:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:08:53.773+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:54 smithi115 ceph-mon[35364]: pgmap v8537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:54 smithi181 ceph-mon[35869]: pgmap v8537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:54 smithi097 ceph-mon[32231]: pgmap v8537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14973.019623 secs 2024-07-26T12:08:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14973.019623 secs 2024-07-26T12:08:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14973.019623 secs 2024-07-26T12:08:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:56 smithi115 ceph-mon[35364]: pgmap v8538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:56 smithi181 ceph-mon[35869]: pgmap v8538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:56 smithi097 ceph-mon[32231]: pgmap v8538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:08:58 smithi115 ceph-mon[35364]: pgmap v8539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:08:58 smithi181 ceph-mon[35869]: pgmap v8539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:08:58 smithi097 ceph-mon[32231]: pgmap v8539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:00 smithi115 ceph-mon[35364]: pgmap v8540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14978.019779 secs 2024-07-26T12:09:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:00 smithi181 ceph-mon[35869]: pgmap v8540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14978.019779 secs 2024-07-26T12:09:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:00 smithi097 ceph-mon[32231]: pgmap v8540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14978.019779 secs 2024-07-26T12:09:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:02 smithi115 ceph-mon[35364]: pgmap v8541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:02 smithi181 ceph-mon[35869]: pgmap v8541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:02 smithi097 ceph-mon[32231]: pgmap v8541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:04 smithi115 ceph-mon[35364]: pgmap v8542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:04 smithi181 ceph-mon[35869]: pgmap v8542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:04 smithi097 ceph-mon[32231]: pgmap v8542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14983.019866 secs 2024-07-26T12:09:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14983.019866 secs 2024-07-26T12:09:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14983.019866 secs 2024-07-26T12:09:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:06 smithi115 ceph-mon[35364]: pgmap v8543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:06 smithi181 ceph-mon[35869]: pgmap v8543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:06 smithi097 ceph-mon[32231]: pgmap v8543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:08.781+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:09:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:08.781+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:09:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:08.781+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:09:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:08.781+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:09:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:08.781+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:08 smithi115 ceph-mon[35364]: pgmap v8544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:08 smithi181 ceph-mon[35869]: pgmap v8544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:08 smithi097 ceph-mon[32231]: pgmap v8544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:10 smithi115 ceph-mon[35364]: pgmap v8545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:10.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14988.020026 secs 2024-07-26T12:09:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:10 smithi181 ceph-mon[35869]: pgmap v8545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14988.020026 secs 2024-07-26T12:09:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:10 smithi097 ceph-mon[32231]: pgmap v8545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14988.020026 secs 2024-07-26T12:09:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:12 smithi115 ceph-mon[35364]: pgmap v8546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:12 smithi181 ceph-mon[35869]: pgmap v8546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:12 smithi097 ceph-mon[32231]: pgmap v8546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:14 smithi115 ceph-mon[35364]: pgmap v8547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:14 smithi181 ceph-mon[35869]: pgmap v8547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:14 smithi097 ceph-mon[32231]: pgmap v8547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:15.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14993.020158 secs 2024-07-26T12:09:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14993.020158 secs 2024-07-26T12:09:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14993.020158 secs 2024-07-26T12:09:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:16 smithi115 ceph-mon[35364]: pgmap v8548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:16 smithi181 ceph-mon[35869]: pgmap v8548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:16 smithi097 ceph-mon[32231]: pgmap v8548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:18.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: pgmap v8549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: pgmap v8550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: pgmap v8551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:19.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: pgmap v8549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: pgmap v8550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: pgmap v8551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:19.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: pgmap v8549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: pgmap v8550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: pgmap v8551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:09:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:20 smithi181 ceph-mon[35869]: pgmap v8552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:09:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:20 smithi097 ceph-mon[32231]: pgmap v8552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:09:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:20 smithi115 ceph-mon[35364]: pgmap v8552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:09:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 14998.020309 secs 2024-07-26T12:09:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 14998.020309 secs 2024-07-26T12:09:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 14998.020309 secs 2024-07-26T12:09:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:22 smithi097 ceph-mon[32231]: pgmap v8553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:09:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:22 smithi181 ceph-mon[35869]: pgmap v8553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:09:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:22 smithi115 ceph-mon[35364]: pgmap v8553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:09:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:23.782+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:09:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:23.784+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:09:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:23.784+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:09:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:23.784+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:09:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:23.784+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:24 smithi181 ceph-mon[35869]: pgmap v8554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:24 smithi097 ceph-mon[32231]: pgmap v8554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:24 smithi115 ceph-mon[35364]: pgmap v8554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15003.020473 secs 2024-07-26T12:09:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15003.020473 secs 2024-07-26T12:09:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15003.020473 secs 2024-07-26T12:09:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:26 smithi115 ceph-mon[35364]: pgmap v8555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:09:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:26 smithi181 ceph-mon[35869]: pgmap v8555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:09:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:26 smithi097 ceph-mon[32231]: pgmap v8555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:09:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:28 smithi181 ceph-mon[35869]: pgmap v8556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:09:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:28 smithi097 ceph-mon[32231]: pgmap v8556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:09:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:28 smithi115 ceph-mon[35364]: pgmap v8556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:09:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:30 smithi181 ceph-mon[35869]: pgmap v8557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:30 smithi097 ceph-mon[32231]: pgmap v8557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:30 smithi115 ceph-mon[35364]: pgmap v8557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15008.020634 secs 2024-07-26T12:09:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15008.020634 secs 2024-07-26T12:09:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15008.020634 secs 2024-07-26T12:09:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:32 smithi115 ceph-mon[35364]: pgmap v8558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:09:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:32 smithi097 ceph-mon[32231]: pgmap v8558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:09:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:32 smithi181 ceph-mon[35869]: pgmap v8558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:09:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:34 smithi181 ceph-mon[35869]: pgmap v8559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:34 smithi097 ceph-mon[32231]: pgmap v8559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:34 smithi115 ceph-mon[35364]: pgmap v8559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15013.020733 secs 2024-07-26T12:09:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15013.020733 secs 2024-07-26T12:09:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15013.020733 secs 2024-07-26T12:09:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:36 smithi097 ceph-mon[32231]: pgmap v8560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:36 smithi115 ceph-mon[35364]: pgmap v8560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:36 smithi181 ceph-mon[35869]: pgmap v8560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:38 smithi181 ceph-mon[35869]: pgmap v8561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:38 smithi097 ceph-mon[32231]: pgmap v8561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:38.791+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:09:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:38.792+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:09:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:38.792+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:09:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:38.792+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:09:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:38.792+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:38 smithi115 ceph-mon[35364]: pgmap v8561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:40 smithi181 ceph-mon[35869]: pgmap v8562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:40 smithi097 ceph-mon[32231]: pgmap v8562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:40 smithi115 ceph-mon[35364]: pgmap v8562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15018.020875 secs 2024-07-26T12:09:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15018.020875 secs 2024-07-26T12:09:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15018.020875 secs 2024-07-26T12:09:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:42 smithi097 ceph-mon[32231]: pgmap v8563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:42 smithi115 ceph-mon[35364]: pgmap v8563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:42 smithi181 ceph-mon[35869]: pgmap v8563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:44 smithi181 ceph-mon[35869]: pgmap v8564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:44 smithi097 ceph-mon[32231]: pgmap v8564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:44 smithi115 ceph-mon[35364]: pgmap v8564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15023.021006 secs 2024-07-26T12:09:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15023.021006 secs 2024-07-26T12:09:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15023.021006 secs 2024-07-26T12:09:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:46 smithi097 ceph-mon[32231]: pgmap v8565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:46 smithi115 ceph-mon[35364]: pgmap v8565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:46 smithi181 ceph-mon[35869]: pgmap v8565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:48 smithi181 ceph-mon[35869]: pgmap v8566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:48 smithi097 ceph-mon[32231]: pgmap v8566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:48 smithi115 ceph-mon[35364]: pgmap v8566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:50 smithi097 ceph-mon[32231]: pgmap v8567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:50.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:50 smithi115 ceph-mon[35364]: pgmap v8567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:50 smithi181 ceph-mon[35869]: pgmap v8567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15028.021164 secs 2024-07-26T12:09:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15028.021164 secs 2024-07-26T12:09:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15028.021164 secs 2024-07-26T12:09:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:52 smithi097 ceph-mon[32231]: pgmap v8568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:52 smithi115 ceph-mon[35364]: pgmap v8568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:52 smithi181 ceph-mon[35869]: pgmap v8568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:53.795+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:09:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:53.796+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:09:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:53.796+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:09:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:53.796+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:09:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:09:53.796+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:54.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:54 smithi097 ceph-mon[32231]: pgmap v8569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:54 smithi115 ceph-mon[35364]: pgmap v8569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:54 smithi181 ceph-mon[35869]: pgmap v8569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:09:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15033.021330 secs 2024-07-26T12:09:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15033.021330 secs 2024-07-26T12:09:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15033.021330 secs 2024-07-26T12:09:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:56 smithi097 ceph-mon[32231]: pgmap v8570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:56 smithi115 ceph-mon[35364]: pgmap v8570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:56 smithi181 ceph-mon[35869]: pgmap v8570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:09:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:09:58 smithi097 ceph-mon[32231]: pgmap v8571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:09:58 smithi115 ceph-mon[35364]: pgmap v8571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:09:58 smithi181 ceph-mon[35869]: pgmap v8571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15038.021467 secs 2024-07-26T12:10:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:01 smithi115 ceph-mon[35364]: pgmap v8572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15038.021467 secs 2024-07-26T12:10:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:01 smithi181 ceph-mon[35869]: pgmap v8572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15038.021467 secs 2024-07-26T12:10:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:01 smithi097 ceph-mon[32231]: pgmap v8572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:02 smithi097 ceph-mon[32231]: pgmap v8573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:02 smithi115 ceph-mon[35364]: pgmap v8573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:02 smithi181 ceph-mon[35869]: pgmap v8573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:04 smithi097 ceph-mon[32231]: pgmap v8574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:04 smithi115 ceph-mon[35364]: pgmap v8574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:04 smithi181 ceph-mon[35869]: pgmap v8574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15043.021598 secs 2024-07-26T12:10:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15043.021598 secs 2024-07-26T12:10:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15043.021598 secs 2024-07-26T12:10:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:06 smithi097 ceph-mon[32231]: pgmap v8575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:06 smithi115 ceph-mon[35364]: pgmap v8575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:06 smithi181 ceph-mon[35869]: pgmap v8575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:08.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:08 smithi097 ceph-mon[32231]: pgmap v8576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:08 smithi181 ceph-mon[35869]: pgmap v8576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:09.480 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:08 smithi115 ceph-mon[35364]: pgmap v8576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:08.803+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:08.804+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:10:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:08.804+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:10:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:08.804+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:08.804+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:09.570+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:10:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:09.570+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:10:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:09.571+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:10:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:09.571+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:10:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:10 smithi097 ceph-mon[32231]: pgmap v8577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:10:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:10 smithi115 ceph-mon[35364]: pgmap v8577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:10:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:10 smithi181 ceph-mon[35869]: pgmap v8577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:10:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15048.021696 secs 2024-07-26T12:10:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15048.021696 secs 2024-07-26T12:10:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15048.021696 secs 2024-07-26T12:10:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:12 smithi115 ceph-mon[35364]: pgmap v8578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:10:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:12 smithi181 ceph-mon[35869]: pgmap v8578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:10:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:12 smithi097 ceph-mon[32231]: pgmap v8578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:10:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:14 smithi097 ceph-mon[32231]: pgmap v8579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:14 smithi115 ceph-mon[35364]: pgmap v8579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:14 smithi181 ceph-mon[35869]: pgmap v8579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15053.021866 secs 2024-07-26T12:10:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15053.021866 secs 2024-07-26T12:10:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15053.021866 secs 2024-07-26T12:10:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:16 smithi115 ceph-mon[35364]: pgmap v8580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:10:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:16 smithi181 ceph-mon[35869]: pgmap v8580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:10:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:16 smithi097 ceph-mon[32231]: pgmap v8580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:10:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:18 smithi097 ceph-mon[32231]: pgmap v8581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:10:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:18 smithi115 ceph-mon[35364]: pgmap v8581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:10:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:18 smithi181 ceph-mon[35869]: pgmap v8581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:10:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:20.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:20.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:20.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:20 smithi115 ceph-mon[35364]: pgmap v8582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:10:20.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:20 smithi115 ceph-mon[35364]: pgmap v8583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:20.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:10:20.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15058.022001 secs 2024-07-26T12:10:20.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:20.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:20.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:20 smithi181 ceph-mon[35869]: pgmap v8582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:10:20.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:20 smithi181 ceph-mon[35869]: pgmap v8583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:20.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:10:20.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15058.022001 secs 2024-07-26T12:10:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:20 smithi097 ceph-mon[32231]: pgmap v8582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:10:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:20 smithi097 ceph-mon[32231]: pgmap v8583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:10:21.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15058.022001 secs 2024-07-26T12:10:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:22 smithi115 ceph-mon[35364]: pgmap v8584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:10:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:22 smithi181 ceph-mon[35869]: pgmap v8584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:10:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:22 smithi097 ceph-mon[32231]: pgmap v8584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:23.806+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:23.807+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:23.807+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:23.807+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:23.807+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:24 smithi115 ceph-mon[35364]: pgmap v8585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:24 smithi181 ceph-mon[35869]: pgmap v8585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:24 smithi097 ceph-mon[32231]: pgmap v8585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15063.022134 secs 2024-07-26T12:10:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15063.022134 secs 2024-07-26T12:10:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15063.022134 secs 2024-07-26T12:10:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:26 smithi115 ceph-mon[35364]: pgmap v8586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:26 smithi181 ceph-mon[35869]: pgmap v8586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:26 smithi097 ceph-mon[32231]: pgmap v8586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:28 smithi115 ceph-mon[35364]: pgmap v8587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:28 smithi181 ceph-mon[35869]: pgmap v8587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:29.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:28 smithi097 ceph-mon[32231]: pgmap v8587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:30 smithi115 ceph-mon[35364]: pgmap v8588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15068.022307 secs 2024-07-26T12:10:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:30 smithi181 ceph-mon[35869]: pgmap v8588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15068.022307 secs 2024-07-26T12:10:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:30 smithi097 ceph-mon[32231]: pgmap v8588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15068.022307 secs 2024-07-26T12:10:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:32 smithi115 ceph-mon[35364]: pgmap v8589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:32 smithi181 ceph-mon[35869]: pgmap v8589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:32 smithi097 ceph-mon[32231]: pgmap v8589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:34 smithi115 ceph-mon[35364]: pgmap v8590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:34 smithi181 ceph-mon[35869]: pgmap v8590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:34 smithi097 ceph-mon[32231]: pgmap v8590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15073.022466 secs 2024-07-26T12:10:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15073.022466 secs 2024-07-26T12:10:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15073.022466 secs 2024-07-26T12:10:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:36 smithi115 ceph-mon[35364]: pgmap v8591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:36 smithi181 ceph-mon[35869]: pgmap v8591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:36 smithi097 ceph-mon[32231]: pgmap v8591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:38.816+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:38.817+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:10:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:38.817+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:10:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:38.817+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:38.817+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:38 smithi115 ceph-mon[35364]: pgmap v8592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:38 smithi181 ceph-mon[35869]: pgmap v8592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:38 smithi097 ceph-mon[32231]: pgmap v8592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:40 smithi115 ceph-mon[35364]: pgmap v8593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15078.022575 secs 2024-07-26T12:10:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:40 smithi181 ceph-mon[35869]: pgmap v8593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15078.022575 secs 2024-07-26T12:10:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:40 smithi097 ceph-mon[32231]: pgmap v8593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15078.022575 secs 2024-07-26T12:10:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:42 smithi115 ceph-mon[35364]: pgmap v8594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:42 smithi181 ceph-mon[35869]: pgmap v8594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:42 smithi097 ceph-mon[32231]: pgmap v8594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:44 smithi115 ceph-mon[35364]: pgmap v8595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:44 smithi181 ceph-mon[35869]: pgmap v8595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:44 smithi097 ceph-mon[32231]: pgmap v8595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15083.022736 secs 2024-07-26T12:10:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15083.022736 secs 2024-07-26T12:10:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15083.022736 secs 2024-07-26T12:10:47.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:46 smithi097 ceph-mon[32231]: pgmap v8596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:46 smithi115 ceph-mon[35364]: pgmap v8596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:46 smithi181 ceph-mon[35869]: pgmap v8596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:48 smithi097 ceph-mon[32231]: pgmap v8597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:48 smithi115 ceph-mon[35364]: pgmap v8597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:48 smithi181 ceph-mon[35869]: pgmap v8597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:51.276 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:51 smithi115 ceph-mon[35364]: pgmap v8598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:51.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15088.022818 secs 2024-07-26T12:10:51.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:51 smithi181 ceph-mon[35869]: pgmap v8598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:51.286 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15088.022818 secs 2024-07-26T12:10:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:51 smithi097 ceph-mon[32231]: pgmap v8598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15088.022818 secs 2024-07-26T12:10:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:53 smithi115 ceph-mon[35364]: pgmap v8599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:53 smithi181 ceph-mon[35869]: pgmap v8599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:53 smithi097 ceph-mon[32231]: pgmap v8599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:53.819+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:53.820+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:10:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:53.820+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:10:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:53.820+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:10:53.820+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:55 smithi115 ceph-mon[35364]: pgmap v8600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:55 smithi181 ceph-mon[35869]: pgmap v8600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:55 smithi097 ceph-mon[32231]: pgmap v8600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15093.022976 secs 2024-07-26T12:10:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15093.022976 secs 2024-07-26T12:10:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15093.022976 secs 2024-07-26T12:10:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:57 smithi115 ceph-mon[35364]: pgmap v8601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:57 smithi181 ceph-mon[35869]: pgmap v8601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:57 smithi097 ceph-mon[32231]: pgmap v8601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:10:59 smithi115 ceph-mon[35364]: pgmap v8602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:10:59 smithi181 ceph-mon[35869]: pgmap v8602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:10:59 smithi097 ceph-mon[32231]: pgmap v8602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:01 smithi115 ceph-mon[35364]: pgmap v8603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:01.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15098.023106 secs 2024-07-26T12:11:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:01 smithi181 ceph-mon[35869]: pgmap v8603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15098.023106 secs 2024-07-26T12:11:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:01 smithi097 ceph-mon[32231]: pgmap v8603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15098.023106 secs 2024-07-26T12:11:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:03 smithi115 ceph-mon[35364]: pgmap v8604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:03 smithi181 ceph-mon[35869]: pgmap v8604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:03 smithi097 ceph-mon[32231]: pgmap v8604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:05 smithi115 ceph-mon[35364]: pgmap v8605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:05 smithi181 ceph-mon[35869]: pgmap v8605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:05 smithi097 ceph-mon[32231]: pgmap v8605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15103.023267 secs 2024-07-26T12:11:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15103.023267 secs 2024-07-26T12:11:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15103.023267 secs 2024-07-26T12:11:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:07 smithi115 ceph-mon[35364]: pgmap v8606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:07 smithi181 ceph-mon[35869]: pgmap v8606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:07 smithi097 ceph-mon[32231]: pgmap v8606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:08.822+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:08.823+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:11:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:08.823+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:11:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:08.823+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:08.823+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:09 smithi115 ceph-mon[35364]: pgmap v8607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:09 smithi181 ceph-mon[35869]: pgmap v8607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:09 smithi097 ceph-mon[32231]: pgmap v8607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:11 smithi115 ceph-mon[35364]: pgmap v8608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15108.023327 secs 2024-07-26T12:11:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:11 smithi181 ceph-mon[35869]: pgmap v8608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15108.023327 secs 2024-07-26T12:11:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:11 smithi097 ceph-mon[32231]: pgmap v8608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15108.023327 secs 2024-07-26T12:11:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:13 smithi115 ceph-mon[35364]: pgmap v8609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:13 smithi181 ceph-mon[35869]: pgmap v8609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:13 smithi097 ceph-mon[32231]: pgmap v8609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:15 smithi115 ceph-mon[35364]: pgmap v8610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:15 smithi181 ceph-mon[35869]: pgmap v8610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:15 smithi097 ceph-mon[32231]: pgmap v8610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15113.023461 secs 2024-07-26T12:11:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15113.023461 secs 2024-07-26T12:11:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15113.023461 secs 2024-07-26T12:11:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:17 smithi115 ceph-mon[35364]: pgmap v8611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:17 smithi181 ceph-mon[35869]: pgmap v8611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:17 smithi097 ceph-mon[32231]: pgmap v8611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:19 smithi115 ceph-mon[35364]: pgmap v8612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:19 smithi181 ceph-mon[35869]: pgmap v8612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:19 smithi097 ceph-mon[32231]: pgmap v8612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:21 smithi115 ceph-mon[35364]: pgmap v8613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15118.023591 secs 2024-07-26T12:11:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:21 smithi181 ceph-mon[35869]: pgmap v8613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15118.023591 secs 2024-07-26T12:11:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:21 smithi097 ceph-mon[32231]: pgmap v8613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15118.023591 secs 2024-07-26T12:11:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:22 smithi115 ceph-mon[35364]: pgmap v8614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:11:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:22 smithi115 ceph-mon[35364]: pgmap v8615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:11:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:22 smithi181 ceph-mon[35869]: pgmap v8614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:11:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:22 smithi181 ceph-mon[35869]: pgmap v8615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:11:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:22 smithi097 ceph-mon[32231]: pgmap v8614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:11:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:22 smithi097 ceph-mon[32231]: pgmap v8615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:11:23.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:23.825+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:23.826+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:11:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:23.826+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:11:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:23.826+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:23.827+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:24 smithi115 ceph-mon[35364]: pgmap v8616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:24 smithi181 ceph-mon[35869]: pgmap v8616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:24 smithi097 ceph-mon[32231]: pgmap v8616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15123.023747 secs 2024-07-26T12:11:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:26 smithi115 ceph-mon[35364]: pgmap v8617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15123.023747 secs 2024-07-26T12:11:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:26 smithi181 ceph-mon[35869]: pgmap v8617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15123.023747 secs 2024-07-26T12:11:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:26 smithi097 ceph-mon[32231]: pgmap v8617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:28 smithi115 ceph-mon[35364]: pgmap v8618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:11:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:28 smithi181 ceph-mon[35869]: pgmap v8618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:11:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:28 smithi097 ceph-mon[32231]: pgmap v8618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:30 smithi115 ceph-mon[35364]: pgmap v8619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:30 smithi181 ceph-mon[35869]: pgmap v8619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:30 smithi097 ceph-mon[32231]: pgmap v8619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15128.023889 secs 2024-07-26T12:11:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15128.023889 secs 2024-07-26T12:11:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15128.023889 secs 2024-07-26T12:11:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:32 smithi115 ceph-mon[35364]: pgmap v8620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:11:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:32 smithi181 ceph-mon[35869]: pgmap v8620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:11:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:32 smithi097 ceph-mon[32231]: pgmap v8620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:11:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:34 smithi115 ceph-mon[35364]: pgmap v8621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:34 smithi181 ceph-mon[35869]: pgmap v8621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:34 smithi097 ceph-mon[32231]: pgmap v8621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15133.024034 secs 2024-07-26T12:11:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:36 smithi115 ceph-mon[35364]: pgmap v8622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15133.024034 secs 2024-07-26T12:11:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:36 smithi181 ceph-mon[35869]: pgmap v8622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15133.024034 secs 2024-07-26T12:11:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:36 smithi097 ceph-mon[32231]: pgmap v8622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:38 smithi115 ceph-mon[35364]: pgmap v8623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:38 smithi181 ceph-mon[35869]: pgmap v8623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:38 smithi097 ceph-mon[32231]: pgmap v8623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:38.835+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:38.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:38.836+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:11:38.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:38.837+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:11:38.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:38.837+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:38.838 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:38.837+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:40 smithi115 ceph-mon[35364]: pgmap v8624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:40 smithi181 ceph-mon[35869]: pgmap v8624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:40 smithi097 ceph-mon[32231]: pgmap v8624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15138.024192 secs 2024-07-26T12:11:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15138.024192 secs 2024-07-26T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15138.024192 secs 2024-07-26T12:11:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:42 smithi115 ceph-mon[35364]: pgmap v8625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:42 smithi181 ceph-mon[35869]: pgmap v8625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:42 smithi097 ceph-mon[32231]: pgmap v8625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:44 smithi115 ceph-mon[35364]: pgmap v8626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:44 smithi181 ceph-mon[35869]: pgmap v8626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:44 smithi097 ceph-mon[32231]: pgmap v8626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15143.024325 secs 2024-07-26T12:11:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:46 smithi115 ceph-mon[35364]: pgmap v8627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15143.024325 secs 2024-07-26T12:11:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:46 smithi181 ceph-mon[35869]: pgmap v8627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15143.024325 secs 2024-07-26T12:11:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:46 smithi097 ceph-mon[32231]: pgmap v8627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:48 smithi115 ceph-mon[35364]: pgmap v8628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:48 smithi181 ceph-mon[35869]: pgmap v8628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:48 smithi097 ceph-mon[32231]: pgmap v8628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:50 smithi115 ceph-mon[35364]: pgmap v8629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:50 smithi181 ceph-mon[35869]: pgmap v8629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:50 smithi097 ceph-mon[32231]: pgmap v8629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15148.024477 secs 2024-07-26T12:11:51.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:51 smithi115 ceph-mon[35364]: pgmap v8630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15148.024477 secs 2024-07-26T12:11:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:51 smithi181 ceph-mon[35869]: pgmap v8630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15148.024477 secs 2024-07-26T12:11:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:51 smithi097 ceph-mon[32231]: pgmap v8630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:53.839+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:53.841+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:11:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:53.841+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:11:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:53.841+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:11:53.841+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:54 smithi115 ceph-mon[35364]: pgmap v8631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:54 smithi181 ceph-mon[35869]: pgmap v8631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:54 smithi097 ceph-mon[32231]: pgmap v8631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15153.024582 secs 2024-07-26T12:11:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:56 smithi115 ceph-mon[35364]: pgmap v8632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15153.024582 secs 2024-07-26T12:11:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:56 smithi181 ceph-mon[35869]: pgmap v8632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15153.024582 secs 2024-07-26T12:11:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:56 smithi097 ceph-mon[32231]: pgmap v8632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:11:58 smithi115 ceph-mon[35364]: pgmap v8633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:11:58 smithi181 ceph-mon[35869]: pgmap v8633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:11:58 smithi097 ceph-mon[32231]: pgmap v8633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:00 smithi115 ceph-mon[35364]: pgmap v8634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:00 smithi181 ceph-mon[35869]: pgmap v8634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:00 smithi097 ceph-mon[32231]: pgmap v8634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15158.024727 secs 2024-07-26T12:12:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15158.024727 secs 2024-07-26T12:12:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15158.024727 secs 2024-07-26T12:12:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:02 smithi115 ceph-mon[35364]: pgmap v8635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:02 smithi181 ceph-mon[35869]: pgmap v8635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:02 smithi097 ceph-mon[32231]: pgmap v8635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:04 smithi115 ceph-mon[35364]: pgmap v8636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:04 smithi181 ceph-mon[35869]: pgmap v8636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:04 smithi097 ceph-mon[32231]: pgmap v8636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15163.024863 secs 2024-07-26T12:12:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:06 smithi115 ceph-mon[35364]: pgmap v8637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15163.024863 secs 2024-07-26T12:12:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:06 smithi181 ceph-mon[35869]: pgmap v8637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15163.024863 secs 2024-07-26T12:12:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:06 smithi097 ceph-mon[32231]: pgmap v8637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:08 smithi115 ceph-mon[35364]: pgmap v8638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:08 smithi181 ceph-mon[35869]: pgmap v8638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:08 smithi097 ceph-mon[32231]: pgmap v8638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:08.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:08.847+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:08.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:08.849+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:12:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:08.849+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:12:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:08.849+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:08.849+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:10 smithi115 ceph-mon[35364]: pgmap v8639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:10 smithi181 ceph-mon[35869]: pgmap v8639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:10 smithi097 ceph-mon[32231]: pgmap v8639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15168.025001 secs 2024-07-26T12:12:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15168.025001 secs 2024-07-26T12:12:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15168.025001 secs 2024-07-26T12:12:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:12 smithi115 ceph-mon[35364]: pgmap v8640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:12 smithi181 ceph-mon[35869]: pgmap v8640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:12 smithi097 ceph-mon[32231]: pgmap v8640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:14 smithi115 ceph-mon[35364]: pgmap v8641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:14 smithi181 ceph-mon[35869]: pgmap v8641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:14 smithi097 ceph-mon[32231]: pgmap v8641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15173.025140 secs 2024-07-26T12:12:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:16 smithi115 ceph-mon[35364]: pgmap v8642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15173.025140 secs 2024-07-26T12:12:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:16 smithi181 ceph-mon[35869]: pgmap v8642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15173.025140 secs 2024-07-26T12:12:16.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:16 smithi097 ceph-mon[32231]: pgmap v8642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:17 smithi115 ceph-mon[35364]: pgmap v8643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:17 smithi181 ceph-mon[35869]: pgmap v8643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:17 smithi097 ceph-mon[32231]: pgmap v8643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:20 smithi115 ceph-mon[35364]: pgmap v8644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:20 smithi181 ceph-mon[35869]: pgmap v8644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:20 smithi097 ceph-mon[32231]: pgmap v8644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:21.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15178.025293 secs 2024-07-26T12:12:21.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15178.025293 secs 2024-07-26T12:12:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15178.025293 secs 2024-07-26T12:12:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:22 smithi115 ceph-mon[35364]: pgmap v8645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:22 smithi181 ceph-mon[35869]: pgmap v8645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:22 smithi097 ceph-mon[32231]: pgmap v8645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:23.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:12:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:12:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:12:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:23.851+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:23.852+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:12:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:23.852+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:12:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:23.852+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:23.853+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:24 smithi115 ceph-mon[35364]: pgmap v8646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:12:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:24 smithi115 ceph-mon[35364]: pgmap v8647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:24 smithi181 ceph-mon[35869]: pgmap v8646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:12:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:24 smithi181 ceph-mon[35869]: pgmap v8647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:24 smithi097 ceph-mon[32231]: pgmap v8646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:12:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:24 smithi097 ceph-mon[32231]: pgmap v8647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:26 smithi115 ceph-mon[35364]: pgmap v8648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15183.025422 secs 2024-07-26T12:12:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:26 smithi181 ceph-mon[35869]: pgmap v8648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15183.025422 secs 2024-07-26T12:12:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:26 smithi097 ceph-mon[32231]: pgmap v8648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15183.025422 secs 2024-07-26T12:12:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:28 smithi115 ceph-mon[35364]: pgmap v8649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:28 smithi181 ceph-mon[35869]: pgmap v8649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:28 smithi097 ceph-mon[32231]: pgmap v8649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:30 smithi115 ceph-mon[35364]: pgmap v8650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:30 smithi181 ceph-mon[35869]: pgmap v8650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:30 smithi097 ceph-mon[32231]: pgmap v8650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15188.025552 secs 2024-07-26T12:12:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15188.025552 secs 2024-07-26T12:12:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15188.025552 secs 2024-07-26T12:12:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:32 smithi115 ceph-mon[35364]: pgmap v8651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:32 smithi181 ceph-mon[35869]: pgmap v8651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:32 smithi097 ceph-mon[32231]: pgmap v8651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:34 smithi115 ceph-mon[35364]: pgmap v8652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:34 smithi181 ceph-mon[35869]: pgmap v8652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:34 smithi097 ceph-mon[32231]: pgmap v8652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:36 smithi115 ceph-mon[35364]: pgmap v8653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15193.025650 secs 2024-07-26T12:12:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:36 smithi181 ceph-mon[35869]: pgmap v8653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15193.025650 secs 2024-07-26T12:12:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:36 smithi097 ceph-mon[32231]: pgmap v8653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15193.025650 secs 2024-07-26T12:12:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:38 smithi115 ceph-mon[35364]: pgmap v8654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:38 smithi181 ceph-mon[35869]: pgmap v8654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:38 smithi097 ceph-mon[32231]: pgmap v8654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:38.855+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:38.856+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:12:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:38.856+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:12:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:38.856+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:38.856+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:40 smithi115 ceph-mon[35364]: pgmap v8655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:40 smithi181 ceph-mon[35869]: pgmap v8655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:40 smithi097 ceph-mon[32231]: pgmap v8655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15198.025783 secs 2024-07-26T12:12:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15198.025783 secs 2024-07-26T12:12:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15198.025783 secs 2024-07-26T12:12:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:42 smithi181 ceph-mon[35869]: pgmap v8656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:42 smithi097 ceph-mon[32231]: pgmap v8656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:42 smithi115 ceph-mon[35364]: pgmap v8656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:43 smithi097 ceph-mon[32231]: pgmap v8657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:43 smithi115 ceph-mon[35364]: pgmap v8657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:43 smithi181 ceph-mon[35869]: pgmap v8657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:45 smithi115 ceph-mon[35364]: pgmap v8658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15203.025919 secs 2024-07-26T12:12:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:45 smithi181 ceph-mon[35869]: pgmap v8658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15203.025919 secs 2024-07-26T12:12:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:45 smithi097 ceph-mon[32231]: pgmap v8658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15203.025919 secs 2024-07-26T12:12:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:47 smithi115 ceph-mon[35364]: pgmap v8659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:47 smithi181 ceph-mon[35869]: pgmap v8659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:47 smithi097 ceph-mon[32231]: pgmap v8659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:49 smithi115 ceph-mon[35364]: pgmap v8660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:49 smithi181 ceph-mon[35869]: pgmap v8660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:49 smithi097 ceph-mon[32231]: pgmap v8660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:51.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15208.026056 secs 2024-07-26T12:12:51.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15208.026056 secs 2024-07-26T12:12:51.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15208.026056 secs 2024-07-26T12:12:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:51 smithi115 ceph-mon[35364]: pgmap v8661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:51 smithi181 ceph-mon[35869]: pgmap v8661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:51 smithi097 ceph-mon[32231]: pgmap v8661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:53.859+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:53.859+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:12:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:53.859+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:12:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:53.859+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:12:53.859+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:53 smithi115 ceph-mon[35364]: pgmap v8662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:53 smithi181 ceph-mon[35869]: pgmap v8662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:53 smithi097 ceph-mon[32231]: pgmap v8662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:55 smithi115 ceph-mon[35364]: pgmap v8663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:56.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15213.026189 secs 2024-07-26T12:12:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:55 smithi181 ceph-mon[35869]: pgmap v8663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15213.026189 secs 2024-07-26T12:12:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:55 smithi097 ceph-mon[32231]: pgmap v8663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15213.026189 secs 2024-07-26T12:12:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:57 smithi115 ceph-mon[35364]: pgmap v8664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:57 smithi181 ceph-mon[35869]: pgmap v8664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:57 smithi097 ceph-mon[32231]: pgmap v8664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:12:59 smithi115 ceph-mon[35364]: pgmap v8665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:12:59 smithi181 ceph-mon[35869]: pgmap v8665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:12:59 smithi097 ceph-mon[32231]: pgmap v8665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15218.026325 secs 2024-07-26T12:13:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15218.026325 secs 2024-07-26T12:13:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15218.026325 secs 2024-07-26T12:13:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:01 smithi115 ceph-mon[35364]: pgmap v8666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:01 smithi181 ceph-mon[35869]: pgmap v8666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:01 smithi097 ceph-mon[32231]: pgmap v8666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:03 smithi115 ceph-mon[35364]: pgmap v8667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:03 smithi181 ceph-mon[35869]: pgmap v8667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:03 smithi097 ceph-mon[32231]: pgmap v8667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:05 smithi115 ceph-mon[35364]: pgmap v8668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15223.026460 secs 2024-07-26T12:13:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:05 smithi181 ceph-mon[35869]: pgmap v8668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15223.026460 secs 2024-07-26T12:13:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:05 smithi097 ceph-mon[32231]: pgmap v8668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15223.026460 secs 2024-07-26T12:13:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.927+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 msgr2=0x7ff3d40c2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T12:13:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.927+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 msgr2=0x7ff3d40c2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:13:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.927+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40c2420 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006300 tx=0x7ff3e007b000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.927+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40c2420 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006300 tx=0x7ff3e007b000 comp rx=0 tx=0).stop 2024-07-26T12:13:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.927+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 msgr2=0x7ff3d40c2420 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T12:13:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.927+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 msgr2=0x7ff3d40c2420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:13:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.927+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40c2420 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:13:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.927+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:13:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.928+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:06.930+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e0071000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:13:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:07 smithi115 ceph-mon[35364]: pgmap v8669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:07 smithi181 ceph-mon[35869]: pgmap v8669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:07 smithi097 ceph-mon[32231]: pgmap v8669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:08.861+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:08.863+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:13:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:08.863+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:13:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:08.863+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:08.863+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:09.602+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:13:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:09.602+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:13:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:09.603+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:13:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:09.604+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:13:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:09 smithi115 ceph-mon[35364]: pgmap v8670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:09 smithi181 ceph-mon[35869]: pgmap v8670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:09 smithi097 ceph-mon[32231]: pgmap v8670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15228.026594 secs 2024-07-26T12:13:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15228.026594 secs 2024-07-26T12:13:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15228.026594 secs 2024-07-26T12:13:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:11 smithi115 ceph-mon[35364]: pgmap v8671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:11 smithi181 ceph-mon[35869]: pgmap v8671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:11 smithi097 ceph-mon[32231]: pgmap v8671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:13 smithi115 ceph-mon[35364]: pgmap v8672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:13 smithi181 ceph-mon[35869]: pgmap v8672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:13 smithi097 ceph-mon[32231]: pgmap v8672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:15 smithi115 ceph-mon[35364]: pgmap v8673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:16.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15233.026730 secs 2024-07-26T12:13:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:15 smithi181 ceph-mon[35869]: pgmap v8673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15233.026730 secs 2024-07-26T12:13:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:15 smithi097 ceph-mon[32231]: pgmap v8673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15233.026730 secs 2024-07-26T12:13:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:17 smithi115 ceph-mon[35364]: pgmap v8674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:17 smithi181 ceph-mon[35869]: pgmap v8674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:17 smithi097 ceph-mon[32231]: pgmap v8674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:19 smithi115 ceph-mon[35364]: pgmap v8675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:19 smithi181 ceph-mon[35869]: pgmap v8675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:19 smithi097 ceph-mon[32231]: pgmap v8675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15238.026861 secs 2024-07-26T12:13:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15238.026861 secs 2024-07-26T12:13:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15238.026861 secs 2024-07-26T12:13:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:21 smithi115 ceph-mon[35364]: pgmap v8676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:21 smithi181 ceph-mon[35869]: pgmap v8676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:21 smithi097 ceph-mon[32231]: pgmap v8676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:23.866+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:23.867+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:13:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:23.867+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:13:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:23.867+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:23.867+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:23 smithi115 ceph-mon[35364]: pgmap v8677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:23 smithi181 ceph-mon[35869]: pgmap v8677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:23 smithi097 ceph-mon[32231]: pgmap v8677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:24 smithi181 ceph-mon[35869]: pgmap v8678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:13:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:24 smithi181 ceph-mon[35869]: pgmap v8679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:13:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:24 smithi097 ceph-mon[32231]: pgmap v8678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:13:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:24 smithi097 ceph-mon[32231]: pgmap v8679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:13:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:24 smithi115 ceph-mon[35364]: pgmap v8678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:13:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:24 smithi115 ceph-mon[35364]: pgmap v8679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:25.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:13:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15243.026998 secs 2024-07-26T12:13:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15243.026998 secs 2024-07-26T12:13:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15243.026998 secs 2024-07-26T12:13:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:26 smithi097 ceph-mon[32231]: pgmap v8680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:26 smithi115 ceph-mon[35364]: pgmap v8680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:26 smithi181 ceph-mon[35869]: pgmap v8680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:29 smithi115 ceph-mon[35364]: pgmap v8681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:29 smithi181 ceph-mon[35869]: pgmap v8681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:29 smithi097 ceph-mon[32231]: pgmap v8681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15248.027133 secs 2024-07-26T12:13:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:31 smithi115 ceph-mon[35364]: pgmap v8682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15248.027133 secs 2024-07-26T12:13:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:31 smithi181 ceph-mon[35869]: pgmap v8682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15248.027133 secs 2024-07-26T12:13:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:31 smithi097 ceph-mon[32231]: pgmap v8682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:33 smithi115 ceph-mon[35364]: pgmap v8683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:13:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:33 smithi181 ceph-mon[35869]: pgmap v8683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:13:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:33 smithi097 ceph-mon[32231]: pgmap v8683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:13:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:35 smithi115 ceph-mon[35364]: pgmap v8684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:35 smithi181 ceph-mon[35869]: pgmap v8684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:35 smithi097 ceph-mon[32231]: pgmap v8684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15253.027269 secs 2024-07-26T12:13:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15253.027269 secs 2024-07-26T12:13:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15253.027269 secs 2024-07-26T12:13:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:37 smithi115 ceph-mon[35364]: pgmap v8685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:37 smithi181 ceph-mon[35869]: pgmap v8685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:37 smithi097 ceph-mon[32231]: pgmap v8685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:38.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:38.875+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:38.876+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:13:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:38.876+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:13:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:38.876+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:38.876+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:39 smithi115 ceph-mon[35364]: pgmap v8686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:13:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:39 smithi181 ceph-mon[35869]: pgmap v8686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:13:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:39 smithi097 ceph-mon[32231]: pgmap v8686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:13:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15258.027406 secs 2024-07-26T12:13:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:41 smithi115 ceph-mon[35364]: pgmap v8687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:13:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15258.027406 secs 2024-07-26T12:13:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:41 smithi181 ceph-mon[35869]: pgmap v8687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:13:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15258.027406 secs 2024-07-26T12:13:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:41 smithi097 ceph-mon[32231]: pgmap v8687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:13:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:43 smithi115 ceph-mon[35364]: pgmap v8688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:13:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:43 smithi181 ceph-mon[35869]: pgmap v8688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:13:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:43 smithi097 ceph-mon[32231]: pgmap v8688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:13:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:45 smithi115 ceph-mon[35364]: pgmap v8689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:13:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:45 smithi181 ceph-mon[35869]: pgmap v8689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:13:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:45 smithi097 ceph-mon[32231]: pgmap v8689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:13:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15263.027529 secs 2024-07-26T12:13:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15263.027529 secs 2024-07-26T12:13:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15263.027529 secs 2024-07-26T12:13:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:47 smithi115 ceph-mon[35364]: pgmap v8690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:47 smithi181 ceph-mon[35869]: pgmap v8690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:47 smithi097 ceph-mon[32231]: pgmap v8690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:49 smithi115 ceph-mon[35364]: pgmap v8691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:49 smithi181 ceph-mon[35869]: pgmap v8691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:49 smithi097 ceph-mon[32231]: pgmap v8691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15268.027688 secs 2024-07-26T12:13:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:51 smithi115 ceph-mon[35364]: pgmap v8692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15268.027688 secs 2024-07-26T12:13:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:51 smithi181 ceph-mon[35869]: pgmap v8692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15268.027688 secs 2024-07-26T12:13:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:51 smithi097 ceph-mon[32231]: pgmap v8692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:53 smithi115 ceph-mon[35364]: pgmap v8693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:53 smithi181 ceph-mon[35869]: pgmap v8693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:53 smithi097 ceph-mon[32231]: pgmap v8693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:53.877+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:13:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:53.878+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:13:53.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:53.878+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:13:53.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:53.878+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:13:53.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:53.878+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:54.611+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:13:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:54.611+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:13:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:54.612+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:13:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:13:54.612+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:13:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:55 smithi115 ceph-mon[35364]: pgmap v8694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:55 smithi181 ceph-mon[35869]: pgmap v8694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:55 smithi097 ceph-mon[32231]: pgmap v8694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15273.027821 secs 2024-07-26T12:13:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15273.027821 secs 2024-07-26T12:13:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15273.027821 secs 2024-07-26T12:13:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:57 smithi115 ceph-mon[35364]: pgmap v8695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:57 smithi181 ceph-mon[35869]: pgmap v8695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:57 smithi097 ceph-mon[32231]: pgmap v8695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:13:59 smithi115 ceph-mon[35364]: pgmap v8696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:13:59 smithi181 ceph-mon[35869]: pgmap v8696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:13:59 smithi097 ceph-mon[32231]: pgmap v8696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15278.027933 secs 2024-07-26T12:14:01.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:01 smithi115 ceph-mon[35364]: pgmap v8697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15278.027933 secs 2024-07-26T12:14:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:01 smithi181 ceph-mon[35869]: pgmap v8697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15278.027933 secs 2024-07-26T12:14:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:01 smithi097 ceph-mon[32231]: pgmap v8697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:03 smithi115 ceph-mon[35364]: pgmap v8698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:03 smithi181 ceph-mon[35869]: pgmap v8698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:03 smithi097 ceph-mon[32231]: pgmap v8698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:05 smithi115 ceph-mon[35364]: pgmap v8699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:05 smithi181 ceph-mon[35869]: pgmap v8699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:05 smithi097 ceph-mon[32231]: pgmap v8699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15283.028062 secs 2024-07-26T12:14:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15283.028062 secs 2024-07-26T12:14:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15283.028062 secs 2024-07-26T12:14:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:07 smithi115 ceph-mon[35364]: pgmap v8700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:07 smithi181 ceph-mon[35869]: pgmap v8700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:07 smithi097 ceph-mon[32231]: pgmap v8700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:08.886+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:14:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:08.887+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:14:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:08.887+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:14:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:08.887+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:14:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:08.887+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:09 smithi181 ceph-mon[35869]: pgmap v8701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:09 smithi097 ceph-mon[32231]: pgmap v8701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:09 smithi115 ceph-mon[35364]: pgmap v8701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15288.028209 secs 2024-07-26T12:14:11.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:11 smithi097 ceph-mon[32231]: pgmap v8702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15288.028209 secs 2024-07-26T12:14:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:11 smithi115 ceph-mon[35364]: pgmap v8702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15288.028209 secs 2024-07-26T12:14:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:11 smithi181 ceph-mon[35869]: pgmap v8702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:12 smithi115 ceph-mon[35364]: pgmap v8703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:12 smithi181 ceph-mon[35869]: pgmap v8703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:12 smithi097 ceph-mon[32231]: pgmap v8703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:15 smithi115 ceph-mon[35364]: pgmap v8704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:15 smithi181 ceph-mon[35869]: pgmap v8704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:15 smithi097 ceph-mon[32231]: pgmap v8704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15293.028353 secs 2024-07-26T12:14:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15293.028353 secs 2024-07-26T12:14:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15293.028353 secs 2024-07-26T12:14:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:17 smithi115 ceph-mon[35364]: pgmap v8705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:17 smithi181 ceph-mon[35869]: pgmap v8705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:17 smithi097 ceph-mon[32231]: pgmap v8705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:19 smithi115 ceph-mon[35364]: pgmap v8706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:19 smithi181 ceph-mon[35869]: pgmap v8706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:19 smithi097 ceph-mon[32231]: pgmap v8706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15298.028489 secs 2024-07-26T12:14:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:21 smithi115 ceph-mon[35364]: pgmap v8707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15298.028489 secs 2024-07-26T12:14:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:21 smithi181 ceph-mon[35869]: pgmap v8707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15298.028489 secs 2024-07-26T12:14:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:21 smithi097 ceph-mon[32231]: pgmap v8707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:23 smithi115 ceph-mon[35364]: pgmap v8708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:23 smithi181 ceph-mon[35869]: pgmap v8708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:23 smithi097 ceph-mon[32231]: pgmap v8708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:23.890+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:14:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:23.891+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:14:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:23.891+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:14:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:23.891+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:14:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:23.891+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:25 smithi115 ceph-mon[35364]: pgmap v8709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:25 smithi181 ceph-mon[35869]: pgmap v8709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:25 smithi097 ceph-mon[32231]: pgmap v8709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15303.028621 secs 2024-07-26T12:14:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:14:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15303.028621 secs 2024-07-26T12:14:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:14:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15303.028621 secs 2024-07-26T12:14:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:14:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:27 smithi115 ceph-mon[35364]: pgmap v8710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:27 smithi115 ceph-mon[35364]: pgmap v8711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:14:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:27 smithi181 ceph-mon[35869]: pgmap v8710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:27 smithi181 ceph-mon[35869]: pgmap v8711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:14:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:27 smithi097 ceph-mon[32231]: pgmap v8710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:27 smithi097 ceph-mon[32231]: pgmap v8711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:14:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:29 smithi115 ceph-mon[35364]: pgmap v8712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:29 smithi181 ceph-mon[35869]: pgmap v8712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:29 smithi097 ceph-mon[32231]: pgmap v8712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:31 smithi115 ceph-mon[35364]: pgmap v8713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:14:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15308.028768 secs 2024-07-26T12:14:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:31 smithi181 ceph-mon[35869]: pgmap v8713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:14:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15308.028768 secs 2024-07-26T12:14:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:31 smithi097 ceph-mon[32231]: pgmap v8713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:14:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15308.028768 secs 2024-07-26T12:14:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:33 smithi115 ceph-mon[35364]: pgmap v8714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:33 smithi181 ceph-mon[35869]: pgmap v8714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:33 smithi097 ceph-mon[32231]: pgmap v8714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:35 smithi115 ceph-mon[35364]: pgmap v8715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:35 smithi181 ceph-mon[35869]: pgmap v8715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:35 smithi097 ceph-mon[32231]: pgmap v8715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15313.028931 secs 2024-07-26T12:14:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15313.028931 secs 2024-07-26T12:14:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15313.028931 secs 2024-07-26T12:14:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:37 smithi115 ceph-mon[35364]: pgmap v8716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:37 smithi181 ceph-mon[35869]: pgmap v8716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:37 smithi097 ceph-mon[32231]: pgmap v8716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:38.898+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:14:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:38.899+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:14:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:38.899+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:14:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:38.899+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:14:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:38.899+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:39 smithi115 ceph-mon[35364]: pgmap v8717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:39 smithi181 ceph-mon[35869]: pgmap v8717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:39 smithi097 ceph-mon[32231]: pgmap v8717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:41 smithi115 ceph-mon[35364]: pgmap v8718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15318.029060 secs 2024-07-26T12:14:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:41 smithi181 ceph-mon[35869]: pgmap v8718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15318.029060 secs 2024-07-26T12:14:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:41 smithi097 ceph-mon[32231]: pgmap v8718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15318.029060 secs 2024-07-26T12:14:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:43 smithi115 ceph-mon[35364]: pgmap v8719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:43 smithi181 ceph-mon[35869]: pgmap v8719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:43 smithi097 ceph-mon[32231]: pgmap v8719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:45 smithi115 ceph-mon[35364]: pgmap v8720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:45 smithi181 ceph-mon[35869]: pgmap v8720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:45 smithi097 ceph-mon[32231]: pgmap v8720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15323.029179 secs 2024-07-26T12:14:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15323.029179 secs 2024-07-26T12:14:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15323.029179 secs 2024-07-26T12:14:47.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:47 smithi181 ceph-mon[35869]: pgmap v8721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:47 smithi097 ceph-mon[32231]: pgmap v8721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:47 smithi115 ceph-mon[35364]: pgmap v8721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:49 smithi097 ceph-mon[32231]: pgmap v8722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:49 smithi115 ceph-mon[35364]: pgmap v8722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:49 smithi181 ceph-mon[35869]: pgmap v8722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:51 smithi097 ceph-mon[32231]: pgmap v8723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15328.029326 secs 2024-07-26T12:14:51.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:51 smithi115 ceph-mon[35364]: pgmap v8723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:51.778 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15328.029326 secs 2024-07-26T12:14:51.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:51 smithi181 ceph-mon[35869]: pgmap v8723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:51.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15328.029326 secs 2024-07-26T12:14:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:52 smithi097 ceph-mon[32231]: pgmap v8724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:52 smithi115 ceph-mon[35364]: pgmap v8724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:52 smithi181 ceph-mon[35869]: pgmap v8724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:53.900+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:14:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:53.902+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:14:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:53.902+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:14:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:53.902+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:14:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:14:53.902+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:54 smithi181 ceph-mon[35869]: pgmap v8725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:54 smithi097 ceph-mon[32231]: pgmap v8725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:54 smithi115 ceph-mon[35364]: pgmap v8725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15333.029421 secs 2024-07-26T12:14:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15333.029421 secs 2024-07-26T12:14:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15333.029421 secs 2024-07-26T12:14:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:56 smithi097 ceph-mon[32231]: pgmap v8726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:56 smithi115 ceph-mon[35364]: pgmap v8726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:56 smithi181 ceph-mon[35869]: pgmap v8726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:14:58 smithi115 ceph-mon[35364]: pgmap v8727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:14:58 smithi181 ceph-mon[35869]: pgmap v8727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:14:58 smithi097 ceph-mon[32231]: pgmap v8727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:00 smithi115 ceph-mon[35364]: pgmap v8728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15338.029560 secs 2024-07-26T12:15:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:00 smithi181 ceph-mon[35869]: pgmap v8728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15338.029560 secs 2024-07-26T12:15:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:00 smithi097 ceph-mon[32231]: pgmap v8728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15338.029560 secs 2024-07-26T12:15:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:02 smithi115 ceph-mon[35364]: pgmap v8729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:02 smithi181 ceph-mon[35869]: pgmap v8729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:02 smithi097 ceph-mon[32231]: pgmap v8729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:04 smithi115 ceph-mon[35364]: pgmap v8730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:04 smithi181 ceph-mon[35869]: pgmap v8730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:04 smithi097 ceph-mon[32231]: pgmap v8730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15343.029702 secs 2024-07-26T12:15:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15343.029702 secs 2024-07-26T12:15:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15343.029702 secs 2024-07-26T12:15:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:06 smithi115 ceph-mon[35364]: pgmap v8731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:06 smithi181 ceph-mon[35869]: pgmap v8731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:06 smithi097 ceph-mon[32231]: pgmap v8731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:08.909+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:15:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:08.910+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:15:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:08.910+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:15:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:08.910+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:15:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:08.910+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:08 smithi115 ceph-mon[35364]: pgmap v8732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:08 smithi181 ceph-mon[35869]: pgmap v8732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:08 smithi097 ceph-mon[32231]: pgmap v8732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:10 smithi115 ceph-mon[35364]: pgmap v8733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15348.029845 secs 2024-07-26T12:15:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:10 smithi181 ceph-mon[35869]: pgmap v8733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15348.029845 secs 2024-07-26T12:15:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:10 smithi097 ceph-mon[32231]: pgmap v8733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15348.029845 secs 2024-07-26T12:15:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:12 smithi115 ceph-mon[35364]: pgmap v8734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:12 smithi181 ceph-mon[35869]: pgmap v8734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:12 smithi097 ceph-mon[32231]: pgmap v8734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:14 smithi115 ceph-mon[35364]: pgmap v8735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:14 smithi181 ceph-mon[35869]: pgmap v8735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:14 smithi097 ceph-mon[32231]: pgmap v8735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15353.029987 secs 2024-07-26T12:15:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15353.029987 secs 2024-07-26T12:15:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15353.029987 secs 2024-07-26T12:15:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:16 smithi115 ceph-mon[35364]: pgmap v8736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:16 smithi181 ceph-mon[35869]: pgmap v8736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:16 smithi097 ceph-mon[32231]: pgmap v8736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:18 smithi115 ceph-mon[35364]: pgmap v8737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:18 smithi181 ceph-mon[35869]: pgmap v8737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:18 smithi097 ceph-mon[32231]: pgmap v8737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:20 smithi115 ceph-mon[35364]: pgmap v8738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:15:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15358.030157 secs 2024-07-26T12:15:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:20 smithi181 ceph-mon[35869]: pgmap v8738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:15:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15358.030157 secs 2024-07-26T12:15:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:20 smithi097 ceph-mon[32231]: pgmap v8738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:15:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15358.030157 secs 2024-07-26T12:15:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:22 smithi115 ceph-mon[35364]: pgmap v8739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:22 smithi181 ceph-mon[35869]: pgmap v8739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:22 smithi097 ceph-mon[32231]: pgmap v8739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:23.913+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:15:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:23.914+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:15:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:23.914+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:15:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:23.914+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:15:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:23.914+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:24 smithi115 ceph-mon[35364]: pgmap v8740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:15:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:24 smithi181 ceph-mon[35869]: pgmap v8740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:15:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:24 smithi097 ceph-mon[32231]: pgmap v8740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:15:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15363.030335 secs 2024-07-26T12:15:26.258 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15363.030335 secs 2024-07-26T12:15:26.259 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15363.030335 secs 2024-07-26T12:15:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:26 smithi115 ceph-mon[35364]: pgmap v8741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:15:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:26 smithi181 ceph-mon[35869]: pgmap v8741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:15:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:26 smithi097 ceph-mon[32231]: pgmap v8741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:15:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: pgmap v8742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:15:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:15:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: pgmap v8742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:15:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:15:28.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:15:28.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:15:28.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:28.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:28.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: pgmap v8742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:15:28.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:15:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:30 smithi115 ceph-mon[35364]: pgmap v8743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:15:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:30 smithi181 ceph-mon[35869]: pgmap v8743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:15:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:30 smithi097 ceph-mon[32231]: pgmap v8743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:15:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15368.030474 secs 2024-07-26T12:15:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15368.030474 secs 2024-07-26T12:15:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15368.030474 secs 2024-07-26T12:15:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:32 smithi097 ceph-mon[32231]: pgmap v8744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:32 smithi115 ceph-mon[35364]: pgmap v8744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:32 smithi181 ceph-mon[35869]: pgmap v8744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:34 smithi097 ceph-mon[32231]: pgmap v8745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:15:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:34 smithi115 ceph-mon[35364]: pgmap v8745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:15:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:34 smithi181 ceph-mon[35869]: pgmap v8745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:15:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15373.030627 secs 2024-07-26T12:15:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:36 smithi115 ceph-mon[35364]: pgmap v8746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15373.030627 secs 2024-07-26T12:15:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:36 smithi181 ceph-mon[35869]: pgmap v8746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15373.030627 secs 2024-07-26T12:15:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:36 smithi097 ceph-mon[32231]: pgmap v8746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:38 smithi115 ceph-mon[35364]: pgmap v8747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:38 smithi181 ceph-mon[35869]: pgmap v8747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:38 smithi097 ceph-mon[32231]: pgmap v8747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:38.917+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:15:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:38.918+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:15:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:38.918+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:15:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:38.918+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:15:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:38.918+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:40 smithi115 ceph-mon[35364]: pgmap v8748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:40 smithi181 ceph-mon[35869]: pgmap v8748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:40 smithi097 ceph-mon[32231]: pgmap v8748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15378.030726 secs 2024-07-26T12:15:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15378.030726 secs 2024-07-26T12:15:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15378.030726 secs 2024-07-26T12:15:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:42 smithi115 ceph-mon[35364]: pgmap v8749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:15:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:42 smithi181 ceph-mon[35869]: pgmap v8749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:15:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:42 smithi097 ceph-mon[32231]: pgmap v8749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:15:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:44 smithi115 ceph-mon[35364]: pgmap v8750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:15:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:44 smithi181 ceph-mon[35869]: pgmap v8750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:15:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:44 smithi097 ceph-mon[32231]: pgmap v8750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:15:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15383.030894 secs 2024-07-26T12:15:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:46 smithi115 ceph-mon[35364]: pgmap v8751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:15:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15383.030894 secs 2024-07-26T12:15:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:46 smithi181 ceph-mon[35869]: pgmap v8751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:15:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15383.030894 secs 2024-07-26T12:15:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:46 smithi097 ceph-mon[32231]: pgmap v8751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:15:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:48 smithi115 ceph-mon[35364]: pgmap v8752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:15:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:48 smithi181 ceph-mon[35869]: pgmap v8752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:15:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:48 smithi097 ceph-mon[32231]: pgmap v8752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:15:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:50 smithi115 ceph-mon[35364]: pgmap v8753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:50 smithi181 ceph-mon[35869]: pgmap v8753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:50 smithi097 ceph-mon[32231]: pgmap v8753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:51.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15388.031057 secs 2024-07-26T12:15:51.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15388.031057 secs 2024-07-26T12:15:51.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15388.031057 secs 2024-07-26T12:15:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:52 smithi115 ceph-mon[35364]: pgmap v8754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:52 smithi181 ceph-mon[35869]: pgmap v8754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:52 smithi097 ceph-mon[32231]: pgmap v8754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:53.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:53.919+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:15:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:53.920+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:15:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:53.920+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:15:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:53.920+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:15:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:15:53.920+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:54 smithi115 ceph-mon[35364]: pgmap v8755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:54 smithi181 ceph-mon[35869]: pgmap v8755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:54 smithi097 ceph-mon[32231]: pgmap v8755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:15:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15393.031208 secs 2024-07-26T12:15:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:56 smithi115 ceph-mon[35364]: pgmap v8756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15393.031208 secs 2024-07-26T12:15:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:56 smithi181 ceph-mon[35869]: pgmap v8756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15393.031208 secs 2024-07-26T12:15:56.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:56 smithi097 ceph-mon[32231]: pgmap v8756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:15:58 smithi115 ceph-mon[35364]: pgmap v8757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:15:58 smithi181 ceph-mon[35869]: pgmap v8757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:15:58 smithi097 ceph-mon[32231]: pgmap v8757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:00 smithi115 ceph-mon[35364]: pgmap v8758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:00 smithi181 ceph-mon[35869]: pgmap v8758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:00 smithi097 ceph-mon[32231]: pgmap v8758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15398.031359 secs 2024-07-26T12:16:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15398.031359 secs 2024-07-26T12:16:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15398.031359 secs 2024-07-26T12:16:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:02 smithi115 ceph-mon[35364]: pgmap v8759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:02 smithi181 ceph-mon[35869]: pgmap v8759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:02 smithi097 ceph-mon[32231]: pgmap v8759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:04 smithi115 ceph-mon[35364]: pgmap v8760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:04 smithi181 ceph-mon[35869]: pgmap v8760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:04 smithi097 ceph-mon[32231]: pgmap v8760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15403.031456 secs 2024-07-26T12:16:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:06 smithi115 ceph-mon[35364]: pgmap v8761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15403.031456 secs 2024-07-26T12:16:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:06 smithi181 ceph-mon[35869]: pgmap v8761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15403.031456 secs 2024-07-26T12:16:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:06 smithi097 ceph-mon[32231]: pgmap v8761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:08 smithi115 ceph-mon[35364]: pgmap v8762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:08 smithi181 ceph-mon[35869]: pgmap v8762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:08 smithi097 ceph-mon[32231]: pgmap v8762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:08.929+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:16:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:08.930+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:16:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:08.930+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:16:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:08.930+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:16:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:08.930+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:09.636+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:16:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:09.636+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:16:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:09.637+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:16:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:09.638+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:16:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:10 smithi115 ceph-mon[35364]: pgmap v8763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:10 smithi181 ceph-mon[35869]: pgmap v8763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:10 smithi097 ceph-mon[32231]: pgmap v8763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15408.031592 secs 2024-07-26T12:16:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15408.031592 secs 2024-07-26T12:16:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15408.031592 secs 2024-07-26T12:16:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:12 smithi115 ceph-mon[35364]: pgmap v8764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:12 smithi181 ceph-mon[35869]: pgmap v8764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:12 smithi097 ceph-mon[32231]: pgmap v8764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:14 smithi115 ceph-mon[35364]: pgmap v8765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:14 smithi181 ceph-mon[35869]: pgmap v8765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:14 smithi097 ceph-mon[32231]: pgmap v8765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15413.031732 secs 2024-07-26T12:16:16.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:16 smithi115 ceph-mon[35364]: pgmap v8766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15413.031732 secs 2024-07-26T12:16:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:16 smithi181 ceph-mon[35869]: pgmap v8766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15413.031732 secs 2024-07-26T12:16:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:16 smithi097 ceph-mon[32231]: pgmap v8766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:18 smithi115 ceph-mon[35364]: pgmap v8767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:18 smithi181 ceph-mon[35869]: pgmap v8767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:18 smithi097 ceph-mon[32231]: pgmap v8767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:20 smithi115 ceph-mon[35364]: pgmap v8768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:20 smithi181 ceph-mon[35869]: pgmap v8768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:20 smithi097 ceph-mon[32231]: pgmap v8768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:16:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15418.031890 secs 2024-07-26T12:16:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15418.031890 secs 2024-07-26T12:16:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15418.031890 secs 2024-07-26T12:16:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:22 smithi115 ceph-mon[35364]: pgmap v8769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:22 smithi181 ceph-mon[35869]: pgmap v8769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:22 smithi097 ceph-mon[32231]: pgmap v8769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:23.932+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:16:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:23.933+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:16:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:23.933+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:16:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:23.933+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:16:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:23.933+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:24 smithi181 ceph-mon[35869]: pgmap v8770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:24 smithi097 ceph-mon[32231]: pgmap v8770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:24 smithi115 ceph-mon[35364]: pgmap v8770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15423.032056 secs 2024-07-26T12:16:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:26 smithi181 ceph-mon[35869]: pgmap v8771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15423.032056 secs 2024-07-26T12:16:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:26 smithi097 ceph-mon[32231]: pgmap v8771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15423.032056 secs 2024-07-26T12:16:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:26 smithi115 ceph-mon[35364]: pgmap v8771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:28 smithi097 ceph-mon[32231]: pgmap v8772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:28 smithi115 ceph-mon[35364]: pgmap v8772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:28 smithi181 ceph-mon[35869]: pgmap v8772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:29.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:16:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:16:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:16:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:16:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:30 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:16:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:30 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:16:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:30 smithi115 ceph-mon[35364]: pgmap v8773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:16:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:30 smithi115 ceph-mon[35364]: pgmap v8774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:30 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:16:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:30 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:16:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:30 smithi181 ceph-mon[35869]: pgmap v8773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:16:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:30 smithi181 ceph-mon[35869]: pgmap v8774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:30 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:16:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:30 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:16:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:30 smithi097 ceph-mon[32231]: pgmap v8773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:16:31.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:30 smithi097 ceph-mon[32231]: pgmap v8774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:16:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15428.032186 secs 2024-07-26T12:16:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15428.032186 secs 2024-07-26T12:16:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15428.032186 secs 2024-07-26T12:16:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:32 smithi115 ceph-mon[35364]: pgmap v8775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:16:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:32 smithi181 ceph-mon[35869]: pgmap v8775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:16:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:32 smithi097 ceph-mon[32231]: pgmap v8775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:16:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:33 smithi115 ceph-mon[35364]: pgmap v8776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:33 smithi181 ceph-mon[35869]: pgmap v8776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:33 smithi097 ceph-mon[32231]: pgmap v8776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:35 smithi115 ceph-mon[35364]: pgmap v8777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:16:35.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15433.032327 secs 2024-07-26T12:16:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:35 smithi181 ceph-mon[35869]: pgmap v8777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:16:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15433.032327 secs 2024-07-26T12:16:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:35 smithi097 ceph-mon[32231]: pgmap v8777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:16:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15433.032327 secs 2024-07-26T12:16:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:37 smithi115 ceph-mon[35364]: pgmap v8778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:37 smithi181 ceph-mon[35869]: pgmap v8778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:37 smithi097 ceph-mon[32231]: pgmap v8778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:38.941+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:16:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:38.942+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:16:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:38.942+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:16:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:38.942+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:16:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:38.942+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:39 smithi115 ceph-mon[35364]: pgmap v8779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:16:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:39 smithi181 ceph-mon[35869]: pgmap v8779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:16:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:39 smithi097 ceph-mon[32231]: pgmap v8779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:16:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15438.032474 secs 2024-07-26T12:16:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15438.032474 secs 2024-07-26T12:16:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15438.032474 secs 2024-07-26T12:16:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:41 smithi115 ceph-mon[35364]: pgmap v8780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:41 smithi181 ceph-mon[35869]: pgmap v8780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:41 smithi097 ceph-mon[32231]: pgmap v8780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:43 smithi115 ceph-mon[35364]: pgmap v8781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:43 smithi181 ceph-mon[35869]: pgmap v8781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:43 smithi097 ceph-mon[32231]: pgmap v8781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:45 smithi115 ceph-mon[35364]: pgmap v8782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:45.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15443.032572 secs 2024-07-26T12:16:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:45 smithi181 ceph-mon[35869]: pgmap v8782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15443.032572 secs 2024-07-26T12:16:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:45 smithi097 ceph-mon[32231]: pgmap v8782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15443.032572 secs 2024-07-26T12:16:47.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:47 smithi115 ceph-mon[35364]: pgmap v8783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:47 smithi181 ceph-mon[35869]: pgmap v8783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:47 smithi097 ceph-mon[32231]: pgmap v8783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:49 smithi115 ceph-mon[35364]: pgmap v8784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:49 smithi181 ceph-mon[35869]: pgmap v8784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:49 smithi097 ceph-mon[32231]: pgmap v8784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15448.032715 secs 2024-07-26T12:16:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15448.032715 secs 2024-07-26T12:16:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15448.032715 secs 2024-07-26T12:16:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:51 smithi181 ceph-mon[35869]: pgmap v8785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:51 smithi097 ceph-mon[32231]: pgmap v8785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:51 smithi115 ceph-mon[35364]: pgmap v8785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:53.945+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:16:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:53.946+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:16:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:53.946+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:16:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:53.946+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:16:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:16:53.946+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:53 smithi097 ceph-mon[32231]: pgmap v8786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:53 smithi115 ceph-mon[35364]: pgmap v8786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:53 smithi181 ceph-mon[35869]: pgmap v8786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:55 smithi097 ceph-mon[32231]: pgmap v8787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:56.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15453.032858 secs 2024-07-26T12:16:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:55 smithi115 ceph-mon[35364]: pgmap v8787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15453.032858 secs 2024-07-26T12:16:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:55 smithi181 ceph-mon[35869]: pgmap v8787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15453.032858 secs 2024-07-26T12:16:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:57 smithi097 ceph-mon[32231]: pgmap v8788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:57 smithi115 ceph-mon[35364]: pgmap v8788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:57 smithi181 ceph-mon[35869]: pgmap v8788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:16:59 smithi115 ceph-mon[35364]: pgmap v8789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:16:59 smithi181 ceph-mon[35869]: pgmap v8789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:16:59 smithi097 ceph-mon[32231]: pgmap v8789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15458.033000 secs 2024-07-26T12:17:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15458.033000 secs 2024-07-26T12:17:01.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15458.033000 secs 2024-07-26T12:17:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:01 smithi115 ceph-mon[35364]: pgmap v8790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:01 smithi181 ceph-mon[35869]: pgmap v8790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:01 smithi097 ceph-mon[32231]: pgmap v8790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:03 smithi115 ceph-mon[35364]: pgmap v8791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:03 smithi181 ceph-mon[35869]: pgmap v8791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:03 smithi097 ceph-mon[32231]: pgmap v8791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:05.800 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:05 smithi097 ceph-mon[32231]: pgmap v8792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:05 smithi115 ceph-mon[35364]: pgmap v8792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15463.033142 secs 2024-07-26T12:17:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:05 smithi181 ceph-mon[35869]: pgmap v8792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15463.033142 secs 2024-07-26T12:17:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15463.033142 secs 2024-07-26T12:17:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:07 smithi115 ceph-mon[35364]: pgmap v8793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:07 smithi181 ceph-mon[35869]: pgmap v8793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:07 smithi097 ceph-mon[32231]: pgmap v8793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:08.954+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:17:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:08.955+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:17:08.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:08.955+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:17:08.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:08.955+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:17:08.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:08.955+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:09 smithi115 ceph-mon[35364]: pgmap v8794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:09 smithi181 ceph-mon[35869]: pgmap v8794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:09 smithi097 ceph-mon[32231]: pgmap v8794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15468.033327 secs 2024-07-26T12:17:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15468.033327 secs 2024-07-26T12:17:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15468.033327 secs 2024-07-26T12:17:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:11 smithi115 ceph-mon[35364]: pgmap v8795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:11 smithi181 ceph-mon[35869]: pgmap v8795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:11 smithi097 ceph-mon[32231]: pgmap v8795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:13 smithi115 ceph-mon[35364]: pgmap v8796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:13 smithi181 ceph-mon[35869]: pgmap v8796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:13 smithi097 ceph-mon[32231]: pgmap v8796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:15 smithi115 ceph-mon[35364]: pgmap v8797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15473.033467 secs 2024-07-26T12:17:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:15 smithi181 ceph-mon[35869]: pgmap v8797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15473.033467 secs 2024-07-26T12:17:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:15 smithi097 ceph-mon[32231]: pgmap v8797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15473.033467 secs 2024-07-26T12:17:18.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:17 smithi115 ceph-mon[35364]: pgmap v8798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:17 smithi181 ceph-mon[35869]: pgmap v8798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:17 smithi097 ceph-mon[32231]: pgmap v8798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:19 smithi115 ceph-mon[35364]: pgmap v8799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:19 smithi181 ceph-mon[35869]: pgmap v8799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:19 smithi097 ceph-mon[32231]: pgmap v8799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:21.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15478.033572 secs 2024-07-26T12:17:21.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15478.033572 secs 2024-07-26T12:17:21.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15478.033572 secs 2024-07-26T12:17:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:21 smithi115 ceph-mon[35364]: pgmap v8800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:21 smithi181 ceph-mon[35869]: pgmap v8800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:21 smithi097 ceph-mon[32231]: pgmap v8800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:23.957+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:17:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:23.959+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:17:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:23.959+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:17:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:23.959+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:17:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:23.959+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:23 smithi115 ceph-mon[35364]: pgmap v8801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:23 smithi181 ceph-mon[35869]: pgmap v8801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:23 smithi097 ceph-mon[32231]: pgmap v8801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:25 smithi115 ceph-mon[35364]: pgmap v8802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15483.033699 secs 2024-07-26T12:17:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:25 smithi181 ceph-mon[35869]: pgmap v8802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15483.033699 secs 2024-07-26T12:17:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:25 smithi097 ceph-mon[32231]: pgmap v8802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15483.033699 secs 2024-07-26T12:17:28.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:27 smithi115 ceph-mon[35364]: pgmap v8803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:27 smithi181 ceph-mon[35869]: pgmap v8803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:27 smithi097 ceph-mon[32231]: pgmap v8803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:29 smithi115 ceph-mon[35364]: pgmap v8804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:30.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:29 smithi181 ceph-mon[35869]: pgmap v8804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:29 smithi097 ceph-mon[32231]: pgmap v8804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15488.033854 secs 2024-07-26T12:17:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:30 smithi115 ceph-mon[35364]: pgmap v8805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:30 smithi115 ceph-mon[35364]: pgmap v8806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15488.033854 secs 2024-07-26T12:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:30 smithi181 ceph-mon[35869]: pgmap v8805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:30 smithi181 ceph-mon[35869]: pgmap v8806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15488.033854 secs 2024-07-26T12:17:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:30 smithi097 ceph-mon[32231]: pgmap v8805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:30 smithi097 ceph-mon[32231]: pgmap v8806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:17:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:32 smithi115 ceph-mon[35364]: pgmap v8807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:32 smithi181 ceph-mon[35869]: pgmap v8807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:32 smithi097 ceph-mon[32231]: pgmap v8807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:34 smithi115 ceph-mon[35364]: pgmap v8808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:34 smithi181 ceph-mon[35869]: pgmap v8808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:34 smithi097 ceph-mon[32231]: pgmap v8808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15493.034011 secs 2024-07-26T12:17:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15493.034011 secs 2024-07-26T12:17:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15493.034011 secs 2024-07-26T12:17:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:36 smithi115 ceph-mon[35364]: pgmap v8809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:36 smithi181 ceph-mon[35869]: pgmap v8809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:36 smithi097 ceph-mon[32231]: pgmap v8809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:38.963+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:17:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:38.964+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:17:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:38.964+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:17:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:38.964+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:17:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:38.964+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:38 smithi115 ceph-mon[35364]: pgmap v8810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:38 smithi181 ceph-mon[35869]: pgmap v8810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:38 smithi097 ceph-mon[32231]: pgmap v8810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:17:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15498.034152 secs 2024-07-26T12:17:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:40 smithi115 ceph-mon[35364]: pgmap v8811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15498.034152 secs 2024-07-26T12:17:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:40 smithi181 ceph-mon[35869]: pgmap v8811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15498.034152 secs 2024-07-26T12:17:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:40 smithi097 ceph-mon[32231]: pgmap v8811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:42 smithi115 ceph-mon[35364]: pgmap v8812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:42 smithi181 ceph-mon[35869]: pgmap v8812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:42 smithi097 ceph-mon[32231]: pgmap v8812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:44 smithi115 ceph-mon[35364]: pgmap v8813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:44 smithi181 ceph-mon[35869]: pgmap v8813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:44 smithi097 ceph-mon[32231]: pgmap v8813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15503.034300 secs 2024-07-26T12:17:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15503.034300 secs 2024-07-26T12:17:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15503.034300 secs 2024-07-26T12:17:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:46 smithi115 ceph-mon[35364]: pgmap v8814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:46 smithi181 ceph-mon[35869]: pgmap v8814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:46 smithi097 ceph-mon[32231]: pgmap v8814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:48 smithi115 ceph-mon[35364]: pgmap v8815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:48 smithi181 ceph-mon[35869]: pgmap v8815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:48 smithi097 ceph-mon[32231]: pgmap v8815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15508.034445 secs 2024-07-26T12:17:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:50 smithi115 ceph-mon[35364]: pgmap v8816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15508.034445 secs 2024-07-26T12:17:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:50 smithi181 ceph-mon[35869]: pgmap v8816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15508.034445 secs 2024-07-26T12:17:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:50 smithi097 ceph-mon[32231]: pgmap v8816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:52 smithi115 ceph-mon[35364]: pgmap v8817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:52 smithi181 ceph-mon[35869]: pgmap v8817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:52 smithi097 ceph-mon[32231]: pgmap v8817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:53.967+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:17:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:53.968+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:17:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:53.968+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:17:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:53.968+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:17:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:17:53.968+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:54 smithi115 ceph-mon[35364]: pgmap v8818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:54 smithi181 ceph-mon[35869]: pgmap v8818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:54 smithi097 ceph-mon[32231]: pgmap v8818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15513.034583 secs 2024-07-26T12:17:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15513.034583 secs 2024-07-26T12:17:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15513.034583 secs 2024-07-26T12:17:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:56 smithi115 ceph-mon[35364]: pgmap v8819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:56 smithi181 ceph-mon[35869]: pgmap v8819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:56 smithi097 ceph-mon[32231]: pgmap v8819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:17:58 smithi115 ceph-mon[35364]: pgmap v8820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:17:58 smithi181 ceph-mon[35869]: pgmap v8820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:17:58 smithi097 ceph-mon[32231]: pgmap v8820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:00 smithi115 ceph-mon[35364]: pgmap v8821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:00 smithi181 ceph-mon[35869]: pgmap v8821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:00 smithi097 ceph-mon[32231]: pgmap v8821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15518.034723 secs 2024-07-26T12:18:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15518.034723 secs 2024-07-26T12:18:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15518.034723 secs 2024-07-26T12:18:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:02 smithi115 ceph-mon[35364]: pgmap v8822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:02 smithi181 ceph-mon[35869]: pgmap v8822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:02 smithi097 ceph-mon[32231]: pgmap v8822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:04 smithi115 ceph-mon[35364]: pgmap v8823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:04 smithi181 ceph-mon[35869]: pgmap v8823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:04 smithi097 ceph-mon[32231]: pgmap v8823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15523.034888 secs 2024-07-26T12:18:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15523.034888 secs 2024-07-26T12:18:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15523.034888 secs 2024-07-26T12:18:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:06 smithi115 ceph-mon[35364]: pgmap v8824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:06 smithi181 ceph-mon[35869]: pgmap v8824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:06 smithi097 ceph-mon[32231]: pgmap v8824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:08 smithi115 ceph-mon[35364]: pgmap v8825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:08.976+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:18:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:08.977+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:18:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:08.977+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:18:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:08.977+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:18:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:08.977+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:08 smithi181 ceph-mon[35869]: pgmap v8825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:08 smithi097 ceph-mon[32231]: pgmap v8825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15528.035023 secs 2024-07-26T12:18:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:10 smithi115 ceph-mon[35364]: pgmap v8826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15528.035023 secs 2024-07-26T12:18:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:10 smithi181 ceph-mon[35869]: pgmap v8826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15528.035023 secs 2024-07-26T12:18:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:10 smithi097 ceph-mon[32231]: pgmap v8826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:12 smithi115 ceph-mon[35364]: pgmap v8827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:12 smithi181 ceph-mon[35869]: pgmap v8827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:12 smithi097 ceph-mon[32231]: pgmap v8827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:14 smithi115 ceph-mon[35364]: pgmap v8828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:14 smithi181 ceph-mon[35869]: pgmap v8828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:14 smithi097 ceph-mon[32231]: pgmap v8828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15533.035107 secs 2024-07-26T12:18:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15533.035107 secs 2024-07-26T12:18:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15533.035107 secs 2024-07-26T12:18:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:16 smithi115 ceph-mon[35364]: pgmap v8829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:16 smithi181 ceph-mon[35869]: pgmap v8829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:16 smithi097 ceph-mon[32231]: pgmap v8829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:18 smithi115 ceph-mon[35364]: pgmap v8830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:18 smithi181 ceph-mon[35869]: pgmap v8830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:18 smithi097 ceph-mon[32231]: pgmap v8830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15538.035292 secs 2024-07-26T12:18:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:20 smithi115 ceph-mon[35364]: pgmap v8831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15538.035292 secs 2024-07-26T12:18:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:20 smithi181 ceph-mon[35869]: pgmap v8831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15538.035292 secs 2024-07-26T12:18:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:20 smithi097 ceph-mon[32231]: pgmap v8831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:22 smithi115 ceph-mon[35364]: pgmap v8832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:22 smithi181 ceph-mon[35869]: pgmap v8832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:22 smithi097 ceph-mon[32231]: pgmap v8832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:23.979+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:18:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:23.980+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:18:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:23.980+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:18:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:23.980+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:18:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:23.980+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:24 smithi115 ceph-mon[35364]: pgmap v8833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:24 smithi181 ceph-mon[35869]: pgmap v8833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:24 smithi097 ceph-mon[32231]: pgmap v8833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15543.035388 secs 2024-07-26T12:18:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15543.035388 secs 2024-07-26T12:18:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15543.035388 secs 2024-07-26T12:18:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:26 smithi115 ceph-mon[35364]: pgmap v8834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:26 smithi181 ceph-mon[35869]: pgmap v8834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:26 smithi097 ceph-mon[32231]: pgmap v8834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:28 smithi115 ceph-mon[35364]: pgmap v8835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:28 smithi181 ceph-mon[35869]: pgmap v8835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:28 smithi097 ceph-mon[32231]: pgmap v8835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15548.035529 secs 2024-07-26T12:18:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:30 smithi115 ceph-mon[35364]: pgmap v8836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15548.035529 secs 2024-07-26T12:18:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:30 smithi181 ceph-mon[35869]: pgmap v8836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15548.035529 secs 2024-07-26T12:18:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:30 smithi097 ceph-mon[32231]: pgmap v8836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:31.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:32 smithi115 ceph-mon[35364]: pgmap v8837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:32 smithi115 ceph-mon[35364]: pgmap v8838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T12:18:32.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:32 smithi181 ceph-mon[35869]: pgmap v8837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:32 smithi181 ceph-mon[35869]: pgmap v8838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T12:18:32.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:32 smithi097 ceph-mon[32231]: pgmap v8837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:32 smithi097 ceph-mon[32231]: pgmap v8838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T12:18:33.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:18:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:34 smithi115 ceph-mon[35364]: pgmap v8839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:34 smithi181 ceph-mon[35869]: pgmap v8839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:34 smithi097 ceph-mon[32231]: pgmap v8839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:35.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15553.035687 secs 2024-07-26T12:18:35.998 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15553.035687 secs 2024-07-26T12:18:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15553.035687 secs 2024-07-26T12:18:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:36 smithi115 ceph-mon[35364]: pgmap v8840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:36 smithi181 ceph-mon[35869]: pgmap v8840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:36 smithi097 ceph-mon[32231]: pgmap v8840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:38 smithi115 ceph-mon[35364]: pgmap v8841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:38 smithi181 ceph-mon[35869]: pgmap v8841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:38.986+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:18:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:38.987+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:18:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:38.987+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:18:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:38.987+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:18:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:38.987+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:38 smithi097 ceph-mon[32231]: pgmap v8841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15558.035835 secs 2024-07-26T12:18:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:40 smithi115 ceph-mon[35364]: pgmap v8842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15558.035835 secs 2024-07-26T12:18:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:40 smithi181 ceph-mon[35869]: pgmap v8842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15558.035835 secs 2024-07-26T12:18:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:40 smithi097 ceph-mon[32231]: pgmap v8842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:18:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:42 smithi115 ceph-mon[35364]: pgmap v8843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:42 smithi181 ceph-mon[35869]: pgmap v8843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:42 smithi097 ceph-mon[32231]: pgmap v8843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:44 smithi115 ceph-mon[35364]: pgmap v8844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:18:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:44 smithi181 ceph-mon[35869]: pgmap v8844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:18:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:44 smithi097 ceph-mon[32231]: pgmap v8844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:18:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15563.035977 secs 2024-07-26T12:18:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15563.035977 secs 2024-07-26T12:18:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15563.035977 secs 2024-07-26T12:18:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:46 smithi115 ceph-mon[35364]: pgmap v8845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:18:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:46 smithi181 ceph-mon[35869]: pgmap v8845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:18:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:46 smithi097 ceph-mon[32231]: pgmap v8845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:18:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:48 smithi115 ceph-mon[35364]: pgmap v8846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:18:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:48 smithi181 ceph-mon[35869]: pgmap v8846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:18:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:48 smithi097 ceph-mon[32231]: pgmap v8846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:18:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15568.036119 secs 2024-07-26T12:18:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:50 smithi115 ceph-mon[35364]: pgmap v8847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:18:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15568.036119 secs 2024-07-26T12:18:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:50 smithi181 ceph-mon[35869]: pgmap v8847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:18:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15568.036119 secs 2024-07-26T12:18:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:50 smithi097 ceph-mon[32231]: pgmap v8847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:18:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:52 smithi115 ceph-mon[35364]: pgmap v8848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:18:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:52 smithi181 ceph-mon[35869]: pgmap v8848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:18:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:52 smithi097 ceph-mon[32231]: pgmap v8848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:18:53.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:53.991+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:18:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:53.992+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:18:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:53.992+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:18:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:53.992+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:18:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:18:53.992+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:54 smithi115 ceph-mon[35364]: pgmap v8849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:18:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:54 smithi181 ceph-mon[35869]: pgmap v8849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:18:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:54 smithi097 ceph-mon[32231]: pgmap v8849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:18:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15573.036267 secs 2024-07-26T12:18:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15573.036267 secs 2024-07-26T12:18:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15573.036267 secs 2024-07-26T12:18:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:56 smithi115 ceph-mon[35364]: pgmap v8850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:56 smithi181 ceph-mon[35869]: pgmap v8850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:56 smithi097 ceph-mon[32231]: pgmap v8850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:18:58 smithi115 ceph-mon[35364]: pgmap v8851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:18:58 smithi181 ceph-mon[35869]: pgmap v8851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:18:58 smithi097 ceph-mon[32231]: pgmap v8851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15578.036414 secs 2024-07-26T12:19:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:00 smithi115 ceph-mon[35364]: pgmap v8852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15578.036414 secs 2024-07-26T12:19:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:00 smithi181 ceph-mon[35869]: pgmap v8852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15578.036414 secs 2024-07-26T12:19:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:00 smithi097 ceph-mon[32231]: pgmap v8852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:02 smithi115 ceph-mon[35364]: pgmap v8853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:02 smithi181 ceph-mon[35869]: pgmap v8853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:02 smithi097 ceph-mon[32231]: pgmap v8853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:04 smithi115 ceph-mon[35364]: pgmap v8854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:04 smithi181 ceph-mon[35869]: pgmap v8854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:04 smithi097 ceph-mon[32231]: pgmap v8854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15583.036554 secs 2024-07-26T12:19:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15583.036554 secs 2024-07-26T12:19:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15583.036554 secs 2024-07-26T12:19:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:06 smithi115 ceph-mon[35364]: pgmap v8855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:06 smithi181 ceph-mon[35869]: pgmap v8855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:06 smithi097 ceph-mon[32231]: pgmap v8855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:08 smithi115 ceph-mon[35364]: pgmap v8856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:08 smithi181 ceph-mon[35869]: pgmap v8856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:08.999+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:19:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:08 smithi097 ceph-mon[32231]: pgmap v8856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:09.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:09.000+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:19:09.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:09.000+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:19:09.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:09.000+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:19:09.002 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:09.000+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:09.670+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:19:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:09.670+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:19:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:09.671+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c00034a0 con 0x7ff3f01061a0 2024-07-26T12:19:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:09.671+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:19:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15588.036694 secs 2024-07-26T12:19:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:10 smithi115 ceph-mon[35364]: pgmap v8857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15588.036694 secs 2024-07-26T12:19:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:10 smithi181 ceph-mon[35869]: pgmap v8857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15588.036694 secs 2024-07-26T12:19:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:10 smithi097 ceph-mon[32231]: pgmap v8857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:12 smithi115 ceph-mon[35364]: pgmap v8858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:12 smithi181 ceph-mon[35869]: pgmap v8858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:12 smithi097 ceph-mon[32231]: pgmap v8858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:14 smithi115 ceph-mon[35364]: pgmap v8859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:14 smithi181 ceph-mon[35869]: pgmap v8859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:14 smithi097 ceph-mon[32231]: pgmap v8859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15593.036850 secs 2024-07-26T12:19:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15593.036850 secs 2024-07-26T12:19:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15593.036850 secs 2024-07-26T12:19:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:16 smithi115 ceph-mon[35364]: pgmap v8860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:16 smithi181 ceph-mon[35869]: pgmap v8860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:16 smithi097 ceph-mon[32231]: pgmap v8860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:18 smithi115 ceph-mon[35364]: pgmap v8861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:18 smithi181 ceph-mon[35869]: pgmap v8861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:18 smithi097 ceph-mon[32231]: pgmap v8861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15598.036989 secs 2024-07-26T12:19:20.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:20 smithi115 ceph-mon[35364]: pgmap v8862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15598.036989 secs 2024-07-26T12:19:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:20 smithi181 ceph-mon[35869]: pgmap v8862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15598.036989 secs 2024-07-26T12:19:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:20 smithi097 ceph-mon[32231]: pgmap v8862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:22 smithi115 ceph-mon[35364]: pgmap v8863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:22 smithi181 ceph-mon[35869]: pgmap v8863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:22 smithi097 ceph-mon[32231]: pgmap v8863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:24.004 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:24.003+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:19:24.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:24.004+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:19:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:24.004+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:19:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:24.004+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:19:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:24.004+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:24 smithi115 ceph-mon[35364]: pgmap v8864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:24 smithi181 ceph-mon[35869]: pgmap v8864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:24 smithi097 ceph-mon[32231]: pgmap v8864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15603.037148 secs 2024-07-26T12:19:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15603.037148 secs 2024-07-26T12:19:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15603.037148 secs 2024-07-26T12:19:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:26 smithi115 ceph-mon[35364]: pgmap v8865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:26 smithi181 ceph-mon[35869]: pgmap v8865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:26 smithi097 ceph-mon[32231]: pgmap v8865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:28 smithi115 ceph-mon[35364]: pgmap v8866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:28 smithi181 ceph-mon[35869]: pgmap v8866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:28 smithi097 ceph-mon[32231]: pgmap v8866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15608.037283 secs 2024-07-26T12:19:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:30 smithi115 ceph-mon[35364]: pgmap v8867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15608.037283 secs 2024-07-26T12:19:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:30 smithi181 ceph-mon[35869]: pgmap v8867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15608.037283 secs 2024-07-26T12:19:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:30 smithi097 ceph-mon[32231]: pgmap v8867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:32 smithi115 ceph-mon[35364]: pgmap v8868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:32 smithi181 ceph-mon[35869]: pgmap v8868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:32 smithi097 ceph-mon[32231]: pgmap v8868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:33.880 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:33.884 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:33.900 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:34 smithi115 ceph-mon[35364]: pgmap v8869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:34 smithi181 ceph-mon[35869]: pgmap v8869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:34 smithi097 ceph-mon[32231]: pgmap v8869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15613.037387 secs 2024-07-26T12:19:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15613.037387 secs 2024-07-26T12:19:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15613.037387 secs 2024-07-26T12:19:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:36 smithi115 ceph-mon[35364]: pgmap v8870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:36 smithi181 ceph-mon[35869]: pgmap v8870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:36 smithi097 ceph-mon[32231]: pgmap v8870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:38.869 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:38 smithi097 ceph-mon[32231]: pgmap v8871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:38 smithi115 ceph-mon[35364]: pgmap v8871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:38 smithi181 ceph-mon[35869]: pgmap v8871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:39.012+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:19:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:39.013+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:19:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:39.013+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:19:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:39.013+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:19:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:39.013+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15618.037532 secs 2024-07-26T12:19:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:40 smithi115 ceph-mon[35364]: pgmap v8872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15618.037532 secs 2024-07-26T12:19:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:40 smithi181 ceph-mon[35869]: pgmap v8872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15618.037532 secs 2024-07-26T12:19:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:40 smithi097 ceph-mon[32231]: pgmap v8872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:43.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:43.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:42 smithi181 ceph-mon[35869]: pgmap v8873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:19:43.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:42 smithi181 ceph-mon[35869]: pgmap v8874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:19:43.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:42 smithi097 ceph-mon[32231]: pgmap v8873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:19:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:42 smithi097 ceph-mon[32231]: pgmap v8874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:19:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:42 smithi115 ceph-mon[35364]: pgmap v8873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:19:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:42 smithi115 ceph-mon[35364]: pgmap v8874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:19:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:19:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:45 smithi115 ceph-mon[35364]: pgmap v8875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:19:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:45 smithi181 ceph-mon[35869]: pgmap v8875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:19:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:45 smithi097 ceph-mon[32231]: pgmap v8875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:19:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15623.037627 secs 2024-07-26T12:19:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15623.037627 secs 2024-07-26T12:19:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15623.037627 secs 2024-07-26T12:19:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:47 smithi115 ceph-mon[35364]: pgmap v8876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:47 smithi181 ceph-mon[35869]: pgmap v8876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:47 smithi097 ceph-mon[32231]: pgmap v8876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:49 smithi115 ceph-mon[35364]: pgmap v8877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:49 smithi181 ceph-mon[35869]: pgmap v8877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:49 smithi097 ceph-mon[32231]: pgmap v8877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:51 smithi115 ceph-mon[35364]: pgmap v8878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:51.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15628.037785 secs 2024-07-26T12:19:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:51 smithi181 ceph-mon[35869]: pgmap v8878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15628.037785 secs 2024-07-26T12:19:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:51 smithi097 ceph-mon[32231]: pgmap v8878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:19:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15628.037785 secs 2024-07-26T12:19:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:53 smithi115 ceph-mon[35364]: pgmap v8879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:53 smithi181 ceph-mon[35869]: pgmap v8879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:53 smithi097 ceph-mon[32231]: pgmap v8879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:54.016+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:19:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:54.017+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:19:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:54.017+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:19:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:54.017+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:19:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:19:54.017+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:55 smithi115 ceph-mon[35364]: pgmap v8880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:19:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:55 smithi181 ceph-mon[35869]: pgmap v8880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:19:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:55 smithi097 ceph-mon[32231]: pgmap v8880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:19:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15633.037931 secs 2024-07-26T12:19:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15633.037931 secs 2024-07-26T12:19:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15633.037931 secs 2024-07-26T12:19:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:57 smithi115 ceph-mon[35364]: pgmap v8881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:19:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:57 smithi181 ceph-mon[35869]: pgmap v8881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:19:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:57 smithi097 ceph-mon[32231]: pgmap v8881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:19:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:19:59 smithi115 ceph-mon[35364]: pgmap v8882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:19:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:19:59 smithi181 ceph-mon[35869]: pgmap v8882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:19:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:19:59 smithi097 ceph-mon[32231]: pgmap v8882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:20:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:01 smithi115 ceph-mon[35364]: pgmap v8883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15638.038085 secs 2024-07-26T12:20:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:01 smithi181 ceph-mon[35869]: pgmap v8883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15638.038085 secs 2024-07-26T12:20:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:01 smithi097 ceph-mon[32231]: pgmap v8883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15638.038085 secs 2024-07-26T12:20:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:03 smithi115 ceph-mon[35364]: pgmap v8884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:20:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:03 smithi181 ceph-mon[35869]: pgmap v8884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:20:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:03 smithi097 ceph-mon[32231]: pgmap v8884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:20:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:05 smithi115 ceph-mon[35364]: pgmap v8885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:20:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:05 smithi181 ceph-mon[35869]: pgmap v8885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:20:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:05 smithi097 ceph-mon[32231]: pgmap v8885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:20:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15643.038267 secs 2024-07-26T12:20:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15643.038267 secs 2024-07-26T12:20:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15643.038267 secs 2024-07-26T12:20:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:07 smithi115 ceph-mon[35364]: pgmap v8886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:07 smithi181 ceph-mon[35869]: pgmap v8886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:07 smithi097 ceph-mon[32231]: pgmap v8886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:09.021+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:09.023+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:20:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:09.023+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:20:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:09.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:09.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:09 smithi115 ceph-mon[35364]: pgmap v8887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:09 smithi181 ceph-mon[35869]: pgmap v8887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:09 smithi097 ceph-mon[32231]: pgmap v8887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:11 smithi115 ceph-mon[35364]: pgmap v8888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15648.038361 secs 2024-07-26T12:20:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:11 smithi181 ceph-mon[35869]: pgmap v8888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15648.038361 secs 2024-07-26T12:20:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:11 smithi097 ceph-mon[32231]: pgmap v8888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15648.038361 secs 2024-07-26T12:20:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:13 smithi115 ceph-mon[35364]: pgmap v8889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:13 smithi181 ceph-mon[35869]: pgmap v8889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:13 smithi097 ceph-mon[32231]: pgmap v8889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:15 smithi115 ceph-mon[35364]: pgmap v8890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:15 smithi181 ceph-mon[35869]: pgmap v8890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:15 smithi097 ceph-mon[32231]: pgmap v8890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15653.038496 secs 2024-07-26T12:20:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15653.038496 secs 2024-07-26T12:20:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15653.038496 secs 2024-07-26T12:20:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:17 smithi115 ceph-mon[35364]: pgmap v8891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:17 smithi181 ceph-mon[35869]: pgmap v8891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:17 smithi097 ceph-mon[32231]: pgmap v8891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:19 smithi181 ceph-mon[35869]: pgmap v8892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:19 smithi097 ceph-mon[32231]: pgmap v8892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:19 smithi115 ceph-mon[35364]: pgmap v8892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:21 smithi097 ceph-mon[32231]: pgmap v8893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:21.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15658.038672 secs 2024-07-26T12:20:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:21 smithi115 ceph-mon[35364]: pgmap v8893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15658.038672 secs 2024-07-26T12:20:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:21 smithi181 ceph-mon[35869]: pgmap v8893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15658.038672 secs 2024-07-26T12:20:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:23 smithi115 ceph-mon[35364]: pgmap v8894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:23 smithi181 ceph-mon[35869]: pgmap v8894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:23 smithi097 ceph-mon[32231]: pgmap v8894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:24.024+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:24.025+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:20:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:24.025+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:20:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:24.025+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:24.025+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:25 smithi115 ceph-mon[35364]: pgmap v8895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:25 smithi181 ceph-mon[35869]: pgmap v8895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:25 smithi097 ceph-mon[32231]: pgmap v8895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15663.038801 secs 2024-07-26T12:20:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15663.038801 secs 2024-07-26T12:20:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15663.038801 secs 2024-07-26T12:20:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:27 smithi115 ceph-mon[35364]: pgmap v8896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:27 smithi181 ceph-mon[35869]: pgmap v8896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:27 smithi097 ceph-mon[32231]: pgmap v8896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:29 smithi115 ceph-mon[35364]: pgmap v8897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:29 smithi181 ceph-mon[35869]: pgmap v8897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:29 smithi097 ceph-mon[32231]: pgmap v8897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:31 smithi115 ceph-mon[35364]: pgmap v8898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15668.038946 secs 2024-07-26T12:20:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:31 smithi181 ceph-mon[35869]: pgmap v8898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15668.038946 secs 2024-07-26T12:20:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:31 smithi097 ceph-mon[32231]: pgmap v8898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15668.038946 secs 2024-07-26T12:20:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:33 smithi115 ceph-mon[35364]: pgmap v8899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:33 smithi181 ceph-mon[35869]: pgmap v8899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:33 smithi097 ceph-mon[32231]: pgmap v8899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:35 smithi115 ceph-mon[35364]: pgmap v8900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:35 smithi181 ceph-mon[35869]: pgmap v8900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:35 smithi097 ceph-mon[32231]: pgmap v8900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15673.039096 secs 2024-07-26T12:20:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15673.039096 secs 2024-07-26T12:20:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15673.039096 secs 2024-07-26T12:20:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:37 smithi115 ceph-mon[35364]: pgmap v8901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:37 smithi181 ceph-mon[35869]: pgmap v8901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:37 smithi097 ceph-mon[32231]: pgmap v8901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:39.032+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:39.034+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:20:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:39.034+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:20:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:39.034+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:39.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:39.034+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:39 smithi115 ceph-mon[35364]: pgmap v8902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:39 smithi181 ceph-mon[35869]: pgmap v8902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:39 smithi097 ceph-mon[32231]: pgmap v8902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:41 smithi115 ceph-mon[35364]: pgmap v8903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15678.039273 secs 2024-07-26T12:20:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:41 smithi181 ceph-mon[35869]: pgmap v8903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15678.039273 secs 2024-07-26T12:20:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:41 smithi097 ceph-mon[32231]: pgmap v8903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15678.039273 secs 2024-07-26T12:20:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:42 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:42 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:42 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:43 smithi115 ceph-mon[35364]: pgmap v8904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:43 smithi181 ceph-mon[35869]: pgmap v8904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:43 smithi097 ceph-mon[32231]: pgmap v8904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:45 smithi181 ceph-mon[35869]: pgmap v8905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:45 smithi115 ceph-mon[35364]: pgmap v8905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:45 smithi097 ceph-mon[32231]: pgmap v8905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15683.039346 secs 2024-07-26T12:20:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15683.039346 secs 2024-07-26T12:20:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15683.039346 secs 2024-07-26T12:20:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:47 smithi115 ceph-mon[35364]: pgmap v8906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:47 smithi181 ceph-mon[35869]: pgmap v8906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:47 smithi097 ceph-mon[32231]: pgmap v8906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: pgmap v8907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: pgmap v8908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:20:49.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: pgmap v8909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:20:49.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: pgmap v8907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:49.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:49.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: pgmap v8908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:20:49.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: pgmap v8909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:20:49.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: pgmap v8907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: pgmap v8908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: pgmap v8909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:20:49.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:20:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15688.039488 secs 2024-07-26T12:20:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:50 smithi115 ceph-mon[35364]: pgmap v8910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:20:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15688.039488 secs 2024-07-26T12:20:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:50 smithi181 ceph-mon[35869]: pgmap v8910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:20:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15688.039488 secs 2024-07-26T12:20:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:50 smithi097 ceph-mon[32231]: pgmap v8910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:20:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:52 smithi115 ceph-mon[35364]: pgmap v8911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:20:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:52 smithi181 ceph-mon[35869]: pgmap v8911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:20:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:52 smithi097 ceph-mon[32231]: pgmap v8911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:20:54.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:54.036+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:20:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:54.037+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:20:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:54.037+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:20:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:54.037+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:20:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:20:54.037+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:54 smithi115 ceph-mon[35364]: pgmap v8912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:20:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:54 smithi181 ceph-mon[35869]: pgmap v8912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:20:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:54 smithi097 ceph-mon[32231]: pgmap v8912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:20:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15693.039620 secs 2024-07-26T12:20:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15693.039620 secs 2024-07-26T12:20:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15693.039620 secs 2024-07-26T12:20:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:56 smithi115 ceph-mon[35364]: pgmap v8913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:20:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:56 smithi181 ceph-mon[35869]: pgmap v8913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:20:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:56 smithi097 ceph-mon[32231]: pgmap v8913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:20:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:20:58 smithi115 ceph-mon[35364]: pgmap v8914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:20:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:20:58 smithi181 ceph-mon[35869]: pgmap v8914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:20:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:20:58 smithi097 ceph-mon[32231]: pgmap v8914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15698.039782 secs 2024-07-26T12:21:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:00 smithi115 ceph-mon[35364]: pgmap v8915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15698.039782 secs 2024-07-26T12:21:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:00 smithi181 ceph-mon[35869]: pgmap v8915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15698.039782 secs 2024-07-26T12:21:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:00 smithi097 ceph-mon[32231]: pgmap v8915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:02 smithi115 ceph-mon[35364]: pgmap v8916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:02 smithi181 ceph-mon[35869]: pgmap v8916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:02 smithi097 ceph-mon[32231]: pgmap v8916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:04 smithi115 ceph-mon[35364]: pgmap v8917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:04 smithi181 ceph-mon[35869]: pgmap v8917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:04 smithi097 ceph-mon[32231]: pgmap v8917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15703.039919 secs 2024-07-26T12:21:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15703.039919 secs 2024-07-26T12:21:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15703.039919 secs 2024-07-26T12:21:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:06 smithi115 ceph-mon[35364]: pgmap v8918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:06 smithi181 ceph-mon[35869]: pgmap v8918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:06 smithi097 ceph-mon[32231]: pgmap v8918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:08 smithi115 ceph-mon[35364]: pgmap v8919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:08 smithi181 ceph-mon[35869]: pgmap v8919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:08 smithi097 ceph-mon[32231]: pgmap v8919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:09.046 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:09.045+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:21:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:09.046+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:21:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:09.046+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:21:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:09.046+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:21:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:09.046+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15708.040057 secs 2024-07-26T12:21:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:10 smithi115 ceph-mon[35364]: pgmap v8920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15708.040057 secs 2024-07-26T12:21:10.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:10 smithi181 ceph-mon[35869]: pgmap v8920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15708.040057 secs 2024-07-26T12:21:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:10 smithi097 ceph-mon[32231]: pgmap v8920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:12 smithi115 ceph-mon[35364]: pgmap v8921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:12 smithi181 ceph-mon[35869]: pgmap v8921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:12 smithi097 ceph-mon[32231]: pgmap v8921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:14 smithi115 ceph-mon[35364]: pgmap v8922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:14 smithi181 ceph-mon[35869]: pgmap v8922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:14 smithi097 ceph-mon[32231]: pgmap v8922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15713.040214 secs 2024-07-26T12:21:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15713.040214 secs 2024-07-26T12:21:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15713.040214 secs 2024-07-26T12:21:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:16 smithi115 ceph-mon[35364]: pgmap v8923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:16 smithi181 ceph-mon[35869]: pgmap v8923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:16 smithi097 ceph-mon[32231]: pgmap v8923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:18 smithi115 ceph-mon[35364]: pgmap v8924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:18 smithi181 ceph-mon[35869]: pgmap v8924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:18 smithi097 ceph-mon[32231]: pgmap v8924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15718.040298 secs 2024-07-26T12:21:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:20 smithi115 ceph-mon[35364]: pgmap v8925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15718.040298 secs 2024-07-26T12:21:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:20 smithi181 ceph-mon[35869]: pgmap v8925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15718.040298 secs 2024-07-26T12:21:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:20 smithi097 ceph-mon[32231]: pgmap v8925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:22 smithi115 ceph-mon[35364]: pgmap v8926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:22 smithi181 ceph-mon[35869]: pgmap v8926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:22 smithi097 ceph-mon[32231]: pgmap v8926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:24.047+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:21:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:24.048+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:21:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:24.048+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:21:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:24.048+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:21:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:24.049+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:24 smithi115 ceph-mon[35364]: pgmap v8927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:24.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:24 smithi181 ceph-mon[35869]: pgmap v8927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:24 smithi097 ceph-mon[32231]: pgmap v8927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15723.040444 secs 2024-07-26T12:21:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15723.040444 secs 2024-07-26T12:21:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15723.040444 secs 2024-07-26T12:21:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:26 smithi115 ceph-mon[35364]: pgmap v8928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:26 smithi181 ceph-mon[35869]: pgmap v8928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:26 smithi097 ceph-mon[32231]: pgmap v8928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:28 smithi115 ceph-mon[35364]: pgmap v8929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:28 smithi181 ceph-mon[35869]: pgmap v8929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:28 smithi097 ceph-mon[32231]: pgmap v8929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15728.040538 secs 2024-07-26T12:21:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:30 smithi115 ceph-mon[35364]: pgmap v8930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15728.040538 secs 2024-07-26T12:21:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:30 smithi181 ceph-mon[35869]: pgmap v8930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15728.040538 secs 2024-07-26T12:21:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:30 smithi097 ceph-mon[32231]: pgmap v8930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:32 smithi115 ceph-mon[35364]: pgmap v8931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:32 smithi181 ceph-mon[35869]: pgmap v8931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:32 smithi097 ceph-mon[32231]: pgmap v8931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:34 smithi115 ceph-mon[35364]: pgmap v8932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:34 smithi181 ceph-mon[35869]: pgmap v8932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:34 smithi097 ceph-mon[32231]: pgmap v8932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15733.040681 secs 2024-07-26T12:21:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15733.040681 secs 2024-07-26T12:21:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15733.040681 secs 2024-07-26T12:21:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:36 smithi115 ceph-mon[35364]: pgmap v8933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:36 smithi181 ceph-mon[35869]: pgmap v8933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:36 smithi097 ceph-mon[32231]: pgmap v8933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:38 smithi115 ceph-mon[35364]: pgmap v8934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:38 smithi181 ceph-mon[35869]: pgmap v8934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:38 smithi097 ceph-mon[32231]: pgmap v8934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:39.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:39.051+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:21:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:39.053+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:21:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:39.053+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:21:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:39.053+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:21:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:39.053+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15738.040771 secs 2024-07-26T12:21:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:40 smithi115 ceph-mon[35364]: pgmap v8935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15738.040771 secs 2024-07-26T12:21:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:40 smithi181 ceph-mon[35869]: pgmap v8935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15738.040771 secs 2024-07-26T12:21:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:40 smithi097 ceph-mon[32231]: pgmap v8935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:42 smithi115 ceph-mon[35364]: pgmap v8936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:42 smithi181 ceph-mon[35869]: pgmap v8936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:42 smithi097 ceph-mon[32231]: pgmap v8936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:44 smithi115 ceph-mon[35364]: pgmap v8937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:44 smithi181 ceph-mon[35869]: pgmap v8937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:44 smithi097 ceph-mon[32231]: pgmap v8937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15743.040924 secs 2024-07-26T12:21:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15743.040924 secs 2024-07-26T12:21:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15743.040924 secs 2024-07-26T12:21:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:46 smithi115 ceph-mon[35364]: pgmap v8938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:46 smithi181 ceph-mon[35869]: pgmap v8938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:46 smithi097 ceph-mon[32231]: pgmap v8938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:48 smithi115 ceph-mon[35364]: pgmap v8939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:48 smithi181 ceph-mon[35869]: pgmap v8939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:48 smithi097 ceph-mon[32231]: pgmap v8939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:49 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:49 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:49 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:50 smithi115 ceph-mon[35364]: pgmap v8940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:21:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:50 smithi115 ceph-mon[35364]: pgmap v8941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:21:50.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15748.041067 secs 2024-07-26T12:21:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:50 smithi181 ceph-mon[35869]: pgmap v8940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:21:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:50 smithi181 ceph-mon[35869]: pgmap v8941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:21:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15748.041067 secs 2024-07-26T12:21:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:50 smithi097 ceph-mon[32231]: pgmap v8940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:21:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:50 smithi097 ceph-mon[32231]: pgmap v8941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:21:51.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15748.041067 secs 2024-07-26T12:21:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:53 smithi115 ceph-mon[35364]: pgmap v8942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:53 smithi181 ceph-mon[35869]: pgmap v8942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:53 smithi097 ceph-mon[32231]: pgmap v8942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:54.056 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:54.055+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:21:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:54.056+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:21:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:54.056+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:21:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:54.056+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:21:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:21:54.056+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:55 smithi115 ceph-mon[35364]: pgmap v8943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:55 smithi181 ceph-mon[35869]: pgmap v8943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:55 smithi097 ceph-mon[32231]: pgmap v8943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15753.041206 secs 2024-07-26T12:21:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15753.041206 secs 2024-07-26T12:21:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15753.041206 secs 2024-07-26T12:21:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:57 smithi115 ceph-mon[35364]: pgmap v8944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:57 smithi181 ceph-mon[35869]: pgmap v8944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:57 smithi097 ceph-mon[32231]: pgmap v8944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:21:59 smithi115 ceph-mon[35364]: pgmap v8945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:21:59 smithi181 ceph-mon[35869]: pgmap v8945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:21:59 smithi097 ceph-mon[32231]: pgmap v8945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:01 smithi115 ceph-mon[35364]: pgmap v8946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15758.041341 secs 2024-07-26T12:22:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:01 smithi181 ceph-mon[35869]: pgmap v8946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15758.041341 secs 2024-07-26T12:22:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:01 smithi097 ceph-mon[32231]: pgmap v8946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15758.041341 secs 2024-07-26T12:22:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:03 smithi115 ceph-mon[35364]: pgmap v8947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:03 smithi181 ceph-mon[35869]: pgmap v8947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:03 smithi097 ceph-mon[32231]: pgmap v8947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:05 smithi115 ceph-mon[35364]: pgmap v8948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:05 smithi181 ceph-mon[35869]: pgmap v8948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:05 smithi097 ceph-mon[32231]: pgmap v8948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15763.041507 secs 2024-07-26T12:22:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15763.041507 secs 2024-07-26T12:22:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15763.041507 secs 2024-07-26T12:22:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:07 smithi115 ceph-mon[35364]: pgmap v8949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:07 smithi181 ceph-mon[35869]: pgmap v8949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:07 smithi097 ceph-mon[32231]: pgmap v8949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.063+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:22:09.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.064+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:22:09.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.064+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:22:09.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.064+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:22:09.066 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.065+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:09 smithi115 ceph-mon[35364]: pgmap v8950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:09 smithi181 ceph-mon[35869]: pgmap v8950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:09 smithi097 ceph-mon[32231]: pgmap v8950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.701+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:22:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.701+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:22:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.702+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:22:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.702+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:22:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.702+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:22:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.702+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:22:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.702+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:22:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:09.703+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:22:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:11 smithi115 ceph-mon[35364]: pgmap v8951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15768.041656 secs 2024-07-26T12:22:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:11 smithi181 ceph-mon[35869]: pgmap v8951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15768.041656 secs 2024-07-26T12:22:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:11 smithi097 ceph-mon[32231]: pgmap v8951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15768.041656 secs 2024-07-26T12:22:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:13 smithi115 ceph-mon[35364]: pgmap v8952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:13 smithi181 ceph-mon[35869]: pgmap v8952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:13 smithi097 ceph-mon[32231]: pgmap v8952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:15 smithi115 ceph-mon[35364]: pgmap v8953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:15 smithi181 ceph-mon[35869]: pgmap v8953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:15 smithi097 ceph-mon[32231]: pgmap v8953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15773.041794 secs 2024-07-26T12:22:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15773.041794 secs 2024-07-26T12:22:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15773.041794 secs 2024-07-26T12:22:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:17 smithi115 ceph-mon[35364]: pgmap v8954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:17 smithi181 ceph-mon[35869]: pgmap v8954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:17 smithi097 ceph-mon[32231]: pgmap v8954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:19.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:19 smithi115 ceph-mon[35364]: pgmap v8955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:19 smithi181 ceph-mon[35869]: pgmap v8955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:19 smithi097 ceph-mon[32231]: pgmap v8955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:21 smithi115 ceph-mon[35364]: pgmap v8956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:21.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15778.041958 secs 2024-07-26T12:22:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:21 smithi181 ceph-mon[35869]: pgmap v8956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15778.041958 secs 2024-07-26T12:22:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:21 smithi097 ceph-mon[32231]: pgmap v8956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15778.041958 secs 2024-07-26T12:22:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:23 smithi115 ceph-mon[35364]: pgmap v8957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:23 smithi181 ceph-mon[35869]: pgmap v8957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:23 smithi097 ceph-mon[32231]: pgmap v8957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:24.068 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:24.067+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:22:24.069 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:24.068+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:22:24.070 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:24.068+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:22:24.070 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:24.068+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:22:24.070 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:24.068+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:25 smithi115 ceph-mon[35364]: pgmap v8958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:25 smithi181 ceph-mon[35869]: pgmap v8958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:25 smithi097 ceph-mon[32231]: pgmap v8958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15783.042096 secs 2024-07-26T12:22:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15783.042096 secs 2024-07-26T12:22:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15783.042096 secs 2024-07-26T12:22:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:27 smithi115 ceph-mon[35364]: pgmap v8959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:27 smithi181 ceph-mon[35869]: pgmap v8959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:27 smithi097 ceph-mon[32231]: pgmap v8959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:29 smithi115 ceph-mon[35364]: pgmap v8960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:29.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:29 smithi181 ceph-mon[35869]: pgmap v8960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:29 smithi097 ceph-mon[32231]: pgmap v8960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:31 smithi115 ceph-mon[35364]: pgmap v8961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15788.042294 secs 2024-07-26T12:22:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:31 smithi181 ceph-mon[35869]: pgmap v8961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15788.042294 secs 2024-07-26T12:22:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:31 smithi097 ceph-mon[32231]: pgmap v8961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15788.042294 secs 2024-07-26T12:22:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:33 smithi115 ceph-mon[35364]: pgmap v8962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:33 smithi181 ceph-mon[35869]: pgmap v8962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:33 smithi097 ceph-mon[32231]: pgmap v8962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:35 smithi115 ceph-mon[35364]: pgmap v8963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:35 smithi181 ceph-mon[35869]: pgmap v8963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:35 smithi097 ceph-mon[32231]: pgmap v8963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15793.042429 secs 2024-07-26T12:22:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15793.042429 secs 2024-07-26T12:22:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15793.042429 secs 2024-07-26T12:22:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:37 smithi115 ceph-mon[35364]: pgmap v8964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:37 smithi181 ceph-mon[35869]: pgmap v8964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:37 smithi097 ceph-mon[32231]: pgmap v8964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:39.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:39.074+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:22:39.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:39.075+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:22:39.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:39.075+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:22:39.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:39.075+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:22:39.077 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:39.075+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:39.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:39 smithi115 ceph-mon[35364]: pgmap v8965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:39 smithi181 ceph-mon[35869]: pgmap v8965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:39 smithi097 ceph-mon[32231]: pgmap v8965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:41 smithi115 ceph-mon[35364]: pgmap v8966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15798.042563 secs 2024-07-26T12:22:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:41 smithi181 ceph-mon[35869]: pgmap v8966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15798.042563 secs 2024-07-26T12:22:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:41 smithi097 ceph-mon[32231]: pgmap v8966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15798.042563 secs 2024-07-26T12:22:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:43 smithi115 ceph-mon[35364]: pgmap v8967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:43 smithi181 ceph-mon[35869]: pgmap v8967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:43 smithi097 ceph-mon[32231]: pgmap v8967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:45 smithi181 ceph-mon[35869]: pgmap v8968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:45 smithi097 ceph-mon[32231]: pgmap v8968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:45 smithi115 ceph-mon[35364]: pgmap v8968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15803.042704 secs 2024-07-26T12:22:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15803.042704 secs 2024-07-26T12:22:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15803.042704 secs 2024-07-26T12:22:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:47 smithi097 ceph-mon[32231]: pgmap v8969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:47 smithi115 ceph-mon[35364]: pgmap v8969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:47 smithi181 ceph-mon[35869]: pgmap v8969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:22:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:49 smithi115 ceph-mon[35364]: pgmap v8970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:49 smithi181 ceph-mon[35869]: pgmap v8970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:49 smithi097 ceph-mon[32231]: pgmap v8970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:51 smithi115 ceph-mon[35364]: pgmap v8971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15808.042849 secs 2024-07-26T12:22:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:22:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:51 smithi181 ceph-mon[35869]: pgmap v8971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15808.042849 secs 2024-07-26T12:22:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:22:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:51 smithi097 ceph-mon[32231]: pgmap v8971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15808.042849 secs 2024-07-26T12:22:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:52.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:22:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:52 smithi115 ceph-mon[35364]: pgmap v8972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:22:52.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:52 smithi115 ceph-mon[35364]: pgmap v8973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:22:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:52 smithi181 ceph-mon[35869]: pgmap v8972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:22:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:52 smithi181 ceph-mon[35869]: pgmap v8973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:22:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:52 smithi097 ceph-mon[32231]: pgmap v8972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:22:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:52 smithi097 ceph-mon[32231]: pgmap v8973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:22:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:54.077+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:22:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:54.079+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:22:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:54.079+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:22:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:54.079+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:22:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:22:54.079+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:54 smithi115 ceph-mon[35364]: pgmap v8974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:22:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:54 smithi181 ceph-mon[35869]: pgmap v8974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:22:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:54 smithi097 ceph-mon[32231]: pgmap v8974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:22:56.806 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15813.043007 secs 2024-07-26T12:22:56.806 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:56 smithi097 ceph-mon[32231]: pgmap v8975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:22:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15813.043007 secs 2024-07-26T12:22:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:56 smithi115 ceph-mon[35364]: pgmap v8975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:22:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15813.043007 secs 2024-07-26T12:22:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:56 smithi181 ceph-mon[35869]: pgmap v8975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:22:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:22:58 smithi115 ceph-mon[35364]: pgmap v8976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:22:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:22:58 smithi181 ceph-mon[35869]: pgmap v8976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:22:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:22:58 smithi097 ceph-mon[32231]: pgmap v8976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:00 smithi115 ceph-mon[35364]: pgmap v8977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:00 smithi181 ceph-mon[35869]: pgmap v8977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:00 smithi097 ceph-mon[32231]: pgmap v8977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15818.043144 secs 2024-07-26T12:23:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15818.043144 secs 2024-07-26T12:23:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15818.043144 secs 2024-07-26T12:23:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:02 smithi115 ceph-mon[35364]: pgmap v8978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:02 smithi181 ceph-mon[35869]: pgmap v8978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:02 smithi097 ceph-mon[32231]: pgmap v8978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:04 smithi115 ceph-mon[35364]: pgmap v8979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:04 smithi181 ceph-mon[35869]: pgmap v8979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:04 smithi097 ceph-mon[32231]: pgmap v8979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15823.043287 secs 2024-07-26T12:23:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:06 smithi115 ceph-mon[35364]: pgmap v8980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15823.043287 secs 2024-07-26T12:23:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:06 smithi181 ceph-mon[35869]: pgmap v8980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15823.043287 secs 2024-07-26T12:23:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:06 smithi097 ceph-mon[32231]: pgmap v8980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:07 smithi115 ceph-mon[35364]: pgmap v8981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:07 smithi181 ceph-mon[35869]: pgmap v8981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:07 smithi097 ceph-mon[32231]: pgmap v8981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:09.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:09.087+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:23:09.089 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:09.088+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:23:09.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:09.088+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:23:09.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:09.088+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:23:09.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:09.088+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:10 smithi097 ceph-mon[32231]: pgmap v8982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:10 smithi115 ceph-mon[35364]: pgmap v8982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:10 smithi181 ceph-mon[35869]: pgmap v8982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15828.043424 secs 2024-07-26T12:23:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15828.043424 secs 2024-07-26T12:23:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15828.043424 secs 2024-07-26T12:23:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:12 smithi115 ceph-mon[35364]: pgmap v8983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:12 smithi181 ceph-mon[35869]: pgmap v8983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:12 smithi097 ceph-mon[32231]: pgmap v8983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:14 smithi115 ceph-mon[35364]: pgmap v8984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:14 smithi181 ceph-mon[35869]: pgmap v8984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:14 smithi097 ceph-mon[32231]: pgmap v8984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15833.043565 secs 2024-07-26T12:23:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:16 smithi115 ceph-mon[35364]: pgmap v8985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15833.043565 secs 2024-07-26T12:23:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:16 smithi181 ceph-mon[35869]: pgmap v8985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15833.043565 secs 2024-07-26T12:23:17.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:16 smithi097 ceph-mon[32231]: pgmap v8985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:18 smithi115 ceph-mon[35364]: pgmap v8986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:18 smithi181 ceph-mon[35869]: pgmap v8986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:18 smithi097 ceph-mon[32231]: pgmap v8986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:20 smithi115 ceph-mon[35364]: pgmap v8987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:20 smithi181 ceph-mon[35869]: pgmap v8987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:20 smithi097 ceph-mon[32231]: pgmap v8987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15838.043727 secs 2024-07-26T12:23:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15838.043727 secs 2024-07-26T12:23:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15838.043727 secs 2024-07-26T12:23:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:22 smithi115 ceph-mon[35364]: pgmap v8988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:22 smithi181 ceph-mon[35869]: pgmap v8988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:22 smithi097 ceph-mon[32231]: pgmap v8988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:24.090+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:23:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:24.091+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:23:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:24.091+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:23:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:24.091+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:23:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:24.092+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:24 smithi115 ceph-mon[35364]: pgmap v8989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:24 smithi181 ceph-mon[35869]: pgmap v8989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:24 smithi097 ceph-mon[32231]: pgmap v8989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15843.043871 secs 2024-07-26T12:23:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:26 smithi115 ceph-mon[35364]: pgmap v8990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15843.043871 secs 2024-07-26T12:23:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:26 smithi181 ceph-mon[35869]: pgmap v8990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15843.043871 secs 2024-07-26T12:23:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:26 smithi097 ceph-mon[32231]: pgmap v8990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:28 smithi115 ceph-mon[35364]: pgmap v8991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:28 smithi181 ceph-mon[35869]: pgmap v8991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:28 smithi097 ceph-mon[32231]: pgmap v8991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:30 smithi115 ceph-mon[35364]: pgmap v8992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:30 smithi181 ceph-mon[35869]: pgmap v8992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:30 smithi097 ceph-mon[32231]: pgmap v8992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15848.044012 secs 2024-07-26T12:23:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15848.044012 secs 2024-07-26T12:23:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15848.044012 secs 2024-07-26T12:23:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:32 smithi115 ceph-mon[35364]: pgmap v8993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:32 smithi181 ceph-mon[35869]: pgmap v8993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:32 smithi097 ceph-mon[32231]: pgmap v8993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:33 smithi115 ceph-mon[35364]: pgmap v8994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:33 smithi181 ceph-mon[35869]: pgmap v8994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:33 smithi097 ceph-mon[32231]: pgmap v8994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15853.044152 secs 2024-07-26T12:23:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:36 smithi115 ceph-mon[35364]: pgmap v8995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15853.044152 secs 2024-07-26T12:23:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:36 smithi181 ceph-mon[35869]: pgmap v8995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15853.044152 secs 2024-07-26T12:23:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:36 smithi097 ceph-mon[32231]: pgmap v8995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:38 smithi115 ceph-mon[35364]: pgmap v8996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:38 smithi181 ceph-mon[35869]: pgmap v8996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:38 smithi097 ceph-mon[32231]: pgmap v8996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:39.098+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:23:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:39.099+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:23:39.101 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:39.099+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:23:39.101 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:39.099+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:23:39.101 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:39.099+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:40 smithi115 ceph-mon[35364]: pgmap v8997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:40 smithi181 ceph-mon[35869]: pgmap v8997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:40 smithi097 ceph-mon[32231]: pgmap v8997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15858.044292 secs 2024-07-26T12:23:41.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15858.044292 secs 2024-07-26T12:23:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15858.044292 secs 2024-07-26T12:23:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:42 smithi115 ceph-mon[35364]: pgmap v8998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:42 smithi181 ceph-mon[35869]: pgmap v8998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:42 smithi097 ceph-mon[32231]: pgmap v8998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:44 smithi115 ceph-mon[35364]: pgmap v8999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:44 smithi181 ceph-mon[35869]: pgmap v8999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:44 smithi097 ceph-mon[32231]: pgmap v8999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15863.044428 secs 2024-07-26T12:23:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:46 smithi115 ceph-mon[35364]: pgmap v9000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15863.044428 secs 2024-07-26T12:23:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:46 smithi181 ceph-mon[35869]: pgmap v9000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15863.044428 secs 2024-07-26T12:23:47.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:46 smithi097 ceph-mon[32231]: pgmap v9000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:48 smithi115 ceph-mon[35364]: pgmap v9001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:48 smithi181 ceph-mon[35869]: pgmap v9001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:48 smithi097 ceph-mon[32231]: pgmap v9001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:50 smithi115 ceph-mon[35364]: pgmap v9002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:50 smithi181 ceph-mon[35869]: pgmap v9002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:50 smithi097 ceph-mon[32231]: pgmap v9002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:23:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15868.044562 secs 2024-07-26T12:23:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15868.044562 secs 2024-07-26T12:23:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15868.044562 secs 2024-07-26T12:23:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:52 smithi115 ceph-mon[35364]: pgmap v9003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:52 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:52 smithi181 ceph-mon[35869]: pgmap v9003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:52 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:52 smithi097 ceph-mon[32231]: pgmap v9003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:23:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:52 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:53.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:23:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:23:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:54.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:23:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:54.102+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:23:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:54.103+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:23:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:54.103+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:23:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:54.103+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:23:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:23:54.103+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:54 smithi115 ceph-mon[35364]: pgmap v9004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:23:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:54 smithi115 ceph-mon[35364]: pgmap v9005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:54 smithi181 ceph-mon[35869]: pgmap v9004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:23:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:54 smithi181 ceph-mon[35869]: pgmap v9005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:54 smithi097 ceph-mon[32231]: pgmap v9004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:23:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:54 smithi097 ceph-mon[32231]: pgmap v9005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:56 smithi115 ceph-mon[35364]: pgmap v9006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15873.044703 secs 2024-07-26T12:23:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:56 smithi181 ceph-mon[35869]: pgmap v9006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15873.044703 secs 2024-07-26T12:23:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:56 smithi097 ceph-mon[32231]: pgmap v9006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15873.044703 secs 2024-07-26T12:23:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:57 smithi115 ceph-mon[35364]: pgmap v9007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:57 smithi181 ceph-mon[35869]: pgmap v9007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:57 smithi097 ceph-mon[32231]: pgmap v9007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:23:59 smithi115 ceph-mon[35364]: pgmap v9008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:23:59 smithi181 ceph-mon[35869]: pgmap v9008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:23:59 smithi097 ceph-mon[32231]: pgmap v9008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15878.044864 secs 2024-07-26T12:24:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15878.044864 secs 2024-07-26T12:24:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15878.044864 secs 2024-07-26T12:24:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:01 smithi115 ceph-mon[35364]: pgmap v9009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:24:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:01 smithi181 ceph-mon[35869]: pgmap v9009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:24:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:01 smithi097 ceph-mon[32231]: pgmap v9009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:24:04.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:03 smithi115 ceph-mon[35364]: pgmap v9010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:24:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:03 smithi181 ceph-mon[35869]: pgmap v9010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:24:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:03 smithi097 ceph-mon[32231]: pgmap v9010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:24:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:05 smithi115 ceph-mon[35364]: pgmap v9011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:06.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15883.045009 secs 2024-07-26T12:24:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:05 smithi181 ceph-mon[35869]: pgmap v9011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15883.045009 secs 2024-07-26T12:24:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:05 smithi097 ceph-mon[32231]: pgmap v9011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15883.045009 secs 2024-07-26T12:24:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:07 smithi115 ceph-mon[35364]: pgmap v9012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:07 smithi181 ceph-mon[35869]: pgmap v9012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:07 smithi097 ceph-mon[32231]: pgmap v9012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:09.111 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:09.109+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:24:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:09.111+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:24:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:09.111+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:24:09.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:09.111+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:24:09.113 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:09.111+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:09 smithi181 ceph-mon[35869]: pgmap v9013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:24:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:09 smithi097 ceph-mon[32231]: pgmap v9013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:24:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:09 smithi115 ceph-mon[35364]: pgmap v9013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:24:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15888.045150 secs 2024-07-26T12:24:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15888.045150 secs 2024-07-26T12:24:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15888.045150 secs 2024-07-26T12:24:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:11 smithi097 ceph-mon[32231]: pgmap v9014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:11 smithi115 ceph-mon[35364]: pgmap v9014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:11 smithi181 ceph-mon[35869]: pgmap v9014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:14 smithi115 ceph-mon[35364]: pgmap v9015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:14 smithi181 ceph-mon[35869]: pgmap v9015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:14 smithi097 ceph-mon[32231]: pgmap v9015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:16 smithi115 ceph-mon[35364]: pgmap v9016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15893.045291 secs 2024-07-26T12:24:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:16 smithi181 ceph-mon[35869]: pgmap v9016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15893.045291 secs 2024-07-26T12:24:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:16 smithi097 ceph-mon[32231]: pgmap v9016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15893.045291 secs 2024-07-26T12:24:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:18 smithi115 ceph-mon[35364]: pgmap v9017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:18 smithi181 ceph-mon[35869]: pgmap v9017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:18 smithi097 ceph-mon[32231]: pgmap v9017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:20.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:20 smithi115 ceph-mon[35364]: pgmap v9018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:20 smithi181 ceph-mon[35869]: pgmap v9018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:20 smithi097 ceph-mon[32231]: pgmap v9018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15898.045430 secs 2024-07-26T12:24:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15898.045430 secs 2024-07-26T12:24:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15898.045430 secs 2024-07-26T12:24:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:22 smithi115 ceph-mon[35364]: pgmap v9019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:22 smithi181 ceph-mon[35869]: pgmap v9019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:22 smithi097 ceph-mon[32231]: pgmap v9019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:24.113+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:24:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:24.114+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:24:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:24.114+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:24:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:24.114+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:24:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:24.114+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:24 smithi115 ceph-mon[35364]: pgmap v9020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:24 smithi181 ceph-mon[35869]: pgmap v9020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:24 smithi097 ceph-mon[32231]: pgmap v9020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:26 smithi115 ceph-mon[35364]: pgmap v9021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15903.045560 secs 2024-07-26T12:24:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:26 smithi181 ceph-mon[35869]: pgmap v9021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15903.045560 secs 2024-07-26T12:24:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:26 smithi097 ceph-mon[32231]: pgmap v9021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:26.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15903.045560 secs 2024-07-26T12:24:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:28 smithi115 ceph-mon[35364]: pgmap v9022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:28 smithi181 ceph-mon[35869]: pgmap v9022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:28 smithi097 ceph-mon[32231]: pgmap v9022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:30 smithi115 ceph-mon[35364]: pgmap v9023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:30 smithi181 ceph-mon[35869]: pgmap v9023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:30 smithi097 ceph-mon[32231]: pgmap v9023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15908.045700 secs 2024-07-26T12:24:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15908.045700 secs 2024-07-26T12:24:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15908.045700 secs 2024-07-26T12:24:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:32 smithi181 ceph-mon[35869]: pgmap v9024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:32 smithi097 ceph-mon[32231]: pgmap v9024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:32 smithi115 ceph-mon[35364]: pgmap v9024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:34 smithi097 ceph-mon[32231]: pgmap v9025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:34 smithi115 ceph-mon[35364]: pgmap v9025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:34 smithi181 ceph-mon[35869]: pgmap v9025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:36 smithi097 ceph-mon[32231]: pgmap v9026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:36.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15913.045835 secs 2024-07-26T12:24:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:36 smithi115 ceph-mon[35364]: pgmap v9026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15913.045835 secs 2024-07-26T12:24:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:36 smithi181 ceph-mon[35869]: pgmap v9026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15913.045835 secs 2024-07-26T12:24:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:38 smithi097 ceph-mon[32231]: pgmap v9027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:38 smithi115 ceph-mon[35364]: pgmap v9027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:38 smithi181 ceph-mon[35869]: pgmap v9027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:39.121 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:39.119+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:24:39.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:39.121+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:24:39.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:39.121+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:24:39.123 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:39.121+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:24:39.123 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:39.121+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:40 smithi115 ceph-mon[35364]: pgmap v9028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:40 smithi181 ceph-mon[35869]: pgmap v9028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:40 smithi097 ceph-mon[32231]: pgmap v9028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15918.045978 secs 2024-07-26T12:24:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15918.045978 secs 2024-07-26T12:24:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15918.045978 secs 2024-07-26T12:24:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:42 smithi115 ceph-mon[35364]: pgmap v9029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:42 smithi181 ceph-mon[35869]: pgmap v9029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:42 smithi097 ceph-mon[32231]: pgmap v9029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:44 smithi115 ceph-mon[35364]: pgmap v9030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:44 smithi181 ceph-mon[35869]: pgmap v9030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:44 smithi097 ceph-mon[32231]: pgmap v9030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:46 smithi115 ceph-mon[35364]: pgmap v9031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15923.046131 secs 2024-07-26T12:24:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:46 smithi181 ceph-mon[35869]: pgmap v9031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15923.046131 secs 2024-07-26T12:24:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:46 smithi097 ceph-mon[32231]: pgmap v9031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15923.046131 secs 2024-07-26T12:24:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:48 smithi115 ceph-mon[35364]: pgmap v9032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:48 smithi181 ceph-mon[35869]: pgmap v9032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:48 smithi097 ceph-mon[32231]: pgmap v9032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:50 smithi115 ceph-mon[35364]: pgmap v9033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:50 smithi181 ceph-mon[35869]: pgmap v9033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:50 smithi097 ceph-mon[32231]: pgmap v9033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:24:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15928.046296 secs 2024-07-26T12:24:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15928.046296 secs 2024-07-26T12:24:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15928.046296 secs 2024-07-26T12:24:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:52 smithi115 ceph-mon[35364]: pgmap v9034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:52 smithi181 ceph-mon[35869]: pgmap v9034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:52 smithi097 ceph-mon[32231]: pgmap v9034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:53 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:53 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:53 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:54.124+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:24:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:54.125+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:24:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:54.125+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:24:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:54.125+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:24:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:24:54.125+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:54 smithi115 ceph-mon[35364]: pgmap v9035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:54.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:54 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:24:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:54 smithi181 ceph-mon[35869]: pgmap v9035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:54 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:24:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:54 smithi097 ceph-mon[32231]: pgmap v9035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:55.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:54 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:24:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:55 smithi115 ceph-mon[35364]: pgmap v9036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:24:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:55 smithi115 ceph-mon[35364]: pgmap v9037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:55 smithi181 ceph-mon[35869]: pgmap v9036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:24:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:55 smithi181 ceph-mon[35869]: pgmap v9037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:55 smithi097 ceph-mon[32231]: pgmap v9036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:24:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:55 smithi097 ceph-mon[32231]: pgmap v9037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:24:56.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15933.046445 secs 2024-07-26T12:24:56.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15933.046445 secs 2024-07-26T12:24:56.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15933.046445 secs 2024-07-26T12:24:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:57 smithi115 ceph-mon[35364]: pgmap v9038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:57 smithi181 ceph-mon[35869]: pgmap v9038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:57 smithi097 ceph-mon[32231]: pgmap v9038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:24:59 smithi115 ceph-mon[35364]: pgmap v9039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:24:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:24:59 smithi181 ceph-mon[35869]: pgmap v9039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:25:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:24:59 smithi097 ceph-mon[32231]: pgmap v9039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:25:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15938.046588 secs 2024-07-26T12:25:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:00 smithi115 ceph-mon[35364]: pgmap v9040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:25:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15938.046588 secs 2024-07-26T12:25:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:00 smithi181 ceph-mon[35869]: pgmap v9040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:25:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15938.046588 secs 2024-07-26T12:25:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:00 smithi097 ceph-mon[32231]: pgmap v9040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:25:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:03 smithi115 ceph-mon[35364]: pgmap v9041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:25:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:03 smithi181 ceph-mon[35869]: pgmap v9041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:25:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:03 smithi097 ceph-mon[32231]: pgmap v9041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:25:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:05 smithi115 ceph-mon[35364]: pgmap v9042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:05.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:05 smithi181 ceph-mon[35869]: pgmap v9042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:05 smithi097 ceph-mon[32231]: pgmap v9042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15943.046680 secs 2024-07-26T12:25:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15943.046680 secs 2024-07-26T12:25:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15943.046680 secs 2024-07-26T12:25:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:07 smithi115 ceph-mon[35364]: pgmap v9043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:07 smithi181 ceph-mon[35869]: pgmap v9043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:07 smithi097 ceph-mon[32231]: pgmap v9043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:09.133 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.132+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.133+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:25:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.133+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:25:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.133+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.133+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:09 smithi115 ceph-mon[35364]: pgmap v9044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:09 smithi181 ceph-mon[35869]: pgmap v9044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.735+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:25:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.735+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0006250 con 0x7ff3f01061a0 2024-07-26T12:25:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.736+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:25:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:09.736+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:25:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:09 smithi097 ceph-mon[32231]: pgmap v9044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15948.046824 secs 2024-07-26T12:25:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:11 smithi115 ceph-mon[35364]: pgmap v9045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15948.046824 secs 2024-07-26T12:25:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:11 smithi181 ceph-mon[35869]: pgmap v9045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15948.046824 secs 2024-07-26T12:25:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:11 smithi097 ceph-mon[32231]: pgmap v9045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:13 smithi115 ceph-mon[35364]: pgmap v9046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:13 smithi181 ceph-mon[35869]: pgmap v9046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:13 smithi097 ceph-mon[32231]: pgmap v9046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:15.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:15 smithi115 ceph-mon[35364]: pgmap v9047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:25:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:15 smithi181 ceph-mon[35869]: pgmap v9047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:25:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:15 smithi097 ceph-mon[32231]: pgmap v9047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:25:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15953.046967 secs 2024-07-26T12:25:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15953.046967 secs 2024-07-26T12:25:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15953.046967 secs 2024-07-26T12:25:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:17 smithi115 ceph-mon[35364]: pgmap v9048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:17 smithi181 ceph-mon[35869]: pgmap v9048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:17 smithi097 ceph-mon[32231]: pgmap v9048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:19 smithi115 ceph-mon[35364]: pgmap v9049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:25:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:19 smithi181 ceph-mon[35869]: pgmap v9049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:25:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:19 smithi097 ceph-mon[32231]: pgmap v9049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:25:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15958.047105 secs 2024-07-26T12:25:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:21 smithi115 ceph-mon[35364]: pgmap v9050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15958.047105 secs 2024-07-26T12:25:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:21 smithi181 ceph-mon[35869]: pgmap v9050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15958.047105 secs 2024-07-26T12:25:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:21 smithi097 ceph-mon[32231]: pgmap v9050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:23 smithi115 ceph-mon[35364]: pgmap v9051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:23 smithi181 ceph-mon[35869]: pgmap v9051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:23 smithi097 ceph-mon[32231]: pgmap v9051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:24.138 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:24.136+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:24.139 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:24.138+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:25:24.139 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:24.138+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:25:24.139 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:24.138+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:24.139 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:24.138+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:25 smithi115 ceph-mon[35364]: pgmap v9052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:25 smithi181 ceph-mon[35869]: pgmap v9052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:25 smithi097 ceph-mon[32231]: pgmap v9052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15963.047268 secs 2024-07-26T12:25:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15963.047268 secs 2024-07-26T12:25:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15963.047268 secs 2024-07-26T12:25:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:27 smithi115 ceph-mon[35364]: pgmap v9053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:27 smithi181 ceph-mon[35869]: pgmap v9053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:27 smithi097 ceph-mon[32231]: pgmap v9053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:29 smithi115 ceph-mon[35364]: pgmap v9054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:29 smithi181 ceph-mon[35869]: pgmap v9054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:29 smithi097 ceph-mon[32231]: pgmap v9054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15968.047410 secs 2024-07-26T12:25:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:31 smithi115 ceph-mon[35364]: pgmap v9055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15968.047410 secs 2024-07-26T12:25:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:31 smithi181 ceph-mon[35869]: pgmap v9055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15968.047410 secs 2024-07-26T12:25:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:31 smithi097 ceph-mon[32231]: pgmap v9055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:33 smithi181 ceph-mon[35869]: pgmap v9056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:33 smithi097 ceph-mon[32231]: pgmap v9056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:33 smithi115 ceph-mon[35364]: pgmap v9056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:35 smithi097 ceph-mon[32231]: pgmap v9057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:35 smithi115 ceph-mon[35364]: pgmap v9057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:35 smithi181 ceph-mon[35869]: pgmap v9057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15973.047551 secs 2024-07-26T12:25:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15973.047551 secs 2024-07-26T12:25:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15973.047551 secs 2024-07-26T12:25:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:37 smithi097 ceph-mon[32231]: pgmap v9058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:37 smithi115 ceph-mon[35364]: pgmap v9058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:37 smithi181 ceph-mon[35869]: pgmap v9058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:39.147 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:39.146+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:39.149 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:39.147+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:25:39.149 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:39.147+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:25:39.149 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:39.147+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:39.149 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:39.147+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:39 smithi115 ceph-mon[35364]: pgmap v9059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:39 smithi181 ceph-mon[35869]: pgmap v9059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:39 smithi097 ceph-mon[32231]: pgmap v9059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15978.047699 secs 2024-07-26T12:25:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:41 smithi115 ceph-mon[35364]: pgmap v9060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15978.047699 secs 2024-07-26T12:25:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:41 smithi181 ceph-mon[35869]: pgmap v9060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15978.047699 secs 2024-07-26T12:25:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:41 smithi097 ceph-mon[32231]: pgmap v9060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:43 smithi115 ceph-mon[35364]: pgmap v9061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:43 smithi181 ceph-mon[35869]: pgmap v9061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:43 smithi097 ceph-mon[32231]: pgmap v9061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:45 smithi115 ceph-mon[35364]: pgmap v9062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:45 smithi181 ceph-mon[35869]: pgmap v9062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:45 smithi097 ceph-mon[32231]: pgmap v9062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15983.047841 secs 2024-07-26T12:25:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15983.047841 secs 2024-07-26T12:25:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15983.047841 secs 2024-07-26T12:25:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:47 smithi115 ceph-mon[35364]: pgmap v9063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:47 smithi181 ceph-mon[35869]: pgmap v9063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:47 smithi097 ceph-mon[32231]: pgmap v9063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:25:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:49 smithi115 ceph-mon[35364]: pgmap v9064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:49 smithi181 ceph-mon[35869]: pgmap v9064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:49 smithi097 ceph-mon[32231]: pgmap v9064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15988.047997 secs 2024-07-26T12:25:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:51 smithi115 ceph-mon[35364]: pgmap v9065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15988.047997 secs 2024-07-26T12:25:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:51 smithi181 ceph-mon[35869]: pgmap v9065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15988.047997 secs 2024-07-26T12:25:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:51 smithi097 ceph-mon[32231]: pgmap v9065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:53 smithi115 ceph-mon[35364]: pgmap v9066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:53.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:53 smithi181 ceph-mon[35869]: pgmap v9066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:53 smithi097 ceph-mon[32231]: pgmap v9066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:54.149+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:54.150+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:25:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:54.150+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:25:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:54.150+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:25:54.150+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:55 smithi115 ceph-mon[35364]: pgmap v9067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:55 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:55 smithi181 ceph-mon[35869]: pgmap v9067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:55 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:55 smithi097 ceph-mon[32231]: pgmap v9067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:25:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:55 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:56.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15993.048082 secs 2024-07-26T12:25:56.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.877 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:56.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:56.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:56 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:25:56.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15993.048082 secs 2024-07-26T12:25:56.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:56.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:56 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:25:56.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15993.048082 secs 2024-07-26T12:25:56.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.904 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:25:56.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:56.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:56.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:56 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:25:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:57 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:25:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:57 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:25:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:57 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:25:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:57 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:25:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:57 smithi115 ceph-mon[35364]: pgmap v9068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:25:57.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:57 smithi115 ceph-mon[35364]: pgmap v9069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:57 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:25:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:57 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:25:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:57 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:25:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:57 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:25:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:57 smithi181 ceph-mon[35869]: pgmap v9068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:25:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:57 smithi181 ceph-mon[35869]: pgmap v9069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:57 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:25:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:57 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:25:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:57 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:25:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:57 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:25:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:57 smithi097 ceph-mon[32231]: pgmap v9068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:25:58.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:57 smithi097 ceph-mon[32231]: pgmap v9069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:25:59 smithi115 ceph-mon[35364]: pgmap v9070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:25:59 smithi181 ceph-mon[35869]: pgmap v9070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:25:59 smithi097 ceph-mon[32231]: pgmap v9070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:01 smithi115 ceph-mon[35364]: pgmap v9071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 15998.048169 secs 2024-07-26T12:26:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:01 smithi181 ceph-mon[35869]: pgmap v9071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 15998.048169 secs 2024-07-26T12:26:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:01 smithi097 ceph-mon[32231]: pgmap v9071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 15998.048169 secs 2024-07-26T12:26:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:03 smithi115 ceph-mon[35364]: pgmap v9072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:03 smithi181 ceph-mon[35869]: pgmap v9072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:03 smithi097 ceph-mon[32231]: pgmap v9072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:05 smithi115 ceph-mon[35364]: pgmap v9073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:26:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:05 smithi181 ceph-mon[35869]: pgmap v9073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:26:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:05 smithi097 ceph-mon[32231]: pgmap v9073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:26:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16003.048321 secs 2024-07-26T12:26:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16003.048321 secs 2024-07-26T12:26:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16003.048321 secs 2024-07-26T12:26:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:07 smithi115 ceph-mon[35364]: pgmap v9074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:07 smithi181 ceph-mon[35869]: pgmap v9074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:07 smithi097 ceph-mon[32231]: pgmap v9074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:08 smithi115 ceph-mon[35364]: pgmap v9075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:08 smithi181 ceph-mon[35869]: pgmap v9075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:08 smithi097 ceph-mon[32231]: pgmap v9075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:09.157+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:09.159+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:26:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:09.159+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:26:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:09.159+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:09.159+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:10 smithi115 ceph-mon[35364]: pgmap v9076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16008.048464 secs 2024-07-26T12:26:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:10 smithi181 ceph-mon[35869]: pgmap v9076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16008.048464 secs 2024-07-26T12:26:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:10 smithi097 ceph-mon[32231]: pgmap v9076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16008.048464 secs 2024-07-26T12:26:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:12 smithi115 ceph-mon[35364]: pgmap v9077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:12 smithi181 ceph-mon[35869]: pgmap v9077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:12 smithi097 ceph-mon[32231]: pgmap v9077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:14 smithi115 ceph-mon[35364]: pgmap v9078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:15.352 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:14 smithi097 ceph-mon[32231]: pgmap v9078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:15.352 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:14 smithi181 ceph-mon[35869]: pgmap v9078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16013.048596 secs 2024-07-26T12:26:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16013.048596 secs 2024-07-26T12:26:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16013.048596 secs 2024-07-26T12:26:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:16 smithi115 ceph-mon[35364]: pgmap v9079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:16 smithi181 ceph-mon[35869]: pgmap v9079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:16 smithi097 ceph-mon[32231]: pgmap v9079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:18 smithi115 ceph-mon[35364]: pgmap v9080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:19.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:18 smithi181 ceph-mon[35869]: pgmap v9080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:18 smithi097 ceph-mon[32231]: pgmap v9080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:20 smithi115 ceph-mon[35364]: pgmap v9081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16018.048740 secs 2024-07-26T12:26:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:20 smithi181 ceph-mon[35869]: pgmap v9081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16018.048740 secs 2024-07-26T12:26:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:20 smithi097 ceph-mon[32231]: pgmap v9081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:21.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16018.048740 secs 2024-07-26T12:26:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:22 smithi115 ceph-mon[35364]: pgmap v9082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:22 smithi181 ceph-mon[35869]: pgmap v9082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:22 smithi097 ceph-mon[32231]: pgmap v9082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:24.162+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:24.163+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:26:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:24.163+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:26:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:24.163+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:24.163+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:24 smithi115 ceph-mon[35364]: pgmap v9083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:24 smithi181 ceph-mon[35869]: pgmap v9083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:24 smithi097 ceph-mon[32231]: pgmap v9083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:26.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16023.048874 secs 2024-07-26T12:26:26.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16023.048874 secs 2024-07-26T12:26:26.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16023.048874 secs 2024-07-26T12:26:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:26 smithi115 ceph-mon[35364]: pgmap v9084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:26 smithi181 ceph-mon[35869]: pgmap v9084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:26 smithi097 ceph-mon[32231]: pgmap v9084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:29.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:28 smithi115 ceph-mon[35364]: pgmap v9085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:28 smithi181 ceph-mon[35869]: pgmap v9085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:28 smithi097 ceph-mon[32231]: pgmap v9085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:30 smithi115 ceph-mon[35364]: pgmap v9086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16028.049009 secs 2024-07-26T12:26:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:30 smithi181 ceph-mon[35869]: pgmap v9086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16028.049009 secs 2024-07-26T12:26:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:30 smithi097 ceph-mon[32231]: pgmap v9086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16028.049009 secs 2024-07-26T12:26:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:32 smithi115 ceph-mon[35364]: pgmap v9087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:32 smithi181 ceph-mon[35869]: pgmap v9087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:32 smithi097 ceph-mon[32231]: pgmap v9087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:34 smithi115 ceph-mon[35364]: pgmap v9088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:34 smithi181 ceph-mon[35869]: pgmap v9088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:34 smithi097 ceph-mon[32231]: pgmap v9088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16033.049151 secs 2024-07-26T12:26:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16033.049151 secs 2024-07-26T12:26:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16033.049151 secs 2024-07-26T12:26:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:36 smithi115 ceph-mon[35364]: pgmap v9089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:36 smithi181 ceph-mon[35869]: pgmap v9089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:36 smithi097 ceph-mon[32231]: pgmap v9089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:39.172+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:39.172+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:26:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:39.172+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:26:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:39.172+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:39.172+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:38 smithi115 ceph-mon[35364]: pgmap v9090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:38 smithi181 ceph-mon[35869]: pgmap v9090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:38 smithi097 ceph-mon[32231]: pgmap v9090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:40 smithi115 ceph-mon[35364]: pgmap v9091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:41.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16038.049296 secs 2024-07-26T12:26:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:40 smithi181 ceph-mon[35869]: pgmap v9091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16038.049296 secs 2024-07-26T12:26:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:40 smithi097 ceph-mon[32231]: pgmap v9091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16038.049296 secs 2024-07-26T12:26:43.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:42 smithi115 ceph-mon[35364]: pgmap v9092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:42 smithi181 ceph-mon[35869]: pgmap v9092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:42 smithi097 ceph-mon[32231]: pgmap v9092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:44 smithi115 ceph-mon[35364]: pgmap v9093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:45.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:44 smithi181 ceph-mon[35869]: pgmap v9093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:44 smithi097 ceph-mon[32231]: pgmap v9093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16043.049432 secs 2024-07-26T12:26:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16043.049432 secs 2024-07-26T12:26:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16043.049432 secs 2024-07-26T12:26:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:46 smithi115 ceph-mon[35364]: pgmap v9094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:46 smithi181 ceph-mon[35869]: pgmap v9094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:46 smithi097 ceph-mon[32231]: pgmap v9094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:48 smithi115 ceph-mon[35364]: pgmap v9095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:48 smithi181 ceph-mon[35869]: pgmap v9095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:48 smithi097 ceph-mon[32231]: pgmap v9095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:50 smithi181 ceph-mon[35869]: pgmap v9096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16048.049596 secs 2024-07-26T12:26:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:50 smithi097 ceph-mon[32231]: pgmap v9096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16048.049596 secs 2024-07-26T12:26:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:50 smithi115 ceph-mon[35364]: pgmap v9096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16048.049596 secs 2024-07-26T12:26:53.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:52 smithi181 ceph-mon[35869]: pgmap v9097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:52 smithi097 ceph-mon[32231]: pgmap v9097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:52 smithi115 ceph-mon[35364]: pgmap v9097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:54.173+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:54.175+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:26:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:54.175+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:26:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:54.175+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:26:54.175+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:54 smithi097 ceph-mon[32231]: pgmap v9098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:54 smithi115 ceph-mon[35364]: pgmap v9098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:54 smithi181 ceph-mon[35869]: pgmap v9098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16053.049745 secs 2024-07-26T12:26:56.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16053.049745 secs 2024-07-26T12:26:56.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16053.049745 secs 2024-07-26T12:26:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:57 smithi115 ceph-mon[35364]: pgmap v9099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:57 smithi181 ceph-mon[35869]: pgmap v9099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:57 smithi097 ceph-mon[32231]: pgmap v9099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: pgmap v9100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T12:26:58.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: pgmap v9101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:58.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:26:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:26:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:26:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:26:58.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:58.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:58.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: pgmap v9100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T12:26:58.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: pgmap v9101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:58.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:26:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:26:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: pgmap v9100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: pgmap v9101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:26:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:27:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:00 smithi115 ceph-mon[35364]: pgmap v9102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:00 smithi181 ceph-mon[35869]: pgmap v9102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:00 smithi097 ceph-mon[32231]: pgmap v9102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16058.049880 secs 2024-07-26T12:27:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16058.049880 secs 2024-07-26T12:27:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16058.049880 secs 2024-07-26T12:27:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:02 smithi115 ceph-mon[35364]: pgmap v9103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:02 smithi181 ceph-mon[35869]: pgmap v9103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:02 smithi097 ceph-mon[32231]: pgmap v9103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:04 smithi115 ceph-mon[35364]: pgmap v9104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:04 smithi181 ceph-mon[35869]: pgmap v9104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:04 smithi097 ceph-mon[32231]: pgmap v9104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16063.050027 secs 2024-07-26T12:27:06.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:06 smithi115 ceph-mon[35364]: pgmap v9105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16063.050027 secs 2024-07-26T12:27:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:06 smithi181 ceph-mon[35869]: pgmap v9105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16063.050027 secs 2024-07-26T12:27:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:06 smithi097 ceph-mon[32231]: pgmap v9105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:08 smithi115 ceph-mon[35364]: pgmap v9106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:08 smithi181 ceph-mon[35869]: pgmap v9106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:08 smithi097 ceph-mon[32231]: pgmap v9106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:09.183+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:09.184+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:27:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:09.184+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:27:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:09.184+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:09.184+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:10 smithi115 ceph-mon[35364]: pgmap v9107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:10 smithi181 ceph-mon[35869]: pgmap v9107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:10 smithi097 ceph-mon[32231]: pgmap v9107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16068.050182 secs 2024-07-26T12:27:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16068.050182 secs 2024-07-26T12:27:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16068.050182 secs 2024-07-26T12:27:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:12 smithi115 ceph-mon[35364]: pgmap v9108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:12.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:12 smithi181 ceph-mon[35869]: pgmap v9108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:12 smithi097 ceph-mon[32231]: pgmap v9108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:14 smithi115 ceph-mon[35364]: pgmap v9109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:14 smithi181 ceph-mon[35869]: pgmap v9109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:14 smithi097 ceph-mon[32231]: pgmap v9109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16073.050325 secs 2024-07-26T12:27:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:16 smithi115 ceph-mon[35364]: pgmap v9110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16073.050325 secs 2024-07-26T12:27:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:16 smithi181 ceph-mon[35869]: pgmap v9110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16073.050325 secs 2024-07-26T12:27:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:16 smithi097 ceph-mon[32231]: pgmap v9110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:18 smithi115 ceph-mon[35364]: pgmap v9111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:18 smithi181 ceph-mon[35869]: pgmap v9111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:18 smithi097 ceph-mon[32231]: pgmap v9111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:20 smithi115 ceph-mon[35364]: pgmap v9112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:20 smithi181 ceph-mon[35869]: pgmap v9112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:20 smithi097 ceph-mon[32231]: pgmap v9112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16078.050464 secs 2024-07-26T12:27:21.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16078.050464 secs 2024-07-26T12:27:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16078.050464 secs 2024-07-26T12:27:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:22 smithi115 ceph-mon[35364]: pgmap v9113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:22 smithi181 ceph-mon[35869]: pgmap v9113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:22 smithi097 ceph-mon[32231]: pgmap v9113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:24.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:24.187+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:24.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:24.188+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:27:24.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:24.188+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:27:24.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:24.188+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:24.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:24.188+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:24 smithi115 ceph-mon[35364]: pgmap v9114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:24 smithi181 ceph-mon[35869]: pgmap v9114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:24 smithi097 ceph-mon[32231]: pgmap v9114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16083.050608 secs 2024-07-26T12:27:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:26 smithi115 ceph-mon[35364]: pgmap v9115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16083.050608 secs 2024-07-26T12:27:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:26 smithi181 ceph-mon[35869]: pgmap v9115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16083.050608 secs 2024-07-26T12:27:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:26 smithi097 ceph-mon[32231]: pgmap v9115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:28 smithi115 ceph-mon[35364]: pgmap v9116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:28 smithi181 ceph-mon[35869]: pgmap v9116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:28 smithi097 ceph-mon[32231]: pgmap v9116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:30 smithi115 ceph-mon[35364]: pgmap v9117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:30 smithi181 ceph-mon[35869]: pgmap v9117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:30 smithi097 ceph-mon[32231]: pgmap v9117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16088.050756 secs 2024-07-26T12:27:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16088.050756 secs 2024-07-26T12:27:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16088.050756 secs 2024-07-26T12:27:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:32 smithi115 ceph-mon[35364]: pgmap v9118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:32 smithi181 ceph-mon[35869]: pgmap v9118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:32 smithi097 ceph-mon[32231]: pgmap v9118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:34 smithi115 ceph-mon[35364]: pgmap v9119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:34 smithi181 ceph-mon[35869]: pgmap v9119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:34 smithi097 ceph-mon[32231]: pgmap v9119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16093.050899 secs 2024-07-26T12:27:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:36 smithi115 ceph-mon[35364]: pgmap v9120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16093.050899 secs 2024-07-26T12:27:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:36 smithi181 ceph-mon[35869]: pgmap v9120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16093.050899 secs 2024-07-26T12:27:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:36 smithi097 ceph-mon[32231]: pgmap v9120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:38 smithi115 ceph-mon[35364]: pgmap v9121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:38 smithi181 ceph-mon[35869]: pgmap v9121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:38 smithi097 ceph-mon[32231]: pgmap v9121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:39.197+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:39.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:39.197+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:27:39.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:39.197+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:27:39.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:39.197+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:39.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:39.197+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:40 smithi115 ceph-mon[35364]: pgmap v9122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:40 smithi181 ceph-mon[35869]: pgmap v9122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:40.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:40 smithi097 ceph-mon[32231]: pgmap v9122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16098.051077 secs 2024-07-26T12:27:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16098.051077 secs 2024-07-26T12:27:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16098.051077 secs 2024-07-26T12:27:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:42 smithi115 ceph-mon[35364]: pgmap v9123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:42 smithi181 ceph-mon[35869]: pgmap v9123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:42 smithi097 ceph-mon[32231]: pgmap v9123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:44 smithi115 ceph-mon[35364]: pgmap v9124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:44 smithi181 ceph-mon[35869]: pgmap v9124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:44 smithi097 ceph-mon[32231]: pgmap v9124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16103.051217 secs 2024-07-26T12:27:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:46 smithi115 ceph-mon[35364]: pgmap v9125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16103.051217 secs 2024-07-26T12:27:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:46 smithi181 ceph-mon[35869]: pgmap v9125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16103.051217 secs 2024-07-26T12:27:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:46 smithi097 ceph-mon[32231]: pgmap v9125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:48.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:48 smithi115 ceph-mon[35364]: pgmap v9126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:48 smithi181 ceph-mon[35869]: pgmap v9126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:48 smithi097 ceph-mon[32231]: pgmap v9126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:50 smithi115 ceph-mon[35364]: pgmap v9127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:50 smithi181 ceph-mon[35869]: pgmap v9127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:50 smithi097 ceph-mon[32231]: pgmap v9127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:51.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16108.051328 secs 2024-07-26T12:27:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16108.051328 secs 2024-07-26T12:27:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16108.051328 secs 2024-07-26T12:27:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:52 smithi097 ceph-mon[32231]: pgmap v9128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:52 smithi115 ceph-mon[35364]: pgmap v9128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:52 smithi181 ceph-mon[35869]: pgmap v9128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:54.199+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:54.201 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:54.200+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:27:54.201 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:54.200+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:27:54.201 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:54.200+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:54.201 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:27:54.200+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:54 smithi097 ceph-mon[32231]: pgmap v9129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:54 smithi115 ceph-mon[35364]: pgmap v9129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:54 smithi181 ceph-mon[35869]: pgmap v9129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:56.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16113.051466 secs 2024-07-26T12:27:56.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:56 smithi115 ceph-mon[35364]: pgmap v9130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:56.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16113.051466 secs 2024-07-26T12:27:56.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:56 smithi181 ceph-mon[35869]: pgmap v9130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16113.051466 secs 2024-07-26T12:27:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:56 smithi097 ceph-mon[32231]: pgmap v9130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:58 smithi115 ceph-mon[35364]: pgmap v9131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:27:58.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:27:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:58 smithi181 ceph-mon[35869]: pgmap v9131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:27:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:27:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:58 smithi097 ceph-mon[32231]: pgmap v9131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:27:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:27:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:59.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:27:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:27:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:27:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:28:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:27:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:00 smithi115 ceph-mon[35364]: pgmap v9132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:00 smithi115 ceph-mon[35364]: pgmap v9133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:00 smithi181 ceph-mon[35869]: pgmap v9132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:00 smithi181 ceph-mon[35869]: pgmap v9133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:00 smithi097 ceph-mon[32231]: pgmap v9132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:00 smithi097 ceph-mon[32231]: pgmap v9133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16118.051557 secs 2024-07-26T12:28:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16118.051557 secs 2024-07-26T12:28:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16118.051557 secs 2024-07-26T12:28:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:02 smithi115 ceph-mon[35364]: pgmap v9134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:02 smithi181 ceph-mon[35869]: pgmap v9134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:02 smithi097 ceph-mon[32231]: pgmap v9134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:04 smithi115 ceph-mon[35364]: pgmap v9135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:04 smithi181 ceph-mon[35869]: pgmap v9135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:04 smithi097 ceph-mon[32231]: pgmap v9135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.930+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T12:28:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.930+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:28:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.930+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e0071000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.930+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e0071000 comp rx=0 tx=0).stop 2024-07-26T12:28:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.930+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:28:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.930+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:28:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.930+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 0x7ff3d40b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:28:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.931+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 0x7ff3d40b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:06.933+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 0x7ff3d40b5df0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005d00 tx=0x7ff3e00a4000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:28:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16123.051650 secs 2024-07-26T12:28:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:06 smithi115 ceph-mon[35364]: pgmap v9136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:28:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16123.051650 secs 2024-07-26T12:28:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:06 smithi181 ceph-mon[35869]: pgmap v9136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:28:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16123.051650 secs 2024-07-26T12:28:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:06 smithi097 ceph-mon[32231]: pgmap v9136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:28:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:08 smithi115 ceph-mon[35364]: pgmap v9137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:08 smithi181 ceph-mon[35869]: pgmap v9137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:08 smithi097 ceph-mon[32231]: pgmap v9137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:28:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.203+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.205+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:28:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.205+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:28:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.205+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.205+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.767+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:28:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.767+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:28:09.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.768+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f070 con 0x7ff3f01061a0 2024-07-26T12:28:09.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:09.768+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:28:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:10 smithi115 ceph-mon[35364]: pgmap v9138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:10 smithi181 ceph-mon[35869]: pgmap v9138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:10 smithi097 ceph-mon[32231]: pgmap v9138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16128.051747 secs 2024-07-26T12:28:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16128.051747 secs 2024-07-26T12:28:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16128.051747 secs 2024-07-26T12:28:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:12 smithi115 ceph-mon[35364]: pgmap v9139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:12 smithi181 ceph-mon[35869]: pgmap v9139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:12 smithi097 ceph-mon[32231]: pgmap v9139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:14 smithi115 ceph-mon[35364]: pgmap v9140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:14 smithi181 ceph-mon[35869]: pgmap v9140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:14 smithi097 ceph-mon[32231]: pgmap v9140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16133.051918 secs 2024-07-26T12:28:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:16 smithi115 ceph-mon[35364]: pgmap v9141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16133.051918 secs 2024-07-26T12:28:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:16 smithi181 ceph-mon[35869]: pgmap v9141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16133.051918 secs 2024-07-26T12:28:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:16 smithi097 ceph-mon[32231]: pgmap v9141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:18 smithi115 ceph-mon[35364]: pgmap v9142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:18 smithi181 ceph-mon[35869]: pgmap v9142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:18 smithi097 ceph-mon[32231]: pgmap v9142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:20 smithi115 ceph-mon[35364]: pgmap v9143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:20 smithi181 ceph-mon[35869]: pgmap v9143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:20 smithi097 ceph-mon[32231]: pgmap v9143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16138.052017 secs 2024-07-26T12:28:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16138.052017 secs 2024-07-26T12:28:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16138.052017 secs 2024-07-26T12:28:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:22 smithi115 ceph-mon[35364]: pgmap v9144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:22 smithi181 ceph-mon[35869]: pgmap v9144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:22 smithi097 ceph-mon[32231]: pgmap v9144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:24.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:24.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:28:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:24.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:28:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:24.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:24.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:24 smithi115 ceph-mon[35364]: pgmap v9145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:24 smithi181 ceph-mon[35869]: pgmap v9145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:24 smithi097 ceph-mon[32231]: pgmap v9145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16143.052164 secs 2024-07-26T12:28:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:26 smithi115 ceph-mon[35364]: pgmap v9146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16143.052164 secs 2024-07-26T12:28:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:26 smithi181 ceph-mon[35869]: pgmap v9146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16143.052164 secs 2024-07-26T12:28:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:26 smithi097 ceph-mon[32231]: pgmap v9146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:28 smithi115 ceph-mon[35364]: pgmap v9147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:28 smithi181 ceph-mon[35869]: pgmap v9147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:28 smithi097 ceph-mon[32231]: pgmap v9147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:30 smithi115 ceph-mon[35364]: pgmap v9148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:30 smithi181 ceph-mon[35869]: pgmap v9148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:30 smithi097 ceph-mon[32231]: pgmap v9148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16148.052309 secs 2024-07-26T12:28:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16148.052309 secs 2024-07-26T12:28:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16148.052309 secs 2024-07-26T12:28:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:32 smithi115 ceph-mon[35364]: pgmap v9149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:32 smithi181 ceph-mon[35869]: pgmap v9149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:32 smithi097 ceph-mon[32231]: pgmap v9149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:33 smithi115 ceph-mon[35364]: pgmap v9150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:33 smithi181 ceph-mon[35869]: pgmap v9150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:33 smithi097 ceph-mon[32231]: pgmap v9150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16153.052402 secs 2024-07-26T12:28:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:36 smithi115 ceph-mon[35364]: pgmap v9151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16153.052402 secs 2024-07-26T12:28:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:36 smithi181 ceph-mon[35869]: pgmap v9151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16153.052402 secs 2024-07-26T12:28:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:36 smithi097 ceph-mon[32231]: pgmap v9151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:38 smithi115 ceph-mon[35364]: pgmap v9152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:38 smithi181 ceph-mon[35869]: pgmap v9152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:38 smithi097 ceph-mon[32231]: pgmap v9152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:39.217 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:39.216+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:39.218 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:39.217+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:28:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:39.217+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:28:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:39.217+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:39.217+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:40 smithi115 ceph-mon[35364]: pgmap v9153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:40 smithi181 ceph-mon[35869]: pgmap v9153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:40 smithi097 ceph-mon[32231]: pgmap v9153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16158.052550 secs 2024-07-26T12:28:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16158.052550 secs 2024-07-26T12:28:41.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16158.052550 secs 2024-07-26T12:28:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:42 smithi115 ceph-mon[35364]: pgmap v9154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:42 smithi181 ceph-mon[35869]: pgmap v9154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:42 smithi097 ceph-mon[32231]: pgmap v9154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:44 smithi115 ceph-mon[35364]: pgmap v9155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:44 smithi181 ceph-mon[35869]: pgmap v9155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:44 smithi097 ceph-mon[32231]: pgmap v9155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16163.052692 secs 2024-07-26T12:28:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:46 smithi115 ceph-mon[35364]: pgmap v9156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16163.052692 secs 2024-07-26T12:28:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:46 smithi181 ceph-mon[35869]: pgmap v9156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16163.052692 secs 2024-07-26T12:28:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:46 smithi097 ceph-mon[32231]: pgmap v9156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:48 smithi115 ceph-mon[35364]: pgmap v9157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:48 smithi181 ceph-mon[35869]: pgmap v9157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:48 smithi097 ceph-mon[32231]: pgmap v9157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:50 smithi115 ceph-mon[35364]: pgmap v9158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:50 smithi181 ceph-mon[35869]: pgmap v9158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:50 smithi097 ceph-mon[32231]: pgmap v9158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16168.052789 secs 2024-07-26T12:28:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16168.052789 secs 2024-07-26T12:28:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16168.052789 secs 2024-07-26T12:28:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:52 smithi115 ceph-mon[35364]: pgmap v9159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:52 smithi181 ceph-mon[35869]: pgmap v9159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:52 smithi097 ceph-mon[32231]: pgmap v9159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:54.221 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:54.219+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:28:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:54.220+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:28:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:54.220+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:28:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:54.220+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:28:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:28:54.221+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:54 smithi115 ceph-mon[35364]: pgmap v9160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:54 smithi181 ceph-mon[35869]: pgmap v9160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:54 smithi097 ceph-mon[32231]: pgmap v9160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16173.052929 secs 2024-07-26T12:28:56.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:56 smithi115 ceph-mon[35364]: pgmap v9161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16173.052929 secs 2024-07-26T12:28:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:56 smithi181 ceph-mon[35869]: pgmap v9161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16173.052929 secs 2024-07-26T12:28:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:56 smithi097 ceph-mon[32231]: pgmap v9161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:58 smithi115 ceph-mon[35364]: pgmap v9162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:58 smithi181 ceph-mon[35869]: pgmap v9162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:58 smithi097 ceph-mon[32231]: pgmap v9162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:28:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:28:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:28:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:00 smithi115 ceph-mon[35364]: pgmap v9163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:00.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:00 smithi181 ceph-mon[35869]: pgmap v9163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:00 smithi097 ceph-mon[32231]: pgmap v9163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:01.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16178.053047 secs 2024-07-26T12:29:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:01.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16178.053047 secs 2024-07-26T12:29:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16178.053047 secs 2024-07-26T12:29:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:29:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:02 smithi115 ceph-mon[35364]: pgmap v9164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:29:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:02 smithi115 ceph-mon[35364]: pgmap v9165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:02 smithi181 ceph-mon[35869]: pgmap v9164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:02 smithi181 ceph-mon[35869]: pgmap v9165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:02 smithi097 ceph-mon[32231]: pgmap v9164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:29:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:02 smithi097 ceph-mon[32231]: pgmap v9165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:04 smithi115 ceph-mon[35364]: pgmap v9166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:04 smithi181 ceph-mon[35869]: pgmap v9166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:04 smithi097 ceph-mon[32231]: pgmap v9166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16183.053188 secs 2024-07-26T12:29:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:06 smithi115 ceph-mon[35364]: pgmap v9167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16183.053188 secs 2024-07-26T12:29:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:06 smithi181 ceph-mon[35869]: pgmap v9167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16183.053188 secs 2024-07-26T12:29:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:06 smithi097 ceph-mon[32231]: pgmap v9167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:07 smithi181 ceph-mon[35869]: pgmap v9168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:07 smithi097 ceph-mon[32231]: pgmap v9168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:07 smithi115 ceph-mon[35364]: pgmap v9168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:09.228+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:29:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:09.230+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:29:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:09.230+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:29:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:09.230+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:29:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:09.230+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:10 smithi115 ceph-mon[35364]: pgmap v9169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:10 smithi181 ceph-mon[35869]: pgmap v9169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:10 smithi097 ceph-mon[32231]: pgmap v9169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:29:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16188.053357 secs 2024-07-26T12:29:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16188.053357 secs 2024-07-26T12:29:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16188.053357 secs 2024-07-26T12:29:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:12 smithi115 ceph-mon[35364]: pgmap v9170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:29:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:12 smithi181 ceph-mon[35869]: pgmap v9170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:29:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:12 smithi097 ceph-mon[32231]: pgmap v9170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:29:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:14 smithi115 ceph-mon[35364]: pgmap v9171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:14 smithi181 ceph-mon[35869]: pgmap v9171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:14 smithi097 ceph-mon[32231]: pgmap v9171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16193.053449 secs 2024-07-26T12:29:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:16 smithi115 ceph-mon[35364]: pgmap v9172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16193.053449 secs 2024-07-26T12:29:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:16 smithi181 ceph-mon[35869]: pgmap v9172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16193.053449 secs 2024-07-26T12:29:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:16 smithi097 ceph-mon[32231]: pgmap v9172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:18 smithi115 ceph-mon[35364]: pgmap v9173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:29:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:18 smithi181 ceph-mon[35869]: pgmap v9173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:29:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:18 smithi097 ceph-mon[32231]: pgmap v9173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:29:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:20 smithi115 ceph-mon[35364]: pgmap v9174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:29:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:20 smithi181 ceph-mon[35869]: pgmap v9174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:29:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:20 smithi097 ceph-mon[32231]: pgmap v9174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:29:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16198.053590 secs 2024-07-26T12:29:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16198.053590 secs 2024-07-26T12:29:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16198.053590 secs 2024-07-26T12:29:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:22 smithi115 ceph-mon[35364]: pgmap v9175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:22 smithi181 ceph-mon[35869]: pgmap v9175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:22 smithi097 ceph-mon[32231]: pgmap v9175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:24.233+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:29:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:24.234+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:29:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:24.234+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:29:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:24.234+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:29:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:24.234+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:24 smithi115 ceph-mon[35364]: pgmap v9176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:29:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:24 smithi181 ceph-mon[35869]: pgmap v9176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:29:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:24 smithi097 ceph-mon[32231]: pgmap v9176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:29:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16203.053730 secs 2024-07-26T12:29:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:26 smithi115 ceph-mon[35364]: pgmap v9177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16203.053730 secs 2024-07-26T12:29:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:26 smithi181 ceph-mon[35869]: pgmap v9177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16203.053730 secs 2024-07-26T12:29:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:26 smithi097 ceph-mon[32231]: pgmap v9177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:28 smithi115 ceph-mon[35364]: pgmap v9178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:28 smithi181 ceph-mon[35869]: pgmap v9178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:28 smithi097 ceph-mon[32231]: pgmap v9178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:30 smithi181 ceph-mon[35869]: pgmap v9179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:30 smithi097 ceph-mon[32231]: pgmap v9179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:30 smithi115 ceph-mon[35364]: pgmap v9179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16208.053870 secs 2024-07-26T12:29:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16208.053870 secs 2024-07-26T12:29:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16208.053870 secs 2024-07-26T12:29:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:32 smithi097 ceph-mon[32231]: pgmap v9180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:32 smithi115 ceph-mon[35364]: pgmap v9180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:32 smithi181 ceph-mon[35869]: pgmap v9180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:34 smithi097 ceph-mon[32231]: pgmap v9181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:34 smithi115 ceph-mon[35364]: pgmap v9181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:34 smithi181 ceph-mon[35869]: pgmap v9181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16213.054017 secs 2024-07-26T12:29:36.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:36 smithi115 ceph-mon[35364]: pgmap v9182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16213.054017 secs 2024-07-26T12:29:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:36 smithi181 ceph-mon[35869]: pgmap v9182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16213.054017 secs 2024-07-26T12:29:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:36 smithi097 ceph-mon[32231]: pgmap v9182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:38 smithi115 ceph-mon[35364]: pgmap v9183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:38 smithi181 ceph-mon[35869]: pgmap v9183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:38 smithi097 ceph-mon[32231]: pgmap v9183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:39.242+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:29:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:39.243+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:29:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:39.243+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:29:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:39.243+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:29:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:39.244+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:40 smithi115 ceph-mon[35364]: pgmap v9184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:40 smithi181 ceph-mon[35869]: pgmap v9184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:40 smithi097 ceph-mon[32231]: pgmap v9184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16218.054181 secs 2024-07-26T12:29:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16218.054181 secs 2024-07-26T12:29:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16218.054181 secs 2024-07-26T12:29:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:42 smithi115 ceph-mon[35364]: pgmap v9185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:42 smithi181 ceph-mon[35869]: pgmap v9185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:42 smithi097 ceph-mon[32231]: pgmap v9185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:44 smithi115 ceph-mon[35364]: pgmap v9186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:44 smithi181 ceph-mon[35869]: pgmap v9186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:44 smithi097 ceph-mon[32231]: pgmap v9186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16223.054342 secs 2024-07-26T12:29:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:46 smithi115 ceph-mon[35364]: pgmap v9187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16223.054342 secs 2024-07-26T12:29:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:46 smithi181 ceph-mon[35869]: pgmap v9187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16223.054342 secs 2024-07-26T12:29:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:46 smithi097 ceph-mon[32231]: pgmap v9187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:48 smithi115 ceph-mon[35364]: pgmap v9188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:48 smithi181 ceph-mon[35869]: pgmap v9188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:48 smithi097 ceph-mon[32231]: pgmap v9188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:50 smithi115 ceph-mon[35364]: pgmap v9189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:50 smithi181 ceph-mon[35869]: pgmap v9189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:50 smithi097 ceph-mon[32231]: pgmap v9189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16228.054482 secs 2024-07-26T12:29:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16228.054482 secs 2024-07-26T12:29:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16228.054482 secs 2024-07-26T12:29:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:52 smithi115 ceph-mon[35364]: pgmap v9190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:52 smithi181 ceph-mon[35869]: pgmap v9190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:52 smithi097 ceph-mon[32231]: pgmap v9190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:54.246+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:29:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:54.248+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:29:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:54.248+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:29:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:54.248+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:29:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:29:54.248+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:54 smithi115 ceph-mon[35364]: pgmap v9191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:54 smithi181 ceph-mon[35869]: pgmap v9191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:54 smithi097 ceph-mon[32231]: pgmap v9191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16233.054617 secs 2024-07-26T12:29:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:56 smithi115 ceph-mon[35364]: pgmap v9192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16233.054617 secs 2024-07-26T12:29:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:56 smithi181 ceph-mon[35869]: pgmap v9192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16233.054617 secs 2024-07-26T12:29:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:56 smithi097 ceph-mon[32231]: pgmap v9192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:29:58 smithi115 ceph-mon[35364]: pgmap v9193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:29:58 smithi181 ceph-mon[35869]: pgmap v9193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:29:58 smithi097 ceph-mon[32231]: pgmap v9193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:00 smithi115 ceph-mon[35364]: pgmap v9194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:00 smithi181 ceph-mon[35869]: pgmap v9194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:00 smithi097 ceph-mon[32231]: pgmap v9194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16238.054761 secs 2024-07-26T12:30:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16238.054761 secs 2024-07-26T12:30:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16238.054761 secs 2024-07-26T12:30:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:02 smithi115 ceph-mon[35364]: pgmap v9195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:02.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:02.705 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:02 smithi097 ceph-mon[32231]: pgmap v9195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:02.705 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:02 smithi181 ceph-mon[35869]: pgmap v9195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:04 smithi115 ceph-mon[35364]: pgmap v9196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:04 smithi181 ceph-mon[35869]: pgmap v9196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:04 smithi097 ceph-mon[32231]: pgmap v9196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16243.054890 secs 2024-07-26T12:30:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:06 smithi097 ceph-mon[32231]: pgmap v9197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:06.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16243.054890 secs 2024-07-26T12:30:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:06 smithi115 ceph-mon[35364]: pgmap v9197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16243.054890 secs 2024-07-26T12:30:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:06 smithi181 ceph-mon[35869]: pgmap v9197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:30:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:07 smithi097 ceph-mon[32231]: pgmap v9198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:07 smithi115 ceph-mon[35364]: pgmap v9198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:07 smithi181 ceph-mon[35869]: pgmap v9198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:09.255+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:30:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:09.257+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:30:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:09.257+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:30:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:09.257+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:30:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:09.257+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:09 smithi115 ceph-mon[35364]: pgmap v9199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:09 smithi181 ceph-mon[35869]: pgmap v9199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:09 smithi097 ceph-mon[32231]: pgmap v9199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:11 smithi115 ceph-mon[35364]: pgmap v9200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16248.055031 secs 2024-07-26T12:30:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:11 smithi181 ceph-mon[35869]: pgmap v9200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16248.055031 secs 2024-07-26T12:30:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:11 smithi097 ceph-mon[32231]: pgmap v9200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16248.055031 secs 2024-07-26T12:30:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:13 smithi115 ceph-mon[35364]: pgmap v9201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:13 smithi181 ceph-mon[35869]: pgmap v9201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:13 smithi097 ceph-mon[32231]: pgmap v9201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:15 smithi115 ceph-mon[35364]: pgmap v9202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:15 smithi181 ceph-mon[35869]: pgmap v9202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:15 smithi097 ceph-mon[32231]: pgmap v9202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16253.055162 secs 2024-07-26T12:30:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16253.055162 secs 2024-07-26T12:30:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16253.055162 secs 2024-07-26T12:30:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:17 smithi115 ceph-mon[35364]: pgmap v9203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:17 smithi181 ceph-mon[35869]: pgmap v9203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:17 smithi097 ceph-mon[32231]: pgmap v9203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:30:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:19 smithi115 ceph-mon[35364]: pgmap v9204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:19 smithi181 ceph-mon[35869]: pgmap v9204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:19 smithi097 ceph-mon[32231]: pgmap v9204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:21 smithi115 ceph-mon[35364]: pgmap v9205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16258.055305 secs 2024-07-26T12:30:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:21 smithi181 ceph-mon[35869]: pgmap v9205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16258.055305 secs 2024-07-26T12:30:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:21 smithi097 ceph-mon[32231]: pgmap v9205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16258.055305 secs 2024-07-26T12:30:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:23 smithi115 ceph-mon[35364]: pgmap v9206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:30:23.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:23 smithi181 ceph-mon[35869]: pgmap v9206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:30:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:23 smithi097 ceph-mon[32231]: pgmap v9206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:30:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.259+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:30:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.261+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:30:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.261+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:30:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.261+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:30:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.261+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.792+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:30:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.792+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:30:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.792+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:30:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:24.792+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:30:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:25 smithi115 ceph-mon[35364]: pgmap v9207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:30:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:25 smithi181 ceph-mon[35869]: pgmap v9207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:30:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:25 smithi097 ceph-mon[32231]: pgmap v9207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:30:26.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16263.055463 secs 2024-07-26T12:30:26.885 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16263.055463 secs 2024-07-26T12:30:26.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16263.055463 secs 2024-07-26T12:30:27.924 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:27 smithi181 ceph-mon[35869]: pgmap v9208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:27.941 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:27 smithi097 ceph-mon[32231]: pgmap v9208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:27 smithi115 ceph-mon[35364]: pgmap v9208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:29.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:29 smithi181 ceph-mon[35869]: pgmap v9209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:30:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:29 smithi115 ceph-mon[35364]: pgmap v9209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:30:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:29 smithi097 ceph-mon[32231]: pgmap v9209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:30:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:31 smithi115 ceph-mon[35364]: pgmap v9210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16268.055606 secs 2024-07-26T12:30:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:31 smithi181 ceph-mon[35869]: pgmap v9210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16268.055606 secs 2024-07-26T12:30:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:31 smithi097 ceph-mon[32231]: pgmap v9210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:30:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16268.055606 secs 2024-07-26T12:30:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:33 smithi115 ceph-mon[35364]: pgmap v9211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:33 smithi181 ceph-mon[35869]: pgmap v9211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:33 smithi097 ceph-mon[32231]: pgmap v9211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:35 smithi115 ceph-mon[35364]: pgmap v9212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:35 smithi181 ceph-mon[35869]: pgmap v9212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:35 smithi097 ceph-mon[32231]: pgmap v9212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16273.055747 secs 2024-07-26T12:30:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16273.055747 secs 2024-07-26T12:30:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16273.055747 secs 2024-07-26T12:30:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:37 smithi115 ceph-mon[35364]: pgmap v9213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:37 smithi181 ceph-mon[35869]: pgmap v9213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:37 smithi097 ceph-mon[32231]: pgmap v9213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:39.266+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:39.267+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:39.267+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:30:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:39.267+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:30:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:39.267+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:39 smithi115 ceph-mon[35364]: pgmap v9214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:39 smithi181 ceph-mon[35869]: pgmap v9214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:39 smithi097 ceph-mon[32231]: pgmap v9214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:41 smithi115 ceph-mon[35364]: pgmap v9215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16278.055891 secs 2024-07-26T12:30:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:41 smithi181 ceph-mon[35869]: pgmap v9215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16278.055891 secs 2024-07-26T12:30:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:41 smithi097 ceph-mon[32231]: pgmap v9215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16278.055891 secs 2024-07-26T12:30:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:43 smithi115 ceph-mon[35364]: pgmap v9216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:43 smithi181 ceph-mon[35869]: pgmap v9216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:43 smithi097 ceph-mon[32231]: pgmap v9216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:45 smithi115 ceph-mon[35364]: pgmap v9217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:45 smithi181 ceph-mon[35869]: pgmap v9217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:45 smithi097 ceph-mon[32231]: pgmap v9217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16283.056051 secs 2024-07-26T12:30:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16283.056051 secs 2024-07-26T12:30:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16283.056051 secs 2024-07-26T12:30:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:47 smithi115 ceph-mon[35364]: pgmap v9218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:47 smithi181 ceph-mon[35869]: pgmap v9218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:47 smithi097 ceph-mon[32231]: pgmap v9218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:49 smithi115 ceph-mon[35364]: pgmap v9219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:49 smithi181 ceph-mon[35869]: pgmap v9219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:49 smithi097 ceph-mon[32231]: pgmap v9219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:51 smithi115 ceph-mon[35364]: pgmap v9220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16288.056189 secs 2024-07-26T12:30:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:51 smithi181 ceph-mon[35869]: pgmap v9220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16288.056189 secs 2024-07-26T12:30:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:51 smithi097 ceph-mon[32231]: pgmap v9220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16288.056189 secs 2024-07-26T12:30:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:53 smithi115 ceph-mon[35364]: pgmap v9221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:53 smithi181 ceph-mon[35869]: pgmap v9221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:53 smithi097 ceph-mon[32231]: pgmap v9221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:54.270+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:30:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:54.272+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:30:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:54.272+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:30:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:54.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:30:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:30:54.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:55 smithi115 ceph-mon[35364]: pgmap v9222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:55 smithi181 ceph-mon[35869]: pgmap v9222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:55 smithi097 ceph-mon[32231]: pgmap v9222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16293.056347 secs 2024-07-26T12:30:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16293.056347 secs 2024-07-26T12:30:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16293.056347 secs 2024-07-26T12:30:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:57 smithi115 ceph-mon[35364]: pgmap v9223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:30:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:57 smithi181 ceph-mon[35869]: pgmap v9223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:30:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:57 smithi097 ceph-mon[32231]: pgmap v9223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:30:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:30:58 smithi115 ceph-mon[35364]: pgmap v9224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:30:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:30:58 smithi181 ceph-mon[35869]: pgmap v9224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:30:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:30:58 smithi097 ceph-mon[32231]: pgmap v9224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:31:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:00 smithi097 ceph-mon[32231]: pgmap v9225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16298.056494 secs 2024-07-26T12:31:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:00 smithi115 ceph-mon[35364]: pgmap v9225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16298.056494 secs 2024-07-26T12:31:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:00 smithi181 ceph-mon[35869]: pgmap v9225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16298.056494 secs 2024-07-26T12:31:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:02 smithi097 ceph-mon[32231]: pgmap v9226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:31:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:02 smithi115 ceph-mon[35364]: pgmap v9226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:31:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:02 smithi181 ceph-mon[35869]: pgmap v9226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:31:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:04 smithi097 ceph-mon[32231]: pgmap v9227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:04 smithi115 ceph-mon[35364]: pgmap v9227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:04 smithi181 ceph-mon[35869]: pgmap v9227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16303.056640 secs 2024-07-26T12:31:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16303.056640 secs 2024-07-26T12:31:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16303.056640 secs 2024-07-26T12:31:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:07 smithi115 ceph-mon[35364]: pgmap v9228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:07 smithi181 ceph-mon[35869]: pgmap v9228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:07 smithi097 ceph-mon[32231]: pgmap v9228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:08 smithi115 ceph-mon[35364]: pgmap v9229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:31:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:08 smithi115 ceph-mon[35364]: pgmap v9230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:08.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:31:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:08 smithi181 ceph-mon[35869]: pgmap v9229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:31:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:08 smithi181 ceph-mon[35869]: pgmap v9230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:08.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:31:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:08 smithi097 ceph-mon[32231]: pgmap v9229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:31:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:08 smithi097 ceph-mon[32231]: pgmap v9230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:31:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.280+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:31:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.281+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:31:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.282+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:31:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.282+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:31:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.282+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:09.800 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.800+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:31:09.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.800+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:31:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.800+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:31:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:09.801+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:31:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:10 smithi115 ceph-mon[35364]: pgmap v9231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:10 smithi181 ceph-mon[35869]: pgmap v9231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:10 smithi097 ceph-mon[32231]: pgmap v9231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16308.056774 secs 2024-07-26T12:31:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16308.056774 secs 2024-07-26T12:31:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16308.056774 secs 2024-07-26T12:31:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:12 smithi181 ceph-mon[35869]: pgmap v9232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:12 smithi097 ceph-mon[32231]: pgmap v9232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:12 smithi115 ceph-mon[35364]: pgmap v9232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:14.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:14 smithi181 ceph-mon[35869]: pgmap v9233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:31:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:14 smithi097 ceph-mon[32231]: pgmap v9233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:31:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:14 smithi115 ceph-mon[35364]: pgmap v9233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:31:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16313.056916 secs 2024-07-26T12:31:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:16 smithi097 ceph-mon[32231]: pgmap v9234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16313.056916 secs 2024-07-26T12:31:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:16 smithi115 ceph-mon[35364]: pgmap v9234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16313.056916 secs 2024-07-26T12:31:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:16 smithi181 ceph-mon[35869]: pgmap v9234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:18 smithi097 ceph-mon[32231]: pgmap v9235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:31:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:18 smithi115 ceph-mon[35364]: pgmap v9235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:31:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:18 smithi181 ceph-mon[35869]: pgmap v9235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:31:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:20 smithi097 ceph-mon[32231]: pgmap v9236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:20 smithi115 ceph-mon[35364]: pgmap v9236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:20 smithi181 ceph-mon[35869]: pgmap v9236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16318.057052 secs 2024-07-26T12:31:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16318.057052 secs 2024-07-26T12:31:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16318.057052 secs 2024-07-26T12:31:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:22 smithi115 ceph-mon[35364]: pgmap v9237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:22 smithi181 ceph-mon[35869]: pgmap v9237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:22 smithi097 ceph-mon[32231]: pgmap v9237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:24.284+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:31:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:24.285+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:31:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:24.285+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:31:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:24.285+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:31:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:24.285+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:24 smithi115 ceph-mon[35364]: pgmap v9238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:24 smithi181 ceph-mon[35869]: pgmap v9238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:24 smithi097 ceph-mon[32231]: pgmap v9238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:26.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16323.057208 secs 2024-07-26T12:31:26.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:26 smithi181 ceph-mon[35869]: pgmap v9239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:26.804 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16323.057208 secs 2024-07-26T12:31:26.804 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:26 smithi097 ceph-mon[32231]: pgmap v9239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16323.057208 secs 2024-07-26T12:31:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:26 smithi115 ceph-mon[35364]: pgmap v9239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:28 smithi115 ceph-mon[35364]: pgmap v9240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:28 smithi181 ceph-mon[35869]: pgmap v9240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:28 smithi097 ceph-mon[32231]: pgmap v9240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:30 smithi115 ceph-mon[35364]: pgmap v9241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:30 smithi181 ceph-mon[35869]: pgmap v9241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:30 smithi097 ceph-mon[32231]: pgmap v9241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16328.057342 secs 2024-07-26T12:31:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16328.057342 secs 2024-07-26T12:31:32.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16328.057342 secs 2024-07-26T12:31:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:32 smithi115 ceph-mon[35364]: pgmap v9242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:32 smithi181 ceph-mon[35869]: pgmap v9242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:32 smithi097 ceph-mon[32231]: pgmap v9242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:34 smithi115 ceph-mon[35364]: pgmap v9243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:34 smithi181 ceph-mon[35869]: pgmap v9243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:34 smithi097 ceph-mon[32231]: pgmap v9243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16333.057471 secs 2024-07-26T12:31:36.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:36 smithi115 ceph-mon[35364]: pgmap v9244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16333.057471 secs 2024-07-26T12:31:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:36 smithi181 ceph-mon[35869]: pgmap v9244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16333.057471 secs 2024-07-26T12:31:37.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:36 smithi097 ceph-mon[32231]: pgmap v9244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:38 smithi115 ceph-mon[35364]: pgmap v9245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:38 smithi181 ceph-mon[35869]: pgmap v9245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:38 smithi097 ceph-mon[32231]: pgmap v9245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:39.293+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:31:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:39.294+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:31:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:39.294+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:31:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:39.294+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:31:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:39.294+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:40 smithi115 ceph-mon[35364]: pgmap v9246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:40 smithi181 ceph-mon[35869]: pgmap v9246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:41.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:40 smithi097 ceph-mon[32231]: pgmap v9246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16338.057612 secs 2024-07-26T12:31:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16338.057612 secs 2024-07-26T12:31:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16338.057612 secs 2024-07-26T12:31:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:42 smithi115 ceph-mon[35364]: pgmap v9247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:42 smithi181 ceph-mon[35869]: pgmap v9247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:42 smithi097 ceph-mon[32231]: pgmap v9247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:44 smithi115 ceph-mon[35364]: pgmap v9248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:44 smithi181 ceph-mon[35869]: pgmap v9248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:44 smithi097 ceph-mon[32231]: pgmap v9248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16343.057753 secs 2024-07-26T12:31:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:46 smithi115 ceph-mon[35364]: pgmap v9249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16343.057753 secs 2024-07-26T12:31:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:46 smithi181 ceph-mon[35869]: pgmap v9249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16343.057753 secs 2024-07-26T12:31:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:46 smithi097 ceph-mon[32231]: pgmap v9249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:48 smithi115 ceph-mon[35364]: pgmap v9250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:48 smithi181 ceph-mon[35869]: pgmap v9250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:48 smithi097 ceph-mon[32231]: pgmap v9250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:50 smithi115 ceph-mon[35364]: pgmap v9251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:50 smithi181 ceph-mon[35869]: pgmap v9251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:50 smithi097 ceph-mon[32231]: pgmap v9251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16348.057917 secs 2024-07-26T12:31:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16348.057917 secs 2024-07-26T12:31:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16348.057917 secs 2024-07-26T12:31:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:52 smithi115 ceph-mon[35364]: pgmap v9252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:52 smithi181 ceph-mon[35869]: pgmap v9252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:52 smithi097 ceph-mon[32231]: pgmap v9252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:31:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:54.296+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:31:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:54.298+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:31:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:54.298+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:31:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:54.298+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:31:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:31:54.298+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:54 smithi115 ceph-mon[35364]: pgmap v9253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:54.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:54 smithi181 ceph-mon[35869]: pgmap v9253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:54 smithi097 ceph-mon[32231]: pgmap v9253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16353.058007 secs 2024-07-26T12:31:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:56 smithi115 ceph-mon[35364]: pgmap v9254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16353.058007 secs 2024-07-26T12:31:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:56 smithi181 ceph-mon[35869]: pgmap v9254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16353.058007 secs 2024-07-26T12:31:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:56 smithi097 ceph-mon[32231]: pgmap v9254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:31:58 smithi115 ceph-mon[35364]: pgmap v9255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:31:58 smithi181 ceph-mon[35869]: pgmap v9255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:31:58 smithi097 ceph-mon[32231]: pgmap v9255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:00 smithi115 ceph-mon[35364]: pgmap v9256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:00 smithi181 ceph-mon[35869]: pgmap v9256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:00 smithi097 ceph-mon[32231]: pgmap v9256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16358.058144 secs 2024-07-26T12:32:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16358.058144 secs 2024-07-26T12:32:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16358.058144 secs 2024-07-26T12:32:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:02 smithi115 ceph-mon[35364]: pgmap v9257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:02 smithi181 ceph-mon[35869]: pgmap v9257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:02 smithi097 ceph-mon[32231]: pgmap v9257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:04 smithi115 ceph-mon[35364]: pgmap v9258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:04 smithi181 ceph-mon[35869]: pgmap v9258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:04 smithi097 ceph-mon[32231]: pgmap v9258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16363.058294 secs 2024-07-26T12:32:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:06 smithi115 ceph-mon[35364]: pgmap v9259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16363.058294 secs 2024-07-26T12:32:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:06 smithi181 ceph-mon[35869]: pgmap v9259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16363.058294 secs 2024-07-26T12:32:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:06 smithi097 ceph-mon[32231]: pgmap v9259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:08 smithi115 ceph-mon[35364]: pgmap v9260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:08 smithi181 ceph-mon[35869]: pgmap v9260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:08.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:08 smithi097 ceph-mon[32231]: pgmap v9260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:09.306+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:32:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:09.307+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:32:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:09.307+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:32:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:09.307+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:32:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:09.308+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:09 smithi181 ceph-mon[35869]: pgmap v9261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:32:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:09 smithi181 ceph-mon[35869]: pgmap v9262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:32:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:09 smithi097 ceph-mon[32231]: pgmap v9261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:32:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:09 smithi097 ceph-mon[32231]: pgmap v9262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:10.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:32:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:09 smithi115 ceph-mon[35364]: pgmap v9261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:32:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:09 smithi115 ceph-mon[35364]: pgmap v9262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:32:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16368.058433 secs 2024-07-26T12:32:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16368.058433 secs 2024-07-26T12:32:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16368.058433 secs 2024-07-26T12:32:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:11 smithi115 ceph-mon[35364]: pgmap v9263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:11 smithi181 ceph-mon[35869]: pgmap v9263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:11 smithi097 ceph-mon[32231]: pgmap v9263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:13 smithi115 ceph-mon[35364]: pgmap v9264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:13 smithi181 ceph-mon[35869]: pgmap v9264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:13 smithi097 ceph-mon[32231]: pgmap v9264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:15 smithi115 ceph-mon[35364]: pgmap v9265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16373.058576 secs 2024-07-26T12:32:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:15 smithi181 ceph-mon[35869]: pgmap v9265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16373.058576 secs 2024-07-26T12:32:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:15 smithi097 ceph-mon[32231]: pgmap v9265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16373.058576 secs 2024-07-26T12:32:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:17 smithi115 ceph-mon[35364]: pgmap v9266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:32:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:17 smithi181 ceph-mon[35869]: pgmap v9266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:32:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:17 smithi097 ceph-mon[32231]: pgmap v9266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:32:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:19 smithi115 ceph-mon[35364]: pgmap v9267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:19 smithi181 ceph-mon[35869]: pgmap v9267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:19 smithi097 ceph-mon[32231]: pgmap v9267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16378.058674 secs 2024-07-26T12:32:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16378.058674 secs 2024-07-26T12:32:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16378.058674 secs 2024-07-26T12:32:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:22 smithi115 ceph-mon[35364]: pgmap v9268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:22 smithi181 ceph-mon[35869]: pgmap v9268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:22 smithi097 ceph-mon[32231]: pgmap v9268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:24.310+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:32:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:24.311+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:32:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:24.311+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:32:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:24.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:32:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:24.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:24 smithi115 ceph-mon[35364]: pgmap v9269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:24 smithi181 ceph-mon[35869]: pgmap v9269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:24 smithi097 ceph-mon[32231]: pgmap v9269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:26 smithi115 ceph-mon[35364]: pgmap v9270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16383.058771 secs 2024-07-26T12:32:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:26 smithi181 ceph-mon[35869]: pgmap v9270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16383.058771 secs 2024-07-26T12:32:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:26 smithi097 ceph-mon[32231]: pgmap v9270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16383.058771 secs 2024-07-26T12:32:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:28 smithi115 ceph-mon[35364]: pgmap v9271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:28 smithi181 ceph-mon[35869]: pgmap v9271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:28 smithi097 ceph-mon[32231]: pgmap v9271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:30 smithi115 ceph-mon[35364]: pgmap v9272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:30 smithi181 ceph-mon[35869]: pgmap v9272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:30 smithi097 ceph-mon[32231]: pgmap v9272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16388.058911 secs 2024-07-26T12:32:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16388.058911 secs 2024-07-26T12:32:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16388.058911 secs 2024-07-26T12:32:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:32 smithi115 ceph-mon[35364]: pgmap v9273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:32 smithi181 ceph-mon[35869]: pgmap v9273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:32 smithi097 ceph-mon[32231]: pgmap v9273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:34 smithi115 ceph-mon[35364]: pgmap v9274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:34 smithi181 ceph-mon[35869]: pgmap v9274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:34 smithi097 ceph-mon[32231]: pgmap v9274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:36 smithi115 ceph-mon[35364]: pgmap v9275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16393.059007 secs 2024-07-26T12:32:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:36 smithi181 ceph-mon[35869]: pgmap v9275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16393.059007 secs 2024-07-26T12:32:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:36 smithi097 ceph-mon[32231]: pgmap v9275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16393.059007 secs 2024-07-26T12:32:38.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:38 smithi181 ceph-mon[35869]: pgmap v9276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:38 smithi097 ceph-mon[32231]: pgmap v9276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:38 smithi115 ceph-mon[35364]: pgmap v9276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:39.313+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:32:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:39.314+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:32:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:39.314+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:32:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:39.314+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:32:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:39.314+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:39 smithi097 ceph-mon[32231]: pgmap v9277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:39 smithi115 ceph-mon[35364]: pgmap v9277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:39 smithi181 ceph-mon[35869]: pgmap v9277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16398.059136 secs 2024-07-26T12:32:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16398.059136 secs 2024-07-26T12:32:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16398.059136 secs 2024-07-26T12:32:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:41 smithi115 ceph-mon[35364]: pgmap v9278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:41 smithi181 ceph-mon[35869]: pgmap v9278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:41 smithi097 ceph-mon[32231]: pgmap v9278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:43 smithi115 ceph-mon[35364]: pgmap v9279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:43 smithi181 ceph-mon[35869]: pgmap v9279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:43 smithi097 ceph-mon[32231]: pgmap v9279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:45 smithi115 ceph-mon[35364]: pgmap v9280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:46.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16403.059275 secs 2024-07-26T12:32:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:45 smithi181 ceph-mon[35869]: pgmap v9280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16403.059275 secs 2024-07-26T12:32:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:45 smithi097 ceph-mon[32231]: pgmap v9280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16403.059275 secs 2024-07-26T12:32:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:47 smithi115 ceph-mon[35364]: pgmap v9281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:47 smithi181 ceph-mon[35869]: pgmap v9281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:47 smithi097 ceph-mon[32231]: pgmap v9281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:49 smithi115 ceph-mon[35364]: pgmap v9282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:49 smithi181 ceph-mon[35869]: pgmap v9282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:49 smithi097 ceph-mon[32231]: pgmap v9282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16408.059420 secs 2024-07-26T12:32:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16408.059420 secs 2024-07-26T12:32:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16408.059420 secs 2024-07-26T12:32:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:51 smithi115 ceph-mon[35364]: pgmap v9283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:51 smithi181 ceph-mon[35869]: pgmap v9283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:51 smithi097 ceph-mon[32231]: pgmap v9283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:53 smithi115 ceph-mon[35364]: pgmap v9284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:53 smithi181 ceph-mon[35869]: pgmap v9284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:53 smithi097 ceph-mon[32231]: pgmap v9284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:54.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:54.316+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:32:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:54.317+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:32:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:54.317+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:32:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:54.317+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:32:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:32:54.318+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:55 smithi181 ceph-mon[35869]: pgmap v9285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16413.059558 secs 2024-07-26T12:32:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:55 smithi097 ceph-mon[32231]: pgmap v9285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16413.059558 secs 2024-07-26T12:32:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:55 smithi115 ceph-mon[35364]: pgmap v9285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:56.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16413.059558 secs 2024-07-26T12:32:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:57 smithi097 ceph-mon[32231]: pgmap v9286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:57 smithi115 ceph-mon[35364]: pgmap v9286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:57 smithi181 ceph-mon[35869]: pgmap v9286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:32:59 smithi097 ceph-mon[32231]: pgmap v9287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:32:59 smithi115 ceph-mon[35364]: pgmap v9287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:32:59 smithi181 ceph-mon[35869]: pgmap v9287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16418.059700 secs 2024-07-26T12:33:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16418.059700 secs 2024-07-26T12:33:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16418.059700 secs 2024-07-26T12:33:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:02 smithi115 ceph-mon[35364]: pgmap v9288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:02 smithi181 ceph-mon[35869]: pgmap v9288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:02 smithi097 ceph-mon[32231]: pgmap v9288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:04 smithi115 ceph-mon[35364]: pgmap v9289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:04 smithi181 ceph-mon[35869]: pgmap v9289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:04 smithi097 ceph-mon[32231]: pgmap v9289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:06 smithi115 ceph-mon[35364]: pgmap v9290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16423.059841 secs 2024-07-26T12:33:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:06 smithi181 ceph-mon[35869]: pgmap v9290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16423.059841 secs 2024-07-26T12:33:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:06 smithi097 ceph-mon[32231]: pgmap v9290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16423.059841 secs 2024-07-26T12:33:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:08 smithi115 ceph-mon[35364]: pgmap v9291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:08 smithi181 ceph-mon[35869]: pgmap v9291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:08 smithi097 ceph-mon[32231]: pgmap v9291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:09.325+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:33:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:09.326+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:33:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:09.326+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:33:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:09.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:33:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:09.326+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:10 smithi115 ceph-mon[35364]: pgmap v9292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:10 smithi181 ceph-mon[35869]: pgmap v9292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:10 smithi097 ceph-mon[32231]: pgmap v9292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16428.059978 secs 2024-07-26T12:33:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:11 smithi115 ceph-mon[35364]: pgmap v9293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:33:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:11 smithi115 ceph-mon[35364]: pgmap v9294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:11.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:33:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16428.059978 secs 2024-07-26T12:33:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:11 smithi181 ceph-mon[35869]: pgmap v9293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:33:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:11 smithi181 ceph-mon[35869]: pgmap v9294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:33:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16428.059978 secs 2024-07-26T12:33:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:11 smithi097 ceph-mon[32231]: pgmap v9293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:33:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:11 smithi097 ceph-mon[32231]: pgmap v9294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:11.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:33:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:13 smithi115 ceph-mon[35364]: pgmap v9295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:33:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:13 smithi181 ceph-mon[35869]: pgmap v9295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:33:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:13 smithi097 ceph-mon[32231]: pgmap v9295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:33:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:15 smithi115 ceph-mon[35364]: pgmap v9296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:15 smithi181 ceph-mon[35869]: pgmap v9296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:15 smithi097 ceph-mon[32231]: pgmap v9296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16433.060115 secs 2024-07-26T12:33:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16433.060115 secs 2024-07-26T12:33:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16433.060115 secs 2024-07-26T12:33:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:17 smithi115 ceph-mon[35364]: pgmap v9297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:17 smithi181 ceph-mon[35869]: pgmap v9297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:17 smithi097 ceph-mon[32231]: pgmap v9297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:19 smithi115 ceph-mon[35364]: pgmap v9298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:33:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:19 smithi181 ceph-mon[35869]: pgmap v9298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:33:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:19 smithi097 ceph-mon[32231]: pgmap v9298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:33:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16438.060284 secs 2024-07-26T12:33:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:21 smithi115 ceph-mon[35364]: pgmap v9299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16438.060284 secs 2024-07-26T12:33:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:21 smithi181 ceph-mon[35869]: pgmap v9299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16438.060284 secs 2024-07-26T12:33:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:21 smithi097 ceph-mon[32231]: pgmap v9299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:23 smithi115 ceph-mon[35364]: pgmap v9300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:23 smithi181 ceph-mon[35869]: pgmap v9300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:23 smithi097 ceph-mon[32231]: pgmap v9300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:24.328+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:33:24.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:24.329+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:33:24.330 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:24.329+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:33:24.331 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:24.329+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:33:24.331 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:24.329+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:25 smithi115 ceph-mon[35364]: pgmap v9301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:25 smithi181 ceph-mon[35869]: pgmap v9301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:25 smithi097 ceph-mon[32231]: pgmap v9301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16443.060425 secs 2024-07-26T12:33:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16443.060425 secs 2024-07-26T12:33:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16443.060425 secs 2024-07-26T12:33:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:27 smithi115 ceph-mon[35364]: pgmap v9302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:27 smithi181 ceph-mon[35869]: pgmap v9302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:27 smithi097 ceph-mon[32231]: pgmap v9302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:29 smithi181 ceph-mon[35869]: pgmap v9303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:29 smithi097 ceph-mon[32231]: pgmap v9303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:29 smithi115 ceph-mon[35364]: pgmap v9303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16448.060569 secs 2024-07-26T12:33:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:31 smithi097 ceph-mon[32231]: pgmap v9304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16448.060569 secs 2024-07-26T12:33:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:31 smithi115 ceph-mon[35364]: pgmap v9304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16448.060569 secs 2024-07-26T12:33:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:31 smithi181 ceph-mon[35869]: pgmap v9304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:32 smithi097 ceph-mon[32231]: pgmap v9305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:32 smithi115 ceph-mon[35364]: pgmap v9305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:32 smithi181 ceph-mon[35869]: pgmap v9305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:35 smithi115 ceph-mon[35364]: pgmap v9306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:35 smithi181 ceph-mon[35869]: pgmap v9306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:35 smithi097 ceph-mon[32231]: pgmap v9306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16453.060713 secs 2024-07-26T12:33:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16453.060713 secs 2024-07-26T12:33:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16453.060713 secs 2024-07-26T12:33:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:37 smithi115 ceph-mon[35364]: pgmap v9307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:37 smithi181 ceph-mon[35869]: pgmap v9307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:37 smithi097 ceph-mon[32231]: pgmap v9307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:39.337+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:33:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:39.338+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:33:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:39.338+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:33:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:39.338+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:33:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:39.338+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:39 smithi115 ceph-mon[35364]: pgmap v9308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:39 smithi181 ceph-mon[35869]: pgmap v9308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:39 smithi097 ceph-mon[32231]: pgmap v9308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16458.060851 secs 2024-07-26T12:33:41.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:41 smithi115 ceph-mon[35364]: pgmap v9309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16458.060851 secs 2024-07-26T12:33:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:41 smithi181 ceph-mon[35869]: pgmap v9309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16458.060851 secs 2024-07-26T12:33:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:41 smithi097 ceph-mon[32231]: pgmap v9309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:43 smithi115 ceph-mon[35364]: pgmap v9310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:43 smithi181 ceph-mon[35869]: pgmap v9310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:43 smithi097 ceph-mon[32231]: pgmap v9310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:45 smithi115 ceph-mon[35364]: pgmap v9311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:45 smithi181 ceph-mon[35869]: pgmap v9311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:45 smithi097 ceph-mon[32231]: pgmap v9311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16463.060988 secs 2024-07-26T12:33:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16463.060988 secs 2024-07-26T12:33:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16463.060988 secs 2024-07-26T12:33:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:47 smithi115 ceph-mon[35364]: pgmap v9312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:47 smithi181 ceph-mon[35869]: pgmap v9312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:47 smithi097 ceph-mon[32231]: pgmap v9312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:49 smithi115 ceph-mon[35364]: pgmap v9313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:49 smithi181 ceph-mon[35869]: pgmap v9313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:49 smithi097 ceph-mon[32231]: pgmap v9313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16468.061133 secs 2024-07-26T12:33:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:51 smithi115 ceph-mon[35364]: pgmap v9314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16468.061133 secs 2024-07-26T12:33:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:51 smithi181 ceph-mon[35869]: pgmap v9314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16468.061133 secs 2024-07-26T12:33:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:51 smithi097 ceph-mon[32231]: pgmap v9314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:53 smithi115 ceph-mon[35364]: pgmap v9315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:53 smithi181 ceph-mon[35869]: pgmap v9315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:53 smithi097 ceph-mon[32231]: pgmap v9315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:54.340+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:33:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:54.342+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:33:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:54.342+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:33:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:54.342+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:33:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:33:54.342+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:55 smithi115 ceph-mon[35364]: pgmap v9316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:55 smithi181 ceph-mon[35869]: pgmap v9316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:55 smithi097 ceph-mon[32231]: pgmap v9316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16473.061269 secs 2024-07-26T12:33:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16473.061269 secs 2024-07-26T12:33:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16473.061269 secs 2024-07-26T12:33:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:57 smithi097 ceph-mon[32231]: pgmap v9317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:57 smithi115 ceph-mon[35364]: pgmap v9317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:57 smithi181 ceph-mon[35869]: pgmap v9317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:33:59 smithi097 ceph-mon[32231]: pgmap v9318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:33:59 smithi115 ceph-mon[35364]: pgmap v9318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:33:59 smithi181 ceph-mon[35869]: pgmap v9318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16478.061409 secs 2024-07-26T12:34:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:01 smithi097 ceph-mon[32231]: pgmap v9319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16478.061409 secs 2024-07-26T12:34:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:01 smithi115 ceph-mon[35364]: pgmap v9319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16478.061409 secs 2024-07-26T12:34:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:01 smithi181 ceph-mon[35869]: pgmap v9319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:03 smithi115 ceph-mon[35364]: pgmap v9320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:03 smithi181 ceph-mon[35869]: pgmap v9320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:03 smithi097 ceph-mon[32231]: pgmap v9320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:05 smithi115 ceph-mon[35364]: pgmap v9321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:05 smithi181 ceph-mon[35869]: pgmap v9321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:05 smithi097 ceph-mon[32231]: pgmap v9321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16483.061568 secs 2024-07-26T12:34:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16483.061568 secs 2024-07-26T12:34:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16483.061568 secs 2024-07-26T12:34:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:07 smithi115 ceph-mon[35364]: pgmap v9322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:07 smithi181 ceph-mon[35869]: pgmap v9322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:07 smithi097 ceph-mon[32231]: pgmap v9322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.349+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:34:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.350+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:34:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.350+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:34:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.350+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:34:09.352 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.351+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.832+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:34:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.832+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:34:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.833+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:34:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:09.833+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:34:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:09 smithi115 ceph-mon[35364]: pgmap v9323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:09 smithi181 ceph-mon[35869]: pgmap v9323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:09 smithi097 ceph-mon[32231]: pgmap v9323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16488.061713 secs 2024-07-26T12:34:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:11 smithi115 ceph-mon[35364]: pgmap v9324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:11 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16488.061713 secs 2024-07-26T12:34:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:11 smithi181 ceph-mon[35869]: pgmap v9324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:11 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16488.061713 secs 2024-07-26T12:34:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:11 smithi097 ceph-mon[32231]: pgmap v9324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:11 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:12.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:12 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:34:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:12 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:34:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:12 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:34:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:13 smithi115 ceph-mon[35364]: pgmap v9325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:34:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:13 smithi115 ceph-mon[35364]: pgmap v9326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:13 smithi181 ceph-mon[35869]: pgmap v9325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:34:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:13 smithi181 ceph-mon[35869]: pgmap v9326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:13 smithi097 ceph-mon[32231]: pgmap v9325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:34:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:13 smithi097 ceph-mon[32231]: pgmap v9326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:15 smithi115 ceph-mon[35364]: pgmap v9327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:15 smithi181 ceph-mon[35869]: pgmap v9327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:15 smithi097 ceph-mon[32231]: pgmap v9327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16493.061797 secs 2024-07-26T12:34:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16493.061797 secs 2024-07-26T12:34:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16493.061797 secs 2024-07-26T12:34:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:17 smithi115 ceph-mon[35364]: pgmap v9328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:17 smithi181 ceph-mon[35869]: pgmap v9328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:17 smithi097 ceph-mon[32231]: pgmap v9328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:19 smithi181 ceph-mon[35869]: pgmap v9329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:19 smithi097 ceph-mon[32231]: pgmap v9329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:19 smithi115 ceph-mon[35364]: pgmap v9329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:21 smithi097 ceph-mon[32231]: pgmap v9330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16498.061941 secs 2024-07-26T12:34:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:21 smithi115 ceph-mon[35364]: pgmap v9330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16498.061941 secs 2024-07-26T12:34:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:21 smithi181 ceph-mon[35869]: pgmap v9330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16498.061941 secs 2024-07-26T12:34:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:23 smithi097 ceph-mon[32231]: pgmap v9331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:23 smithi115 ceph-mon[35364]: pgmap v9331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:23 smithi181 ceph-mon[35869]: pgmap v9331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:24.354+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:34:24.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:24.355+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:34:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:24.355+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:34:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:24.355+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:34:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:24.355+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:24 smithi115 ceph-mon[35364]: pgmap v9332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:24 smithi181 ceph-mon[35869]: pgmap v9332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:24 smithi097 ceph-mon[32231]: pgmap v9332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:26.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16503.062100 secs 2024-07-26T12:34:26.135 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16503.062100 secs 2024-07-26T12:34:26.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16503.062100 secs 2024-07-26T12:34:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:26 smithi115 ceph-mon[35364]: pgmap v9333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:26 smithi181 ceph-mon[35869]: pgmap v9333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:27.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:26 smithi097 ceph-mon[32231]: pgmap v9333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:28 smithi115 ceph-mon[35364]: pgmap v9334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:28 smithi181 ceph-mon[35869]: pgmap v9334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:28 smithi097 ceph-mon[32231]: pgmap v9334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:30 smithi115 ceph-mon[35364]: pgmap v9335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16508.062279 secs 2024-07-26T12:34:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:30 smithi181 ceph-mon[35869]: pgmap v9335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16508.062279 secs 2024-07-26T12:34:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:30 smithi097 ceph-mon[32231]: pgmap v9335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16508.062279 secs 2024-07-26T12:34:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:32 smithi115 ceph-mon[35364]: pgmap v9336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:32 smithi181 ceph-mon[35869]: pgmap v9336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:32 smithi097 ceph-mon[32231]: pgmap v9336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:34 smithi115 ceph-mon[35364]: pgmap v9337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:34 smithi181 ceph-mon[35869]: pgmap v9337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:34 smithi097 ceph-mon[32231]: pgmap v9337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16513.062375 secs 2024-07-26T12:34:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16513.062375 secs 2024-07-26T12:34:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16513.062375 secs 2024-07-26T12:34:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:36 smithi115 ceph-mon[35364]: pgmap v9338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:36 smithi181 ceph-mon[35869]: pgmap v9338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:36 smithi097 ceph-mon[32231]: pgmap v9338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:38 smithi115 ceph-mon[35364]: pgmap v9339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:38 smithi181 ceph-mon[35869]: pgmap v9339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:38 smithi097 ceph-mon[32231]: pgmap v9339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:39.362+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:39.364+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:39.364+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:39.364+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:39.364+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:40 smithi115 ceph-mon[35364]: pgmap v9340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16518.062517 secs 2024-07-26T12:34:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:40 smithi181 ceph-mon[35869]: pgmap v9340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16518.062517 secs 2024-07-26T12:34:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:40 smithi097 ceph-mon[32231]: pgmap v9340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16518.062517 secs 2024-07-26T12:34:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:42 smithi115 ceph-mon[35364]: pgmap v9341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:42 smithi181 ceph-mon[35869]: pgmap v9341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:42 smithi097 ceph-mon[32231]: pgmap v9341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:44 smithi115 ceph-mon[35364]: pgmap v9342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:44 smithi181 ceph-mon[35869]: pgmap v9342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:44 smithi097 ceph-mon[32231]: pgmap v9342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16523.062679 secs 2024-07-26T12:34:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16523.062679 secs 2024-07-26T12:34:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16523.062679 secs 2024-07-26T12:34:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:46 smithi115 ceph-mon[35364]: pgmap v9343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:46 smithi181 ceph-mon[35869]: pgmap v9343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:46 smithi097 ceph-mon[32231]: pgmap v9343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:48 smithi115 ceph-mon[35364]: pgmap v9344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:48 smithi181 ceph-mon[35869]: pgmap v9344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:48 smithi097 ceph-mon[32231]: pgmap v9344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:50 smithi115 ceph-mon[35364]: pgmap v9345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16528.062819 secs 2024-07-26T12:34:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:50 smithi181 ceph-mon[35869]: pgmap v9345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16528.062819 secs 2024-07-26T12:34:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:50 smithi097 ceph-mon[32231]: pgmap v9345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16528.062819 secs 2024-07-26T12:34:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:52 smithi115 ceph-mon[35364]: pgmap v9346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:53.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:52 smithi181 ceph-mon[35869]: pgmap v9346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:52 smithi097 ceph-mon[32231]: pgmap v9346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:54.366+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:34:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:54.367+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:34:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:54.367+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:34:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:54.367+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:34:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:34:54.368+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:54 smithi115 ceph-mon[35364]: pgmap v9347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:54 smithi181 ceph-mon[35869]: pgmap v9347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:54 smithi097 ceph-mon[32231]: pgmap v9347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16533.062962 secs 2024-07-26T12:34:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16533.062962 secs 2024-07-26T12:34:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16533.062962 secs 2024-07-26T12:34:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:56 smithi115 ceph-mon[35364]: pgmap v9348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:56 smithi181 ceph-mon[35869]: pgmap v9348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:56 smithi097 ceph-mon[32231]: pgmap v9348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:34:58 smithi115 ceph-mon[35364]: pgmap v9349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:34:58 smithi181 ceph-mon[35869]: pgmap v9349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:34:58 smithi097 ceph-mon[32231]: pgmap v9349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:35:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:00 smithi115 ceph-mon[35364]: pgmap v9350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16538.063103 secs 2024-07-26T12:35:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:00 smithi181 ceph-mon[35869]: pgmap v9350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16538.063103 secs 2024-07-26T12:35:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:00 smithi097 ceph-mon[32231]: pgmap v9350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16538.063103 secs 2024-07-26T12:35:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:02 smithi115 ceph-mon[35364]: pgmap v9351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:03.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:02 smithi181 ceph-mon[35869]: pgmap v9351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:02 smithi097 ceph-mon[32231]: pgmap v9351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:04 smithi181 ceph-mon[35869]: pgmap v9352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:04 smithi097 ceph-mon[32231]: pgmap v9352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:04 smithi115 ceph-mon[35364]: pgmap v9352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16543.063288 secs 2024-07-26T12:35:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16543.063288 secs 2024-07-26T12:35:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16543.063288 secs 2024-07-26T12:35:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:06 smithi097 ceph-mon[32231]: pgmap v9353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:06 smithi115 ceph-mon[35364]: pgmap v9353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:06 smithi181 ceph-mon[35869]: pgmap v9353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:08 smithi097 ceph-mon[32231]: pgmap v9354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:09.373+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:09.374+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:35:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:09.374+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:35:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:09.374+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:09.374+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:08 smithi115 ceph-mon[35364]: pgmap v9354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:08 smithi181 ceph-mon[35869]: pgmap v9354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:11 smithi115 ceph-mon[35364]: pgmap v9355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16548.063422 secs 2024-07-26T12:35:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:11 smithi181 ceph-mon[35869]: pgmap v9355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16548.063422 secs 2024-07-26T12:35:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:11 smithi097 ceph-mon[32231]: pgmap v9355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16548.063422 secs 2024-07-26T12:35:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:13 smithi115 ceph-mon[35364]: pgmap v9356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:13 smithi181 ceph-mon[35869]: pgmap v9356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:13 smithi097 ceph-mon[32231]: pgmap v9356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:14 smithi115 ceph-mon[35364]: pgmap v9357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:35:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:14 smithi115 ceph-mon[35364]: pgmap v9358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:35:14.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:35:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:14 smithi181 ceph-mon[35869]: pgmap v9357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:35:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:14 smithi181 ceph-mon[35869]: pgmap v9358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:35:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:35:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:14 smithi097 ceph-mon[32231]: pgmap v9357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:35:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:14 smithi097 ceph-mon[32231]: pgmap v9358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:35:14.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:35:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16553.063564 secs 2024-07-26T12:35:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:16 smithi115 ceph-mon[35364]: pgmap v9359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:35:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16553.063564 secs 2024-07-26T12:35:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:16 smithi181 ceph-mon[35869]: pgmap v9359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:35:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16553.063564 secs 2024-07-26T12:35:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:16 smithi097 ceph-mon[32231]: pgmap v9359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:35:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:18 smithi115 ceph-mon[35364]: pgmap v9360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:18 smithi181 ceph-mon[35869]: pgmap v9360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:18 smithi097 ceph-mon[32231]: pgmap v9360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:20 smithi115 ceph-mon[35364]: pgmap v9361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:20 smithi181 ceph-mon[35869]: pgmap v9361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:20 smithi097 ceph-mon[32231]: pgmap v9361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16558.063710 secs 2024-07-26T12:35:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16558.063710 secs 2024-07-26T12:35:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16558.063710 secs 2024-07-26T12:35:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:22 smithi115 ceph-mon[35364]: pgmap v9362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:22 smithi181 ceph-mon[35869]: pgmap v9362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:22 smithi097 ceph-mon[32231]: pgmap v9362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:24.377+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:24.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:35:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:24.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:35:24.380 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:24.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:24.380 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:24.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:24 smithi115 ceph-mon[35364]: pgmap v9363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:35:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:24 smithi181 ceph-mon[35869]: pgmap v9363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:35:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:24 smithi097 ceph-mon[32231]: pgmap v9363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:35:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16563.063877 secs 2024-07-26T12:35:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:26 smithi115 ceph-mon[35364]: pgmap v9364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:35:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16563.063877 secs 2024-07-26T12:35:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:26 smithi181 ceph-mon[35869]: pgmap v9364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:35:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16563.063877 secs 2024-07-26T12:35:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:26 smithi097 ceph-mon[32231]: pgmap v9364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:35:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:28 smithi115 ceph-mon[35364]: pgmap v9365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:28 smithi181 ceph-mon[35869]: pgmap v9365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:28 smithi097 ceph-mon[32231]: pgmap v9365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:30 smithi115 ceph-mon[35364]: pgmap v9366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:35:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:30 smithi181 ceph-mon[35869]: pgmap v9366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:35:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:30 smithi097 ceph-mon[32231]: pgmap v9366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:35:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16568.064037 secs 2024-07-26T12:35:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16568.064037 secs 2024-07-26T12:35:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16568.064037 secs 2024-07-26T12:35:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:32 smithi115 ceph-mon[35364]: pgmap v9367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:35:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:32 smithi181 ceph-mon[35869]: pgmap v9367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:35:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:32 smithi097 ceph-mon[32231]: pgmap v9367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:35:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:34 smithi115 ceph-mon[35364]: pgmap v9368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:34 smithi181 ceph-mon[35869]: pgmap v9368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:34 smithi097 ceph-mon[32231]: pgmap v9368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16573.064210 secs 2024-07-26T12:35:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:36 smithi115 ceph-mon[35364]: pgmap v9369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16573.064210 secs 2024-07-26T12:35:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:36 smithi181 ceph-mon[35869]: pgmap v9369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16573.064210 secs 2024-07-26T12:35:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:36 smithi097 ceph-mon[32231]: pgmap v9369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:38 smithi115 ceph-mon[35364]: pgmap v9370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:38 smithi181 ceph-mon[35869]: pgmap v9370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:38 smithi097 ceph-mon[32231]: pgmap v9370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:39.384+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:39.385+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:35:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:39.385+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:35:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:39.385+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:39.385+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:40 smithi115 ceph-mon[35364]: pgmap v9371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:40 smithi181 ceph-mon[35869]: pgmap v9371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:40 smithi097 ceph-mon[32231]: pgmap v9371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16578.064347 secs 2024-07-26T12:35:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16578.064347 secs 2024-07-26T12:35:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16578.064347 secs 2024-07-26T12:35:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:42 smithi115 ceph-mon[35364]: pgmap v9372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:42 smithi181 ceph-mon[35869]: pgmap v9372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:42 smithi097 ceph-mon[32231]: pgmap v9372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:44 smithi115 ceph-mon[35364]: pgmap v9373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:44 smithi181 ceph-mon[35869]: pgmap v9373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:44 smithi097 ceph-mon[32231]: pgmap v9373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16583.064487 secs 2024-07-26T12:35:46.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:46 smithi115 ceph-mon[35364]: pgmap v9374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16583.064487 secs 2024-07-26T12:35:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:46 smithi181 ceph-mon[35869]: pgmap v9374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16583.064487 secs 2024-07-26T12:35:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:46 smithi097 ceph-mon[32231]: pgmap v9374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:48 smithi115 ceph-mon[35364]: pgmap v9375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:48 smithi181 ceph-mon[35869]: pgmap v9375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:48 smithi097 ceph-mon[32231]: pgmap v9375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:50 smithi115 ceph-mon[35364]: pgmap v9376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:50 smithi181 ceph-mon[35869]: pgmap v9376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:50 smithi097 ceph-mon[32231]: pgmap v9376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16588.064588 secs 2024-07-26T12:35:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16588.064588 secs 2024-07-26T12:35:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16588.064588 secs 2024-07-26T12:35:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:52 smithi115 ceph-mon[35364]: pgmap v9377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:52 smithi181 ceph-mon[35869]: pgmap v9377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:52 smithi097 ceph-mon[32231]: pgmap v9377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:54.387+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:54.388+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:54.388+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:54.388+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:35:54.389+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:54 smithi115 ceph-mon[35364]: pgmap v9378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:54 smithi181 ceph-mon[35869]: pgmap v9378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:54 smithi097 ceph-mon[32231]: pgmap v9378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16593.064716 secs 2024-07-26T12:35:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:56 smithi115 ceph-mon[35364]: pgmap v9379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16593.064716 secs 2024-07-26T12:35:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:56 smithi181 ceph-mon[35869]: pgmap v9379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16593.064716 secs 2024-07-26T12:35:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:56 smithi097 ceph-mon[32231]: pgmap v9379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:35:58 smithi115 ceph-mon[35364]: pgmap v9380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:35:58 smithi181 ceph-mon[35869]: pgmap v9380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:35:58 smithi097 ceph-mon[32231]: pgmap v9380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:00 smithi115 ceph-mon[35364]: pgmap v9381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:00 smithi181 ceph-mon[35869]: pgmap v9381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:00 smithi097 ceph-mon[32231]: pgmap v9381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16598.064861 secs 2024-07-26T12:36:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16598.064861 secs 2024-07-26T12:36:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16598.064861 secs 2024-07-26T12:36:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:02 smithi115 ceph-mon[35364]: pgmap v9382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:02 smithi181 ceph-mon[35869]: pgmap v9382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:02 smithi097 ceph-mon[32231]: pgmap v9382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:04 smithi181 ceph-mon[35869]: pgmap v9383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:04 smithi097 ceph-mon[32231]: pgmap v9383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:04 smithi115 ceph-mon[35364]: pgmap v9383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16603.065003 secs 2024-07-26T12:36:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:06 smithi097 ceph-mon[32231]: pgmap v9384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16603.065003 secs 2024-07-26T12:36:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:06 smithi115 ceph-mon[35364]: pgmap v9384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16603.065003 secs 2024-07-26T12:36:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:06 smithi181 ceph-mon[35869]: pgmap v9384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:08 smithi097 ceph-mon[32231]: pgmap v9385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:08 smithi115 ceph-mon[35364]: pgmap v9385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:08 smithi181 ceph-mon[35869]: pgmap v9385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:09.396+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:09.397+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:36:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:09.397+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:36:09.399 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:09.397+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:36:09.399 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:09.397+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:10 smithi097 ceph-mon[32231]: pgmap v9386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:10 smithi115 ceph-mon[35364]: pgmap v9386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:10 smithi181 ceph-mon[35869]: pgmap v9386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16608.065163 secs 2024-07-26T12:36:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16608.065163 secs 2024-07-26T12:36:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16608.065163 secs 2024-07-26T12:36:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:12 smithi115 ceph-mon[35364]: pgmap v9387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:12 smithi181 ceph-mon[35869]: pgmap v9387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:12 smithi097 ceph-mon[32231]: pgmap v9387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:14 smithi115 ceph-mon[35364]: pgmap v9388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:14 smithi181 ceph-mon[35869]: pgmap v9388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:14 smithi097 ceph-mon[32231]: pgmap v9388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: pgmap v9389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: pgmap v9390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:15.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:36:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: pgmap v9389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: pgmap v9390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:15.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:36:16.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:36:16.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:36:16.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:36:16.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:36:16.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:16.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:16.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: pgmap v9389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:36:16.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: pgmap v9390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:16.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:36:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16613.065304 secs 2024-07-26T12:36:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16613.065304 secs 2024-07-26T12:36:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16613.065304 secs 2024-07-26T12:36:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:17 smithi115 ceph-mon[35364]: pgmap v9391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:17 smithi181 ceph-mon[35869]: pgmap v9391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:17 smithi097 ceph-mon[32231]: pgmap v9391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:19 smithi115 ceph-mon[35364]: pgmap v9392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:19 smithi181 ceph-mon[35869]: pgmap v9392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:19 smithi097 ceph-mon[32231]: pgmap v9392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16618.065442 secs 2024-07-26T12:36:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:21 smithi115 ceph-mon[35364]: pgmap v9393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:36:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16618.065442 secs 2024-07-26T12:36:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:21 smithi181 ceph-mon[35869]: pgmap v9393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:36:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16618.065442 secs 2024-07-26T12:36:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:21 smithi097 ceph-mon[32231]: pgmap v9393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:36:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:22 smithi115 ceph-mon[35364]: pgmap v9394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:22 smithi181 ceph-mon[35869]: pgmap v9394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:22 smithi097 ceph-mon[32231]: pgmap v9394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:24.400+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:24.401+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:36:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:24.401+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:36:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:24.401+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:36:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:24.401+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:25 smithi115 ceph-mon[35364]: pgmap v9395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:25 smithi181 ceph-mon[35869]: pgmap v9395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:25 smithi097 ceph-mon[32231]: pgmap v9395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16623.065593 secs 2024-07-26T12:36:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16623.065593 secs 2024-07-26T12:36:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16623.065593 secs 2024-07-26T12:36:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:27 smithi115 ceph-mon[35364]: pgmap v9396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:27 smithi181 ceph-mon[35869]: pgmap v9396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:27 smithi097 ceph-mon[32231]: pgmap v9396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:29 smithi115 ceph-mon[35364]: pgmap v9397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:29 smithi181 ceph-mon[35869]: pgmap v9397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:29 smithi097 ceph-mon[32231]: pgmap v9397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16628.065737 secs 2024-07-26T12:36:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:31 smithi115 ceph-mon[35364]: pgmap v9398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16628.065737 secs 2024-07-26T12:36:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:31 smithi181 ceph-mon[35869]: pgmap v9398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16628.065737 secs 2024-07-26T12:36:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:31 smithi097 ceph-mon[32231]: pgmap v9398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:33 smithi115 ceph-mon[35364]: pgmap v9399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:33 smithi181 ceph-mon[35869]: pgmap v9399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:33 smithi097 ceph-mon[32231]: pgmap v9399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:35 smithi181 ceph-mon[35869]: pgmap v9400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:35 smithi097 ceph-mon[32231]: pgmap v9400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:35 smithi115 ceph-mon[35364]: pgmap v9400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16633.065922 secs 2024-07-26T12:36:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16633.065922 secs 2024-07-26T12:36:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16633.065922 secs 2024-07-26T12:36:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:37 smithi097 ceph-mon[32231]: pgmap v9401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:37 smithi115 ceph-mon[35364]: pgmap v9401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:37 smithi181 ceph-mon[35869]: pgmap v9401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:39.405+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:39.406+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:36:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:39.406+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:36:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:39.406+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:36:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:39.406+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:39 smithi115 ceph-mon[35364]: pgmap v9402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:39 smithi181 ceph-mon[35869]: pgmap v9402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:39 smithi097 ceph-mon[32231]: pgmap v9402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16638.066075 secs 2024-07-26T12:36:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:41 smithi115 ceph-mon[35364]: pgmap v9403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16638.066075 secs 2024-07-26T12:36:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:41 smithi181 ceph-mon[35869]: pgmap v9403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16638.066075 secs 2024-07-26T12:36:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:41 smithi097 ceph-mon[32231]: pgmap v9403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:43 smithi115 ceph-mon[35364]: pgmap v9404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:43 smithi181 ceph-mon[35869]: pgmap v9404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:43 smithi097 ceph-mon[32231]: pgmap v9404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:45 smithi115 ceph-mon[35364]: pgmap v9405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:45 smithi181 ceph-mon[35869]: pgmap v9405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:45 smithi097 ceph-mon[32231]: pgmap v9405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16643.066280 secs 2024-07-26T12:36:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16643.066280 secs 2024-07-26T12:36:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16643.066280 secs 2024-07-26T12:36:48.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:47 smithi115 ceph-mon[35364]: pgmap v9406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:47 smithi181 ceph-mon[35869]: pgmap v9406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:47 smithi097 ceph-mon[32231]: pgmap v9406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:48 smithi115 ceph-mon[35364]: pgmap v9407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:48 smithi181 ceph-mon[35869]: pgmap v9407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:48 smithi097 ceph-mon[32231]: pgmap v9407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16648.066377 secs 2024-07-26T12:36:51.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:51 smithi115 ceph-mon[35364]: pgmap v9408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16648.066377 secs 2024-07-26T12:36:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:51 smithi181 ceph-mon[35869]: pgmap v9408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16648.066377 secs 2024-07-26T12:36:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:51 smithi097 ceph-mon[32231]: pgmap v9408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:53 smithi115 ceph-mon[35364]: pgmap v9409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:53 smithi181 ceph-mon[35869]: pgmap v9409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:53 smithi097 ceph-mon[32231]: pgmap v9409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:36:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:54.408+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:54.409+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:36:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:54.409+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:36:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:54.409+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:36:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:36:54.410+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:55 smithi115 ceph-mon[35364]: pgmap v9410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:55 smithi181 ceph-mon[35869]: pgmap v9410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:55 smithi097 ceph-mon[32231]: pgmap v9410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16653.066475 secs 2024-07-26T12:36:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16653.066475 secs 2024-07-26T12:36:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16653.066475 secs 2024-07-26T12:36:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:57 smithi181 ceph-mon[35869]: pgmap v9411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:57 smithi097 ceph-mon[32231]: pgmap v9411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:57 smithi115 ceph-mon[35364]: pgmap v9411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:36:59 smithi097 ceph-mon[32231]: pgmap v9412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:36:59 smithi115 ceph-mon[35364]: pgmap v9412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:36:59 smithi181 ceph-mon[35869]: pgmap v9412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16658.066656 secs 2024-07-26T12:37:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:01 smithi097 ceph-mon[32231]: pgmap v9413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16658.066656 secs 2024-07-26T12:37:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:01 smithi115 ceph-mon[35364]: pgmap v9413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16658.066656 secs 2024-07-26T12:37:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:01 smithi181 ceph-mon[35869]: pgmap v9413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:02 smithi115 ceph-mon[35364]: pgmap v9414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:02 smithi181 ceph-mon[35869]: pgmap v9414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:02 smithi097 ceph-mon[32231]: pgmap v9414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:05 smithi115 ceph-mon[35364]: pgmap v9415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:05 smithi181 ceph-mon[35869]: pgmap v9415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:05 smithi097 ceph-mon[32231]: pgmap v9415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16663.066757 secs 2024-07-26T12:37:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16663.066757 secs 2024-07-26T12:37:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16663.066757 secs 2024-07-26T12:37:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:07 smithi115 ceph-mon[35364]: pgmap v9416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:07 smithi181 ceph-mon[35869]: pgmap v9416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:07 smithi097 ceph-mon[32231]: pgmap v9416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:09.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.418+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.419+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:37:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.419+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:37:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.419+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:37:09.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.419+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.865+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:37:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.865+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:37:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.866+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:37:09.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:09.866+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:37:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:09 smithi181 ceph-mon[35869]: pgmap v9417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:09 smithi097 ceph-mon[32231]: pgmap v9417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:09 smithi115 ceph-mon[35364]: pgmap v9417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16668.066862 secs 2024-07-26T12:37:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:11 smithi097 ceph-mon[32231]: pgmap v9418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16668.066862 secs 2024-07-26T12:37:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:11 smithi115 ceph-mon[35364]: pgmap v9418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16668.066862 secs 2024-07-26T12:37:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:11 smithi181 ceph-mon[35869]: pgmap v9418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:13 smithi097 ceph-mon[32231]: pgmap v9419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:13 smithi115 ceph-mon[35364]: pgmap v9419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:13 smithi181 ceph-mon[35869]: pgmap v9419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:15 smithi097 ceph-mon[32231]: pgmap v9420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:15 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:15 smithi115 ceph-mon[35364]: pgmap v9420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:15 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:15 smithi181 ceph-mon[35869]: pgmap v9420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:15 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16673.066995 secs 2024-07-26T12:37:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:17.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16673.066995 secs 2024-07-26T12:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:37:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16673.066995 secs 2024-07-26T12:37:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:17.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:37:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:17 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:37:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:17 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:37:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:17 smithi115 ceph-mon[35364]: pgmap v9421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:37:18.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:17 smithi115 ceph-mon[35364]: pgmap v9422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:37:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:17 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:37:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:17 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:37:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:17 smithi181 ceph-mon[35869]: pgmap v9421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:37:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:17 smithi181 ceph-mon[35869]: pgmap v9422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:17 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:17 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:17 smithi097 ceph-mon[32231]: pgmap v9421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:37:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:17 smithi097 ceph-mon[32231]: pgmap v9422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:37:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:19 smithi115 ceph-mon[35364]: pgmap v9423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:37:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:19 smithi181 ceph-mon[35869]: pgmap v9423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:37:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:19 smithi097 ceph-mon[32231]: pgmap v9423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:37:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:20 smithi115 ceph-mon[35364]: pgmap v9424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:37:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:20 smithi181 ceph-mon[35869]: pgmap v9424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:37:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:20 smithi097 ceph-mon[32231]: pgmap v9424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:37:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16678.067163 secs 2024-07-26T12:37:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16678.067163 secs 2024-07-26T12:37:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16678.067163 secs 2024-07-26T12:37:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:22 smithi115 ceph-mon[35364]: pgmap v9425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:37:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:22 smithi181 ceph-mon[35869]: pgmap v9425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:37:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:22 smithi097 ceph-mon[32231]: pgmap v9425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:37:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:24.422+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:24.423+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:37:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:24.423+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:37:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:24.423+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:37:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:24.423+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:25 smithi115 ceph-mon[35364]: pgmap v9426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:25 smithi181 ceph-mon[35869]: pgmap v9426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:25 smithi097 ceph-mon[32231]: pgmap v9426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:26.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16683.067322 secs 2024-07-26T12:37:26.385 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16683.067322 secs 2024-07-26T12:37:26.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16683.067322 secs 2024-07-26T12:37:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:27 smithi115 ceph-mon[35364]: pgmap v9427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:37:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:27 smithi181 ceph-mon[35869]: pgmap v9427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:37:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:27 smithi097 ceph-mon[32231]: pgmap v9427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:37:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:29 smithi115 ceph-mon[35364]: pgmap v9428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:29 smithi181 ceph-mon[35869]: pgmap v9428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:29 smithi097 ceph-mon[32231]: pgmap v9428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:31 smithi115 ceph-mon[35364]: pgmap v9429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:31.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16688.067479 secs 2024-07-26T12:37:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:31 smithi181 ceph-mon[35869]: pgmap v9429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16688.067479 secs 2024-07-26T12:37:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:31 smithi097 ceph-mon[32231]: pgmap v9429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16688.067479 secs 2024-07-26T12:37:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:33 smithi115 ceph-mon[35364]: pgmap v9430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:33 smithi181 ceph-mon[35869]: pgmap v9430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:33 smithi097 ceph-mon[32231]: pgmap v9430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:35 smithi115 ceph-mon[35364]: pgmap v9431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:35 smithi181 ceph-mon[35869]: pgmap v9431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:35 smithi097 ceph-mon[32231]: pgmap v9431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16693.067629 secs 2024-07-26T12:37:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16693.067629 secs 2024-07-26T12:37:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16693.067629 secs 2024-07-26T12:37:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:37 smithi115 ceph-mon[35364]: pgmap v9432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:37 smithi181 ceph-mon[35869]: pgmap v9432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:37 smithi097 ceph-mon[32231]: pgmap v9432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:39.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:39.425+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:39.426+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:37:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:39.426+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:37:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:39.426+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:37:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:39.426+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:39 smithi115 ceph-mon[35364]: pgmap v9433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:39 smithi181 ceph-mon[35869]: pgmap v9433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:39 smithi097 ceph-mon[32231]: pgmap v9433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:41 smithi115 ceph-mon[35364]: pgmap v9434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16698.067786 secs 2024-07-26T12:37:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:41 smithi181 ceph-mon[35869]: pgmap v9434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16698.067786 secs 2024-07-26T12:37:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:41 smithi097 ceph-mon[32231]: pgmap v9434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16698.067786 secs 2024-07-26T12:37:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:43 smithi115 ceph-mon[35364]: pgmap v9435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:43 smithi181 ceph-mon[35869]: pgmap v9435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:43.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:43 smithi097 ceph-mon[32231]: pgmap v9435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:45 smithi115 ceph-mon[35364]: pgmap v9436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:45 smithi181 ceph-mon[35869]: pgmap v9436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:45 smithi097 ceph-mon[32231]: pgmap v9436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16703.067936 secs 2024-07-26T12:37:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16703.067936 secs 2024-07-26T12:37:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16703.067936 secs 2024-07-26T12:37:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:47 smithi115 ceph-mon[35364]: pgmap v9437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:47 smithi181 ceph-mon[35869]: pgmap v9437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:47 smithi097 ceph-mon[32231]: pgmap v9437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:49 smithi115 ceph-mon[35364]: pgmap v9438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:49 smithi181 ceph-mon[35869]: pgmap v9438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:49 smithi097 ceph-mon[32231]: pgmap v9438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:51 smithi115 ceph-mon[35364]: pgmap v9439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16708.068086 secs 2024-07-26T12:37:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:51 smithi181 ceph-mon[35869]: pgmap v9439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16708.068086 secs 2024-07-26T12:37:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:51 smithi097 ceph-mon[32231]: pgmap v9439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16708.068086 secs 2024-07-26T12:37:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:53 smithi115 ceph-mon[35364]: pgmap v9440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:53 smithi181 ceph-mon[35869]: pgmap v9440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:53 smithi097 ceph-mon[32231]: pgmap v9440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:54.428+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:54.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:54.429+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:37:54.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:54.429+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:37:54.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:54.429+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:37:54.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:37:54.429+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:55.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:55 smithi181 ceph-mon[35869]: pgmap v9441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:55 smithi097 ceph-mon[32231]: pgmap v9441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:55 smithi115 ceph-mon[35364]: pgmap v9441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16713.068305 secs 2024-07-26T12:37:56.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16713.068305 secs 2024-07-26T12:37:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16713.068305 secs 2024-07-26T12:37:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:57 smithi097 ceph-mon[32231]: pgmap v9442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:57 smithi115 ceph-mon[35364]: pgmap v9442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:57 smithi181 ceph-mon[35869]: pgmap v9442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:37:59 smithi115 ceph-mon[35364]: pgmap v9443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:37:59 smithi181 ceph-mon[35869]: pgmap v9443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:37:59 smithi097 ceph-mon[32231]: pgmap v9443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:01 smithi115 ceph-mon[35364]: pgmap v9444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16718.068463 secs 2024-07-26T12:38:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:01 smithi181 ceph-mon[35869]: pgmap v9444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16718.068463 secs 2024-07-26T12:38:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:01 smithi097 ceph-mon[32231]: pgmap v9444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16718.068463 secs 2024-07-26T12:38:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:03 smithi115 ceph-mon[35364]: pgmap v9445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:03 smithi181 ceph-mon[35869]: pgmap v9445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:03.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:03 smithi097 ceph-mon[32231]: pgmap v9445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:05 smithi115 ceph-mon[35364]: pgmap v9446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:05 smithi181 ceph-mon[35869]: pgmap v9446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:05 smithi097 ceph-mon[32231]: pgmap v9446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16723.068619 secs 2024-07-26T12:38:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16723.068619 secs 2024-07-26T12:38:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16723.068619 secs 2024-07-26T12:38:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:07 smithi115 ceph-mon[35364]: pgmap v9447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:07 smithi181 ceph-mon[35869]: pgmap v9447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:07 smithi097 ceph-mon[32231]: pgmap v9447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:09.437+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:09.438+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:38:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:09.438+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:38:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:09.438+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:38:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:09.438+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:09 smithi115 ceph-mon[35364]: pgmap v9448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:09 smithi181 ceph-mon[35869]: pgmap v9448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:09 smithi097 ceph-mon[32231]: pgmap v9448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:11 smithi115 ceph-mon[35364]: pgmap v9449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16728.068799 secs 2024-07-26T12:38:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:11 smithi181 ceph-mon[35869]: pgmap v9449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16728.068799 secs 2024-07-26T12:38:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:11 smithi097 ceph-mon[32231]: pgmap v9449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16728.068799 secs 2024-07-26T12:38:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:13 smithi115 ceph-mon[35364]: pgmap v9450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:13 smithi181 ceph-mon[35869]: pgmap v9450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:13 smithi097 ceph-mon[32231]: pgmap v9450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:15 smithi115 ceph-mon[35364]: pgmap v9451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:15 smithi181 ceph-mon[35869]: pgmap v9451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:15 smithi097 ceph-mon[32231]: pgmap v9451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16733.068959 secs 2024-07-26T12:38:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16733.068959 secs 2024-07-26T12:38:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16733.068959 secs 2024-07-26T12:38:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:17 smithi115 ceph-mon[35364]: pgmap v9452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:17.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:17 smithi181 ceph-mon[35869]: pgmap v9452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:17 smithi097 ceph-mon[32231]: pgmap v9452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:19 smithi115 ceph-mon[35364]: pgmap v9453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:19 smithi115 ceph-mon[35364]: pgmap v9454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:38:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:19 smithi115 ceph-mon[35364]: pgmap v9455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:19.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:19 smithi181 ceph-mon[35869]: pgmap v9453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:19 smithi181 ceph-mon[35869]: pgmap v9454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:19 smithi181 ceph-mon[35869]: pgmap v9455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:19 smithi097 ceph-mon[32231]: pgmap v9453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:19 smithi097 ceph-mon[32231]: pgmap v9454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:38:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:19 smithi097 ceph-mon[32231]: pgmap v9455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:19.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:38:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16738.069116 secs 2024-07-26T12:38:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:21 smithi115 ceph-mon[35364]: pgmap v9456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:38:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16738.069116 secs 2024-07-26T12:38:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:21 smithi181 ceph-mon[35869]: pgmap v9456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:38:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16738.069116 secs 2024-07-26T12:38:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:21 smithi097 ceph-mon[32231]: pgmap v9456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:38:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:23 smithi115 ceph-mon[35364]: pgmap v9457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:23 smithi181 ceph-mon[35869]: pgmap v9457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:23 smithi097 ceph-mon[32231]: pgmap v9457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:24.440+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:24.442+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:38:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:24.442+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:38:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:24.442+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:38:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:24.442+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:25 smithi115 ceph-mon[35364]: pgmap v9458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:25 smithi181 ceph-mon[35869]: pgmap v9458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:25 smithi097 ceph-mon[32231]: pgmap v9458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:26.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16743.069293 secs 2024-07-26T12:38:26.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16743.069293 secs 2024-07-26T12:38:26.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16743.069293 secs 2024-07-26T12:38:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:27 smithi115 ceph-mon[35364]: pgmap v9459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:38:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:27 smithi181 ceph-mon[35869]: pgmap v9459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:38:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:27 smithi097 ceph-mon[32231]: pgmap v9459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:38:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:29 smithi115 ceph-mon[35364]: pgmap v9460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:38:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:29 smithi181 ceph-mon[35869]: pgmap v9460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:38:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:29 smithi097 ceph-mon[32231]: pgmap v9460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:38:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16748.069389 secs 2024-07-26T12:38:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:31 smithi115 ceph-mon[35364]: pgmap v9461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16748.069389 secs 2024-07-26T12:38:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:31 smithi181 ceph-mon[35869]: pgmap v9461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16748.069389 secs 2024-07-26T12:38:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:31 smithi097 ceph-mon[32231]: pgmap v9461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:33 smithi115 ceph-mon[35364]: pgmap v9462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:33 smithi181 ceph-mon[35869]: pgmap v9462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:33 smithi097 ceph-mon[32231]: pgmap v9462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:35 smithi115 ceph-mon[35364]: pgmap v9463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:35 smithi181 ceph-mon[35869]: pgmap v9463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:35 smithi097 ceph-mon[32231]: pgmap v9463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16753.069490 secs 2024-07-26T12:38:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16753.069490 secs 2024-07-26T12:38:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16753.069490 secs 2024-07-26T12:38:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:37 smithi115 ceph-mon[35364]: pgmap v9464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:37 smithi181 ceph-mon[35869]: pgmap v9464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:37 smithi097 ceph-mon[32231]: pgmap v9464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.449+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.451+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:38:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.451+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:38:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.451+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:38:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.451+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:39.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.881+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:38:39.883 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.881+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:38:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.883+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:38:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:39.883+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:38:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:39 smithi115 ceph-mon[35364]: pgmap v9465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:39 smithi181 ceph-mon[35869]: pgmap v9465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:39 smithi097 ceph-mon[32231]: pgmap v9465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16758.069583 secs 2024-07-26T12:38:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:41 smithi115 ceph-mon[35364]: pgmap v9466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16758.069583 secs 2024-07-26T12:38:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:41 smithi181 ceph-mon[35869]: pgmap v9466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:42.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16758.069583 secs 2024-07-26T12:38:42.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:41 smithi097 ceph-mon[32231]: pgmap v9466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:43 smithi115 ceph-mon[35364]: pgmap v9467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:43 smithi181 ceph-mon[35869]: pgmap v9467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:43 smithi097 ceph-mon[32231]: pgmap v9467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:45 smithi115 ceph-mon[35364]: pgmap v9468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:45 smithi181 ceph-mon[35869]: pgmap v9468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:45 smithi097 ceph-mon[32231]: pgmap v9468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16763.069725 secs 2024-07-26T12:38:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16763.069725 secs 2024-07-26T12:38:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16763.069725 secs 2024-07-26T12:38:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:47 smithi115 ceph-mon[35364]: pgmap v9469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:47 smithi181 ceph-mon[35869]: pgmap v9469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:47 smithi097 ceph-mon[32231]: pgmap v9469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:49 smithi115 ceph-mon[35364]: pgmap v9470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:49 smithi181 ceph-mon[35869]: pgmap v9470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:49 smithi097 ceph-mon[32231]: pgmap v9470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16768.069812 secs 2024-07-26T12:38:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:51 smithi115 ceph-mon[35364]: pgmap v9471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16768.069812 secs 2024-07-26T12:38:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:51 smithi181 ceph-mon[35869]: pgmap v9471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16768.069812 secs 2024-07-26T12:38:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:51 smithi097 ceph-mon[32231]: pgmap v9471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:53 smithi115 ceph-mon[35364]: pgmap v9472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:53 smithi181 ceph-mon[35869]: pgmap v9472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:53 smithi097 ceph-mon[32231]: pgmap v9472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:54.455 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:54.453+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:54.455+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:54.455+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:54.455+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:38:54.455+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:55 smithi115 ceph-mon[35364]: pgmap v9473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:55 smithi181 ceph-mon[35869]: pgmap v9473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:55 smithi097 ceph-mon[32231]: pgmap v9473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16773.069952 secs 2024-07-26T12:38:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16773.069952 secs 2024-07-26T12:38:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16773.069952 secs 2024-07-26T12:38:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:57 smithi115 ceph-mon[35364]: pgmap v9474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:57 smithi181 ceph-mon[35869]: pgmap v9474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:57 smithi097 ceph-mon[32231]: pgmap v9474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:38:59 smithi115 ceph-mon[35364]: pgmap v9475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:38:59 smithi181 ceph-mon[35869]: pgmap v9475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:38:59 smithi097 ceph-mon[32231]: pgmap v9475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16778.070116 secs 2024-07-26T12:39:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:01 smithi181 ceph-mon[35869]: pgmap v9476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16778.070116 secs 2024-07-26T12:39:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:01 smithi097 ceph-mon[32231]: pgmap v9476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16778.070116 secs 2024-07-26T12:39:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:01 smithi115 ceph-mon[35364]: pgmap v9476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:03 smithi181 ceph-mon[35869]: pgmap v9477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:03 smithi097 ceph-mon[32231]: pgmap v9477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:03 smithi115 ceph-mon[35364]: pgmap v9477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:05 smithi097 ceph-mon[32231]: pgmap v9478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:05 smithi115 ceph-mon[35364]: pgmap v9478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:05 smithi181 ceph-mon[35869]: pgmap v9478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16783.070304 secs 2024-07-26T12:39:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16783.070304 secs 2024-07-26T12:39:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16783.070304 secs 2024-07-26T12:39:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:07 smithi115 ceph-mon[35364]: pgmap v9479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:07 smithi181 ceph-mon[35869]: pgmap v9479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:07 smithi097 ceph-mon[32231]: pgmap v9479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:09.463+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:09.464+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:39:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:09.464+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:39:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:09.464+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:39:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:09.464+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:09 smithi115 ceph-mon[35364]: pgmap v9480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:09 smithi181 ceph-mon[35869]: pgmap v9480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:09 smithi097 ceph-mon[32231]: pgmap v9480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16788.070456 secs 2024-07-26T12:39:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:11 smithi115 ceph-mon[35364]: pgmap v9481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16788.070456 secs 2024-07-26T12:39:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:11 smithi181 ceph-mon[35869]: pgmap v9481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16788.070456 secs 2024-07-26T12:39:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:11 smithi097 ceph-mon[32231]: pgmap v9481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:13 smithi115 ceph-mon[35364]: pgmap v9482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:13 smithi181 ceph-mon[35869]: pgmap v9482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:13 smithi097 ceph-mon[32231]: pgmap v9482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:15 smithi115 ceph-mon[35364]: pgmap v9483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:15 smithi181 ceph-mon[35869]: pgmap v9483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:15 smithi097 ceph-mon[32231]: pgmap v9483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16793.070622 secs 2024-07-26T12:39:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16793.070622 secs 2024-07-26T12:39:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16793.070622 secs 2024-07-26T12:39:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:17 smithi115 ceph-mon[35364]: pgmap v9484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:17 smithi181 ceph-mon[35869]: pgmap v9484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:17 smithi097 ceph-mon[32231]: pgmap v9484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:19 smithi115 ceph-mon[35364]: pgmap v9485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:19 smithi181 ceph-mon[35869]: pgmap v9485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:19 smithi097 ceph-mon[32231]: pgmap v9485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:20.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16798.070734 secs 2024-07-26T12:39:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:20 smithi115 ceph-mon[35364]: pgmap v9486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16798.070734 secs 2024-07-26T12:39:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:20 smithi181 ceph-mon[35869]: pgmap v9486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16798.070734 secs 2024-07-26T12:39:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:20 smithi097 ceph-mon[32231]: pgmap v9486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:22 smithi115 ceph-mon[35364]: pgmap v9487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:39:22.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:22 smithi181 ceph-mon[35869]: pgmap v9487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:22 smithi097 ceph-mon[32231]: pgmap v9487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:39:22.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:39:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:24.467+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:24.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:24.468+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:39:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:24.468+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:39:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:24.468+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:39:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:24.468+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:24 smithi115 ceph-mon[35364]: pgmap v9488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:24 smithi181 ceph-mon[35869]: pgmap v9488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:24 smithi097 ceph-mon[32231]: pgmap v9488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16803.070887 secs 2024-07-26T12:39:26.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:26 smithi115 ceph-mon[35364]: pgmap v9489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:39:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16803.070887 secs 2024-07-26T12:39:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:26 smithi181 ceph-mon[35869]: pgmap v9489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:39:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16803.070887 secs 2024-07-26T12:39:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:26 smithi097 ceph-mon[32231]: pgmap v9489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:39:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:27 smithi115 ceph-mon[35364]: pgmap v9490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:27 smithi181 ceph-mon[35869]: pgmap v9490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:27 smithi097 ceph-mon[32231]: pgmap v9490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:30 smithi115 ceph-mon[35364]: pgmap v9491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:39:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:30 smithi181 ceph-mon[35869]: pgmap v9491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:39:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:30 smithi097 ceph-mon[32231]: pgmap v9491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:39:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16808.070992 secs 2024-07-26T12:39:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16808.070992 secs 2024-07-26T12:39:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16808.070992 secs 2024-07-26T12:39:32.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:32 smithi115 ceph-mon[35364]: pgmap v9492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:39:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:32 smithi181 ceph-mon[35869]: pgmap v9492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:39:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:32 smithi097 ceph-mon[32231]: pgmap v9492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:39:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:34 smithi115 ceph-mon[35364]: pgmap v9493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:34 smithi181 ceph-mon[35869]: pgmap v9493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:34 smithi097 ceph-mon[32231]: pgmap v9493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16813.071152 secs 2024-07-26T12:39:36.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:36 smithi115 ceph-mon[35364]: pgmap v9494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16813.071152 secs 2024-07-26T12:39:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:36 smithi181 ceph-mon[35869]: pgmap v9494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16813.071152 secs 2024-07-26T12:39:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:36 smithi097 ceph-mon[32231]: pgmap v9494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:38 smithi115 ceph-mon[35364]: pgmap v9495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:38 smithi181 ceph-mon[35869]: pgmap v9495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:38 smithi097 ceph-mon[32231]: pgmap v9495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:39.470+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:39.471+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:39:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:39.471+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:39:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:39.471+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:39:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:39.471+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:40 smithi115 ceph-mon[35364]: pgmap v9496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:40 smithi181 ceph-mon[35869]: pgmap v9496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:40 smithi097 ceph-mon[32231]: pgmap v9496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16818.071307 secs 2024-07-26T12:39:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16818.071307 secs 2024-07-26T12:39:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16818.071307 secs 2024-07-26T12:39:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:42 smithi115 ceph-mon[35364]: pgmap v9497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:42 smithi181 ceph-mon[35869]: pgmap v9497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:42 smithi097 ceph-mon[32231]: pgmap v9497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:44.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:44 smithi181 ceph-mon[35869]: pgmap v9498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:39:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:44 smithi097 ceph-mon[32231]: pgmap v9498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:39:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:44 smithi115 ceph-mon[35364]: pgmap v9498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:39:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16823.071419 secs 2024-07-26T12:39:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:46 smithi097 ceph-mon[32231]: pgmap v9499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:39:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16823.071419 secs 2024-07-26T12:39:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:46 smithi115 ceph-mon[35364]: pgmap v9499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:39:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16823.071419 secs 2024-07-26T12:39:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:46 smithi181 ceph-mon[35869]: pgmap v9499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:39:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:48 smithi097 ceph-mon[32231]: pgmap v9500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:48 smithi115 ceph-mon[35364]: pgmap v9500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:48.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:48 smithi181 ceph-mon[35869]: pgmap v9500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:50 smithi097 ceph-mon[32231]: pgmap v9501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:39:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:50 smithi115 ceph-mon[35364]: pgmap v9501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:39:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:50 smithi181 ceph-mon[35869]: pgmap v9501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:39:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16828.071566 secs 2024-07-26T12:39:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16828.071566 secs 2024-07-26T12:39:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16828.071566 secs 2024-07-26T12:39:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:52 smithi115 ceph-mon[35364]: pgmap v9502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:52 smithi181 ceph-mon[35869]: pgmap v9502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:52 smithi097 ceph-mon[32231]: pgmap v9502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:54.474+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:54.475+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:39:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:54.475+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:39:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:54.475+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:39:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:39:54.475+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:54 smithi115 ceph-mon[35364]: pgmap v9503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:54 smithi181 ceph-mon[35869]: pgmap v9503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:54 smithi097 ceph-mon[32231]: pgmap v9503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:56.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16833.071724 secs 2024-07-26T12:39:56.528 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:56 smithi115 ceph-mon[35364]: pgmap v9504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:56.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16833.071724 secs 2024-07-26T12:39:56.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:56 smithi181 ceph-mon[35869]: pgmap v9504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:56.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16833.071724 secs 2024-07-26T12:39:56.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:56 smithi097 ceph-mon[32231]: pgmap v9504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:39:58 smithi115 ceph-mon[35364]: pgmap v9505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:39:58 smithi181 ceph-mon[35869]: pgmap v9505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:39:58 smithi097 ceph-mon[32231]: pgmap v9505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:00 smithi115 ceph-mon[35364]: pgmap v9506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:00 smithi181 ceph-mon[35869]: pgmap v9506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:00 smithi097 ceph-mon[32231]: pgmap v9506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16838.071898 secs 2024-07-26T12:40:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16838.071898 secs 2024-07-26T12:40:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16838.071898 secs 2024-07-26T12:40:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:02 smithi115 ceph-mon[35364]: pgmap v9507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:02 smithi181 ceph-mon[35869]: pgmap v9507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:02 smithi097 ceph-mon[32231]: pgmap v9507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:04 smithi115 ceph-mon[35364]: pgmap v9508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:04 smithi181 ceph-mon[35869]: pgmap v9508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:04 smithi097 ceph-mon[32231]: pgmap v9508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16843.072056 secs 2024-07-26T12:40:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:06 smithi115 ceph-mon[35364]: pgmap v9509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:40:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16843.072056 secs 2024-07-26T12:40:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:06 smithi181 ceph-mon[35869]: pgmap v9509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:40:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16843.072056 secs 2024-07-26T12:40:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:06 smithi097 ceph-mon[32231]: pgmap v9509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:40:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:08 smithi115 ceph-mon[35364]: pgmap v9510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:40:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:08 smithi181 ceph-mon[35869]: pgmap v9510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:40:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:08 smithi097 ceph-mon[32231]: pgmap v9510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:40:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.484+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:09.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.485+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:40:09.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.485+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:40:09.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.485+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:40:09.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.485+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.898+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:40:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.898+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:40:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.899+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:40:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:09.899+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:40:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:10 smithi115 ceph-mon[35364]: pgmap v9511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:40:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:10 smithi181 ceph-mon[35869]: pgmap v9511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:40:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:10 smithi097 ceph-mon[32231]: pgmap v9511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:40:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16848.072216 secs 2024-07-26T12:40:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16848.072216 secs 2024-07-26T12:40:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16848.072216 secs 2024-07-26T12:40:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:12 smithi115 ceph-mon[35364]: pgmap v9512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:12 smithi181 ceph-mon[35869]: pgmap v9512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:12 smithi097 ceph-mon[32231]: pgmap v9512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:14 smithi115 ceph-mon[35364]: pgmap v9513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:40:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:14 smithi181 ceph-mon[35869]: pgmap v9513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:40:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:14 smithi097 ceph-mon[32231]: pgmap v9513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:40:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16853.072371 secs 2024-07-26T12:40:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:16 smithi115 ceph-mon[35364]: pgmap v9514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:40:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16853.072371 secs 2024-07-26T12:40:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:16 smithi181 ceph-mon[35869]: pgmap v9514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:40:16.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16853.072371 secs 2024-07-26T12:40:16.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:16 smithi097 ceph-mon[32231]: pgmap v9514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:40:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:18 smithi115 ceph-mon[35364]: pgmap v9515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:18 smithi181 ceph-mon[35869]: pgmap v9515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:18 smithi097 ceph-mon[32231]: pgmap v9515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:20 smithi115 ceph-mon[35364]: pgmap v9516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:20 smithi181 ceph-mon[35869]: pgmap v9516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:20 smithi097 ceph-mon[32231]: pgmap v9516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16858.072522 secs 2024-07-26T12:40:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16858.072522 secs 2024-07-26T12:40:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16858.072522 secs 2024-07-26T12:40:22.704 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:22 smithi097 ceph-mon[32231]: pgmap v9517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:22.704 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:22 smithi115 ceph-mon[35364]: pgmap v9517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:22 smithi181 ceph-mon[35869]: pgmap v9517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:22.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:24.489 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:24.488+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:24.490 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:24.489+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:40:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:24.489+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:40:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:24.489+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:40:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:24.489+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:24 smithi115 ceph-mon[35364]: pgmap v9518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:24 smithi181 ceph-mon[35869]: pgmap v9518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:24 smithi097 ceph-mon[32231]: pgmap v9518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:25.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:26.682 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16863.072617 secs 2024-07-26T12:40:26.682 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:26 smithi097 ceph-mon[32231]: pgmap v9519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:26.683 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:26.683 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:26.683 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:26 smithi097 ceph-mon[32231]: pgmap v9520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:26.683 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:26 smithi097 ceph-mon[32231]: pgmap v9521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:26.683 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16863.072617 secs 2024-07-26T12:40:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:26 smithi115 ceph-mon[35364]: pgmap v9519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:26.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:26 smithi115 ceph-mon[35364]: pgmap v9520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:26.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:26 smithi115 ceph-mon[35364]: pgmap v9521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:26.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16863.072617 secs 2024-07-26T12:40:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:26 smithi181 ceph-mon[35869]: pgmap v9519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:26.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:26.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:26 smithi181 ceph-mon[35869]: pgmap v9520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:26.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:26 smithi181 ceph-mon[35869]: pgmap v9521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:26.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:40:28.218 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:27 smithi181 ceph-mon[35869]: pgmap v9522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:27 smithi115 ceph-mon[35364]: pgmap v9522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:27 smithi097 ceph-mon[32231]: pgmap v9522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:29 smithi115 ceph-mon[35364]: pgmap v9523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:29 smithi181 ceph-mon[35869]: pgmap v9523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:29 smithi097 ceph-mon[32231]: pgmap v9523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16868.072721 secs 2024-07-26T12:40:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16868.072721 secs 2024-07-26T12:40:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16868.072721 secs 2024-07-26T12:40:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:31 smithi115 ceph-mon[35364]: pgmap v9524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:40:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:31 smithi181 ceph-mon[35869]: pgmap v9524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:40:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:31 smithi097 ceph-mon[32231]: pgmap v9524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:40:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:33 smithi115 ceph-mon[35364]: pgmap v9525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:40:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:33 smithi181 ceph-mon[35869]: pgmap v9525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:40:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:33 smithi097 ceph-mon[32231]: pgmap v9525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:40:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16873.072875 secs 2024-07-26T12:40:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:35 smithi115 ceph-mon[35364]: pgmap v9526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16873.072875 secs 2024-07-26T12:40:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:35 smithi181 ceph-mon[35869]: pgmap v9526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16873.072875 secs 2024-07-26T12:40:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:35 smithi097 ceph-mon[32231]: pgmap v9526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:37 smithi115 ceph-mon[35364]: pgmap v9527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:37 smithi181 ceph-mon[35869]: pgmap v9527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:37 smithi097 ceph-mon[32231]: pgmap v9527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:39.496+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:39.497+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:40:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:39.497+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:40:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:39.497+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:40:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:39.497+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:39 smithi115 ceph-mon[35364]: pgmap v9528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:39 smithi181 ceph-mon[35869]: pgmap v9528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:40.255 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:39 smithi097 ceph-mon[32231]: pgmap v9528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16878.073040 secs 2024-07-26T12:40:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16878.073040 secs 2024-07-26T12:40:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16878.073040 secs 2024-07-26T12:40:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:41 smithi115 ceph-mon[35364]: pgmap v9529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:41 smithi181 ceph-mon[35869]: pgmap v9529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:41 smithi097 ceph-mon[32231]: pgmap v9529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:43 smithi115 ceph-mon[35364]: pgmap v9530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:43 smithi181 ceph-mon[35869]: pgmap v9530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:43 smithi097 ceph-mon[32231]: pgmap v9530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16883.073184 secs 2024-07-26T12:40:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:45 smithi115 ceph-mon[35364]: pgmap v9531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16883.073184 secs 2024-07-26T12:40:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:45 smithi181 ceph-mon[35869]: pgmap v9531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16883.073184 secs 2024-07-26T12:40:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:45 smithi097 ceph-mon[32231]: pgmap v9531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:47 smithi115 ceph-mon[35364]: pgmap v9532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:47 smithi181 ceph-mon[35869]: pgmap v9532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:47 smithi097 ceph-mon[32231]: pgmap v9532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:50 smithi115 ceph-mon[35364]: pgmap v9533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16888.073335 secs 2024-07-26T12:40:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:50 smithi181 ceph-mon[35869]: pgmap v9533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16888.073335 secs 2024-07-26T12:40:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:50 smithi097 ceph-mon[32231]: pgmap v9533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16888.073335 secs 2024-07-26T12:40:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:52 smithi115 ceph-mon[35364]: pgmap v9534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:40:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:52 smithi181 ceph-mon[35869]: pgmap v9534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:40:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:52 smithi097 ceph-mon[32231]: pgmap v9534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:40:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:54.500+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:54.501+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:40:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:54.501+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:54.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:40:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:40:54.501+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:54 smithi115 ceph-mon[35364]: pgmap v9535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:54 smithi181 ceph-mon[35869]: pgmap v9535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:54 smithi097 ceph-mon[32231]: pgmap v9535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16893.073450 secs 2024-07-26T12:40:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16893.073450 secs 2024-07-26T12:40:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16893.073450 secs 2024-07-26T12:40:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:56 smithi115 ceph-mon[35364]: pgmap v9536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:56 smithi181 ceph-mon[35869]: pgmap v9536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:56 smithi097 ceph-mon[32231]: pgmap v9536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:40:58 smithi115 ceph-mon[35364]: pgmap v9537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:40:58 smithi181 ceph-mon[35869]: pgmap v9537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:40:58 smithi097 ceph-mon[32231]: pgmap v9537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:00 smithi115 ceph-mon[35364]: pgmap v9538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16898.073558 secs 2024-07-26T12:41:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:00 smithi181 ceph-mon[35869]: pgmap v9538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16898.073558 secs 2024-07-26T12:41:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:00 smithi097 ceph-mon[32231]: pgmap v9538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16898.073558 secs 2024-07-26T12:41:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:02 smithi115 ceph-mon[35364]: pgmap v9539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:02 smithi181 ceph-mon[35869]: pgmap v9539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:02 smithi097 ceph-mon[32231]: pgmap v9539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:04 smithi115 ceph-mon[35364]: pgmap v9540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:04 smithi181 ceph-mon[35869]: pgmap v9540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:04 smithi097 ceph-mon[32231]: pgmap v9540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16903.073724 secs 2024-07-26T12:41:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16903.073724 secs 2024-07-26T12:41:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16903.073724 secs 2024-07-26T12:41:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:06 smithi115 ceph-mon[35364]: pgmap v9541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:06 smithi181 ceph-mon[35869]: pgmap v9541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:06 smithi097 ceph-mon[32231]: pgmap v9541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:08 smithi115 ceph-mon[35364]: pgmap v9542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:08 smithi181 ceph-mon[35869]: pgmap v9542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:08 smithi097 ceph-mon[32231]: pgmap v9542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:09.510 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:09.509+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:09.512 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:09.510+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:41:09.512 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:09.511+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:41:09.512 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:09.511+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:09.512 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:09.511+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:10 smithi115 ceph-mon[35364]: pgmap v9543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16908.073893 secs 2024-07-26T12:41:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:10 smithi181 ceph-mon[35869]: pgmap v9543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16908.073893 secs 2024-07-26T12:41:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:10 smithi097 ceph-mon[32231]: pgmap v9543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16908.073893 secs 2024-07-26T12:41:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:12 smithi181 ceph-mon[35869]: pgmap v9544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:12 smithi097 ceph-mon[32231]: pgmap v9544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:12 smithi115 ceph-mon[35364]: pgmap v9544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:14 smithi097 ceph-mon[32231]: pgmap v9545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:14 smithi115 ceph-mon[35364]: pgmap v9545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:14 smithi181 ceph-mon[35869]: pgmap v9545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16913.074002 secs 2024-07-26T12:41:16.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16913.074002 secs 2024-07-26T12:41:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16913.074002 secs 2024-07-26T12:41:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:17 smithi115 ceph-mon[35364]: pgmap v9546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:17.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:17 smithi181 ceph-mon[35869]: pgmap v9546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:17 smithi097 ceph-mon[32231]: pgmap v9546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:19 smithi115 ceph-mon[35364]: pgmap v9547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:19 smithi181 ceph-mon[35869]: pgmap v9547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:19 smithi097 ceph-mon[32231]: pgmap v9547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:21 smithi115 ceph-mon[35364]: pgmap v9548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16918.074152 secs 2024-07-26T12:41:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:21 smithi181 ceph-mon[35869]: pgmap v9548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16918.074152 secs 2024-07-26T12:41:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:21 smithi097 ceph-mon[32231]: pgmap v9548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16918.074152 secs 2024-07-26T12:41:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:23 smithi115 ceph-mon[35364]: pgmap v9549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:23 smithi181 ceph-mon[35869]: pgmap v9549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:23 smithi097 ceph-mon[32231]: pgmap v9549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:24.514+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:24.516+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:41:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:24.516+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:41:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:24.516+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:24.516+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:25 smithi115 ceph-mon[35364]: pgmap v9550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:25.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:25 smithi181 ceph-mon[35869]: pgmap v9550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:25 smithi097 ceph-mon[32231]: pgmap v9550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:26.375 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16923.074300 secs 2024-07-26T12:41:26.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16923.074300 secs 2024-07-26T12:41:26.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16923.074300 secs 2024-07-26T12:41:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:27 smithi115 ceph-mon[35364]: pgmap v9551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:27 smithi181 ceph-mon[35869]: pgmap v9551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:27 smithi097 ceph-mon[32231]: pgmap v9551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:28 smithi115 ceph-mon[35364]: pgmap v9552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T12:41:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:28 smithi115 ceph-mon[35364]: pgmap v9553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:41:28.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:41:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:28 smithi181 ceph-mon[35869]: pgmap v9552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T12:41:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:28 smithi181 ceph-mon[35869]: pgmap v9553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:41:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:41:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:28 smithi097 ceph-mon[32231]: pgmap v9552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T12:41:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:28 smithi097 ceph-mon[32231]: pgmap v9553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:41:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:41:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:30 smithi115 ceph-mon[35364]: pgmap v9554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:30 smithi181 ceph-mon[35869]: pgmap v9554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:30 smithi097 ceph-mon[32231]: pgmap v9554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16928.074430 secs 2024-07-26T12:41:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16928.074430 secs 2024-07-26T12:41:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16928.074430 secs 2024-07-26T12:41:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:32 smithi115 ceph-mon[35364]: pgmap v9555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:32 smithi181 ceph-mon[35869]: pgmap v9555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:32 smithi097 ceph-mon[32231]: pgmap v9555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:34 smithi115 ceph-mon[35364]: pgmap v9556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:34 smithi181 ceph-mon[35869]: pgmap v9556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:34 smithi097 ceph-mon[32231]: pgmap v9556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16933.074515 secs 2024-07-26T12:41:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:36 smithi115 ceph-mon[35364]: pgmap v9557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16933.074515 secs 2024-07-26T12:41:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:36 smithi181 ceph-mon[35869]: pgmap v9557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16933.074515 secs 2024-07-26T12:41:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:36 smithi097 ceph-mon[32231]: pgmap v9557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:38 smithi115 ceph-mon[35364]: pgmap v9558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:41:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:38 smithi181 ceph-mon[35869]: pgmap v9558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:41:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:38 smithi097 ceph-mon[32231]: pgmap v9558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:41:39.524 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:39.523+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:39.525 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:39.524+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:41:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:39.524+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:41:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:39.524+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:39.524+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:40 smithi115 ceph-mon[35364]: pgmap v9559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:41:40.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:40 smithi181 ceph-mon[35869]: pgmap v9559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:41:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:40 smithi097 ceph-mon[32231]: pgmap v9559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:41:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16938.074659 secs 2024-07-26T12:41:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16938.074659 secs 2024-07-26T12:41:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16938.074659 secs 2024-07-26T12:41:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:42 smithi115 ceph-mon[35364]: pgmap v9560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:42 smithi181 ceph-mon[35869]: pgmap v9560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:42 smithi097 ceph-mon[32231]: pgmap v9560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:44 smithi115 ceph-mon[35364]: pgmap v9561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:44 smithi181 ceph-mon[35869]: pgmap v9561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:44 smithi097 ceph-mon[32231]: pgmap v9561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16943.074814 secs 2024-07-26T12:41:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:46 smithi115 ceph-mon[35364]: pgmap v9562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16943.074814 secs 2024-07-26T12:41:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:46 smithi181 ceph-mon[35869]: pgmap v9562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16943.074814 secs 2024-07-26T12:41:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:46 smithi097 ceph-mon[32231]: pgmap v9562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:48 smithi115 ceph-mon[35364]: pgmap v9563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:48 smithi181 ceph-mon[35869]: pgmap v9563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:48 smithi097 ceph-mon[32231]: pgmap v9563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:50 smithi115 ceph-mon[35364]: pgmap v9564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:41:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:50 smithi181 ceph-mon[35869]: pgmap v9564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:41:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:50 smithi097 ceph-mon[32231]: pgmap v9564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:41:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16948.074958 secs 2024-07-26T12:41:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16948.074958 secs 2024-07-26T12:41:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16948.074958 secs 2024-07-26T12:41:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:52 smithi115 ceph-mon[35364]: pgmap v9565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:52 smithi181 ceph-mon[35869]: pgmap v9565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:52 smithi097 ceph-mon[32231]: pgmap v9565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:41:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:54.526+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:54.528+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:41:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:54.528+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:41:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:54.528+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:41:54.528+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:54 smithi115 ceph-mon[35364]: pgmap v9566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:54 smithi181 ceph-mon[35869]: pgmap v9566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:54 smithi097 ceph-mon[32231]: pgmap v9566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16953.075090 secs 2024-07-26T12:41:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:56 smithi115 ceph-mon[35364]: pgmap v9567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16953.075090 secs 2024-07-26T12:41:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:56 smithi181 ceph-mon[35869]: pgmap v9567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16953.075090 secs 2024-07-26T12:41:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:56 smithi097 ceph-mon[32231]: pgmap v9567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:41:58 smithi181 ceph-mon[35869]: pgmap v9568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:41:58 smithi097 ceph-mon[32231]: pgmap v9568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:41:58 smithi115 ceph-mon[35364]: pgmap v9568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:00 smithi181 ceph-mon[35869]: pgmap v9569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:00 smithi097 ceph-mon[32231]: pgmap v9569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:00 smithi115 ceph-mon[35364]: pgmap v9569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16958.075287 secs 2024-07-26T12:42:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16958.075287 secs 2024-07-26T12:42:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16958.075287 secs 2024-07-26T12:42:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:02 smithi097 ceph-mon[32231]: pgmap v9570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:02 smithi115 ceph-mon[35364]: pgmap v9570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:02 smithi181 ceph-mon[35869]: pgmap v9570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:04 smithi115 ceph-mon[35364]: pgmap v9571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:04 smithi181 ceph-mon[35869]: pgmap v9571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:04 smithi097 ceph-mon[32231]: pgmap v9571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16963.075380 secs 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:06 smithi115 ceph-mon[35364]: pgmap v9572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16963.075380 secs 2024-07-26T12:42:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:06 smithi181 ceph-mon[35869]: pgmap v9572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16963.075380 secs 2024-07-26T12:42:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:06 smithi097 ceph-mon[32231]: pgmap v9572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:08 smithi115 ceph-mon[35364]: pgmap v9573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:08 smithi181 ceph-mon[35869]: pgmap v9573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:08 smithi097 ceph-mon[32231]: pgmap v9573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:09.536+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:09.537+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:42:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:09.538+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:42:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:09.538+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:09.538+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:10 smithi115 ceph-mon[35364]: pgmap v9574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:10 smithi181 ceph-mon[35869]: pgmap v9574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:10 smithi097 ceph-mon[32231]: pgmap v9574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16968.075520 secs 2024-07-26T12:42:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16968.075520 secs 2024-07-26T12:42:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16968.075520 secs 2024-07-26T12:42:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:12 smithi115 ceph-mon[35364]: pgmap v9575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:12 smithi181 ceph-mon[35869]: pgmap v9575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:12 smithi097 ceph-mon[32231]: pgmap v9575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:14 smithi115 ceph-mon[35364]: pgmap v9576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:14 smithi181 ceph-mon[35869]: pgmap v9576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:14 smithi097 ceph-mon[32231]: pgmap v9576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16973.075662 secs 2024-07-26T12:42:16.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:16 smithi115 ceph-mon[35364]: pgmap v9577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16973.075662 secs 2024-07-26T12:42:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:16 smithi181 ceph-mon[35869]: pgmap v9577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16973.075662 secs 2024-07-26T12:42:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:16 smithi097 ceph-mon[32231]: pgmap v9577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:18 smithi115 ceph-mon[35364]: pgmap v9578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:18 smithi181 ceph-mon[35869]: pgmap v9578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:18 smithi097 ceph-mon[32231]: pgmap v9578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:20 smithi115 ceph-mon[35364]: pgmap v9579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:20 smithi181 ceph-mon[35869]: pgmap v9579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:20 smithi097 ceph-mon[32231]: pgmap v9579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:42:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16978.075812 secs 2024-07-26T12:42:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16978.075812 secs 2024-07-26T12:42:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16978.075812 secs 2024-07-26T12:42:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:22 smithi115 ceph-mon[35364]: pgmap v9580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:22 smithi181 ceph-mon[35869]: pgmap v9580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:22 smithi097 ceph-mon[32231]: pgmap v9580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:24.541+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:24.542+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:42:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:24.542+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:42:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:24.542+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:24.542+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:24 smithi115 ceph-mon[35364]: pgmap v9581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:24 smithi181 ceph-mon[35869]: pgmap v9581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:24 smithi097 ceph-mon[32231]: pgmap v9581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16983.075960 secs 2024-07-26T12:42:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:26 smithi115 ceph-mon[35364]: pgmap v9582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16983.075960 secs 2024-07-26T12:42:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:26 smithi181 ceph-mon[35869]: pgmap v9582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16983.075960 secs 2024-07-26T12:42:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:26 smithi097 ceph-mon[32231]: pgmap v9582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:28 smithi115 ceph-mon[35364]: pgmap v9583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:28 smithi181 ceph-mon[35869]: pgmap v9583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:28.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:28 smithi097 ceph-mon[32231]: pgmap v9583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:29 smithi115 ceph-mon[35364]: pgmap v9584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:29 smithi115 ceph-mon[35364]: pgmap v9585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:29.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:42:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:29 smithi181 ceph-mon[35869]: pgmap v9584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:29 smithi181 ceph-mon[35869]: pgmap v9585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:29.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:42:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:29 smithi097 ceph-mon[32231]: pgmap v9584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:29 smithi097 ceph-mon[32231]: pgmap v9585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:42:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:42:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16988.076096 secs 2024-07-26T12:42:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:31 smithi115 ceph-mon[35364]: pgmap v9586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16988.076096 secs 2024-07-26T12:42:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:31 smithi181 ceph-mon[35869]: pgmap v9586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16988.076096 secs 2024-07-26T12:42:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:31 smithi097 ceph-mon[32231]: pgmap v9586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:33 smithi115 ceph-mon[35364]: pgmap v9587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:33 smithi181 ceph-mon[35869]: pgmap v9587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:33 smithi097 ceph-mon[32231]: pgmap v9587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:35 smithi115 ceph-mon[35364]: pgmap v9588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:35 smithi181 ceph-mon[35869]: pgmap v9588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:35 smithi097 ceph-mon[32231]: pgmap v9588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16993.076281 secs 2024-07-26T12:42:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16993.076281 secs 2024-07-26T12:42:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16993.076281 secs 2024-07-26T12:42:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:37 smithi115 ceph-mon[35364]: pgmap v9589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:37 smithi181 ceph-mon[35869]: pgmap v9589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:37 smithi097 ceph-mon[32231]: pgmap v9589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:38 smithi115 ceph-mon[35364]: pgmap v9590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:38 smithi181 ceph-mon[35869]: pgmap v9590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:38 smithi097 ceph-mon[32231]: pgmap v9590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:39.551 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:39.549+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:39.551+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:42:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:39.551+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:42:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:39.551+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:39.551+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 16998.076416 secs 2024-07-26T12:42:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:41 smithi115 ceph-mon[35364]: pgmap v9591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 16998.076416 secs 2024-07-26T12:42:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:41 smithi181 ceph-mon[35869]: pgmap v9591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 16998.076416 secs 2024-07-26T12:42:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:41 smithi097 ceph-mon[32231]: pgmap v9591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:43 smithi115 ceph-mon[35364]: pgmap v9592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:43 smithi181 ceph-mon[35869]: pgmap v9592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:43 smithi097 ceph-mon[32231]: pgmap v9592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:45 smithi115 ceph-mon[35364]: pgmap v9593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:45 smithi181 ceph-mon[35869]: pgmap v9593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:45 smithi097 ceph-mon[32231]: pgmap v9593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17003.076514 secs 2024-07-26T12:42:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17003.076514 secs 2024-07-26T12:42:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17003.076514 secs 2024-07-26T12:42:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:47 smithi115 ceph-mon[35364]: pgmap v9594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:47 smithi181 ceph-mon[35869]: pgmap v9594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:47 smithi097 ceph-mon[32231]: pgmap v9594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:48 smithi115 ceph-mon[35364]: pgmap v9595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:48 smithi181 ceph-mon[35869]: pgmap v9595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:48 smithi097 ceph-mon[32231]: pgmap v9595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17008.076655 secs 2024-07-26T12:42:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:51 smithi115 ceph-mon[35364]: pgmap v9596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17008.076655 secs 2024-07-26T12:42:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:51 smithi181 ceph-mon[35869]: pgmap v9596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17008.076655 secs 2024-07-26T12:42:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:51 smithi097 ceph-mon[32231]: pgmap v9596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:53 smithi115 ceph-mon[35364]: pgmap v9597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:53 smithi181 ceph-mon[35869]: pgmap v9597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:53 smithi097 ceph-mon[32231]: pgmap v9597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:54.555 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:54.554+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:54.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:54.555+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:42:54.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:54.555+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:42:54.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:54.555+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:54.557 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:42:54.555+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:55 smithi115 ceph-mon[35364]: pgmap v9598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:55 smithi181 ceph-mon[35869]: pgmap v9598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:55 smithi097 ceph-mon[32231]: pgmap v9598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17013.076802 secs 2024-07-26T12:42:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17013.076802 secs 2024-07-26T12:42:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17013.076802 secs 2024-07-26T12:42:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:57 smithi115 ceph-mon[35364]: pgmap v9599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:57 smithi181 ceph-mon[35869]: pgmap v9599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:57 smithi097 ceph-mon[32231]: pgmap v9599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:42:58 smithi115 ceph-mon[35364]: pgmap v9600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:42:58 smithi181 ceph-mon[35869]: pgmap v9600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:42:58 smithi097 ceph-mon[32231]: pgmap v9600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17018.076937 secs 2024-07-26T12:43:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:01 smithi115 ceph-mon[35364]: pgmap v9601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17018.076937 secs 2024-07-26T12:43:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:01 smithi181 ceph-mon[35869]: pgmap v9601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17018.076937 secs 2024-07-26T12:43:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:01 smithi097 ceph-mon[32231]: pgmap v9601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:03 smithi115 ceph-mon[35364]: pgmap v9602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:03 smithi181 ceph-mon[35869]: pgmap v9602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:03 smithi097 ceph-mon[32231]: pgmap v9602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:05 smithi115 ceph-mon[35364]: pgmap v9603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:06.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:05 smithi181 ceph-mon[35869]: pgmap v9603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:05 smithi097 ceph-mon[32231]: pgmap v9603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.934+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 msgr2=0x7ff3d40b5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T12:43:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.934+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 msgr2=0x7ff3d40b5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:43:06.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.934+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 0x7ff3d40b5df0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005d00 tx=0x7ff3e00a4000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:06.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.934+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 0x7ff3d40b5df0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005d00 tx=0x7ff3e00a4000 comp rx=0 tx=0).stop 2024-07-26T12:43:06.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.934+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 msgr2=0x7ff3d40b5df0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T12:43:06.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.934+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 msgr2=0x7ff3d40b5df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:43:06.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.934+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b59e0 0x7ff3d40b5df0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:06.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.934+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:06.936 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.935+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:06.936+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057c0 tx=0x7ff3e006a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17023.077096 secs 2024-07-26T12:43:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:06 smithi115 ceph-mon[35364]: pgmap v9604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17023.077096 secs 2024-07-26T12:43:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:06 smithi181 ceph-mon[35869]: pgmap v9604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17023.077096 secs 2024-07-26T12:43:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:06 smithi097 ceph-mon[32231]: pgmap v9604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.557+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.558+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:43:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.558+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:43:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.558+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.558+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:09.931 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.930+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:43:09.931 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.930+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:43:09.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.931+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:43:09.932 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:09.931+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:43:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:09 smithi115 ceph-mon[35364]: pgmap v9605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:09 smithi181 ceph-mon[35869]: pgmap v9605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:09 smithi097 ceph-mon[32231]: pgmap v9605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17028.077277 secs 2024-07-26T12:43:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:11 smithi115 ceph-mon[35364]: pgmap v9606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17028.077277 secs 2024-07-26T12:43:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:11 smithi181 ceph-mon[35869]: pgmap v9606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17028.077277 secs 2024-07-26T12:43:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:11 smithi097 ceph-mon[32231]: pgmap v9606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:12 smithi115 ceph-mon[35364]: pgmap v9607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:13.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:12 smithi181 ceph-mon[35869]: pgmap v9607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:12 smithi097 ceph-mon[32231]: pgmap v9607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:15 smithi115 ceph-mon[35364]: pgmap v9608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:15 smithi181 ceph-mon[35869]: pgmap v9608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:15 smithi097 ceph-mon[32231]: pgmap v9608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17033.077431 secs 2024-07-26T12:43:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17033.077431 secs 2024-07-26T12:43:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17033.077431 secs 2024-07-26T12:43:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:17 smithi115 ceph-mon[35364]: pgmap v9609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:17 smithi181 ceph-mon[35869]: pgmap v9609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:17 smithi097 ceph-mon[32231]: pgmap v9609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:18 smithi115 ceph-mon[35364]: pgmap v9610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:18 smithi181 ceph-mon[35869]: pgmap v9610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:18 smithi097 ceph-mon[32231]: pgmap v9610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17038.077586 secs 2024-07-26T12:43:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:21 smithi115 ceph-mon[35364]: pgmap v9611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17038.077586 secs 2024-07-26T12:43:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:21 smithi181 ceph-mon[35869]: pgmap v9611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17038.077586 secs 2024-07-26T12:43:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:21 smithi097 ceph-mon[32231]: pgmap v9611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:22 smithi115 ceph-mon[35364]: pgmap v9612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:22 smithi181 ceph-mon[35869]: pgmap v9612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:22 smithi097 ceph-mon[32231]: pgmap v9612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:24.561 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:24.560+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:24.562 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:24.561+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:43:24.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:24.561+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:43:24.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:24.561+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:24.563 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:24.561+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:25 smithi115 ceph-mon[35364]: pgmap v9613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:25 smithi181 ceph-mon[35869]: pgmap v9613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:25 smithi097 ceph-mon[32231]: pgmap v9613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:27.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17043.077752 secs 2024-07-26T12:43:27.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17043.077752 secs 2024-07-26T12:43:27.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17043.077752 secs 2024-07-26T12:43:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:27 smithi115 ceph-mon[35364]: pgmap v9614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:27 smithi181 ceph-mon[35869]: pgmap v9614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:27 smithi097 ceph-mon[32231]: pgmap v9614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:28 smithi115 ceph-mon[35364]: pgmap v9615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:28 smithi181 ceph-mon[35869]: pgmap v9615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:28 smithi097 ceph-mon[32231]: pgmap v9615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:30.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17048.077902 secs 2024-07-26T12:43:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:30 smithi115 ceph-mon[35364]: pgmap v9616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:43:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:30 smithi115 ceph-mon[35364]: pgmap v9617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:43:31.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:30 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:43:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17048.077902 secs 2024-07-26T12:43:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:30 smithi181 ceph-mon[35869]: pgmap v9616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:43:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:30 smithi181 ceph-mon[35869]: pgmap v9617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:43:31.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:30 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:43:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17048.077902 secs 2024-07-26T12:43:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:30 smithi097 ceph-mon[32231]: pgmap v9616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:43:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:30 smithi097 ceph-mon[32231]: pgmap v9617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:43:31.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:30 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:43:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:33 smithi097 ceph-mon[32231]: pgmap v9618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:43:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:33 smithi115 ceph-mon[35364]: pgmap v9618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:43:33.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:33 smithi181 ceph-mon[35869]: pgmap v9618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:43:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:35 smithi097 ceph-mon[32231]: pgmap v9619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:35 smithi115 ceph-mon[35364]: pgmap v9619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:35 smithi181 ceph-mon[35869]: pgmap v9619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17053.078043 secs 2024-07-26T12:43:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17053.078043 secs 2024-07-26T12:43:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17053.078043 secs 2024-07-26T12:43:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:37 smithi115 ceph-mon[35364]: pgmap v9620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:37 smithi181 ceph-mon[35869]: pgmap v9620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:37 smithi097 ceph-mon[32231]: pgmap v9620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:39.568+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:39.570+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:43:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:39.570+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:43:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:39.570+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:39.570+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:39 smithi115 ceph-mon[35364]: pgmap v9621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:39 smithi181 ceph-mon[35869]: pgmap v9621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:39 smithi097 ceph-mon[32231]: pgmap v9621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17058.078178 secs 2024-07-26T12:43:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:41 smithi115 ceph-mon[35364]: pgmap v9622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:43:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17058.078178 secs 2024-07-26T12:43:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:41 smithi181 ceph-mon[35869]: pgmap v9622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:43:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17058.078178 secs 2024-07-26T12:43:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:41 smithi097 ceph-mon[32231]: pgmap v9622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:43:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:43 smithi115 ceph-mon[35364]: pgmap v9623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:43:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:43 smithi181 ceph-mon[35869]: pgmap v9623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:43:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:43 smithi097 ceph-mon[32231]: pgmap v9623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:43:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:45 smithi115 ceph-mon[35364]: pgmap v9624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:45 smithi181 ceph-mon[35869]: pgmap v9624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:45 smithi097 ceph-mon[32231]: pgmap v9624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17063.078344 secs 2024-07-26T12:43:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17063.078344 secs 2024-07-26T12:43:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17063.078344 secs 2024-07-26T12:43:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:47 smithi115 ceph-mon[35364]: pgmap v9625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:47 smithi181 ceph-mon[35869]: pgmap v9625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:47 smithi097 ceph-mon[32231]: pgmap v9625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:49 smithi115 ceph-mon[35364]: pgmap v9626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:49 smithi181 ceph-mon[35869]: pgmap v9626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:49.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:49 smithi097 ceph-mon[32231]: pgmap v9626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17068.078503 secs 2024-07-26T12:43:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:51 smithi115 ceph-mon[35364]: pgmap v9627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17068.078503 secs 2024-07-26T12:43:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:51 smithi181 ceph-mon[35869]: pgmap v9627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17068.078503 secs 2024-07-26T12:43:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:51 smithi097 ceph-mon[32231]: pgmap v9627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:53 smithi115 ceph-mon[35364]: pgmap v9628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:53 smithi181 ceph-mon[35869]: pgmap v9628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:53 smithi097 ceph-mon[32231]: pgmap v9628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:54.573+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:54.574+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:43:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:54.574+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:43:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:54.574+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:43:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:43:54.574+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:55 smithi115 ceph-mon[35364]: pgmap v9629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:55 smithi181 ceph-mon[35869]: pgmap v9629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:55 smithi097 ceph-mon[32231]: pgmap v9629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17073.078590 secs 2024-07-26T12:43:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17073.078590 secs 2024-07-26T12:43:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17073.078590 secs 2024-07-26T12:43:57.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:57 smithi115 ceph-mon[35364]: pgmap v9630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:57.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:57 smithi181 ceph-mon[35869]: pgmap v9630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:57.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:57 smithi097 ceph-mon[32231]: pgmap v9630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:43:59 smithi115 ceph-mon[35364]: pgmap v9631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:43:59 smithi181 ceph-mon[35869]: pgmap v9631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:43:59 smithi097 ceph-mon[32231]: pgmap v9631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17078.078685 secs 2024-07-26T12:44:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:01 smithi115 ceph-mon[35364]: pgmap v9632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17078.078685 secs 2024-07-26T12:44:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:01 smithi181 ceph-mon[35869]: pgmap v9632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17078.078685 secs 2024-07-26T12:44:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:01 smithi097 ceph-mon[32231]: pgmap v9632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:03 smithi115 ceph-mon[35364]: pgmap v9633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:03 smithi181 ceph-mon[35869]: pgmap v9633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:03 smithi097 ceph-mon[32231]: pgmap v9633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:05 smithi115 ceph-mon[35364]: pgmap v9634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:05 smithi181 ceph-mon[35869]: pgmap v9634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:05 smithi097 ceph-mon[32231]: pgmap v9634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17083.078773 secs 2024-07-26T12:44:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17083.078773 secs 2024-07-26T12:44:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17083.078773 secs 2024-07-26T12:44:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:07 smithi115 ceph-mon[35364]: pgmap v9635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:07 smithi181 ceph-mon[35869]: pgmap v9635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:07 smithi097 ceph-mon[32231]: pgmap v9635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:09.575+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:09.576+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:44:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:09.577+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:44:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:09.577+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:44:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:09.577+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:09 smithi115 ceph-mon[35364]: pgmap v9636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:09.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:09 smithi181 ceph-mon[35869]: pgmap v9636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:09 smithi097 ceph-mon[32231]: pgmap v9636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17088.078932 secs 2024-07-26T12:44:11.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:11 smithi115 ceph-mon[35364]: pgmap v9637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17088.078932 secs 2024-07-26T12:44:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:11 smithi181 ceph-mon[35869]: pgmap v9637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17088.078932 secs 2024-07-26T12:44:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:11 smithi097 ceph-mon[32231]: pgmap v9637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:13 smithi115 ceph-mon[35364]: pgmap v9638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:44:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:13 smithi181 ceph-mon[35869]: pgmap v9638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:44:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:13 smithi097 ceph-mon[32231]: pgmap v9638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:44:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:15 smithi115 ceph-mon[35364]: pgmap v9639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:15 smithi181 ceph-mon[35869]: pgmap v9639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:15 smithi097 ceph-mon[32231]: pgmap v9639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17093.079068 secs 2024-07-26T12:44:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17093.079068 secs 2024-07-26T12:44:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17093.079068 secs 2024-07-26T12:44:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:17 smithi115 ceph-mon[35364]: pgmap v9640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:17 smithi181 ceph-mon[35869]: pgmap v9640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:17 smithi097 ceph-mon[32231]: pgmap v9640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:19 smithi115 ceph-mon[35364]: pgmap v9641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:19 smithi181 ceph-mon[35869]: pgmap v9641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:19 smithi097 ceph-mon[32231]: pgmap v9641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17098.079279 secs 2024-07-26T12:44:21.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:21 smithi115 ceph-mon[35364]: pgmap v9642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17098.079279 secs 2024-07-26T12:44:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:21 smithi181 ceph-mon[35869]: pgmap v9642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17098.079279 secs 2024-07-26T12:44:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:21 smithi097 ceph-mon[32231]: pgmap v9642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:23 smithi115 ceph-mon[35364]: pgmap v9643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:23 smithi181 ceph-mon[35869]: pgmap v9643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:23 smithi097 ceph-mon[32231]: pgmap v9643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:24.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:24.579+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:24.580+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:44:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:24.580+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:44:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:24.580+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:44:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:24.580+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:25 smithi115 ceph-mon[35364]: pgmap v9644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:25 smithi181 ceph-mon[35869]: pgmap v9644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:25 smithi097 ceph-mon[32231]: pgmap v9644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:44:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17103.079389 secs 2024-07-26T12:44:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17103.079389 secs 2024-07-26T12:44:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17103.079389 secs 2024-07-26T12:44:27.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:27 smithi115 ceph-mon[35364]: pgmap v9645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:27.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:27 smithi181 ceph-mon[35869]: pgmap v9645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:27.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:27 smithi097 ceph-mon[32231]: pgmap v9645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:29 smithi115 ceph-mon[35364]: pgmap v9646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:29 smithi181 ceph-mon[35869]: pgmap v9646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:29 smithi097 ceph-mon[32231]: pgmap v9646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17108.079532 secs 2024-07-26T12:44:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:31 smithi115 ceph-mon[35364]: pgmap v9647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:31.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:31 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17108.079532 secs 2024-07-26T12:44:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:31 smithi181 ceph-mon[35869]: pgmap v9647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:31 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17108.079532 secs 2024-07-26T12:44:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:31 smithi097 ceph-mon[32231]: pgmap v9647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:31 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:32 smithi115 ceph-mon[35364]: pgmap v9648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:44:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:32 smithi115 ceph-mon[35364]: pgmap v9649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:32.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:44:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:32 smithi181 ceph-mon[35869]: pgmap v9648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:44:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:32 smithi181 ceph-mon[35869]: pgmap v9649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:44:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:32 smithi097 ceph-mon[32231]: pgmap v9648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:44:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:32 smithi097 ceph-mon[32231]: pgmap v9649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:32.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:44:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:34 smithi115 ceph-mon[35364]: pgmap v9650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:34.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:34 smithi181 ceph-mon[35869]: pgmap v9650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:34 smithi097 ceph-mon[32231]: pgmap v9650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:44:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17113.079693 secs 2024-07-26T12:44:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:36 smithi115 ceph-mon[35364]: pgmap v9651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17113.079693 secs 2024-07-26T12:44:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:36 smithi181 ceph-mon[35869]: pgmap v9651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17113.079693 secs 2024-07-26T12:44:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:36 smithi097 ceph-mon[32231]: pgmap v9651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:38 smithi115 ceph-mon[35364]: pgmap v9652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:38 smithi181 ceph-mon[35869]: pgmap v9652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:38 smithi097 ceph-mon[32231]: pgmap v9652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:39.588+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:39.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:39.590+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:44:39.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:39.590+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:44:39.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:39.590+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:44:39.591 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:39.590+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:40 smithi115 ceph-mon[35364]: pgmap v9653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:40 smithi181 ceph-mon[35869]: pgmap v9653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:40 smithi097 ceph-mon[32231]: pgmap v9653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17118.079829 secs 2024-07-26T12:44:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17118.079829 secs 2024-07-26T12:44:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17118.079829 secs 2024-07-26T12:44:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:42 smithi115 ceph-mon[35364]: pgmap v9654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:42.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:42 smithi181 ceph-mon[35869]: pgmap v9654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:42 smithi097 ceph-mon[32231]: pgmap v9654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:44 smithi115 ceph-mon[35364]: pgmap v9655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:44 smithi181 ceph-mon[35869]: pgmap v9655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:44 smithi097 ceph-mon[32231]: pgmap v9655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17123.079916 secs 2024-07-26T12:44:46.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:46 smithi115 ceph-mon[35364]: pgmap v9656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17123.079916 secs 2024-07-26T12:44:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:46 smithi181 ceph-mon[35869]: pgmap v9656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17123.079916 secs 2024-07-26T12:44:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:46 smithi097 ceph-mon[32231]: pgmap v9656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:48 smithi115 ceph-mon[35364]: pgmap v9657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:48 smithi181 ceph-mon[35869]: pgmap v9657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:48 smithi097 ceph-mon[32231]: pgmap v9657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:50 smithi181 ceph-mon[35869]: pgmap v9658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:50 smithi097 ceph-mon[32231]: pgmap v9658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:50 smithi115 ceph-mon[35364]: pgmap v9658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17128.080073 secs 2024-07-26T12:44:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17128.080073 secs 2024-07-26T12:44:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17128.080073 secs 2024-07-26T12:44:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:52 smithi097 ceph-mon[32231]: pgmap v9659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:52 smithi115 ceph-mon[35364]: pgmap v9659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:52 smithi181 ceph-mon[35869]: pgmap v9659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:54.593+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:54.594+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:44:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:54.594+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:44:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:54.594+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:44:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:44:54.594+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:54 smithi097 ceph-mon[32231]: pgmap v9660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:54 smithi115 ceph-mon[35364]: pgmap v9660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:55.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:54 smithi181 ceph-mon[35869]: pgmap v9660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:57.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17133.080221 secs 2024-07-26T12:44:57.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:56 smithi115 ceph-mon[35364]: pgmap v9661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:57.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17133.080221 secs 2024-07-26T12:44:57.036 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:56 smithi181 ceph-mon[35869]: pgmap v9661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17133.080221 secs 2024-07-26T12:44:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:56 smithi097 ceph-mon[32231]: pgmap v9661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:59.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:44:58 smithi097 ceph-mon[32231]: pgmap v9662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:44:58 smithi115 ceph-mon[35364]: pgmap v9662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:44:58 smithi181 ceph-mon[35869]: pgmap v9662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:00 smithi115 ceph-mon[35364]: pgmap v9663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:00 smithi181 ceph-mon[35869]: pgmap v9663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:00 smithi097 ceph-mon[32231]: pgmap v9663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17138.080296 secs 2024-07-26T12:45:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17138.080296 secs 2024-07-26T12:45:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17138.080296 secs 2024-07-26T12:45:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:02 smithi115 ceph-mon[35364]: pgmap v9664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:02 smithi181 ceph-mon[35869]: pgmap v9664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:02 smithi097 ceph-mon[32231]: pgmap v9664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:04 smithi115 ceph-mon[35364]: pgmap v9665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:04 smithi181 ceph-mon[35869]: pgmap v9665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:04 smithi097 ceph-mon[32231]: pgmap v9665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17143.080408 secs 2024-07-26T12:45:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:06 smithi115 ceph-mon[35364]: pgmap v9666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17143.080408 secs 2024-07-26T12:45:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:06 smithi181 ceph-mon[35869]: pgmap v9666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17143.080408 secs 2024-07-26T12:45:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:06 smithi097 ceph-mon[32231]: pgmap v9666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:08 smithi115 ceph-mon[35364]: pgmap v9667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:08 smithi181 ceph-mon[35869]: pgmap v9667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:08 smithi097 ceph-mon[32231]: pgmap v9667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:09.602+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:09.604+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:45:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:09.604+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:45:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:09.604+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:45:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:09.604+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:10 smithi115 ceph-mon[35364]: pgmap v9668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:10 smithi181 ceph-mon[35869]: pgmap v9668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:10 smithi097 ceph-mon[32231]: pgmap v9668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17148.080554 secs 2024-07-26T12:45:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17148.080554 secs 2024-07-26T12:45:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17148.080554 secs 2024-07-26T12:45:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:12 smithi115 ceph-mon[35364]: pgmap v9669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:12 smithi181 ceph-mon[35869]: pgmap v9669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:12 smithi097 ceph-mon[32231]: pgmap v9669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:14 smithi115 ceph-mon[35364]: pgmap v9670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:14 smithi181 ceph-mon[35869]: pgmap v9670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:14 smithi097 ceph-mon[32231]: pgmap v9670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17153.080693 secs 2024-07-26T12:45:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:16 smithi115 ceph-mon[35364]: pgmap v9671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17153.080693 secs 2024-07-26T12:45:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:16 smithi181 ceph-mon[35869]: pgmap v9671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17153.080693 secs 2024-07-26T12:45:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:16 smithi097 ceph-mon[32231]: pgmap v9671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:18 smithi115 ceph-mon[35364]: pgmap v9672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:18 smithi181 ceph-mon[35869]: pgmap v9672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:18 smithi097 ceph-mon[32231]: pgmap v9672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:20 smithi115 ceph-mon[35364]: pgmap v9673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:20 smithi181 ceph-mon[35869]: pgmap v9673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:20 smithi097 ceph-mon[32231]: pgmap v9673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17158.080826 secs 2024-07-26T12:45:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17158.080826 secs 2024-07-26T12:45:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17158.080826 secs 2024-07-26T12:45:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:22 smithi115 ceph-mon[35364]: pgmap v9674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:22.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:22 smithi181 ceph-mon[35869]: pgmap v9674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:22 smithi097 ceph-mon[32231]: pgmap v9674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:24.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:24.607+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:24.608+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:45:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:24.608+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:45:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:24.608+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:45:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:24.608+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:24 smithi115 ceph-mon[35364]: pgmap v9675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:24 smithi181 ceph-mon[35869]: pgmap v9675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:24 smithi097 ceph-mon[32231]: pgmap v9675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17163.080997 secs 2024-07-26T12:45:26.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:26 smithi115 ceph-mon[35364]: pgmap v9676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:45:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17163.080997 secs 2024-07-26T12:45:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:26 smithi181 ceph-mon[35869]: pgmap v9676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:45:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17163.080997 secs 2024-07-26T12:45:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:26 smithi097 ceph-mon[32231]: pgmap v9676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:45:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:28 smithi115 ceph-mon[35364]: pgmap v9677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:45:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:28 smithi181 ceph-mon[35869]: pgmap v9677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:45:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:28 smithi097 ceph-mon[32231]: pgmap v9677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:45:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:30 smithi181 ceph-mon[35869]: pgmap v9678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:30 smithi097 ceph-mon[32231]: pgmap v9678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:30 smithi115 ceph-mon[35364]: pgmap v9678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17168.081134 secs 2024-07-26T12:45:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17168.081134 secs 2024-07-26T12:45:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17168.081134 secs 2024-07-26T12:45:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:32 smithi097 ceph-mon[32231]: pgmap v9679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:32 smithi115 ceph-mon[35364]: pgmap v9679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:32 smithi181 ceph-mon[35869]: pgmap v9679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:33.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:45:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:45:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:45:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:34 smithi115 ceph-mon[35364]: pgmap v9680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:45:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:34 smithi115 ceph-mon[35364]: pgmap v9681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:34 smithi181 ceph-mon[35869]: pgmap v9680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:45:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:34 smithi181 ceph-mon[35869]: pgmap v9681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:34 smithi097 ceph-mon[32231]: pgmap v9680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:45:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:34 smithi097 ceph-mon[32231]: pgmap v9681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17173.081282 secs 2024-07-26T12:45:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:36 smithi115 ceph-mon[35364]: pgmap v9682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17173.081282 secs 2024-07-26T12:45:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:36 smithi181 ceph-mon[35869]: pgmap v9682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17173.081282 secs 2024-07-26T12:45:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:36 smithi097 ceph-mon[32231]: pgmap v9682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:38 smithi115 ceph-mon[35364]: pgmap v9683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:38 smithi181 ceph-mon[35869]: pgmap v9683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:38 smithi097 ceph-mon[32231]: pgmap v9683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:39.616+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:39.618 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:39.617+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:45:39.618 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:39.617+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:45:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:39.617+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:45:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:39.617+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:40 smithi115 ceph-mon[35364]: pgmap v9684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:40 smithi181 ceph-mon[35869]: pgmap v9684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:40 smithi097 ceph-mon[32231]: pgmap v9684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17178.081422 secs 2024-07-26T12:45:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17178.081422 secs 2024-07-26T12:45:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17178.081422 secs 2024-07-26T12:45:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:42 smithi115 ceph-mon[35364]: pgmap v9685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:42 smithi181 ceph-mon[35869]: pgmap v9685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:42 smithi097 ceph-mon[32231]: pgmap v9685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:44 smithi115 ceph-mon[35364]: pgmap v9686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:44 smithi181 ceph-mon[35869]: pgmap v9686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:44 smithi097 ceph-mon[32231]: pgmap v9686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17183.081554 secs 2024-07-26T12:45:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:46 smithi115 ceph-mon[35364]: pgmap v9687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17183.081554 secs 2024-07-26T12:45:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:46 smithi181 ceph-mon[35869]: pgmap v9687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17183.081554 secs 2024-07-26T12:45:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:46 smithi097 ceph-mon[32231]: pgmap v9687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:48 smithi115 ceph-mon[35364]: pgmap v9688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:48 smithi181 ceph-mon[35869]: pgmap v9688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:48 smithi097 ceph-mon[32231]: pgmap v9688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:50 smithi115 ceph-mon[35364]: pgmap v9689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:50 smithi181 ceph-mon[35869]: pgmap v9689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:50 smithi097 ceph-mon[32231]: pgmap v9689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17188.081689 secs 2024-07-26T12:45:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17188.081689 secs 2024-07-26T12:45:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17188.081689 secs 2024-07-26T12:45:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:52 smithi115 ceph-mon[35364]: pgmap v9690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:52 smithi181 ceph-mon[35869]: pgmap v9690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:52 smithi097 ceph-mon[32231]: pgmap v9690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:54.619+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:54.620+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:45:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:54.621+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:45:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:54.621+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:45:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:45:54.621+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:54 smithi115 ceph-mon[35364]: pgmap v9691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:54 smithi181 ceph-mon[35869]: pgmap v9691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:54 smithi097 ceph-mon[32231]: pgmap v9691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17193.081824 secs 2024-07-26T12:45:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:56 smithi115 ceph-mon[35364]: pgmap v9692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17193.081824 secs 2024-07-26T12:45:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:56 smithi181 ceph-mon[35869]: pgmap v9692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17193.081824 secs 2024-07-26T12:45:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:56 smithi097 ceph-mon[32231]: pgmap v9692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:45:58 smithi115 ceph-mon[35364]: pgmap v9693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:45:58 smithi181 ceph-mon[35869]: pgmap v9693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:45:58 smithi097 ceph-mon[32231]: pgmap v9693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:00 smithi115 ceph-mon[35364]: pgmap v9694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:00 smithi181 ceph-mon[35869]: pgmap v9694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:00 smithi097 ceph-mon[32231]: pgmap v9694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17198.081962 secs 2024-07-26T12:46:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17198.081962 secs 2024-07-26T12:46:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17198.081962 secs 2024-07-26T12:46:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:02 smithi115 ceph-mon[35364]: pgmap v9695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:02 smithi181 ceph-mon[35869]: pgmap v9695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:02 smithi097 ceph-mon[32231]: pgmap v9695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:04 smithi115 ceph-mon[35364]: pgmap v9696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:04 smithi181 ceph-mon[35869]: pgmap v9696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:04 smithi097 ceph-mon[32231]: pgmap v9696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17203.082106 secs 2024-07-26T12:46:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:05 smithi115 ceph-mon[35364]: pgmap v9697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17203.082106 secs 2024-07-26T12:46:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:05 smithi181 ceph-mon[35869]: pgmap v9697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17203.082106 secs 2024-07-26T12:46:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:05 smithi097 ceph-mon[32231]: pgmap v9697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:08 smithi115 ceph-mon[35364]: pgmap v9698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:08 smithi181 ceph-mon[35869]: pgmap v9698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:08 smithi097 ceph-mon[32231]: pgmap v9698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.622+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.623+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:46:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.623+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:46:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.623+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:46:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.624+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.962+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:46:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.962+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:46:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.963+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:46:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:09.963+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:46:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:10 smithi115 ceph-mon[35364]: pgmap v9699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:10 smithi181 ceph-mon[35869]: pgmap v9699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:10 smithi097 ceph-mon[32231]: pgmap v9699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17208.082283 secs 2024-07-26T12:46:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17208.082283 secs 2024-07-26T12:46:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17208.082283 secs 2024-07-26T12:46:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:12 smithi115 ceph-mon[35364]: pgmap v9700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:12 smithi181 ceph-mon[35869]: pgmap v9700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:12 smithi097 ceph-mon[32231]: pgmap v9700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:14 smithi181 ceph-mon[35869]: pgmap v9701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:14 smithi097 ceph-mon[32231]: pgmap v9701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:14 smithi115 ceph-mon[35364]: pgmap v9701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17213.082423 secs 2024-07-26T12:46:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:16 smithi097 ceph-mon[32231]: pgmap v9702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:46:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17213.082423 secs 2024-07-26T12:46:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:16 smithi115 ceph-mon[35364]: pgmap v9702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:46:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17213.082423 secs 2024-07-26T12:46:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:16 smithi181 ceph-mon[35869]: pgmap v9702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:46:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:18 smithi097 ceph-mon[32231]: pgmap v9703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:46:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:18 smithi115 ceph-mon[35364]: pgmap v9703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:46:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:18 smithi181 ceph-mon[35869]: pgmap v9703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:46:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:20 smithi097 ceph-mon[32231]: pgmap v9704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:20 smithi115 ceph-mon[35364]: pgmap v9704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:20 smithi181 ceph-mon[35869]: pgmap v9704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17218.082562 secs 2024-07-26T12:46:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17218.082562 secs 2024-07-26T12:46:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17218.082562 secs 2024-07-26T12:46:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:22 smithi115 ceph-mon[35364]: pgmap v9705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:46:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:22 smithi181 ceph-mon[35869]: pgmap v9705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:46:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:22 smithi097 ceph-mon[32231]: pgmap v9705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:46:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:24.627+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:24.628+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:46:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:24.628+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:46:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:24.628+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:46:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:24.628+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:24 smithi115 ceph-mon[35364]: pgmap v9706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:24 smithi181 ceph-mon[35869]: pgmap v9706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:24 smithi097 ceph-mon[32231]: pgmap v9706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17223.082647 secs 2024-07-26T12:46:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:26 smithi115 ceph-mon[35364]: pgmap v9707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17223.082647 secs 2024-07-26T12:46:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:26 smithi181 ceph-mon[35869]: pgmap v9707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17223.082647 secs 2024-07-26T12:46:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:26 smithi097 ceph-mon[32231]: pgmap v9707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:28 smithi115 ceph-mon[35364]: pgmap v9708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:28 smithi181 ceph-mon[35869]: pgmap v9708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:28 smithi097 ceph-mon[32231]: pgmap v9708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:30 smithi115 ceph-mon[35364]: pgmap v9709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:30 smithi181 ceph-mon[35869]: pgmap v9709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:30 smithi097 ceph-mon[32231]: pgmap v9709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:31.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17228.082791 secs 2024-07-26T12:46:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17228.082791 secs 2024-07-26T12:46:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17228.082791 secs 2024-07-26T12:46:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:32 smithi115 ceph-mon[35364]: pgmap v9710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:32 smithi181 ceph-mon[35869]: pgmap v9710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:32 smithi097 ceph-mon[32231]: pgmap v9710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:34 smithi115 ceph-mon[35364]: pgmap v9711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:34 smithi181 ceph-mon[35869]: pgmap v9711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:34 smithi097 ceph-mon[32231]: pgmap v9711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: pgmap v9712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: pgmap v9713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:35.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:46:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: pgmap v9712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: pgmap v9713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:35.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:46:35.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:46:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:46:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: pgmap v9712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: pgmap v9713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:35.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:46:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17233.082927 secs 2024-07-26T12:46:36.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17233.082927 secs 2024-07-26T12:46:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17233.082927 secs 2024-07-26T12:46:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:37 smithi115 ceph-mon[35364]: pgmap v9714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:37 smithi181 ceph-mon[35869]: pgmap v9714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:37 smithi097 ceph-mon[32231]: pgmap v9714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:39.636+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:39.637+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:46:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:39.637+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:46:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:39.637+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:46:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:39.638+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:39 smithi115 ceph-mon[35364]: pgmap v9715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:39 smithi181 ceph-mon[35869]: pgmap v9715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:39 smithi097 ceph-mon[32231]: pgmap v9715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17238.083066 secs 2024-07-26T12:46:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:41 smithi115 ceph-mon[35364]: pgmap v9716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17238.083066 secs 2024-07-26T12:46:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:41 smithi181 ceph-mon[35869]: pgmap v9716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17238.083066 secs 2024-07-26T12:46:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:41 smithi097 ceph-mon[32231]: pgmap v9716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:43 smithi115 ceph-mon[35364]: pgmap v9717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:43 smithi181 ceph-mon[35869]: pgmap v9717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:43 smithi097 ceph-mon[32231]: pgmap v9717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:45 smithi115 ceph-mon[35364]: pgmap v9718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:45 smithi181 ceph-mon[35869]: pgmap v9718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:45 smithi097 ceph-mon[32231]: pgmap v9718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17243.083272 secs 2024-07-26T12:46:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17243.083272 secs 2024-07-26T12:46:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17243.083272 secs 2024-07-26T12:46:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:47 smithi115 ceph-mon[35364]: pgmap v9719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:47 smithi181 ceph-mon[35869]: pgmap v9719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:47 smithi097 ceph-mon[32231]: pgmap v9719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:49 smithi115 ceph-mon[35364]: pgmap v9720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:49 smithi181 ceph-mon[35869]: pgmap v9720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:49 smithi097 ceph-mon[32231]: pgmap v9720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17248.083403 secs 2024-07-26T12:46:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:51 smithi115 ceph-mon[35364]: pgmap v9721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17248.083403 secs 2024-07-26T12:46:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:51 smithi181 ceph-mon[35869]: pgmap v9721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17248.083403 secs 2024-07-26T12:46:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:51 smithi097 ceph-mon[32231]: pgmap v9721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:53 smithi115 ceph-mon[35364]: pgmap v9722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:53 smithi181 ceph-mon[35869]: pgmap v9722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:53 smithi097 ceph-mon[32231]: pgmap v9722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.641+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.642+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:46:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.642+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:46:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.642+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:46:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.642+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:54.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.970+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:46:54.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.970+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:46:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.971+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:46:54.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:46:54.971+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:46:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:55 smithi181 ceph-mon[35869]: pgmap v9723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:55 smithi097 ceph-mon[32231]: pgmap v9723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:55 smithi115 ceph-mon[35364]: pgmap v9723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17253.083536 secs 2024-07-26T12:46:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17253.083536 secs 2024-07-26T12:46:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17253.083536 secs 2024-07-26T12:46:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:57 smithi097 ceph-mon[32231]: pgmap v9724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:57 smithi115 ceph-mon[35364]: pgmap v9724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:57 smithi181 ceph-mon[35869]: pgmap v9724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:46:59 smithi097 ceph-mon[32231]: pgmap v9725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:00.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:46:59 smithi115 ceph-mon[35364]: pgmap v9725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:00.035 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:46:59 smithi181 ceph-mon[35869]: pgmap v9725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17258.083677 secs 2024-07-26T12:47:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:01 smithi115 ceph-mon[35364]: pgmap v9726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17258.083677 secs 2024-07-26T12:47:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:01 smithi181 ceph-mon[35869]: pgmap v9726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17258.083677 secs 2024-07-26T12:47:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:01 smithi097 ceph-mon[32231]: pgmap v9726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:03 smithi115 ceph-mon[35364]: pgmap v9727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:03 smithi181 ceph-mon[35869]: pgmap v9727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:03 smithi097 ceph-mon[32231]: pgmap v9727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:05 smithi115 ceph-mon[35364]: pgmap v9728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:05 smithi181 ceph-mon[35869]: pgmap v9728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:05 smithi097 ceph-mon[32231]: pgmap v9728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17263.083818 secs 2024-07-26T12:47:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17263.083818 secs 2024-07-26T12:47:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17263.083818 secs 2024-07-26T12:47:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:07 smithi115 ceph-mon[35364]: pgmap v9729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:07 smithi181 ceph-mon[35869]: pgmap v9729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:07 smithi097 ceph-mon[32231]: pgmap v9729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:09.651 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:09.649+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:09.651+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:47:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:09.651+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:47:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:09.651+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:47:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:09.651+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:09 smithi115 ceph-mon[35364]: pgmap v9730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:09 smithi181 ceph-mon[35869]: pgmap v9730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:09 smithi097 ceph-mon[32231]: pgmap v9730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17268.083957 secs 2024-07-26T12:47:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:11 smithi115 ceph-mon[35364]: pgmap v9731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17268.083957 secs 2024-07-26T12:47:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:11 smithi181 ceph-mon[35869]: pgmap v9731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17268.083957 secs 2024-07-26T12:47:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:11 smithi097 ceph-mon[32231]: pgmap v9731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:13 smithi115 ceph-mon[35364]: pgmap v9732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:13 smithi181 ceph-mon[35869]: pgmap v9732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:13 smithi097 ceph-mon[32231]: pgmap v9732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:15 smithi115 ceph-mon[35364]: pgmap v9733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:15 smithi181 ceph-mon[35869]: pgmap v9733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:15 smithi097 ceph-mon[32231]: pgmap v9733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17273.084117 secs 2024-07-26T12:47:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17273.084117 secs 2024-07-26T12:47:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17273.084117 secs 2024-07-26T12:47:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:17 smithi115 ceph-mon[35364]: pgmap v9734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:17 smithi181 ceph-mon[35869]: pgmap v9734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:17 smithi097 ceph-mon[32231]: pgmap v9734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:19 smithi115 ceph-mon[35364]: pgmap v9735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:19 smithi181 ceph-mon[35869]: pgmap v9735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:19 smithi097 ceph-mon[32231]: pgmap v9735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17278.084308 secs 2024-07-26T12:47:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:21 smithi115 ceph-mon[35364]: pgmap v9736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17278.084308 secs 2024-07-26T12:47:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:21 smithi181 ceph-mon[35869]: pgmap v9736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17278.084308 secs 2024-07-26T12:47:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:21 smithi097 ceph-mon[32231]: pgmap v9736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:23 smithi115 ceph-mon[35364]: pgmap v9737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:23 smithi181 ceph-mon[35869]: pgmap v9737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:23 smithi097 ceph-mon[32231]: pgmap v9737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:24.654+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:24.655+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:47:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:24.655+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:47:24.657 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:24.655+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:47:24.657 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:24.655+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:25 smithi115 ceph-mon[35364]: pgmap v9738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:25 smithi181 ceph-mon[35869]: pgmap v9738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:25 smithi097 ceph-mon[32231]: pgmap v9738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17283.084446 secs 2024-07-26T12:47:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17283.084446 secs 2024-07-26T12:47:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17283.084446 secs 2024-07-26T12:47:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:27 smithi115 ceph-mon[35364]: pgmap v9739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:27 smithi181 ceph-mon[35869]: pgmap v9739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:27 smithi097 ceph-mon[32231]: pgmap v9739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:29 smithi115 ceph-mon[35364]: pgmap v9740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:29 smithi181 ceph-mon[35869]: pgmap v9740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:29 smithi097 ceph-mon[32231]: pgmap v9740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17288.084592 secs 2024-07-26T12:47:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:31 smithi115 ceph-mon[35364]: pgmap v9741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17288.084592 secs 2024-07-26T12:47:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:31 smithi181 ceph-mon[35869]: pgmap v9741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17288.084592 secs 2024-07-26T12:47:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:31 smithi097 ceph-mon[32231]: pgmap v9741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:33 smithi115 ceph-mon[35364]: pgmap v9742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:33 smithi181 ceph-mon[35869]: pgmap v9742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:33 smithi097 ceph-mon[32231]: pgmap v9742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:35 smithi115 ceph-mon[35364]: pgmap v9743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:35 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:35 smithi181 ceph-mon[35869]: pgmap v9743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:35 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:35 smithi097 ceph-mon[32231]: pgmap v9743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:35 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17293.084670 secs 2024-07-26T12:47:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:37.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17293.084670 secs 2024-07-26T12:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:47:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17293.084670 secs 2024-07-26T12:47:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:47:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:37.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:47:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:37 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:47:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:37 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:47:38.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:37 smithi115 ceph-mon[35364]: pgmap v9744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:37 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:47:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:37 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:47:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:37 smithi181 ceph-mon[35869]: pgmap v9744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:37 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:47:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:37 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:47:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:37 smithi097 ceph-mon[32231]: pgmap v9744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:38 smithi115 ceph-mon[35364]: pgmap v9745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:47:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:38 smithi181 ceph-mon[35869]: pgmap v9745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:47:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:38 smithi097 ceph-mon[32231]: pgmap v9745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:47:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:39.658+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:39.659+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:47:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:39.659+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:47:39.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:39.659+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:47:39.661 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:39.659+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:41 smithi115 ceph-mon[35364]: pgmap v9746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:47:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17298.084762 secs 2024-07-26T12:47:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:41 smithi181 ceph-mon[35869]: pgmap v9746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:47:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17298.084762 secs 2024-07-26T12:47:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:41 smithi097 ceph-mon[32231]: pgmap v9746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:47:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17298.084762 secs 2024-07-26T12:47:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:43 smithi115 ceph-mon[35364]: pgmap v9747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:43 smithi181 ceph-mon[35869]: pgmap v9747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:43 smithi097 ceph-mon[32231]: pgmap v9747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:45 smithi115 ceph-mon[35364]: pgmap v9748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:47:45.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:45 smithi181 ceph-mon[35869]: pgmap v9748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:47:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:45 smithi097 ceph-mon[32231]: pgmap v9748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T12:47:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17303.084857 secs 2024-07-26T12:47:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17303.084857 secs 2024-07-26T12:47:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17303.084857 secs 2024-07-26T12:47:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:47 smithi115 ceph-mon[35364]: pgmap v9749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:47 smithi181 ceph-mon[35869]: pgmap v9749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:47 smithi097 ceph-mon[32231]: pgmap v9749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:47:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:49 smithi115 ceph-mon[35364]: pgmap v9750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:49 smithi181 ceph-mon[35869]: pgmap v9750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:49 smithi097 ceph-mon[32231]: pgmap v9750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:51 smithi115 ceph-mon[35364]: pgmap v9751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17308.084960 secs 2024-07-26T12:47:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:51 smithi181 ceph-mon[35869]: pgmap v9751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17308.084960 secs 2024-07-26T12:47:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:51 smithi097 ceph-mon[32231]: pgmap v9751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17308.084960 secs 2024-07-26T12:47:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:53 smithi115 ceph-mon[35364]: pgmap v9752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:53 smithi181 ceph-mon[35869]: pgmap v9752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:53 smithi097 ceph-mon[32231]: pgmap v9752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:54.662+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:54.663+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:47:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:54.663+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:47:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:54.663+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:47:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:47:54.663+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:55 smithi115 ceph-mon[35364]: pgmap v9753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:55 smithi181 ceph-mon[35869]: pgmap v9753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:55 smithi097 ceph-mon[32231]: pgmap v9753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17313.085107 secs 2024-07-26T12:47:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17313.085107 secs 2024-07-26T12:47:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17313.085107 secs 2024-07-26T12:47:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:57 smithi115 ceph-mon[35364]: pgmap v9754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:47:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:57 smithi181 ceph-mon[35869]: pgmap v9754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:47:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:57 smithi097 ceph-mon[32231]: pgmap v9754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:47:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:47:59 smithi115 ceph-mon[35364]: pgmap v9755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:47:59 smithi181 ceph-mon[35869]: pgmap v9755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:47:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:47:59 smithi097 ceph-mon[32231]: pgmap v9755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:01 smithi115 ceph-mon[35364]: pgmap v9756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17318.085297 secs 2024-07-26T12:48:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:01 smithi181 ceph-mon[35869]: pgmap v9756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17318.085297 secs 2024-07-26T12:48:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:01 smithi097 ceph-mon[32231]: pgmap v9756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17318.085297 secs 2024-07-26T12:48:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:03 smithi115 ceph-mon[35364]: pgmap v9757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:03 smithi181 ceph-mon[35869]: pgmap v9757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:03 smithi097 ceph-mon[32231]: pgmap v9757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:05 smithi115 ceph-mon[35364]: pgmap v9758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:05 smithi181 ceph-mon[35869]: pgmap v9758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:05 smithi097 ceph-mon[32231]: pgmap v9758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17323.085397 secs 2024-07-26T12:48:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17323.085397 secs 2024-07-26T12:48:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17323.085397 secs 2024-07-26T12:48:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:07 smithi181 ceph-mon[35869]: pgmap v9759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:07 smithi097 ceph-mon[32231]: pgmap v9759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:07 smithi115 ceph-mon[35364]: pgmap v9759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:09 smithi097 ceph-mon[32231]: pgmap v9760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:09.670+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:09.671+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:48:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:09.671+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:48:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:09.671+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:48:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:09.671+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:09 smithi115 ceph-mon[35364]: pgmap v9760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:09 smithi181 ceph-mon[35869]: pgmap v9760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:11 smithi097 ceph-mon[32231]: pgmap v9761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:48:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17328.085499 secs 2024-07-26T12:48:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:11 smithi115 ceph-mon[35364]: pgmap v9761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:48:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17328.085499 secs 2024-07-26T12:48:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:11 smithi181 ceph-mon[35869]: pgmap v9761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:48:11.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17328.085499 secs 2024-07-26T12:48:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:13 smithi097 ceph-mon[32231]: pgmap v9762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:48:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:13 smithi115 ceph-mon[35364]: pgmap v9762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:48:13.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:13 smithi181 ceph-mon[35869]: pgmap v9762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:48:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:15 smithi115 ceph-mon[35364]: pgmap v9763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:48:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:15 smithi181 ceph-mon[35869]: pgmap v9763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:48:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:15 smithi097 ceph-mon[32231]: pgmap v9763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:48:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17333.085587 secs 2024-07-26T12:48:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17333.085587 secs 2024-07-26T12:48:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17333.085587 secs 2024-07-26T12:48:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:17 smithi115 ceph-mon[35364]: pgmap v9764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:17 smithi181 ceph-mon[35869]: pgmap v9764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:17 smithi097 ceph-mon[32231]: pgmap v9764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:19 smithi115 ceph-mon[35364]: pgmap v9765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:48:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:19 smithi181 ceph-mon[35869]: pgmap v9765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:48:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:19 smithi097 ceph-mon[32231]: pgmap v9765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:48:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:21 smithi115 ceph-mon[35364]: pgmap v9766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:48:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17338.085743 secs 2024-07-26T12:48:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:21 smithi181 ceph-mon[35869]: pgmap v9766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:48:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17338.085743 secs 2024-07-26T12:48:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:21 smithi097 ceph-mon[32231]: pgmap v9766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:48:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17338.085743 secs 2024-07-26T12:48:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:23 smithi115 ceph-mon[35364]: pgmap v9767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:23 smithi181 ceph-mon[35869]: pgmap v9767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:23 smithi097 ceph-mon[32231]: pgmap v9767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:24.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:24.674+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:24.675+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:48:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:24.675+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:48:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:24.675+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:48:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:24.676+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:25 smithi115 ceph-mon[35364]: pgmap v9768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:25 smithi181 ceph-mon[35869]: pgmap v9768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:25 smithi097 ceph-mon[32231]: pgmap v9768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17343.085909 secs 2024-07-26T12:48:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17343.085909 secs 2024-07-26T12:48:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17343.085909 secs 2024-07-26T12:48:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:27 smithi115 ceph-mon[35364]: pgmap v9769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:27 smithi181 ceph-mon[35869]: pgmap v9769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:27 smithi097 ceph-mon[32231]: pgmap v9769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:29 smithi115 ceph-mon[35364]: pgmap v9770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:29 smithi181 ceph-mon[35869]: pgmap v9770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:29 smithi097 ceph-mon[32231]: pgmap v9770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:31.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:31 smithi115 ceph-mon[35364]: pgmap v9771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:31.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17348.086066 secs 2024-07-26T12:48:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:31 smithi181 ceph-mon[35869]: pgmap v9771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17348.086066 secs 2024-07-26T12:48:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:31 smithi097 ceph-mon[32231]: pgmap v9771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:31.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17348.086066 secs 2024-07-26T12:48:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:33 smithi115 ceph-mon[35364]: pgmap v9772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:33 smithi181 ceph-mon[35869]: pgmap v9772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:33 smithi097 ceph-mon[32231]: pgmap v9772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:35 smithi115 ceph-mon[35364]: pgmap v9773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:35 smithi181 ceph-mon[35869]: pgmap v9773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:35 smithi097 ceph-mon[32231]: pgmap v9773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17353.086170 secs 2024-07-26T12:48:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17353.086170 secs 2024-07-26T12:48:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17353.086170 secs 2024-07-26T12:48:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:37 smithi115 ceph-mon[35364]: pgmap v9774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:37.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:37 smithi181 ceph-mon[35869]: pgmap v9774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:37.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:37 smithi097 ceph-mon[32231]: pgmap v9774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:48:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:39.684+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:39.686+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:48:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:39.686+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:48:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:39.686+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:48:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:39.686+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:39 smithi115 ceph-mon[35364]: pgmap v9775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:39 smithi115 ceph-mon[35364]: pgmap v9776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:39 smithi181 ceph-mon[35869]: pgmap v9775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:39.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:39 smithi181 ceph-mon[35869]: pgmap v9776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:39 smithi097 ceph-mon[32231]: pgmap v9775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:39.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:39 smithi097 ceph-mon[32231]: pgmap v9776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:41 smithi115 ceph-mon[35364]: pgmap v9777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17358.086310 secs 2024-07-26T12:48:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:41 smithi181 ceph-mon[35869]: pgmap v9777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17358.086310 secs 2024-07-26T12:48:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:41 smithi097 ceph-mon[32231]: pgmap v9777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17358.086310 secs 2024-07-26T12:48:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:43 smithi115 ceph-mon[35364]: pgmap v9778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:43 smithi181 ceph-mon[35869]: pgmap v9778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:43 smithi097 ceph-mon[32231]: pgmap v9778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:45 smithi115 ceph-mon[35364]: pgmap v9779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:45 smithi181 ceph-mon[35869]: pgmap v9779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:45 smithi097 ceph-mon[32231]: pgmap v9779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17363.086448 secs 2024-07-26T12:48:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17363.086448 secs 2024-07-26T12:48:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17363.086448 secs 2024-07-26T12:48:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:47 smithi115 ceph-mon[35364]: pgmap v9780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:47 smithi181 ceph-mon[35869]: pgmap v9780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:47 smithi097 ceph-mon[32231]: pgmap v9780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:48:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:49 smithi115 ceph-mon[35364]: pgmap v9781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:49 smithi181 ceph-mon[35869]: pgmap v9781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:49 smithi097 ceph-mon[32231]: pgmap v9781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:51 smithi115 ceph-mon[35364]: pgmap v9782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17368.086593 secs 2024-07-26T12:48:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:51 smithi181 ceph-mon[35869]: pgmap v9782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:51.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17368.086593 secs 2024-07-26T12:48:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:51 smithi097 ceph-mon[32231]: pgmap v9782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17368.086593 secs 2024-07-26T12:48:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:53 smithi115 ceph-mon[35364]: pgmap v9783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:53 smithi181 ceph-mon[35869]: pgmap v9783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:53 smithi097 ceph-mon[32231]: pgmap v9783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:54.687+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:54.689+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:48:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:54.689+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:48:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:54.689+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:48:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:48:54.689+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:55 smithi115 ceph-mon[35364]: pgmap v9784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:55 smithi181 ceph-mon[35869]: pgmap v9784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:55 smithi097 ceph-mon[32231]: pgmap v9784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:48:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17373.086749 secs 2024-07-26T12:48:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17373.086749 secs 2024-07-26T12:48:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17373.086749 secs 2024-07-26T12:48:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:57 smithi115 ceph-mon[35364]: pgmap v9785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:57 smithi181 ceph-mon[35869]: pgmap v9785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:57 smithi097 ceph-mon[32231]: pgmap v9785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:48:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:48:59 smithi115 ceph-mon[35364]: pgmap v9786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:48:59 smithi181 ceph-mon[35869]: pgmap v9786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:48:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:48:59 smithi097 ceph-mon[32231]: pgmap v9786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:01 smithi115 ceph-mon[35364]: pgmap v9787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17378.086919 secs 2024-07-26T12:49:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:01 smithi181 ceph-mon[35869]: pgmap v9787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17378.086919 secs 2024-07-26T12:49:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:01 smithi097 ceph-mon[32231]: pgmap v9787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17378.086919 secs 2024-07-26T12:49:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:03 smithi181 ceph-mon[35869]: pgmap v9788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:03 smithi097 ceph-mon[32231]: pgmap v9788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:03 smithi115 ceph-mon[35364]: pgmap v9788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:05 smithi097 ceph-mon[32231]: pgmap v9789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:05 smithi115 ceph-mon[35364]: pgmap v9789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:05 smithi181 ceph-mon[35869]: pgmap v9789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17383.087053 secs 2024-07-26T12:49:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17383.087053 secs 2024-07-26T12:49:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17383.087053 secs 2024-07-26T12:49:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:07 smithi115 ceph-mon[35364]: pgmap v9790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:07 smithi181 ceph-mon[35869]: pgmap v9790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:07 smithi097 ceph-mon[32231]: pgmap v9790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.697+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.698+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.698+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.698+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.699+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:09 smithi115 ceph-mon[35364]: pgmap v9791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:09 smithi181 ceph-mon[35869]: pgmap v9791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.993+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:49:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.993+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:49:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.994+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:49:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:09.994+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:49:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:09 smithi097 ceph-mon[32231]: pgmap v9791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:11 smithi115 ceph-mon[35364]: pgmap v9792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17388.087186 secs 2024-07-26T12:49:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:11 smithi181 ceph-mon[35869]: pgmap v9792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17388.087186 secs 2024-07-26T12:49:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:11 smithi097 ceph-mon[32231]: pgmap v9792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17388.087186 secs 2024-07-26T12:49:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:13 smithi115 ceph-mon[35364]: pgmap v9793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:13 smithi181 ceph-mon[35869]: pgmap v9793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:13 smithi097 ceph-mon[32231]: pgmap v9793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:15 smithi115 ceph-mon[35364]: pgmap v9794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:15 smithi181 ceph-mon[35869]: pgmap v9794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:15 smithi097 ceph-mon[32231]: pgmap v9794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17393.087299 secs 2024-07-26T12:49:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17393.087299 secs 2024-07-26T12:49:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17393.087299 secs 2024-07-26T12:49:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:17 smithi115 ceph-mon[35364]: pgmap v9795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:17 smithi181 ceph-mon[35869]: pgmap v9795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:17 smithi097 ceph-mon[32231]: pgmap v9795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:19 smithi115 ceph-mon[35364]: pgmap v9796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:19 smithi181 ceph-mon[35869]: pgmap v9796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:19 smithi097 ceph-mon[32231]: pgmap v9796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:21 smithi115 ceph-mon[35364]: pgmap v9797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:21.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17398.087441 secs 2024-07-26T12:49:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:21 smithi181 ceph-mon[35869]: pgmap v9797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17398.087441 secs 2024-07-26T12:49:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:21 smithi097 ceph-mon[32231]: pgmap v9797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17398.087441 secs 2024-07-26T12:49:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:23 smithi115 ceph-mon[35364]: pgmap v9798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:23 smithi181 ceph-mon[35869]: pgmap v9798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:23 smithi097 ceph-mon[32231]: pgmap v9798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:24.702+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:24.703+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:49:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:24.703+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:49:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:24.703+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:49:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:24.703+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:25 smithi115 ceph-mon[35364]: pgmap v9799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:25 smithi181 ceph-mon[35869]: pgmap v9799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:25 smithi097 ceph-mon[32231]: pgmap v9799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17403.087574 secs 2024-07-26T12:49:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17403.087574 secs 2024-07-26T12:49:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17403.087574 secs 2024-07-26T12:49:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:27 smithi115 ceph-mon[35364]: pgmap v9800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:27 smithi181 ceph-mon[35869]: pgmap v9800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:27 smithi097 ceph-mon[32231]: pgmap v9800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:29 smithi115 ceph-mon[35364]: pgmap v9801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:29 smithi181 ceph-mon[35869]: pgmap v9801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:29 smithi097 ceph-mon[32231]: pgmap v9801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:31 smithi115 ceph-mon[35364]: pgmap v9802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17408.087713 secs 2024-07-26T12:49:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:31 smithi181 ceph-mon[35869]: pgmap v9802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17408.087713 secs 2024-07-26T12:49:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:31 smithi097 ceph-mon[32231]: pgmap v9802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17408.087713 secs 2024-07-26T12:49:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:33 smithi115 ceph-mon[35364]: pgmap v9803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:33.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:33 smithi181 ceph-mon[35869]: pgmap v9803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:33 smithi097 ceph-mon[32231]: pgmap v9803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:35 smithi115 ceph-mon[35364]: pgmap v9804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:35 smithi181 ceph-mon[35869]: pgmap v9804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:35 smithi097 ceph-mon[32231]: pgmap v9804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17413.087847 secs 2024-07-26T12:49:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17413.087847 secs 2024-07-26T12:49:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17413.087847 secs 2024-07-26T12:49:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:37 smithi115 ceph-mon[35364]: pgmap v9805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:37 smithi181 ceph-mon[35869]: pgmap v9805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:37 smithi097 ceph-mon[32231]: pgmap v9805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:39.711+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:39.712+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:49:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:39.712+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:49:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:39.712+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:49:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:39.712+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:39 smithi115 ceph-mon[35364]: pgmap v9806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:39.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:39 smithi181 ceph-mon[35869]: pgmap v9806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:39.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:39 smithi097 ceph-mon[32231]: pgmap v9806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:40.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:49:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:41 smithi115 ceph-mon[35364]: pgmap v9807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:41 smithi115 ceph-mon[35364]: pgmap v9808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:41.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17418.088008 secs 2024-07-26T12:49:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:41 smithi181 ceph-mon[35869]: pgmap v9807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:41 smithi181 ceph-mon[35869]: pgmap v9808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17418.088008 secs 2024-07-26T12:49:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:41 smithi097 ceph-mon[32231]: pgmap v9807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:41 smithi097 ceph-mon[32231]: pgmap v9808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17418.088008 secs 2024-07-26T12:49:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:43 smithi115 ceph-mon[35364]: pgmap v9809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:43.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:43 smithi181 ceph-mon[35869]: pgmap v9809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:43 smithi097 ceph-mon[32231]: pgmap v9809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:45 smithi115 ceph-mon[35364]: pgmap v9810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:45 smithi181 ceph-mon[35869]: pgmap v9810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:45 smithi097 ceph-mon[32231]: pgmap v9810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17423.088164 secs 2024-07-26T12:49:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17423.088164 secs 2024-07-26T12:49:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17423.088164 secs 2024-07-26T12:49:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:47 smithi115 ceph-mon[35364]: pgmap v9811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:47 smithi181 ceph-mon[35869]: pgmap v9811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:47 smithi097 ceph-mon[32231]: pgmap v9811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:49 smithi115 ceph-mon[35364]: pgmap v9812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:49 smithi181 ceph-mon[35869]: pgmap v9812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:49 smithi097 ceph-mon[32231]: pgmap v9812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:51 smithi115 ceph-mon[35364]: pgmap v9813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:49:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17428.088304 secs 2024-07-26T12:49:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:51 smithi181 ceph-mon[35869]: pgmap v9813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:49:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17428.088304 secs 2024-07-26T12:49:52.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:51 smithi097 ceph-mon[32231]: pgmap v9813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:49:52.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17428.088304 secs 2024-07-26T12:49:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:53 smithi115 ceph-mon[35364]: pgmap v9814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:53 smithi181 ceph-mon[35869]: pgmap v9814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:53 smithi097 ceph-mon[32231]: pgmap v9814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:54.714+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:54.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:54.716+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:49:54.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:54.716+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:49:54.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:54.716+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:49:54.717 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:49:54.716+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:55 smithi181 ceph-mon[35869]: pgmap v9815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:55 smithi097 ceph-mon[32231]: pgmap v9815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:55 smithi115 ceph-mon[35364]: pgmap v9815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17433.088453 secs 2024-07-26T12:49:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17433.088453 secs 2024-07-26T12:49:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17433.088453 secs 2024-07-26T12:49:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:57 smithi097 ceph-mon[32231]: pgmap v9816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:57 smithi115 ceph-mon[35364]: pgmap v9816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:57 smithi181 ceph-mon[35869]: pgmap v9816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:49:59 smithi097 ceph-mon[32231]: pgmap v9817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:49:59 smithi115 ceph-mon[35364]: pgmap v9817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:49:59 smithi181 ceph-mon[35869]: pgmap v9817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:01.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:01.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:01 smithi115 ceph-mon[35364]: pgmap v9818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17438.088593 secs 2024-07-26T12:50:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:01 smithi181 ceph-mon[35869]: pgmap v9818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17438.088593 secs 2024-07-26T12:50:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:01 smithi097 ceph-mon[32231]: pgmap v9818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17438.088593 secs 2024-07-26T12:50:04.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:03 smithi097 ceph-mon[32231]: pgmap v9819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:03 smithi115 ceph-mon[35364]: pgmap v9819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:03 smithi181 ceph-mon[35869]: pgmap v9819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:05 smithi115 ceph-mon[35364]: pgmap v9820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:05 smithi181 ceph-mon[35869]: pgmap v9820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:05 smithi097 ceph-mon[32231]: pgmap v9820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17443.088735 secs 2024-07-26T12:50:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17443.088735 secs 2024-07-26T12:50:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17443.088735 secs 2024-07-26T12:50:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:07 smithi115 ceph-mon[35364]: pgmap v9821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:07 smithi181 ceph-mon[35869]: pgmap v9821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:07 smithi097 ceph-mon[32231]: pgmap v9821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:09.725 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:09.724+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:09.725+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:09.725+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:09.725+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:09.727 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:09.725+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:09 smithi115 ceph-mon[35364]: pgmap v9822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:09 smithi181 ceph-mon[35869]: pgmap v9822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:09 smithi097 ceph-mon[32231]: pgmap v9822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:11 smithi115 ceph-mon[35364]: pgmap v9823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17448.088883 secs 2024-07-26T12:50:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:11 smithi181 ceph-mon[35869]: pgmap v9823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17448.088883 secs 2024-07-26T12:50:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:11 smithi097 ceph-mon[32231]: pgmap v9823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17448.088883 secs 2024-07-26T12:50:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:13 smithi115 ceph-mon[35364]: pgmap v9824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:13 smithi181 ceph-mon[35869]: pgmap v9824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:13 smithi097 ceph-mon[32231]: pgmap v9824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:15 smithi115 ceph-mon[35364]: pgmap v9825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:15 smithi181 ceph-mon[35869]: pgmap v9825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:15 smithi097 ceph-mon[32231]: pgmap v9825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17453.089052 secs 2024-07-26T12:50:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17453.089052 secs 2024-07-26T12:50:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17453.089052 secs 2024-07-26T12:50:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:17 smithi115 ceph-mon[35364]: pgmap v9826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:17 smithi181 ceph-mon[35869]: pgmap v9826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:17 smithi097 ceph-mon[32231]: pgmap v9826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:19 smithi115 ceph-mon[35364]: pgmap v9827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:19 smithi181 ceph-mon[35869]: pgmap v9827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:19 smithi097 ceph-mon[32231]: pgmap v9827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:21 smithi115 ceph-mon[35364]: pgmap v9828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17458.089214 secs 2024-07-26T12:50:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:21 smithi181 ceph-mon[35869]: pgmap v9828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17458.089214 secs 2024-07-26T12:50:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:21 smithi097 ceph-mon[32231]: pgmap v9828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17458.089214 secs 2024-07-26T12:50:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:23 smithi115 ceph-mon[35364]: pgmap v9829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:23 smithi181 ceph-mon[35869]: pgmap v9829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:23 smithi097 ceph-mon[32231]: pgmap v9829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:24.728+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:24.729+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:50:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:24.729+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:50:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:24.729+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:24.729+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:25 smithi115 ceph-mon[35364]: pgmap v9830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:25 smithi181 ceph-mon[35869]: pgmap v9830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:25 smithi097 ceph-mon[32231]: pgmap v9830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17463.089351 secs 2024-07-26T12:50:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17463.089351 secs 2024-07-26T12:50:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17463.089351 secs 2024-07-26T12:50:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:27 smithi115 ceph-mon[35364]: pgmap v9831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:27 smithi181 ceph-mon[35869]: pgmap v9831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:27 smithi097 ceph-mon[32231]: pgmap v9831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:30.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:29 smithi115 ceph-mon[35364]: pgmap v9832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:29 smithi181 ceph-mon[35869]: pgmap v9832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:29 smithi097 ceph-mon[32231]: pgmap v9832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:31 smithi115 ceph-mon[35364]: pgmap v9833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17468.089491 secs 2024-07-26T12:50:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:31 smithi181 ceph-mon[35869]: pgmap v9833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17468.089491 secs 2024-07-26T12:50:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:31 smithi097 ceph-mon[32231]: pgmap v9833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17468.089491 secs 2024-07-26T12:50:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:33 smithi115 ceph-mon[35364]: pgmap v9834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:33 smithi181 ceph-mon[35869]: pgmap v9834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:33 smithi097 ceph-mon[32231]: pgmap v9834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:35 smithi097 ceph-mon[32231]: pgmap v9835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:50:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:35 smithi115 ceph-mon[35364]: pgmap v9835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:50:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:35 smithi181 ceph-mon[35869]: pgmap v9835: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:50:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17473.089624 secs 2024-07-26T12:50:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17473.089624 secs 2024-07-26T12:50:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17473.089624 secs 2024-07-26T12:50:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:37 smithi115 ceph-mon[35364]: pgmap v9836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:37 smithi181 ceph-mon[35869]: pgmap v9836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:37 smithi097 ceph-mon[32231]: pgmap v9836: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:39.736+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:39.737+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:50:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:39.738+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:50:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:39.738+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:39.738+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:39 smithi115 ceph-mon[35364]: pgmap v9837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:50:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:39 smithi181 ceph-mon[35869]: pgmap v9837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:50:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:39 smithi097 ceph-mon[32231]: pgmap v9837: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:50:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:41 smithi115 ceph-mon[35364]: pgmap v9838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:50:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17478.089765 secs 2024-07-26T12:50:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:41 smithi181 ceph-mon[35869]: pgmap v9838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:50:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17478.089765 secs 2024-07-26T12:50:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:41 smithi097 ceph-mon[32231]: pgmap v9838: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:50:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17478.089765 secs 2024-07-26T12:50:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:43 smithi115 ceph-mon[35364]: pgmap v9839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:50:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:43 smithi181 ceph-mon[35869]: pgmap v9839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:50:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:43 smithi097 ceph-mon[32231]: pgmap v9839: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:50:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:45 smithi115 ceph-mon[35364]: pgmap v9840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:50:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:45 smithi181 ceph-mon[35869]: pgmap v9840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:50:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:45 smithi097 ceph-mon[32231]: pgmap v9840: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:50:46.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17483.089926 secs 2024-07-26T12:50:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17483.089926 secs 2024-07-26T12:50:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17483.089926 secs 2024-07-26T12:50:47.283 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:47 smithi097 ceph-mon[32231]: pgmap v9841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:47 smithi115 ceph-mon[35364]: pgmap v9841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:47 smithi181 ceph-mon[35869]: pgmap v9841: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:49.464 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:49 smithi097 ceph-mon[32231]: pgmap v9842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:49 smithi115 ceph-mon[35364]: pgmap v9842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:49 smithi181 ceph-mon[35869]: pgmap v9842: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:50 smithi115 ceph-mon[35364]: pgmap v9843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:50 smithi115 ceph-mon[35364]: pgmap v9844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:51.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:50 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:50 smithi181 ceph-mon[35869]: pgmap v9843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:50 smithi181 ceph-mon[35869]: pgmap v9844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:50 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:50 smithi097 ceph-mon[32231]: pgmap v9843: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:50 smithi097 ceph-mon[32231]: pgmap v9844: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:51.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:50 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:50:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17488.090065 secs 2024-07-26T12:50:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17488.090065 secs 2024-07-26T12:50:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17488.090065 secs 2024-07-26T12:50:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:52 smithi115 ceph-mon[35364]: pgmap v9845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:52 smithi181 ceph-mon[35869]: pgmap v9845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:52 smithi097 ceph-mon[32231]: pgmap v9845: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:54.742 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:54.741+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:54.742+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:50:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:54.742+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:50:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:54.742+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:50:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:50:54.742+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:54 smithi115 ceph-mon[35364]: pgmap v9846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:50:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:54 smithi181 ceph-mon[35869]: pgmap v9846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:50:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:54 smithi097 ceph-mon[32231]: pgmap v9846: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:50:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17493.090207 secs 2024-07-26T12:50:57.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:56 smithi115 ceph-mon[35364]: pgmap v9847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17493.090207 secs 2024-07-26T12:50:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:56 smithi181 ceph-mon[35869]: pgmap v9847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17493.090207 secs 2024-07-26T12:50:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:56 smithi097 ceph-mon[32231]: pgmap v9847: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:50:58 smithi115 ceph-mon[35364]: pgmap v9848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:50:58 smithi181 ceph-mon[35869]: pgmap v9848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:50:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:50:58 smithi097 ceph-mon[32231]: pgmap v9848: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:00 smithi115 ceph-mon[35364]: pgmap v9849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:00 smithi181 ceph-mon[35869]: pgmap v9849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:00 smithi097 ceph-mon[32231]: pgmap v9849: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17498.090351 secs 2024-07-26T12:51:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17498.090351 secs 2024-07-26T12:51:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17498.090351 secs 2024-07-26T12:51:02.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:02 smithi115 ceph-mon[35364]: pgmap v9850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:02.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:02 smithi181 ceph-mon[35869]: pgmap v9850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:02 smithi097 ceph-mon[32231]: pgmap v9850: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:04.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:04 smithi097 ceph-mon[32231]: pgmap v9851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:04.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:04 smithi115 ceph-mon[35364]: pgmap v9851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:04 smithi181 ceph-mon[35869]: pgmap v9851: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17503.090495 secs 2024-07-26T12:51:06.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:06 smithi097 ceph-mon[32231]: pgmap v9852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17503.090495 secs 2024-07-26T12:51:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:06 smithi115 ceph-mon[35364]: pgmap v9852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17503.090495 secs 2024-07-26T12:51:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:06 smithi181 ceph-mon[35869]: pgmap v9852: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:08 smithi115 ceph-mon[35364]: pgmap v9853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:08 smithi181 ceph-mon[35869]: pgmap v9853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:08 smithi097 ceph-mon[32231]: pgmap v9853: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:09.752 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:09.751+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:09.752+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:51:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:09.752+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:09.752+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:09.752+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:10 smithi115 ceph-mon[35364]: pgmap v9854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:10 smithi181 ceph-mon[35869]: pgmap v9854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:10 smithi097 ceph-mon[32231]: pgmap v9854: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17508.090636 secs 2024-07-26T12:51:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17508.090636 secs 2024-07-26T12:51:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17508.090636 secs 2024-07-26T12:51:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:12 smithi115 ceph-mon[35364]: pgmap v9855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:12 smithi181 ceph-mon[35869]: pgmap v9855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:12 smithi097 ceph-mon[32231]: pgmap v9855: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:14.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:14 smithi115 ceph-mon[35364]: pgmap v9856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:14 smithi181 ceph-mon[35869]: pgmap v9856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:14 smithi097 ceph-mon[32231]: pgmap v9856: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17513.090775 secs 2024-07-26T12:51:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:16 smithi115 ceph-mon[35364]: pgmap v9857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17513.090775 secs 2024-07-26T12:51:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:16 smithi181 ceph-mon[35869]: pgmap v9857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17513.090775 secs 2024-07-26T12:51:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:16 smithi097 ceph-mon[32231]: pgmap v9857: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:18 smithi115 ceph-mon[35364]: pgmap v9858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:18 smithi181 ceph-mon[35869]: pgmap v9858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:18 smithi097 ceph-mon[32231]: pgmap v9858: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:20 smithi115 ceph-mon[35364]: pgmap v9859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:20 smithi181 ceph-mon[35869]: pgmap v9859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:20 smithi097 ceph-mon[32231]: pgmap v9859: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17518.090921 secs 2024-07-26T12:51:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17518.090921 secs 2024-07-26T12:51:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17518.090921 secs 2024-07-26T12:51:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:22 smithi115 ceph-mon[35364]: pgmap v9860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:22 smithi181 ceph-mon[35869]: pgmap v9860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:22 smithi097 ceph-mon[32231]: pgmap v9860: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:24 smithi115 ceph-mon[35364]: pgmap v9861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:24 smithi181 ceph-mon[35869]: pgmap v9861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:24 smithi097 ceph-mon[32231]: pgmap v9861: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:24.755+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:24.756+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:51:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:24.756+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:24.756+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:24.756+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17523.091026 secs 2024-07-26T12:51:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:26 smithi115 ceph-mon[35364]: pgmap v9862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17523.091026 secs 2024-07-26T12:51:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:26 smithi181 ceph-mon[35869]: pgmap v9862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17523.091026 secs 2024-07-26T12:51:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:26 smithi097 ceph-mon[32231]: pgmap v9862: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:28 smithi115 ceph-mon[35364]: pgmap v9863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:28.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:28 smithi181 ceph-mon[35869]: pgmap v9863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:28 smithi097 ceph-mon[32231]: pgmap v9863: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:30 smithi115 ceph-mon[35364]: pgmap v9864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:30 smithi181 ceph-mon[35869]: pgmap v9864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:30 smithi097 ceph-mon[32231]: pgmap v9864: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17528.091174 secs 2024-07-26T12:51:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17528.091174 secs 2024-07-26T12:51:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17528.091174 secs 2024-07-26T12:51:32.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:32 smithi115 ceph-mon[35364]: pgmap v9865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:32.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:32 smithi181 ceph-mon[35869]: pgmap v9865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:32 smithi097 ceph-mon[32231]: pgmap v9865: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:34.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:34 smithi097 ceph-mon[32231]: pgmap v9866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:34 smithi115 ceph-mon[35364]: pgmap v9866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:34 smithi181 ceph-mon[35869]: pgmap v9866: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17533.091311 secs 2024-07-26T12:51:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:36 smithi115 ceph-mon[35364]: pgmap v9867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17533.091311 secs 2024-07-26T12:51:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:36 smithi181 ceph-mon[35869]: pgmap v9867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17533.091311 secs 2024-07-26T12:51:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:36 smithi097 ceph-mon[32231]: pgmap v9867: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:38 smithi115 ceph-mon[35364]: pgmap v9868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:38 smithi181 ceph-mon[35869]: pgmap v9868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:38 smithi097 ceph-mon[32231]: pgmap v9868: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:39.764+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:39.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:51:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:39.765+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:39.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:39.765+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:40 smithi115 ceph-mon[35364]: pgmap v9869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:40 smithi181 ceph-mon[35869]: pgmap v9869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:40 smithi097 ceph-mon[32231]: pgmap v9869: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17538.091470 secs 2024-07-26T12:51:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17538.091470 secs 2024-07-26T12:51:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17538.091470 secs 2024-07-26T12:51:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:43 smithi115 ceph-mon[35364]: pgmap v9870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:51:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:43 smithi181 ceph-mon[35869]: pgmap v9870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:51:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:43 smithi097 ceph-mon[32231]: pgmap v9870: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:51:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:45 smithi115 ceph-mon[35364]: pgmap v9871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:45 smithi181 ceph-mon[35869]: pgmap v9871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:45 smithi097 ceph-mon[32231]: pgmap v9871: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17543.091615 secs 2024-07-26T12:51:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17543.091615 secs 2024-07-26T12:51:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17543.091615 secs 2024-07-26T12:51:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:47 smithi115 ceph-mon[35364]: pgmap v9872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:47 smithi181 ceph-mon[35869]: pgmap v9872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:47 smithi097 ceph-mon[32231]: pgmap v9872: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:51:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:49 smithi115 ceph-mon[35364]: pgmap v9873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:49 smithi181 ceph-mon[35869]: pgmap v9873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:49 smithi097 ceph-mon[32231]: pgmap v9873: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:51 smithi115 ceph-mon[35364]: pgmap v9874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17548.091755 secs 2024-07-26T12:51:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:51 smithi181 ceph-mon[35869]: pgmap v9874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17548.091755 secs 2024-07-26T12:51:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:51 smithi097 ceph-mon[32231]: pgmap v9874: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17548.091755 secs 2024-07-26T12:51:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:53.201 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:53 smithi115 ceph-mon[35364]: pgmap v9875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:53.439 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:53 smithi181 ceph-mon[35869]: pgmap v9875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:53 smithi097 ceph-mon[32231]: pgmap v9875: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:54.768+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:54.769+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:51:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:54.769+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:54.769+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:51:54.769+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:55.341 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:55 smithi115 ceph-mon[35364]: pgmap v9876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:55 smithi181 ceph-mon[35869]: pgmap v9876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:55 smithi097 ceph-mon[32231]: pgmap v9876: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:56.467 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17553.091858 secs 2024-07-26T12:51:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17553.091858 secs 2024-07-26T12:51:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17553.091858 secs 2024-07-26T12:51:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: pgmap v9877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: pgmap v9878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: pgmap v9879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:51:57.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: pgmap v9877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: pgmap v9878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:51:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: pgmap v9879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:51:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: pgmap v9877: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: pgmap v9878: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: pgmap v9879: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:51:57.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:51:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:51:59 smithi115 ceph-mon[35364]: pgmap v9880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:51:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:51:59 smithi181 ceph-mon[35869]: pgmap v9880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:52:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:51:59 smithi097 ceph-mon[32231]: pgmap v9880: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T12:52:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17558.091999 secs 2024-07-26T12:52:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:01 smithi115 ceph-mon[35364]: pgmap v9881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17558.091999 secs 2024-07-26T12:52:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:01 smithi181 ceph-mon[35869]: pgmap v9881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17558.091999 secs 2024-07-26T12:52:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:01 smithi097 ceph-mon[32231]: pgmap v9881: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:03 smithi115 ceph-mon[35364]: pgmap v9882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:03 smithi181 ceph-mon[35869]: pgmap v9882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:03 smithi097 ceph-mon[32231]: pgmap v9882: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:05 smithi115 ceph-mon[35364]: pgmap v9883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:05 smithi181 ceph-mon[35869]: pgmap v9883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:05 smithi097 ceph-mon[32231]: pgmap v9883: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:52:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17563.092136 secs 2024-07-26T12:52:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17563.092136 secs 2024-07-26T12:52:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17563.092136 secs 2024-07-26T12:52:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:07 smithi115 ceph-mon[35364]: pgmap v9884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:07 smithi181 ceph-mon[35869]: pgmap v9884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:07 smithi097 ceph-mon[32231]: pgmap v9884: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:09.777+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:09.779+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:52:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:09.779+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:09.779+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:09.779+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:09 smithi115 ceph-mon[35364]: pgmap v9885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:09 smithi181 ceph-mon[35869]: pgmap v9885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:09 smithi097 ceph-mon[32231]: pgmap v9885: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:10.023+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:52:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:10.023+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:52:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:10.023+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:52:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:10.023+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:52:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17568.092293 secs 2024-07-26T12:52:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:11 smithi115 ceph-mon[35364]: pgmap v9886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17568.092293 secs 2024-07-26T12:52:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:11 smithi181 ceph-mon[35869]: pgmap v9886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17568.092293 secs 2024-07-26T12:52:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:11 smithi097 ceph-mon[32231]: pgmap v9886: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:13 smithi115 ceph-mon[35364]: pgmap v9887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:13 smithi181 ceph-mon[35869]: pgmap v9887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:13 smithi097 ceph-mon[32231]: pgmap v9887: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:15 smithi115 ceph-mon[35364]: pgmap v9888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:15 smithi181 ceph-mon[35869]: pgmap v9888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:15 smithi097 ceph-mon[32231]: pgmap v9888: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:16.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17573.092429 secs 2024-07-26T12:52:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17573.092429 secs 2024-07-26T12:52:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17573.092429 secs 2024-07-26T12:52:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:17 smithi181 ceph-mon[35869]: pgmap v9889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:17 smithi097 ceph-mon[32231]: pgmap v9889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:17 smithi115 ceph-mon[35364]: pgmap v9889: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:19 smithi097 ceph-mon[32231]: pgmap v9890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:19 smithi115 ceph-mon[35364]: pgmap v9890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:19 smithi181 ceph-mon[35869]: pgmap v9890: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17578.092570 secs 2024-07-26T12:52:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:21 smithi097 ceph-mon[32231]: pgmap v9891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17578.092570 secs 2024-07-26T12:52:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:21 smithi115 ceph-mon[35364]: pgmap v9891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17578.092570 secs 2024-07-26T12:52:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:21 smithi181 ceph-mon[35869]: pgmap v9891: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:23 smithi181 ceph-mon[35869]: pgmap v9892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:23 smithi097 ceph-mon[32231]: pgmap v9892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:23 smithi115 ceph-mon[35364]: pgmap v9892: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:24.783+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:24.783+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:52:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:24.783+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:24.783+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:24.783+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:25 smithi115 ceph-mon[35364]: pgmap v9893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:25 smithi181 ceph-mon[35869]: pgmap v9893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:25 smithi097 ceph-mon[32231]: pgmap v9893: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17583.092712 secs 2024-07-26T12:52:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17583.092712 secs 2024-07-26T12:52:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17583.092712 secs 2024-07-26T12:52:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:27 smithi115 ceph-mon[35364]: pgmap v9894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:27 smithi181 ceph-mon[35869]: pgmap v9894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:27 smithi097 ceph-mon[32231]: pgmap v9894: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:29 smithi115 ceph-mon[35364]: pgmap v9895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:29 smithi181 ceph-mon[35869]: pgmap v9895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:29 smithi097 ceph-mon[32231]: pgmap v9895: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17588.092871 secs 2024-07-26T12:52:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:31 smithi115 ceph-mon[35364]: pgmap v9896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17588.092871 secs 2024-07-26T12:52:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:31 smithi181 ceph-mon[35869]: pgmap v9896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17588.092871 secs 2024-07-26T12:52:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:31 smithi097 ceph-mon[32231]: pgmap v9896: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:33 smithi115 ceph-mon[35364]: pgmap v9897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:33 smithi181 ceph-mon[35869]: pgmap v9897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:33 smithi097 ceph-mon[32231]: pgmap v9897: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:35 smithi115 ceph-mon[35364]: pgmap v9898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:35 smithi181 ceph-mon[35869]: pgmap v9898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:35 smithi097 ceph-mon[32231]: pgmap v9898: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17593.092999 secs 2024-07-26T12:52:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17593.092999 secs 2024-07-26T12:52:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17593.092999 secs 2024-07-26T12:52:37.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:37 smithi181 ceph-mon[35869]: pgmap v9899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:37 smithi097 ceph-mon[32231]: pgmap v9899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:37 smithi115 ceph-mon[35364]: pgmap v9899: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:39.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:39.790+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:39.791+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:52:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:39.791+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:39.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:39.791+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:39.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:39.791+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:39 smithi181 ceph-mon[35869]: pgmap v9900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:40.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:39 smithi097 ceph-mon[32231]: pgmap v9900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:39 smithi115 ceph-mon[35364]: pgmap v9900: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17598.093142 secs 2024-07-26T12:52:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:41 smithi097 ceph-mon[32231]: pgmap v9901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17598.093142 secs 2024-07-26T12:52:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:41 smithi115 ceph-mon[35364]: pgmap v9901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17598.093142 secs 2024-07-26T12:52:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:41 smithi181 ceph-mon[35869]: pgmap v9901: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:43 smithi097 ceph-mon[32231]: pgmap v9902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:43 smithi115 ceph-mon[35364]: pgmap v9902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:43 smithi181 ceph-mon[35869]: pgmap v9902: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:45 smithi097 ceph-mon[32231]: pgmap v9903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:45 smithi115 ceph-mon[35364]: pgmap v9903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:45 smithi181 ceph-mon[35869]: pgmap v9903: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17603.093285 secs 2024-07-26T12:52:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17603.093285 secs 2024-07-26T12:52:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17603.093285 secs 2024-07-26T12:52:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:47 smithi115 ceph-mon[35364]: pgmap v9904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:48.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:47 smithi181 ceph-mon[35869]: pgmap v9904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:47 smithi097 ceph-mon[32231]: pgmap v9904: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:49 smithi115 ceph-mon[35364]: pgmap v9905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:49 smithi181 ceph-mon[35869]: pgmap v9905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:49 smithi097 ceph-mon[32231]: pgmap v9905: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17608.093429 secs 2024-07-26T12:52:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:51 smithi115 ceph-mon[35364]: pgmap v9906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17608.093429 secs 2024-07-26T12:52:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:51 smithi181 ceph-mon[35869]: pgmap v9906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17608.093429 secs 2024-07-26T12:52:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:51 smithi097 ceph-mon[32231]: pgmap v9906: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:53 smithi115 ceph-mon[35364]: pgmap v9907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:53 smithi181 ceph-mon[35869]: pgmap v9907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:53 smithi097 ceph-mon[32231]: pgmap v9907: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:52:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:54.794+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:54.795+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:52:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:54.795+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:54.795+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:52:54.795+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:55 smithi115 ceph-mon[35364]: pgmap v9908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:55 smithi181 ceph-mon[35869]: pgmap v9908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:55 smithi097 ceph-mon[32231]: pgmap v9908: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17613.093566 secs 2024-07-26T12:52:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17613.093566 secs 2024-07-26T12:52:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17613.093566 secs 2024-07-26T12:52:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:57 smithi115 ceph-mon[35364]: pgmap v9909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:57 smithi181 ceph-mon[35869]: pgmap v9909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:57 smithi097 ceph-mon[32231]: pgmap v9909: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:58 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:52:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:58 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:52:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:58 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:53:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:59 smithi115 ceph-mon[35364]: pgmap v9910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:53:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:52:59 smithi115 ceph-mon[35364]: pgmap v9911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:53:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:59 smithi181 ceph-mon[35869]: pgmap v9910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:53:00.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:52:59 smithi181 ceph-mon[35869]: pgmap v9911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:53:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:59 smithi097 ceph-mon[32231]: pgmap v9910: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:53:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:52:59 smithi097 ceph-mon[32231]: pgmap v9911: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:53:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:01 smithi115 ceph-mon[35364]: pgmap v9912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:53:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17618.093711 secs 2024-07-26T12:53:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:01 smithi181 ceph-mon[35869]: pgmap v9912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:53:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17618.093711 secs 2024-07-26T12:53:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:01 smithi097 ceph-mon[32231]: pgmap v9912: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:53:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17618.093711 secs 2024-07-26T12:53:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:03 smithi115 ceph-mon[35364]: pgmap v9913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:03 smithi181 ceph-mon[35869]: pgmap v9913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:03 smithi097 ceph-mon[32231]: pgmap v9913: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:05 smithi115 ceph-mon[35364]: pgmap v9914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:05 smithi181 ceph-mon[35869]: pgmap v9914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:05 smithi097 ceph-mon[32231]: pgmap v9914: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17623.093840 secs 2024-07-26T12:53:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17623.093840 secs 2024-07-26T12:53:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17623.093840 secs 2024-07-26T12:53:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:07 smithi115 ceph-mon[35364]: pgmap v9915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:53:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:07 smithi181 ceph-mon[35869]: pgmap v9915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:53:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:07 smithi097 ceph-mon[32231]: pgmap v9915: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:53:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:09.803+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:09.804+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:53:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:09.804+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:09.804+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:09.805+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:09 smithi115 ceph-mon[35364]: pgmap v9916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:53:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:09 smithi181 ceph-mon[35869]: pgmap v9916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:53:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:09 smithi097 ceph-mon[32231]: pgmap v9916: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:53:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:11 smithi115 ceph-mon[35364]: pgmap v9917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17628.093998 secs 2024-07-26T12:53:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:11 smithi181 ceph-mon[35869]: pgmap v9917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17628.093998 secs 2024-07-26T12:53:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:11 smithi097 ceph-mon[32231]: pgmap v9917: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17628.093998 secs 2024-07-26T12:53:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:13 smithi115 ceph-mon[35364]: pgmap v9918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:13 smithi181 ceph-mon[35869]: pgmap v9918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:13 smithi097 ceph-mon[32231]: pgmap v9918: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:15 smithi181 ceph-mon[35869]: pgmap v9919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:15 smithi115 ceph-mon[35364]: pgmap v9919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:15 smithi097 ceph-mon[32231]: pgmap v9919: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17633.094178 secs 2024-07-26T12:53:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17633.094178 secs 2024-07-26T12:53:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17633.094178 secs 2024-07-26T12:53:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:17 smithi115 ceph-mon[35364]: pgmap v9920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:17 smithi181 ceph-mon[35869]: pgmap v9920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:17 smithi097 ceph-mon[32231]: pgmap v9920: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:19 smithi115 ceph-mon[35364]: pgmap v9921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:19 smithi181 ceph-mon[35869]: pgmap v9921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:19 smithi097 ceph-mon[32231]: pgmap v9921: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:21 smithi115 ceph-mon[35364]: pgmap v9922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17638.094343 secs 2024-07-26T12:53:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:21 smithi181 ceph-mon[35869]: pgmap v9922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17638.094343 secs 2024-07-26T12:53:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:21 smithi097 ceph-mon[32231]: pgmap v9922: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17638.094343 secs 2024-07-26T12:53:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:23 smithi115 ceph-mon[35364]: pgmap v9923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:24.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:23 smithi181 ceph-mon[35869]: pgmap v9923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:23 smithi097 ceph-mon[32231]: pgmap v9923: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:24.808+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:24.809+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:53:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:24.809+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:24.809+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:24.809+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:25 smithi115 ceph-mon[35364]: pgmap v9924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:25 smithi181 ceph-mon[35869]: pgmap v9924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:25 smithi097 ceph-mon[32231]: pgmap v9924: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17643.094482 secs 2024-07-26T12:53:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17643.094482 secs 2024-07-26T12:53:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17643.094482 secs 2024-07-26T12:53:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:27 smithi115 ceph-mon[35364]: pgmap v9925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:27 smithi181 ceph-mon[35869]: pgmap v9925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:27 smithi097 ceph-mon[32231]: pgmap v9925: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:29 smithi115 ceph-mon[35364]: pgmap v9926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:53:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:29 smithi181 ceph-mon[35869]: pgmap v9926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:53:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:29 smithi097 ceph-mon[32231]: pgmap v9926: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:53:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:31 smithi115 ceph-mon[35364]: pgmap v9927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:53:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17648.094616 secs 2024-07-26T12:53:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:31 smithi181 ceph-mon[35869]: pgmap v9927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:53:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17648.094616 secs 2024-07-26T12:53:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:31 smithi097 ceph-mon[32231]: pgmap v9927: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:53:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17648.094616 secs 2024-07-26T12:53:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:33 smithi181 ceph-mon[35869]: pgmap v9928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:33 smithi097 ceph-mon[32231]: pgmap v9928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:33 smithi115 ceph-mon[35364]: pgmap v9928: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:35 smithi097 ceph-mon[32231]: pgmap v9929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:53:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:35 smithi115 ceph-mon[35364]: pgmap v9929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:53:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:35 smithi181 ceph-mon[35869]: pgmap v9929: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:53:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17653.094763 secs 2024-07-26T12:53:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17653.094763 secs 2024-07-26T12:53:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17653.094763 secs 2024-07-26T12:53:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:38 smithi097 ceph-mon[32231]: pgmap v9930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:38 smithi115 ceph-mon[35364]: pgmap v9930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:38.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:38 smithi181 ceph-mon[35869]: pgmap v9930: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:39 smithi115 ceph-mon[35364]: pgmap v9931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:39 smithi181 ceph-mon[35869]: pgmap v9931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:39 smithi097 ceph-mon[32231]: pgmap v9931: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:39.816+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:39.817+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:53:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:39.817+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:39.817+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:39.817+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:41 smithi115 ceph-mon[35364]: pgmap v9932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17658.094906 secs 2024-07-26T12:53:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:41 smithi181 ceph-mon[35869]: pgmap v9932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17658.094906 secs 2024-07-26T12:53:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:41 smithi097 ceph-mon[32231]: pgmap v9932: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17658.094906 secs 2024-07-26T12:53:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:43 smithi115 ceph-mon[35364]: pgmap v9933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:43 smithi181 ceph-mon[35869]: pgmap v9933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:43 smithi097 ceph-mon[32231]: pgmap v9933: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:45 smithi115 ceph-mon[35364]: pgmap v9934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:45 smithi181 ceph-mon[35869]: pgmap v9934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:45 smithi097 ceph-mon[32231]: pgmap v9934: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17663.095004 secs 2024-07-26T12:53:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17663.095004 secs 2024-07-26T12:53:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17663.095004 secs 2024-07-26T12:53:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:47 smithi115 ceph-mon[35364]: pgmap v9935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:47 smithi181 ceph-mon[35869]: pgmap v9935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:47 smithi097 ceph-mon[32231]: pgmap v9935: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:49 smithi115 ceph-mon[35364]: pgmap v9936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:49 smithi181 ceph-mon[35869]: pgmap v9936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:49 smithi097 ceph-mon[32231]: pgmap v9936: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:51 smithi115 ceph-mon[35364]: pgmap v9937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17668.095100 secs 2024-07-26T12:53:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:51 smithi181 ceph-mon[35869]: pgmap v9937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17668.095100 secs 2024-07-26T12:53:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:51 smithi097 ceph-mon[32231]: pgmap v9937: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17668.095100 secs 2024-07-26T12:53:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:53 smithi115 ceph-mon[35364]: pgmap v9938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:53 smithi181 ceph-mon[35869]: pgmap v9938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:53 smithi097 ceph-mon[32231]: pgmap v9938: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:54.820+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:54.822+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:53:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:54.822+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:54.822+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:53:54.822+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:55 smithi115 ceph-mon[35364]: pgmap v9939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:55 smithi181 ceph-mon[35869]: pgmap v9939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:55 smithi097 ceph-mon[32231]: pgmap v9939: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17673.095270 secs 2024-07-26T12:53:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17673.095270 secs 2024-07-26T12:53:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17673.095270 secs 2024-07-26T12:53:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:57 smithi115 ceph-mon[35364]: pgmap v9940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:57 smithi181 ceph-mon[35869]: pgmap v9940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:57 smithi097 ceph-mon[32231]: pgmap v9940: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:59 smithi115 ceph-mon[35364]: pgmap v9941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:53:59 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:59 smithi181 ceph-mon[35869]: pgmap v9941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:59.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:53:59 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:59 smithi097 ceph-mon[32231]: pgmap v9941: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:53:59 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:00 smithi115 ceph-mon[35364]: pgmap v9942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:00 smithi115 ceph-mon[35364]: pgmap v9943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:00.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:54:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:00 smithi181 ceph-mon[35869]: pgmap v9942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:00 smithi181 ceph-mon[35869]: pgmap v9943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:54:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:00 smithi097 ceph-mon[32231]: pgmap v9942: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:00 smithi097 ceph-mon[32231]: pgmap v9943: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:54:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17678.095411 secs 2024-07-26T12:54:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17678.095411 secs 2024-07-26T12:54:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17678.095411 secs 2024-07-26T12:54:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:02 smithi115 ceph-mon[35364]: pgmap v9944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:02 smithi181 ceph-mon[35869]: pgmap v9944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:02 smithi097 ceph-mon[32231]: pgmap v9944: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:04 smithi115 ceph-mon[35364]: pgmap v9945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:04 smithi181 ceph-mon[35869]: pgmap v9945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:04 smithi097 ceph-mon[32231]: pgmap v9945: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:06.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17683.095542 secs 2024-07-26T12:54:06.805 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:06 smithi097 ceph-mon[32231]: pgmap v9946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17683.095542 secs 2024-07-26T12:54:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:06 smithi115 ceph-mon[35364]: pgmap v9946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17683.095542 secs 2024-07-26T12:54:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:06 smithi181 ceph-mon[35869]: pgmap v9946: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:08 smithi115 ceph-mon[35364]: pgmap v9947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:08 smithi181 ceph-mon[35869]: pgmap v9947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:08 smithi097 ceph-mon[32231]: pgmap v9947: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:09.826+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:54:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:09.827+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:54:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:09.827+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:54:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:09.827+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:54:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:09.827+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:10 smithi115 ceph-mon[35364]: pgmap v9948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:10 smithi181 ceph-mon[35869]: pgmap v9948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:10 smithi097 ceph-mon[32231]: pgmap v9948: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17688.095682 secs 2024-07-26T12:54:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17688.095682 secs 2024-07-26T12:54:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17688.095682 secs 2024-07-26T12:54:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:12 smithi115 ceph-mon[35364]: pgmap v9949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:12 smithi181 ceph-mon[35869]: pgmap v9949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:12 smithi097 ceph-mon[32231]: pgmap v9949: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:14.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:14 smithi115 ceph-mon[35364]: pgmap v9950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:14 smithi181 ceph-mon[35869]: pgmap v9950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:14 smithi097 ceph-mon[32231]: pgmap v9950: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17693.095815 secs 2024-07-26T12:54:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:16 smithi115 ceph-mon[35364]: pgmap v9951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17693.095815 secs 2024-07-26T12:54:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:16 smithi181 ceph-mon[35869]: pgmap v9951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17693.095815 secs 2024-07-26T12:54:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:16 smithi097 ceph-mon[32231]: pgmap v9951: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:18 smithi115 ceph-mon[35364]: pgmap v9952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:18.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:18 smithi181 ceph-mon[35869]: pgmap v9952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:18 smithi097 ceph-mon[32231]: pgmap v9952: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:20 smithi115 ceph-mon[35364]: pgmap v9953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:20 smithi181 ceph-mon[35869]: pgmap v9953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:20 smithi097 ceph-mon[32231]: pgmap v9953: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17698.095956 secs 2024-07-26T12:54:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17698.095956 secs 2024-07-26T12:54:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17698.095956 secs 2024-07-26T12:54:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:22 smithi115 ceph-mon[35364]: pgmap v9954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:22 smithi181 ceph-mon[35869]: pgmap v9954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:22 smithi097 ceph-mon[32231]: pgmap v9954: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:24.829+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:54:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:24.831+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:54:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:24.831+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:54:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:24.831+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:54:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:24.831+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:24 smithi115 ceph-mon[35364]: pgmap v9955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:24 smithi181 ceph-mon[35869]: pgmap v9955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:24 smithi097 ceph-mon[32231]: pgmap v9955: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17703.096098 secs 2024-07-26T12:54:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:26 smithi115 ceph-mon[35364]: pgmap v9956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17703.096098 secs 2024-07-26T12:54:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:26 smithi181 ceph-mon[35869]: pgmap v9956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17703.096098 secs 2024-07-26T12:54:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:26 smithi097 ceph-mon[32231]: pgmap v9956: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:28 smithi115 ceph-mon[35364]: pgmap v9957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:28 smithi181 ceph-mon[35869]: pgmap v9957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:28 smithi097 ceph-mon[32231]: pgmap v9957: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:30.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:30 smithi115 ceph-mon[35364]: pgmap v9958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:30 smithi181 ceph-mon[35869]: pgmap v9958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:30 smithi097 ceph-mon[32231]: pgmap v9958: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17708.096268 secs 2024-07-26T12:54:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17708.096268 secs 2024-07-26T12:54:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17708.096268 secs 2024-07-26T12:54:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:32 smithi115 ceph-mon[35364]: pgmap v9959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:32 smithi181 ceph-mon[35869]: pgmap v9959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:32 smithi097 ceph-mon[32231]: pgmap v9959: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:34 smithi115 ceph-mon[35364]: pgmap v9960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:34 smithi181 ceph-mon[35869]: pgmap v9960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:34 smithi097 ceph-mon[32231]: pgmap v9960: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17713.096405 secs 2024-07-26T12:54:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:36 smithi115 ceph-mon[35364]: pgmap v9961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17713.096405 secs 2024-07-26T12:54:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:36 smithi181 ceph-mon[35869]: pgmap v9961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17713.096405 secs 2024-07-26T12:54:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:36 smithi097 ceph-mon[32231]: pgmap v9961: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:38 smithi115 ceph-mon[35364]: pgmap v9962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:38 smithi181 ceph-mon[35869]: pgmap v9962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:38 smithi097 ceph-mon[32231]: pgmap v9962: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:39.839+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:54:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:39.841+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:54:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:39.841+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:54:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:39.841+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:54:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:39.841+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:40 smithi115 ceph-mon[35364]: pgmap v9963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:40 smithi181 ceph-mon[35869]: pgmap v9963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:40 smithi097 ceph-mon[32231]: pgmap v9963: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17718.096553 secs 2024-07-26T12:54:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17718.096553 secs 2024-07-26T12:54:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17718.096553 secs 2024-07-26T12:54:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:42 smithi115 ceph-mon[35364]: pgmap v9964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:42 smithi181 ceph-mon[35869]: pgmap v9964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:42 smithi097 ceph-mon[32231]: pgmap v9964: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:44 smithi115 ceph-mon[35364]: pgmap v9965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:44 smithi181 ceph-mon[35869]: pgmap v9965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:44 smithi097 ceph-mon[32231]: pgmap v9965: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:46.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17723.096697 secs 2024-07-26T12:54:46.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:46 smithi115 ceph-mon[35364]: pgmap v9966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17723.096697 secs 2024-07-26T12:54:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:46 smithi181 ceph-mon[35869]: pgmap v9966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17723.096697 secs 2024-07-26T12:54:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:46 smithi097 ceph-mon[32231]: pgmap v9966: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:48 smithi115 ceph-mon[35364]: pgmap v9967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:48 smithi181 ceph-mon[35869]: pgmap v9967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:48 smithi097 ceph-mon[32231]: pgmap v9967: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:50 smithi181 ceph-mon[35869]: pgmap v9968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:50 smithi097 ceph-mon[32231]: pgmap v9968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:50 smithi115 ceph-mon[35364]: pgmap v9968: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17728.096854 secs 2024-07-26T12:54:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17728.096854 secs 2024-07-26T12:54:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17728.096854 secs 2024-07-26T12:54:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:52 smithi097 ceph-mon[32231]: pgmap v9969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:52 smithi115 ceph-mon[35364]: pgmap v9969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:52 smithi181 ceph-mon[35869]: pgmap v9969: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:54.844+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:54:54.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:54.845+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:54:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:54.845+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:54:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:54.845+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:54:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:54:54.845+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:54 smithi097 ceph-mon[32231]: pgmap v9970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:54 smithi115 ceph-mon[35364]: pgmap v9970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:54 smithi181 ceph-mon[35869]: pgmap v9970: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17733.096999 secs 2024-07-26T12:54:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:56 smithi115 ceph-mon[35364]: pgmap v9971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17733.096999 secs 2024-07-26T12:54:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:56 smithi181 ceph-mon[35869]: pgmap v9971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17733.096999 secs 2024-07-26T12:54:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:56 smithi097 ceph-mon[32231]: pgmap v9971: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:54:58 smithi115 ceph-mon[35364]: pgmap v9972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:54:58 smithi181 ceph-mon[35869]: pgmap v9972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:54:58 smithi097 ceph-mon[32231]: pgmap v9972: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:00 smithi115 ceph-mon[35364]: pgmap v9973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:01.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:00 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:00 smithi181 ceph-mon[35869]: pgmap v9973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:00 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:00 smithi097 ceph-mon[32231]: pgmap v9973: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:00 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17738.097124 secs 2024-07-26T12:55:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:01 smithi115 ceph-mon[35364]: pgmap v9974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:55:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:01 smithi115 ceph-mon[35364]: pgmap v9975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:02.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:01 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:55:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17738.097124 secs 2024-07-26T12:55:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:01 smithi181 ceph-mon[35869]: pgmap v9974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:55:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:01 smithi181 ceph-mon[35869]: pgmap v9975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:01 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:55:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17738.097124 secs 2024-07-26T12:55:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:01 smithi097 ceph-mon[32231]: pgmap v9974: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:55:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:01 smithi097 ceph-mon[32231]: pgmap v9975: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:02.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:01 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:55:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:03 smithi115 ceph-mon[35364]: pgmap v9976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:55:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:03 smithi181 ceph-mon[35869]: pgmap v9976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:55:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:03 smithi097 ceph-mon[32231]: pgmap v9976: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:55:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:05 smithi115 ceph-mon[35364]: pgmap v9977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:55:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:05 smithi181 ceph-mon[35869]: pgmap v9977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:55:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:05 smithi097 ceph-mon[32231]: pgmap v9977: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:55:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17743.097275 secs 2024-07-26T12:55:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17743.097275 secs 2024-07-26T12:55:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17743.097275 secs 2024-07-26T12:55:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:07 smithi097 ceph-mon[32231]: pgmap v9978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:07 smithi115 ceph-mon[35364]: pgmap v9978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:07 smithi181 ceph-mon[35869]: pgmap v9978: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:09.853+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:55:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:09.854+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:55:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:09.854+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:55:09.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:09.854+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:55:09.856 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:09.854+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:10.056+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:10.056+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:55:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:10.056+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff3c0004980 con 0x7ff3f01061a0 2024-07-26T12:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:10.056+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:10.056+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff3bc003690 con 0x55d25070b2f0 2024-07-26T12:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:10.056+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:10.057+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f100 con 0x7ff3f01061a0 2024-07-26T12:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:10.057+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:55:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:09 smithi097 ceph-mon[32231]: pgmap v9979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:09 smithi115 ceph-mon[35364]: pgmap v9979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:09 smithi181 ceph-mon[35869]: pgmap v9979: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:12 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17748.097415 secs 2024-07-26T12:55:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:12 smithi115 ceph-mon[35364]: pgmap v9980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:12 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17748.097415 secs 2024-07-26T12:55:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:12 smithi181 ceph-mon[35869]: pgmap v9980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:12 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17748.097415 secs 2024-07-26T12:55:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:12 smithi097 ceph-mon[32231]: pgmap v9980: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:14 smithi115 ceph-mon[35364]: pgmap v9981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:14 smithi181 ceph-mon[35869]: pgmap v9981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:14 smithi097 ceph-mon[32231]: pgmap v9981: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:15 smithi115 ceph-mon[35364]: pgmap v9982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:15 smithi181 ceph-mon[35869]: pgmap v9982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:15 smithi097 ceph-mon[32231]: pgmap v9982: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17753.097550 secs 2024-07-26T12:55:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17753.097550 secs 2024-07-26T12:55:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17753.097550 secs 2024-07-26T12:55:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:17 smithi115 ceph-mon[35364]: pgmap v9983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:17 smithi181 ceph-mon[35869]: pgmap v9983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:17 smithi097 ceph-mon[32231]: pgmap v9983: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:19 smithi181 ceph-mon[35869]: pgmap v9984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:19 smithi097 ceph-mon[32231]: pgmap v9984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:19 smithi115 ceph-mon[35364]: pgmap v9984: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17758.097650 secs 2024-07-26T12:55:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:21 smithi181 ceph-mon[35869]: pgmap v9985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17758.097650 secs 2024-07-26T12:55:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:21 smithi097 ceph-mon[32231]: pgmap v9985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17758.097650 secs 2024-07-26T12:55:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:21 smithi115 ceph-mon[35364]: pgmap v9985: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:23 smithi115 ceph-mon[35364]: pgmap v9986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:23 smithi181 ceph-mon[35869]: pgmap v9986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:23 smithi097 ceph-mon[32231]: pgmap v9986: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:24.856+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:55:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:24.857+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:55:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:24.857+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:55:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:24.857+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:55:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:24.857+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:25 smithi181 ceph-mon[35869]: pgmap v9987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:25 smithi097 ceph-mon[32231]: pgmap v9987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:25 smithi115 ceph-mon[35364]: pgmap v9987: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17763.097740 secs 2024-07-26T12:55:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17763.097740 secs 2024-07-26T12:55:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17763.097740 secs 2024-07-26T12:55:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:27 smithi097 ceph-mon[32231]: pgmap v9988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:27 smithi115 ceph-mon[35364]: pgmap v9988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:27 smithi181 ceph-mon[35869]: pgmap v9988: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:30 smithi115 ceph-mon[35364]: pgmap v9989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:30 smithi181 ceph-mon[35869]: pgmap v9989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:30 smithi097 ceph-mon[32231]: pgmap v9989: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17768.097914 secs 2024-07-26T12:55:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:31 smithi115 ceph-mon[35364]: pgmap v9990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17768.097914 secs 2024-07-26T12:55:31.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:31 smithi181 ceph-mon[35869]: pgmap v9990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17768.097914 secs 2024-07-26T12:55:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:31 smithi097 ceph-mon[32231]: pgmap v9990: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:33 smithi097 ceph-mon[32231]: pgmap v9991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:33 smithi115 ceph-mon[35364]: pgmap v9991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:33 smithi181 ceph-mon[35869]: pgmap v9991: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:35 smithi097 ceph-mon[32231]: pgmap v9992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:35 smithi115 ceph-mon[35364]: pgmap v9992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:35 smithi181 ceph-mon[35869]: pgmap v9992: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:37 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17773.098046 secs 2024-07-26T12:55:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:37 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17773.098046 secs 2024-07-26T12:55:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:37 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17773.098046 secs 2024-07-26T12:55:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:38 smithi115 ceph-mon[35364]: pgmap v9993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:38 smithi181 ceph-mon[35869]: pgmap v9993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:38 smithi097 ceph-mon[32231]: pgmap v9993: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:39.862+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:55:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:39.863+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:55:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:39.863+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:55:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:39.863+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:55:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:39.863+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:40 smithi115 ceph-mon[35364]: pgmap v9994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:40 smithi181 ceph-mon[35869]: pgmap v9994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:40 smithi097 ceph-mon[32231]: pgmap v9994: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17778.098189 secs 2024-07-26T12:55:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17778.098189 secs 2024-07-26T12:55:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17778.098189 secs 2024-07-26T12:55:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:42 smithi115 ceph-mon[35364]: pgmap v9995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:42 smithi181 ceph-mon[35869]: pgmap v9995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:42 smithi097 ceph-mon[32231]: pgmap v9995: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:43 smithi115 ceph-mon[35364]: pgmap v9996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:43 smithi181 ceph-mon[35869]: pgmap v9996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:43 smithi097 ceph-mon[32231]: pgmap v9996: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:45 smithi097 ceph-mon[32231]: pgmap v9997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:45 smithi115 ceph-mon[35364]: pgmap v9997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:45 smithi181 ceph-mon[35869]: pgmap v9997: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:47 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17783.098323 secs 2024-07-26T12:55:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:47 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17783.098323 secs 2024-07-26T12:55:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:47 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17783.098323 secs 2024-07-26T12:55:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:48 smithi115 ceph-mon[35364]: pgmap v9998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:48 smithi181 ceph-mon[35869]: pgmap v9998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:48 smithi097 ceph-mon[32231]: pgmap v9998: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:50 smithi115 ceph-mon[35364]: pgmap v9999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:50 smithi181 ceph-mon[35869]: pgmap v9999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:50 smithi097 ceph-mon[32231]: pgmap v9999: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17788.098416 secs 2024-07-26T12:55:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17788.098416 secs 2024-07-26T12:55:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17788.098416 secs 2024-07-26T12:55:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:52 smithi115 ceph-mon[35364]: pgmap v10000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:52 smithi181 ceph-mon[35869]: pgmap v10000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:52 smithi097 ceph-mon[32231]: pgmap v10000: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:53 smithi115 ceph-mon[35364]: pgmap v10001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:55:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:53 smithi181 ceph-mon[35869]: pgmap v10001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:55:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:53 smithi097 ceph-mon[32231]: pgmap v10001: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:55:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:54.865+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:55:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:54.866+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:55:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:54.866+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:55:54.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:54.866+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:55:54.868 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:55:54.866+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:55 smithi097 ceph-mon[32231]: pgmap v10002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:55 smithi115 ceph-mon[35364]: pgmap v10002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:55 smithi181 ceph-mon[35869]: pgmap v10002: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:57 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17793.098564 secs 2024-07-26T12:55:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:57 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17793.098564 secs 2024-07-26T12:55:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:57 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17793.098564 secs 2024-07-26T12:55:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:55:58 smithi115 ceph-mon[35364]: pgmap v10003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:55:58 smithi181 ceph-mon[35869]: pgmap v10003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:55:58 smithi097 ceph-mon[32231]: pgmap v10003: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:56:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:00 smithi115 ceph-mon[35364]: pgmap v10004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:56:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:00 smithi181 ceph-mon[35869]: pgmap v10004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:56:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:00 smithi097 ceph-mon[32231]: pgmap v10004: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:56:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17798.098702 secs 2024-07-26T12:56:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17798.098702 secs 2024-07-26T12:56:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17798.098702 secs 2024-07-26T12:56:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:02 smithi115 ceph-mon[35364]: pgmap v10005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:56:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:02 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:02 smithi181 ceph-mon[35869]: pgmap v10005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:56:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:02 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:02 smithi097 ceph-mon[32231]: pgmap v10005: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:56:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:02 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:03 smithi115 ceph-mon[35364]: pgmap v10006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:56:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:03 smithi115 ceph-mon[35364]: pgmap v10007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:56:03.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:56:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:03 smithi181 ceph-mon[35869]: pgmap v10006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:56:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:03 smithi181 ceph-mon[35869]: pgmap v10007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:56:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:56:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:03 smithi097 ceph-mon[32231]: pgmap v10006: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:56:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:03 smithi097 ceph-mon[32231]: pgmap v10007: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:56:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:56:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:05 smithi115 ceph-mon[35364]: pgmap v10008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:05 smithi181 ceph-mon[35869]: pgmap v10008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:05 smithi097 ceph-mon[32231]: pgmap v10008: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:06.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17803.098792 secs 2024-07-26T12:56:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17803.098792 secs 2024-07-26T12:56:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17803.098792 secs 2024-07-26T12:56:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:07 smithi115 ceph-mon[35364]: pgmap v10009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:07.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:07 smithi181 ceph-mon[35869]: pgmap v10009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:07 smithi097 ceph-mon[32231]: pgmap v10009: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:09 smithi115 ceph-mon[35364]: pgmap v10010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:56:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:09 smithi181 ceph-mon[35869]: pgmap v10010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:56:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:09 smithi097 ceph-mon[32231]: pgmap v10010: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:56:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:09.874+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:09.875+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:56:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:09.875+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:56:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:09.875+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:09.876+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17808.098933 secs 2024-07-26T12:56:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:11 smithi115 ceph-mon[35364]: pgmap v10011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:56:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17808.098933 secs 2024-07-26T12:56:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:11 smithi181 ceph-mon[35869]: pgmap v10011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:56:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17808.098933 secs 2024-07-26T12:56:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:11 smithi097 ceph-mon[32231]: pgmap v10011: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:56:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:13 smithi115 ceph-mon[35364]: pgmap v10012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:13 smithi181 ceph-mon[35869]: pgmap v10012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:13 smithi097 ceph-mon[32231]: pgmap v10012: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:15 smithi115 ceph-mon[35364]: pgmap v10013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:15 smithi181 ceph-mon[35869]: pgmap v10013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:15 smithi097 ceph-mon[32231]: pgmap v10013: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17813.099097 secs 2024-07-26T12:56:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17813.099097 secs 2024-07-26T12:56:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17813.099097 secs 2024-07-26T12:56:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:17 smithi115 ceph-mon[35364]: pgmap v10014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:17 smithi181 ceph-mon[35869]: pgmap v10014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:17 smithi097 ceph-mon[32231]: pgmap v10014: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:19 smithi115 ceph-mon[35364]: pgmap v10015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:19 smithi181 ceph-mon[35869]: pgmap v10015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:19 smithi097 ceph-mon[32231]: pgmap v10015: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17818.099226 secs 2024-07-26T12:56:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:21 smithi115 ceph-mon[35364]: pgmap v10016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17818.099226 secs 2024-07-26T12:56:21.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:21 smithi181 ceph-mon[35869]: pgmap v10016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17818.099226 secs 2024-07-26T12:56:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:21 smithi097 ceph-mon[32231]: pgmap v10016: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:23 smithi115 ceph-mon[35364]: pgmap v10017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:23 smithi181 ceph-mon[35869]: pgmap v10017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:23 smithi097 ceph-mon[32231]: pgmap v10017: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:24.879+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:24.880+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:56:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:24.880+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:56:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:24.880+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:24.880+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:25 smithi115 ceph-mon[35364]: pgmap v10018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:25 smithi181 ceph-mon[35869]: pgmap v10018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:25 smithi097 ceph-mon[32231]: pgmap v10018: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17823.099348 secs 2024-07-26T12:56:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17823.099348 secs 2024-07-26T12:56:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17823.099348 secs 2024-07-26T12:56:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:27 smithi181 ceph-mon[35869]: pgmap v10019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:27 smithi097 ceph-mon[32231]: pgmap v10019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:27 smithi115 ceph-mon[35364]: pgmap v10019: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:29 smithi097 ceph-mon[32231]: pgmap v10020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:29 smithi115 ceph-mon[35364]: pgmap v10020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:29 smithi181 ceph-mon[35869]: pgmap v10020: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17828.099489 secs 2024-07-26T12:56:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:31 smithi097 ceph-mon[32231]: pgmap v10021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17828.099489 secs 2024-07-26T12:56:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:31 smithi115 ceph-mon[35364]: pgmap v10021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17828.099489 secs 2024-07-26T12:56:31.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:31 smithi181 ceph-mon[35869]: pgmap v10021: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:33 smithi097 ceph-mon[32231]: pgmap v10022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:33 smithi115 ceph-mon[35364]: pgmap v10022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:33 smithi181 ceph-mon[35869]: pgmap v10022: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:35.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:35 smithi115 ceph-mon[35364]: pgmap v10023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:35.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:35 smithi181 ceph-mon[35869]: pgmap v10023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:35 smithi097 ceph-mon[32231]: pgmap v10023: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:36.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17833.099594 secs 2024-07-26T12:56:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17833.099594 secs 2024-07-26T12:56:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17833.099594 secs 2024-07-26T12:56:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:37 smithi115 ceph-mon[35364]: pgmap v10024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:37 smithi181 ceph-mon[35869]: pgmap v10024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:37 smithi097 ceph-mon[32231]: pgmap v10024: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:39.889 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:39.888+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:39.890 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:39.889+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:56:39.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:39.889+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:56:39.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:39.889+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:39.891 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:39.889+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:39 smithi115 ceph-mon[35364]: pgmap v10025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:39 smithi181 ceph-mon[35869]: pgmap v10025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:39 smithi097 ceph-mon[32231]: pgmap v10025: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17838.099732 secs 2024-07-26T12:56:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:41 smithi115 ceph-mon[35364]: pgmap v10026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17838.099732 secs 2024-07-26T12:56:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:41 smithi181 ceph-mon[35869]: pgmap v10026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17838.099732 secs 2024-07-26T12:56:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:41 smithi097 ceph-mon[32231]: pgmap v10026: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:43 smithi115 ceph-mon[35364]: pgmap v10027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:43 smithi181 ceph-mon[35869]: pgmap v10027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:43 smithi097 ceph-mon[32231]: pgmap v10027: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:45 smithi115 ceph-mon[35364]: pgmap v10028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:45 smithi181 ceph-mon[35869]: pgmap v10028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:45 smithi097 ceph-mon[32231]: pgmap v10028: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17843.099872 secs 2024-07-26T12:56:46.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17843.099872 secs 2024-07-26T12:56:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17843.099872 secs 2024-07-26T12:56:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:47 smithi115 ceph-mon[35364]: pgmap v10029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:47 smithi181 ceph-mon[35869]: pgmap v10029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:47 smithi097 ceph-mon[32231]: pgmap v10029: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:49 smithi115 ceph-mon[35364]: pgmap v10030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:49.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:49 smithi181 ceph-mon[35869]: pgmap v10030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:49 smithi097 ceph-mon[32231]: pgmap v10030: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17848.100026 secs 2024-07-26T12:56:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:51 smithi115 ceph-mon[35364]: pgmap v10031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17848.100026 secs 2024-07-26T12:56:51.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:51 smithi181 ceph-mon[35869]: pgmap v10031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17848.100026 secs 2024-07-26T12:56:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:51 smithi097 ceph-mon[32231]: pgmap v10031: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:53 smithi115 ceph-mon[35364]: pgmap v10032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:53 smithi181 ceph-mon[35869]: pgmap v10032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:53 smithi097 ceph-mon[32231]: pgmap v10032: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:54.891+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:54.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:54.893+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:56:54.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:54.893+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:56:54.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:54.893+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:54.894 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:56:54.893+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:55 smithi115 ceph-mon[35364]: pgmap v10033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:55 smithi181 ceph-mon[35869]: pgmap v10033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:55 smithi097 ceph-mon[32231]: pgmap v10033: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17853.100177 secs 2024-07-26T12:56:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17853.100177 secs 2024-07-26T12:56:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17853.100177 secs 2024-07-26T12:56:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:57 smithi115 ceph-mon[35364]: pgmap v10034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:57 smithi181 ceph-mon[35869]: pgmap v10034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:57 smithi097 ceph-mon[32231]: pgmap v10034: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:56:59 smithi115 ceph-mon[35364]: pgmap v10035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:59.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:56:59 smithi181 ceph-mon[35869]: pgmap v10035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:56:59 smithi097 ceph-mon[32231]: pgmap v10035: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17858.100320 secs 2024-07-26T12:57:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:01 smithi115 ceph-mon[35364]: pgmap v10036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17858.100320 secs 2024-07-26T12:57:01.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:01 smithi181 ceph-mon[35869]: pgmap v10036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17858.100320 secs 2024-07-26T12:57:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:01 smithi097 ceph-mon[32231]: pgmap v10036: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:03 smithi115 ceph-mon[35364]: pgmap v10037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:03 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:03 smithi181 ceph-mon[35869]: pgmap v10037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:03.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:03 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:03 smithi097 ceph-mon[32231]: pgmap v10037: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:03 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:57:04.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:57:04.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:57:04.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:04.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:04.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: pgmap v10038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:04.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: pgmap v10039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:57:04.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:04 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:57:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: pgmap v10038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:04.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: pgmap v10039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:57:04.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:04 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T12:57:05.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:05.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:05.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: pgmap v10038: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:05.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: pgmap v10039: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:57:05.003 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:04 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:57:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17863.100459 secs 2024-07-26T12:57:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:06 smithi115 ceph-mon[35364]: pgmap v10040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17863.100459 secs 2024-07-26T12:57:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:06 smithi181 ceph-mon[35869]: pgmap v10040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17863.100459 secs 2024-07-26T12:57:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:06 smithi097 ceph-mon[32231]: pgmap v10040: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:08 smithi115 ceph-mon[35364]: pgmap v10041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:57:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:08 smithi181 ceph-mon[35869]: pgmap v10041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:57:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:08 smithi097 ceph-mon[32231]: pgmap v10041: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:57:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:09.901+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:09.902+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:57:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:09.902+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:57:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:09.902+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:09.902+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:10 smithi115 ceph-mon[35364]: pgmap v10042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:10 smithi181 ceph-mon[35869]: pgmap v10042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:10 smithi097 ceph-mon[32231]: pgmap v10042: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17868.100604 secs 2024-07-26T12:57:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17868.100604 secs 2024-07-26T12:57:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17868.100604 secs 2024-07-26T12:57:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:12 smithi115 ceph-mon[35364]: pgmap v10043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:57:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:12 smithi181 ceph-mon[35869]: pgmap v10043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:57:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:12 smithi097 ceph-mon[32231]: pgmap v10043: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:57:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:14 smithi115 ceph-mon[35364]: pgmap v10044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:14 smithi181 ceph-mon[35869]: pgmap v10044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:14 smithi097 ceph-mon[32231]: pgmap v10044: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17873.100707 secs 2024-07-26T12:57:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:16 smithi115 ceph-mon[35364]: pgmap v10045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17873.100707 secs 2024-07-26T12:57:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:16 smithi181 ceph-mon[35869]: pgmap v10045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17873.100707 secs 2024-07-26T12:57:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:16 smithi097 ceph-mon[32231]: pgmap v10045: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:18 smithi115 ceph-mon[35364]: pgmap v10046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:18 smithi181 ceph-mon[35869]: pgmap v10046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:18 smithi097 ceph-mon[32231]: pgmap v10046: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:20 smithi115 ceph-mon[35364]: pgmap v10047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:20 smithi181 ceph-mon[35869]: pgmap v10047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:20 smithi097 ceph-mon[32231]: pgmap v10047: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17878.100795 secs 2024-07-26T12:57:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17878.100795 secs 2024-07-26T12:57:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17878.100795 secs 2024-07-26T12:57:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:22 smithi115 ceph-mon[35364]: pgmap v10048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:22 smithi181 ceph-mon[35869]: pgmap v10048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:22 smithi097 ceph-mon[32231]: pgmap v10048: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:24.905+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:24.906+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:57:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:24.906+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:57:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:24.906+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:24.906+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:24 smithi115 ceph-mon[35364]: pgmap v10049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:24 smithi181 ceph-mon[35869]: pgmap v10049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:24 smithi097 ceph-mon[32231]: pgmap v10049: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17883.100936 secs 2024-07-26T12:57:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:26 smithi115 ceph-mon[35364]: pgmap v10050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17883.100936 secs 2024-07-26T12:57:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:26 smithi181 ceph-mon[35869]: pgmap v10050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17883.100936 secs 2024-07-26T12:57:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:26 smithi097 ceph-mon[32231]: pgmap v10050: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:28 smithi115 ceph-mon[35364]: pgmap v10051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:28 smithi181 ceph-mon[35869]: pgmap v10051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:28 smithi097 ceph-mon[32231]: pgmap v10051: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:30 smithi115 ceph-mon[35364]: pgmap v10052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:30 smithi181 ceph-mon[35869]: pgmap v10052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:30 smithi097 ceph-mon[32231]: pgmap v10052: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17888.101035 secs 2024-07-26T12:57:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17888.101035 secs 2024-07-26T12:57:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17888.101035 secs 2024-07-26T12:57:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:32 smithi115 ceph-mon[35364]: pgmap v10053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:32 smithi181 ceph-mon[35869]: pgmap v10053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:32 smithi097 ceph-mon[32231]: pgmap v10053: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:34 smithi115 ceph-mon[35364]: pgmap v10054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:34 smithi181 ceph-mon[35869]: pgmap v10054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:34 smithi097 ceph-mon[32231]: pgmap v10054: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17893.101182 secs 2024-07-26T12:57:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:36 smithi115 ceph-mon[35364]: pgmap v10055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17893.101182 secs 2024-07-26T12:57:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:36 smithi181 ceph-mon[35869]: pgmap v10055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17893.101182 secs 2024-07-26T12:57:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:36 smithi097 ceph-mon[32231]: pgmap v10055: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:38 smithi115 ceph-mon[35364]: pgmap v10056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:38 smithi181 ceph-mon[35869]: pgmap v10056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:38 smithi097 ceph-mon[32231]: pgmap v10056: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:39.912 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:39.910+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:39.913 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:39.912+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:57:39.913 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:39.912+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:57:39.913 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:39.912+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:39.913 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:39.912+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:40 smithi115 ceph-mon[35364]: pgmap v10057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:40 smithi181 ceph-mon[35869]: pgmap v10057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:40 smithi097 ceph-mon[32231]: pgmap v10057: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17898.101332 secs 2024-07-26T12:57:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17898.101332 secs 2024-07-26T12:57:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17898.101332 secs 2024-07-26T12:57:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:42 smithi115 ceph-mon[35364]: pgmap v10058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:42 smithi181 ceph-mon[35869]: pgmap v10058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:42 smithi097 ceph-mon[32231]: pgmap v10058: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:44 smithi115 ceph-mon[35364]: pgmap v10059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:44 smithi181 ceph-mon[35869]: pgmap v10059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:44 smithi097 ceph-mon[32231]: pgmap v10059: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17903.101480 secs 2024-07-26T12:57:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:46 smithi181 ceph-mon[35869]: pgmap v10060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17903.101480 secs 2024-07-26T12:57:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:46 smithi097 ceph-mon[32231]: pgmap v10060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17903.101480 secs 2024-07-26T12:57:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:46 smithi115 ceph-mon[35364]: pgmap v10060: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:48 smithi097 ceph-mon[32231]: pgmap v10061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:48 smithi115 ceph-mon[35364]: pgmap v10061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:48 smithi181 ceph-mon[35869]: pgmap v10061: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:50 smithi097 ceph-mon[32231]: pgmap v10062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:50 smithi115 ceph-mon[35364]: pgmap v10062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:50 smithi181 ceph-mon[35869]: pgmap v10062: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:52 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17908.101567 secs 2024-07-26T12:57:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:52 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17908.101567 secs 2024-07-26T12:57:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:52 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17908.101567 secs 2024-07-26T12:57:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:53 smithi115 ceph-mon[35364]: pgmap v10063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:53 smithi181 ceph-mon[35869]: pgmap v10063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:53 smithi097 ceph-mon[32231]: pgmap v10063: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:54.916 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:54.914+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:54.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:54.916+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:57:54.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:54.916+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:57:54.917 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:54.916+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:57:54.916+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:55 smithi115 ceph-mon[35364]: pgmap v10064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:55 smithi181 ceph-mon[35869]: pgmap v10064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:55 smithi097 ceph-mon[32231]: pgmap v10064: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:57 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17913.101713 secs 2024-07-26T12:57:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:57 smithi115 ceph-mon[35364]: pgmap v10065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:57 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17913.101713 secs 2024-07-26T12:57:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:57 smithi181 ceph-mon[35869]: pgmap v10065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:57 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17913.101713 secs 2024-07-26T12:57:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:57 smithi097 ceph-mon[32231]: pgmap v10065: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:57:59 smithi115 ceph-mon[35364]: pgmap v10066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:57:59 smithi181 ceph-mon[35869]: pgmap v10066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:57:59 smithi097 ceph-mon[32231]: pgmap v10066: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:01 smithi115 ceph-mon[35364]: pgmap v10067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:01 smithi181 ceph-mon[35869]: pgmap v10067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:01 smithi097 ceph-mon[32231]: pgmap v10067: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:02 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17918.101847 secs 2024-07-26T12:58:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:02 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17918.101847 secs 2024-07-26T12:58:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:02 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17918.101847 secs 2024-07-26T12:58:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:03 smithi115 ceph-mon[35364]: pgmap v10068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:03 smithi181 ceph-mon[35869]: pgmap v10068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:03 smithi097 ceph-mon[32231]: pgmap v10068: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:04 smithi115 ceph-mon[35364]: pgmap v10069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:04 smithi181 ceph-mon[35869]: pgmap v10069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:04 smithi097 ceph-mon[32231]: pgmap v10069: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:05 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:05 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:05 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17923.102033 secs 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:06.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: pgmap v10070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:06.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:58:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17923.102033 secs 2024-07-26T12:58:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:58:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:58:06.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:06.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:06.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: pgmap v10070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:06.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:58:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17923.102033 secs 2024-07-26T12:58:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:58:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T12:58:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T12:58:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: pgmap v10070: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:06.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:58:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.936+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T12:58:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.936+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:58:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.936+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057c0 tx=0x7ff3e006a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.936+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057c0 tx=0x7ff3e006a000 comp rx=0 tx=0).stop 2024-07-26T12:58:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.936+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T12:58:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.936+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 msgr2=0x7ff3d40b4650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:58:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.936+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4650 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:58:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.936+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:58:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.937+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:06.938+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40b5be0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e008b070 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:58:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:08 smithi115 ceph-mon[35364]: pgmap v10071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:58:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:08 smithi181 ceph-mon[35869]: pgmap v10071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:58:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:08 smithi097 ceph-mon[32231]: pgmap v10071: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:58:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:09.919+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:09.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:09.920+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:58:09.921 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:09.920+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:58:09.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:09.920+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:09.922 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:09.920+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:10.089 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:10.088+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:58:10.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:10.088+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T12:58:10.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:10.089+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T12:58:10.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:10.089+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T12:58:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:10 smithi181 ceph-mon[35869]: pgmap v10072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:58:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:10 smithi097 ceph-mon[32231]: pgmap v10072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:58:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:10 smithi115 ceph-mon[35364]: pgmap v10072: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:58:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17928.102134 secs 2024-07-26T12:58:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17928.102134 secs 2024-07-26T12:58:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17928.102134 secs 2024-07-26T12:58:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:12 smithi097 ceph-mon[32231]: pgmap v10073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:58:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:12 smithi115 ceph-mon[35364]: pgmap v10073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:58:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:12 smithi181 ceph-mon[35869]: pgmap v10073: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:58:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:14 smithi097 ceph-mon[32231]: pgmap v10074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:14 smithi115 ceph-mon[35364]: pgmap v10074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:14 smithi181 ceph-mon[35869]: pgmap v10074: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17933.102277 secs 2024-07-26T12:58:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:16 smithi115 ceph-mon[35364]: pgmap v10075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17933.102277 secs 2024-07-26T12:58:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:16 smithi181 ceph-mon[35869]: pgmap v10075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17933.102277 secs 2024-07-26T12:58:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:16 smithi097 ceph-mon[32231]: pgmap v10075: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:18 smithi115 ceph-mon[35364]: pgmap v10076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:18 smithi181 ceph-mon[35869]: pgmap v10076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:18 smithi097 ceph-mon[32231]: pgmap v10076: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:20 smithi115 ceph-mon[35364]: pgmap v10077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:20 smithi181 ceph-mon[35869]: pgmap v10077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:20 smithi097 ceph-mon[32231]: pgmap v10077: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17938.102412 secs 2024-07-26T12:58:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17938.102412 secs 2024-07-26T12:58:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17938.102412 secs 2024-07-26T12:58:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:22 smithi115 ceph-mon[35364]: pgmap v10078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:22 smithi181 ceph-mon[35869]: pgmap v10078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:22 smithi097 ceph-mon[32231]: pgmap v10078: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:24 smithi115 ceph-mon[35364]: pgmap v10079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:24 smithi181 ceph-mon[35869]: pgmap v10079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:24 smithi097 ceph-mon[32231]: pgmap v10079: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:24.923+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:24.924+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:58:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:24.924+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:58:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:24.924+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:24.924+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17943.102542 secs 2024-07-26T12:58:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:26 smithi115 ceph-mon[35364]: pgmap v10080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17943.102542 secs 2024-07-26T12:58:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:26 smithi181 ceph-mon[35869]: pgmap v10080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17943.102542 secs 2024-07-26T12:58:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:26 smithi097 ceph-mon[32231]: pgmap v10080: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:28 smithi115 ceph-mon[35364]: pgmap v10081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:28 smithi181 ceph-mon[35869]: pgmap v10081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:28 smithi097 ceph-mon[32231]: pgmap v10081: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:30 smithi115 ceph-mon[35364]: pgmap v10082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:30.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:30 smithi181 ceph-mon[35869]: pgmap v10082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:30 smithi097 ceph-mon[32231]: pgmap v10082: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17948.102682 secs 2024-07-26T12:58:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17948.102682 secs 2024-07-26T12:58:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17948.102682 secs 2024-07-26T12:58:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:32 smithi115 ceph-mon[35364]: pgmap v10083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:32 smithi181 ceph-mon[35869]: pgmap v10083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:32 smithi097 ceph-mon[32231]: pgmap v10083: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:34 smithi115 ceph-mon[35364]: pgmap v10084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:34 smithi181 ceph-mon[35869]: pgmap v10084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:34 smithi097 ceph-mon[32231]: pgmap v10084: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17953.102820 secs 2024-07-26T12:58:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:36 smithi115 ceph-mon[35364]: pgmap v10085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17953.102820 secs 2024-07-26T12:58:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:36 smithi181 ceph-mon[35869]: pgmap v10085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17953.102820 secs 2024-07-26T12:58:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:36 smithi097 ceph-mon[32231]: pgmap v10085: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:38 smithi115 ceph-mon[35364]: pgmap v10086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:38 smithi181 ceph-mon[35869]: pgmap v10086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:38 smithi097 ceph-mon[32231]: pgmap v10086: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:39.933 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:39.932+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:39.933+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:58:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:39.933+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:58:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:39.933+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:39.933+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:40 smithi115 ceph-mon[35364]: pgmap v10087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:40 smithi181 ceph-mon[35869]: pgmap v10087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:40 smithi097 ceph-mon[32231]: pgmap v10087: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17958.102908 secs 2024-07-26T12:58:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17958.102908 secs 2024-07-26T12:58:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17958.102908 secs 2024-07-26T12:58:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:42 smithi115 ceph-mon[35364]: pgmap v10088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:42 smithi181 ceph-mon[35869]: pgmap v10088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:42 smithi097 ceph-mon[32231]: pgmap v10088: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:44 smithi115 ceph-mon[35364]: pgmap v10089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:58:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:44 smithi181 ceph-mon[35869]: pgmap v10089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:58:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:44 smithi097 ceph-mon[32231]: pgmap v10089: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:58:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17963.102989 secs 2024-07-26T12:58:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:46 smithi115 ceph-mon[35364]: pgmap v10090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17963.102989 secs 2024-07-26T12:58:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:46 smithi181 ceph-mon[35869]: pgmap v10090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17963.102989 secs 2024-07-26T12:58:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:46 smithi097 ceph-mon[32231]: pgmap v10090: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:48 smithi115 ceph-mon[35364]: pgmap v10091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:48 smithi181 ceph-mon[35869]: pgmap v10091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:48 smithi097 ceph-mon[32231]: pgmap v10091: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:50 smithi115 ceph-mon[35364]: pgmap v10092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:50 smithi181 ceph-mon[35869]: pgmap v10092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:50 smithi097 ceph-mon[32231]: pgmap v10092: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17968.103129 secs 2024-07-26T12:58:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17968.103129 secs 2024-07-26T12:58:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17968.103129 secs 2024-07-26T12:58:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:52 smithi115 ceph-mon[35364]: pgmap v10093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:52 smithi181 ceph-mon[35869]: pgmap v10093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:52 smithi097 ceph-mon[32231]: pgmap v10093: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:54 smithi115 ceph-mon[35364]: pgmap v10094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:54 smithi181 ceph-mon[35869]: pgmap v10094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:54.753 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:54 smithi097 ceph-mon[32231]: pgmap v10094: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:54.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:54.935+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:58:54.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:54.936+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:58:54.937 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:54.936+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:58:54.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:54.936+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:58:54.938 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:58:54.936+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17973.103279 secs 2024-07-26T12:58:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:56 smithi115 ceph-mon[35364]: pgmap v10095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17973.103279 secs 2024-07-26T12:58:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:56 smithi181 ceph-mon[35869]: pgmap v10095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17973.103279 secs 2024-07-26T12:58:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:56 smithi097 ceph-mon[32231]: pgmap v10095: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:58:58 smithi115 ceph-mon[35364]: pgmap v10096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:58:58 smithi181 ceph-mon[35869]: pgmap v10096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:58:58 smithi097 ceph-mon[32231]: pgmap v10096: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:00 smithi115 ceph-mon[35364]: pgmap v10097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:00 smithi181 ceph-mon[35869]: pgmap v10097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:00 smithi097 ceph-mon[32231]: pgmap v10097: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17978.103372 secs 2024-07-26T12:59:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17978.103372 secs 2024-07-26T12:59:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17978.103372 secs 2024-07-26T12:59:02.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:02 smithi181 ceph-mon[35869]: pgmap v10098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:02 smithi097 ceph-mon[32231]: pgmap v10098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:02 smithi115 ceph-mon[35364]: pgmap v10098: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:04 smithi181 ceph-mon[35869]: pgmap v10099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:04 smithi097 ceph-mon[32231]: pgmap v10099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:04.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:04 smithi115 ceph-mon[35364]: pgmap v10099: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17983.103508 secs 2024-07-26T12:59:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:06 smithi097 ceph-mon[32231]: pgmap v10100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:06 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17983.103508 secs 2024-07-26T12:59:06.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:06 smithi115 ceph-mon[35364]: pgmap v10100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:06.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:06 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17983.103508 secs 2024-07-26T12:59:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:06 smithi181 ceph-mon[35869]: pgmap v10100: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:06.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:06 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T12:59:08.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:08 smithi115 ceph-mon[35364]: pgmap v10101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:59:08.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:08 smithi115 ceph-mon[35364]: pgmap v10102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:08.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:08 smithi181 ceph-mon[35869]: pgmap v10101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:59:08.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:08 smithi181 ceph-mon[35869]: pgmap v10102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:08.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:08 smithi097 ceph-mon[32231]: pgmap v10101: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:59:08.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:08 smithi097 ceph-mon[32231]: pgmap v10102: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:09.938+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:59:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:09.939+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:59:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:09.939+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:59:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:09.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:59:09.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:09.939+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:10 smithi115 ceph-mon[35364]: pgmap v10103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:10 smithi181 ceph-mon[35869]: pgmap v10103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:10 smithi097 ceph-mon[32231]: pgmap v10103: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17988.103653 secs 2024-07-26T12:59:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17988.103653 secs 2024-07-26T12:59:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17988.103653 secs 2024-07-26T12:59:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:12 smithi115 ceph-mon[35364]: pgmap v10104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:12 smithi181 ceph-mon[35869]: pgmap v10104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:13.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:12 smithi097 ceph-mon[32231]: pgmap v10104: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:14 smithi115 ceph-mon[35364]: pgmap v10105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:14 smithi181 ceph-mon[35869]: pgmap v10105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:14 smithi097 ceph-mon[32231]: pgmap v10105: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:16 smithi115 ceph-mon[35364]: pgmap v10106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17993.103812 secs 2024-07-26T12:59:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:16 smithi181 ceph-mon[35869]: pgmap v10106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17993.103812 secs 2024-07-26T12:59:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:16 smithi097 ceph-mon[32231]: pgmap v10106: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:59:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17993.103812 secs 2024-07-26T12:59:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:18 smithi115 ceph-mon[35364]: pgmap v10107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:18 smithi181 ceph-mon[35869]: pgmap v10107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:18 smithi097 ceph-mon[32231]: pgmap v10107: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:20 smithi115 ceph-mon[35364]: pgmap v10108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:20 smithi181 ceph-mon[35869]: pgmap v10108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:20 smithi097 ceph-mon[32231]: pgmap v10108: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 17998.103949 secs 2024-07-26T12:59:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 17998.103949 secs 2024-07-26T12:59:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 17998.103949 secs 2024-07-26T12:59:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:22 smithi115 ceph-mon[35364]: pgmap v10109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:22 smithi181 ceph-mon[35869]: pgmap v10109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:22 smithi097 ceph-mon[32231]: pgmap v10109: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:24.941+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:59:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:24.942+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:59:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:24.942+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:59:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:24.942+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:59:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:24.942+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:24 smithi115 ceph-mon[35364]: pgmap v10110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:24 smithi181 ceph-mon[35869]: pgmap v10110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:24 smithi097 ceph-mon[32231]: pgmap v10110: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18003.104089 secs 2024-07-26T12:59:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:26 smithi115 ceph-mon[35364]: pgmap v10111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18003.104089 secs 2024-07-26T12:59:26.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:26 smithi181 ceph-mon[35869]: pgmap v10111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18003.104089 secs 2024-07-26T12:59:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:26 smithi097 ceph-mon[32231]: pgmap v10111: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:28 smithi115 ceph-mon[35364]: pgmap v10112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:28 smithi181 ceph-mon[35869]: pgmap v10112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:29.002 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:28 smithi097 ceph-mon[32231]: pgmap v10112: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:30 smithi115 ceph-mon[35364]: pgmap v10113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:30 smithi181 ceph-mon[35869]: pgmap v10113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:30 smithi097 ceph-mon[32231]: pgmap v10113: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18008.104268 secs 2024-07-26T12:59:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18008.104268 secs 2024-07-26T12:59:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18008.104268 secs 2024-07-26T12:59:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:32 smithi115 ceph-mon[35364]: pgmap v10114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:32 smithi181 ceph-mon[35869]: pgmap v10114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:32 smithi097 ceph-mon[32231]: pgmap v10114: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:34 smithi115 ceph-mon[35364]: pgmap v10115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:34 smithi181 ceph-mon[35869]: pgmap v10115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:34 smithi097 ceph-mon[32231]: pgmap v10115: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18013.104408 secs 2024-07-26T12:59:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:36 smithi115 ceph-mon[35364]: pgmap v10116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18013.104408 secs 2024-07-26T12:59:36.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:36 smithi181 ceph-mon[35869]: pgmap v10116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18013.104408 secs 2024-07-26T12:59:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:36 smithi097 ceph-mon[32231]: pgmap v10116: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:38 smithi115 ceph-mon[35364]: pgmap v10117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:38 smithi181 ceph-mon[35869]: pgmap v10117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:38 smithi097 ceph-mon[32231]: pgmap v10117: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:39.951+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:59:39.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:39.952+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:59:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:39.952+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:59:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:39.952+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:59:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:39.952+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:40 smithi115 ceph-mon[35364]: pgmap v10118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:40 smithi181 ceph-mon[35869]: pgmap v10118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:40 smithi097 ceph-mon[32231]: pgmap v10118: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18018.104547 secs 2024-07-26T12:59:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18018.104547 secs 2024-07-26T12:59:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18018.104547 secs 2024-07-26T12:59:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:42 smithi115 ceph-mon[35364]: pgmap v10119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:42 smithi181 ceph-mon[35869]: pgmap v10119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:42 smithi097 ceph-mon[32231]: pgmap v10119: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:44 smithi115 ceph-mon[35364]: pgmap v10120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:44 smithi181 ceph-mon[35869]: pgmap v10120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:44 smithi097 ceph-mon[32231]: pgmap v10120: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18023.104638 secs 2024-07-26T12:59:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:46 smithi115 ceph-mon[35364]: pgmap v10121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18023.104638 secs 2024-07-26T12:59:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:46 smithi181 ceph-mon[35869]: pgmap v10121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18023.104638 secs 2024-07-26T12:59:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:46 smithi097 ceph-mon[32231]: pgmap v10121: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:48 smithi115 ceph-mon[35364]: pgmap v10122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:48 smithi181 ceph-mon[35869]: pgmap v10122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:48 smithi097 ceph-mon[32231]: pgmap v10122: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:50 smithi115 ceph-mon[35364]: pgmap v10123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:50 smithi181 ceph-mon[35869]: pgmap v10123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:50 smithi097 ceph-mon[32231]: pgmap v10123: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18028.104783 secs 2024-07-26T12:59:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18028.104783 secs 2024-07-26T12:59:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18028.104783 secs 2024-07-26T12:59:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:52 smithi115 ceph-mon[35364]: pgmap v10124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:52 smithi181 ceph-mon[35869]: pgmap v10124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:52 smithi097 ceph-mon[32231]: pgmap v10124: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:54.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:54.954+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:59:54.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:54.956+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T12:59:54.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:54.956+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:59:54.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:54.956+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:59:54.957 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T12:59:54.956+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:54 smithi115 ceph-mon[35364]: pgmap v10125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:54 smithi181 ceph-mon[35869]: pgmap v10125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:54 smithi097 ceph-mon[32231]: pgmap v10125: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18033.104925 secs 2024-07-26T12:59:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:56 smithi115 ceph-mon[35364]: pgmap v10126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18033.104925 secs 2024-07-26T12:59:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:56 smithi181 ceph-mon[35869]: pgmap v10126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18033.104925 secs 2024-07-26T12:59:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:56 smithi097 ceph-mon[32231]: pgmap v10126: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:58.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 12:59:58 smithi181 ceph-mon[35869]: pgmap v10127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 12:59:58 smithi097 ceph-mon[32231]: pgmap v10127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 12:59:58 smithi115 ceph-mon[35364]: pgmap v10127: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:00 smithi181 ceph-mon[35869]: pgmap v10128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:00 smithi097 ceph-mon[32231]: pgmap v10128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:00 smithi115 ceph-mon[35364]: pgmap v10128: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18038.105070 secs 2024-07-26T13:00:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18038.105070 secs 2024-07-26T13:00:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18038.105070 secs 2024-07-26T13:00:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:02 smithi097 ceph-mon[32231]: pgmap v10129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:02 smithi115 ceph-mon[35364]: pgmap v10129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:02 smithi181 ceph-mon[35869]: pgmap v10129: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:04 smithi115 ceph-mon[35364]: pgmap v10130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:04 smithi181 ceph-mon[35869]: pgmap v10130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:04 smithi097 ceph-mon[32231]: pgmap v10130: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18043.105207 secs 2024-07-26T13:00:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:06 smithi115 ceph-mon[35364]: pgmap v10131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18043.105207 secs 2024-07-26T13:00:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:06 smithi181 ceph-mon[35869]: pgmap v10131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18043.105207 secs 2024-07-26T13:00:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:06 smithi097 ceph-mon[32231]: pgmap v10131: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:08.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:07 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:08.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:07 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:08.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:07 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:08 smithi115 ceph-mon[35364]: pgmap v10132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:08 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:08 smithi181 ceph-mon[35869]: pgmap v10132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:08 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:08 smithi097 ceph-mon[32231]: pgmap v10132: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:08 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:09.964+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:00:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:09.965+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:00:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:09.965+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:00:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:09.965+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:00:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:09.966+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:10.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:09 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:09 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:09 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:00:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:10 smithi115 ceph-mon[35364]: pgmap v10133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:10 smithi115 ceph-mon[35364]: pgmap v10134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:10 smithi181 ceph-mon[35869]: pgmap v10133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:10 smithi181 ceph-mon[35869]: pgmap v10134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:10 smithi097 ceph-mon[32231]: pgmap v10133: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:10 smithi097 ceph-mon[32231]: pgmap v10134: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18048.105355 secs 2024-07-26T13:00:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18048.105355 secs 2024-07-26T13:00:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18048.105355 secs 2024-07-26T13:00:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:12 smithi115 ceph-mon[35364]: pgmap v10135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:12 smithi181 ceph-mon[35869]: pgmap v10135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:13.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:12 smithi097 ceph-mon[32231]: pgmap v10135: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:14 smithi115 ceph-mon[35364]: pgmap v10136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:14 smithi181 ceph-mon[35869]: pgmap v10136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:14 smithi097 ceph-mon[32231]: pgmap v10136: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:16 smithi115 ceph-mon[35364]: pgmap v10137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18053.105489 secs 2024-07-26T13:00:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:16 smithi181 ceph-mon[35869]: pgmap v10137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18053.105489 secs 2024-07-26T13:00:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:16 smithi097 ceph-mon[32231]: pgmap v10137: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18053.105489 secs 2024-07-26T13:00:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:18 smithi115 ceph-mon[35364]: pgmap v10138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:18 smithi181 ceph-mon[35869]: pgmap v10138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:18 smithi097 ceph-mon[32231]: pgmap v10138: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:20 smithi115 ceph-mon[35364]: pgmap v10139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:00:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:20 smithi181 ceph-mon[35869]: pgmap v10139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:00:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:20 smithi097 ceph-mon[32231]: pgmap v10139: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:00:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18058.105620 secs 2024-07-26T13:00:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18058.105620 secs 2024-07-26T13:00:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18058.105620 secs 2024-07-26T13:00:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:22 smithi115 ceph-mon[35364]: pgmap v10140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:22 smithi181 ceph-mon[35869]: pgmap v10140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:22 smithi097 ceph-mon[32231]: pgmap v10140: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:24.969+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:00:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:24.971+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:00:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:24.971+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:00:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:24.971+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:00:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:24.971+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:24 smithi115 ceph-mon[35364]: pgmap v10141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:00:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:24 smithi181 ceph-mon[35869]: pgmap v10141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:00:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:24 smithi097 ceph-mon[32231]: pgmap v10141: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:00:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:26 smithi115 ceph-mon[35364]: pgmap v10142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:00:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18063.105761 secs 2024-07-26T13:00:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:26 smithi181 ceph-mon[35869]: pgmap v10142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:00:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18063.105761 secs 2024-07-26T13:00:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:26 smithi097 ceph-mon[32231]: pgmap v10142: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:00:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18063.105761 secs 2024-07-26T13:00:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:28 smithi115 ceph-mon[35364]: pgmap v10143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:28 smithi181 ceph-mon[35869]: pgmap v10143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:28 smithi097 ceph-mon[32231]: pgmap v10143: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:30 smithi115 ceph-mon[35364]: pgmap v10144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:00:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:30 smithi181 ceph-mon[35869]: pgmap v10144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:00:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:30 smithi097 ceph-mon[32231]: pgmap v10144: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:00:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18068.105908 secs 2024-07-26T13:00:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18068.105908 secs 2024-07-26T13:00:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18068.105908 secs 2024-07-26T13:00:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:32 smithi115 ceph-mon[35364]: pgmap v10145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:32 smithi181 ceph-mon[35869]: pgmap v10145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:32 smithi097 ceph-mon[32231]: pgmap v10145: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:34 smithi115 ceph-mon[35364]: pgmap v10146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:34 smithi181 ceph-mon[35869]: pgmap v10146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:34 smithi097 ceph-mon[32231]: pgmap v10146: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18073.106048 secs 2024-07-26T13:00:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:36 smithi115 ceph-mon[35364]: pgmap v10147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18073.106048 secs 2024-07-26T13:00:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:36 smithi181 ceph-mon[35869]: pgmap v10147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18073.106048 secs 2024-07-26T13:00:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:36 smithi097 ceph-mon[32231]: pgmap v10147: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:38 smithi115 ceph-mon[35364]: pgmap v10148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:38 smithi181 ceph-mon[35869]: pgmap v10148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:38 smithi097 ceph-mon[32231]: pgmap v10148: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:39.981 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:39.980+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:00:39.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:39.981+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:00:39.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:39.981+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:00:39.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:39.981+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:00:39.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:39.981+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:40 smithi115 ceph-mon[35364]: pgmap v10149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:40 smithi181 ceph-mon[35869]: pgmap v10149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:40 smithi097 ceph-mon[32231]: pgmap v10149: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18078.106177 secs 2024-07-26T13:00:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18078.106177 secs 2024-07-26T13:00:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18078.106177 secs 2024-07-26T13:00:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:42 smithi115 ceph-mon[35364]: pgmap v10150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:42 smithi181 ceph-mon[35869]: pgmap v10150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:42 smithi097 ceph-mon[32231]: pgmap v10150: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:44 smithi115 ceph-mon[35364]: pgmap v10151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:44 smithi181 ceph-mon[35869]: pgmap v10151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:44 smithi097 ceph-mon[32231]: pgmap v10151: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18083.106321 secs 2024-07-26T13:00:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:46 smithi115 ceph-mon[35364]: pgmap v10152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:47.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18083.106321 secs 2024-07-26T13:00:47.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:46 smithi181 ceph-mon[35869]: pgmap v10152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18083.106321 secs 2024-07-26T13:00:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:46 smithi097 ceph-mon[32231]: pgmap v10152: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:48 smithi115 ceph-mon[35364]: pgmap v10153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:48 smithi181 ceph-mon[35869]: pgmap v10153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:48 smithi097 ceph-mon[32231]: pgmap v10153: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:50 smithi181 ceph-mon[35869]: pgmap v10154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:50 smithi097 ceph-mon[32231]: pgmap v10154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:50 smithi115 ceph-mon[35364]: pgmap v10154: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18088.106465 secs 2024-07-26T13:00:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18088.106465 secs 2024-07-26T13:00:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18088.106465 secs 2024-07-26T13:00:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:52 smithi097 ceph-mon[32231]: pgmap v10155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:52 smithi115 ceph-mon[35364]: pgmap v10155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:52 smithi181 ceph-mon[35869]: pgmap v10155: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:54.983+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:00:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:54.984+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:00:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:54.984+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:00:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:54.984+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:00:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:00:54.984+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:54 smithi097 ceph-mon[32231]: pgmap v10156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:54 smithi115 ceph-mon[35364]: pgmap v10156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:54 smithi181 ceph-mon[35869]: pgmap v10156: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:57 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18093.106550 secs 2024-07-26T13:00:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:57 smithi115 ceph-mon[35364]: pgmap v10157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:57 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18093.106550 secs 2024-07-26T13:00:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:57 smithi181 ceph-mon[35869]: pgmap v10157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:57 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18093.106550 secs 2024-07-26T13:00:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:57 smithi097 ceph-mon[32231]: pgmap v10157: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:00:59 smithi115 ceph-mon[35364]: pgmap v10158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:00:59 smithi181 ceph-mon[35869]: pgmap v10158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:00:59 smithi097 ceph-mon[32231]: pgmap v10158: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:01:01.269 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:01 smithi115 ceph-mon[35364]: pgmap v10159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:01:01.437 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:01 smithi097 ceph-mon[32231]: pgmap v10159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:01:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:01 smithi181 ceph-mon[35869]: pgmap v10159: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:01:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:02 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18098.106681 secs 2024-07-26T13:01:02.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:02 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18098.106681 secs 2024-07-26T13:01:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:02 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18098.106681 secs 2024-07-26T13:01:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:03 smithi115 ceph-mon[35364]: pgmap v10160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:01:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:03 smithi181 ceph-mon[35869]: pgmap v10160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:01:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:03 smithi097 ceph-mon[32231]: pgmap v10160: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:01:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:05 smithi115 ceph-mon[35364]: pgmap v10161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:05 smithi181 ceph-mon[35869]: pgmap v10161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:05 smithi097 ceph-mon[32231]: pgmap v10161: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:07 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18103.106825 secs 2024-07-26T13:01:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:07 smithi115 ceph-mon[35364]: pgmap v10162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:07 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18103.106825 secs 2024-07-26T13:01:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:07 smithi181 ceph-mon[35869]: pgmap v10162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:07 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18103.106825 secs 2024-07-26T13:01:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:07 smithi097 ceph-mon[32231]: pgmap v10162: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:09.305 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:09 smithi097 ceph-mon[32231]: pgmap v10163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:09 smithi115 ceph-mon[35364]: pgmap v10163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:09 smithi181 ceph-mon[35869]: pgmap v10163: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:09.991+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:01:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:09.992+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:01:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:09.992+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:01:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:09.992+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:01:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:09.992+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:10.120+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:01:10.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:10.120+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:01:10.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:10.121+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:01:10.122 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:10.121+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:01:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:10 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:10 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:10.483 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:10 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:11 smithi115 ceph-mon[35364]: pgmap v10164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:11 smithi181 ceph-mon[35869]: pgmap v10164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:11 smithi097 ceph-mon[32231]: pgmap v10164: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:12 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18108.106904 secs 2024-07-26T13:01:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:12 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18108.106904 secs 2024-07-26T13:01:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:12 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18108.106904 secs 2024-07-26T13:01:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:13 smithi115 ceph-mon[35364]: pgmap v10165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:13 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:13 smithi181 ceph-mon[35869]: pgmap v10165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:13 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:13 smithi097 ceph-mon[32231]: pgmap v10165: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:13 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:14.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:01:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:15 smithi115 ceph-mon[35364]: pgmap v10166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:15 smithi115 ceph-mon[35364]: pgmap v10167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:15 smithi115 ceph-mon[35364]: pgmap v10168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:01:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:15 smithi181 ceph-mon[35869]: pgmap v10166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:15 smithi181 ceph-mon[35869]: pgmap v10167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:15 smithi181 ceph-mon[35869]: pgmap v10168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:01:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:15 smithi097 ceph-mon[32231]: pgmap v10166: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:15 smithi097 ceph-mon[32231]: pgmap v10167: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:15 smithi097 ceph-mon[32231]: pgmap v10168: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:01:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18113.107058 secs 2024-07-26T13:01:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:16 smithi115 ceph-mon[35364]: pgmap v10169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18113.107058 secs 2024-07-26T13:01:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:16 smithi181 ceph-mon[35869]: pgmap v10169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18113.107058 secs 2024-07-26T13:01:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:16 smithi097 ceph-mon[32231]: pgmap v10169: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:18 smithi115 ceph-mon[35364]: pgmap v10170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:18 smithi181 ceph-mon[35869]: pgmap v10170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:18 smithi097 ceph-mon[32231]: pgmap v10170: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:20 smithi115 ceph-mon[35364]: pgmap v10171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:20 smithi181 ceph-mon[35869]: pgmap v10171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:20 smithi097 ceph-mon[32231]: pgmap v10171: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18118.107218 secs 2024-07-26T13:01:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18118.107218 secs 2024-07-26T13:01:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18118.107218 secs 2024-07-26T13:01:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:22 smithi115 ceph-mon[35364]: pgmap v10172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T13:01:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:22 smithi181 ceph-mon[35869]: pgmap v10172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T13:01:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:22 smithi097 ceph-mon[32231]: pgmap v10172: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T13:01:24.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:24.996+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:01:24.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:24.996+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:01:24.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:24.996+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:01:24.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:24.996+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:01:24.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:24.996+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:24 smithi115 ceph-mon[35364]: pgmap v10173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:24 smithi181 ceph-mon[35869]: pgmap v10173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:24 smithi097 ceph-mon[32231]: pgmap v10173: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18123.107375 secs 2024-07-26T13:01:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:26 smithi115 ceph-mon[35364]: pgmap v10174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18123.107375 secs 2024-07-26T13:01:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:26 smithi181 ceph-mon[35869]: pgmap v10174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18123.107375 secs 2024-07-26T13:01:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:26 smithi097 ceph-mon[32231]: pgmap v10174: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:28 smithi115 ceph-mon[35364]: pgmap v10175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:28 smithi181 ceph-mon[35869]: pgmap v10175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:28 smithi097 ceph-mon[32231]: pgmap v10175: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:30 smithi115 ceph-mon[35364]: pgmap v10176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:30 smithi181 ceph-mon[35869]: pgmap v10176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:30 smithi097 ceph-mon[32231]: pgmap v10176: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18128.107510 secs 2024-07-26T13:01:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18128.107510 secs 2024-07-26T13:01:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18128.107510 secs 2024-07-26T13:01:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:32 smithi115 ceph-mon[35364]: pgmap v10177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:32 smithi181 ceph-mon[35869]: pgmap v10177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:32 smithi097 ceph-mon[32231]: pgmap v10177: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:34 smithi115 ceph-mon[35364]: pgmap v10178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:34 smithi181 ceph-mon[35869]: pgmap v10178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:34 smithi097 ceph-mon[32231]: pgmap v10178: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18133.107610 secs 2024-07-26T13:01:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:36 smithi115 ceph-mon[35364]: pgmap v10179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18133.107610 secs 2024-07-26T13:01:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:36 smithi181 ceph-mon[35869]: pgmap v10179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18133.107610 secs 2024-07-26T13:01:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:36 smithi097 ceph-mon[32231]: pgmap v10179: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:38 smithi115 ceph-mon[35364]: pgmap v10180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:38 smithi181 ceph-mon[35869]: pgmap v10180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:38 smithi097 ceph-mon[32231]: pgmap v10180: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:40.004+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:01:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:40.005+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:01:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:40.005+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:01:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:40.005+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:01:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:40.005+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:40 smithi115 ceph-mon[35364]: pgmap v10181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:40 smithi181 ceph-mon[35869]: pgmap v10181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:40 smithi097 ceph-mon[32231]: pgmap v10181: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18138.107747 secs 2024-07-26T13:01:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18138.107747 secs 2024-07-26T13:01:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18138.107747 secs 2024-07-26T13:01:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:42 smithi097 ceph-mon[32231]: pgmap v10182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:42 smithi115 ceph-mon[35364]: pgmap v10182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:42 smithi181 ceph-mon[35869]: pgmap v10182: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:45 smithi115 ceph-mon[35364]: pgmap v10183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:45 smithi181 ceph-mon[35869]: pgmap v10183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:45 smithi097 ceph-mon[32231]: pgmap v10183: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:47 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18143.107903 secs 2024-07-26T13:01:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:47 smithi115 ceph-mon[35364]: pgmap v10184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:47 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18143.107903 secs 2024-07-26T13:01:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:47 smithi181 ceph-mon[35869]: pgmap v10184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:47 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18143.107903 secs 2024-07-26T13:01:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:47 smithi097 ceph-mon[32231]: pgmap v10184: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:49 smithi115 ceph-mon[35364]: pgmap v10185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:49 smithi181 ceph-mon[35869]: pgmap v10185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:49 smithi097 ceph-mon[32231]: pgmap v10185: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:51 smithi115 ceph-mon[35364]: pgmap v10186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:51 smithi181 ceph-mon[35869]: pgmap v10186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:51 smithi097 ceph-mon[32231]: pgmap v10186: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:52 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18148.108046 secs 2024-07-26T13:01:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:52 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18148.108046 secs 2024-07-26T13:01:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:52 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18148.108046 secs 2024-07-26T13:01:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:53 smithi115 ceph-mon[35364]: pgmap v10187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:53 smithi181 ceph-mon[35869]: pgmap v10187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:53 smithi097 ceph-mon[32231]: pgmap v10187: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:55.010 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:55.009+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:01:55.011 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:55.010+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:01:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:55.010+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:01:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:55.010+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:01:55.012 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:01:55.010+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:55 smithi115 ceph-mon[35364]: pgmap v10188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:55 smithi181 ceph-mon[35869]: pgmap v10188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:55 smithi097 ceph-mon[32231]: pgmap v10188: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:57 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18153.108188 secs 2024-07-26T13:01:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:57 smithi115 ceph-mon[35364]: pgmap v10189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:57 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18153.108188 secs 2024-07-26T13:01:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:57 smithi181 ceph-mon[35869]: pgmap v10189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:57 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18153.108188 secs 2024-07-26T13:01:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:57 smithi097 ceph-mon[32231]: pgmap v10189: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:01:58 smithi097 ceph-mon[32231]: pgmap v10190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:01:58 smithi115 ceph-mon[35364]: pgmap v10190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:01:58 smithi181 ceph-mon[35869]: pgmap v10190: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:00 smithi115 ceph-mon[35364]: pgmap v10191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:00 smithi181 ceph-mon[35869]: pgmap v10191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:00 smithi097 ceph-mon[32231]: pgmap v10191: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18158.108329 secs 2024-07-26T13:02:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18158.108329 secs 2024-07-26T13:02:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18158.108329 secs 2024-07-26T13:02:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:02 smithi115 ceph-mon[35364]: pgmap v10192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:02 smithi181 ceph-mon[35869]: pgmap v10192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:02 smithi097 ceph-mon[32231]: pgmap v10192: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:04 smithi115 ceph-mon[35364]: pgmap v10193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:04 smithi181 ceph-mon[35869]: pgmap v10193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:04 smithi097 ceph-mon[32231]: pgmap v10193: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18163.108426 secs 2024-07-26T13:02:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:06 smithi115 ceph-mon[35364]: pgmap v10194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18163.108426 secs 2024-07-26T13:02:07.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:06 smithi181 ceph-mon[35869]: pgmap v10194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18163.108426 secs 2024-07-26T13:02:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:06 smithi097 ceph-mon[32231]: pgmap v10194: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:08 smithi115 ceph-mon[35364]: pgmap v10195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:08 smithi181 ceph-mon[35869]: pgmap v10195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:08 smithi097 ceph-mon[32231]: pgmap v10195: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:10.018+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:02:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:10.019+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:02:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:10.019+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:02:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:10.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:02:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:10.019+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:10 smithi115 ceph-mon[35364]: pgmap v10196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:10 smithi181 ceph-mon[35869]: pgmap v10196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:10 smithi097 ceph-mon[32231]: pgmap v10196: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18168.108568 secs 2024-07-26T13:02:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18168.108568 secs 2024-07-26T13:02:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18168.108568 secs 2024-07-26T13:02:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:12 smithi097 ceph-mon[32231]: pgmap v10197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:12 smithi115 ceph-mon[35364]: pgmap v10197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:13.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:12 smithi181 ceph-mon[35869]: pgmap v10197: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:14 smithi097 ceph-mon[32231]: pgmap v10198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:14 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:14 smithi115 ceph-mon[35364]: pgmap v10198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:14 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:14 smithi181 ceph-mon[35869]: pgmap v10198: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:14 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:02:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:16.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:02:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:02:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:17 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18173.108697 secs 2024-07-26T13:02:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:17 smithi115 ceph-mon[35364]: pgmap v10199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:02:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:17 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18173.108697 secs 2024-07-26T13:02:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:17 smithi181 ceph-mon[35869]: pgmap v10199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:02:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:17 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18173.108697 secs 2024-07-26T13:02:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:17 smithi097 ceph-mon[32231]: pgmap v10199: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:02:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:19 smithi115 ceph-mon[35364]: pgmap v10200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:02:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:19 smithi181 ceph-mon[35869]: pgmap v10200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:02:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:19 smithi097 ceph-mon[32231]: pgmap v10200: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:02:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:21 smithi115 ceph-mon[35364]: pgmap v10201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:02:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:21 smithi181 ceph-mon[35869]: pgmap v10201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:02:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:21 smithi097 ceph-mon[32231]: pgmap v10201: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:02:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:22 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18178.108845 secs 2024-07-26T13:02:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:22 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18178.108845 secs 2024-07-26T13:02:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:22 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18178.108845 secs 2024-07-26T13:02:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:23 smithi115 ceph-mon[35364]: pgmap v10202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:02:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:23 smithi181 ceph-mon[35869]: pgmap v10202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:02:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:23 smithi097 ceph-mon[32231]: pgmap v10202: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:02:25.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:25.022+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:02:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:25.023+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:02:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:25.023+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:02:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:25.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:02:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:25.023+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:25 smithi115 ceph-mon[35364]: pgmap v10203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:02:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:25 smithi181 ceph-mon[35869]: pgmap v10203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:02:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:25 smithi097 ceph-mon[32231]: pgmap v10203: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:02:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:27 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18183.108989 secs 2024-07-26T13:02:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:27 smithi115 ceph-mon[35364]: pgmap v10204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:02:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:27 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18183.108989 secs 2024-07-26T13:02:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:27 smithi181 ceph-mon[35869]: pgmap v10204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:02:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:27 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18183.108989 secs 2024-07-26T13:02:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:27 smithi097 ceph-mon[32231]: pgmap v10204: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:02:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:29 smithi115 ceph-mon[35364]: pgmap v10205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:29 smithi181 ceph-mon[35869]: pgmap v10205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:29 smithi097 ceph-mon[32231]: pgmap v10205: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:31 smithi115 ceph-mon[35364]: pgmap v10206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:31 smithi181 ceph-mon[35869]: pgmap v10206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:31 smithi097 ceph-mon[32231]: pgmap v10206: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:32 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18188.109137 secs 2024-07-26T13:02:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:32 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18188.109137 secs 2024-07-26T13:02:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:32 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18188.109137 secs 2024-07-26T13:02:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:33 smithi115 ceph-mon[35364]: pgmap v10207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:33 smithi181 ceph-mon[35869]: pgmap v10207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:33 smithi097 ceph-mon[32231]: pgmap v10207: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:35 smithi115 ceph-mon[35364]: pgmap v10208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:35 smithi181 ceph-mon[35869]: pgmap v10208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:35 smithi097 ceph-mon[32231]: pgmap v10208: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18193.109318 secs 2024-07-26T13:02:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:36 smithi115 ceph-mon[35364]: pgmap v10209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18193.109318 secs 2024-07-26T13:02:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:36 smithi181 ceph-mon[35869]: pgmap v10209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18193.109318 secs 2024-07-26T13:02:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:36 smithi097 ceph-mon[32231]: pgmap v10209: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:38 smithi115 ceph-mon[35364]: pgmap v10210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:38 smithi181 ceph-mon[35869]: pgmap v10210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:38 smithi097 ceph-mon[32231]: pgmap v10210: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:40.024+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:02:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:40.025+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:02:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:40.025+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:02:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:40.025+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:02:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:40.025+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:40 smithi115 ceph-mon[35364]: pgmap v10211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:40 smithi181 ceph-mon[35869]: pgmap v10211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:40 smithi097 ceph-mon[32231]: pgmap v10211: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18198.109465 secs 2024-07-26T13:02:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18198.109465 secs 2024-07-26T13:02:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18198.109465 secs 2024-07-26T13:02:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:42 smithi115 ceph-mon[35364]: pgmap v10212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:42 smithi181 ceph-mon[35869]: pgmap v10212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:42 smithi097 ceph-mon[32231]: pgmap v10212: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:44 smithi115 ceph-mon[35364]: pgmap v10213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:44 smithi181 ceph-mon[35869]: pgmap v10213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:44 smithi097 ceph-mon[32231]: pgmap v10213: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18203.109605 secs 2024-07-26T13:02:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:46 smithi115 ceph-mon[35364]: pgmap v10214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18203.109605 secs 2024-07-26T13:02:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:46 smithi181 ceph-mon[35869]: pgmap v10214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18203.109605 secs 2024-07-26T13:02:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:46 smithi097 ceph-mon[32231]: pgmap v10214: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:48 smithi115 ceph-mon[35364]: pgmap v10215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:48 smithi181 ceph-mon[35869]: pgmap v10215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:48 smithi097 ceph-mon[32231]: pgmap v10215: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:50 smithi115 ceph-mon[35364]: pgmap v10216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:50 smithi181 ceph-mon[35869]: pgmap v10216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:50 smithi097 ceph-mon[32231]: pgmap v10216: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18208.109741 secs 2024-07-26T13:02:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18208.109741 secs 2024-07-26T13:02:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18208.109741 secs 2024-07-26T13:02:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:52 smithi115 ceph-mon[35364]: pgmap v10217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:52 smithi181 ceph-mon[35869]: pgmap v10217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:52 smithi097 ceph-mon[32231]: pgmap v10217: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:54 smithi115 ceph-mon[35364]: pgmap v10218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:54 smithi181 ceph-mon[35869]: pgmap v10218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:54 smithi097 ceph-mon[32231]: pgmap v10218: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:02:55.030 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:55.029+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:02:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:55.030+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:02:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:55.030+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:02:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:55.030+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:02:55.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:02:55.030+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18213.109852 secs 2024-07-26T13:02:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:56 smithi115 ceph-mon[35364]: pgmap v10219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18213.109852 secs 2024-07-26T13:02:56.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:56 smithi181 ceph-mon[35869]: pgmap v10219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18213.109852 secs 2024-07-26T13:02:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:56 smithi097 ceph-mon[32231]: pgmap v10219: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:02:58 smithi115 ceph-mon[35364]: pgmap v10220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:02:58 smithi181 ceph-mon[35869]: pgmap v10220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:02:58 smithi097 ceph-mon[32231]: pgmap v10220: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:00 smithi115 ceph-mon[35364]: pgmap v10221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:03:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:00 smithi181 ceph-mon[35869]: pgmap v10221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:03:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:00 smithi097 ceph-mon[32231]: pgmap v10221: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:03:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18218.110005 secs 2024-07-26T13:03:01.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18218.110005 secs 2024-07-26T13:03:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18218.110005 secs 2024-07-26T13:03:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:02 smithi115 ceph-mon[35364]: pgmap v10222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:02 smithi181 ceph-mon[35869]: pgmap v10222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:02 smithi097 ceph-mon[32231]: pgmap v10222: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:04 smithi115 ceph-mon[35364]: pgmap v10223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:04 smithi181 ceph-mon[35869]: pgmap v10223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:04 smithi097 ceph-mon[32231]: pgmap v10223: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18223.110145 secs 2024-07-26T13:03:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:06 smithi115 ceph-mon[35364]: pgmap v10224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18223.110145 secs 2024-07-26T13:03:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:06 smithi181 ceph-mon[35869]: pgmap v10224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:06.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18223.110145 secs 2024-07-26T13:03:06.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:06 smithi097 ceph-mon[32231]: pgmap v10224: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:08 smithi181 ceph-mon[35869]: pgmap v10225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:08 smithi097 ceph-mon[32231]: pgmap v10225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:08 smithi115 ceph-mon[35364]: pgmap v10225: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:10.032+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:03:10.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:10.033+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:03:10.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:10.033+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:03:10.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:10.033+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:03:10.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:10.034+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:10 smithi181 ceph-mon[35869]: pgmap v10226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:03:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:10 smithi097 ceph-mon[32231]: pgmap v10226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:03:10.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:10 smithi115 ceph-mon[35364]: pgmap v10226: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:03:11.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18228.110295 secs 2024-07-26T13:03:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18228.110295 secs 2024-07-26T13:03:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18228.110295 secs 2024-07-26T13:03:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:12 smithi097 ceph-mon[32231]: pgmap v10227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:12 smithi115 ceph-mon[35364]: pgmap v10227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:12 smithi181 ceph-mon[35869]: pgmap v10227: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:14 smithi115 ceph-mon[35364]: pgmap v10228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:14 smithi181 ceph-mon[35869]: pgmap v10228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:15.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:14 smithi097 ceph-mon[32231]: pgmap v10228: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18233.110436 secs 2024-07-26T13:03:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:16 smithi115 ceph-mon[35364]: pgmap v10229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:16 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18233.110436 secs 2024-07-26T13:03:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:16 smithi181 ceph-mon[35869]: pgmap v10229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:16 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18233.110436 secs 2024-07-26T13:03:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:16 smithi097 ceph-mon[32231]: pgmap v10229: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:16 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:17 smithi115 ceph-mon[35364]: pgmap v10230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:03:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:17 smithi115 ceph-mon[35364]: pgmap v10231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:03:17.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:03:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:17 smithi181 ceph-mon[35869]: pgmap v10230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:03:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:17 smithi181 ceph-mon[35869]: pgmap v10231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:03:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:03:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:17 smithi097 ceph-mon[32231]: pgmap v10230: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:03:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:17 smithi097 ceph-mon[32231]: pgmap v10231: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:03:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:03:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:19 smithi097 ceph-mon[32231]: pgmap v10232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:19 smithi115 ceph-mon[35364]: pgmap v10232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:19 smithi181 ceph-mon[35869]: pgmap v10232: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18238.110606 secs 2024-07-26T13:03:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:21 smithi097 ceph-mon[32231]: pgmap v10233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:03:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18238.110606 secs 2024-07-26T13:03:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:21 smithi115 ceph-mon[35364]: pgmap v10233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:03:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18238.110606 secs 2024-07-26T13:03:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:21 smithi181 ceph-mon[35869]: pgmap v10233: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:03:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:23 smithi115 ceph-mon[35364]: pgmap v10234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:03:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:23 smithi181 ceph-mon[35869]: pgmap v10234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:03:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:23 smithi097 ceph-mon[32231]: pgmap v10234: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:03:25.038 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.037+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:03:25.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.038+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:03:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.038+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:03:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.038+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:03:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.038+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.144+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:03:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.144+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:03:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.144+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:03:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:25.145+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:03:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:25 smithi115 ceph-mon[35364]: pgmap v10235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:03:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:25 smithi181 ceph-mon[35869]: pgmap v10235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:03:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:25 smithi097 ceph-mon[32231]: pgmap v10235: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:03:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18243.110746 secs 2024-07-26T13:03:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18243.110746 secs 2024-07-26T13:03:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18243.110746 secs 2024-07-26T13:03:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:27 smithi115 ceph-mon[35364]: pgmap v10236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:27 smithi181 ceph-mon[35869]: pgmap v10236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:27 smithi097 ceph-mon[32231]: pgmap v10236: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:29 smithi115 ceph-mon[35364]: pgmap v10237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:29 smithi181 ceph-mon[35869]: pgmap v10237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:29 smithi097 ceph-mon[32231]: pgmap v10237: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18248.110885 secs 2024-07-26T13:03:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:31 smithi115 ceph-mon[35364]: pgmap v10238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18248.110885 secs 2024-07-26T13:03:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:31 smithi181 ceph-mon[35869]: pgmap v10238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18248.110885 secs 2024-07-26T13:03:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:31 smithi097 ceph-mon[32231]: pgmap v10238: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:33 smithi115 ceph-mon[35364]: pgmap v10239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:33 smithi181 ceph-mon[35869]: pgmap v10239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:33 smithi097 ceph-mon[32231]: pgmap v10239: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:35 smithi115 ceph-mon[35364]: pgmap v10240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:35 smithi181 ceph-mon[35869]: pgmap v10240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:35 smithi097 ceph-mon[32231]: pgmap v10240: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18253.111019 secs 2024-07-26T13:03:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18253.111019 secs 2024-07-26T13:03:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18253.111019 secs 2024-07-26T13:03:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:37 smithi115 ceph-mon[35364]: pgmap v10241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:37 smithi181 ceph-mon[35869]: pgmap v10241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:37 smithi097 ceph-mon[32231]: pgmap v10241: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:40.045+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:03:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:40.047+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:03:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:40.047+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:03:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:40.047+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:03:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:40.047+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:39 smithi115 ceph-mon[35364]: pgmap v10242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:39 smithi181 ceph-mon[35869]: pgmap v10242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:39 smithi097 ceph-mon[32231]: pgmap v10242: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18258.111149 secs 2024-07-26T13:03:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:41 smithi115 ceph-mon[35364]: pgmap v10243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18258.111149 secs 2024-07-26T13:03:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:41 smithi181 ceph-mon[35869]: pgmap v10243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18258.111149 secs 2024-07-26T13:03:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:41 smithi097 ceph-mon[32231]: pgmap v10243: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:43 smithi115 ceph-mon[35364]: pgmap v10244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:43 smithi181 ceph-mon[35869]: pgmap v10244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:43 smithi097 ceph-mon[32231]: pgmap v10244: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:03:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:45 smithi115 ceph-mon[35364]: pgmap v10245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:45 smithi181 ceph-mon[35869]: pgmap v10245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:45 smithi097 ceph-mon[32231]: pgmap v10245: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18263.111287 secs 2024-07-26T13:03:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18263.111287 secs 2024-07-26T13:03:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18263.111287 secs 2024-07-26T13:03:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:47 smithi115 ceph-mon[35364]: pgmap v10246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:47 smithi181 ceph-mon[35869]: pgmap v10246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:47 smithi097 ceph-mon[32231]: pgmap v10246: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:49 smithi115 ceph-mon[35364]: pgmap v10247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:49 smithi181 ceph-mon[35869]: pgmap v10247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:49 smithi097 ceph-mon[32231]: pgmap v10247: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18268.111423 secs 2024-07-26T13:03:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:51 smithi115 ceph-mon[35364]: pgmap v10248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:03:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18268.111423 secs 2024-07-26T13:03:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:51 smithi181 ceph-mon[35869]: pgmap v10248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:03:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18268.111423 secs 2024-07-26T13:03:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:51 smithi097 ceph-mon[32231]: pgmap v10248: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:03:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:53 smithi115 ceph-mon[35364]: pgmap v10249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:03:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:53 smithi181 ceph-mon[35869]: pgmap v10249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:03:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:53 smithi097 ceph-mon[32231]: pgmap v10249: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:03:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:55.050+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:03:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:55.052+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:03:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:55.052+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:03:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:55.052+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:03:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:03:55.052+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:55 smithi115 ceph-mon[35364]: pgmap v10250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:55 smithi181 ceph-mon[35869]: pgmap v10250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:55 smithi097 ceph-mon[32231]: pgmap v10250: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:03:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18273.111563 secs 2024-07-26T13:03:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18273.111563 secs 2024-07-26T13:03:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18273.111563 secs 2024-07-26T13:03:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:57 smithi115 ceph-mon[35364]: pgmap v10251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:57 smithi181 ceph-mon[35869]: pgmap v10251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:57 smithi097 ceph-mon[32231]: pgmap v10251: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:03:59 smithi115 ceph-mon[35364]: pgmap v10252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:03:59 smithi181 ceph-mon[35869]: pgmap v10252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:03:59 smithi097 ceph-mon[32231]: pgmap v10252: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18278.111702 secs 2024-07-26T13:04:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:01 smithi115 ceph-mon[35364]: pgmap v10253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18278.111702 secs 2024-07-26T13:04:02.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:01 smithi181 ceph-mon[35869]: pgmap v10253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18278.111702 secs 2024-07-26T13:04:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:01 smithi097 ceph-mon[32231]: pgmap v10253: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:03 smithi115 ceph-mon[35364]: pgmap v10254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:03 smithi181 ceph-mon[35869]: pgmap v10254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:03 smithi097 ceph-mon[32231]: pgmap v10254: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:05 smithi115 ceph-mon[35364]: pgmap v10255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:05 smithi181 ceph-mon[35869]: pgmap v10255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:05 smithi097 ceph-mon[32231]: pgmap v10255: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18283.111841 secs 2024-07-26T13:04:07.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18283.111841 secs 2024-07-26T13:04:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18283.111841 secs 2024-07-26T13:04:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:07 smithi115 ceph-mon[35364]: pgmap v10256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:07 smithi181 ceph-mon[35869]: pgmap v10256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:07 smithi097 ceph-mon[32231]: pgmap v10256: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.060+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:04:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.061+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:04:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.061+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:04:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.061+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:04:10.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.061+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.152+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:04:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.152+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:04:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.153+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:04:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:10.153+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:04:10.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:09 smithi115 ceph-mon[35364]: pgmap v10257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:09 smithi181 ceph-mon[35869]: pgmap v10257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:09 smithi097 ceph-mon[32231]: pgmap v10257: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18288.112005 secs 2024-07-26T13:04:12.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:11 smithi115 ceph-mon[35364]: pgmap v10258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18288.112005 secs 2024-07-26T13:04:12.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:11 smithi181 ceph-mon[35869]: pgmap v10258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18288.112005 secs 2024-07-26T13:04:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:11 smithi097 ceph-mon[32231]: pgmap v10258: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:13 smithi181 ceph-mon[35869]: pgmap v10259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:13 smithi097 ceph-mon[32231]: pgmap v10259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:13 smithi115 ceph-mon[35364]: pgmap v10259: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:15 smithi181 ceph-mon[35869]: pgmap v10260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:15 smithi097 ceph-mon[32231]: pgmap v10260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:15 smithi115 ceph-mon[35364]: pgmap v10260: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18293.112170 secs 2024-07-26T13:04:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18293.112170 secs 2024-07-26T13:04:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18293.112170 secs 2024-07-26T13:04:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:17 smithi097 ceph-mon[32231]: pgmap v10261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:18.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:17 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:17 smithi115 ceph-mon[35364]: pgmap v10261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:17 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:17 smithi181 ceph-mon[35869]: pgmap v10261: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:17 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:19.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:04:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:19.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:04:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:04:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:20 smithi115 ceph-mon[35364]: pgmap v10262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:04:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:20 smithi115 ceph-mon[35364]: pgmap v10263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:20 smithi181 ceph-mon[35869]: pgmap v10262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:04:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:20 smithi181 ceph-mon[35869]: pgmap v10263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:20 smithi097 ceph-mon[32231]: pgmap v10262: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:04:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:20 smithi097 ceph-mon[32231]: pgmap v10263: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:22 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18298.112315 secs 2024-07-26T13:04:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:22 smithi115 ceph-mon[35364]: pgmap v10264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:22 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18298.112315 secs 2024-07-26T13:04:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:22 smithi181 ceph-mon[35869]: pgmap v10264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:22 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18298.112315 secs 2024-07-26T13:04:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:22 smithi097 ceph-mon[32231]: pgmap v10264: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:24 smithi115 ceph-mon[35364]: pgmap v10265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:24 smithi181 ceph-mon[35869]: pgmap v10265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:24 smithi097 ceph-mon[32231]: pgmap v10265: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:25.063+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:04:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:25.064+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:04:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:25.064+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:04:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:25.064+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:04:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:25.064+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:26 smithi115 ceph-mon[35364]: pgmap v10266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:26 smithi181 ceph-mon[35869]: pgmap v10266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:26 smithi097 ceph-mon[32231]: pgmap v10266: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:27 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18303.112464 secs 2024-07-26T13:04:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:27 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18303.112464 secs 2024-07-26T13:04:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:27 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18303.112464 secs 2024-07-26T13:04:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:28 smithi115 ceph-mon[35364]: pgmap v10267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:04:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:28 smithi181 ceph-mon[35869]: pgmap v10267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:04:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:28 smithi097 ceph-mon[32231]: pgmap v10267: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:04:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:30 smithi115 ceph-mon[35364]: pgmap v10268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:04:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:30 smithi181 ceph-mon[35869]: pgmap v10268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:04:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:30 smithi097 ceph-mon[32231]: pgmap v10268: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:04:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:32 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18308.112605 secs 2024-07-26T13:04:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:32 smithi115 ceph-mon[35364]: pgmap v10269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:04:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:32 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18308.112605 secs 2024-07-26T13:04:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:32 smithi181 ceph-mon[35869]: pgmap v10269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:04:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:32 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18308.112605 secs 2024-07-26T13:04:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:32 smithi097 ceph-mon[32231]: pgmap v10269: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:04:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:34 smithi115 ceph-mon[35364]: pgmap v10270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:34 smithi181 ceph-mon[35869]: pgmap v10270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:34 smithi097 ceph-mon[32231]: pgmap v10270: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:36 smithi115 ceph-mon[35364]: pgmap v10271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:04:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:36 smithi181 ceph-mon[35869]: pgmap v10271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:04:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:36 smithi097 ceph-mon[32231]: pgmap v10271: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:04:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:37 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18313.112749 secs 2024-07-26T13:04:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:37 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18313.112749 secs 2024-07-26T13:04:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:37 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18313.112749 secs 2024-07-26T13:04:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:38 smithi115 ceph-mon[35364]: pgmap v10272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:38 smithi181 ceph-mon[35869]: pgmap v10272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:38 smithi097 ceph-mon[32231]: pgmap v10272: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:40.072+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:04:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:40.073+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:04:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:40.073+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:04:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:40.073+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:04:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:40.074+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:40 smithi115 ceph-mon[35364]: pgmap v10273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:40 smithi181 ceph-mon[35869]: pgmap v10273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:40 smithi097 ceph-mon[32231]: pgmap v10273: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18318.112860 secs 2024-07-26T13:04:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:41 smithi115 ceph-mon[35364]: pgmap v10274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18318.112860 secs 2024-07-26T13:04:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:41 smithi181 ceph-mon[35869]: pgmap v10274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18318.112860 secs 2024-07-26T13:04:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:41 smithi097 ceph-mon[32231]: pgmap v10274: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:43 smithi115 ceph-mon[35364]: pgmap v10275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:43 smithi181 ceph-mon[35869]: pgmap v10275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:43 smithi097 ceph-mon[32231]: pgmap v10275: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:45 smithi115 ceph-mon[35364]: pgmap v10276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:45.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:45 smithi181 ceph-mon[35869]: pgmap v10276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:45 smithi097 ceph-mon[32231]: pgmap v10276: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18323.112995 secs 2024-07-26T13:04:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18323.112995 secs 2024-07-26T13:04:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18323.112995 secs 2024-07-26T13:04:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:47 smithi115 ceph-mon[35364]: pgmap v10277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:47 smithi181 ceph-mon[35869]: pgmap v10277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:47 smithi097 ceph-mon[32231]: pgmap v10277: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:49 smithi115 ceph-mon[35364]: pgmap v10278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:49 smithi181 ceph-mon[35869]: pgmap v10278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:49 smithi097 ceph-mon[32231]: pgmap v10278: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18328.113138 secs 2024-07-26T13:04:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:51 smithi115 ceph-mon[35364]: pgmap v10279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18328.113138 secs 2024-07-26T13:04:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:51 smithi181 ceph-mon[35869]: pgmap v10279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18328.113138 secs 2024-07-26T13:04:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:51 smithi097 ceph-mon[32231]: pgmap v10279: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:53 smithi115 ceph-mon[35364]: pgmap v10280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:53 smithi181 ceph-mon[35869]: pgmap v10280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:53 smithi097 ceph-mon[32231]: pgmap v10280: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:55.077+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:04:55.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:55.078+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:04:55.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:55.078+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:04:55.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:55.078+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:04:55.079 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:04:55.078+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:55 smithi115 ceph-mon[35364]: pgmap v10281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:55 smithi181 ceph-mon[35869]: pgmap v10281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:55 smithi097 ceph-mon[32231]: pgmap v10281: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18333.113282 secs 2024-07-26T13:04:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18333.113282 secs 2024-07-26T13:04:56.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18333.113282 secs 2024-07-26T13:04:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:57 smithi097 ceph-mon[32231]: pgmap v10282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:57.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:57 smithi115 ceph-mon[35364]: pgmap v10282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:57 smithi181 ceph-mon[35869]: pgmap v10282: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:04:59 smithi097 ceph-mon[32231]: pgmap v10283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:04:59 smithi115 ceph-mon[35364]: pgmap v10283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:04:59 smithi181 ceph-mon[35869]: pgmap v10283: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18338.113420 secs 2024-07-26T13:05:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:01 smithi115 ceph-mon[35364]: pgmap v10284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18338.113420 secs 2024-07-26T13:05:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:01 smithi181 ceph-mon[35869]: pgmap v10284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18338.113420 secs 2024-07-26T13:05:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:01 smithi097 ceph-mon[32231]: pgmap v10284: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:03.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:03 smithi115 ceph-mon[35364]: pgmap v10285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:03 smithi181 ceph-mon[35869]: pgmap v10285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:03.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:03 smithi097 ceph-mon[32231]: pgmap v10285: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:05 smithi115 ceph-mon[35364]: pgmap v10286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:05:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:05 smithi181 ceph-mon[35869]: pgmap v10286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:05:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:05 smithi097 ceph-mon[32231]: pgmap v10286: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:05:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18343.113560 secs 2024-07-26T13:05:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18343.113560 secs 2024-07-26T13:05:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18343.113560 secs 2024-07-26T13:05:07.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:07 smithi115 ceph-mon[35364]: pgmap v10287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:07 smithi181 ceph-mon[35869]: pgmap v10287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:07.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:07 smithi097 ceph-mon[32231]: pgmap v10287: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:09 smithi115 ceph-mon[35364]: pgmap v10288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:09 smithi181 ceph-mon[35869]: pgmap v10288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:09 smithi097 ceph-mon[32231]: pgmap v10288: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:10.085 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:10.083+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:10.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:10.084+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:05:10.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:10.084+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:05:10.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:10.084+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:10.086 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:10.085+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:11.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18348.113705 secs 2024-07-26T13:05:11.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:11 smithi097 ceph-mon[32231]: pgmap v10289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18348.113705 secs 2024-07-26T13:05:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:11 smithi115 ceph-mon[35364]: pgmap v10289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18348.113705 secs 2024-07-26T13:05:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:11 smithi181 ceph-mon[35869]: pgmap v10289: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:13 smithi115 ceph-mon[35364]: pgmap v10290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:13 smithi181 ceph-mon[35869]: pgmap v10290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:13.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:13 smithi097 ceph-mon[32231]: pgmap v10290: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:15 smithi115 ceph-mon[35364]: pgmap v10291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:05:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:15 smithi181 ceph-mon[35869]: pgmap v10291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:05:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:15 smithi097 ceph-mon[32231]: pgmap v10291: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:05:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18353.113845 secs 2024-07-26T13:05:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18353.113845 secs 2024-07-26T13:05:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18353.113845 secs 2024-07-26T13:05:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:17 smithi115 ceph-mon[35364]: pgmap v10292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:17 smithi181 ceph-mon[35869]: pgmap v10292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:17.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:17 smithi097 ceph-mon[32231]: pgmap v10292: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:19 smithi115 ceph-mon[35364]: pgmap v10293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:19.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:19 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:19 smithi181 ceph-mon[35869]: pgmap v10293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:19.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:19 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:19 smithi097 ceph-mon[32231]: pgmap v10293: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:19 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:20 smithi115 ceph-mon[35364]: pgmap v10294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:05:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:20 smithi115 ceph-mon[35364]: pgmap v10295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:20.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:05:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:20 smithi181 ceph-mon[35869]: pgmap v10294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:05:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:20 smithi181 ceph-mon[35869]: pgmap v10295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:05:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:20 smithi097 ceph-mon[32231]: pgmap v10294: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:05:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:20 smithi097 ceph-mon[32231]: pgmap v10295: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:05:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18358.113978 secs 2024-07-26T13:05:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18358.113978 secs 2024-07-26T13:05:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18358.113978 secs 2024-07-26T13:05:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:22 smithi115 ceph-mon[35364]: pgmap v10296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:22.758 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:22 smithi097 ceph-mon[32231]: pgmap v10296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:22.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:22 smithi181 ceph-mon[35869]: pgmap v10296: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:24 smithi115 ceph-mon[35364]: pgmap v10297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:05:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:24 smithi181 ceph-mon[35869]: pgmap v10297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:05:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:24 smithi097 ceph-mon[32231]: pgmap v10297: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:05:25.088 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:25.087+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:25.088+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:05:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:25.088+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:05:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:25.088+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:25.088+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18363.114118 secs 2024-07-26T13:05:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:26 smithi115 ceph-mon[35364]: pgmap v10298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18363.114118 secs 2024-07-26T13:05:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:26 smithi181 ceph-mon[35869]: pgmap v10298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18363.114118 secs 2024-07-26T13:05:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:26 smithi097 ceph-mon[32231]: pgmap v10298: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:28 smithi115 ceph-mon[35364]: pgmap v10299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:28 smithi181 ceph-mon[35869]: pgmap v10299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:28 smithi097 ceph-mon[32231]: pgmap v10299: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:30 smithi115 ceph-mon[35364]: pgmap v10300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:30 smithi181 ceph-mon[35869]: pgmap v10300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:30 smithi097 ceph-mon[32231]: pgmap v10300: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18368.114266 secs 2024-07-26T13:05:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18368.114266 secs 2024-07-26T13:05:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18368.114266 secs 2024-07-26T13:05:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:32 smithi115 ceph-mon[35364]: pgmap v10301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:32 smithi181 ceph-mon[35869]: pgmap v10301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:32 smithi097 ceph-mon[32231]: pgmap v10301: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:34 smithi115 ceph-mon[35364]: pgmap v10302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:34 smithi181 ceph-mon[35869]: pgmap v10302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:34 smithi097 ceph-mon[32231]: pgmap v10302: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18373.114401 secs 2024-07-26T13:05:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:36 smithi115 ceph-mon[35364]: pgmap v10303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18373.114401 secs 2024-07-26T13:05:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:36 smithi181 ceph-mon[35869]: pgmap v10303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18373.114401 secs 2024-07-26T13:05:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:36 smithi097 ceph-mon[32231]: pgmap v10303: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:38 smithi115 ceph-mon[35364]: pgmap v10304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:38 smithi181 ceph-mon[35869]: pgmap v10304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:38 smithi097 ceph-mon[32231]: pgmap v10304: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:40.096+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:40.097+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:05:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:40.097+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:05:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:40.097+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:40.097+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:40 smithi115 ceph-mon[35364]: pgmap v10305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:40 smithi181 ceph-mon[35869]: pgmap v10305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:40 smithi097 ceph-mon[32231]: pgmap v10305: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:42.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18378.114563 secs 2024-07-26T13:05:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18378.114563 secs 2024-07-26T13:05:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18378.114563 secs 2024-07-26T13:05:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:42 smithi097 ceph-mon[32231]: pgmap v10306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:43.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:42 smithi115 ceph-mon[35364]: pgmap v10306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:42 smithi181 ceph-mon[35869]: pgmap v10306: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:44 smithi097 ceph-mon[32231]: pgmap v10307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:45.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:44 smithi115 ceph-mon[35364]: pgmap v10307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:44 smithi181 ceph-mon[35869]: pgmap v10307: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18383.114692 secs 2024-07-26T13:05:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18383.114692 secs 2024-07-26T13:05:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18383.114692 secs 2024-07-26T13:05:47.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:46 smithi115 ceph-mon[35364]: pgmap v10308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:47.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:46 smithi181 ceph-mon[35869]: pgmap v10308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:47.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:46 smithi097 ceph-mon[32231]: pgmap v10308: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:49.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:48 smithi115 ceph-mon[35364]: pgmap v10309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:48 smithi181 ceph-mon[35869]: pgmap v10309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:49.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:48 smithi097 ceph-mon[32231]: pgmap v10309: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:50 smithi115 ceph-mon[35364]: pgmap v10310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18388.114821 secs 2024-07-26T13:05:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:50 smithi181 ceph-mon[35869]: pgmap v10310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18388.114821 secs 2024-07-26T13:05:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:50 smithi097 ceph-mon[32231]: pgmap v10310: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18388.114821 secs 2024-07-26T13:05:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:52 smithi115 ceph-mon[35364]: pgmap v10311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:52 smithi181 ceph-mon[35869]: pgmap v10311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:53.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:52 smithi097 ceph-mon[32231]: pgmap v10311: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:55.101 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:55.100+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:05:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:55.101+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:05:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:55.101+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:05:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:55.101+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:05:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:05:55.101+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:54 smithi115 ceph-mon[35364]: pgmap v10312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:54 smithi181 ceph-mon[35869]: pgmap v10312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:54 smithi097 ceph-mon[32231]: pgmap v10312: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18393.114962 secs 2024-07-26T13:05:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18393.114962 secs 2024-07-26T13:05:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18393.114962 secs 2024-07-26T13:05:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:56 smithi115 ceph-mon[35364]: pgmap v10313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:56 smithi181 ceph-mon[35869]: pgmap v10313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:56 smithi097 ceph-mon[32231]: pgmap v10313: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:05:58 smithi115 ceph-mon[35364]: pgmap v10314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:05:58 smithi181 ceph-mon[35869]: pgmap v10314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:05:58 smithi097 ceph-mon[32231]: pgmap v10314: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:00 smithi115 ceph-mon[35364]: pgmap v10315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:01.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18398.115103 secs 2024-07-26T13:06:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:00 smithi181 ceph-mon[35869]: pgmap v10315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18398.115103 secs 2024-07-26T13:06:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:00 smithi097 ceph-mon[32231]: pgmap v10315: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18398.115103 secs 2024-07-26T13:06:03.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:02 smithi115 ceph-mon[35364]: pgmap v10316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:02 smithi181 ceph-mon[35869]: pgmap v10316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:02 smithi097 ceph-mon[32231]: pgmap v10316: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:04 smithi115 ceph-mon[35364]: pgmap v10317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:04 smithi181 ceph-mon[35869]: pgmap v10317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:04 smithi097 ceph-mon[32231]: pgmap v10317: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18403.115277 secs 2024-07-26T13:06:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18403.115277 secs 2024-07-26T13:06:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18403.115277 secs 2024-07-26T13:06:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:06 smithi115 ceph-mon[35364]: pgmap v10318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:06 smithi181 ceph-mon[35869]: pgmap v10318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:06 smithi097 ceph-mon[32231]: pgmap v10318: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:08 smithi115 ceph-mon[35364]: pgmap v10319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:08 smithi181 ceph-mon[35869]: pgmap v10319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:08 smithi097 ceph-mon[32231]: pgmap v10319: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:10.110 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:10.109+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:10.110+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:06:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:10.110+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:06:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:10.110+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:10.110+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:10 smithi115 ceph-mon[35364]: pgmap v10320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18408.115422 secs 2024-07-26T13:06:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:10 smithi181 ceph-mon[35869]: pgmap v10320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:11.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18408.115422 secs 2024-07-26T13:06:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:10 smithi097 ceph-mon[32231]: pgmap v10320: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18408.115422 secs 2024-07-26T13:06:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:12 smithi115 ceph-mon[35364]: pgmap v10321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:12 smithi181 ceph-mon[35869]: pgmap v10321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:12 smithi097 ceph-mon[32231]: pgmap v10321: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:14 smithi115 ceph-mon[35364]: pgmap v10322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:14 smithi181 ceph-mon[35869]: pgmap v10322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:15.253 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:14 smithi097 ceph-mon[32231]: pgmap v10322: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:06:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18413.115551 secs 2024-07-26T13:06:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18413.115551 secs 2024-07-26T13:06:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18413.115551 secs 2024-07-26T13:06:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:16 smithi115 ceph-mon[35364]: pgmap v10323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:16 smithi181 ceph-mon[35869]: pgmap v10323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:16 smithi097 ceph-mon[32231]: pgmap v10323: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:18 smithi115 ceph-mon[35364]: pgmap v10324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:18 smithi181 ceph-mon[35869]: pgmap v10324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:18 smithi097 ceph-mon[32231]: pgmap v10324: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:20 smithi115 ceph-mon[35364]: pgmap v10325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:20 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18418.115660 secs 2024-07-26T13:06:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:20 smithi181 ceph-mon[35869]: pgmap v10325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:20 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18418.115660 secs 2024-07-26T13:06:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:20 smithi097 ceph-mon[32231]: pgmap v10325: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:20 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18418.115660 secs 2024-07-26T13:06:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:21 smithi115 ceph-mon[35364]: pgmap v10326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:06:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:21 smithi115 ceph-mon[35364]: pgmap v10327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:22.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:21 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:06:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:21 smithi181 ceph-mon[35869]: pgmap v10326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:06:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:21 smithi181 ceph-mon[35869]: pgmap v10327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:22.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:21 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:06:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:21 smithi097 ceph-mon[32231]: pgmap v10326: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:06:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:21 smithi097 ceph-mon[32231]: pgmap v10327: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:22.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:21 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:06:23.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:23 smithi115 ceph-mon[35364]: pgmap v10328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:23.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:23 smithi181 ceph-mon[35869]: pgmap v10328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:24.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:23 smithi097 ceph-mon[32231]: pgmap v10328: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:25.113+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:25.115+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:06:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:25.115+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:06:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:25.115+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:25.115+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:26 smithi115 ceph-mon[35364]: pgmap v10329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18423.115756 secs 2024-07-26T13:06:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:26 smithi181 ceph-mon[35869]: pgmap v10329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18423.115756 secs 2024-07-26T13:06:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:26 smithi097 ceph-mon[32231]: pgmap v10329: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18423.115756 secs 2024-07-26T13:06:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:28 smithi115 ceph-mon[35364]: pgmap v10330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:28 smithi181 ceph-mon[35869]: pgmap v10330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:28 smithi097 ceph-mon[32231]: pgmap v10330: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:30 smithi115 ceph-mon[35364]: pgmap v10331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:30 smithi181 ceph-mon[35869]: pgmap v10331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:30 smithi097 ceph-mon[32231]: pgmap v10331: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18428.115890 secs 2024-07-26T13:06:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18428.115890 secs 2024-07-26T13:06:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18428.115890 secs 2024-07-26T13:06:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:32 smithi115 ceph-mon[35364]: pgmap v10332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:32 smithi181 ceph-mon[35869]: pgmap v10332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:32 smithi097 ceph-mon[32231]: pgmap v10332: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:34 smithi115 ceph-mon[35364]: pgmap v10333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:34 smithi181 ceph-mon[35869]: pgmap v10333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:34 smithi097 ceph-mon[32231]: pgmap v10333: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18433.116050 secs 2024-07-26T13:06:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18433.116050 secs 2024-07-26T13:06:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18433.116050 secs 2024-07-26T13:06:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:36 smithi115 ceph-mon[35364]: pgmap v10334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:36 smithi181 ceph-mon[35869]: pgmap v10334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:36 smithi097 ceph-mon[32231]: pgmap v10334: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:37 smithi115 ceph-mon[35364]: pgmap v10335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:37 smithi181 ceph-mon[35869]: pgmap v10335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:37 smithi097 ceph-mon[32231]: pgmap v10335: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:40.123 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:40.122+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:40.125 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:40.123+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:06:40.125 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:40.123+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:06:40.125 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:40.123+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:40.125 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:40.124+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:40 smithi115 ceph-mon[35364]: pgmap v10336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:40 smithi181 ceph-mon[35869]: pgmap v10336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:40 smithi097 ceph-mon[32231]: pgmap v10336: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:41.383 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18438.116169 secs 2024-07-26T13:06:41.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18438.116169 secs 2024-07-26T13:06:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18438.116169 secs 2024-07-26T13:06:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:42 smithi115 ceph-mon[35364]: pgmap v10337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:42 smithi181 ceph-mon[35869]: pgmap v10337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:42 smithi097 ceph-mon[32231]: pgmap v10337: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:43 smithi115 ceph-mon[35364]: pgmap v10338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:43 smithi181 ceph-mon[35869]: pgmap v10338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:43 smithi097 ceph-mon[32231]: pgmap v10338: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:46 smithi115 ceph-mon[35364]: pgmap v10339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18443.116307 secs 2024-07-26T13:06:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:46 smithi181 ceph-mon[35869]: pgmap v10339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18443.116307 secs 2024-07-26T13:06:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:46 smithi097 ceph-mon[32231]: pgmap v10339: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18443.116307 secs 2024-07-26T13:06:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:48 smithi115 ceph-mon[35364]: pgmap v10340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:48 smithi181 ceph-mon[35869]: pgmap v10340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:48 smithi097 ceph-mon[32231]: pgmap v10340: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:50.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:50 smithi115 ceph-mon[35364]: pgmap v10341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:50 smithi181 ceph-mon[35869]: pgmap v10341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:50 smithi097 ceph-mon[32231]: pgmap v10341: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18448.116453 secs 2024-07-26T13:06:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18448.116453 secs 2024-07-26T13:06:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18448.116453 secs 2024-07-26T13:06:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:52 smithi115 ceph-mon[35364]: pgmap v10342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:52 smithi181 ceph-mon[35869]: pgmap v10342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:52 smithi097 ceph-mon[32231]: pgmap v10342: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:06:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:54 smithi115 ceph-mon[35364]: pgmap v10343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:54.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:54 smithi181 ceph-mon[35869]: pgmap v10343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:54 smithi097 ceph-mon[32231]: pgmap v10343: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:55.127+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:55.128+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:06:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:55.128+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:06:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:55.128+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:06:55.129+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:55 smithi115 ceph-mon[35364]: pgmap v10344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18453.116593 secs 2024-07-26T13:06:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:55 smithi181 ceph-mon[35869]: pgmap v10344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18453.116593 secs 2024-07-26T13:06:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:55 smithi097 ceph-mon[32231]: pgmap v10344: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18453.116593 secs 2024-07-26T13:06:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:06:58 smithi115 ceph-mon[35364]: pgmap v10345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:06:58 smithi181 ceph-mon[35869]: pgmap v10345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:58.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:06:58 smithi097 ceph-mon[32231]: pgmap v10345: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:00 smithi115 ceph-mon[35364]: pgmap v10346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:00 smithi181 ceph-mon[35869]: pgmap v10346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:00 smithi097 ceph-mon[32231]: pgmap v10346: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18458.116759 secs 2024-07-26T13:07:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18458.116759 secs 2024-07-26T13:07:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18458.116759 secs 2024-07-26T13:07:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:02 smithi115 ceph-mon[35364]: pgmap v10347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:02 smithi181 ceph-mon[35869]: pgmap v10347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:02 smithi097 ceph-mon[32231]: pgmap v10347: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:04 smithi115 ceph-mon[35364]: pgmap v10348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:04 smithi181 ceph-mon[35869]: pgmap v10348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:04 smithi097 ceph-mon[32231]: pgmap v10348: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18463.116920 secs 2024-07-26T13:07:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18463.116920 secs 2024-07-26T13:07:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18463.116920 secs 2024-07-26T13:07:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:06 smithi115 ceph-mon[35364]: pgmap v10349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:06 smithi181 ceph-mon[35869]: pgmap v10349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:06 smithi097 ceph-mon[32231]: pgmap v10349: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:07 smithi115 ceph-mon[35364]: pgmap v10350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:07 smithi181 ceph-mon[35869]: pgmap v10350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:07 smithi097 ceph-mon[32231]: pgmap v10350: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.137+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:07:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.138+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:07:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.138+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:07:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.138+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:07:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.138+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.182+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:07:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.182+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:07:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.183+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:07:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:10.183+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:07:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:10 smithi115 ceph-mon[35364]: pgmap v10351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:10 smithi181 ceph-mon[35869]: pgmap v10351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:10 smithi097 ceph-mon[32231]: pgmap v10351: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:11.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18468.117076 secs 2024-07-26T13:07:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18468.117076 secs 2024-07-26T13:07:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18468.117076 secs 2024-07-26T13:07:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:12 smithi115 ceph-mon[35364]: pgmap v10352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:12 smithi181 ceph-mon[35869]: pgmap v10352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:12 smithi097 ceph-mon[32231]: pgmap v10352: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:14 smithi115 ceph-mon[35364]: pgmap v10353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:14 smithi181 ceph-mon[35869]: pgmap v10353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:14 smithi097 ceph-mon[32231]: pgmap v10353: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18473.117220 secs 2024-07-26T13:07:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18473.117220 secs 2024-07-26T13:07:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18473.117220 secs 2024-07-26T13:07:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:16 smithi115 ceph-mon[35364]: pgmap v10354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:16 smithi181 ceph-mon[35869]: pgmap v10354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:16 smithi097 ceph-mon[32231]: pgmap v10354: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:17 smithi115 ceph-mon[35364]: pgmap v10355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:17 smithi181 ceph-mon[35869]: pgmap v10355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:17 smithi097 ceph-mon[32231]: pgmap v10355: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:20 smithi115 ceph-mon[35364]: pgmap v10356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:20 smithi181 ceph-mon[35869]: pgmap v10356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:20 smithi097 ceph-mon[32231]: pgmap v10356: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18478.117359 secs 2024-07-26T13:07:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18478.117359 secs 2024-07-26T13:07:21.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18478.117359 secs 2024-07-26T13:07:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:22 smithi115 ceph-mon[35364]: pgmap v10357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:22 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:22 smithi181 ceph-mon[35869]: pgmap v10357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:22 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:22 smithi097 ceph-mon[32231]: pgmap v10357: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:22 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:23.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:23.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:23.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:07:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:23.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:07:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:24 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T13:07:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:24 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T13:07:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:24 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T13:07:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:24 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T13:07:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:24 smithi115 ceph-mon[35364]: pgmap v10358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:07:24.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:24 smithi115 ceph-mon[35364]: pgmap v10359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:24.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:24 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T13:07:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:24 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T13:07:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:24 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T13:07:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:24 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T13:07:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:24 smithi181 ceph-mon[35869]: pgmap v10358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:07:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:24 smithi181 ceph-mon[35869]: pgmap v10359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:24 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T13:07:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:24 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T13:07:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:24 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T13:07:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:24 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T13:07:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:24 smithi097 ceph-mon[32231]: pgmap v10358: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:07:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:24 smithi097 ceph-mon[32231]: pgmap v10359: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:25.141+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:07:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:25.142+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:07:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:25.142+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:07:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:25.142+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:07:25.144 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:25.142+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:25 smithi115 ceph-mon[35364]: pgmap v10360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18483.117457 secs 2024-07-26T13:07:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:25 smithi181 ceph-mon[35869]: pgmap v10360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18483.117457 secs 2024-07-26T13:07:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:25 smithi097 ceph-mon[32231]: pgmap v10360: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18483.117457 secs 2024-07-26T13:07:27.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:27 smithi115 ceph-mon[35364]: pgmap v10361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:27 smithi181 ceph-mon[35869]: pgmap v10361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:28.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:27 smithi097 ceph-mon[32231]: pgmap v10361: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:29 smithi115 ceph-mon[35364]: pgmap v10362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:29 smithi181 ceph-mon[35869]: pgmap v10362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:29 smithi097 ceph-mon[32231]: pgmap v10362: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18488.117543 secs 2024-07-26T13:07:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18488.117543 secs 2024-07-26T13:07:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18488.117543 secs 2024-07-26T13:07:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:31 smithi115 ceph-mon[35364]: pgmap v10363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:07:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:31 smithi181 ceph-mon[35869]: pgmap v10363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:07:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:31 smithi097 ceph-mon[32231]: pgmap v10363: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:07:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:33 smithi115 ceph-mon[35364]: pgmap v10364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:33 smithi181 ceph-mon[35869]: pgmap v10364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:33 smithi097 ceph-mon[32231]: pgmap v10364: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:35 smithi115 ceph-mon[35364]: pgmap v10365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18493.117638 secs 2024-07-26T13:07:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:35 smithi181 ceph-mon[35869]: pgmap v10365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18493.117638 secs 2024-07-26T13:07:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:35 smithi097 ceph-mon[32231]: pgmap v10365: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18493.117638 secs 2024-07-26T13:07:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:37 smithi115 ceph-mon[35364]: pgmap v10366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:37 smithi181 ceph-mon[35869]: pgmap v10366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:37 smithi097 ceph-mon[32231]: pgmap v10366: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:39 smithi115 ceph-mon[35364]: pgmap v10367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:39 smithi181 ceph-mon[35869]: pgmap v10367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:39 smithi097 ceph-mon[32231]: pgmap v10367: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:40.150+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:07:40.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:40.151+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:07:40.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:40.151+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:07:40.152 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:40.151+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:07:40.153 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:40.151+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18498.117781 secs 2024-07-26T13:07:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18498.117781 secs 2024-07-26T13:07:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18498.117781 secs 2024-07-26T13:07:41.876 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:41 smithi115 ceph-mon[35364]: pgmap v10368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:41.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:41 smithi181 ceph-mon[35869]: pgmap v10368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:41.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:41 smithi097 ceph-mon[32231]: pgmap v10368: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:43 smithi115 ceph-mon[35364]: pgmap v10369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:43 smithi181 ceph-mon[35869]: pgmap v10369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:43 smithi097 ceph-mon[32231]: pgmap v10369: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:45 smithi115 ceph-mon[35364]: pgmap v10370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18503.117926 secs 2024-07-26T13:07:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:45 smithi181 ceph-mon[35869]: pgmap v10370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18503.117926 secs 2024-07-26T13:07:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:45 smithi097 ceph-mon[32231]: pgmap v10370: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18503.117926 secs 2024-07-26T13:07:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:47 smithi115 ceph-mon[35364]: pgmap v10371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:47 smithi181 ceph-mon[35869]: pgmap v10371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:47 smithi097 ceph-mon[32231]: pgmap v10371: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:49 smithi115 ceph-mon[35364]: pgmap v10372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:49 smithi181 ceph-mon[35869]: pgmap v10372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:50.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:49 smithi097 ceph-mon[32231]: pgmap v10372: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18508.118089 secs 2024-07-26T13:07:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18508.118089 secs 2024-07-26T13:07:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18508.118089 secs 2024-07-26T13:07:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:51 smithi115 ceph-mon[35364]: pgmap v10373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:51 smithi181 ceph-mon[35869]: pgmap v10373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:51 smithi097 ceph-mon[32231]: pgmap v10373: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:53 smithi115 ceph-mon[35364]: pgmap v10374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:53 smithi181 ceph-mon[35869]: pgmap v10374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:53 smithi097 ceph-mon[32231]: pgmap v10374: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:55.154+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:07:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:55.155+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:07:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:55.155+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:07:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:55.156+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:07:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:07:55.156+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:55 smithi115 ceph-mon[35364]: pgmap v10375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18513.118287 secs 2024-07-26T13:07:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:55 smithi181 ceph-mon[35869]: pgmap v10375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:55.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18513.118287 secs 2024-07-26T13:07:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:55 smithi097 ceph-mon[32231]: pgmap v10375: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18513.118287 secs 2024-07-26T13:07:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:57 smithi115 ceph-mon[35364]: pgmap v10376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:57 smithi181 ceph-mon[35869]: pgmap v10376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:57 smithi097 ceph-mon[32231]: pgmap v10376: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:07:59 smithi115 ceph-mon[35364]: pgmap v10377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:07:59 smithi181 ceph-mon[35869]: pgmap v10377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:07:59 smithi097 ceph-mon[32231]: pgmap v10377: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18518.118426 secs 2024-07-26T13:08:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18518.118426 secs 2024-07-26T13:08:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18518.118426 secs 2024-07-26T13:08:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:01 smithi115 ceph-mon[35364]: pgmap v10378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:01 smithi181 ceph-mon[35869]: pgmap v10378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:01 smithi097 ceph-mon[32231]: pgmap v10378: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:03 smithi115 ceph-mon[35364]: pgmap v10379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:03 smithi181 ceph-mon[35869]: pgmap v10379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:03 smithi097 ceph-mon[32231]: pgmap v10379: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:05 smithi115 ceph-mon[35364]: pgmap v10380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18523.118563 secs 2024-07-26T13:08:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:05 smithi181 ceph-mon[35869]: pgmap v10380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18523.118563 secs 2024-07-26T13:08:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:05 smithi097 ceph-mon[32231]: pgmap v10380: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18523.118563 secs 2024-07-26T13:08:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:07 smithi115 ceph-mon[35364]: pgmap v10381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:07 smithi181 ceph-mon[35869]: pgmap v10381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:07 smithi097 ceph-mon[32231]: pgmap v10381: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:09 smithi115 ceph-mon[35364]: pgmap v10382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:09 smithi181 ceph-mon[35869]: pgmap v10382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:09 smithi097 ceph-mon[32231]: pgmap v10382: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:10.163+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:08:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:10.164+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:08:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:10.164+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:08:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:10.164+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:08:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:10.164+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18528.118701 secs 2024-07-26T13:08:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18528.118701 secs 2024-07-26T13:08:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18528.118701 secs 2024-07-26T13:08:11.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:11 smithi181 ceph-mon[35869]: pgmap v10383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:11 smithi097 ceph-mon[32231]: pgmap v10383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:12.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:11 smithi115 ceph-mon[35364]: pgmap v10383: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:13 smithi097 ceph-mon[32231]: pgmap v10384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:14.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:13 smithi115 ceph-mon[35364]: pgmap v10384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:13 smithi181 ceph-mon[35869]: pgmap v10384: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:15 smithi097 ceph-mon[32231]: pgmap v10385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18533.118832 secs 2024-07-26T13:08:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:15 smithi115 ceph-mon[35364]: pgmap v10385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18533.118832 secs 2024-07-26T13:08:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:15 smithi181 ceph-mon[35869]: pgmap v10385: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18533.118832 secs 2024-07-26T13:08:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:17 smithi097 ceph-mon[32231]: pgmap v10386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:17 smithi115 ceph-mon[35364]: pgmap v10386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:17 smithi181 ceph-mon[35869]: pgmap v10386: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:19 smithi115 ceph-mon[35364]: pgmap v10387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:19 smithi181 ceph-mon[35869]: pgmap v10387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:20.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:19 smithi097 ceph-mon[32231]: pgmap v10387: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18538.118969 secs 2024-07-26T13:08:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18538.118969 secs 2024-07-26T13:08:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18538.118969 secs 2024-07-26T13:08:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:21 smithi115 ceph-mon[35364]: pgmap v10388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:22.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:21 smithi181 ceph-mon[35869]: pgmap v10388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:21 smithi097 ceph-mon[32231]: pgmap v10388: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:23 smithi115 ceph-mon[35364]: pgmap v10389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:23 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:23 smithi181 ceph-mon[35869]: pgmap v10389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:23 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:23 smithi097 ceph-mon[32231]: pgmap v10389: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:23 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:25.168 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:25.167+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:08:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:25.168+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:08:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:25.168+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:08:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:25.168+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:08:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:25.168+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:25.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:24 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:25.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:24 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:08:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T13:08:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T13:08:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:25.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:24 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:08:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:25 smithi115 ceph-mon[35364]: pgmap v10390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:08:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:25 smithi115 ceph-mon[35364]: pgmap v10391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:26.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18543.119128 secs 2024-07-26T13:08:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:25 smithi181 ceph-mon[35869]: pgmap v10390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:08:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:25 smithi181 ceph-mon[35869]: pgmap v10391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18543.119128 secs 2024-07-26T13:08:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:25 smithi097 ceph-mon[32231]: pgmap v10390: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:08:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:25 smithi097 ceph-mon[32231]: pgmap v10391: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18543.119128 secs 2024-07-26T13:08:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:27 smithi115 ceph-mon[35364]: pgmap v10392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:27 smithi181 ceph-mon[35869]: pgmap v10392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:27 smithi097 ceph-mon[32231]: pgmap v10392: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:29 smithi115 ceph-mon[35364]: pgmap v10393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:29 smithi181 ceph-mon[35869]: pgmap v10393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:29 smithi097 ceph-mon[32231]: pgmap v10393: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18548.119280 secs 2024-07-26T13:08:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18548.119280 secs 2024-07-26T13:08:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18548.119280 secs 2024-07-26T13:08:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:31 smithi115 ceph-mon[35364]: pgmap v10394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:31 smithi181 ceph-mon[35869]: pgmap v10394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:31 smithi097 ceph-mon[32231]: pgmap v10394: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:33 smithi115 ceph-mon[35364]: pgmap v10395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:33 smithi181 ceph-mon[35869]: pgmap v10395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:33 smithi097 ceph-mon[32231]: pgmap v10395: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:35 smithi115 ceph-mon[35364]: pgmap v10396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18553.119417 secs 2024-07-26T13:08:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:35 smithi181 ceph-mon[35869]: pgmap v10396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18553.119417 secs 2024-07-26T13:08:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:35 smithi097 ceph-mon[32231]: pgmap v10396: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18553.119417 secs 2024-07-26T13:08:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:37 smithi115 ceph-mon[35364]: pgmap v10397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:37 smithi181 ceph-mon[35869]: pgmap v10397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:37 smithi097 ceph-mon[32231]: pgmap v10397: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:40.173 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:40.171+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:08:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:40.172+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:08:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:40.172+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:08:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:40.172+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:08:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:40.173+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:39 smithi115 ceph-mon[35364]: pgmap v10398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:39 smithi181 ceph-mon[35869]: pgmap v10398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:39 smithi097 ceph-mon[32231]: pgmap v10398: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:41.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18558.119551 secs 2024-07-26T13:08:41.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18558.119551 secs 2024-07-26T13:08:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18558.119551 secs 2024-07-26T13:08:42.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:41 smithi115 ceph-mon[35364]: pgmap v10399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:41 smithi181 ceph-mon[35869]: pgmap v10399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:41 smithi097 ceph-mon[32231]: pgmap v10399: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:43 smithi115 ceph-mon[35364]: pgmap v10400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:43 smithi181 ceph-mon[35869]: pgmap v10400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:43 smithi097 ceph-mon[32231]: pgmap v10400: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:45 smithi115 ceph-mon[35364]: pgmap v10401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18563.119698 secs 2024-07-26T13:08:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:45 smithi181 ceph-mon[35869]: pgmap v10401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18563.119698 secs 2024-07-26T13:08:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:45 smithi097 ceph-mon[32231]: pgmap v10401: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18563.119698 secs 2024-07-26T13:08:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:47 smithi115 ceph-mon[35364]: pgmap v10402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:47 smithi181 ceph-mon[35869]: pgmap v10402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:47 smithi097 ceph-mon[32231]: pgmap v10402: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:49 smithi115 ceph-mon[35364]: pgmap v10403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:49 smithi181 ceph-mon[35869]: pgmap v10403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:49 smithi097 ceph-mon[32231]: pgmap v10403: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18568.119793 secs 2024-07-26T13:08:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18568.119793 secs 2024-07-26T13:08:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18568.119793 secs 2024-07-26T13:08:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:51 smithi115 ceph-mon[35364]: pgmap v10404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:51 smithi181 ceph-mon[35869]: pgmap v10404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:51 smithi097 ceph-mon[32231]: pgmap v10404: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:53 smithi115 ceph-mon[35364]: pgmap v10405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:53 smithi181 ceph-mon[35869]: pgmap v10405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:53 smithi097 ceph-mon[32231]: pgmap v10405: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:55.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:55.175+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:08:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:55.176+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:08:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:55.176+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:08:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:55.176+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:08:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:08:55.177+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:55 smithi115 ceph-mon[35364]: pgmap v10406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18573.119928 secs 2024-07-26T13:08:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:55 smithi181 ceph-mon[35869]: pgmap v10406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18573.119928 secs 2024-07-26T13:08:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:55 smithi097 ceph-mon[32231]: pgmap v10406: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18573.119928 secs 2024-07-26T13:08:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:57 smithi115 ceph-mon[35364]: pgmap v10407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:57 smithi181 ceph-mon[35869]: pgmap v10407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:57 smithi097 ceph-mon[32231]: pgmap v10407: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:08:59 smithi115 ceph-mon[35364]: pgmap v10408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:08:59 smithi181 ceph-mon[35869]: pgmap v10408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:08:59 smithi097 ceph-mon[32231]: pgmap v10408: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18578.120070 secs 2024-07-26T13:09:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18578.120070 secs 2024-07-26T13:09:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18578.120070 secs 2024-07-26T13:09:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:01 smithi115 ceph-mon[35364]: pgmap v10409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:01 smithi181 ceph-mon[35869]: pgmap v10409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:01 smithi097 ceph-mon[32231]: pgmap v10409: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:03 smithi115 ceph-mon[35364]: pgmap v10410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:03 smithi181 ceph-mon[35869]: pgmap v10410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:03 smithi097 ceph-mon[32231]: pgmap v10410: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:05 smithi115 ceph-mon[35364]: pgmap v10411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18583.120210 secs 2024-07-26T13:09:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:05 smithi181 ceph-mon[35869]: pgmap v10411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:06.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18583.120210 secs 2024-07-26T13:09:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:05 smithi097 ceph-mon[32231]: pgmap v10411: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18583.120210 secs 2024-07-26T13:09:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:07 smithi115 ceph-mon[35364]: pgmap v10412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:07 smithi181 ceph-mon[35869]: pgmap v10412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:07 smithi097 ceph-mon[32231]: pgmap v10412: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:10.185+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:09:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:10.186+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:09:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:10.186+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:09:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:10.186+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:09:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:10.186+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:09 smithi181 ceph-mon[35869]: pgmap v10413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:09 smithi097 ceph-mon[32231]: pgmap v10413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:09 smithi115 ceph-mon[35364]: pgmap v10413: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18588.120353 secs 2024-07-26T13:09:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18588.120353 secs 2024-07-26T13:09:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18588.120353 secs 2024-07-26T13:09:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:11 smithi097 ceph-mon[32231]: pgmap v10414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:12.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:11 smithi115 ceph-mon[35364]: pgmap v10414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:11 smithi181 ceph-mon[35869]: pgmap v10414: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:14.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:14 smithi115 ceph-mon[35364]: pgmap v10415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:14 smithi181 ceph-mon[35869]: pgmap v10415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:14 smithi097 ceph-mon[32231]: pgmap v10415: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:16 smithi115 ceph-mon[35364]: pgmap v10416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18593.120458 secs 2024-07-26T13:09:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:16 smithi181 ceph-mon[35869]: pgmap v10416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18593.120458 secs 2024-07-26T13:09:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:16 smithi097 ceph-mon[32231]: pgmap v10416: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18593.120458 secs 2024-07-26T13:09:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:18 smithi115 ceph-mon[35364]: pgmap v10417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:18 smithi181 ceph-mon[35869]: pgmap v10417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:18 smithi097 ceph-mon[32231]: pgmap v10417: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:20 smithi115 ceph-mon[35364]: pgmap v10418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:20 smithi181 ceph-mon[35869]: pgmap v10418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:20 smithi097 ceph-mon[32231]: pgmap v10418: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18598.120592 secs 2024-07-26T13:09:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18598.120592 secs 2024-07-26T13:09:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18598.120592 secs 2024-07-26T13:09:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:22 smithi115 ceph-mon[35364]: pgmap v10419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:22 smithi181 ceph-mon[35869]: pgmap v10419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:22 smithi097 ceph-mon[32231]: pgmap v10419: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:24 smithi115 ceph-mon[35364]: pgmap v10420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:24 smithi181 ceph-mon[35869]: pgmap v10420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:24 smithi097 ceph-mon[32231]: pgmap v10420: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:25.189+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:09:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:25.190+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:09:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:25.190+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:09:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:25.190+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:09:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:25.190+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:25 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:25 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:25 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:26 smithi115 ceph-mon[35364]: pgmap v10421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18603.120699 secs 2024-07-26T13:09:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:26 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:26 smithi181 ceph-mon[35869]: pgmap v10421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18603.120699 secs 2024-07-26T13:09:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:26 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:26 smithi097 ceph-mon[32231]: pgmap v10421: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18603.120699 secs 2024-07-26T13:09:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:26 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:27 smithi115 ceph-mon[35364]: pgmap v10422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:27.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:27.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:27 smithi181 ceph-mon[35869]: pgmap v10422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:27.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:27 smithi097 ceph-mon[32231]: pgmap v10422: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:27.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:09:29.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:29 smithi115 ceph-mon[35364]: pgmap v10423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:29 smithi181 ceph-mon[35869]: pgmap v10423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:30.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:29 smithi097 ceph-mon[32231]: pgmap v10423: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18608.120795 secs 2024-07-26T13:09:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18608.120795 secs 2024-07-26T13:09:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18608.120795 secs 2024-07-26T13:09:31.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:31 smithi115 ceph-mon[35364]: pgmap v10424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:31 smithi181 ceph-mon[35869]: pgmap v10424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:32.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:31 smithi097 ceph-mon[32231]: pgmap v10424: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:33.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:33 smithi115 ceph-mon[35364]: pgmap v10425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:33 smithi181 ceph-mon[35869]: pgmap v10425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:34.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:33 smithi097 ceph-mon[32231]: pgmap v10425: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:35 smithi115 ceph-mon[35364]: pgmap v10426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18613.120874 secs 2024-07-26T13:09:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:35 smithi181 ceph-mon[35869]: pgmap v10426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18613.120874 secs 2024-07-26T13:09:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:35 smithi097 ceph-mon[32231]: pgmap v10426: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18613.120874 secs 2024-07-26T13:09:37.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:37 smithi115 ceph-mon[35364]: pgmap v10427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:37 smithi181 ceph-mon[35869]: pgmap v10427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:38.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:37 smithi097 ceph-mon[32231]: pgmap v10427: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:39.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:39 smithi115 ceph-mon[35364]: pgmap v10428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:39 smithi181 ceph-mon[35869]: pgmap v10428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:40.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:39 smithi097 ceph-mon[32231]: pgmap v10428: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:40.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:40.195+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:09:40.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:40.196+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:09:40.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:40.196+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:09:40.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:40.196+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:09:40.198 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:40.197+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18618.121041 secs 2024-07-26T13:09:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18618.121041 secs 2024-07-26T13:09:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18618.121041 secs 2024-07-26T13:09:41.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:41 smithi181 ceph-mon[35869]: pgmap v10429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:41.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:41 smithi097 ceph-mon[32231]: pgmap v10429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:41.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:41 smithi115 ceph-mon[35364]: pgmap v10429: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:43.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:43 smithi115 ceph-mon[35364]: pgmap v10430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:43 smithi181 ceph-mon[35869]: pgmap v10430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:44.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:43 smithi097 ceph-mon[32231]: pgmap v10430: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:45 smithi115 ceph-mon[35364]: pgmap v10431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18623.121174 secs 2024-07-26T13:09:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:45 smithi181 ceph-mon[35869]: pgmap v10431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18623.121174 secs 2024-07-26T13:09:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:45 smithi097 ceph-mon[32231]: pgmap v10431: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18623.121174 secs 2024-07-26T13:09:47.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:47 smithi115 ceph-mon[35364]: pgmap v10432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:47 smithi181 ceph-mon[35869]: pgmap v10432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:48.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:47 smithi097 ceph-mon[32231]: pgmap v10432: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:49.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:49 smithi115 ceph-mon[35364]: pgmap v10433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:50.006 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:49 smithi097 ceph-mon[32231]: pgmap v10433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:50.006 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:49 smithi181 ceph-mon[35869]: pgmap v10433: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:09:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18628.121309 secs 2024-07-26T13:09:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18628.121309 secs 2024-07-26T13:09:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18628.121309 secs 2024-07-26T13:09:51.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:51 smithi115 ceph-mon[35364]: pgmap v10434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:51.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:51 smithi181 ceph-mon[35869]: pgmap v10434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:52.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:51 smithi097 ceph-mon[32231]: pgmap v10434: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:09:53.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:53 smithi115 ceph-mon[35364]: pgmap v10435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:53 smithi181 ceph-mon[35869]: pgmap v10435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:54.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:53 smithi097 ceph-mon[32231]: pgmap v10435: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:55.199+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:09:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:55.200+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:09:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:55.200+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:09:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:55.200+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:09:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:09:55.200+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:55 smithi115 ceph-mon[35364]: pgmap v10436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18633.121393 secs 2024-07-26T13:09:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:55 smithi181 ceph-mon[35869]: pgmap v10436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18633.121393 secs 2024-07-26T13:09:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:55 smithi097 ceph-mon[32231]: pgmap v10436: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18633.121393 secs 2024-07-26T13:09:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:57 smithi115 ceph-mon[35364]: pgmap v10437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:57 smithi181 ceph-mon[35869]: pgmap v10437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:57 smithi097 ceph-mon[32231]: pgmap v10437: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:09:59 smithi115 ceph-mon[35364]: pgmap v10438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:09:59 smithi181 ceph-mon[35869]: pgmap v10438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:09:59 smithi097 ceph-mon[32231]: pgmap v10438: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18638.121475 secs 2024-07-26T13:10:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18638.121475 secs 2024-07-26T13:10:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18638.121475 secs 2024-07-26T13:10:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:01 smithi115 ceph-mon[35364]: pgmap v10439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:01 smithi181 ceph-mon[35869]: pgmap v10439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:01 smithi097 ceph-mon[32231]: pgmap v10439: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:03 smithi115 ceph-mon[35364]: pgmap v10440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:03 smithi181 ceph-mon[35869]: pgmap v10440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:03 smithi097 ceph-mon[32231]: pgmap v10440: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:05 smithi115 ceph-mon[35364]: pgmap v10441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18643.121609 secs 2024-07-26T13:10:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:05 smithi181 ceph-mon[35869]: pgmap v10441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18643.121609 secs 2024-07-26T13:10:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:05 smithi097 ceph-mon[32231]: pgmap v10441: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18643.121609 secs 2024-07-26T13:10:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:07 smithi115 ceph-mon[35364]: pgmap v10442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:07.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:07 smithi181 ceph-mon[35869]: pgmap v10442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:07 smithi097 ceph-mon[32231]: pgmap v10442: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:09 smithi115 ceph-mon[35364]: pgmap v10443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:09 smithi181 ceph-mon[35869]: pgmap v10443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:09 smithi097 ceph-mon[32231]: pgmap v10443: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.208+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:10:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:10:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.209+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:10:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.209+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:10:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.210+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.213+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:10:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.213+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:10:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.214+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:10:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:10.214+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:10:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18648.121769 secs 2024-07-26T13:10:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18648.121769 secs 2024-07-26T13:10:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18648.121769 secs 2024-07-26T13:10:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:11 smithi115 ceph-mon[35364]: pgmap v10444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:11 smithi181 ceph-mon[35869]: pgmap v10444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:11 smithi097 ceph-mon[32231]: pgmap v10444: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:13 smithi115 ceph-mon[35364]: pgmap v10445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:13 smithi181 ceph-mon[35869]: pgmap v10445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:13 smithi097 ceph-mon[32231]: pgmap v10445: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:15 smithi181 ceph-mon[35869]: pgmap v10446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18653.121908 secs 2024-07-26T13:10:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:15 smithi097 ceph-mon[32231]: pgmap v10446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18653.121908 secs 2024-07-26T13:10:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:15 smithi115 ceph-mon[35364]: pgmap v10446: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18653.121908 secs 2024-07-26T13:10:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:17 smithi097 ceph-mon[32231]: pgmap v10447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:18.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:17 smithi115 ceph-mon[35364]: pgmap v10447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:17 smithi181 ceph-mon[35869]: pgmap v10447: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:19 smithi097 ceph-mon[32231]: pgmap v10448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:20.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:19 smithi115 ceph-mon[35364]: pgmap v10448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:19 smithi181 ceph-mon[35869]: pgmap v10448: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18658.122054 secs 2024-07-26T13:10:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18658.122054 secs 2024-07-26T13:10:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18658.122054 secs 2024-07-26T13:10:22.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:21 smithi115 ceph-mon[35364]: pgmap v10449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:21 smithi181 ceph-mon[35869]: pgmap v10449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:22.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:21 smithi097 ceph-mon[32231]: pgmap v10449: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:24.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:23 smithi115 ceph-mon[35364]: pgmap v10450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:23 smithi181 ceph-mon[35869]: pgmap v10450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:23 smithi097 ceph-mon[32231]: pgmap v10450: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:25.213+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:10:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:25.215+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:10:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:25.215+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:10:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:25.215+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:10:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:25.215+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:25 smithi115 ceph-mon[35364]: pgmap v10451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18663.122191 secs 2024-07-26T13:10:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:25 smithi181 ceph-mon[35869]: pgmap v10451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18663.122191 secs 2024-07-26T13:10:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:25 smithi097 ceph-mon[32231]: pgmap v10451: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18663.122191 secs 2024-07-26T13:10:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:27 smithi115 ceph-mon[35364]: pgmap v10452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:27 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:27 smithi181 ceph-mon[35869]: pgmap v10452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:27 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:27 smithi097 ceph-mon[32231]: pgmap v10452: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:27 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:28 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:28 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:28 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:10:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:29 smithi115 ceph-mon[35364]: pgmap v10453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:29 smithi115 ceph-mon[35364]: pgmap v10454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:29 smithi181 ceph-mon[35869]: pgmap v10453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:29 smithi181 ceph-mon[35869]: pgmap v10454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:29 smithi097 ceph-mon[32231]: pgmap v10453: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:29 smithi097 ceph-mon[32231]: pgmap v10454: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18668.122341 secs 2024-07-26T13:10:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18668.122341 secs 2024-07-26T13:10:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18668.122341 secs 2024-07-26T13:10:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:31 smithi115 ceph-mon[35364]: pgmap v10455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:31 smithi181 ceph-mon[35869]: pgmap v10455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:31 smithi097 ceph-mon[32231]: pgmap v10455: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:33 smithi115 ceph-mon[35364]: pgmap v10456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T13:10:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:33 smithi181 ceph-mon[35869]: pgmap v10456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T13:10:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:33 smithi097 ceph-mon[32231]: pgmap v10456: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T13:10:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:35 smithi115 ceph-mon[35364]: pgmap v10457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18673.122435 secs 2024-07-26T13:10:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:35 smithi181 ceph-mon[35869]: pgmap v10457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18673.122435 secs 2024-07-26T13:10:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:35 smithi097 ceph-mon[32231]: pgmap v10457: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18673.122435 secs 2024-07-26T13:10:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:37 smithi115 ceph-mon[35364]: pgmap v10458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:37 smithi181 ceph-mon[35869]: pgmap v10458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:37 smithi097 ceph-mon[32231]: pgmap v10458: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:10:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:40.219+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:10:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:40.220+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:10:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:40.220+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:10:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:40.220+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:10:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:40.220+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:39 smithi115 ceph-mon[35364]: pgmap v10459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:39 smithi181 ceph-mon[35869]: pgmap v10459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:39 smithi097 ceph-mon[32231]: pgmap v10459: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:41.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18678.122523 secs 2024-07-26T13:10:41.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18678.122523 secs 2024-07-26T13:10:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18678.122523 secs 2024-07-26T13:10:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:41 smithi115 ceph-mon[35364]: pgmap v10460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:41 smithi181 ceph-mon[35869]: pgmap v10460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:41 smithi097 ceph-mon[32231]: pgmap v10460: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:43 smithi115 ceph-mon[35364]: pgmap v10461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:43 smithi181 ceph-mon[35869]: pgmap v10461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:43 smithi097 ceph-mon[32231]: pgmap v10461: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:45 smithi115 ceph-mon[35364]: pgmap v10462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:10:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18683.122671 secs 2024-07-26T13:10:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:45 smithi181 ceph-mon[35869]: pgmap v10462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:10:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18683.122671 secs 2024-07-26T13:10:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:45 smithi097 ceph-mon[32231]: pgmap v10462: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:10:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18683.122671 secs 2024-07-26T13:10:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:47 smithi115 ceph-mon[35364]: pgmap v10463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:47 smithi181 ceph-mon[35869]: pgmap v10463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:47 smithi097 ceph-mon[32231]: pgmap v10463: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:49 smithi115 ceph-mon[35364]: pgmap v10464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:49 smithi181 ceph-mon[35869]: pgmap v10464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:49 smithi097 ceph-mon[32231]: pgmap v10464: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18688.122818 secs 2024-07-26T13:10:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18688.122818 secs 2024-07-26T13:10:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18688.122818 secs 2024-07-26T13:10:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:51 smithi115 ceph-mon[35364]: pgmap v10465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:51 smithi181 ceph-mon[35869]: pgmap v10465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:51 smithi097 ceph-mon[32231]: pgmap v10465: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:53 smithi115 ceph-mon[35364]: pgmap v10466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:53 smithi181 ceph-mon[35869]: pgmap v10466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:53 smithi097 ceph-mon[32231]: pgmap v10466: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:10:55.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:55.222+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:10:55.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:55.223+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:10:55.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:55.223+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:10:55.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:55.223+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:10:55.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:10:55.223+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:55 smithi115 ceph-mon[35364]: pgmap v10467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:10:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18693.122976 secs 2024-07-26T13:10:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:55 smithi181 ceph-mon[35869]: pgmap v10467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:10:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18693.122976 secs 2024-07-26T13:10:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:55 smithi097 ceph-mon[32231]: pgmap v10467: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:10:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18693.122976 secs 2024-07-26T13:10:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:57 smithi115 ceph-mon[35364]: pgmap v10468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:57 smithi181 ceph-mon[35869]: pgmap v10468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:57 smithi097 ceph-mon[32231]: pgmap v10468: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:10:59 smithi115 ceph-mon[35364]: pgmap v10469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:10:59 smithi181 ceph-mon[35869]: pgmap v10469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:10:59 smithi097 ceph-mon[32231]: pgmap v10469: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18698.123124 secs 2024-07-26T13:11:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18698.123124 secs 2024-07-26T13:11:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18698.123124 secs 2024-07-26T13:11:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:01 smithi115 ceph-mon[35364]: pgmap v10470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:01 smithi181 ceph-mon[35869]: pgmap v10470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:01 smithi097 ceph-mon[32231]: pgmap v10470: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:04.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:03 smithi115 ceph-mon[35364]: pgmap v10471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:03 smithi181 ceph-mon[35869]: pgmap v10471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:03 smithi097 ceph-mon[32231]: pgmap v10471: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:05 smithi115 ceph-mon[35364]: pgmap v10472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18703.123305 secs 2024-07-26T13:11:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:05 smithi181 ceph-mon[35869]: pgmap v10472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18703.123305 secs 2024-07-26T13:11:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:05 smithi097 ceph-mon[32231]: pgmap v10472: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18703.123305 secs 2024-07-26T13:11:08.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:07 smithi115 ceph-mon[35364]: pgmap v10473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:07 smithi181 ceph-mon[35869]: pgmap v10473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:07 smithi097 ceph-mon[32231]: pgmap v10473: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:10.231+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:09 smithi181 ceph-mon[35869]: pgmap v10474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:10.232+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:11:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:10.232+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:11:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:10.232+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:11:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:10.232+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:09 smithi097 ceph-mon[32231]: pgmap v10474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:09 smithi115 ceph-mon[35364]: pgmap v10474: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18708.123447 secs 2024-07-26T13:11:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18708.123447 secs 2024-07-26T13:11:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18708.123447 secs 2024-07-26T13:11:12.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:11 smithi097 ceph-mon[32231]: pgmap v10475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:11 smithi115 ceph-mon[35364]: pgmap v10475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:11 smithi181 ceph-mon[35869]: pgmap v10475: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:14.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:13 smithi097 ceph-mon[32231]: pgmap v10476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:14.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:13 smithi115 ceph-mon[35364]: pgmap v10476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:13 smithi181 ceph-mon[35869]: pgmap v10476: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:16 smithi115 ceph-mon[35364]: pgmap v10477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18713.123557 secs 2024-07-26T13:11:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:16 smithi181 ceph-mon[35869]: pgmap v10477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18713.123557 secs 2024-07-26T13:11:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:16 smithi097 ceph-mon[32231]: pgmap v10477: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18713.123557 secs 2024-07-26T13:11:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:18 smithi115 ceph-mon[35364]: pgmap v10478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:18 smithi181 ceph-mon[35869]: pgmap v10478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:18 smithi097 ceph-mon[32231]: pgmap v10478: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:20 smithi115 ceph-mon[35364]: pgmap v10479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:20 smithi181 ceph-mon[35869]: pgmap v10479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:20 smithi097 ceph-mon[32231]: pgmap v10479: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18718.123700 secs 2024-07-26T13:11:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18718.123700 secs 2024-07-26T13:11:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18718.123700 secs 2024-07-26T13:11:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:22 smithi115 ceph-mon[35364]: pgmap v10480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:22 smithi181 ceph-mon[35869]: pgmap v10480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:22 smithi097 ceph-mon[32231]: pgmap v10480: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:24 smithi115 ceph-mon[35364]: pgmap v10481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:24 smithi181 ceph-mon[35869]: pgmap v10481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:24 smithi097 ceph-mon[32231]: pgmap v10481: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:25.234+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:25.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:25.235+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:11:25.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:25.235+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:11:25.236 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:25.235+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:11:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:25.235+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:26 smithi115 ceph-mon[35364]: pgmap v10482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18723.123866 secs 2024-07-26T13:11:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:26 smithi181 ceph-mon[35869]: pgmap v10482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18723.123866 secs 2024-07-26T13:11:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:26 smithi097 ceph-mon[32231]: pgmap v10482: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18723.123866 secs 2024-07-26T13:11:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:28 smithi115 ceph-mon[35364]: pgmap v10483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:28 smithi181 ceph-mon[35869]: pgmap v10483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:28 smithi097 ceph-mon[32231]: pgmap v10483: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:29 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:29 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:29 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:30 smithi115 ceph-mon[35364]: pgmap v10484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:30 smithi181 ceph-mon[35869]: pgmap v10484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:30 smithi097 ceph-mon[32231]: pgmap v10484: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18728.124009 secs 2024-07-26T13:11:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18728.124009 secs 2024-07-26T13:11:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18728.124009 secs 2024-07-26T13:11:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:32 smithi115 ceph-mon[35364]: pgmap v10485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:11:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:32 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:32 smithi181 ceph-mon[35869]: pgmap v10485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:11:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:32 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:32 smithi097 ceph-mon[32231]: pgmap v10485: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:11:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:32 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:34 smithi115 ceph-mon[35364]: pgmap v10486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:34 smithi181 ceph-mon[35869]: pgmap v10486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:34.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:34 smithi097 ceph-mon[32231]: pgmap v10486: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:11:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:35 smithi115 ceph-mon[35364]: pgmap v10487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:11:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:35 smithi115 ceph-mon[35364]: pgmap v10488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18733.124148 secs 2024-07-26T13:11:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:35 smithi181 ceph-mon[35869]: pgmap v10487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:11:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:35 smithi181 ceph-mon[35869]: pgmap v10488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18733.124148 secs 2024-07-26T13:11:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:35 smithi097 ceph-mon[32231]: pgmap v10487: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:11:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:35 smithi097 ceph-mon[32231]: pgmap v10488: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18733.124148 secs 2024-07-26T13:11:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:37 smithi115 ceph-mon[35364]: pgmap v10489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:11:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:37 smithi181 ceph-mon[35869]: pgmap v10489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:11:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:37 smithi097 ceph-mon[32231]: pgmap v10489: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:11:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:39 smithi115 ceph-mon[35364]: pgmap v10490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:11:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:39 smithi181 ceph-mon[35869]: pgmap v10490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:11:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:39 smithi097 ceph-mon[32231]: pgmap v10490: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:11:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.228+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:11:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.228+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:11:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.231+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:11:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.231+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:11:40.237 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.236+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.237+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:11:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.237+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:11:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.237+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:11:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:40.237+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18738.124294 secs 2024-07-26T13:11:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18738.124294 secs 2024-07-26T13:11:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18738.124294 secs 2024-07-26T13:11:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:41 smithi097 ceph-mon[32231]: pgmap v10491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:41.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:41 smithi181 ceph-mon[35869]: pgmap v10491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:41 smithi115 ceph-mon[35364]: pgmap v10491: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:42.669 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:42 smithi181 ceph-mon[35869]: pgmap v10492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:11:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:42 smithi115 ceph-mon[35364]: pgmap v10492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:11:42.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:42 smithi097 ceph-mon[32231]: pgmap v10492: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:11:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:44 smithi115 ceph-mon[35364]: pgmap v10493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:44 smithi181 ceph-mon[35869]: pgmap v10493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:44 smithi097 ceph-mon[32231]: pgmap v10493: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18743.124433 secs 2024-07-26T13:11:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18743.124433 secs 2024-07-26T13:11:45.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18743.124433 secs 2024-07-26T13:11:46.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:46 smithi115 ceph-mon[35364]: pgmap v10494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:46 smithi181 ceph-mon[35869]: pgmap v10494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:46.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:46 smithi097 ceph-mon[32231]: pgmap v10494: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:48 smithi115 ceph-mon[35364]: pgmap v10495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:48 smithi181 ceph-mon[35869]: pgmap v10495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:48 smithi097 ceph-mon[32231]: pgmap v10495: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:50 smithi115 ceph-mon[35364]: pgmap v10496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18748.124575 secs 2024-07-26T13:11:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:50 smithi181 ceph-mon[35869]: pgmap v10496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18748.124575 secs 2024-07-26T13:11:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:50 smithi097 ceph-mon[32231]: pgmap v10496: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18748.124575 secs 2024-07-26T13:11:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:52 smithi115 ceph-mon[35364]: pgmap v10497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:52 smithi181 ceph-mon[35869]: pgmap v10497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:52.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:52 smithi097 ceph-mon[32231]: pgmap v10497: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:54 smithi115 ceph-mon[35364]: pgmap v10498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:54 smithi181 ceph-mon[35869]: pgmap v10498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:54 smithi097 ceph-mon[32231]: pgmap v10498: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:11:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:55.240+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:55.242+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:11:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:55.242+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:11:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:55.242+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:11:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:11:55.242+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18753.124677 secs 2024-07-26T13:11:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18753.124677 secs 2024-07-26T13:11:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18753.124677 secs 2024-07-26T13:11:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:56 smithi115 ceph-mon[35364]: pgmap v10499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:56 smithi181 ceph-mon[35869]: pgmap v10499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:56 smithi097 ceph-mon[32231]: pgmap v10499: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:11:58 smithi115 ceph-mon[35364]: pgmap v10500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:11:58 smithi181 ceph-mon[35869]: pgmap v10500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:11:58 smithi097 ceph-mon[32231]: pgmap v10500: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:00 smithi115 ceph-mon[35364]: pgmap v10501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18758.124778 secs 2024-07-26T13:12:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:00 smithi181 ceph-mon[35869]: pgmap v10501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18758.124778 secs 2024-07-26T13:12:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:00 smithi097 ceph-mon[32231]: pgmap v10501: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18758.124778 secs 2024-07-26T13:12:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:02 smithi115 ceph-mon[35364]: pgmap v10502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:02 smithi181 ceph-mon[35869]: pgmap v10502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:02 smithi097 ceph-mon[32231]: pgmap v10502: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:04 smithi115 ceph-mon[35364]: pgmap v10503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:04 smithi181 ceph-mon[35869]: pgmap v10503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:04 smithi097 ceph-mon[32231]: pgmap v10503: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18763.124861 secs 2024-07-26T13:12:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18763.124861 secs 2024-07-26T13:12:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18763.124861 secs 2024-07-26T13:12:06.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:06 smithi115 ceph-mon[35364]: pgmap v10504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:06 smithi181 ceph-mon[35869]: pgmap v10504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:06 smithi097 ceph-mon[32231]: pgmap v10504: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:08.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:08 smithi115 ceph-mon[35364]: pgmap v10505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:08 smithi181 ceph-mon[35869]: pgmap v10505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:08 smithi097 ceph-mon[32231]: pgmap v10505: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:10.247+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:10.248+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:12:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:10.248+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:12:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:10.248+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:10.248+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:10 smithi115 ceph-mon[35364]: pgmap v10506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18768.124989 secs 2024-07-26T13:12:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:10 smithi181 ceph-mon[35869]: pgmap v10506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:10.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18768.124989 secs 2024-07-26T13:12:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:10 smithi097 ceph-mon[32231]: pgmap v10506: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18768.124989 secs 2024-07-26T13:12:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:12 smithi115 ceph-mon[35364]: pgmap v10507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:12 smithi181 ceph-mon[35869]: pgmap v10507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:12 smithi097 ceph-mon[32231]: pgmap v10507: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:14 smithi115 ceph-mon[35364]: pgmap v10508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:14 smithi181 ceph-mon[35869]: pgmap v10508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:14 smithi097 ceph-mon[32231]: pgmap v10508: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18773.125138 secs 2024-07-26T13:12:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18773.125138 secs 2024-07-26T13:12:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18773.125138 secs 2024-07-26T13:12:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:16 smithi097 ceph-mon[32231]: pgmap v10509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:16 smithi115 ceph-mon[35364]: pgmap v10509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:16 smithi181 ceph-mon[35869]: pgmap v10509: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:18 smithi097 ceph-mon[32231]: pgmap v10510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:18.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:18 smithi115 ceph-mon[35364]: pgmap v10510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:18 smithi181 ceph-mon[35869]: pgmap v10510: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:20 smithi115 ceph-mon[35364]: pgmap v10511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18778.125327 secs 2024-07-26T13:12:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:20 smithi181 ceph-mon[35869]: pgmap v10511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18778.125327 secs 2024-07-26T13:12:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:20 smithi097 ceph-mon[32231]: pgmap v10511: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18778.125327 secs 2024-07-26T13:12:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:22 smithi115 ceph-mon[35364]: pgmap v10512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:22 smithi181 ceph-mon[35869]: pgmap v10512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:22 smithi097 ceph-mon[32231]: pgmap v10512: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:24 smithi115 ceph-mon[35364]: pgmap v10513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:24 smithi181 ceph-mon[35869]: pgmap v10513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:24 smithi097 ceph-mon[32231]: pgmap v10513: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:25.251+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:25.252+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:12:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:25.252+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:12:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:25.252+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:25.252+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18783.125472 secs 2024-07-26T13:12:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18783.125472 secs 2024-07-26T13:12:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18783.125472 secs 2024-07-26T13:12:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:26 smithi115 ceph-mon[35364]: pgmap v10514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:26 smithi181 ceph-mon[35869]: pgmap v10514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:26 smithi097 ceph-mon[32231]: pgmap v10514: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:28 smithi115 ceph-mon[35364]: pgmap v10515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:28 smithi181 ceph-mon[35869]: pgmap v10515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:28 smithi097 ceph-mon[32231]: pgmap v10515: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:30 smithi115 ceph-mon[35364]: pgmap v10516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18788.125620 secs 2024-07-26T13:12:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:30 smithi181 ceph-mon[35869]: pgmap v10516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18788.125620 secs 2024-07-26T13:12:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:30 smithi097 ceph-mon[32231]: pgmap v10516: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18788.125620 secs 2024-07-26T13:12:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:32 smithi115 ceph-mon[35364]: pgmap v10517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:32 smithi181 ceph-mon[35869]: pgmap v10517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:32 smithi097 ceph-mon[32231]: pgmap v10517: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:33 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:33 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:33 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:34 smithi115 ceph-mon[35364]: pgmap v10518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:34 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:12:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:34 smithi181 ceph-mon[35869]: pgmap v10518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:35.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:34 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:12:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:34 smithi097 ceph-mon[32231]: pgmap v10518: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:34 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:12:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:35 smithi115 ceph-mon[35364]: pgmap v10519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:12:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:35 smithi115 ceph-mon[35364]: pgmap v10520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18793.125719 secs 2024-07-26T13:12:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:35 smithi181 ceph-mon[35869]: pgmap v10519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:12:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:35 smithi181 ceph-mon[35869]: pgmap v10520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18793.125719 secs 2024-07-26T13:12:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:35 smithi097 ceph-mon[32231]: pgmap v10519: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:12:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:35 smithi097 ceph-mon[32231]: pgmap v10520: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18793.125719 secs 2024-07-26T13:12:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:37 smithi115 ceph-mon[35364]: pgmap v10521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:37 smithi181 ceph-mon[35869]: pgmap v10521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:37 smithi097 ceph-mon[32231]: pgmap v10521: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:39 smithi115 ceph-mon[35364]: pgmap v10522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:39 smithi181 ceph-mon[35869]: pgmap v10522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:39 smithi097 ceph-mon[32231]: pgmap v10522: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:40.259+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:40.261+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:12:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:40.261+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:12:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:40.261+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:40.261+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18798.125862 secs 2024-07-26T13:12:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18798.125862 secs 2024-07-26T13:12:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18798.125862 secs 2024-07-26T13:12:42.134 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:41 smithi181 ceph-mon[35869]: pgmap v10523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:42.153 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:41 smithi097 ceph-mon[32231]: pgmap v10523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:41 smithi115 ceph-mon[35364]: pgmap v10523: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:44.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:43 smithi115 ceph-mon[35364]: pgmap v10524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:43 smithi181 ceph-mon[35869]: pgmap v10524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:44.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:43 smithi097 ceph-mon[32231]: pgmap v10524: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:45 smithi115 ceph-mon[35364]: pgmap v10525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:46.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18803.126002 secs 2024-07-26T13:12:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:45 smithi181 ceph-mon[35869]: pgmap v10525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18803.126002 secs 2024-07-26T13:12:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:45 smithi097 ceph-mon[32231]: pgmap v10525: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:46.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18803.126002 secs 2024-07-26T13:12:48.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:47 smithi115 ceph-mon[35364]: pgmap v10526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:47 smithi181 ceph-mon[35869]: pgmap v10526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:48.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:47 smithi097 ceph-mon[32231]: pgmap v10526: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:50.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:49 smithi115 ceph-mon[35364]: pgmap v10527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:49 smithi181 ceph-mon[35869]: pgmap v10527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:50.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:49 smithi097 ceph-mon[32231]: pgmap v10527: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:51.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18808.126142 secs 2024-07-26T13:12:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18808.126142 secs 2024-07-26T13:12:51.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18808.126142 secs 2024-07-26T13:12:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:51 smithi115 ceph-mon[35364]: pgmap v10528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:51 smithi181 ceph-mon[35869]: pgmap v10528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:51 smithi097 ceph-mon[32231]: pgmap v10528: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:54.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:53 smithi115 ceph-mon[35364]: pgmap v10529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:53 smithi181 ceph-mon[35869]: pgmap v10529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:54.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:53 smithi097 ceph-mon[32231]: pgmap v10529: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:55.264+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:55.266+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:12:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:55.266+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:12:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:55.266+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:12:55.266+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:55 smithi115 ceph-mon[35364]: pgmap v10530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18813.126274 secs 2024-07-26T13:12:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:55 smithi181 ceph-mon[35869]: pgmap v10530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18813.126274 secs 2024-07-26T13:12:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:55 smithi097 ceph-mon[32231]: pgmap v10530: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:56.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18813.126274 secs 2024-07-26T13:12:58.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:57 smithi115 ceph-mon[35364]: pgmap v10531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:57 smithi181 ceph-mon[35869]: pgmap v10531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:58.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:57 smithi097 ceph-mon[32231]: pgmap v10531: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:00.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:12:59 smithi115 ceph-mon[35364]: pgmap v10532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:12:59 smithi181 ceph-mon[35869]: pgmap v10532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:00.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:12:59 smithi097 ceph-mon[32231]: pgmap v10532: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18818.126446 secs 2024-07-26T13:13:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18818.126446 secs 2024-07-26T13:13:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18818.126446 secs 2024-07-26T13:13:02.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:01 smithi115 ceph-mon[35364]: pgmap v10533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:01 smithi181 ceph-mon[35869]: pgmap v10533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:02.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:01 smithi097 ceph-mon[32231]: pgmap v10533: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:03 smithi181 ceph-mon[35869]: pgmap v10534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:04.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:03 smithi097 ceph-mon[32231]: pgmap v10534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:03 smithi115 ceph-mon[35364]: pgmap v10534: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:05 smithi097 ceph-mon[32231]: pgmap v10535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18823.126577 secs 2024-07-26T13:13:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:05 smithi115 ceph-mon[35364]: pgmap v10535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18823.126577 secs 2024-07-26T13:13:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:05 smithi181 ceph-mon[35869]: pgmap v10535: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18823.126577 secs 2024-07-26T13:13:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.939+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 msgr2=0x7ff3d40b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T13:13:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.939+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 msgr2=0x7ff3d40b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:13:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.939+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40b5be0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e008b070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.939+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40b5be0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005820 tx=0x7ff3e008b070 comp rx=0 tx=0).stop 2024-07-26T13:13:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.939+0000 7ff3efe00640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 msgr2=0x7ff3d40b5be0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.97:6800/2663280026, fault. 2024-07-26T13:13:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.939+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 msgr2=0x7ff3d40b5be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:13:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.939+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b57d0 0x7ff3d40b5be0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:13:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.939+0000 7ff3ed600640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:13:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.940+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:06.942+0000 7ff3efe00640 1 --2- 172.21.15.97:0/530760114 >> [v2:172.21.15.97:6800/2663280026,v1:172.21.15.97:6801/2663280026] conn(0x7ff3d40b4fa0 0x7ff3d40b4440 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00313d0 tx=0x7ff3e00a8000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:13:08.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:07 smithi097 ceph-mon[32231]: pgmap v10536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:07 smithi115 ceph-mon[35364]: pgmap v10536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:07 smithi181 ceph-mon[35869]: pgmap v10536: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.245+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:13:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.245+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:13:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.245+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:13:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.245+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:13:10.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:09 smithi097 ceph-mon[32231]: pgmap v10537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.267+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.268+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:13:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.268+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:13:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.268+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:10.269 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:10.268+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:09 smithi115 ceph-mon[35364]: pgmap v10537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:09 smithi181 ceph-mon[35869]: pgmap v10537: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18828.126711 secs 2024-07-26T13:13:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18828.126711 secs 2024-07-26T13:13:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18828.126711 secs 2024-07-26T13:13:12.384 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:12 smithi181 ceph-mon[35869]: pgmap v10538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:12.403 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:12 smithi097 ceph-mon[32231]: pgmap v10538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:12.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:12 smithi115 ceph-mon[35364]: pgmap v10538: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:14.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:14 smithi115 ceph-mon[35364]: pgmap v10539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:14 smithi181 ceph-mon[35869]: pgmap v10539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:14 smithi097 ceph-mon[32231]: pgmap v10539: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:16 smithi115 ceph-mon[35364]: pgmap v10540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18833.126814 secs 2024-07-26T13:13:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:16 smithi181 ceph-mon[35869]: pgmap v10540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18833.126814 secs 2024-07-26T13:13:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:16 smithi097 ceph-mon[32231]: pgmap v10540: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18833.126814 secs 2024-07-26T13:13:18.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:18 smithi115 ceph-mon[35364]: pgmap v10541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:18 smithi181 ceph-mon[35869]: pgmap v10541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:18 smithi097 ceph-mon[32231]: pgmap v10541: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:20.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:20 smithi115 ceph-mon[35364]: pgmap v10542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:20 smithi181 ceph-mon[35869]: pgmap v10542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:20.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:20 smithi097 ceph-mon[32231]: pgmap v10542: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18838.126956 secs 2024-07-26T13:13:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18838.126956 secs 2024-07-26T13:13:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18838.126956 secs 2024-07-26T13:13:22.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:22 smithi115 ceph-mon[35364]: pgmap v10543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:22 smithi181 ceph-mon[35869]: pgmap v10543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:22.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:22 smithi097 ceph-mon[32231]: pgmap v10543: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:24.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:24 smithi115 ceph-mon[35364]: pgmap v10544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:24 smithi181 ceph-mon[35869]: pgmap v10544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:24.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:24 smithi097 ceph-mon[32231]: pgmap v10544: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:25.272 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:25.271+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:25.272+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:13:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:25.272+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:13:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:25.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:25.272+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:26 smithi115 ceph-mon[35364]: pgmap v10545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18843.127114 secs 2024-07-26T13:13:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:26 smithi181 ceph-mon[35869]: pgmap v10545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:26.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18843.127114 secs 2024-07-26T13:13:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:26 smithi097 ceph-mon[32231]: pgmap v10545: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18843.127114 secs 2024-07-26T13:13:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:28 smithi115 ceph-mon[35364]: pgmap v10546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:28 smithi181 ceph-mon[35869]: pgmap v10546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:28 smithi097 ceph-mon[32231]: pgmap v10546: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:30.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:30 smithi115 ceph-mon[35364]: pgmap v10547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:30.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:30 smithi181 ceph-mon[35869]: pgmap v10547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:30.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:30 smithi097 ceph-mon[32231]: pgmap v10547: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18848.127311 secs 2024-07-26T13:13:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18848.127311 secs 2024-07-26T13:13:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18848.127311 secs 2024-07-26T13:13:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:32 smithi115 ceph-mon[35364]: pgmap v10548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:32 smithi181 ceph-mon[35869]: pgmap v10548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:32 smithi097 ceph-mon[32231]: pgmap v10548: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:34 smithi115 ceph-mon[35364]: pgmap v10549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:34 smithi181 ceph-mon[35869]: pgmap v10549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:34 smithi097 ceph-mon[32231]: pgmap v10549: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:36 smithi115 ceph-mon[35364]: pgmap v10550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:13:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:36 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18853.127471 secs 2024-07-26T13:13:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:36 smithi181 ceph-mon[35869]: pgmap v10550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:13:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:36 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18853.127471 secs 2024-07-26T13:13:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:36 smithi097 ceph-mon[32231]: pgmap v10550: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:13:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:36 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18853.127471 secs 2024-07-26T13:13:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:37 smithi115 ceph-mon[35364]: pgmap v10551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:13:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:13:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:37 smithi181 ceph-mon[35869]: pgmap v10551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:13:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:13:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:37 smithi097 ceph-mon[32231]: pgmap v10551: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:13:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:13:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:39 smithi097 ceph-mon[32231]: pgmap v10552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:13:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:39 smithi115 ceph-mon[35364]: pgmap v10552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:13:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:39 smithi181 ceph-mon[35869]: pgmap v10552: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:13:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:40.280+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:40.281+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:13:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:40.281+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:13:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:40.281+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:40.281+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18858.127604 secs 2024-07-26T13:13:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18858.127604 secs 2024-07-26T13:13:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18858.127604 secs 2024-07-26T13:13:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:41 smithi115 ceph-mon[35364]: pgmap v10553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:13:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:41 smithi181 ceph-mon[35869]: pgmap v10553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:13:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:41 smithi097 ceph-mon[32231]: pgmap v10553: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:13:42.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:42 smithi181 ceph-mon[35869]: pgmap v10554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:13:42.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:42 smithi097 ceph-mon[32231]: pgmap v10554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:13:42.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:42 smithi115 ceph-mon[35364]: pgmap v10554: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:13:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:45 smithi097 ceph-mon[32231]: pgmap v10555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:13:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18863.127744 secs 2024-07-26T13:13:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:45 smithi115 ceph-mon[35364]: pgmap v10555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:13:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18863.127744 secs 2024-07-26T13:13:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:45 smithi181 ceph-mon[35869]: pgmap v10555: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:13:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18863.127744 secs 2024-07-26T13:13:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:47 smithi115 ceph-mon[35364]: pgmap v10556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:13:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:47 smithi181 ceph-mon[35869]: pgmap v10556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:13:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:47 smithi097 ceph-mon[32231]: pgmap v10556: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:13:48.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:48 smithi115 ceph-mon[35364]: pgmap v10557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:48 smithi181 ceph-mon[35869]: pgmap v10557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:48.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:48 smithi097 ceph-mon[32231]: pgmap v10557: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18868.127917 secs 2024-07-26T13:13:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:50 smithi115 ceph-mon[35364]: pgmap v10558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18868.127917 secs 2024-07-26T13:13:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:50 smithi181 ceph-mon[35869]: pgmap v10558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18868.127917 secs 2024-07-26T13:13:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:50 smithi097 ceph-mon[32231]: pgmap v10558: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:53 smithi097 ceph-mon[32231]: pgmap v10559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:53 smithi115 ceph-mon[35364]: pgmap v10559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:53 smithi181 ceph-mon[35869]: pgmap v10559: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:54.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:54 smithi115 ceph-mon[35364]: pgmap v10560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:54.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:54 smithi181 ceph-mon[35869]: pgmap v10560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:54.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:54 smithi097 ceph-mon[32231]: pgmap v10560: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:55.285+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:13:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:55.286+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:13:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:55.286+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:13:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:55.286+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:13:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:13:55.286+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18873.128086 secs 2024-07-26T13:13:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18873.128086 secs 2024-07-26T13:13:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18873.128086 secs 2024-07-26T13:13:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:56 smithi115 ceph-mon[35364]: pgmap v10561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:56 smithi181 ceph-mon[35869]: pgmap v10561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:56 smithi097 ceph-mon[32231]: pgmap v10561: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:59.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:13:59 smithi115 ceph-mon[35364]: pgmap v10562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:13:59 smithi181 ceph-mon[35869]: pgmap v10562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:59.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:13:59 smithi097 ceph-mon[32231]: pgmap v10562: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18878.128276 secs 2024-07-26T13:14:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18878.128276 secs 2024-07-26T13:14:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18878.128276 secs 2024-07-26T13:14:01.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:01 smithi115 ceph-mon[35364]: pgmap v10563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:01 smithi181 ceph-mon[35869]: pgmap v10563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:01.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:01 smithi097 ceph-mon[32231]: pgmap v10563: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:02 smithi115 ceph-mon[35364]: pgmap v10564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:02 smithi181 ceph-mon[35869]: pgmap v10564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:02 smithi097 ceph-mon[32231]: pgmap v10564: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:05 smithi115 ceph-mon[35364]: pgmap v10565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18883.128368 secs 2024-07-26T13:14:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:05 smithi181 ceph-mon[35869]: pgmap v10565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18883.128368 secs 2024-07-26T13:14:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:05 smithi097 ceph-mon[32231]: pgmap v10565: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18883.128368 secs 2024-07-26T13:14:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:06 smithi115 ceph-mon[35364]: pgmap v10566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:06 smithi181 ceph-mon[35869]: pgmap v10566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:06 smithi097 ceph-mon[32231]: pgmap v10566: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:09 smithi115 ceph-mon[35364]: pgmap v10567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:09 smithi181 ceph-mon[35869]: pgmap v10567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:09.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:09 smithi097 ceph-mon[32231]: pgmap v10567: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:10.294+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:14:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:10.296+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:14:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:10.296+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:14:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:10.296+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:14:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:10.296+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18888.128513 secs 2024-07-26T13:14:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:10 smithi115 ceph-mon[35364]: pgmap v10568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18888.128513 secs 2024-07-26T13:14:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:10 smithi181 ceph-mon[35869]: pgmap v10568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18888.128513 secs 2024-07-26T13:14:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:10 smithi097 ceph-mon[32231]: pgmap v10568: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:12 smithi115 ceph-mon[35364]: pgmap v10569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:12 smithi181 ceph-mon[35869]: pgmap v10569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:12 smithi097 ceph-mon[32231]: pgmap v10569: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:14 smithi115 ceph-mon[35364]: pgmap v10570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:14 smithi181 ceph-mon[35869]: pgmap v10570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:14 smithi097 ceph-mon[32231]: pgmap v10570: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:15.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18893.128613 secs 2024-07-26T13:14:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18893.128613 secs 2024-07-26T13:14:15.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18893.128613 secs 2024-07-26T13:14:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:16 smithi115 ceph-mon[35364]: pgmap v10571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:16 smithi181 ceph-mon[35869]: pgmap v10571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:16 smithi097 ceph-mon[32231]: pgmap v10571: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:18 smithi115 ceph-mon[35364]: pgmap v10572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:18 smithi181 ceph-mon[35869]: pgmap v10572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:18.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:18 smithi097 ceph-mon[32231]: pgmap v10572: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18898.128754 secs 2024-07-26T13:14:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:20 smithi115 ceph-mon[35364]: pgmap v10573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18898.128754 secs 2024-07-26T13:14:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:20 smithi181 ceph-mon[35869]: pgmap v10573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18898.128754 secs 2024-07-26T13:14:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:20 smithi097 ceph-mon[32231]: pgmap v10573: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:22 smithi115 ceph-mon[35364]: pgmap v10574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:22 smithi181 ceph-mon[35869]: pgmap v10574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:22 smithi097 ceph-mon[32231]: pgmap v10574: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:24 smithi115 ceph-mon[35364]: pgmap v10575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:24 smithi181 ceph-mon[35869]: pgmap v10575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:24 smithi097 ceph-mon[32231]: pgmap v10575: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:25.300 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:25.298+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:14:25.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:25.300+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:14:25.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:25.300+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:14:25.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:25.300+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:14:25.301 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:25.300+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18903.128904 secs 2024-07-26T13:14:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18903.128904 secs 2024-07-26T13:14:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18903.128904 secs 2024-07-26T13:14:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:26 smithi115 ceph-mon[35364]: pgmap v10576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:26 smithi181 ceph-mon[35869]: pgmap v10576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:26 smithi097 ceph-mon[32231]: pgmap v10576: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:28 smithi115 ceph-mon[35364]: pgmap v10577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:28 smithi181 ceph-mon[35869]: pgmap v10577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:28 smithi097 ceph-mon[32231]: pgmap v10577: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18908.129053 secs 2024-07-26T13:14:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:30 smithi115 ceph-mon[35364]: pgmap v10578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18908.129053 secs 2024-07-26T13:14:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:30 smithi181 ceph-mon[35869]: pgmap v10578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18908.129053 secs 2024-07-26T13:14:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:30 smithi097 ceph-mon[32231]: pgmap v10578: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:32 smithi115 ceph-mon[35364]: pgmap v10579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:32 smithi181 ceph-mon[35869]: pgmap v10579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:32 smithi097 ceph-mon[32231]: pgmap v10579: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:34 smithi115 ceph-mon[35364]: pgmap v10580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:34 smithi181 ceph-mon[35869]: pgmap v10580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:34 smithi097 ceph-mon[32231]: pgmap v10580: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18913.129200 secs 2024-07-26T13:14:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18913.129200 secs 2024-07-26T13:14:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18913.129200 secs 2024-07-26T13:14:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:36 smithi115 ceph-mon[35364]: pgmap v10581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:36 smithi181 ceph-mon[35869]: pgmap v10581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:36 smithi097 ceph-mon[32231]: pgmap v10581: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:37 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:37 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:37 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:38 smithi115 ceph-mon[35364]: pgmap v10582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:14:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:38 smithi115 ceph-mon[35364]: pgmap v10583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:38.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:14:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:38.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:38.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:38 smithi181 ceph-mon[35869]: pgmap v10582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:14:38.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:38 smithi181 ceph-mon[35869]: pgmap v10583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:38.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:14:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:38 smithi097 ceph-mon[32231]: pgmap v10582: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:14:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:38 smithi097 ceph-mon[32231]: pgmap v10583: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:38.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:14:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:40.304+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:14:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:40.306+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:14:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:40.306+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:14:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:40.306+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:14:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:40.306+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:40 smithi115 ceph-mon[35364]: pgmap v10584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:14:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18918.129343 secs 2024-07-26T13:14:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:40 smithi181 ceph-mon[35869]: pgmap v10584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:14:40.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18918.129343 secs 2024-07-26T13:14:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:40 smithi097 ceph-mon[32231]: pgmap v10584: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:14:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18918.129343 secs 2024-07-26T13:14:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:42 smithi115 ceph-mon[35364]: pgmap v10585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:42 smithi181 ceph-mon[35869]: pgmap v10585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:42 smithi097 ceph-mon[32231]: pgmap v10585: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:44 smithi115 ceph-mon[35364]: pgmap v10586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:44 smithi181 ceph-mon[35869]: pgmap v10586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:44 smithi097 ceph-mon[32231]: pgmap v10586: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18923.129490 secs 2024-07-26T13:14:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18923.129490 secs 2024-07-26T13:14:46.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18923.129490 secs 2024-07-26T13:14:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:46 smithi115 ceph-mon[35364]: pgmap v10587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:14:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:46 smithi181 ceph-mon[35869]: pgmap v10587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:14:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:46 smithi097 ceph-mon[32231]: pgmap v10587: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:14:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:48 smithi115 ceph-mon[35364]: pgmap v10588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:48 smithi181 ceph-mon[35869]: pgmap v10588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:48 smithi097 ceph-mon[32231]: pgmap v10588: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:50 smithi115 ceph-mon[35364]: pgmap v10589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18928.129633 secs 2024-07-26T13:14:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:50 smithi181 ceph-mon[35869]: pgmap v10589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18928.129633 secs 2024-07-26T13:14:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:50 smithi097 ceph-mon[32231]: pgmap v10589: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18928.129633 secs 2024-07-26T13:14:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:52 smithi181 ceph-mon[35869]: pgmap v10590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:52 smithi097 ceph-mon[32231]: pgmap v10590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:53.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:52 smithi115 ceph-mon[35364]: pgmap v10590: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:54 smithi097 ceph-mon[32231]: pgmap v10591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:55.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:54 smithi115 ceph-mon[35364]: pgmap v10591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:54 smithi181 ceph-mon[35869]: pgmap v10591: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:55.311 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:55.309+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:55.311+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:55.311+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:55.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:14:55.311+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18933.129729 secs 2024-07-26T13:14:56.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18933.129729 secs 2024-07-26T13:14:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18933.129729 secs 2024-07-26T13:14:57.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:56 smithi115 ceph-mon[35364]: pgmap v10592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:56 smithi181 ceph-mon[35869]: pgmap v10592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:57.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:56 smithi097 ceph-mon[32231]: pgmap v10592: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:59.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:14:58 smithi115 ceph-mon[35364]: pgmap v10593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:14:58 smithi181 ceph-mon[35869]: pgmap v10593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:59.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:14:58 smithi097 ceph-mon[32231]: pgmap v10593: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:00 smithi115 ceph-mon[35364]: pgmap v10594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:01.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18938.129827 secs 2024-07-26T13:15:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:00 smithi181 ceph-mon[35869]: pgmap v10594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18938.129827 secs 2024-07-26T13:15:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:00 smithi097 ceph-mon[32231]: pgmap v10594: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:01.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18938.129827 secs 2024-07-26T13:15:03.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:02 smithi115 ceph-mon[35364]: pgmap v10595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:02 smithi181 ceph-mon[35869]: pgmap v10595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:03.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:02 smithi097 ceph-mon[32231]: pgmap v10595: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:04 smithi115 ceph-mon[35364]: pgmap v10596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:04 smithi181 ceph-mon[35869]: pgmap v10596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:05.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:04 smithi097 ceph-mon[32231]: pgmap v10596: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18943.129968 secs 2024-07-26T13:15:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18943.129968 secs 2024-07-26T13:15:06.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18943.129968 secs 2024-07-26T13:15:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:06 smithi115 ceph-mon[35364]: pgmap v10597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:15:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:06 smithi181 ceph-mon[35869]: pgmap v10597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:15:07.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:06 smithi097 ceph-mon[32231]: pgmap v10597: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:15:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:08 smithi115 ceph-mon[35364]: pgmap v10598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:15:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:08 smithi181 ceph-mon[35869]: pgmap v10598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:15:09.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:08 smithi097 ceph-mon[32231]: pgmap v10598: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:15:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:10.319+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:15:10.322 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:10.321+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:15:10.322 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:10.321+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:15:10.322 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:10.321+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:15:10.322 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:10.321+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:10 smithi115 ceph-mon[35364]: pgmap v10599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18948.130111 secs 2024-07-26T13:15:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:10 smithi181 ceph-mon[35869]: pgmap v10599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18948.130111 secs 2024-07-26T13:15:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:10 smithi097 ceph-mon[32231]: pgmap v10599: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:11.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18948.130111 secs 2024-07-26T13:15:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:12 smithi115 ceph-mon[35364]: pgmap v10600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:12 smithi181 ceph-mon[35869]: pgmap v10600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:12 smithi097 ceph-mon[32231]: pgmap v10600: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:14 smithi115 ceph-mon[35364]: pgmap v10601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:14 smithi181 ceph-mon[35869]: pgmap v10601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:15.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:14 smithi097 ceph-mon[32231]: pgmap v10601: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18953.130283 secs 2024-07-26T13:15:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18953.130283 secs 2024-07-26T13:15:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18953.130283 secs 2024-07-26T13:15:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:16 smithi115 ceph-mon[35364]: pgmap v10602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:17.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:16 smithi181 ceph-mon[35869]: pgmap v10602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:16 smithi097 ceph-mon[32231]: pgmap v10602: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:19.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:18 smithi115 ceph-mon[35364]: pgmap v10603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:18 smithi181 ceph-mon[35869]: pgmap v10603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:18 smithi097 ceph-mon[32231]: pgmap v10603: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:20 smithi115 ceph-mon[35364]: pgmap v10604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:21.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18958.130433 secs 2024-07-26T13:15:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:20 smithi181 ceph-mon[35869]: pgmap v10604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18958.130433 secs 2024-07-26T13:15:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:20 smithi097 ceph-mon[32231]: pgmap v10604: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18958.130433 secs 2024-07-26T13:15:23.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:22 smithi115 ceph-mon[35364]: pgmap v10605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:22 smithi181 ceph-mon[35869]: pgmap v10605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:23.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:22 smithi097 ceph-mon[32231]: pgmap v10605: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:25.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:24 smithi115 ceph-mon[35364]: pgmap v10606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:24 smithi181 ceph-mon[35869]: pgmap v10606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:25.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:24 smithi097 ceph-mon[32231]: pgmap v10606: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:25.323+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:15:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:25.324+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:15:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:25.324+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:15:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:25.324+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:15:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:25.325+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18963.130588 secs 2024-07-26T13:15:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18963.130588 secs 2024-07-26T13:15:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18963.130588 secs 2024-07-26T13:15:27.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:26 smithi115 ceph-mon[35364]: pgmap v10607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:26 smithi181 ceph-mon[35869]: pgmap v10607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:27.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:26 smithi097 ceph-mon[32231]: pgmap v10607: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:29.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:28 smithi115 ceph-mon[35364]: pgmap v10608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:28 smithi181 ceph-mon[35869]: pgmap v10608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:29.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:28 smithi097 ceph-mon[32231]: pgmap v10608: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:31.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:30 smithi115 ceph-mon[35364]: pgmap v10609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:31.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18968.130676 secs 2024-07-26T13:15:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:30 smithi181 ceph-mon[35869]: pgmap v10609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18968.130676 secs 2024-07-26T13:15:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:30 smithi097 ceph-mon[32231]: pgmap v10609: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18968.130676 secs 2024-07-26T13:15:33.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:32 smithi115 ceph-mon[35364]: pgmap v10610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:32 smithi181 ceph-mon[35869]: pgmap v10610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:33.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:32 smithi097 ceph-mon[32231]: pgmap v10610: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:35.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:34 smithi115 ceph-mon[35364]: pgmap v10611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:35.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:34 smithi181 ceph-mon[35869]: pgmap v10611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:35.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:34 smithi097 ceph-mon[32231]: pgmap v10611: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18973.130817 secs 2024-07-26T13:15:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18973.130817 secs 2024-07-26T13:15:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18973.130817 secs 2024-07-26T13:15:37.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:36 smithi115 ceph-mon[35364]: pgmap v10612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:36 smithi181 ceph-mon[35869]: pgmap v10612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:37.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:36 smithi097 ceph-mon[32231]: pgmap v10612: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:38 smithi115 ceph-mon[35364]: pgmap v10613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:39.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:38 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:38 smithi181 ceph-mon[35869]: pgmap v10613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:38 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:38 smithi097 ceph-mon[32231]: pgmap v10613: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:39.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:38 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:39 smithi115 ceph-mon[35364]: pgmap v10614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:15:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:39 smithi115 ceph-mon[35364]: pgmap v10615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:40.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:39 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:15:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:39 smithi181 ceph-mon[35869]: pgmap v10614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:15:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:39 smithi181 ceph-mon[35869]: pgmap v10615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:39 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:15:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:39 smithi097 ceph-mon[32231]: pgmap v10614: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:15:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:39 smithi097 ceph-mon[32231]: pgmap v10615: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:40.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:39 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:15:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:40.333+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:15:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:40.334+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:15:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:40.334+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:15:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:40.334+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:15:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:40.334+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:41.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18978.130956 secs 2024-07-26T13:15:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18978.130956 secs 2024-07-26T13:15:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18978.130956 secs 2024-07-26T13:15:42.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:41 smithi115 ceph-mon[35364]: pgmap v10616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:41 smithi181 ceph-mon[35869]: pgmap v10616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:41 smithi097 ceph-mon[32231]: pgmap v10616: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:44 smithi115 ceph-mon[35364]: pgmap v10617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:15:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:44 smithi181 ceph-mon[35869]: pgmap v10617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:15:44.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:44 smithi097 ceph-mon[32231]: pgmap v10617: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:15:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:46 smithi115 ceph-mon[35364]: pgmap v10618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:15:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18983.131115 secs 2024-07-26T13:15:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:46 smithi181 ceph-mon[35869]: pgmap v10618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:15:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18983.131115 secs 2024-07-26T13:15:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:46 smithi097 ceph-mon[32231]: pgmap v10618: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:15:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18983.131115 secs 2024-07-26T13:15:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:48 smithi115 ceph-mon[35364]: pgmap v10619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:48 smithi181 ceph-mon[35869]: pgmap v10619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:48 smithi097 ceph-mon[32231]: pgmap v10619: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:50.476 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:50 smithi115 ceph-mon[35364]: pgmap v10620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:50 smithi181 ceph-mon[35869]: pgmap v10620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:50 smithi097 ceph-mon[32231]: pgmap v10620: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18988.131297 secs 2024-07-26T13:15:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18988.131297 secs 2024-07-26T13:15:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18988.131297 secs 2024-07-26T13:15:52.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:52 smithi115 ceph-mon[35364]: pgmap v10621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:52 smithi181 ceph-mon[35869]: pgmap v10621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:52 smithi097 ceph-mon[32231]: pgmap v10621: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:54.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:54 smithi115 ceph-mon[35364]: pgmap v10622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:54 smithi181 ceph-mon[35869]: pgmap v10622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:54 smithi097 ceph-mon[32231]: pgmap v10622: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:55.337+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:15:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:55.339+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:15:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:55.339+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:15:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:55.339+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:15:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:15:55.339+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:56 smithi115 ceph-mon[35364]: pgmap v10623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18993.131459 secs 2024-07-26T13:15:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:56 smithi181 ceph-mon[35869]: pgmap v10623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18993.131459 secs 2024-07-26T13:15:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:56 smithi097 ceph-mon[32231]: pgmap v10623: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18993.131459 secs 2024-07-26T13:15:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:15:58 smithi115 ceph-mon[35364]: pgmap v10624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:15:58 smithi181 ceph-mon[35869]: pgmap v10624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:15:58 smithi097 ceph-mon[32231]: pgmap v10624: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:00.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:00 smithi115 ceph-mon[35364]: pgmap v10625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:00 smithi181 ceph-mon[35869]: pgmap v10625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:00 smithi097 ceph-mon[32231]: pgmap v10625: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 18998.131595 secs 2024-07-26T13:16:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 18998.131595 secs 2024-07-26T13:16:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 18998.131595 secs 2024-07-26T13:16:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:02 smithi115 ceph-mon[35364]: pgmap v10626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:02 smithi181 ceph-mon[35869]: pgmap v10626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:02 smithi097 ceph-mon[32231]: pgmap v10626: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:04 smithi115 ceph-mon[35364]: pgmap v10627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:04 smithi181 ceph-mon[35869]: pgmap v10627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:04 smithi097 ceph-mon[32231]: pgmap v10627: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:06 smithi115 ceph-mon[35364]: pgmap v10628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19003.131678 secs 2024-07-26T13:16:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:06 smithi181 ceph-mon[35869]: pgmap v10628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19003.131678 secs 2024-07-26T13:16:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:06 smithi097 ceph-mon[32231]: pgmap v10628: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19003.131678 secs 2024-07-26T13:16:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:08 smithi115 ceph-mon[35364]: pgmap v10629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:08 smithi181 ceph-mon[35869]: pgmap v10629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:08 smithi097 ceph-mon[32231]: pgmap v10629: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.275+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:16:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.275+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:16:10.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.275+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:16:10.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.276+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:16:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:16:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.343+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:16:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.343+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:16:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:16:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:10.343+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:10.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:10 smithi115 ceph-mon[35364]: pgmap v10630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:10 smithi181 ceph-mon[35869]: pgmap v10630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:10.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:10 smithi097 ceph-mon[32231]: pgmap v10630: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19008.131855 secs 2024-07-26T13:16:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19008.131855 secs 2024-07-26T13:16:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19008.131855 secs 2024-07-26T13:16:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:12 smithi181 ceph-mon[35869]: pgmap v10631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:12 smithi097 ceph-mon[32231]: pgmap v10631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:12 smithi115 ceph-mon[35364]: pgmap v10631: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:14.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:14 smithi097 ceph-mon[32231]: pgmap v10632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:14 smithi115 ceph-mon[35364]: pgmap v10632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:14 smithi181 ceph-mon[35869]: pgmap v10632: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:16 smithi097 ceph-mon[32231]: pgmap v10633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19013.132005 secs 2024-07-26T13:16:16.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:16 smithi115 ceph-mon[35364]: pgmap v10633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:16.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19013.132005 secs 2024-07-26T13:16:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:16 smithi181 ceph-mon[35869]: pgmap v10633: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:16.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19013.132005 secs 2024-07-26T13:16:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:18 smithi097 ceph-mon[32231]: pgmap v10634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:18 smithi115 ceph-mon[35364]: pgmap v10634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:18 smithi181 ceph-mon[35869]: pgmap v10634: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:20 smithi115 ceph-mon[35364]: pgmap v10635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:20.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19018.132148 secs 2024-07-26T13:16:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:20 smithi181 ceph-mon[35869]: pgmap v10635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19018.132148 secs 2024-07-26T13:16:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:20 smithi097 ceph-mon[32231]: pgmap v10635: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19018.132148 secs 2024-07-26T13:16:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:22 smithi115 ceph-mon[35364]: pgmap v10636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:22 smithi181 ceph-mon[35869]: pgmap v10636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:22.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:22 smithi097 ceph-mon[32231]: pgmap v10636: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:23 smithi115 ceph-mon[35364]: pgmap v10637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:23 smithi181 ceph-mon[35869]: pgmap v10637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:23 smithi097 ceph-mon[32231]: pgmap v10637: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:25.345+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:16:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:25.347+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:16:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:25.347+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:16:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:25.347+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:16:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:25.347+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:26 smithi115 ceph-mon[35364]: pgmap v10638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19023.132299 secs 2024-07-26T13:16:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:26 smithi181 ceph-mon[35869]: pgmap v10638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19023.132299 secs 2024-07-26T13:16:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:26 smithi097 ceph-mon[32231]: pgmap v10638: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19023.132299 secs 2024-07-26T13:16:28.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:28 smithi115 ceph-mon[35364]: pgmap v10639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:28 smithi181 ceph-mon[35869]: pgmap v10639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:28.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:28 smithi097 ceph-mon[32231]: pgmap v10639: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:30.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:30 smithi115 ceph-mon[35364]: pgmap v10640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:30 smithi181 ceph-mon[35869]: pgmap v10640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:30.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:30 smithi097 ceph-mon[32231]: pgmap v10640: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19028.132407 secs 2024-07-26T13:16:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19028.132407 secs 2024-07-26T13:16:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19028.132407 secs 2024-07-26T13:16:32.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:32 smithi115 ceph-mon[35364]: pgmap v10641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:32 smithi181 ceph-mon[35869]: pgmap v10641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:32.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:32 smithi097 ceph-mon[32231]: pgmap v10641: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:34.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:34 smithi115 ceph-mon[35364]: pgmap v10642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:34 smithi181 ceph-mon[35869]: pgmap v10642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:34.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:34 smithi097 ceph-mon[32231]: pgmap v10642: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:36 smithi115 ceph-mon[35364]: pgmap v10643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19033.132514 secs 2024-07-26T13:16:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:36 smithi181 ceph-mon[35869]: pgmap v10643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19033.132514 secs 2024-07-26T13:16:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:36 smithi097 ceph-mon[32231]: pgmap v10643: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19033.132514 secs 2024-07-26T13:16:38.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:38 smithi115 ceph-mon[35364]: pgmap v10644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:38 smithi181 ceph-mon[35869]: pgmap v10644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:38.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:38 smithi097 ceph-mon[32231]: pgmap v10644: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:40.355+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:16:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:40.356+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:16:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:40.356+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:16:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:40.356+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:16:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:40.356+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:40 smithi115 ceph-mon[35364]: pgmap v10645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:40.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:40 smithi181 ceph-mon[35869]: pgmap v10645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:40 smithi097 ceph-mon[32231]: pgmap v10645: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:40.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19038.132618 secs 2024-07-26T13:16:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:16:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19038.132618 secs 2024-07-26T13:16:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:16:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19038.132618 secs 2024-07-26T13:16:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:16:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:42 smithi115 ceph-mon[35364]: pgmap v10646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:16:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:42 smithi115 ceph-mon[35364]: pgmap v10647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:42 smithi181 ceph-mon[35869]: pgmap v10646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:16:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:42 smithi181 ceph-mon[35869]: pgmap v10647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:42 smithi097 ceph-mon[32231]: pgmap v10646: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:16:42.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:42 smithi097 ceph-mon[32231]: pgmap v10647: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:44.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:44 smithi115 ceph-mon[35364]: pgmap v10648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:44.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:44 smithi181 ceph-mon[35869]: pgmap v10648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:44.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:44 smithi097 ceph-mon[32231]: pgmap v10648: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:46 smithi115 ceph-mon[35364]: pgmap v10649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19043.132713 secs 2024-07-26T13:16:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:46 smithi181 ceph-mon[35869]: pgmap v10649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:46.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19043.132713 secs 2024-07-26T13:16:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:46 smithi097 ceph-mon[32231]: pgmap v10649: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19043.132713 secs 2024-07-26T13:16:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:48 smithi115 ceph-mon[35364]: pgmap v10650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:48 smithi181 ceph-mon[35869]: pgmap v10650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:48 smithi097 ceph-mon[32231]: pgmap v10650: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:50.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:50 smithi181 ceph-mon[35869]: pgmap v10651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:50.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:50 smithi097 ceph-mon[32231]: pgmap v10651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:50 smithi115 ceph-mon[35364]: pgmap v10651: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19048.132849 secs 2024-07-26T13:16:51.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19048.132849 secs 2024-07-26T13:16:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19048.132849 secs 2024-07-26T13:16:52.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:52 smithi097 ceph-mon[32231]: pgmap v10652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:52.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:52 smithi115 ceph-mon[35364]: pgmap v10652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:52 smithi181 ceph-mon[35869]: pgmap v10652: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:54.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:54 smithi097 ceph-mon[32231]: pgmap v10653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:54.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:54 smithi115 ceph-mon[35364]: pgmap v10653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:54 smithi181 ceph-mon[35869]: pgmap v10653: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:55.361 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:55.360+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:16:55.362 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:55.361+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:16:55.362 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:55.361+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:16:55.363 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:55.361+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:16:55.363 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:16:55.361+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:55 smithi115 ceph-mon[35364]: pgmap v10654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19053.132938 secs 2024-07-26T13:16:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:55 smithi181 ceph-mon[35869]: pgmap v10654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19053.132938 secs 2024-07-26T13:16:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:55 smithi097 ceph-mon[32231]: pgmap v10654: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19053.132938 secs 2024-07-26T13:16:57.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:57 smithi115 ceph-mon[35364]: pgmap v10655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:57 smithi181 ceph-mon[35869]: pgmap v10655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:58.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:57 smithi097 ceph-mon[32231]: pgmap v10655: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:59.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:16:59 smithi115 ceph-mon[35364]: pgmap v10656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:16:59 smithi181 ceph-mon[35869]: pgmap v10656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:17:00.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:16:59 smithi097 ceph-mon[32231]: pgmap v10656: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:17:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19058.133091 secs 2024-07-26T13:17:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19058.133091 secs 2024-07-26T13:17:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19058.133091 secs 2024-07-26T13:17:01.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:01 smithi115 ceph-mon[35364]: pgmap v10657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:01 smithi181 ceph-mon[35869]: pgmap v10657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:02.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:01 smithi097 ceph-mon[32231]: pgmap v10657: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:03.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:03 smithi115 ceph-mon[35364]: pgmap v10658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:03.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:03 smithi181 ceph-mon[35869]: pgmap v10658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:04.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:03 smithi097 ceph-mon[32231]: pgmap v10658: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:05 smithi115 ceph-mon[35364]: pgmap v10659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:05.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19063.133273 secs 2024-07-26T13:17:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:05 smithi181 ceph-mon[35869]: pgmap v10659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:05.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19063.133273 secs 2024-07-26T13:17:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:05 smithi097 ceph-mon[32231]: pgmap v10659: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19063.133273 secs 2024-07-26T13:17:07.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:07 smithi115 ceph-mon[35364]: pgmap v10660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:07 smithi181 ceph-mon[35869]: pgmap v10660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:08.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:07 smithi097 ceph-mon[32231]: pgmap v10660: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:09.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:09 smithi115 ceph-mon[35364]: pgmap v10661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:09 smithi181 ceph-mon[35869]: pgmap v10661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:10.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:09 smithi097 ceph-mon[32231]: pgmap v10661: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:10.363 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:10.362+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:17:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:10.363+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:17:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:10.363+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:17:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:10.363+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:17:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:10.363+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19068.133419 secs 2024-07-26T13:17:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19068.133419 secs 2024-07-26T13:17:11.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19068.133419 secs 2024-07-26T13:17:11.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:11 smithi115 ceph-mon[35364]: pgmap v10662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:11 smithi181 ceph-mon[35869]: pgmap v10662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:12.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:11 smithi097 ceph-mon[32231]: pgmap v10662: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:13.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:13 smithi115 ceph-mon[35364]: pgmap v10663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:13 smithi181 ceph-mon[35869]: pgmap v10663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:14.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:13 smithi097 ceph-mon[32231]: pgmap v10663: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:15 smithi115 ceph-mon[35364]: pgmap v10664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19073.133511 secs 2024-07-26T13:17:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:15 smithi181 ceph-mon[35869]: pgmap v10664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19073.133511 secs 2024-07-26T13:17:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:15 smithi097 ceph-mon[32231]: pgmap v10664: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19073.133511 secs 2024-07-26T13:17:17.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:17 smithi115 ceph-mon[35364]: pgmap v10665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:17 smithi181 ceph-mon[35869]: pgmap v10665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:18.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:17 smithi097 ceph-mon[32231]: pgmap v10665: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:19.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:19 smithi115 ceph-mon[35364]: pgmap v10666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:19 smithi181 ceph-mon[35869]: pgmap v10666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:20.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:19 smithi097 ceph-mon[32231]: pgmap v10666: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19078.133653 secs 2024-07-26T13:17:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19078.133653 secs 2024-07-26T13:17:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19078.133653 secs 2024-07-26T13:17:21.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:21 smithi115 ceph-mon[35364]: pgmap v10667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:21 smithi181 ceph-mon[35869]: pgmap v10667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:22.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:21 smithi097 ceph-mon[32231]: pgmap v10667: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:24.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:23 smithi115 ceph-mon[35364]: pgmap v10668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:23 smithi181 ceph-mon[35869]: pgmap v10668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:24.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:23 smithi097 ceph-mon[32231]: pgmap v10668: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:25.368 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:25.366+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:17:25.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:25.368+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:17:25.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:25.368+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:17:25.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:25.368+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:17:25.369 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:25.368+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:25 smithi115 ceph-mon[35364]: pgmap v10669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:26.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19083.133797 secs 2024-07-26T13:17:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:25 smithi181 ceph-mon[35869]: pgmap v10669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19083.133797 secs 2024-07-26T13:17:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:25 smithi097 ceph-mon[32231]: pgmap v10669: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:26.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19083.133797 secs 2024-07-26T13:17:28.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:27 smithi115 ceph-mon[35364]: pgmap v10670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:27 smithi181 ceph-mon[35869]: pgmap v10670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:28.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:27 smithi097 ceph-mon[32231]: pgmap v10670: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:30.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:29 smithi115 ceph-mon[35364]: pgmap v10671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:29 smithi181 ceph-mon[35869]: pgmap v10671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:30.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:29 smithi097 ceph-mon[32231]: pgmap v10671: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:31.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19088.133940 secs 2024-07-26T13:17:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19088.133940 secs 2024-07-26T13:17:31.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19088.133940 secs 2024-07-26T13:17:32.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:31 smithi115 ceph-mon[35364]: pgmap v10672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:17:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:31 smithi181 ceph-mon[35869]: pgmap v10672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:17:32.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:31 smithi097 ceph-mon[32231]: pgmap v10672: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:17:34.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:33 smithi115 ceph-mon[35364]: pgmap v10673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:17:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:33 smithi181 ceph-mon[35869]: pgmap v10673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:17:34.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:33 smithi097 ceph-mon[32231]: pgmap v10673: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:17:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:35 smithi115 ceph-mon[35364]: pgmap v10674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:36.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19093.134091 secs 2024-07-26T13:17:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:35 smithi181 ceph-mon[35869]: pgmap v10674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19093.134091 secs 2024-07-26T13:17:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:35 smithi097 ceph-mon[32231]: pgmap v10674: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:17:36.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19093.134091 secs 2024-07-26T13:17:38.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:37 smithi115 ceph-mon[35364]: pgmap v10675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:37 smithi181 ceph-mon[35869]: pgmap v10675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:38.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:37 smithi097 ceph-mon[32231]: pgmap v10675: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:40.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:39 smithi115 ceph-mon[35364]: pgmap v10676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:39 smithi181 ceph-mon[35869]: pgmap v10676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:40.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:39 smithi097 ceph-mon[32231]: pgmap v10676: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:40.376+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:17:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:40.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:17:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:40.378+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:17:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:40.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:17:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:40.378+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19098.134279 secs 2024-07-26T13:17:41.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:40 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19098.134279 secs 2024-07-26T13:17:41.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:40 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19098.134279 secs 2024-07-26T13:17:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:40 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:42.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: pgmap v10677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:42.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:41 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:17:42.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: pgmap v10677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:42.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:41 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: pgmap v10677: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:17:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:41 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:17:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:42 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T13:17:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:42 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T13:17:43.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:42 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T13:17:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:42 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T13:17:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:42 smithi097 ceph-mon[32231]: pgmap v10678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:17:43.252 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:42 smithi097 ceph-mon[32231]: pgmap v10679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:42 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T13:17:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:42 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T13:17:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:42 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T13:17:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:42 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T13:17:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:42 smithi115 ceph-mon[35364]: pgmap v10678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:17:43.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:42 smithi115 ceph-mon[35364]: pgmap v10679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:42 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi097 to 176.4M 2024-07-26T13:17:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:42 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi115 to 176.4M 2024-07-26T13:17:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:42 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi097 to 185043353: error parsing value: Value '185043353' is below minimum 939524096 2024-07-26T13:17:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:42 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi115 to 185028300: error parsing value: Value '185028300' is below minimum 939524096 2024-07-26T13:17:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:42 smithi181 ceph-mon[35869]: pgmap v10678: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:17:43.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:42 smithi181 ceph-mon[35869]: pgmap v10679: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:45.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:44 smithi097 ceph-mon[32231]: pgmap v10680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:45.285 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:45 smithi181 ceph-mon[35869]: pgmap v10680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:45 smithi115 ceph-mon[35364]: pgmap v10680: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19103.134440 secs 2024-07-26T13:17:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19103.134440 secs 2024-07-26T13:17:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19103.134440 secs 2024-07-26T13:17:47.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:47 smithi115 ceph-mon[35364]: pgmap v10681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:47 smithi181 ceph-mon[35869]: pgmap v10681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:47 smithi097 ceph-mon[32231]: pgmap v10681: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:49 smithi115 ceph-mon[35364]: pgmap v10682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:17:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:49 smithi181 ceph-mon[35869]: pgmap v10682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:17:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:49 smithi097 ceph-mon[32231]: pgmap v10682: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:17:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:51 smithi115 ceph-mon[35364]: pgmap v10683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:17:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19108.134583 secs 2024-07-26T13:17:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:51 smithi181 ceph-mon[35869]: pgmap v10683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:17:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19108.134583 secs 2024-07-26T13:17:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:51 smithi097 ceph-mon[32231]: pgmap v10683: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:17:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19108.134583 secs 2024-07-26T13:17:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:53 smithi115 ceph-mon[35364]: pgmap v10684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:53 smithi181 ceph-mon[35869]: pgmap v10684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:53 smithi097 ceph-mon[32231]: pgmap v10684: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:55.383 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:55.381+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:17:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:55.382+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:17:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:55.382+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:17:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:55.382+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:17:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:17:55.382+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:55 smithi115 ceph-mon[35364]: pgmap v10685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:55 smithi181 ceph-mon[35869]: pgmap v10685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:55 smithi097 ceph-mon[32231]: pgmap v10685: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19113.134689 secs 2024-07-26T13:17:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19113.134689 secs 2024-07-26T13:17:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19113.134689 secs 2024-07-26T13:17:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:57 smithi115 ceph-mon[35364]: pgmap v10686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:57 smithi181 ceph-mon[35869]: pgmap v10686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:57 smithi097 ceph-mon[32231]: pgmap v10686: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:17:59 smithi115 ceph-mon[35364]: pgmap v10687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:17:59 smithi181 ceph-mon[35869]: pgmap v10687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:17:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:17:59 smithi097 ceph-mon[32231]: pgmap v10687: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:01 smithi115 ceph-mon[35364]: pgmap v10688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19118.134827 secs 2024-07-26T13:18:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:01 smithi181 ceph-mon[35869]: pgmap v10688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19118.134827 secs 2024-07-26T13:18:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:01 smithi097 ceph-mon[32231]: pgmap v10688: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19118.134827 secs 2024-07-26T13:18:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:03 smithi115 ceph-mon[35364]: pgmap v10689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:03 smithi181 ceph-mon[35869]: pgmap v10689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:03 smithi097 ceph-mon[32231]: pgmap v10689: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:05 smithi115 ceph-mon[35364]: pgmap v10690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:05 smithi181 ceph-mon[35869]: pgmap v10690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:05 smithi097 ceph-mon[32231]: pgmap v10690: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19123.134980 secs 2024-07-26T13:18:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19123.134980 secs 2024-07-26T13:18:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19123.134980 secs 2024-07-26T13:18:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:07 smithi115 ceph-mon[35364]: pgmap v10691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:07 smithi181 ceph-mon[35869]: pgmap v10691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:07 smithi097 ceph-mon[32231]: pgmap v10691: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:09.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:09 smithi115 ceph-mon[35364]: pgmap v10692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:09 smithi181 ceph-mon[35869]: pgmap v10692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:09 smithi097 ceph-mon[32231]: pgmap v10692: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:10.389+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:18:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:10.390+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:18:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:10.390+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:18:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:10.390+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:18:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:10.390+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:11 smithi115 ceph-mon[35364]: pgmap v10693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:11.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19128.135123 secs 2024-07-26T13:18:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:11 smithi181 ceph-mon[35869]: pgmap v10693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19128.135123 secs 2024-07-26T13:18:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:11 smithi097 ceph-mon[32231]: pgmap v10693: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19128.135123 secs 2024-07-26T13:18:13.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:13 smithi115 ceph-mon[35364]: pgmap v10694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:13 smithi181 ceph-mon[35869]: pgmap v10694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:13 smithi097 ceph-mon[32231]: pgmap v10694: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:15.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:15 smithi115 ceph-mon[35364]: pgmap v10695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:15 smithi181 ceph-mon[35869]: pgmap v10695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:15 smithi097 ceph-mon[32231]: pgmap v10695: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:16.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19133.135309 secs 2024-07-26T13:18:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19133.135309 secs 2024-07-26T13:18:16.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19133.135309 secs 2024-07-26T13:18:17.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:17 smithi115 ceph-mon[35364]: pgmap v10696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:17 smithi181 ceph-mon[35869]: pgmap v10696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:17.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:17 smithi097 ceph-mon[32231]: pgmap v10696: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:19 smithi115 ceph-mon[35364]: pgmap v10697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:19 smithi181 ceph-mon[35869]: pgmap v10697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:19.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:19 smithi097 ceph-mon[32231]: pgmap v10697: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:21 smithi115 ceph-mon[35364]: pgmap v10698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19138.135453 secs 2024-07-26T13:18:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:21 smithi181 ceph-mon[35869]: pgmap v10698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19138.135453 secs 2024-07-26T13:18:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:21 smithi097 ceph-mon[32231]: pgmap v10698: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:21.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19138.135453 secs 2024-07-26T13:18:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:23 smithi115 ceph-mon[35364]: pgmap v10699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:23 smithi181 ceph-mon[35869]: pgmap v10699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:23 smithi097 ceph-mon[32231]: pgmap v10699: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:25.394+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:18:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:25.395+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:18:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:25.395+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:18:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:25.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:18:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:25.395+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:25 smithi115 ceph-mon[35364]: pgmap v10700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:25 smithi181 ceph-mon[35869]: pgmap v10700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:25 smithi097 ceph-mon[32231]: pgmap v10700: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19143.135592 secs 2024-07-26T13:18:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19143.135592 secs 2024-07-26T13:18:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19143.135592 secs 2024-07-26T13:18:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:27 smithi115 ceph-mon[35364]: pgmap v10701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:27 smithi181 ceph-mon[35869]: pgmap v10701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:27 smithi097 ceph-mon[32231]: pgmap v10701: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:29 smithi115 ceph-mon[35364]: pgmap v10702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:29 smithi181 ceph-mon[35869]: pgmap v10702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:29 smithi097 ceph-mon[32231]: pgmap v10702: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:31 smithi115 ceph-mon[35364]: pgmap v10703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19148.135710 secs 2024-07-26T13:18:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:31 smithi181 ceph-mon[35869]: pgmap v10703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19148.135710 secs 2024-07-26T13:18:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:31 smithi097 ceph-mon[32231]: pgmap v10703: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:31.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19148.135710 secs 2024-07-26T13:18:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:33 smithi115 ceph-mon[35364]: pgmap v10704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:33 smithi181 ceph-mon[35869]: pgmap v10704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:33 smithi097 ceph-mon[32231]: pgmap v10704: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:35 smithi115 ceph-mon[35364]: pgmap v10705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:35 smithi181 ceph-mon[35869]: pgmap v10705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:35 smithi097 ceph-mon[32231]: pgmap v10705: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19153.135807 secs 2024-07-26T13:18:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19153.135807 secs 2024-07-26T13:18:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19153.135807 secs 2024-07-26T13:18:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:37 smithi097 ceph-mon[32231]: pgmap v10706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:37 smithi115 ceph-mon[35364]: pgmap v10706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:37 smithi181 ceph-mon[35869]: pgmap v10706: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:39 smithi097 ceph-mon[32231]: pgmap v10707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:39.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:39 smithi115 ceph-mon[35364]: pgmap v10707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:39 smithi181 ceph-mon[35869]: pgmap v10707: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:40.402+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:18:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:40.403+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:18:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:40.403+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:18:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:40.403+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:18:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:40.403+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:41 smithi097 ceph-mon[32231]: pgmap v10708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19158.135967 secs 2024-07-26T13:18:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:41 smithi115 ceph-mon[35364]: pgmap v10708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19158.135967 secs 2024-07-26T13:18:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:41 smithi181 ceph-mon[35869]: pgmap v10708: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19158.135967 secs 2024-07-26T13:18:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:43 smithi115 ceph-mon[35364]: pgmap v10709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:43.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:43 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:43 smithi181 ceph-mon[35869]: pgmap v10709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:43 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:43 smithi097 ceph-mon[32231]: pgmap v10709: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:43.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:43 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:44.725 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:18:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:18:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:18:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:44.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:44.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:18:45.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:45 smithi181 ceph-mon[35869]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T13:18:45.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:45 smithi181 ceph-mon[35869]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T13:18:45.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:45 smithi181 ceph-mon[35869]: pgmap v10710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:18:45.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:45 smithi181 ceph-mon[35869]: pgmap v10711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:18:45.635 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19163.136117 secs 2024-07-26T13:18:45.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:45 smithi097 ceph-mon[32231]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T13:18:45.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:45 smithi097 ceph-mon[32231]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T13:18:45.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:45 smithi097 ceph-mon[32231]: pgmap v10710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:18:45.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:45 smithi097 ceph-mon[32231]: pgmap v10711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:18:45.653 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19163.136117 secs 2024-07-26T13:18:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:45 smithi115 ceph-mon[35364]: Adjusting osd_memory_target on smithi181 to 176.4M 2024-07-26T13:18:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:45 smithi115 ceph-mon[35364]: Unable to set osd_memory_target on smithi181 to 185013964: error parsing value: Value '185013964' is below minimum 939524096 2024-07-26T13:18:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:45 smithi115 ceph-mon[35364]: pgmap v10710: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:18:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:45 smithi115 ceph-mon[35364]: pgmap v10711: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:18:45.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19163.136117 secs 2024-07-26T13:18:47.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:47 smithi115 ceph-mon[35364]: pgmap v10712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:18:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:47 smithi181 ceph-mon[35869]: pgmap v10712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:18:47.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:47 smithi097 ceph-mon[32231]: pgmap v10712: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:18:49.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:49 smithi115 ceph-mon[35364]: pgmap v10713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:49 smithi181 ceph-mon[35869]: pgmap v10713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:49.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:49 smithi097 ceph-mon[32231]: pgmap v10713: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:50.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19168.136299 secs 2024-07-26T13:18:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19168.136299 secs 2024-07-26T13:18:50.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19168.136299 secs 2024-07-26T13:18:51.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:51 smithi115 ceph-mon[35364]: pgmap v10714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:51 smithi181 ceph-mon[35869]: pgmap v10714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:51.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:51 smithi097 ceph-mon[32231]: pgmap v10714: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:53.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:53 smithi115 ceph-mon[35364]: pgmap v10715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:53 smithi181 ceph-mon[35869]: pgmap v10715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:53.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:53 smithi097 ceph-mon[32231]: pgmap v10715: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:55.407+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:18:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:55.407+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:18:55.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:55.407+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:18:55.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:55.407+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:18:55.409 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:18:55.407+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:55 smithi115 ceph-mon[35364]: pgmap v10716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:55.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19173.136464 secs 2024-07-26T13:18:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:55 smithi181 ceph-mon[35869]: pgmap v10716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19173.136464 secs 2024-07-26T13:18:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:55 smithi097 ceph-mon[32231]: pgmap v10716: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:55.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19173.136464 secs 2024-07-26T13:18:56.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:56 smithi115 ceph-mon[35364]: pgmap v10717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:56 smithi181 ceph-mon[35869]: pgmap v10717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:56.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:56 smithi097 ceph-mon[32231]: pgmap v10717: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:58.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:18:58 smithi115 ceph-mon[35364]: pgmap v10718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:18:58 smithi181 ceph-mon[35869]: pgmap v10718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:58.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:18:58 smithi097 ceph-mon[32231]: pgmap v10718: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:00 smithi115 ceph-mon[35364]: pgmap v10719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:00.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19178.136562 secs 2024-07-26T13:19:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:00 smithi181 ceph-mon[35869]: pgmap v10719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19178.136562 secs 2024-07-26T13:19:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:00 smithi097 ceph-mon[32231]: pgmap v10719: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:00.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19178.136562 secs 2024-07-26T13:19:02.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:02 smithi115 ceph-mon[35364]: pgmap v10720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:02 smithi181 ceph-mon[35869]: pgmap v10720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:02.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:02 smithi097 ceph-mon[32231]: pgmap v10720: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:04.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:04 smithi115 ceph-mon[35364]: pgmap v10721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:04 smithi181 ceph-mon[35869]: pgmap v10721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:04.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:04 smithi097 ceph-mon[32231]: pgmap v10721: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:05.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19183.136699 secs 2024-07-26T13:19:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19183.136699 secs 2024-07-26T13:19:05.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19183.136699 secs 2024-07-26T13:19:06.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:06 smithi115 ceph-mon[35364]: pgmap v10722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:06 smithi181 ceph-mon[35869]: pgmap v10722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:06.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:06 smithi097 ceph-mon[32231]: pgmap v10722: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:08 smithi181 ceph-mon[35869]: pgmap v10723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:08.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:08 smithi097 ceph-mon[32231]: pgmap v10723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:08.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:08 smithi115 ceph-mon[35364]: pgmap v10723: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.308+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:19:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.308+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:19:10.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.309+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:19:10.310 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.309+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:19:10.410 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.408+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:19:10.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.410+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:19:10.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.410+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:19:10.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.410+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:19:10.411 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:10.410+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:10 smithi097 ceph-mon[32231]: pgmap v10724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:10.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19188.136840 secs 2024-07-26T13:19:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:10 smithi115 ceph-mon[35364]: pgmap v10724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:10.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19188.136840 secs 2024-07-26T13:19:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:10 smithi181 ceph-mon[35869]: pgmap v10724: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19188.136840 secs 2024-07-26T13:19:12.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:12 smithi097 ceph-mon[32231]: pgmap v10725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:12.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:12 smithi115 ceph-mon[35364]: pgmap v10725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:12 smithi181 ceph-mon[35869]: pgmap v10725: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:14 smithi097 ceph-mon[32231]: pgmap v10726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:14.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:14 smithi115 ceph-mon[35364]: pgmap v10726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:14 smithi181 ceph-mon[35869]: pgmap v10726: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:15.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19193.137011 secs 2024-07-26T13:19:15.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19193.137011 secs 2024-07-26T13:19:16.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19193.137011 secs 2024-07-26T13:19:16.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:16 smithi115 ceph-mon[35364]: pgmap v10727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:16 smithi181 ceph-mon[35869]: pgmap v10727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:17.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:16 smithi097 ceph-mon[32231]: pgmap v10727: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:18.777 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:18 smithi115 ceph-mon[35364]: pgmap v10728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:18 smithi181 ceph-mon[35869]: pgmap v10728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:19.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:18 smithi097 ceph-mon[32231]: pgmap v10728: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:20 smithi115 ceph-mon[35364]: pgmap v10729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:20.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19198.137160 secs 2024-07-26T13:19:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:20 smithi181 ceph-mon[35869]: pgmap v10729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:20.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19198.137160 secs 2024-07-26T13:19:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:20 smithi097 ceph-mon[32231]: pgmap v10729: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:21.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19198.137160 secs 2024-07-26T13:19:22.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:22 smithi115 ceph-mon[35364]: pgmap v10730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:22 smithi181 ceph-mon[35869]: pgmap v10730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:23.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:22 smithi097 ceph-mon[32231]: pgmap v10730: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:24.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:24 smithi115 ceph-mon[35364]: pgmap v10731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:24.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:24 smithi181 ceph-mon[35869]: pgmap v10731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:25.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:24 smithi097 ceph-mon[32231]: pgmap v10731: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:25.413+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:19:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:25.414+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:19:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:25.414+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:19:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:25.414+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:19:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:25.415+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:25.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:25 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19203.137330 secs 2024-07-26T13:19:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:25 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19203.137330 secs 2024-07-26T13:19:26.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:25 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19203.137330 secs 2024-07-26T13:19:26.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:26 smithi115 ceph-mon[35364]: pgmap v10732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:26 smithi181 ceph-mon[35869]: pgmap v10732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:27.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:26 smithi097 ceph-mon[32231]: pgmap v10732: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:28.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:28 smithi115 ceph-mon[35364]: pgmap v10733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:28 smithi181 ceph-mon[35869]: pgmap v10733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:29.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:28 smithi097 ceph-mon[32231]: pgmap v10733: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:30 smithi115 ceph-mon[35364]: pgmap v10734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:30.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19208.137430 secs 2024-07-26T13:19:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:30 smithi181 ceph-mon[35869]: pgmap v10734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:30.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19208.137430 secs 2024-07-26T13:19:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:30 smithi097 ceph-mon[32231]: pgmap v10734: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:31.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19208.137430 secs 2024-07-26T13:19:32.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:32 smithi115 ceph-mon[35364]: pgmap v10735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:32 smithi181 ceph-mon[35869]: pgmap v10735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:33.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:32 smithi097 ceph-mon[32231]: pgmap v10735: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:34.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:34 smithi115 ceph-mon[35364]: pgmap v10736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:34.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:34 smithi181 ceph-mon[35869]: pgmap v10736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:35.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:34 smithi097 ceph-mon[32231]: pgmap v10736: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:35.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19213.137567 secs 2024-07-26T13:19:35.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19213.137567 secs 2024-07-26T13:19:36.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19213.137567 secs 2024-07-26T13:19:36.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:36 smithi115 ceph-mon[35364]: pgmap v10737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:36 smithi181 ceph-mon[35869]: pgmap v10737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:37.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:36 smithi097 ceph-mon[32231]: pgmap v10737: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:38.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:38 smithi115 ceph-mon[35364]: pgmap v10738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:38 smithi181 ceph-mon[35869]: pgmap v10738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:39.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:38 smithi097 ceph-mon[32231]: pgmap v10738: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:40.420+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:19:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:40.421+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:19:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:40.421+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:19:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:40.421+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:19:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:40.421+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:40 smithi115 ceph-mon[35364]: pgmap v10739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19218.137719 secs 2024-07-26T13:19:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:40 smithi181 ceph-mon[35869]: pgmap v10739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19218.137719 secs 2024-07-26T13:19:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:40 smithi097 ceph-mon[32231]: pgmap v10739: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:41.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19218.137719 secs 2024-07-26T13:19:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:42 smithi115 ceph-mon[35364]: pgmap v10740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:42 smithi181 ceph-mon[35869]: pgmap v10740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:42 smithi097 ceph-mon[32231]: pgmap v10740: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:44 smithi115 ceph-mon[35364]: pgmap v10741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:44 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:44 smithi181 ceph-mon[35869]: pgmap v10741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:44.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:44 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:44 smithi097 ceph-mon[32231]: pgmap v10741: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:44 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:45.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19223.137852 secs 2024-07-26T13:19:45.940 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:45.941 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:45.941 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:45 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19223.137852 secs 2024-07-26T13:19:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:45 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19223.137852 secs 2024-07-26T13:19:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:45.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:45 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:19:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:46 smithi115 ceph-mon[35364]: pgmap v10742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:46 smithi115 ceph-mon[35364]: pgmap v10743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:46 smithi181 ceph-mon[35869]: pgmap v10742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:46 smithi181 ceph-mon[35869]: pgmap v10743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:46 smithi097 ceph-mon[32231]: pgmap v10742: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:46 smithi097 ceph-mon[32231]: pgmap v10743: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:48.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:48 smithi115 ceph-mon[35364]: pgmap v10744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:48 smithi181 ceph-mon[35869]: pgmap v10744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:49.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:48 smithi097 ceph-mon[32231]: pgmap v10744: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:50 smithi115 ceph-mon[35364]: pgmap v10745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19228.137994 secs 2024-07-26T13:19:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:50 smithi181 ceph-mon[35869]: pgmap v10745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19228.137994 secs 2024-07-26T13:19:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:50 smithi097 ceph-mon[32231]: pgmap v10745: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19228.137994 secs 2024-07-26T13:19:52.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:52 smithi115 ceph-mon[35364]: pgmap v10746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:52 smithi181 ceph-mon[35869]: pgmap v10746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:53.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:52 smithi097 ceph-mon[32231]: pgmap v10746: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:54.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:54 smithi115 ceph-mon[35364]: pgmap v10747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:54 smithi181 ceph-mon[35869]: pgmap v10747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:55.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:54 smithi097 ceph-mon[32231]: pgmap v10747: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.317+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:19:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.317+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:19:55.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.318+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:19:55.319 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.318+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:19:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.422+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:19:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.423+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:19:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.423+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:19:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.423+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:19:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:19:55.423+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:55.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:55 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19233.138153 secs 2024-07-26T13:19:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:55 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19233.138153 secs 2024-07-26T13:19:56.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:55 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19233.138153 secs 2024-07-26T13:19:56.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:56 smithi115 ceph-mon[35364]: pgmap v10748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:19:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:56 smithi181 ceph-mon[35869]: pgmap v10748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:19:57.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:56 smithi097 ceph-mon[32231]: pgmap v10748: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:19:58.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:19:58 smithi115 ceph-mon[35364]: pgmap v10749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:58.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:19:58 smithi181 ceph-mon[35869]: pgmap v10749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:59.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:19:58 smithi097 ceph-mon[32231]: pgmap v10749: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:00 smithi115 ceph-mon[35364]: pgmap v10750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:00 smithi115 ceph-mon[35364]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:00.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:00 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19238.138345 secs 2024-07-26T13:20:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:00 smithi181 ceph-mon[35869]: pgmap v10750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:00 smithi181 ceph-mon[35869]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:00 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19238.138345 secs 2024-07-26T13:20:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:00 smithi097 ceph-mon[32231]: pgmap v10750: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:00 smithi097 ceph-mon[32231]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:01.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:00 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19238.138345 secs 2024-07-26T13:20:02.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:02 smithi115 ceph-mon[35364]: pgmap v10751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:02 smithi181 ceph-mon[35869]: pgmap v10751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:03.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:02 smithi097 ceph-mon[32231]: pgmap v10751: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:04.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:04 smithi181 ceph-mon[35869]: pgmap v10752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:05.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:04 smithi097 ceph-mon[32231]: pgmap v10752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:05.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:04 smithi115 ceph-mon[35364]: pgmap v10752: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:06.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:05 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19243.138480 secs 2024-07-26T13:20:06.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:05 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19243.138480 secs 2024-07-26T13:20:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:05 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19243.138480 secs 2024-07-26T13:20:07.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:06 smithi097 ceph-mon[32231]: pgmap v10753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:07.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:06 smithi115 ceph-mon[35364]: pgmap v10753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:06 smithi181 ceph-mon[35869]: pgmap v10753: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:09.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:08 smithi097 ceph-mon[32231]: pgmap v10754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:09.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:08 smithi115 ceph-mon[35364]: pgmap v10754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:08 smithi181 ceph-mon[35869]: pgmap v10754: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:10.426 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:10.425+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:10.426+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:20:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:10.426+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:20:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:10.426+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:10.426+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:10 smithi115 ceph-mon[35364]: pgmap v10755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:20:11.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:10 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19248.138622 secs 2024-07-26T13:20:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:10 smithi181 ceph-mon[35869]: pgmap v10755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:20:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:10 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19248.138622 secs 2024-07-26T13:20:11.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:10 smithi097 ceph-mon[32231]: pgmap v10755: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:20:11.254 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:10 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19248.138622 secs 2024-07-26T13:20:13.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:12 smithi115 ceph-mon[35364]: pgmap v10756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:12 smithi181 ceph-mon[35869]: pgmap v10756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:13.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:12 smithi097 ceph-mon[32231]: pgmap v10756: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:15.035 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:14 smithi181 ceph-mon[35869]: pgmap v10757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:20:15.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:14 smithi097 ceph-mon[32231]: pgmap v10757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:20:15.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:14 smithi115 ceph-mon[35364]: pgmap v10757: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:20:16.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:15 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19253.138752 secs 2024-07-26T13:20:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:15 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19253.138752 secs 2024-07-26T13:20:16.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:15 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19253.138752 secs 2024-07-26T13:20:17.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:16 smithi115 ceph-mon[35364]: pgmap v10758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:20:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:16 smithi181 ceph-mon[35869]: pgmap v10758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:20:17.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:16 smithi097 ceph-mon[32231]: pgmap v10758: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:20:19.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:18 smithi097 ceph-mon[32231]: pgmap v10759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:20:19.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:18 smithi115 ceph-mon[35364]: pgmap v10759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:20:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:18 smithi181 ceph-mon[35869]: pgmap v10759: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:20:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:20 smithi097 ceph-mon[32231]: pgmap v10760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:20:21.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:20 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19258.138900 secs 2024-07-26T13:20:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:20 smithi115 ceph-mon[35364]: pgmap v10760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:20:21.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:20 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19258.138900 secs 2024-07-26T13:20:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:20 smithi181 ceph-mon[35869]: pgmap v10760: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:20:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:20 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19258.138900 secs 2024-07-26T13:20:23.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:23 smithi115 ceph-mon[35364]: pgmap v10761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:23 smithi181 ceph-mon[35869]: pgmap v10761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:23.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:23 smithi097 ceph-mon[32231]: pgmap v10761: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:25.429+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:25.431+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:20:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:25.431+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:20:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:25.431+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:25.431+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:25.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:25 smithi115 ceph-mon[35364]: pgmap v10762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:25 smithi181 ceph-mon[35869]: pgmap v10762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:25.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:25 smithi097 ceph-mon[32231]: pgmap v10762: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:26.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19263.139043 secs 2024-07-26T13:20:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19263.139043 secs 2024-07-26T13:20:26.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19263.139043 secs 2024-07-26T13:20:27.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:27 smithi115 ceph-mon[35364]: pgmap v10763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:27 smithi181 ceph-mon[35869]: pgmap v10763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:27.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:27 smithi097 ceph-mon[32231]: pgmap v10763: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:29.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:29 smithi115 ceph-mon[35364]: pgmap v10764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:29 smithi181 ceph-mon[35869]: pgmap v10764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:29.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:29 smithi097 ceph-mon[32231]: pgmap v10764: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:31 smithi115 ceph-mon[35364]: pgmap v10765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:31.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:31 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19268.139178 secs 2024-07-26T13:20:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:31 smithi181 ceph-mon[35869]: pgmap v10765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:31 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19268.139178 secs 2024-07-26T13:20:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:31 smithi097 ceph-mon[32231]: pgmap v10765: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:31.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:31 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19268.139178 secs 2024-07-26T13:20:33.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:33 smithi115 ceph-mon[35364]: pgmap v10766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:33 smithi181 ceph-mon[35869]: pgmap v10766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:33.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:33 smithi097 ceph-mon[32231]: pgmap v10766: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:35.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:35 smithi115 ceph-mon[35364]: pgmap v10767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:35 smithi181 ceph-mon[35869]: pgmap v10767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:35.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:35 smithi097 ceph-mon[32231]: pgmap v10767: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:36.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:36 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19273.139344 secs 2024-07-26T13:20:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:36 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19273.139344 secs 2024-07-26T13:20:36.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:36 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19273.139344 secs 2024-07-26T13:20:37.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:37 smithi115 ceph-mon[35364]: pgmap v10768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:37 smithi181 ceph-mon[35869]: pgmap v10768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:37.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:37 smithi097 ceph-mon[32231]: pgmap v10768: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:39.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:39 smithi115 ceph-mon[35364]: pgmap v10769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:39 smithi181 ceph-mon[35869]: pgmap v10769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:39.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:39 smithi097 ceph-mon[32231]: pgmap v10769: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:40.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:40.439+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:40.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:40.441+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:20:40.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:40.441+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:20:40.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:40.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:40.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:40.441+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:41 smithi115 ceph-mon[35364]: pgmap v10770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:41.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:41 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19278.139478 secs 2024-07-26T13:20:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:41 smithi181 ceph-mon[35869]: pgmap v10770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:41 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19278.139478 secs 2024-07-26T13:20:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:41 smithi097 ceph-mon[32231]: pgmap v10770: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:41.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:41 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19278.139478 secs 2024-07-26T13:20:43.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:43 smithi115 ceph-mon[35364]: pgmap v10771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:43 smithi181 ceph-mon[35869]: pgmap v10771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:43.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:43 smithi097 ceph-mon[32231]: pgmap v10771: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:45.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:45 smithi115 ceph-mon[35364]: pgmap v10772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:45 smithi181 ceph-mon[35869]: pgmap v10772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:45.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:45 smithi097 ceph-mon[32231]: pgmap v10772: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:46 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19283.139576 secs 2024-07-26T13:20:46.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:46 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:46 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19283.139576 secs 2024-07-26T13:20:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:46 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:46 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19283.139576 secs 2024-07-26T13:20:46.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:46 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:47 smithi115 ceph-mon[35364]: pgmap v10773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:47 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:47 smithi181 ceph-mon[35869]: pgmap v10773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:47 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:47 smithi097 ceph-mon[32231]: pgmap v10773: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:47.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:47 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:48.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:48.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:20:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:49 smithi115 ceph-mon[35364]: pgmap v10774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:49 smithi115 ceph-mon[35364]: pgmap v10775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:49.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:49 smithi115 ceph-mon[35364]: pgmap v10776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:20:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:49 smithi181 ceph-mon[35869]: pgmap v10774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:49 smithi181 ceph-mon[35869]: pgmap v10775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:49 smithi181 ceph-mon[35869]: pgmap v10776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:20:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:49 smithi097 ceph-mon[32231]: pgmap v10774: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:49 smithi097 ceph-mon[32231]: pgmap v10775: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:49.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:49 smithi097 ceph-mon[32231]: pgmap v10776: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:20:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:51 smithi115 ceph-mon[35364]: pgmap v10777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:51.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:51 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19288.139676 secs 2024-07-26T13:20:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:51 smithi181 ceph-mon[35869]: pgmap v10777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:51 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19288.139676 secs 2024-07-26T13:20:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:51 smithi097 ceph-mon[32231]: pgmap v10777: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:51.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:51 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19288.139676 secs 2024-07-26T13:20:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:53 smithi115 ceph-mon[35364]: pgmap v10778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:20:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:53 smithi181 ceph-mon[35869]: pgmap v10778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:20:53.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:53 smithi097 ceph-mon[32231]: pgmap v10778: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:20:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:55.443+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:20:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:55.444+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:20:55.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:55.444+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:20:55.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:55.444+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:20:55.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:20:55.445+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:55 smithi115 ceph-mon[35364]: pgmap v10779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:20:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:55 smithi181 ceph-mon[35869]: pgmap v10779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:20:55.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:55 smithi097 ceph-mon[32231]: pgmap v10779: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:20:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19293.139762 secs 2024-07-26T13:20:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19293.139762 secs 2024-07-26T13:20:56.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19293.139762 secs 2024-07-26T13:20:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:57 smithi115 ceph-mon[35364]: pgmap v10780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:57 smithi181 ceph-mon[35869]: pgmap v10780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:57.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:57 smithi097 ceph-mon[32231]: pgmap v10780: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:59.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:20:59 smithi115 ceph-mon[35364]: pgmap v10781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:20:59 smithi181 ceph-mon[35869]: pgmap v10781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:59.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:20:59 smithi097 ceph-mon[32231]: pgmap v10781: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:01 smithi115 ceph-mon[35364]: pgmap v10782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19298.139873 secs 2024-07-26T13:21:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:01 smithi181 ceph-mon[35869]: pgmap v10782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19298.139873 secs 2024-07-26T13:21:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:01 smithi097 ceph-mon[32231]: pgmap v10782: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19298.139873 secs 2024-07-26T13:21:03.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:03 smithi115 ceph-mon[35364]: pgmap v10783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:03.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:03 smithi181 ceph-mon[35869]: pgmap v10783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:03.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:03 smithi097 ceph-mon[32231]: pgmap v10783: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:05.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:05 smithi115 ceph-mon[35364]: pgmap v10784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:05 smithi181 ceph-mon[35869]: pgmap v10784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:05.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:05 smithi097 ceph-mon[32231]: pgmap v10784: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19303.140014 secs 2024-07-26T13:21:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19303.140014 secs 2024-07-26T13:21:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19303.140014 secs 2024-07-26T13:21:07.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:07 smithi115 ceph-mon[35364]: pgmap v10785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:07 smithi181 ceph-mon[35869]: pgmap v10785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:07.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:07 smithi097 ceph-mon[32231]: pgmap v10785: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:09 smithi181 ceph-mon[35869]: pgmap v10786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:09.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:09 smithi097 ceph-mon[32231]: pgmap v10786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:09.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:09 smithi115 ceph-mon[35364]: pgmap v10786: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:10.446+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:10.448+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:21:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:10.448+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:21:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:10.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:10.448+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:11 smithi181 ceph-mon[35869]: pgmap v10787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19308.140154 secs 2024-07-26T13:21:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:11 smithi097 ceph-mon[32231]: pgmap v10787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19308.140154 secs 2024-07-26T13:21:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:11 smithi115 ceph-mon[35364]: pgmap v10787: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:11.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19308.140154 secs 2024-07-26T13:21:13.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:13 smithi097 ceph-mon[32231]: pgmap v10788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:13.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:13 smithi115 ceph-mon[35364]: pgmap v10788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:13 smithi181 ceph-mon[35869]: pgmap v10788: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:15.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:15 smithi097 ceph-mon[32231]: pgmap v10789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:15.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:15 smithi181 ceph-mon[35869]: pgmap v10789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:15.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:15 smithi115 ceph-mon[35364]: pgmap v10789: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:16.683 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19313.140298 secs 2024-07-26T13:21:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19313.140298 secs 2024-07-26T13:21:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19313.140298 secs 2024-07-26T13:21:17.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:17 smithi115 ceph-mon[35364]: pgmap v10790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:17 smithi181 ceph-mon[35869]: pgmap v10790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:17.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:17 smithi097 ceph-mon[32231]: pgmap v10790: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:19.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:19 smithi115 ceph-mon[35364]: pgmap v10791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:19 smithi181 ceph-mon[35869]: pgmap v10791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:19.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:19 smithi097 ceph-mon[32231]: pgmap v10791: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:21 smithi115 ceph-mon[35364]: pgmap v10792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19318.140437 secs 2024-07-26T13:21:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:21 smithi181 ceph-mon[35869]: pgmap v10792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19318.140437 secs 2024-07-26T13:21:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:21 smithi097 ceph-mon[32231]: pgmap v10792: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19318.140437 secs 2024-07-26T13:21:23.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:23 smithi115 ceph-mon[35364]: pgmap v10793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:23 smithi181 ceph-mon[35869]: pgmap v10793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:23.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:23 smithi097 ceph-mon[32231]: pgmap v10793: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:25.451+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:25.452+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:21:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:25.452+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:21:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:25.452+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:25.452+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:25.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:25 smithi115 ceph-mon[35364]: pgmap v10794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:25 smithi181 ceph-mon[35869]: pgmap v10794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:25.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:25 smithi097 ceph-mon[32231]: pgmap v10794: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19323.140593 secs 2024-07-26T13:21:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19323.140593 secs 2024-07-26T13:21:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19323.140593 secs 2024-07-26T13:21:27.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:27 smithi115 ceph-mon[35364]: pgmap v10795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:27 smithi181 ceph-mon[35869]: pgmap v10795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:27.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:27 smithi097 ceph-mon[32231]: pgmap v10795: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:29 smithi115 ceph-mon[35364]: pgmap v10796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:29 smithi181 ceph-mon[35869]: pgmap v10796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:29.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:29 smithi097 ceph-mon[32231]: pgmap v10796: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19328.140773 secs 2024-07-26T13:21:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19328.140773 secs 2024-07-26T13:21:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19328.140773 secs 2024-07-26T13:21:31.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:31 smithi115 ceph-mon[35364]: pgmap v10797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:31 smithi181 ceph-mon[35869]: pgmap v10797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:31.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:31 smithi097 ceph-mon[32231]: pgmap v10797: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:33.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:33 smithi115 ceph-mon[35364]: pgmap v10798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:33 smithi181 ceph-mon[35869]: pgmap v10798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:33.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:33 smithi097 ceph-mon[32231]: pgmap v10798: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:35 smithi115 ceph-mon[35364]: pgmap v10799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19333.140871 secs 2024-07-26T13:21:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:35 smithi181 ceph-mon[35869]: pgmap v10799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19333.140871 secs 2024-07-26T13:21:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:35 smithi097 ceph-mon[32231]: pgmap v10799: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19333.140871 secs 2024-07-26T13:21:37.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:37 smithi115 ceph-mon[35364]: pgmap v10800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:37 smithi181 ceph-mon[35869]: pgmap v10800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:37.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:37 smithi097 ceph-mon[32231]: pgmap v10800: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:38 smithi115 ceph-mon[35364]: pgmap v10801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:38 smithi181 ceph-mon[35869]: pgmap v10801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:38 smithi097 ceph-mon[32231]: pgmap v10801: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:40.461+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:40.462+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:21:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:40.462+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:21:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:40.462+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:40.462+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:40 smithi097 ceph-mon[32231]: pgmap v10802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19338.141027 secs 2024-07-26T13:21:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:40 smithi115 ceph-mon[35364]: pgmap v10802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:40.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19338.141027 secs 2024-07-26T13:21:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:40 smithi181 ceph-mon[35869]: pgmap v10802: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19338.141027 secs 2024-07-26T13:21:42.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:42 smithi115 ceph-mon[35364]: pgmap v10803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:42 smithi181 ceph-mon[35869]: pgmap v10803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:43.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:42 smithi097 ceph-mon[32231]: pgmap v10803: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:44.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:44 smithi115 ceph-mon[35364]: pgmap v10804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:44 smithi181 ceph-mon[35869]: pgmap v10804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:45.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:44 smithi097 ceph-mon[32231]: pgmap v10804: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:45.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:45 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19343.141187 secs 2024-07-26T13:21:45.903 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:45 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19343.141187 secs 2024-07-26T13:21:45.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:45 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19343.141187 secs 2024-07-26T13:21:46.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:46 smithi115 ceph-mon[35364]: pgmap v10805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:46 smithi181 ceph-mon[35869]: pgmap v10805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:47.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:46 smithi097 ceph-mon[32231]: pgmap v10805: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:48.795 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:48 smithi097 ceph-mon[32231]: pgmap v10806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:48.795 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:48 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:48.862 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:48 smithi181 ceph-mon[35869]: pgmap v10806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:48.862 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:48 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:48.962 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:48 smithi115 ceph-mon[35364]: pgmap v10806: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:48.962 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:48 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:50 smithi115 ceph-mon[35364]: pgmap v10807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:50.974 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:50 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19348.141327 secs 2024-07-26T13:21:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:50 smithi181 ceph-mon[35869]: pgmap v10807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:50 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19348.141327 secs 2024-07-26T13:21:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:50 smithi097 ceph-mon[32231]: pgmap v10807: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:51.001 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:50 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19348.141327 secs 2024-07-26T13:21:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.224 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:51 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:51 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.250 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:52.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:51 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:53.405 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:53 smithi097 ceph-mon[32231]: pgmap v10808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:53.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:53 smithi115 ceph-mon[35364]: pgmap v10808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:53 smithi181 ceph-mon[35869]: pgmap v10808: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:55.251 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:55 smithi097 ceph-mon[32231]: pgmap v10809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:55.464+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:55.465+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:21:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:55.466+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:21:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:55.466+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:21:55.466+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:55.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:55 smithi115 ceph-mon[35364]: pgmap v10809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:55 smithi181 ceph-mon[35869]: pgmap v10809: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:56.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:56 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19353.141468 secs 2024-07-26T13:21:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:56 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19353.141468 secs 2024-07-26T13:21:56.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:56 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19353.141468 secs 2024-07-26T13:21:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:57 smithi115 ceph-mon[35364]: pgmap v10810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:57.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:57.475 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:57 smithi115 ceph-mon[35364]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:57 smithi181 ceph-mon[35869]: pgmap v10810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:57 smithi181 ceph-mon[35869]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:57.494 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:57 smithi097 ceph-mon[32231]: pgmap v10810: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:57.494 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:57.494 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:57.494 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:57.494 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:57.494 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:57 smithi097 ceph-mon[32231]: from='mgr.14150 172.21.15.97:0/273941148' entity='mgr.x' 2024-07-26T13:21:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:58 smithi115 ceph-mon[35364]: pgmap v10811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:21:58.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:21:58 smithi115 ceph-mon[35364]: pgmap v10812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:21:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:58 smithi181 ceph-mon[35869]: pgmap v10811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:21:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:21:58 smithi181 ceph-mon[35869]: pgmap v10812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:21:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:58 smithi097 ceph-mon[32231]: pgmap v10811: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:21:58.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:21:58 smithi097 ceph-mon[32231]: pgmap v10812: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:00.477 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:00 smithi115 ceph-mon[35364]: pgmap v10813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:00 smithi181 ceph-mon[35869]: pgmap v10813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:00.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:00 smithi097 ceph-mon[32231]: pgmap v10813: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:01.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:01 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19358.141612 secs 2024-07-26T13:22:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:01 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19358.141612 secs 2024-07-26T13:22:01.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:01 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19358.141612 secs 2024-07-26T13:22:02.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:02 smithi115 ceph-mon[35364]: pgmap v10814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:02 smithi181 ceph-mon[35869]: pgmap v10814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:02.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:02 smithi097 ceph-mon[32231]: pgmap v10814: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:04.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:04 smithi115 ceph-mon[35364]: pgmap v10815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:04 smithi181 ceph-mon[35869]: pgmap v10815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:04.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:04 smithi097 ceph-mon[32231]: pgmap v10815: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:22:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:06 smithi115 ceph-mon[35364]: pgmap v10816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:22:06.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:06 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19363.141743 secs 2024-07-26T13:22:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:06 smithi181 ceph-mon[35869]: pgmap v10816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:22:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:06 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19363.141743 secs 2024-07-26T13:22:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:06 smithi097 ceph-mon[32231]: pgmap v10816: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:22:06.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:06 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19363.141743 secs 2024-07-26T13:22:08.474 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:08 smithi115 ceph-mon[35364]: pgmap v10817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:08 smithi181 ceph-mon[35869]: pgmap v10817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:08.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:08 smithi097 ceph-mon[32231]: pgmap v10817: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.343+0000 7ff3e7e00640 1 -- 172.21.15.97:0/530760114 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3c0004740 con 0x7ff3f01061a0 2024-07-26T13:22:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.343+0000 7ff3dcc00640 1 -- 172.21.15.97:0/1934891825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:22:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.343+0000 7ff3de000640 1 -- 172.21.15.97:0/1934891825 <== mon.0 v2:172.21.15.97:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff3bc006500 con 0x55d25070b2f0 2024-07-26T13:22:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.344+0000 7ff3ed600640 1 -- 172.21.15.97:0/530760114 <== mon.0 v2:172.21.15.97:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff3d807f040 con 0x7ff3f01061a0 2024-07-26T13:22:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.468+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:22:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.469+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:22:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.469+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:22:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.469+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:22:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:10.469+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:10 smithi181 ceph-mon[35869]: pgmap v10818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:10.502 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:10 smithi097 ceph-mon[32231]: pgmap v10818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:10.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:10 smithi115 ceph-mon[35364]: pgmap v10818: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:11.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:11 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19368.141880 secs 2024-07-26T13:22:11.726 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:11 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19368.141880 secs 2024-07-26T13:22:11.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:11 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19368.141880 secs 2024-07-26T13:22:12.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:12 smithi097 ceph-mon[32231]: pgmap v10819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:12.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:12 smithi115 ceph-mon[35364]: pgmap v10819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:12 smithi181 ceph-mon[35869]: pgmap v10819: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:14.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:14 smithi115 ceph-mon[35364]: pgmap v10820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:14 smithi181 ceph-mon[35869]: pgmap v10820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:14.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:14 smithi097 ceph-mon[32231]: pgmap v10820: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:16.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:16 smithi181 ceph-mon[35869]: pgmap v10821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:16.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:16 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19373.142009 secs 2024-07-26T13:22:16.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:16 smithi097 ceph-mon[32231]: pgmap v10821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:16.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:16 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19373.142009 secs 2024-07-26T13:22:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:16 smithi115 ceph-mon[35364]: pgmap v10821: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:16.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:16 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19373.142009 secs 2024-07-26T13:22:18.501 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:18 smithi097 ceph-mon[32231]: pgmap v10822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:18 smithi115 ceph-mon[35364]: pgmap v10822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:18 smithi181 ceph-mon[35869]: pgmap v10822: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:20.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:20 smithi115 ceph-mon[35364]: pgmap v10823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:20 smithi181 ceph-mon[35869]: pgmap v10823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:20.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:20 smithi097 ceph-mon[32231]: pgmap v10823: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:21.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:21 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19378.142162 secs 2024-07-26T13:22:21.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:21 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19378.142162 secs 2024-07-26T13:22:21.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:21 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19378.142162 secs 2024-07-26T13:22:22.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:22 smithi115 ceph-mon[35364]: pgmap v10824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:22.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:22 smithi181 ceph-mon[35869]: pgmap v10824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:22.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:22 smithi097 ceph-mon[32231]: pgmap v10824: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:24.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:24 smithi115 ceph-mon[35364]: pgmap v10825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:24 smithi181 ceph-mon[35869]: pgmap v10825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:24.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:24 smithi097 ceph-mon[32231]: pgmap v10825: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:25.471+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:22:25.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:25.472+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:22:25.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:25.472+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:22:25.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:25.472+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:22:25.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:25.472+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:26 smithi115 ceph-mon[35364]: pgmap v10826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:22:26.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:26 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19383.142291 secs 2024-07-26T13:22:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:26 smithi181 ceph-mon[35869]: pgmap v10826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:22:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:26 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19383.142291 secs 2024-07-26T13:22:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:26 smithi097 ceph-mon[32231]: pgmap v10826: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:22:26.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:26 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19383.142291 secs 2024-07-26T13:22:28.610 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:28 smithi181 ceph-mon[35869]: pgmap v10827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:28.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:28 smithi115 ceph-mon[35364]: pgmap v10827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:28.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:28 smithi097 ceph-mon[32231]: pgmap v10827: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:30 smithi115 ceph-mon[35364]: pgmap v10828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:30.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:30 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19388.142435 secs 2024-07-26T13:22:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:30 smithi181 ceph-mon[35869]: pgmap v10828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:30.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:30 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19388.142435 secs 2024-07-26T13:22:30.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:30 smithi097 ceph-mon[32231]: pgmap v10828: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:30.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:30 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19388.142435 secs 2024-07-26T13:22:32.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:32 smithi115 ceph-mon[35364]: pgmap v10829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:22:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:32 smithi181 ceph-mon[35869]: pgmap v10829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:22:32.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:32 smithi097 ceph-mon[32231]: pgmap v10829: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:22:34.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:34 smithi115 ceph-mon[35364]: pgmap v10830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:34 smithi181 ceph-mon[35869]: pgmap v10830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:34.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:34 smithi097 ceph-mon[32231]: pgmap v10830: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:35.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:35 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19393.142569 secs 2024-07-26T13:22:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:35 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19393.142569 secs 2024-07-26T13:22:35.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:35 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19393.142569 secs 2024-07-26T13:22:36.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:36 smithi115 ceph-mon[35364]: pgmap v10831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:36 smithi181 ceph-mon[35869]: pgmap v10831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:36.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:36 smithi097 ceph-mon[32231]: pgmap v10831: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:38.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:38 smithi115 ceph-mon[35364]: pgmap v10832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:38 smithi181 ceph-mon[35869]: pgmap v10832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:38.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:38 smithi097 ceph-mon[32231]: pgmap v10832: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:40.480+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:22:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:40.481+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T13:22:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:40.481+0000 7ff3e7400640 1 -- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 msgr2=0x55d250af15a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T13:22:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:40.481+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:22:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-07-26T13:22:40.481+0000 7ff3e7400640 1 --2- 172.21.15.97:0/1934891825 >> [v2:172.21.15.115:6838/1903638498,v1:172.21.15.115:6839/1903638498] conn(0x55d2506f2d80 0x55d250af15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:40 smithi115 ceph-mon[35364]: pgmap v10833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:40.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:40 smithi115 ceph-mon[35364]: 1 slow requests, 0 included below; oldest blocked for > 19398.142733 secs 2024-07-26T13:22:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:40 smithi181 ceph-mon[35869]: pgmap v10833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:40 smithi181 ceph-mon[35869]: 1 slow requests, 0 included below; oldest blocked for > 19398.142733 secs 2024-07-26T13:22:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:40 smithi097 ceph-mon[32231]: pgmap v10833: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:40.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:40 smithi097 ceph-mon[32231]: 1 slow requests, 0 included below; oldest blocked for > 19398.142733 secs 2024-07-26T13:22:41.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Jul 26 13:22:41 smithi115 ceph-mon[35364]: pgmap v10834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Jul 26 13:22:41 smithi181 ceph-mon[35869]: pgmap v10834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:41.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Jul 26 13:22:41 smithi097 ceph-mon[32231]: pgmap v10834: 97 pgs: 97 active+clean; 211 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:42.535 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-07-26T13:22:42.535 DEBUG:teuthology.task.console_log:Killing console logger for smithi097 2024-07-26T13:22:42.536 DEBUG:teuthology.task.console_log:Killing console logger for smithi115 2024-07-26T13:22:42.536 DEBUG:teuthology.task.console_log:Killing console logger for smithi181 2024-07-26T13:22:42.536 DEBUG:teuthology.exit:Finished running handlers